MaxStocker.com

My Secret List of Toronto Driving Shortcuts

Fri, 04 Mar 2011, 18:47:00 EST

The other day a friend asked me about a driving route that is currently fouled up in a major way by construction, and it got me thinking about my list of driving shortcuts in the city. Now these are time shortcuts, some of them my wife refers to as "long cuts" and longer in distance some of them may be but I promise you that, especially in rush hour, knowing these routes can save you plenty of time.

So without further ado here are some of my secret shortcuts to Toronto driving. "Not so secret any more!", you say and you'd be right, which is why I can't give you all of them. Sorry.

Some General Advice

With the exception of some areas and routes (discussed below) in general Toronto traffic, at least the worst of it, is very rush hour centric. It seems obvious but do remember just because a street or route is bad at one time of day doesn't mean it always is, and in fact you can be free and clear even when it is bad as long as you're going in the "right" direction. The point here being that when driving you shouldn't dismiss a route just because it might be clogged at certain times of the day.

North-South in the Eastern part of the city

The secret route here revolves around one of the best hidden little secrets of Toronto driving, Leslie Street. Leslie Street from Eglinton to the 401 is a driving timesavers gold mine. There aren't many lights in comparison to alternate routes, you can get on and off at convienent spots and the traffic is much lighter than alternates as well. If you're headed north or south anywhere between Yonge and the DVP at pretty much any time of day this is the route you should be on. Even in the worst of rush hour and going with the traffic you will beat Yonge, Bayview, Don Mills or the DVP on Leslie.

From the north end Leslie connects right to the 401 which is useful, at the south end it's a bit trickier but easy enough to get right downtown. Heading south on Leslie when you get to Eglinton turn right. Then get in the left hand lane to make a left turn at the very next light (Brentcliffe Road). You may arrive at this light and discover that the left hand turn lane is backed up for a long, long way. That's okay because I'm telling you this is the longest advance green in Toronto. Turn left onto Brentcliffe and then right onto Wicksteed Avenue and in moments you're at the intersection of McRae and Laird.

From there depending on where you want to go you can make your next move. Heading to Riverdale/The Beaches? Head south on Laird, across Millwood and on into Riverdale. Going downtown? South on Laird to Southvale/Moore Avenue and then head down the Bayview extension all the way to King. Heading uptown? Take McRae to Millwood, Millwood to across Bayview and then cut down to Davisvile Ave across all the way to Yonge and beyond.

South in the Central-Western part of the city

This route isn't as long (or as good) as the one above but I blame the fact that the western half of Toronto just sucks to get around, maybe if the Spadina expressway went where it was supposed to things would be different but the fact remains it's just more tricky and there's less you can do about it. It also suffers from being a one-way only route. It helps if you're going south but can't help you coming north. Nevertheless there is a handy little shortcut from Eglinton/Bathurst to Dupont and Davenport that uses Russell Hill Road.

On an aside I recently became aware that this road is named after one of the most notorious slave owners in Toronto history. There have been a few attempts to rename the street because of this but they have all failed. I think that's pretty shameful to be honest with you, hopefully at some point this will be properly addressed.

Moving on... Russell Hill is tricky to get to from Eglinton, although the best way is probably off of Old Forest Hill Road (which intersects Eglinton at the top of the hill just east of Bathurst. It's relatively easy to get to on St Clair but part of the time savings is the north part so if you can get on it before that's good. (Note that some/all of the parts north of St Clair are two ways, so be careful and don't think it's one way all the way). Once you get south of St Clair you're smooth sailing, with a few twists and turns, until you get dumped north of the railway bridge that crosses Dupont/Spadina right north of where Dupont and Davenport meet up. From there if you're headed downtown just go over to Avenue Road and then down University.

East-West Across the North and West of the City (401 from Avenue to 427 area)

Nope. Sorry, you're outta luck. Nobody has ever found a good route across that part of the city and I'm inclined to believe there just isn't one. The only thing that can ever help you in that area is the 407, but depending on where you are actually going that just takes you way too far out of the way to be effective.

A few months back I took a taxi to the airport at 6am. There was a traffic jam, heading west mind you which is against the main traffic flow, from the Allen Road to the 409. At 6 am! So you're just screwed, leave early, hope for the best and be glad you don't actually live in Etobicoke. If you do live in Etobicoke, especially north Etobicoke, really, you need to move.

East-West on Bloor/Danforth from Yonge to Broadview (and area)

There are two secrets on this route and both will save you a lot of time if you have to go from Broadview/Danforth to Yonge/Bloor at rush hour or if you even only can use one part of these routes from different sources and destinations.

I should mention here that this route is certainly one of the ones where it's not always bad. If you're going at non-peak times, or headed the opposite way to rush hour just stay on Bloor/Danforth and it's great. When you are with rush hour traffic though... it's ridiculous, it's not uncommon for what should be a 10 minute drive to take well over 30 (and sometimes even longer). This route will cut the time down to a better 17-18. So yes, this is longer than things will take at the best of times but when traffic is bad it's a great one.

Rosedale Valley Road. Rosedale Valley Road is your friend on this route, running from the Bayview extension south of Bloor right up to almost Church and Davenport. There can sometimes be a lot of traffic here too, and waiting for the 4 way stop can take some time, don't worry once past that it's non-stop to Bayview and away you go. Plus there is some nice scenery along the way, certainly more picturesque than a view of the back of St Jamestown so it's a win all around.

Once you get to Bayview turn left and head up to the second secret part of the route Pottery Road. Pottery Road will take you across and up the valley to the intersection of Broadview and Mortimer. Mortimer is a great east/west route if you need to keep going that way, and if you need to go south Broadview is okay. Pottery is a great little road for connecting Leaside and North Riverdale in general so you can use it for those purposes as well, much better than anything involving Pape Avenue for sure. The only caveat with Pottery is if you have a manual transmission and you're not always the best of friends with it you might want to give it a pass. Sometimes you can be stuck on the hill waiting for the light, and it's pretty serious hill to be honest with you.

Beaches/South Riverdale East/West to Downtown

Adelaide/Richmond to Eastern Avenue are your friends here. Look the reality is that when you're on any east west street between University and Church traffic, in rush hour, is just going to suck. There's nothing you can do about that, but once you get past there Adelaide (out of downtown) and Richmond (in to downtown) are really fast routes across to Eastern and from there all the way to Leslie if you want. Very few lights and a lot less traffic than any alternatives.

A final word of advice

Whatever you do never take the DVP. The DVP is for tourists and idiots, so unless you're using it at 3am you're taking the wrong route. Because even then it's 50/50 that there will be a traffic jam from 401 to Lawrence in both directions.

I would say the same generally applies for Allen Road, the 401 from 427 to Avenue Road and anywhere on the Gardiner except that sometimes you just have no choice. So if you can at all avoid any of those, I'd do that too.

Google Analytics vs Reality

Thu, 02 Dec 2010, 21:49:00 EST

Like many others I am using Google Analytics on many sites to track traffic, and have generally been happy with it. Recently though I have encountered an issue that is making me seriously reconsider it.

In short the issue is that there appears to be a truly massive gulf between what Google Analytics is reporting and reality.

What is happening in my case is that I have a site that is running Analytics on all it's pages. This site also has a number of "special" content pages, to get a general idea the basic site consists of about 50 "regular" pages and about 750 special ones and the traffic split between is about 40 to 60. At any rate for these special pages I am doing my own logging and tracking because views and interactions and statistics on them is important for me to have close at hand. And here is where I found the discrepency.

Now the basic analytics report for November for the site claims about 25,000 page views. But my tracking and logging for the special pages alone for November shows 44,000 views! That's a huge difference and only worse when you remember I am not even tracking all the pages on the site. Based on what both Analytics reports and what I suspect in terms of the ratio of basic to special pages views, if my recorded numbers are extrapolated to the whole site I get a November total page view count of 73,000.

When I first discovered this I was shocked, Google can't really be losing 50,000 page views can it?

Well following the first rule of good development (strange bugs are your fault) I assumed the problem was in my code and began rexamining exactly what is being logged and when. I reviewed exactly what my code is doing and the specifics of what I am logging. The problem is not at my end. I compared results for a specific special page that I log to Google's and it's clear Google is simply not seeing (or recording) all the requests that I am. In one example I saw a special page that had 18 views by my count over a 2 day period. By Google's count that same page, over the same time period had 6 views. It's a pattern that's fairly consistent across the board, in all cases Google has less page views then I do, sometimes by a small number but in many cases a large gap between them.

So what is happening? Well to cover off what it is not first

I am filtering Analytics and not seeing all traffic. No I checked, I am not doing that.
I am looking at unique page views in Analytics and not totals. Again no and I did check.
I am logging more views because users are refreshing in the middle of a page load (or similar behaviour leading to "false" views). Again no. I log the time and other information about the requests and when I look at the pattern it looks like normal usage. For the example listed above I can see 17 different "users" looked at that page a total of 18 times and these requests are spread out throughout the day.

So that leaves me wondering about bots and users who have disabled either JavaScript or Analytics specifically. I do see some bot requests certainly, again to return the example above there is one confirmed bot in the 18 and another two I think might be. But again let's just say for arguments sake that those three are all bots and that I am missing another three as well (just to be on the safe side, this of course makes the assumption that a full third of the traffic to my site is coming from bots but we'll go with that for now). So that gives us the following.

My total logged requests	18
Subtract user who viewed page twice	-1
Subtract known or suspected bots	-6
My adjusted total views	11
Google Analytics	6

As you can see even with the adjustments which I am fairly sure are too high we are still left with Analytics tracking barely half the actual traffic. And that leaves us with the possibilities that eitherhalf the site users have JavaScript disabled or blocked or Analytics is in some way pretty broken. I don't really know what it can be. The JavaScript or blocking seems to be the most palatable alternative left but I find it difficult to think that a full 50% of the audience for this site would fit into that category. I guess there is a possibility that because of the content there are really bots swarming the site at all times and that is the difference but again the usage patterns I see don't look like that and really the idea that a site with 25,000 page views is getting swamped by another almost 50,000 views from bots in every month is a bit incredulous.

The one interesting thing I have realized though from this is that may help explain some other odd behaviour I have noticed in terms of traffic as reported by Google. I have noticed for some time that the number of active sessions for the site at any one time sometimes exceeds the total number of visitors that Google reports on the site. Even when that doesn't happen though the number of visitors by session at one time for peak hours (from 8 to 5) will represent 1/4 to 1/3 of the total Google reports which is also suspicious. I mean it's not too likely that users are spending 2, 3 or more hours on the site at any one time. But if Analytics is truly missing 1/2 or more of the site traffic then those numbers start to make more sense.

I am now doing more extensive logging and tracking on my end so that I can better identify and elminate bots as well as compare my results more closely with Google's. I am hoping by Monday to have enough data to perform a meaningful analysis and maybe get a better handle on what exactly is going on. We shall see. It really is... odd.

Twitter makes me mental or how not to write an API

Fri, 08 Oct 2010, 22:30:00 EST

Sometimes in the land of development you'll find yourself working on painful tasks, usually involving some half-baked library with shoddy documentation. Lord knows I've had a few of these. And I have to say the twitter API ranks right up there with the worst of them.

The overly complicated setup, the lack of a step-by-step guide that isn't missing the all important steps 3 and 17, the fact that you can do everything right and twitter will still throw random 401, 403 and 500 errors at you and the handy-dandy undocumented "features" (like locking out an entire netblock at a time if you attempt to use single use authentication) all contribute to making it a real "winner". But the documentation in general is what makes this crap stand out above the rest.

As an example I would like to present a page that I nominate for the 2010 worst page of the year wand is featured on the dev.twitter.com site. This page isn't just bad because it's poorly structured (which it is) and needlessly complicated (which it also is) but more than anything because I think the most direct results of it will be to actually "uneducate" the masses. No. Really. Think about it. Most documentation/guides are written to help people understand something better. In this case most readers will actually be dumber for reading it.

Before I delve into the many different problems with this page I think I need to give a little background on the issue. The question related to how Twitter counts "characters" aka the 140 limit.

Character encoding is a thing I know a goodly amount about (which saved me from being infected by the stupidity on the page in question) and the basics are as follows.

Bytes are not the same as characters. Characters you can think of as the letters you see on the screen. Like A B C etc
Bytes are the computer representation of characters. They are how computers are actually storing, comparing etc characters
Some alphabets (or character sets), like the one used for English (called latin) will have a one to one character to byte ratio. So one character equals one byte
Some character sets have multiple bytes per character (for example Chinese). So one character might take two, three or four bytes.

The reason all this is important is that when you talk about data limits of any sort, sending, buffering, receiving etc, you are almost always referring to bytes. Because, sensibly, bytes are what is actually going on at a low level and no matter what alphabet (character set) you use 4 (for example) bytes is always going to be 4 bytes regardless of if that represents 4 characters or 2 or 1.

So keeping that in mind let's look at all the things wrong with http://dev.twitter.com/...characters. And again the question is a simple one, does Twitter count to 140 bytes (which would make sense) or 140 characters. A fairly simple question to answer, or one would think anyway...

To give a bit of credit here the answer is actually in paragraph 2. And since I understood it I should, according to Twitter, be working for Twitter. Sadly though the explanation is a poor one because a short non-geeky version of what it says could (and should) easily be given. Here is the short non-geeky version, "Twitter counts characters not bytes".
The page then veers off into a random attack on Wikipedia. Which is to say the least odd, I mean why point out a link to Wikipedia only then to say how bad it is? Not co-incidentally this is where my real upset started. Character encodings are one of the most chronically misunderstood computer topics and the nonsense that appears in this section is just contributing to it. To begin with the Wikipedia explanation is better than what actually follows and statements like "For many Tweets all characters are a single byte and this page is of no use" is just so much unneeded confusion. What they mean by that by the way is using a latin (English) character set. And it's not "of no use", it's of plenty of use you just have to understand that for most latin characters it's a one to one byte to character ratio.
I have to jump back slightly here to award what is one the stupidest sentences I have ever read which follows the Wikipedia reference. In all it's glory it is
The definition we're interested in here is not the general definition of a character in computing but rather the definition of what "character" means when we say "140 characters".
I have no idea what the author of the page was trying to say at that point but they badly need to have some sense knocked back into them. The whole "defintion" of what "140 characters" is *IS* a computing definition. Further the understanding of the nature of bytes vs characters is key to that definition no matter how much you try and make out like it's different.
Now we get to the "meat" of the page which can be found starting with the counting characters section. At this point can be found some (rather poor) explanation of the relationship of bytes vs characters (which the page just went through some convolutions to try and say didn't matter) followed by diving right into codepoints. Now excepting what comes before and after this would be passable but in context... it just finished saying that none of this was important *and* it turns out later it is unimportant but for different reasons than originally stated. So essentially so much cruft that annoys those with actual knowledge and will confuse the hell (and make it appear all very complicated) to those who don't.
The next section, on Combining Diacritical Marks is just more useless cruft. It won't do anything to help those who don't know and is just bumphf if you do. At least it avoids outright harmful stupidity.
Unicode Normalization is the section that follows and is really the key to understanding the whole page and there are two sentences that actually give the answer which are "Twitter also counts the number of codepoints in the text rather than UTF-8 bytes." and "Twitter counts the length of a Tweet using the Normalization Form C (NFC) version of the text.". To translate "Twitter counts characters. The character count is derived using Normalization Form C of the text". Sadly though in the real version the two sentences do not appear in that order nor are they consecutive. And in fact the most important sentence (the first one saying what is counted) is the one that's also buried the deepest. Considering that it is the answer to what the whole page is supposedly about it is remarkable that it is buried so deep.

In short. Terrible.

Now look, you can make the argument that a complex description is required, and no doubt Twitter suffers from having to support APIs for languages used by "programmers" who wouldn't know a byte if it rose up and took one out of their ass (I'm looking at you PHP) but honest to god I could have written a much better version in two paragraphs and with a bunch of links for "further reading". It seems to me that at least part of the problem is that whoever wrote that page doesn't know who the audience will be. The technically competent and savvy, some kid who thinks programming means "I installed WordPress", people who just enjoy reading poorly structured documents on any subject and/or my grandmother. This page seems to try and reach all these groups and with a predictable result.

And back to my original point, and pain. This document/page is pretty typical of the "documentation" that Twitter provides. Just cluttered with cruft and statements that are at the least disingenuous, important items are buried and all in all you have to read things 10 times before you can decide if you are crazy or the author was. (Hint: it's the author). Really, really dreadful stuff. My hat goes off to all those who came before and have written the actual low-level Twitter APIs that all these apps use.

3 random things

Tue, 06 Jul 2010, 09:53:00 EST

As the title implies there is no "theme" to this post it's just a collection of a few thoughts I've had or encountered recently.

First to get a bit of housekeeping out of the way I have a new Twitter account. http://twitter.com/MaxStocker. I am still currently responsible for the previous (TalentOyster) one but I will be transitioning away from one to the other over the next short while. (Note: I do know I need to update this in a number of places and I will do so over the next number of days. If after say the 9th of July you see an old account linked somewhere please send me an email and let me know).

On to the tech stuff. Url-rewriting in Apache. I generally think it's a good thing but boy, sometimes I think it can be abused. I've seen some really ugly stuff in the wild, and lord knows even for myself while regex is not my special calling in life I can write complex enough ones to make it tricky to follow.

But the one thought I had with this recently is that it *is* interesting to use it for validation purposes. And by validation I mean type-checking values passed for processing. The question is, is this a good idea. My feeling is that it seems like a nice plus, but also has the danger that it could lure the unsuspecting into thinking it's "enough" which it isn't.

Finally foreign keys, are they really needed? This is one of those ideas that I thought would make a good post on its own. And it probably would, but I lack the time at the moment to expand on it so a short entry will have to suffice.

I am not sure how "necessary" foreign keys really are any more. Now do note I am talking primary keys here so let's not go crazy. And I'm not talking about fields that *act* as foreign keys either but actual database "constraint" foreign keys that will only let you use value in X where it's the primary key of Y.

In theory it's a nice idea because you move data relationship into the database and out of the application. But do you really? No. Because in your application you end up having the same steps to deal with missing relationships, or changes to them, that you would have in the database regardless. The other side to this is that there is a "cost" in having foreign keys, I'm not going to talk about performance here because hopefully the database has that one sorted out but a cost in making the persistence system more restrictive then it perhaps needs to be.

I don't doubt that much of feelings on this has got to do with my experience, both with using databases that didn't have or support foreign keys or in having terrible problems migrating databases that did have them. And I also wouldn't recommend these ideas to someone who is new to databases or database development but... if you know what you are doing then I wonder.

This all may be a really stupid idea, I'm still thinking about it but ask yourself these questions. When is the last time that the constraints of your foreign keys weren't duplicated in your codebase? And do you really use cascading updates or deletes on foreign key changes? If the answer to those are no and not really then I don't really think you need foreign keys either.

Whose blog is it anyway?

Wed, 05 May 2010, 21:10:00 EST

I had a discussion today on the topic of people stealing content from blogs, twitter feeds and the like. As it was the second time, and second different person, I had a discussion on this very topic about this week I thought I might offer some commentary on it.

In short. I don't care.

Look I do get that you put effort and thought into the content you provide. I mean between here and various forums over the years I don't know how much content I've provided, nor if it's all exactly high-quality riveting stuff, but a lot of it? For sure yes. And some of it good? I'd like to think so. So at any rate, I do empathize with you.

And still...

I should probably elborate what I mean by stealing content. If there is some, reasonable level of credit given then for the purposes of this post it's not stealing. This isn't a debate about the legal intricacies of intellectual property. That's a fine debate to have but what I post on a public internet site I don't need to have my permission asked to use the content. But do give credit where it's due. If the content is copied then say so and put the source person's name or company along with that statement. A link would be nice too but I don't consider that a make or break point. But if you don't mention it's copied, or the mention or name is in a 2 point font or in the HTML comments, well, then that's not really credit.

But even with all that I still don't care. Why? Well for starters it happens all the time, and much more than you probably want to know to be honest with you. But the reality is there is less than zero that you can actually do about it. You can make your content private and that might help, but even that's not a gaurantee and let's face it private content means a smaller audience for whatever thoughts, inspiration, rants, ideas or general ramblings you bring to the internet buffet.

And while I guess that's somewhat a cynical view there is another side to my views on this and it is this. The public internet is about information sharing. The things that you post on your blog, or your facebook page or on twitter are thoughts that scatter to the wind like so many dandelion seeds. Some are going to blow onto concrete and die. Some are going to be swept down into dark crevices you don't really want to know about. But some are going to take root and grow.

The internet empowers all of us with the power of mass-messaging. Anyone at any time can read your thoughts and agree or disagree with them, be inspired by them or find an answer in them. But at the same time it means that shady characters can try and claim them as their own. Does that drawback make it less empowering? I don't think so and more I tend to believe that shady characters tend to be shady in everything they do, and in the end get called out for it in one way or another.

I guess I think that more than anything I (or you or anyone else for that matter) has better things to do with my time than try and track down people who have so little in the way of original ideas that they have to try and steal mine.

Random thoughts from the week

Sat, 10 Apr 2010, 10:28:00 EST

I had a number of thoughts this week that at some point I thought could sprout into blog posts but it turned out that no, they really couldn't. On their own at least. So here is just a collection of random thoughts that occurred to me during the past week...

***

Sometimes MySQL drives me bonkers. It's not a terrible database but it sure does have issues that prevent it from being a good one. Earlier this week I made a boo-boo and wrote a query that made MySQL try and make a cartesian join on three tables containing roughly 4000, 200 and 100 records and so attempting to build a cursor of about 80 million records.

It was quite the scene.

And yeah it was my fault but it proved to be impossible to get MySQL back (it had to be humanely put down) and that shouldn't be.

***

I really dislike the word artisan. I mean the word itself is okay but it seems everything now is artisan. There was a time that it was confined largely to bread and while a bit pretentious I could live with that but now apparently any foodstuff isn't worth anything if it isn't artisan. Enough already.

***

I am personally not a huge twitter fan. I have tried. It's just not really "me". But having said that if you and/or your business isn't on twitter you're doing yourself a *huge* disservice.

***

Sometimes people are scared of numbers. It's puzzling to me at times. How can you measure for success or failure, or to know where you are in relation to those if you don't have real numbers? And refuse to do anything about it?

I attribute this fear to a fear of what the real numbers actually are. I suppose that in that case you can safely assume you're well entrenched on the path to ruin.

***

If you have a full-blown web application then you *must* provide decent online documentation on how it works. And decent can in some cases mean "it exists".

Look, I'm sure you feel your app is so simple a brain dead monkey could figure it out but I don't understand the terminology you use and so I can't get started. A simple glossary of the terms you use and how they fit together in your little world would be excellent. "Projects", "Lists" and "Campaigns" are too generic as terms for me to discern the subtle means they have when it relates to your software.

SEO and the magic beans

Mon, 05 Apr 2010, 20:13:00 EST

It seems like not a week goes by without someone, somewhere asking me a question about using "keywords" on a page. It seems there are still people, apparently stuck in 1997, who just know, or at least heard from a friend-of-a-friend or some spam (whichever comes first) that adding a few keywords (META tag keywords for those of you scoring at home) is all that stands between their site and billions of untold riches.

Let me answer that question succinctly.

NO! IT DOESN'T WORK THAT WAY!

And now for the longer version.

I want to start this off by discussing the actual term, "SEO". Let's face some facts here. SEO is an obsolete term because really there is only one search engine that matters to any site and that's Google. Yes, I am aware of Bing and Yahoo, but the fact is that if your site has a strong ranking (high listing) on Google it will have a strong ranking on those as well. And any other search engine that actually matters.

So aren't keywords the magic beans that will grow your site like an overreaching beanstalk to the city in the clouds?

No. The fact is that Google places no relevance on any content in keywords tags. And for good reason! Google is popular because it produces useful results and it doesn't produce useful results by relying on the "optimized" keywords that some scam artist produces. I mean, think about it. If Google was just a summary of the garbage that "SEO" companies produced it would be useless. And nobody would actually use it. (As an aside this was a popular way to do things in 1997. And it's why you haven't heard the terms "AltaVista" or "Northern Light" in about 10 years).

If at this point you are 100% sure that I have totally lost the plot read over a few excerpts from the Wikipedia entry on Meta-elements (and the keywords specifically).

"With respect to Google, thirty-seven leaders in search engine optimization concluded in April 2007 that the relevance of having your keywords in the meta-attribute keywords is little to none"

Little to none... but wait, it gets worse

"in September 2009 Matt Cutts of Google announced that they are no longer taking keywords into account whatsoever."

And if you want to disbelieve Wikipedia check out this page on the Google webmaster central blog.

"Google doesn't use the "keywords" meta tag in our web search ranking."

I honestly am not sure I can make it any clearer. Keywords will not help your Google page ranking. In fact what evidence there is suggests it can only make it worse. Putting misleading or fraudulent keywords on your pages in an attempt to increase traffic can actually hurt your page because Google may decide your site is just a scam and lower its page rank score.

So what can you do to increase your Google rank? Before I answer this question let me ask you a couple. What is the purpose of your website? Do you expect to actually have online sales? Will that be a major part of your business? Are your answers to any of the preceding questions realistic?

Before you can determine the ultimate strategy for increasing your sites Google rank you need to figure out what you want to do with your site. Too many people start "optimization" projects with nebulous goals. To paraphrase the Cheshire cat, if you don't know where you want to go what path you take is entirely irrelevant. You need to start by defining actual goals for your business and your site. Define how your site actually fits into your business goals. Anything else is a waste of your time.

And more than anything else time is what you'll need to invest to increase your Google ranking. Real links from respected sites is what will increase your Google ranking. Establishing those relationships and spreading awareness of your brand takes time and effort and there is no magic bean shortcut to that.

Having said all that I will share some code tips to help make your site as Google-friendly as possible. Again, these actions won't be themselves skyrocket your site to the top of search results but if you are making the effort to grow your presence organically they will help.

Make sure the terms you want to be found by are actually in the content of your pages. It seems simple and yet... If you want people to be able to find your site by searching for "apple" then you need to actually use the word "apple" in your site content and not just "fruit". Just because your personal opinion is that "fruit" is a more inclusive term than "apple" won't help you in your Google ranking. Use the terms you want to be found by. Use them in the context of your content. And use them often.
Use descriptive titles and add useful description (meta tags). These steps may or may not help your results move up in Google rank but they will help your content be categorized by both Google and your users.
When possible use unique URLs for content. Either through URL re-writing or the magic that is enterprise platforms like J2EE giving unique (not entirely querystring based) URLs can help your site getting all it's content indexed. (To translate if you have links that look like http://yoursite.com/index.php?page=1 and http://yoursite.com/index.php?page=2 or you can have links that look like http://yoursite.com/AboutMyCompany and http://yoursite.com/ContactMyCompany the latter form is better.)
If you have content that is frequently updated do add the cache and expiry META tags to your content to appropriate values. This will let Google know that the page should be checked at regular intervals and updated appropriately. Don't abuse these tags if your content is largely static because that can penalize you, but do take the time to add it if your content often changes.
Last but not least make sure your site is accessible to search engines. Adding a link to every page that takes you to an up-to-date site map is often a good idea. And if your site is all Flash... well... you made a mistake. Make sure the pages of your site can be viewed by a user who doesn't have flash and has JavaScript turned off in their browser. If your site is still at least navigable with those restrictions then you're looking good to have your site fully indexed by Google.

Most of these steps are in my mind just common sense. At the end of the day your site will only get the Google rank you need because you invest the time and effort needed to grow your ranking organically. But since it's 2010 and it still comes up let me be totally clear, meta keywords are not the magic beans of e-commerce riches. And if you are sure you know different, I would urge you to consider the source of your information. SEO "Advice" that comes in spam emails isn't worth any more than the advice on increasing your manhood or your stock portfolio that you also got through spam.

I really think

Sat, 27 Mar 2010, 14:04:00 EST

that there is no quicker path to crappy code then having to screen scrape data. I really do.

One of the things I have to do in my current job is to have software that connects to other sites (that are owned/used by our clients) and scrape data from those sites to pull into our database. It's somewhat a thankless task because it's ugly and awful and nobody really notices when it works but it can cause problems when it fails.

And the code. Well. I tried. But the fact is it's just ugly. There is some abstraction at least so that the parsing/scraping mess is an interface used by the sorting and loading code. But still...

And I'm not sure what I could do different or better to be honest. Every scrape is different and some of them involve a bunch of JavaScript parsing so it's not even so simple as just loading an HTML DOM.

And I'm not aided by the fact that some of these systems are truly dreadful from an user-interaction perspective. The kind of systems that rely on sessions for navigation thus resulting in unrecoverable errors should you hit back in your browser or attempt to see some data in any order but the one officially proscribed. I don't want to go off on a total rant here but it's 2010! Who writes web applications like that anymore? This isn't a new paradigm anymore guys, and frankly if your app can't handle a user hitting the back button on any page ever without complete collapse your app just plain out sucks.

I'll tell you, if it wasn't for fiddler I'd be totally, well, you know. (On an aside fiddler now supports HTTPS, I think it has for a little while now although it didn't used to and I hadn't used it before now, and while your browser will (rightly) complain about what is essentially a man-in-the-middle attack you can use it to fully trace HTTPS interactions)

On a funny note though I had one of our clients (whose data we have been scraping successfully for months) call and ask me if I could explain/help them to do the same because they need to scrape the data and couldn't figure out the magic I did to do it. In fairness it was one of the JavaScript related scrapes and took some painful reverse engineering on my part to figure out what the heck was going on.

Blackberry development

Sat, 06 Mar 2010, 20:23:00 EST

So... I started playing with doing some blackberry development. I'm not entirely sure what I will do with it but it's sort of cool in a way.

But I am finding some painful things. Like the simulator is buggy as all get out, at least on my platform (Vista 64 bit). I have to end task to kill it for one thing and the reloading of apps is at best flaky so I end up having to kill it quite often because it locks up.

And the documentation is not great. There is documentation. And there are tutorials but while it is Java it's a whole new world when it comes to UI and the behaviour with that is not well documented. It seems to be "Swing-ish" in some respects but there are some behaviours I don't understand. My current problem is that I have a "window" (screen) I push a screen on top but when I pop that screen the main menu appears on the first window. Why? I don't know. But I wish it would stop. :/

Right way, wrong way

Thu, 11 Feb 2010, 11:26:00 EST

I think when I am finished working on my current project I could write a whole book on tips and tricks for internationalization of websites.

But here's the most important lesson I've learned so far. The technical challenges are the very, very, very, very least of your hurdles. Really.

If there was one thing I wish I had done differently so far is that I wish I had been able to more clearly identify and emphasize the difficulties raised by these challenges and what the options are for overcoming them. With regards to the latter point it's a bit of a problem because I'm still not sure how to really overcome some of them, or at least which way is best.

An update for December

Thu, 31 Dec 2009, 18:30:00 EST

As it's the last day of December (and of 2009) and I haven't written one blog entry yet for December I kind of felt I should.

Well obviously I have been pretty busy work-wise these last few months but I am hoping it will settle down at the end of January for a bit. I feel that in general I need to get into better shape and have less stress in the coming year. The first three weeks of January will be pretty busy with finishing off (for now) the big project at work but things should be okay after that.

I also do want to return to blogging more regularly. I think it's good for me at least to get some of my ideas out and this gives me an opportunity to do that. It's also good for turning on the non-coding parts of my brain once in a while at least.

MySQL, JDBC, Unicode and You

Sun, 29 Nov 2009, 10:13:00 EST

A few notes on my experiences with using MySQL from JDBC with unicode support that might help others.

Yes, MySQL and the JDBC driver fully support UTF-8 and really it's quite easy to do but there are a few things to be aware of.

First you are best off if you create your database to use utf8 as the character set from the get go. If the database is set to use another character set by default then you have to be careful with your CREATE/ALTER table statements to make sure you're using utf8 there. Also the driver can get confused (if you're not careful) when the database and table character sets don't match.

Second on the driver front you should stick the following parameters onto your JDBC connection URL (I am assuming you are using the MySQL supplied JDBC driver, I can't imagine why you wouldn't really).

useUnicode=true&characterEncoding=UTF-8

These parameters will make sure that the driver uses the correct encoding. As mentioned above if the database is set to use utf8 the driver will auto-detect this but there is a difference between database and table character sets in MySQL which can trip you up if not careful. So setting these parameters ensures it will work regardless of your setup.

In my opinion you should never use another character set in MySQL besides latin or utf8. If you are only ever going to store "ASCII" text than latin is fine but if you are supporting any other character set or sets just use utf8. It keeps things simple.

And to wrap up, a few notes on testing and displaying. If you are having problems with UTF-8 data and your database you should test the data before you insert it to make sure it is what you think. A large amount of JDBC related encoding issues are caused by the data being mangled well before it is event stored into the database. Also make sure that you are identifying the data correctly on the way to display it as well.

For quick and dirty stand-alone tests JOptionPanes work well in seeing data (as long as you have fonts that can display the glyphs in question). For web projects there are two common mistakes that can happen, one is not correctly identifying the outbound data as UTF-8 (which you can do very simply in your JSP page directive). The second is mangling the data in which is solved by setting the request encoding correctly in your servlet.

request.setCharacterEncoding("UTF-8");

Whatever doesn't kill me will make me stronger

Thu, 05 Nov 2009, 20:43:00 EST

Words to live by.

I hope.

I'm having a bit of a time of it of late. Work is not going swimmingly well. I mean I am working well but not getting everything I need to get done actually done as I get sucked into other projects. Unfortunatley at work they have found my weak spot. Ego. Appealing to it, and me for help is hard for me to turn down.

Meanwhile the fires of The Flu ™ swirl around me, especially on the home front where I am trying to avoid getting sick as best I can because that will really sink my work schedule.

And on top of all that Sarah is having more troubles with her Mac. The latest fiasco is that caused by simply this. The TCP stack in OSX is an EPIC FAIL. Previously I had encountered a bunch of shenanigans with AirPort, whereby when the connection would drop and reset, which happened frequently (for mysterious reasons known only to Mr Jobs) the whole OS would lock up. Now hard-wired it's having problems when it resets the DHCP lease. Namely it is somehow fouling up the MTU size so that it ping and email work but everything else goes out in too large a packet size and vanishes into the ether.

So well done on Apple on successfully implementing a full-on implementation of the stability of the Windows 96 TCP stack.

Pah-thetic.

So that's me.

On a plus side I believe I have done things with CSS now, for internationalization and cross-browser purposes that are actually pretty cutting edge. So that's a plus.

Somewhat random thoughts

Sat, 17 Oct 2009, 18:49:00 EST

I've been having an interesting but mostly good experience at my new job. As a person with lots of contracting experience previously I've seen a real mix in the ways company operate internally. My new company is no exception and there are both good and bad things I see. Basically on the plus side there is good product and people with good experience, on the down side process management needs improvement. The latter has been frustating both to experience and witness a bit at times because like anything I do I am invested (in many ways) in my job so I like when things go efficiently. However, this issue was also one of the reasons I was hired.

Specifically focusing on the technical things are going fairly well. I am still positive about delivering things as advertised and on time but there are some elements of unknown because frankly there are aspects of what is being developed that have never been done before. And there are always little hitches you don't anticipate. A small example from this past week, the IT staff (who have been good and responsive) are configuring the servers (staging and production) and we hit a small snag when I decided to yank Apache from the chain and just deploy on Tomcat. It works fine except that I forgot about the privledged port restriction on Linux so a bit of time was lost to work around that.

Outside the technical I have been getting plenty of positive feedback from people and that's really nice. It's always nice to be appreciated although I also know that besides having lots of experience and knowing what I am doing I also do work very hard so I don't exactly feel it's undeserved praise. At any rate there is already concern being expressed then when initial development is complete I remain happy. As in, what will I want to do, what will it take, to keep me once the initial development cycle is complete?

It's an interesting question because among other things it's a question I haven't really asked myself much. To be honest the project needs to be a success for me to stay in the long-term because part of my compensation is tied to sales and my compensation needs to be more competitive. But beyond that I really hadn't thought much.

Now is not a time for me to spending a lot of time dwelling on it because I have a lot already on my plate, but it is proving interesting food for thought.

Strange SSL woes

Wed, 14 Oct 2009, 08:46:00 EST

I encountered a strange problem yesterday that I am still trying to resolve. Basically I went to an HTTPS url in Firefox and it got very upset. Essentially tells me that the connection is not secure because the certificate can't be found for the CA. Running a test on the verisign site also produces a fail that says "The intermediate CA certificate cannot be found for the following certificate chain".

What I really don't understand is that IE on various machines at various locations don't have a problem. And FireFox (same version on same OS and arch) doesn't have a problem either in another location.

Yet running the versign test from any location produces the same results. Fail.

So either Verisign (and my Firefox) has lost their minds or a bunch of browsers are trusting a certificate they shouldn't be. Either one of those is bad. Or there's something entirely else I'm missing.

This doesn't make any sense to me.

Because everybody has a Mom

Sat, 03 Oct 2009, 11:12:00 EST

Okay, if you've already seen this I apologize but every chance to get the word out.

I am participating in the "Run for the Cure" for the Canadian Breast Cancer foundation which takes place tomorrow, Sunday October 4th 2009. I am doing this for a number of reasons including for my mom who was a victim of breast cancer in 2001.

If you would like to sponsor me in this excellent cause please visit https://www.cibcrunforthecure.com/html/personal_page.asp?track=3887953&languageid=1.

Thanks for reading.

Sometimes I wonder

Tue, 29 Sep 2009, 21:41:00 EST

I wonder about people sometimes. I really do. I have been recently witnessing a slow train wreck on the Java forums with a new user. Now I should start off by pointing out that the user admits to be very new to Java, in their thread of last week they claimed to be programming in it for all of 24 hours thus far (later expanded to 48). Which is fine, there are new programmers all the time, it's one of the main uses of the forum. But of course this one is trying to write some sort of application for business use that in short is a multi-threaded socket server.

In two words. Uh oh.

The ultimate point of failure in this, and the part that really makes me scratch my head is this users arrogance. They claim to have programmed before in some language. Which I think is true, although they have obviously never dealt with either threading or sockets before in any language. But the real kicker is that they don't seem to be listening to any really good advice. Like for example the kind I gave in the first thread they started.

In the first thread they started I gave the user the proper threading model to use for their program. What threads will do what and where synchronized work queues will go. And what will be hard and what will be easy. Now sure it was high level, but I was willing to give more if asked (and of course wasn't) and more importantly I know it's actually right.

So from the looks of more recent threads by the user, which I have declined to participate in, the project is moving forward, directly to the rocky shoals. If it ever works at all, which is unlikely, it will not scale and most likely will deadlock and crash.

And that's about what you can expect to happen when someone who doesn't know what they are doing writes a multithreaded socket server.

And the question that I am left wondering, as I always am, is why is this person asking for "advice"? Their arrogance blinds them from accepting good advice, they somehow think "they know" the way the app should be written... even though they have no idea of how to write it. Here's a clue, if you don't know how to implement your design then you shouldn't be designing in the first place.

I am just truly perplexed at the logic that leads one to make these sorts of decisions.

Many updates

Sat, 26 Sep 2009, 19:45:00 EST

As I noted previously I have been busy with a new job and I am bit saddened at how much my blogging has fallen off recently because I've tended to be pretty good at keeping up with it.

The main problem is not the job so much as it is that the current work schedule and timelines means that I am ultra-busy there and with time this should reduce to a more manageable level. The bottom line is that the web product I am working on needs to be launched ASAP and while there will of course be ongoing maintainence and further development needs these will be less taxing than the current situation. It is exciting and interesting work but just a little busy.

I have though made a few updates to my site here to reflect my changing career and needs. Essentially I am continuing to work with existing clients, and existing partners, but I am no longer looking for new additional clients. If you are looking for web development work and are not an existing client please contact my marketing partner StayAwake.

There are also now links to my Twitter account, Facebook and LinkedIn profile pages on all my pages, besides my blog here, which I promise to update more often, Twitter is probably a good way to see what I am up to at any given time.

I have had some blog ideas, and items of interest that I should be blogging about more, so lack of content is not an issue here, just time. But I'll try and be more on top of that.

IE Rant

Sun, 06 Sep 2009, 20:10:00 EST

I wish I could think of a better (read: more humour) title for this post but I'm tired and frustrated so I can't.

IE 7 is an abomination and I really, really don't get it. I wish that it was just the IE 7 wrapper around the IE 6 rendering engine, that would have been far better and that's saying something. Yes IE 6 is not close to CSS and DOM standards but at least it was close to IE 5. And IE 8 gets credit for being "close enough" to the standards to actually work with the same code that works on Firefox, Opera and Safari.

But no. Instead we have a totally different set of very quirky behaviour that's not like 6 or 8 either. What in the .... ?!?

Now some might say well 6 isn't the standard either (or for that matter 8) and yeah okay. But what excuse was there to develop a browser that is (a) incompatible with your previous and (b) isn't the standard either? None. That's what. And that's the mystery here.

Honestly if Microsoft's different IE versions weren't so bloody difficult to test I probably wouldn't be so frustrated. But they are so it's relatively painful to test and to top it off, as pointed out, for some very stupid reason you need to test each version independently because apparently nobody at Microsoft would know "backward compatibility" if it jumped up and smacked them in the face. Oh and that "Compatibility View" in IE 8. Well. Yeah. Whatever it actually does, and I've heard various theories about that, it's certainly not either IE 6 or 7 behaviour. And it doesn't seem to really match IE 5 either. So who knows what that is supposed to be but suffice to say it doesn't really work either. What a shock.

Status update

Sun, 30 Aug 2009, 21:59:00 EST

I feel bad for not blogging for awhile but I've been pretty busy as of late. I have just started working, a full-time gig, on a pretty complex J2EE project. It's pretty exciting for me because there are many interesting parts to it but it's also keeping me fairly business.

I am going to have to update my site, probably this week, because while I continue to do some work for existing clients, specifically those I am providing hosting and related services to, I am otherwise available.

I will be blogging about some parts of the project I am working on because I think they would be of interest to some people with J2EE. The main part adding complexity to this project is internationalization which I have well in hand but I think is worth discussing here.

On an unrelated (mostly) side note I have discovered that the JSTL <c:out tag does not really seem to work for producing RSS reader content. I'm not entirely sure why that is yet, I suspect the quotes are being escaped incorrectly for RSS.

Blog reuse tips

Mon, 17 Aug 2009, 12:35:00 EST

Re-using and making available for reuse content from a variety of different sources on the web is becoming a bigger thing for most companies I know. It just makes a better use of employees time to simply reuse content from a corporate blog, or news feeds on the rest of the website and vice-versa.

It's pretty easy to set things like this up as well. Attaching widgets to RSS feed and displaying selected content is not too difficult technically and then different parts of your site will both have dynamic content and all get updated at the same time. As any programmer will tell you, code reuse is a good thing, doing it with content has the same kinds of benefits.

In this line I want to offer two tips to people writing or contributing to blogs in the business world. These tips will help you later be able to reuse the content in multiple places, because it's not just about the technical side to implementing this, the content has to be there and ready to go as well.

Lot's of entries and updates

Your company featured in a newspaper article? Make sure to mention it in a blog post. Your site just been updated? Make sure to mention that in a blog post. Short, news-related blog posts are fine and you should make them. One problem that corporate blogs often have is that every entry is a long essay or opinion piece. That sort of thing is great and can do wonders for your brand in terms of thought leadership, but your blog doesn't have to be, and really shouldn't be just that.

Putting short entries with this kind of update information is great for keeping people informed about your company's activities and more importantly it will help you later when it comes to finding blog content that you can reuse on other portions of your site.

Always tag ALL your entries

I really can't emphasize this enough but you should always, always be tagging all your blog entries. Being able to clearly identify entries by tag will be vital later to allowing you to select specific content from your blog feed for later use. You simply can't reuse content well if your content isn't tagged.

I think these two tips are pretty simple, and add to the value of your basic blog content regardless of what else you do with it. But following these tips also means your blog content is ready to be reused on the rest of your site as well.

Will Canada take Facebook to court?

Sun, 16 Aug 2009, 11:37:00 EST

It will be interesting to see what happens this week, and especially Monday now that the 30 grace period Canada's privacy commissioner gave to Facebook to clear up it's privacy issues has expired.

This is a very long tale that started last May when the privacy commissioner began an investigation after receiving a few complaints but one in particular. Facebook has always had "issues" surrounding privacy related issues, and as I pointed out last year I think they seem to have a corporate culture of finding the fine line by crossing over and letting user outrage push them back. It's happened again and again so it really seems that is the case.

Regardless they certainly have privacy issues and sure enough the findings of the privacy commissioner found no less than 12 violations of Canadian law relating to privacy on Facebook. After the findings were made public the commissioner gave Facebook 30 days to address the issues or face further sanctions/steps.

The 30 days are up and zero has been done.

If the privacy commissioner has any power, and is to be respected in the slightest, they really need to move quickly now to (a) force Facebook into changes and (b) punish via fines Facebook for the inaction. I really think they need to do both because while I am all for social networking I am also acutely aware that privacy is not just an issue but a real problem. Far too many companies take privacy issues only as an afterthought, and the results are plain to see. Facebook is a large entity and a well known privacy abuser, if they "get away with it" then it doesn't inspire other companies to review and update their stance on privacy issues, and to me that's more what this is about.

We'll see what happens.

Death knell for software patents?

Wed, 12 Aug 2009, 11:38:00 EST

Today's breaking news, that Microsoft has had an injunction against them, preventing them from selling Word due to patent "infringements" is I think possibly the beginning of the end of software patents in general. Or one can only hope.

The reason I say this is that this action is reaching new heights in nuisance lawsuits filed by weasel companies looking to cash in on the basis of confusion and FUD at the U.S. Patent Office and civil justice systems.

You see Microsoft is accused of patent infringement for the XML format files used by new versions of Word. Now I am all for standing up for the little guy against the big, bad corporation but the problem is that the patent is absolute rubbish. I actually read the damn thing and it claims to be an "invention" of SGML with "mapped metacodes", otherwise known as tags. So it is essentially a patent of XML.

There are a million reasons this dog won't hunt, including that the patent is dated for July of 1998, a full 6 months after the XML 1.0 standard was officially decreed by the W3C (an international standards body). So the holders of the patent in question took a new standard and managed to somehow get a patent for it. It is to say the least laughable.

This isn't the first time a small company has gotten hold of a highly dubious patent and used it to try and extort money from a successful company but it's got to be one of the loudest actions, in terms of coverage and based on one of the flimsiest patents that I have yet seen. Whether XML was ever patentable or not, and that's questionable since as the patent admits it's an relatively simple application of an existing idea (SGML) it certainly wasn't patentable after the standard was made official.

I think this shows in a bright light all that is wrong with the ideas of patenting software. I am fine with people protecting their work and efforts, after all at the end of the day most of us who write software need to actually get paid for doing so. But the patent route is one of obvious failure. I am not sure it actually "protects" anyone and it seems to more than not just lead to rubbish like this.

I hope the company who has filed this suit is in the end squashed like the roaches they are. I am ashamed to note that they are located in Canada, but I guess it just goes to show you that parasitic vermin lurk in every place you can go.

Updating JDoctopdf

Sat, 08 Aug 2009, 10:11:00 EST

I am working on cleaning up a few projects this weekend, including JDoctopdf. There are two items that need attention. One is that I would like the pictures to maintain their size better, I am not sure if that is really fixable or not. The second is that I would like to change the way I handle table borders. As it is the table border issue is pretty hopeless, there are some border and cell "style" properties but my attempts to reverse engineer this have all failed. Sometimes I can almost detect whether a border exists or not, but further testing shows it just doesn't work 100% of the time either way.

The problem is that normally tables have and need borders. It makes their content easier to read. However sometimes tables are used for formatting, for example creating two columns for a header, and in this case adding borders looks stupid. So I had an idea, which is that since I can't detect table borders I will make any table with one row have no border and and tables with two or more rows have a border.

It's not perfect but I think it's the best I can do with the libraries as is.

And I'll probably look at integrating docx support as well. That would be pretty useful I think.

Privacy and Security

Fri, 07 Aug 2009, 08:45:00 EST

It seems that recently privacy and security issues are all over the place and more than ever. Facebook is being threatened by the Canadian privacy commissioner. Toronto Hydro seems to have lost, and not know how, a bunch of contact and billing data. Even I had an interaction recently with a case of identity theft.

On the hydro issue, it seems someone got hold of a bunch of contact and billing information and is using it to send threatening invoices/bills, perhaps posturing as a collection agency, and trying to collect on "overdue" bills. I know about this mainly because I, along with everyone else I know got a letter recently from Toronto Hydro saying this.

I honestly find this kind of thing pretty worrisome. It's not so much what has happened but that it happened at all. Who got this information and how?

Not that long ago I was helping a client fill out a privacy related questionnaire as part of their work with a very large Canadian company (as in one of the top 10 largest companies in Canada by value). At first I thought it was a good idea but I quickly came to the conclusion that it was in fact a bad sign. Why? Because most of the questions didn't apply to relationship between my client and the company to whom they are a vendor. And the troubling part of that is that it raises the question about how much that company understands their processes, specifically when it comes to data flow and privacy related issues.

I think in the end this is where the problems really come out. It's not lack of security per se that does things in and causes problems, it's lack of understanding of what a particular business's model and flow is.

Three IE AJAX gotchas

Sun, 26 Jul 2009, 16:01:00 EST

I have been working recently on a web-application (J2EE) that uses a bunch of AJAX for the front end and I have come to realize a few things about browsers and their varying support for DOM manipulation and other AJAX related points. Well one thing really. IE in versions before 8 is really, really, really terrible.

I did know about the mess with CSS and various browsers, and in IE8 this mess still continues, for example the ways in which IE vs all other browsers add padding and margins to elements with relation to their size. I didn't really know what a hash Microsoft had made of the DOM support in IE previous to 8 though.

The one good thing I can say though is that at least with IE 8 you can now build a site with the same CSS and JS scripts that will actually work, and work the same on IE 8 and all other browsers. One thing to note, when I say all other browsers I mean: Firefox, Chrome and Opera on Windows, Firefox and Opera on Ubuntu and Firefox and Safari on Mac, so maybe not all browsers, but certainly a good selection on multiple platforms.

At any rate a few bugs/quirks to note in implementing AJAX on IE 7 and 6, maybe this will save someone else some frustration.

AJAX objects can only be used once. I am not entirely sure why this is but once you use (aka send a request and get a response back) from an XMLHttpRequest object you cannot use it ever again. You must create a new instance of the object to make another call.
Again it's worth noting that any other browser (see above list) and IE 8, do not work like this, you can reuse these objects, and frankly I'm not sure why you shouldn't or couldn't.

In IE 7 and 6 at least though you can't do this. Worse, there is no error/exception raised when you try, but simply nothing happens. If you send a second request with the same object then it will send it and just never, ever come back, aka never fire any events.

This was a strange one, and not incredibly well documented as far as I can tell and it caught me out because I couldn't understand why the browser was loading data successfully the first time but seemed to hang on further requests after.
You must set CSS classes via the className property. This is one that I frankly knew about already but it's worth noting I got caught out when IE 8 properly supports the setAttribute methods, including setting the class of an element and previous IE versions just don't work with those methods.

To be honest a good rule of thumb to mention here is that if you are doing DOM manipulation at all, you'll really want one set of scripts for IE 6,7 (and earlier) and another set for other browsers including IE8. This is because in general all the standard DOM manipulation methods are supported in other browsers and IE 8 but either not at all or very buggily so in IE 7 or less.
You must use properties and inline (anonymous) functions to add functions to DOM elements. The very simple DOM method setAttribute which let's you easily add event handlers with functions and parameters you want is sorely missed here but you have to set event handler functions as properties in older IE versions. Where this gets more complicated is when you want to specify your own parameters.

A quick example with some code to show the difference and "how-to" do it both ways. Let's say I have a function inserting some div's into a DOM and I want that on onmouseover events to fire an alert that says which div this is. In IE 8 (and other browsers) the code would look like this...
function sayHello(ID){ alert("Hello from "+ID); } // in the div creation code... var aDiv = document.createElement("div"); aDiv.setAttribute("class","someCSSclass"); aDiv.setAttribute("onmouseover","sayHello("+someUniqueValue+");");
And in IE 6 and 7 you need this...
function sayHello(ID){ alert("Hello from "+ID); } // in the div creation code... var aDiv = document.createElement("div"); aDiv.className = "someCSSclass"; aDiv.onmouseover = function(e){ sayHello(someUniqueValue); }};
I don't know about you but I know which of those styles I like better (aka the one without the anonymous inline function).
Make sure your DOCTYPE is set. This one actually relates to IE8 but it's worth mentioning for the bizarre behaviour that otherwise happens. If your document does not have a DOCTYPE declared (and you can put any junk in it, just as long as you have one) then IE 8 drops into "compatibility" mode which means everything starts behaving totally incorrectly.
Both CSS and DOM that is correct, and was otherwise working in IE 8 will then break. In theory, this behaviour is supposed to be helpful, maybe, but it's also supposed to be more like operating IE 7, at least based on what IE 8 starts saying it is (IE7) when operating in compatibility/quirks mode, but actually it isn't very much like IE 7 at all. More like IE 5, which all in all can lead to some really bizarre behaviour when you least expect it.

Well, I hope that someone searching finds this helpful. There are a million articles, tutorials etc on a lot of these things but it's hard to find all the major quirks documented in one place so this might just help. I can only hope that soon enough most IE users will have moved on to 8 and left a great number of these problems behind. It's actually kind of more frustrating than ever before now that Microsoft has produced a browser (8) that while not entirely correct is order of magnitudes closer to other browsers in implementations of basic specifications than ones they have produced before. If only more people actually used it...

Max has internet!

Mon, 20 Jul 2009, 10:31:00 EST

After several months of intermittent connections, much slower than normal speeds and some periods where I was without any connection for days at a stretch (including most of last week) I now have a, proper, working internet connection.

Whoo-hoo!

I resolved my problems by hiring an electrician. What I discovered was very interesting. I would say about 50% of the problem was with inside wiring and 50% by Bell. So in fact there were two problems, which explains a great deal of why the problems were so hard to isolate.

To be honest it appears that the slowness was due to the inside problems and the outages due to the Bell problems. And that actually surprises me a good deal. Four different Bell technicians came and examined at least parts of the setup including two who I know looked at the problem area and didn't seem to see it. The upshot being that Bell seems to really be suffering from raging incompetence.

I have to say, it's a little difficult being without internet, especially when one is a web developer by trade. So all in all I am very happy that it is working now but I am a little annoyed I had so many problems for so very long.

Having fun with AJAX and DHTML

Mon, 13 Jul 2009, 11:55:00 EST

Working on a new version of an existing site that is using AJAX and DHTML. I'm really happy with the progress thus far. You can really do some pretty cool kinds of things especially when it comes to letting users search for data and paging the results etc.

My only complaint is that when it comes to testing this sort of thing it's both more involved and complicated. You simply *have* to test in different browsers rather than just letting that fall under the design area.

But I think it's worth it. Going pretty well so far anyway.

Good and bad

Thu, 09 Jul 2009, 00:35:00 EST

It's been a bit topsy-turvy as of late, which is the main reason I haven't been blogging for a few weeks.

First I got the flu. That was unpleasant. For a few days I was just lying flat out, weak, sweating (with fever) and coughing. Sometimes though I think having a fever can be good when you are sick because I think being somewhat delirious and sleepy can help you avoid feeling how rotten you really feel. Or something.
Then I went on vacation to the cottage. Hooray. Although it did rain a lot. All the same the break was good and probably good to be off the computer for awhile.
When I came back I got my new netbook from Dell delivered. It's an inspiron mini 10. It's running Ubuntu and I upgraded to the solid state drive and larger battery. It's not bad, I am using it to write this very blog entry in fact. I am considering installing Eclipse on it. Which I know is odd but then I could maybe take it to the park with the dog and work on things there. Have to think about that.
And last but sadly not least, the criminal racketeering ring that operates under the guise of the post office has struck me again. About a week and half ago I mailed two old, non-optimally functioning modems back to my ISP. They haven't got them yet. Guess I forgot to pay Canada Post their protection money. At any rate not good. I think if they don't show up I am on the hook for two DSL modems.

So kind of busy times and a real mix of up and down. I also think my cellphone, which is due for replacement soon is slowly dying. I don't know. The new netbook is nice though.

JDoctopdf

Mon, 15 Jun 2009, 12:32:00 EST

Hooray!

I have launched my Jdoctopdf project which you can find here.

This is a free, open-source, library for converting Word 97 files to PDFs in Java. Some other input and ouput file types are both available and under development. The unique thing about this library is that there are no platform, or third party application dependencies, it is 100% pure Java. You don't require a Windows OS or an installation of Microsoft Office or OpenOffice for the library to use. As such I believe this library is suitable for use in a servlet container environment (this was one of my original requirements in fact).

At any rate if you have ever looked for such a thing and, like me, been frustrated with your search, you may find it useful.

Ahhhhhhh

Thu, 11 Jun 2009, 16:48:00 EST

So a few weeks ago, someone, I don't know who but I have some ideas, nominated me as a featured contributor on the SDN Java forums. So I answered some questions about myself, relating mostly to Java and/or the forums. I was half-expecting it (my profile/interview thingy) to go up at the start of June but that didn't seem to happen (I suspect JavaOne interference).

Anyway. So today I go and was going to check on a few questions I had answered while between other things and *blam* there I am staring back at me. Which was a bit. Eerie. Or something.

At any rate I think it provides a good answer to the question of "what's the difference between the JDC/SDN and JavaRanch?" because nobody is going to be featuring me on JavaRanch anytime soon.

Hooray. I think.

Connection resolution (hopefully) and other ISP issues

Wed, 10 Jun 2009, 22:46:00 EST

Well, two modems and three service calls later I seem to have a working internet connection. I hope it lasts.

I actually now have a Dry loop (or naked) DSL connection and I chucked my phone service entirely so that will cut about $70 off my monthly bills. So that's good. I'm not entirely sure I won't be having more problems with my connection but this does cut out a lot of garbage of things-that-might-be-the-problem-but-aren't and should help isolate the problem in future. Frankly I think that crap modems are the number one culprit in my recent problems.

One thing that is bad though is that my ISP has recently (within the last few months) offshored their support services with the kind of decline in quality you'd expect. Wait times have increased quite a bit and my experience has really declined. Along with ESL related issues, I am put on hold repeatedly and for lengthy times, I have had my call "redirected" and then ended up back in the main queue and to top it all off the front line support people are down the standard of most, that is clueless idiots who don't have the skills or authority to actually check or fix anything but are just there to filter out noise calls.

I have been with my ISP for several years now. I stayed with them, recommended them, and brought them more customers because of their service. They actually charge me more than I could get other places, but again the service experience was worth it to me.

This is not.

Normally at this point I'd be complaining to the ISP and my sale's rep there. I may do that but really I don't expect that to help. I mean they are unlikely to bring back the good, Canadian support workers to replace the offshored ones just because I complain. But I do think this is a big mistake they are making, it's the kind of choice that was made by someone to "cut costs" without understanding what their business model actually is.

Well anyway, it's going to be an unpleasant mess to resolve because of the services I have there. I am going to not really think about it for now, at least until I get some other work related things squared away. Then I guess I'll be moving somewhere else. Really disappointed though.

Signs of Bad Code : Threading

Sat, 06 Jun 2009, 13:10:00 EST

I often find myself explaining why some particular code or database design etc has been done badly. This is mainly because much of my work is referral based so projects I get called into often have a bunch of code that isn't working in some way and in such cases it's not uncommon to find some disaster lurking.

So I thought as a semi-regular feature I'd explain a few points about different aspects of software that for me throw up red flags. These are the kinds of things I look for in code and help me determine quickly if the code is "good" or "bad".

One thing by itself may not be too bad but having several of them in the code/design is a bad sign and almost always indicitive that the original coder/designer was not competent enough. These bad code ideas can cause all sorts of problems ranging from further development being really painful to simply not working and not having a hope of working either. If I see enough of these problems then my recommendation is a re-write because at some point you have a design that is so broken it simply can't be fixed.

With that said I thought I'd start by looking at multi-threaded Java code. Some of the concepts here are Java specific but some actually can be applied to any multi-threaded code.

#1 Extending Thread instead of implementing Runnable

This is a Java specific issue and it's not overly fatal but it is all the same a bad sign. What we're looking for here is how the threads are implemented in the program. There are two ways of doing this, first by extending Thread ...

public MyThread extends Thread{

and the second by implementing Runnable

public MyThread implements Runnable{

The second is preferrable from an OO design perspective if for no other reason that the first locks you into a class hierarchy and the second does not. If you don't know what that means, essentially the second allows for more flexible design later and the first does not and all other things being equal why would you not choose the more flexible approach. Further, actually the first approach does have some other drawbacks that have to do with thread signalling (wait/notify).

The point is that the second approach is better and the first approach is often a sign of someone either new to threading in Java or new to Java in general. This isn't a fatal flaw, although because of the lack of flexibility created it can cause issues with future development, but it is a bad sign.

#2 Threading code using join

This point again refers to a Java specific threading implementation issue however the basic concept here (and what is wrong with it) actually applies to multi-threaded code in general. join is a java method that let's you "attach" one thread to another and what that means is that the execution of thread A will stop and wait for the execution of thread B to complete.

Now if you stop a minute and think about it there's something very wrong-headed with the idea of join. Threading allows a program to do work in parallel whereas join implies doing work in serial, aka one thing at a time. The concepts of parallel execution versus serial execution are of course as contradictory as they appear and that's why this is a sign of bad threading code.

There are some proper uses of join but they are very few and far between. In every single case I have seen where join is being used either the code should be designed to use a queue or it simply should not be multi-threaded at all. The queue related issue is most common, it is fine to have multithreaded code where different parts wait for output from other parts but using join is the wrong solution because among other things it doesn't scale very well and it demonstrates that the original coder/designer didn't really understand basic threading patterns and design.

#3 Using Thread.setPriority

Again this is a Java specific implementation detail but similar concepts exist in other languages. Setting the priority of a thread versus other threads acts as a hint to the VM and the OS that one thread should be given more resources (processor time) than another thread.

Use of Thread.setPriority is a major red flag that in 99.999% of cases means that the code in question is very, very broken.

Why? Well because code that uses Thread.setPriority is almost always doing it to control program flow and execution aka the programmer wants one thread to start or finish it's work before another lower priority one. And let me be totally clear, that is very wrong and means the code is broken.

To begin with it's worth noting above where I said "acts as a hint" because that's a key point in understanding why this is a bad idea. Setting a thread's priority may in fact do nothing so relying on it for timing in order to get the program execution order correct is, by specification, incorrect. Where this gets particularly nasty is that people don't seem to understand the specification and sometimes, it can work the way the programmer might expect, a higher priority thread finishing it's work before another but when it works "correctly" that is based purely on luck and not because the code is actually correct.

This all leaves the question of why setPriority exists as a method, there is a use for it but it is very rare. It is of course hilarious to note that any time you find setPriority and mention that it's wrong the coder in question will tell you that this is one of the rare cases. Rubbish. The only correct use of setPriority is when, for performance reasons a certain task (thread) may be fine running only during "idle" times but at some other times it is better to run at a higher priority than other threads. An example of the correct use is a task like garbage collection in Java. The fact is though that almost all threading tasks do not fit in this very narrow use case.

Ultimately you can be sure that if a program isn't working correctly and if it uses Thread.setPriority it shouldn't be. It simply is only to be used in very specific cases in very specific code by expert level programmers.

Use of Thread.setPriority in non-functioning code is a sign a complete re-write will almost certainly be needed. In most cases the problem the code was trying to solve was properly handled with a queue of some sort.

#4 Multiple execution flows

This topic is hard to pin down to one thing to exactly look for but it's a critical topic all the same. Here I am looking for code that obtains "locks" on objects in different orders. For example if a thread sometimes obtains locks on object A and then object B but other times obtains locks on object B and then object A.

The reason this is a bad sign is that these sorts of multiple execution flows/paths are what leads to threads deadlocking. Code that acquires locks on the same objects but in differing orders is prone to thread deadlocking issues because thread 1 has a lock on A and is waiting for B while thread 2 has a lock on B and is waiting for A. If the code is better designed then the locks will only happen in the one order, thus deadlock will not occur because you won't have two threads waiting on resources locked by each other.

As I said this one can be difficult to spot, in many ways the easiest way to spot it is when a program locks up because of thread deadlocking but it's a particulaly bad sign when it's there. Code and programs that can deadlock is a sign of a fundamentally broken design that must be re-written. In short if code is deadlocking you can almost expect that code to have be totally redone to fix what may seem as a minor problem

Recovering from OpenOffice

Tue, 02 Jun 2009, 19:49:00 EST

Following up from a previous post I actually discovered that the OpenOffice installation had done even more damage then I had thought.

Essentially outside of running directly from a cmd prompt no program that used Java worked any more. This included Tomcat, Limewire and a number of stand-alone programs deployed as self-executing JARs. What was really brutal is what happened to both IE and Firefox browsers. Going to any page with any applet caused IE to enter some sort of loop and eventually throw up an error page saying the site in question was broken. This was actually better than Firefox which would just silently crash and vanish. In a surprising turn of events Eclipse was not affected, I think I would have discovered the problem earlier if it was.

I have now fixed my system and the steps I took to do that were

Uninstall OpenOffice
Uninstall Java Runtime installed by OpenOffice (jre 6 update 13)
Uninstall older Java Runtime (jre 6 update 12)
Restart system
Install Java Runtime (jre 6 update 13)
Restart system

And now everything works.

So as near as I can tell the OpenOffice installer installed a new runtime (update 13) but did something very odd with the configuration. Now the idea that it would break Tomcat for example is not good but I sort of understand it. Breaking Java in all browsers though? I am fairly unimpressed.

So as it turns out OpenOffice couldn't really help me the way I wanted anyway but on the basis of my experience I would really recommend people to not use it, at least on Windows. It's a pretty nice little product, and free, and it does work well on Ubuntu at least and it did work well on Vista but this installer nonsense is just unforgivable amateur hour.

The other day...

Thu, 28 May 2009, 22:43:00 EST

I installed OpenOffice. I didn't want to use it really but wanted to see about using some of it's libraries for something else. That turned out to be useless.

Today I discovered that the Open Office installation program broke my Tomcat.

Ho Ho Ho

I am not pleased. I think I may have to uninstall OpenOffice and Java and reinstall Java before it will work again. Tomcat seems totally lost and a reinstall didn't help.

Added Java to Category List

Tue, 26 May 2009, 21:01:00 EST

I was finding, fairly unsurprisingly that the number of Java tagged posts was increasing so I have created a category for it instead. I migrated all the existing Java related posts that were tagged as such (and a couple that weren't) to this new category.

As an aside my internet connection is still pretty flaky, I was hoping for a fix today but no such luck, maybe later this week I'll get lucky. It's pretty hard to work on remote projects when your connection is down more than up.

Proper exception handling can't wait

Mon, 25 May 2009, 11:09:00 EST

The most overall common mistake I see in Java code posted in forums and discussion groups is bad exception handling. By bad exception handling I mean seeing the following.

try{ //some code here }catch(SomeException se){ }

In other words, no actual handling of the exception, no printing of the stack trace. Nothing.

When told to fix this it's not uncommon to get back responses like

I know the exception handling is bad but I'm going to fix it later

I find it very difficult to understand the sort of errors in logic that can lead one to make such a statement. But let's clear this up. No. You don't add "better" or "fixed" exception handling later, you add it now.

The fact is that the sort of nonsense posted above is not just useless but actually hides problems from you, and during development and testing, well, you really do want to know about problems. An empty catch block is just laziness but worse a sign of some stupidity because swallowing exceptions only means that you will have no real clue what is working and what is not.

Does that mean you have to do "everything" that you will eventually do in your catch block from the get-go? No. It's okay to improve exception handling later, throwing up dialogs to the user, logging the exception, whatever. But at the very minimum a catch block must always have a call to printStackTrace. That will at least tell you, while you are developing and testing, that a problem actually happened and where it happened.

And if you don't feel like doing that? Then quite simply don't catch the exception at all. Simply add a throws clause to the method and let the exception be propagated to the caller, etc. Quite frankly you can throw right back out of main during testing if you like. It's not perfect but again at least you get a stack trace.

But whatever you do, don't swallow exceptions. Ever. I don't care that you're going to "fix" it later or whatever other lame excuse you have, you simply cannot debug code that swallows exceptions so just don't.

Connection woes

Sat, 23 May 2009, 09:17:00 EST

sigh

A sure sign of summer here is that my internet connection has gone haywire. It's a bit ... not good.

I am having what is known as a dry loop DSL installed because I believe a good portion of the problem here is the physical phone line, which is of not very good quality. But there actually seem to be multiple gremlins at play. I believe both the modem and my wireless router are overheating, and I have a feeling that at least one of my ethernet cables really needs replacement.

Which brings us back to why the weather. Well. I think when the humidity reaches a certain point the line quality decays even further. This isn't totally impossible I don't think and my working experience goes a long ways to proving that theory. But I think what's really happening is that the degrading in DSL signal is sort of the straw that breaks the camel's back. So I think part of the solution will be figuring out how to keep the modem and router cooler as well.

Anyway, not really fun stuff and pretty annoying to be honest. Hopefully by this time next week I'll have a connection that works for more than 5 minutes at a time once an hour.

Why I hate Apple

Wed, 20 May 2009, 11:56:00 EST

Before I get started here I just want to say this. If you have a Mac and like it, then that's great and personally I would prefer just to run some Linux variant all the time myself. This post is not about why an existing and happy Mac user should use another kind of computer.

What this post is about is blowing up some stupid myths that are trotted out time and time again by Apple fanbois (and girls) to try and convince PC users that they should drop their PC for a Mac. Because frankly I get tired reading the same old nonsense over and over. So from now on when I see it I am just going to link to this post.

Myth : Mac hardware is better/different

Truth : No, Mac uses the same basic hardware as any old PC.

This myth is a good place to start because there was a time when it was actually true. Once upon a time Macs did use a different processor architecture (ppc) than PCs (x86). The RISC (Mac) and CISC (Intel) architecture debate is really beyond the scope of discussion here but suffice to say that RISC did offer certain benefits at a certain level of performance but with increases in speed and improvements and other changes the long term win has gone to CISC. This is why, among other reasons that since 2006 now Apple has been using Intel (x86-CISC) style processors in their products. All of which means the hardware is the same stuff as is in any PC.

Now there are still a few complications, like the fact that Apple likes to use proprietary outputs for video and various data exchange formats but the bottom line is that (a) the value of these proprietary formats from a customer perspective is at best highly dubious and (b) these relatively minor differences aside the hardware in a Mac and a PC are in fact the same and are the same in the most important way, the processor.

And a quick note on the dubious comment above. It would seem that most of the reason that Apple uses so much proprietary bits is so that they can force obsolescence quickly and force customers to constantly purchase the "newest" Apple hardware bits that will then in turn be made "obsolete" soon enough. Essentially Apple uses proprietary hardware so that they can use obsolescence as a marketing and sales tool.

Myth : Mac is cost comparitive with PCs

Truth : No, to get a reasonably equivalent system on a Mac vs a PC will cost you at least twice as much and usually much, much more.

This myth is a funny one because it doesn't get explictly trotted out by Mac cheerleaders but is almost always implictly present, along the lines of "Mac might cost a *bit* more". Double or triple the price is a bit? It's no wonder that the cheerleaders don't want you to know the truth about this one, although frankly I think most of the time a lot of these people are pretty divorced from the truth/reality to begin with.

At any rate apples to apples (ha ha), or as close as one can get, price comparisons are important because for a lot of standard users managing costs will mean reducing the performance capability of the system. It does nobody any favours to try and price compare a Mac and PC when the PC has two or three times the raw processing power. But let's look at a real world example shall we? (All prices listed below are Canadian dollars)

A new PC from Dell, with a quad core 2.66 GHz processor, 8GB of memory running at 1066MHz, 1 TB (1000 GB) hard drive, 512 MB graphics card and 24" flat panel display prices in today at $1914 before tax.

The same hardware from Apple, same processor (quad core at same speed), same memory (speed and amount), same size hard drive, graphics card and display prices in today at $4617 before tax.

Yes that's right, for the exact same hardware configurations Apple charges $2,703 more than Dell. And while $2,700 may or may not sound like a lot to you consider this. For the same price as the Apple equivalent you could purchase 2 of the exact same hardware based machines from Dell and still have $789 left over. Any way you look at it that is not an insignificant price difference.

But is it worth it? Well it's hard to see why it would be. If you are a long term Apple user and use Apple based software then really the Apple is probably a better choice for you, and the more than double cost is simply the cost of your doing business. But for someone choosing between a PC and Mac with all other things being equal? Simply put the answer is no. The more than double price markup is due to one thing, the Apple logo on the side of the case. Now for me a logo stamped on the side of a case is not worth $2,700 but perhaps it is for you.

One last point on this myth I want to make, just to short cut any whining. I chose a Dell machine with Vista to get a reasonably well known company and some equivalence with support and additional services between the two prices but you can do much better in price with a PC if you like. Starting by running a Linux OS instead. So the truth of the matter is if you feel the Mac price I listed is somehow unfair to Apple you should keep in mind that the PC price is actually relatively high but I tried to be as fair as possible to Mac.

Myth : Macs don't crash and PCs do

Truth : Happy little commericals aside this is pure and simply marketing rubbish.

It currently seems pretty "cool" to trash PCs, or at least Windows based PCs because unlike Macs they just crash "all the time". But is that actually true?

This myth actually has two parts, the first part of the myth is that PC == Windows, which considering Macs are running PC hardware has a certain irony. I'm pretty sure though that when Mac fanbois preach this myth they're not including my PC running Ubuntu. But for the purposes of argument let's just pretend, like those fanbois, that Linux doesn't exist and move on to the second part of the myth.

So does Windows crash more than Mac? This is a tricky one to answer, not because it's true but because it's difficult to prove either way. Certainly both sides have put out lots of propaganda studies to show that the statement is true or false. My personal experience suggest that the opposite is in fact true (Macs crash more than PCs). But putting aside propaganda and my personal experience what are we left with?

The reality of crashes is that much more than the OS the software used and how it is used matters. One of the problems in proving this myth one way or another is that one would need to compare equals and you just can't. Software that was originally written for Mac and ported to Windows is just going to be less buggy on Mac. Conversely software written on Windows and later ported to Mac is just going to be more buggy on Mac. And the expertise and behaviour of individual users is very hard to quantify. All of this provides the grains of truth that fuel this myth and those who would argue the opposite, it's not hard to picture how a long term Mac user with Mac originating software could find Windows a crashy and frustrating experience, but the fact is the reverse is true as well.

At the end of the day what one is really left with is that in terms of stability there is no major inherent difference, improvement or flaw between Mac and Windows. The OS's are different, and so is the software, and so is the experience of users interacting with the systems. But different just means different, not "better".

Myth: Macs are safer than PCs

Truth: Partially and in a significantly noticably way true but in total mostly false

There is no place to begin other than discussing viruses and there's no question that an unprotected PC is far, far, far more likely to be infected with a virus than an equivalent Mac. So on this front the myth is in fact true.

The problem is, and this is where this myth becomes the most dangerous of all, is that viruses alone aren't the only issue and often the reasons for PCs being more likely to be infected is badly misunderstood.

The reason that a PC is more likely to get a virus than a Mac is simply this, there are far more viruses written to infect PCs than there are to infect Mac. This is really a simple point but the truth of it seems to be lost on a lot of fanbois. Here's a quote from the Apple site in a FAQ question about are Macs secure...

While no computer connected to the Internet is 100 percent immune to viruses and spyware, the Mac is built on a solid UNIX foundation and designed with security in mind.

There are some truth in that statement that seem to slip fanbois attention. Starting with the fact that you cannot make a claim that Apple's are 100% virus, hacker, etc proof. They are not. And it's also interesting to note the UNIX reference which really means that any time one trots out a Linux version as an alternate to Mac the "security" issue dies a quick, inglorious death.

But the sad part of this myth is how so not true it really is. The virus issue is very noticable point, but is virus protection all that makes a computer secure. No.

Let me repeat that in case it isn't fully clear. No.

No operating system for example is going to protect a user from phishing attacks, or personal identity theft, etc. The fact is that the only person who can protect a user from those sorts of problems is, the user. And this is where this myth gets dangerous because there seems to be this idea among some that if you use a Mac you are somehow "safe" from all these sorts of unpleasantnesses but you are not. To my mind encouraging users to think that they don't have to be or shouldn't have to be vigilant because they chose OS X over OS Y is a really, serious mistake. Security of your personal information is something every user must always be vigilant about no matter what sort of computer they have.

Which leads into my final point on this myth and that is network security and patch and update frequency. At this point the most major security problems are arguably networks themselves and not the actual computers on them. You can have a locked down Mac but if you have an open wireless network I can do nasty things to your DNS and do a lot of very not nice things to you including stealing your personal information and otherwise getting control of your machine outright. Significantly on this specific issue Apple has actually proved to be pathetically slow in releasing patches for network security related issues. In the DNS cache poisoning crisis of last year Apple took significantly much more time than any other vendor, including Microsoft to introduce a patch to fix this serious security hole.

So all in all, yes it's true, a Windows PC has a much better chance of getting infected with a virus, but despite what fanbois might want you to know security doesn't end with virus protection. And the fact is that when security flaws are discovered Apple is actually worse than Windows most of the time in addressing these flaws.

Conclusion

I want to restate what I said in the first place. If you have been using Mac for some time, and/or you just use a lot of Mac based software, well, then you should continue to do so. For such a user Apple is a good choice and I don't doubt that migrating to a Windows PC would be a frustrating and annoying experience for you. But let's not confuse the real issue here, which is this, if you are not in the above category a Mac is not for you. Buying a Mac will cost you more money and has very few benefits and a whole pile of drawbacks. Different is different, but different does not mean better and I hope in this post that some of the myths espoused by Mac lovers as to why it's "better" have been exposed for the lies that they are.

Extracting DOCX content with Java

Tue, 19 May 2009, 15:00:00 EST

I'm trying to read DOCX (Word 2007 file format) files in Java. My goal is just to be able to index the content so I only need to be able to extract the content, without formatting concerns. I looked around a bit but Apache POI doesn't have DOCX support yet (coming soon and in theory in some sort of pre-alpha preview) and the one other tool I found has a site that's broken.

But I figured it's basically XML so how hard could it be. And the answer is that if you want to extract the raw textual content it's pretty simple.

If you don't already know a DOCX file is really a zip archive. The zip contains a number of files but the key one for content is word/document.xml Then it's simply a matter of parsing that XML document and reading all the content type tags.

So below is some code that shows how to do exactly that. Again, no formatting data is preserved and I should note that this will not pick up contents from headers and footers. Picking up header and footer content is a bit more complex because those are in different XML files and may or may not exist depending on if headers and footers actually exist in the original document.

Hope this helps someone else get started with whatever they need too.

import java.util.zip.ZipFile; import java.util.zip.ZipException; import java.util.zip.ZipEntry; import java.io.InputStream; import javax.xml.parsers.DocumentBuilder; import javax.xml.parsers.DocumentBuilderFactory; import javax.xml.parsers.FactoryConfigurationError; import javax.xml.parsers.ParserConfigurationException; import org.xml.sax.SAXException; import org.xml.sax.SAXParseException; import java.io.File; import java.io.IOException; import org.w3c.dom.Document; import org.w3c.dom.DOMException; import org.w3c.dom.NodeList; import org.w3c.dom.Node; import java.util.List; import java.util.ArrayList; public class DocxExtractor { public static void main(String args[]){ ZipFile docxfile = null; try{ docxfile = new ZipFile(args[0]); }catch(Exception e){ // file corrupt or otherwise could not be found e.printStackTrace(); return; } InputStream in = null; try{ ZipEntry ze = docxfile.getEntry("word/document.xml"); in = docxfile.getInputStream(ze); }catch(NullPointerException nulle){ System.err.println("Expected entry word/document.xml does not exist"); nulle.printStackTrace(); return; }catch(IOException ioe){ ioe.printStackTrace(); return; } Document document = null; try{ DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance(); DocumentBuilder builder = factory.newDocumentBuilder(); document = builder.parse(in); }catch(ParserConfigurationException pce){ pce.printStackTrace(); return; }catch(SAXException sex){ sex.printStackTrace(); return; }catch(IOException ioe){ ioe.printStackTrace(); return; }finally{ try{ docxfile.close(); }catch(IOException ioe){ System.err.println("Exception closing file."); ioe.printStackTrace(); } } NodeList list = document.getElementsByTagName("w:t"); List<String> content = new ArrayList<String>(); for(int i=0;i Node aNode = list.item(i); content.add(aNode.getFirstChild().getNodeValue()); } for(String s : content){ System.out.println(s); } } }

I notice

Fri, 15 May 2009, 22:56:00 EST

... that over time the "community content" sections of the MSDN docs are filling up with stupid people posting questions. Which is pretty much what I said would happen.

"Community content" or comments sections do not belong in documentation. I don't know why this is such a hard concept to understand. Seems some people feel that if users can't post content to a page (preferrably with AJAX of course) that it just isn't a proper webpage.

Just so much nonsense.

JDBC Best Practices

Fri, 15 May 2009, 08:51:00 EST

I have had a few posts before relating to some JDBC best practices but I thought I'd list the top five all in one post. These are good ideas for making your JDBC code safe, secure, stable and portable.

Properly Close Your Resources

I discussed this point in a previous entry you can find here but is bears repeating as this is the biggest source of bugs in deployed applications that I have seen.

You must always close all your JDBC resources (by calling close) in the opposite order that you create them. Failing to properly close your JDBC resources can lead to dire consequences including the total lockup/failure of your application. Or worse.

And properly closing means calling close. Setting JDBC resources to null is a major mistake and a sign of a very unprofessional programmer. Setting a JDBC resource to null is not equivalent to calling close (again for details see this).

Use a Connection Pool

Unless your application is a one connection loading/filtering/reporting stand-alone special using a connection pool is a good idea. For everything else use a connection pool, and in case it isn't clear if you have a J2EE project then that means you. When you have multiple connections, connection pools improve performance. And really enough said.

What I should add though is that you should use an existing pool, don't try and implement your own, which some people seem to try for some unknown reason. There are many existing, working implementations, including many free and open-source ones. These pools work properly, in managing stale and closed connections. Use them.

Use PreparedStatements

This is another point I have addressed repeatedly including the SDN Share site but it can't go mentioned enough.

Proper use of PreparedStatements makes your code more portable, *may* improve performance but most importantly makes your code safe. Any time you have any query with user supplied parameter it is a JDBC best practice to use a PreparedStatement (or CallableStatement) and bind the parameter at runtime to it. Doing anything else is a serious error and anyone who tells you differently (and sadly there are people who do) doesn't know what they are talking about.

Making your code safe from SQL injection attacks is a simple thing to do and it's always the right thing.

Don't use SELECT *

This practice is one that is not specific to JDBC really but a general best practice for SQL programming. When you have a SELECT query you should always specifically list the columns you want and not use *. Listing your columns means your code is protected against changes to the layout of the table (which does happen) and it's also more self-documenting.

The bottom line is that using SELECT * is just lazy and it makes your code more brittle than it needs to be so why do it. It just makes sense to take the 5 seconds and actually list the columns you want.

Keep your SQL Portable

Finally it's a JDBC best practice to make your SQL as vendor (aka database type) neutral as possible. You can't always be perfect, sometimes, especially with stored procedures it simply isn't possible, but you should try as much as you can to only use SQL that can be used on any SQL compliant database.

There are three advantages to this. One, your code will be easier to port if the database ever does change. Two, your code is less likely to run into problems on database upgrades. This is a point not too many people seem to consider but upgrades and updates do happen and syntax for vendor specific implementations can change between versions. Three, your code is easier to pick up and follow. In theory JDBC code can be read by any JDBC programmer, you don't want to have to have a SQL Server programmer because you heavily used Transact-SQL.

In line with the above, if you really do need to use vendor specific SQL then consider using stored procedures (CallableStatements) as much as possible. The idea here being that if you must use vendor specific syntax you can properly refactor the database specific code into the database and leave your generic JDBC code intact.

Simple PHP RSS loader

Sun, 10 May 2009, 10:22:00 EST

A little while ago I wrote a simple RSS parser and loader in PHP for a client. The purpose for me was to read an RSS feed and copy the contents all so the clients blogspot blog can also be presented on their own website.

Anyway here is the main guts of the script, it uses cURL and XML/DOM.

<?php $ch = curl_init(); $feedurl = "http://yourfeedurlhere"; curl_setopt($ch, CURLOPT_URL, $feedurl); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); $blogfeed = curl_exec($ch); curl_close($ch); $blog = DOMDocument::loadXML($blogfeed); $entries = $blog->getElementsByTagName("item"); foreach($entries as $entry){ $pubdatenode = $entry->getElementsByTagName("pubDate"); $pubdate = $pubdatenode -> item(0)->nodeValue; $adate = date("Y-m-d H:i:s",strtotime($pubdate)); $titlenode = $entry->getElementsByTagName("title"); $title = $titlenode -> item(0)->nodeValue; $descriptionnode = $entry->getElementsByTagName("description"); $description = $descriptionnode -> item(0)->nodeValue; } ?>

So inside the foreach loop there are three variables parsed out of each item in the feed. One for the title (title), the posting date (adate) and the description or content field (description). I then take that date and optionally update a database with the information but you could do whatever you need to at that point. I haven't extracted the link or comment fields because I either already know that information or don't even really want it, again you could extract that as well if you wished, the same applies for other channel entities.

At any rate there are other, more complex libraries, out there that will turn the feed into a list of objects for you but for my needs a simple solution works well so it might help someone else as well.

MySQL and DBCP for Tomcat 5

Sat, 09 May 2009, 19:35:00 EST

I find that between each version of Tomcat working totally differently and the fact that the documentation is wildly inconsistent and often misleading setting up a connection pool on Tomcat can be a big headache. On top of everything else I found when Googling that you get all these hits that aren't always totally useful or point you at the documentation that's not quite all there in the first place. Now that I got a DBCP pool up and working on Tomcat 5.5 with MySQL I thought I'd describe it here.

So if you're looking for help with a Tomcat 5.5 DBCP and MySQL connection pool you are in the right place. If you have another Tomcat version sorry but you're likely out of luck. Again different versions have different ways of doing all this.

Step One - Get all the Jars in the right place

You need two jars for a DBCP connection pool with MySQL. One is the DBCP jar called naming-factory-dbcp.jar. Tomcat seems to ship with this now so you should have it. It should be in CATALINA_HOME/common/lib

The second jar is a MySQL connector jar with the MySQL JDBC driver. When you download this jar make sure you download the correct (aka non debugging) driver. This jar also goes into CATALINA_HOME/common/lib

Step Two - configure your context

There are a couple of ways to configure your context (see the Tomcat docs for alternatives) but the way I will describe here is to create your own context.xml. This goes into your META-INF directory in your project, WAR and when deployed. For this example imagine the context is named DBTest and in a similarly named code base. Here are the contents for your context

auth="Container" type="javax.sql.DataSource" username="USERNAME" password="PASSWORD" driverClassName="com.mysql.jdbc.Driver" url="jdbc:mysql://HOSTNAME:3306/DATABASENAME" validationQuery="select 1" maxActive="5" maxIdle="2"/>

Step Three - configure your web.xml

You need to add the following to the web.xml for your application. This snippet goes inside the web-app element of your web.xml.

jdbc/mypool javax.sql.DataSource Container

Step Four - write some testing code

Here is a small snippet of code showing how to get a java.sql.Connection using what has been defined above. Note how the name above "jdbc/mypool" is translated into one we can lookup from the InitialContext.

Context ctx = new InitialContext(); DataSource ds = (DataSource) ctx.lookup("java:/comp/env/jdbc/mypool"); Connection conn = ds.getConnection();

Step Five - Deploy

You can now build and deploy your web application to Tomcat 5.5 and you should now have a functional DBCP connection pool running to MySQL.

I hope this helps someone, I think it would have helped me.

Tracing 316.70.50.1

Mon, 04 May 2009, 09:45:00 EST

As computers and the internet are so much a part of our lives it's not a great surprise to see technology becoming more a part of the fabric of stories in entertainment media. Television shows, movies, both show the use of technology in good and bad lights. So given that and the current popularity of police and forensic work themed shows it's not terribly surprising that the use of the internet and computers are often shown helping to capture criminals.

One of the most popular uses in this regard shows the police (or the like) using posts on a website, or emails, or other internet activity to find and locate a criminal, right down to their front door. For example

Person A : Oh our suspect posted on forum X yesterday.

Person B : Can you find them?

Person A : Sure no problem. We'll just *click* *click* get their IP address and now find them (picture of geolocation map zooming inexorably in to a specific address. Got 'em

There are actually a number of myths prevelant in the approaches shown and I think I'd like to bust them here. I should be clear I'm not picking at the myths here because I want to nit-pick every piece of staged entertainment. But I do think that the false portrayal given in examples like above are harmful to the general public in terms of understanding real issues regarding privacy, big brother and security on the internet.

Myth #1 - Anything you do on the internet identifies your IP address clearly

This the first problem with the way the use of technology is portrayed, the ease with which the investigators can get a valid IP address. Often enough in the scenarios presented investigators are able, in seconds to get an IP address based on a forum post or blog entry. The problem is that if that information is retrievable (and there are some cases in which it really wouldn't be) more often then not it's going to take significant time to find it. For about 99% of cases it would take serving the related ISP with a summons to produce the log files for the relevant period. In other words this information isn't available one click away, again assuming it's available at all.

Sometimes other sources are used for identifying IP addresses, email seems to be a popular one. If you've ever looked at the headers for an email messages you may have noticed that each message contains a number of IP addresses and this information is used by mail servers. So in this example surely if you have an email address you can identify the IP address of the sender? Right?

Well, not so much. An email may well contain the IP address of the sender but for a whole bunch of quite common reasons it may not. For one thing it is possibly (and relatively easy) to just outright lie. The ease with which one can insert or alter fake email headers and still have email sent is proved by the amount of email spam in existence. Any sort of competent criminal wouldn't be traceable via email headers.

Even with a less conspiratorial act emails can be useless for tracking orginal senders because the user used another system to send. As an example of this think of hotmail or gmail. Someone using one of these systems will not be traceable by email headers, the originating IP address used by the sender is never included as part of the headers. Which leaves one at best back to getting an IP from the company running that webservice, if it can be found again at all.

In short, it's not usually impossible to identify an IP address based on an email or forum post or other activity but more often then not it's going to be much more involved then portrayed. And in some cases it will be impossible.

Myth #2 - Using an IP address I can track you to your door

Assuming investigators were able to find a real IP address for you, overcoming the above and assuming you didn't use a proxy of some sort and number of other assumptions the use of this IP address to track you to your door is truly fantastic.

There are plenty of sites and services that use geolocation by IP to identify your country of origin. This does work, with some caveats. IP geolocation is done by looking up the location of the netblock owner. A netblock is a range of IP addresses and IP addresses are assigned at the internet level of things by netblocks. The problems with this though are two-fold. One the information is not always correct. Sometimes netblocks have been re-assigned and re-sold so many times that it's near impossible to find out what country the IP is actually in. Second though and more importantly is that again what that information tells you (when it's correct that is) is the location of the netblock owner. That is it will probably help to identify the location of the ISP that the suspect uses, it does not contain the information that could be used to supply a specific address for the suspect. Taking the IP and a summons to the ISP might enable one to identify the suspect address but again that's not a point, click and zoom excersise and involves the use of several assumptions.

At any rate, entertainment is great, and I'm all for the suspension of disbelief but don't get caught thinking that either big brother government or criminal elements can use information you "leak" when using the internet in any form to quickly identify and find you. It just doesn't actually work that way.

The flip side

Tue, 28 Apr 2009, 09:43:00 EST

Of my ongoing Ubuntu issues is of course the standard Windows security by hijinks.

Trying to install Tomcat 6 on my Vista box. Launching the install and Vista checks my permission, which is fine and the install runs okay until the service installation part where it dies with a fairly ambiguous error message. I was able to get it installed eventually by disabling the UAC (security in Vista), which also took several reboots to disable and re-enable.

So, I did get it working but what kind of OS

Requires reboots to modify running security models?
Requires me to disable all security in order to allow one thing?
Doesn't allow me, running as administrator, to do whatever I want even after asking me permission on top?
Has very little documentation, logs, usable error messages for figuring all that out?

And the answer is Vista. So it's the same old security rubbish as always in Windows, sort of useful but not really because it lacks granularity, requires totally needless rebooting and is totally counter-intuitive.

Faaaan-tastic.

Starting to irritate me

Fri, 24 Apr 2009, 09:31:00 EST

Ubuntu.

Like an modern operating system there are software updates and patches available for Ubuntu regularly. Sadly it seems every second or third update seems to hammer all my X-settings and drivers resulting in the display being limited to 1024 by 768 or usually worse.

It would be much better if it would stop doing this.

I won't say that Windows doesn't have its share of problems, certainly it does, but they tend not to have problems like this. To be honest it's this sort of nonsense that relegates Ubuntu to hobbyists and strong computer users and not becoming a real choice for most users.

It's annoying and I just wish I could use Ubuntu for more than a week at a time without the display (or sometimes other visual settings) going all awry on me. Is that really too much to hope for? Frankly, I'm disappointed and I certainly wouldn't recommend anyone to use Ubuntu instead of another OS at present.

As seen on the internet

Wed, 22 Apr 2009, 12:23:00 EST

I am quoted a few times in an article that was published today in Microsoft Home Magazine. The article can be found here.

It's an article on password security and directed at novice users but I think I came off fine for the target.

I was actually contacted about being a source for the article in December and I think the article was finished in February but it was just published today. They seem to take it (the online magazine)seriously, I was actually contacted by a third-party for fact checking purposes in February as well.

At any rate I think I offered some good tips, and password security is important to everyone so worth giving it a read.

Wildcat 2

Tue, 21 Apr 2009, 23:30:00 EST

I put up a new game, Pacific Wildcat 2, a sequel game to my original from last year. This version is longer, has new enemies and I think is more difficult to play. By way of comparison the last level of the first game has 59 enemies while the new one has 115.

I will leave the old one up as well because it's still fun to play and if you struggle with the new one you can always return to that one to win.

It was interesting picking up my old code and working with it. The game engine I wrote for the original is not too bad actually, it's not too often I find that you can pick up old code and not find a lot of stupid things in it. There are some things I would change for sure but mostly it's pretty good. The hardest part in developing the game is making the maps. If I am ever to do any more work on this then I would start by working on that.

Anyway, have fun. :)

Don't panic

Tue, 21 Apr 2009, 10:30:00 EST

Since the news broke about Oracle trying to purchase Sun I have seen more than one article or opinion piece about how this is the end of the world. To which I would say, absolute rubbish.

The panic itself centers around MySQL, purchased by Sun recently and I have seen articles claiming that the purchase of Sun by Oracle will kill blogs, many websites and even the internet itself (through some mysterious process). Couldn't possibly be more wrong. Here's why.

The deal isn't done yet

Before any guesses can be made about what might happen to Sun in general and MySQL in particular it's probably important to wait until this deal actually closes. The current economic climate makes it more difficult than usual to consummate large transactions such as this and let's not forget it was only a few weeks ago that IBM made an offer on Sun in a deal that then fell apart.

Credit difficulties could scupper this deal yet, never minding either or both parties getting cold feet. It's not outside the realm of possibility that upon closer examination of Sun's books Oracle decides against this deal or lowers the price, ala IBM and the deal breaks on that. I'm not saying that this deal won't complete, it may well, but it's more than a little presumptious to make dire claims about it until it does.

Oracle doesn't want to kill MySQL

One of the common assumptions made by those panicking is that Oracle wants to "kill" MySQL. In some conspiratorial theories this is one of the reasons that Oracle is buying Sun. Again this is so much nonsense.

For starters Oracle isn't going to buy Sun for 7.5 billion dollars to turn around and destroy 1 billion of their new asset which is what Sun paid for MySQL not that long ago. It would be an incredibly stupid business decision. Second, MySQL isn't competing with Oracle anyway. More on this later but the idea that MySQL is truly an competitor to full enterprise databases is, and always has been, a myth.

The bottom line on this point is that anyone claiming that Oracle is buying Sun to kill MySQL is making two really bad assumptions. One that Oracle is willing to destroy 1 billion worth of assets and two that they see MySQL as a competitor, neither of which are very sensible.

Even the worst case scenario is not that bad

So let's for a moment put aside all the above and assume Oracle does buy Sun and then does actively seek to sink MySQL. Perhaps they stop development. Or put forward a date in which there will no longer be support offered. Is this really the end of the world?

No.

For one thing existing deployments are not really going to be affected by such decisions, at least in the short-term. So the idea that this will "kill" existing blogs, for example, is just so much poppycock. Such a decision would affect future development for sure but purposefully confusing future development and existing solutions is pure FUD.

In some ways the long-term death of MySQL wouldn't really be such a bad thing. There are plenty of other, real enterprise, databases out there, including other open-source and free ones like postgreSQL.

Look, there is no doubt that MySQL did fill in an important role in the open-source market but that doesn't mean that it is the open-source market. And to be honest there are too many people, in my view, who see MySQL as something more than it is or was and think it's the solution to all life's problems. Or at least database problems. And despite the myths it never ever was.

All in all, if this deal goes through there may well be some changes. But to make outlandish claims based on three ifs and a maybe with some dark conspiracies thrown in is uncalled for and way off-base.

Added Categories to RSS feed

Sat, 18 Apr 2009, 00:00:00 EST

I added category tags to the entries in my RSS feed. This means that all the entries are rss-tagged with the category and any tags they have. Adding these tags means that for readers that support them at least the blog content is easier to find/search because it acts like an index.

In conjunction with this I improved my AJAX based RSS reader so that the index of category tags is actually displayed in alphabetical order, like, an index. (Duh!)

On that note if you haven't seen my RSS reader try it out, it's pretty cool. I think I should work on the CSS a little more and perhaps add some sort of way for users to customize it. I mean the display needs work, but the functionality at least is there.

Week of hell

Wed, 15 Apr 2009, 00:00:00 EST

Next week will be my week of hell. On Tuesday morning I am having a cavity filled and on Friday I'm having surgery to have a wisdom tooth removed.

That's at least two too many dental appointments in a given week.

I'm hoping I'll feel better at the end of it but I am not looking forward to it at all.

Cross-site-scripting

Fri, 10 Apr 2009, 00:00:00 EST

While there are many resources and discussions about cross-site scripting attacks (sometimes pretentiously described as XSS) the number of major websites that continue to be plagued by problems relating to it suggest it isn't a very well understood issue.

In that light I'd start by stating how very serious cross-site scripting injections are. If an attacker can inject any JavaScript code, in any form into your site then your site can be used by an attacker as if it were their own site and worse with the additional power of any credentials you use to authenticate users on your site. That means cross site scripting injection can be used by attackers to:

turn your site into a deployment vector for malware and spyware
use your site to exploit browser bugs to attack users operating systems
access information you store about your users, including reading, changing and otherwise stealing information that is considered private

In short, this is pretty serious stuff.

So it really is remarkable how many serious, commerical, web-application products are open to such attacks. For example Jive Software products (look at their client list).

From the programming side of things cross-site injection is actually a more complicated topic then it first appears. As a general rule of thumb though any time you allow vistors to post content that will appear in some form on your site you need to be very careful. I'd encourage more companies to hire experts (like me) to look for flaws like this.

Most people (although sadly not all) do have the basics covered in terms of not allowing script tags (and content) through wholesale but there are many ways of injecting scripts into code. As one example allowing users to post links can be very dangerous and I would suggest from what I have seen that most software designed to accept user supplied links does not do this correctly. As a second rule of thumb I'd suggest that one should closely examine any code that takes user supplied content to create HTML tag attribute values.

And what should you do if you're not a programmer. You should make sure your site is cross site scripting injection proof, again many aren't, and if your site allows user supplied content in some form chances are it's probably insecure. It's also noteworthy I think to consider the rammifications of both exploits and the cost in time and effort to make your code and site safe.

Do the benefits of making your site an interactive Web 2.0 special actually outweigh the costs? It would seem be the large amounts of insecure code out there that most people seem to think it does, but I would argue that's more a result of not knowing what the risks really are.

AJAX RSS Reader update

Mon, 06 Apr 2009, 00:00:00 EST

Following up on my AJAX RSS reader I made some improvements so now it can be used to read any RSS compatible feed. The reader is available here.

Now there is support for supplying your own URLs. Just type or copy/paste the URL of the feed you want into the Feed URL box and click the Load/Refresh Feed link.

An AJAX RSS reader

Sun, 05 Apr 2009, 00:00:00 EST

I threw together a little AJAX based RSS reader. You can see it at here.

My idea when doing this was that I wanted to make a simple reader that worked the same in different browsers. The default reader built in to browsers all do some very different things and I find that a bit annoying. My reader is loosely based on the IE reader which I find at least has the most functionality (more than FireFox or Opera anyway).

Some next steps with this would be

Fancy up the CSS. It all looks very simple right now but the L&F could be improved easily enough
Add server support for piping other feeds through. Ideally you could just type in the URL for a feed and this page would display it. There's a problem with that though because AJAX (rightly) restricts you to only talking to URLS from the same domain as the page. So I would need to make a special server-side script that will read the URL contents for you and pass it back to the page.

Still finding the way

Sat, 04 Apr 2009, 00:00:00 EST

If had to choose an internet enabled fad that we could all best do without it would have to be comments on stories posted on news (newspaper, tv, etc) sites. What a waste of space.

I recently saw a story about the baseball team and stadium in Toronto. The liquor control board has suspended the stadiums license to serve alcohol for two games for serving infractions. To translate, the stadium alcohol concessions were caught serving alcohol to obviously inebriated fans and likely some minors as well. So as punishment they won't be able to sell alcohol at couple of games. This is a system that has been around for a long time in Ontario (and likely other places as well but I can't speak to the details there) and it generally works. Restaurants, bars and public settings like stadiums make a good deal of money from selling alcoholic beverages but they must obey the law. Suspending licenses for infractions embarasses the serving company and hurts them where it counts, in the balance sheet.

The comments made by Joe Q Public about this story on the newspaper site where I read it mainly demonstrate that Joe Q Public seems to have a great deal of difficulty in basic reading skills. The comments were almost all raging about how dare the government interfere with their right to drink beer. Neverminding the confusion that would cause someone to claim drinking alcohol anywhere they want as a "right" it's mostly just obvious that the people supplying the comments don't understand how the system works at all or that the real villains in the piece are the stadium. I have to wonder how those claiming that the world is turning into 1984 because "big brother" is "interfering with their right to drink" would feel if their 14 year old daughter came home from a baseball game drunk. I rather suspect they'd be railing against the lack of government intervention in that case.

Point of all this being, as I am sure we have all seen, comments sections on news sites seem to act as a magnet for the ignorant to share their ill-thought out opinions. And quite frankly if all the comments sections were removed from such sites it wouldn't be any kind of loss.

I feel in a nutshell this case is representative of what's wrong with "the internet" as a whole. There is a general sense that "the internet", which really means www, is a tool/technology that promotes "democratic" ideals and empowers people. Well. It certainly empowers a lot of people to make fools of themselves.

I think to some degree there are uses for the internet for education of current events and involving people more in the world around them, we just don't have the right formula yet for doing so. I don't know what the right formula is either, I do know that allowing anyone passing by to leave an ignorant comment about a news story is not part of the right formula.

Upgraded zip tool

Sun, 29 Mar 2009, 00:00:00 EST

I have posted up version 2 of the MaxZip tool available here.

As I originally said on posting the first version, I wrote this tool because I wanted a zip tool that worked the same way on different systems and sometimes I work on systems that don't have a built in zip tool so it's helpful there as well.

The new version, which I re-wrote to be Java 1.5 compatible, features a number of improvements including

More (and I think better) options for directory structure preservation
Support for WAR file decompression
Support for console/shell only mode for both compression and decompression

The pure-shell mode I know I will find useful, I hope you do too.

And if you're looking for a good reason to use this tool yourself I would say this, for whatever reasons it performs both compression and decompression operations faster than the built in zip tools on the two platforms I tried it against (Vista and Ubuntu).

Stop blaming the computer

Fri, 27 Mar 2009, 00:00:00 EST

One thing that irritates me no end is hearing people say "I can't because the computer won't let me" or "I can't because the database won't let me" and variations on that. You hear it in person and sometimes you see it in print too as various "computer errors" are listed as the cause of some evil or another.

The reason it irritates me is this, it's not the computer's fault, or even the fault of "the database". The problem is with the software that's being used and/or the training the person has on the software. And the reason that's a problem for me is that putting the blame in places it doesn't belong has two drawbacks. One, it makes people think that computers can often have errors and two it makes people have needlessly low expectations when it comes to software.

The reality is that software and computers can work and can work flawlessly. Look no further than the skies for proof of that. Auto-pilot, crash-avoidance and other systems used in flight use systems that work, and have to work all the time. Interestingly when non-mechanical induced accidents happen they happen because of human error, the pilot made a mistake, didn't trust the computers or didn't understand how the computers were helping them.

Now obviously not all software has needs as demanding as flight control and so there's going to be a certain amount of bugs in any system.

Design flaws on the other hand. Almost all the time when people blame "the system" it's a flaw in the design of that system. I find it interesting how often when I present project estimate budgets the section on design and requirements gathering is the part that clients want to remove or lower. I find it funny at times because it's as if the client who doesn't understand the process of software development wants to dictate to me, the software developer, how the process should work.

Here's the thing, writing the code is obviously important, as is testing, and as is training. But if the design is wrong you're hosed.

KVM joy

Mon, 23 Mar 2009, 00:00:00 EST

Well I hooked up my new KVM and all is mostly well with both systems. My Ubuntu screen is still not as high a resolution as Vista but at least it's not squashed.

Vista is okay except for some strange networking problems that I can't explain. There's no error but I get served blank pages from websites (in IE and Firefox) and looking at the same site in Ubuntu (in Firefox and Opera) there is no problem. The connection is the same so there's something bizarre at an OS level here.

In other news, kind of a mixed bag.

I need to update a program I don't have the latest source for. :( I believe it was lost in the great XP crash of December. I decompiled what I have and compared it to the source I do have but there's a number of things that changed and decompiling doesn't make for the greatest source.

On the plus side of the ledger I am now a SCJP. I got my pretty certificate and a strange credit-card looking thing in the mail on Friday.

Finally, I need to go to the dentist. :| I think I have a cavity.

Trouble in tech town

Thu, 19 Mar 2009, 00:00:00 EST

So I just got a new computer. I really needed a new screen because my old one was a-quitting. But also I need Windows for two things, Photoshop and Office.

Anyway computer came today, I hooked it up to my KVM booted Windows switched back to Ubuntu and the KVM promptly died. I don't know why but I was trapped in Ubuntu.

Sigh

So got a new KVM now. It works but for reasons known only to the angry computer spirits and the creators of X (these are possibly the same) Ubuntu doesn't seem to "like" my new screen as much with the new KVM. So I'm stuck with a 1024 by whatever size in Ubuntu apparently and the aspect ratio is all wrong so everything in Ubuntu looks like Wile E Coyote after a 10 tonne weight has been dropped on him. That is to say rather squished.

So I guess tomorrow I'll be editing my x config files and hoping my new screen doesn't catch fire. Why must everything be so complicated! He cried plaintively...

On the plus side I found my Fiona Apple CD. The good one. I guess this is karma making some sort of effort.

Stopping Blog Spammers

Tue, 10 Mar 2009, 00:00:00 EST

Well, after some delay I have put up a sub-site with a blog spammer blacklist. You can find the sub-site at http://maxstocker.com/blogspam/.

There are actually a few resources on that site, including a FAQ, some tips for stopping blog spam and the list which is available in both HTML and CSV formats. This list is a blacklist of known blog comment spammer addresses, so you can use the list to help filter out spam comments on your blog or any blog you administer really.

Thoughts on selecting an IT vendor

Mon, 09 Mar 2009, 00:00:00 EST

One of the roles I am often asked to advise clients in is IT vendor selection. As such I have given thus a lot of thought and some experience and it is my belief that nothing is better than experience or failing that a good referral.

Really.

There is certainly some separation of wheat from chaff that one can do by looking at the vendors qualifications and having well thought out requirements. No question. But how can you really know that the final deliverables will be met, on time and within your expectations? There doesn't seem to be a good answer for that.

I think the common thinking is that requirements are where "it's at" for this process but that simply doesn't cover all of it. Particularly when contracting for a service. How can one properly iterate requirements for future services that are really unknown? Time for delivery? Time for response? And how about what the requirements in terms of time and experience your users need to have? I'm not sure how possible it really is to capture that all in writing.

At the end of the day there is no substitute for first hand experience with the vendor or at least a referral based on experience from someone you trust about that vendor.

These guys didn't make $11,668.00 on Google and neither will you

Sat, 07 Mar 2009, 00:00:00 EST

As part of my ongoing effort to expose shady operators for the scum they are a brief note about some of the ads displaying on Facebook.

One of the current shady ads reads something like "Would You Like to know how I got $11,668.00 by Posting Links on Google?" and links here. Not even sure where to start with this trash but it begins with the picture of a Government of Canada cheque for revenue that is supposedly coming from Google. What the product actually is is another problem. The links from the site lead to another page, I suppose eventually one would pay for information about Google AdSense. Information that's freely available by the way.

Well if you're not sure why this is a scam here are the reasons

The ad landing page misrepresents how Google AdSense works to begin with. You don't get paid for "posting links on Google" (whatever that gibberish is supposed to mean). You get AdSense revenue by hosting Google ads on sites with decent amounts of unique web traffic.
The site also implies that the money will start flowing right away. It won't. Google isn't stupid. And there is no, and I can't emphasize this enough, NO quick path to untold riches with AdSense. Google isn't stupid and this is a business and the real world. Getting $11,000 in AdSense revenue per month is certainly possible but you have to have actual real traffic and a lot of it in order to see numbers like that. To get an idea 50,000 pages served to real traffic in a month will get you around $100. (These numbers will vary 10-20% for a variety of reasons) So to actually get $11,000 in monthly revenue will require about 5.5 million page hits per month. And let's be clear, that's a lot of page hits and a lot of users. Scams like this often have "advice" about generating traffic in a variety of artificial ways but... you won't be fooling anyone. It takes 2 months after the end of a month before Google sends you any money. Why? So they can look for fraud, and when they find your fraud your account will be closed and you will get zero.
Even if there weren't the problems listed above this is a scam because there is no "real" information that they are selling that isn't already available for free. The only information that isn't available on the AdSense site itself would be the "advice" and "tricks" that violate the AdSense terms of service and if followed will see the termination of your account.

But wait. There's more. If you looked at the page you'd see it was "written" by some dude named "Richard Tucker" and there's a nice picture of him enjoying his new lifestyle with his family in Disneyland. Charming. Here's something though, there's another one of these ads running on Facebook titled like "Let me show you how to get $11,668.05 Free from the Canadian Government in 30 days" and that links to another site linked here. This page by one "Jeffrey Donahue" has a plan for scanning the government to get "free" grant money. Now I shouldn't be having to point this out but you aren't going to be scamming Revenue Canada for $11,000. More though we have the same scanned photo of a government cheque as on "Richard Tucker"s site and, oh, what is this, the same picture of "the family on vacation at Disneyland".

Some of these scammers, or at least these clowns, make such a poor effort it's wonder that anyone needs to be warned at all. But sadly they wouldn't be in the scamming business if there weren't people willing to get scammed. Don't be one of those people. I honestly hope that shining bright lights into the dark corners of the internet will help reduce the number of these frauds. Likely not, but I remain hopeful.

Light at the end of the tunnel

Wed, 04 Mar 2009, 00:00:00 EST

For the last day or so I have been interacting with a different IT support company for another client. A different support company from the one that has been giving me a great deal of heartache recently.

I am pleased to note that (a) they actually answer the phone/response times are good and (b) I am not being subjected to incoherent technical jargon from less-than-competent technical people when I ask a question. It feels like some sort of miracle compared to the long-delayed, gibberish ridden, incorrect nonsense I have been subjected to recently.

I think I may recommend this company to my other client as a replacement for their current outfit. I have a lot of things to do. I don't really want to have to spend a lot of time fighting with inept liars.

So maybe there is hope yet.

Another sign of doom

Mon, 02 Mar 2009, 00:00:00 EST

Previously, I mentioned how the use of the phrase "XML database" was a sign of portending doom. Well, I have another one.

Wiki based documentation.

If you're at all technical, or even if you're not, you've probably by now encountered a manual for something that is in a wiki format. Here's the bottom line on this. There is no surer sign that the vendor of the product in question was too lazy and/or cheap to produce proper documentation then having the primary documentation exist in wiki form. In every case I have seen of this it seems that the day before launch someone said "Hey we don't have any documentation!" and someone else said "wiki?" and the reply came back "Yeah, great idea, we can make it a collaborative effort". Collaborative effort in this case translates as "the documentation will be a incomplete, highly muddled mess but nobody will be specifically to blame".

The fact is that the wiki format is actually a pretty poor paradigm for technical documentation in the first place. Wikis are intended, and do have a more "organic" structure for starters, a concept that goes against the formal organization that is needed to make technical documentation actually usable.

Wikis also don't lend themselves well to the style of searching that documentation requires. Both of these elements make for a frustrating experience as users have to jump from topic to topic, and search often in vain for content that is misfiled, not easy to find at least and in some cases doesn't even exist.

It's also a failure to not have a team or person responsible for the content of documentation. An all too common feature of documentation in wiki form that I have seen is incredibly poor language and lacking key elements, steps and descriptions. Wiki documentation seems to rapidly become a dumping ground for the developers or salespeople to talk about the wonderfulness of the product rather than focusing on how to actually use it.

Finally, one of benefits of wikis is the collaboration they can make possible. Which is great, except again this has zero role in technical documentation. The last thing a technical manual needs is the confusion of being written by different authors, with different styles and voices. And allowing contributions by developers using the product? Even worse. The wikis I have seen like that rapidly turn into collections of "worst practices" and generally bad ideas of the kind that make me want to weep.

Look a wiki is a fine tool but it isn't a suitable tool for documentation in the first place and regardless no content will just "write itself". Don't be lazy. Write proper documentation that is organized by a central person or team who manages the structure, style and content. That makes for usable documentation, which in the end makes your product that much better.

Facebook API

Fri, 27 Feb 2009, 00:00:00 EST

Man.

I've been working for a bit on an application targeted at Facebook. Sadly this requires being able to understand the Facebook API. I've spent several days reading and gotten exactly nowhere.

My biggest problem is that there are two ways in which to build Facebook apps. You can use a thing called FBML or you can use iframes. I want (need) to use iframes. Guess which one is documented?

I'd rate my current frustration level at 8.5 out of 10.

More CRTC bungling

Thu, 26 Feb 2009, 00:00:00 EST

I have been watching the ongoing failure of the Canadian Do Not Call List with some bemusement.

In short, many people who signed up to the do not call list find themselves getting more calls than ever, including a lot more calls from criminal/fraudulent elements.

So why am I bemused? Because it's a government operation, and even worse a CRTC project, so the fact it's been totally bungled should really come as a surprise to nobody.

There are ever so many problems with the list, including what it is and how it works, which in some cases actually conflicts with out existing privacy laws (PIPEDA). And it was never really explained in terms of how it should work to Joe Consumer. So to begin with the expectations raised in the public mind weren't going to happen even if things did work out.

But they haven't worked out because in what has to be one of the most stunning acts of incompetence yet the CRTC is essentially providing a free, clean, list of contact information to tele-marketing agencies. Which seems a fine idea except that they made this list available to just about anyone which means that criminal set-ups were drooling all over themselves in a race to get the list.

Well done idiots.

There was of course a much better way to handle this. In fact several better ways. One was they could have imposed some regulation on the industry (you know, being a regulatory agency and all). For example they could have required that every tele-marketing company operating in Canada register with them (CRTC), and make a security deposit with them and then and only then get access to the list. Companies that then violated the DNC registry lose their deposit and their status as a legal tele-marketing operation. Then Joe Public can always inquire as to the legal status of a company phoning him.

Honestly this isn't that hard.

The CRTC meanwhile is reviewing the process and promises a quick response. By government standards this means that by mid-October they'll have a new and even worse idea.

Sometimes people ask me in my capacity as a database guy if I am scared of "big brother" government. And I tell them no. Because the government is way too clueless to be an effective monitoring agency of themselves, let alone anyone else. Bottom line, even if they have nefarious intent (which by and large I don't believe) they just aren't competent enough to pull it off.

But incompetence is a double edged sword, as in this case. While you don't need to spend many sleepless nights worrying about what "big brother" knows about you, you really, really, really shouldn't trust "big brother" to help you to protect your information either. Keeping your information private and safe really begins and ends with you.

Up and down

Mon, 23 Feb 2009, 00:00:00 EST

Well my mug arrived today. I need to use a bigger picture, the quality though turned out not too bad at all. I'm pretty pleased with it.

On the down side I feel like dreck. Some sort of cold/flu, I'm like the before picture in a medicine ad. Congested, achy, feverish and just quietly moaning.

At which point you say "Why are you blogging now then?" and I say "Yeah". I'm tired of sleeping and my body feels less achy when I sit, but all in all I think I'm going back to bed now.

Ick.

Don't lie

Fri, 20 Feb 2009, 00:00:00 EST

As a general word of advice for IT support people the world over.

Don't lie.

I can't be any clearer than that.

I've had a running battle with a fairly inept outsourced IT support company this past week. The stunning lack of competence is not good, the lying is making it all much worse.

The lying started when the issue was first reported and the response was that they (support company) had not made any changes, which they in fact had. That might be forgivable but as various attempts at jargon-speak and garbled information came along I asked days ago for them to stop that. Such nonsense would probably fool most people but it wasn't going to be fooling me.

Today the person responsible for the ticket decided to escalate it both internally (to their management) and externally (with the management at the company being supported). Sadly they did so with seemingly the intent of making me look bad but yet managed to get several major technical details totally wrong.

I somehow don't feel I should be having to explain how internet e-mail works to an IT support company in order to get an e-mail support issue resolved. Nevertheless if the IT support company doesn't know then stringing a collection of terms you don't understand together in the hope of appearing expert is really a mistake.

Especially when I already asked for such nonsense to stop.

It would be great if you know but if you don't know for god's sake don't try and pretend like you do. Good advice at any time, but even more when dealing with someone who clearly does know.

Blog layout updates

Wed, 18 Feb 2009, 00:00:00 EST

I added a couple of new navigation elements to my blog pages.

One I added the list of categories a blog entry is filed in to the end of the entry (below the list of tags for the post). Previously you could click the categories to see posts filed in each category but you couldn't otherwise see what categories a post was filed in.

Second I added a section titled "Recent Comments" to the left side navigation bar below the list of recent posts (or posts from a specific month). I mainly did this so that I can see recent comments as I missed one made fairly recently that was on an older post. This was now I can see when comments are made even on older posts.

All in all good I think although it makes the page a bit long on the left. I might remove the category listings from the side now as I have added them to the posts directly. We'll see.

100!

Mon, 16 Feb 2009, 00:00:00 EST

This is my 100th entry in this blog. Whoo-hoo.

I don't really have a lot of actual content for this entry because I thought 100 was enough of something to celebrate. I am happy that I have gotten this far and that I have been fairly consistent in updating this blog as well.

On a technical note this means my next entry will push the first one off my RSS feed (I think). I wonder how readers like Facebook will handle that?

On with the next hundred...

I predict the future (and try not to look foolish)

Sun, 15 Feb 2009, 00:00:00 EST

I am pretty avid reader of history and if there is one thing I know it's that as soon as people think they have "it" figured out, it turns out, they don't. It doesn't really matter what "it" is. It could be natural science, economics, politics or technology and computers. Actually it seems there's no more sure way of looking like a fool then declaring that something has been resolved forever or that a field has reached it's pinnacle and there will be no more room for improvement.

In this light I have been thinking about what the future holds for computers. I actually do think that when it comes to the desktop computing we've all grown accustomed to over the last 20 years a peak of sorts has been reached and perhaps even passed. It's unclear to me for example how the last 10 years of MS Office updates have made for a richer experience, shifting buttons and menus around isn't exactly a new paradigm.

I am however not so stupid as to think that there won't be in 10-20 years a lot of changes to computers and their uses so in that light here are my five predictions for where things are in 15 years time (2024).

Increased power

I suspect in 15 years that the power of computers will increase tremendously from the current state. Seems a safe guess when looking at changes over the last 15 years. :) I predict though that in 15 years we'll actually be seeing quantum based computers with some regularity and that coupled with all the multiple core stuff and increasing memory etc will mean computers that are 100s of times faster and more powerful then the ones today.

More AI

I think given large increases in processing power that it seems likely that there are changes to how we interact with computers in 2024.

I think in 2024, AI becomes a lot more common place and a part of basic interactions with computers. Essentially I suspect that brute-force type of AI systems will become more practical. The standard desktop-type computer will be able to co-relate data at speeds unthinkable today. Cross-checking massive amounts of historical data will mean that our systems will be better able to guess what we want, are doing, need help with etc. I realize this sounds sort of like clippy but imagine if it actually worked? Because it was based on your usage and even then could "learn" or appear to learn anyway based on your use? I think that's really my point here, being able co-relate data 100s of times faster will I think mean that computers and systems will have at least the appearance of learning, enough so that the difference between the appearance and actual learning isn't human distinguishable.

Dramatic interface changes

I think 2024 computers will be unrecognizable to us today mainly for their lack of keyboards. Now again I have a feeling that keyboards will still exist (they've existed in one form or another for a couple of hundred years and I don't see that changing in 15 years) but I think that computers of the future will be much more voice driven. Here's the thing, I type fast but if I could dictate this blog entry? Okay, so you can do that now, but not the way I want, or foresee. I'd like to be able to dictate at my rapid speech rate, have the computer get it all down correctly all while having my music playing like it is right now. This isn't really possible right now at least without being badly error prone but I think in 2024 faster computers will mean that speech recognition software will be able to handle difficulties like these much better than currently.

I also think our computers are likely to much more mobile and this includes the display as well. I imagine a setup whereby the computer is somewhere in my place and I can talk to it and view the display of the current application I am working on wherever I am inside. This means I can work in one place as now if I wish but I also could be working or doing other things while "typing" a blog entry.

The display particularly I don't know how it would work but I do think it would be a good thing and possible in the not too distant future, it means among other things that computers really fulfil their potential of "freeing" us up from menial tasks like typing and allow us to do other things at the same time.

Dramatic changes to security models

This change is one I think will come as a negative consequence of some of the above, namely computing power. It seems to me that the things like the encryption ideas we have today will have to change if everyday computers really do become a lot more powerful. Brute force techniques for example, while thoroughly impractical today might become a matter of much shorter time by then. Beyond that I also suspect that with better systems and more AI there may perhaps be all sorts of new attacks that reduce brute-force decryption times of current systems in ways we just don't know about yet. All in all I think it means that the security models we use now will be thought of as quaintly primitive in 2024 and the models used then will be unrecognizable to us now.

Dramatic increases in integration

I do think that one of the changes that will happen given increases in raw power and interfaces is that our computers will be generally tightly integrated into our homes and work. For example I think things as simple as light switches and wall thermostats will be almost gone and certainly obsolete by 2024. The same goes for things like keys for the office or home. I predict when you come home from work in 2024 you'll use your voice to unlock the door, the computer will turn on lights and change the climate according to the time-of-day, time-of-year and your known preferences. You'll start making dinner and ask the computer to play feeds or read news about items you've indicated interest in. And you'll cook a dish from recipe your system searches for when you give it a general idea of what you want or even just a list of ingredients that you have on hand.

Well there they are then. I wonder which ones will turn out to be actually true...

Why fulfilling demand is sometimes a mistake

Thu, 12 Feb 2009, 00:00:00 EST

When it comes to anything really but particularly software development, sometimes people feel that every whim should be catered to, every need filled and every request served. But so often it pays much better to address the real, underlying issue that is the source of the "need" in the first place.

As an example for sometime there was no way in Java to get the MAC address of a network adapter. Now to begin with as a caveat, "no way" isn't entirely true. One could do it, but it required native code or using executing another process (like ipconfig). But true enough there wasn't a Java API method to do it.

But there sure were alot of requests (read demands) for this feature.

So lo and behold in Java 1.6 Sun added a method getHardwareAddress() to the java.net.NetworkInterface class that returns the MAC address.

And all was well.

Or not

What's actually happened is the questions about how to get and use the MAC address have actually increased. The problem is that exactly zero of these questions now (or ever before) were based on a valid idea. Nineteen times out of twenty the "requirements" that lead to the MAC address questions are security related. That is that the questioner is attempting to use the MAC address to authenticate a user. This couldn't be more wrong-headed.

As a quick aside it's wrong headed for a bunch of reasons, not the least of which being it provides exactly zero-security. It is trivial to change a MAC address on most computers these days. Plus it has the added drawbacks of not working when it otherwise might and generally frustrating the real users all while doing nothing to discourage the malicious ones.

Back on point, I'm hard pressed to imagine a better example of when giving the masses what they "need" is actually a step-backward. It's never a good idea in my opinion to encourage bad security ideas and thoughts rather than addressing the real underlying problem. Which in this case is a total lack of understanding of what authentication means, how it works, and what it must have to be successful. Namely an authentication model that allows trivial spoofing is a failure.

Besides all this what about that one in twenty who does have a real need for the MAC address? Well to be honest one in twenty is high because I haven't seen one yet, but nevertheless such information was always available really. One just needed to spend a bit of time figuring out how.

Looking for that special valentine gift?

Wed, 11 Feb 2009, 00:00:00 EST

Not sure what gift to give your loved one this valentines? Want to show that you care and have good taste? Well look no further, just check this out.

Okay. Kind of lame. But I think it might make for a good t-shirt. I have to see how the image turns out, I think I might need a better quality one. I ordered a mug, we'll see how it goes.

Grrrrrr

Wed, 04 Feb 2009, 00:00:00 EST

I have to go soon to see some clients. These clients use the inept IT outsourcing company I mentioned in an earlier post.

Suffice to say that the IT company has broken the email setup on multiple computers. I don't know why. A few weeks ago they decided to change their own email authentication schemes for incoming mail (still not using SPF for some reason) and then when some people were having issues sending from home denied they had changed anything and tried to blame anyone else. That's the problem with one client and I suspect the problem here again with the second.

And the Exchange mess is still not resolved so there is increasing anxiety about that. I really think it may be about the time when I give the idiots at this IT company an earful.

It's one thing to be incompetent and break your own stuff, it's another when you start lying and fiddling with other people's things you don't understand until you break those as well.

We shall see how it goes...

Office 2007 Ribbon et al

Sun, 01 Feb 2009, 00:00:00 EST

In a word.

Why?

I had occasion to use Access 2007 recently, it was truly a terrible experience. To begin with the replacement of the standard menu items with what is essentially a collection of tabs is a good example of "User Experience" run amok.

I am sure that someone has done some study and shown that people who have never used a computer before in their life find the ribbon more user friendly. But of course people who have never used a computer before are not the ones who have been using Office previously. So what this really amounts to is the imposition of a new user interface paradigm, totally different than what's been done in previous versions of Office and totally different from the way any other application work has worked on the system in question.

I cringe thinking what the reaction of some of the long term, heavy users of Office will be when they encounter this mess the first time.

I also really wonder who exactly is being tested in user experience studies when it comes to the intuitiveness of the entire application. How is that I have used Windows alot, and Access alot, and even other windowing systems to some degree but I found Access 2007 really painful to use? They seem to have totally dispatched dialogs and instead have menus open quietly in the different quadrants of the screen. It really shouldn't take me 5 minutes and a lot of random clicking to figure out how to create a new database (file)!

Some of this might be forgivable if things actually worked seamlessly, but of course, they do not. Trying to open a database created with an older version of Access was a real exercise frustration. Upon opening it informed me it needed to convert the database ( a fairly common request from Access) but it then proceeded to quietly fail for reasons still unknown to me. I eventually discovered that while Access 2007 would not, for whatever reason convert the database it would import it. But for all the fantastic user experience-ness of this new version I had to figure that out on my own.

I'd certainly find it difficult recommending anyone "upgrade" to this nonsense.

Chrome Server : First impressions

Mon, 26 Jan 2009, 00:00:00 EST

Someone I know beta launched a new web-platform "thingy" today called Chrome Server. I've only looked at it for a little bit but here are my thought so far.

Okay, to start on a positive note it seems cool and the little I tried seems to work and for a beta both of those are good things.

On the other hand I'm not really sure what the point is. To begin with what exactly the service/platform/product is, well that's not entirely clear to me. Currently it seems to be a mix of (a) a product that lets you edit, manage a website through a browser, (b) a free (at least for now) hosting service, and (c) a new web application development language. I have a feeling the developers see points (a) and (c) together as a unique offering.

At any rate neither (a) nor (b) are anything new really. Web based editors of varying complexity have been around for some time now. So really what we're left with is (c). And how is the new development language? Well it appears to be JavaScript based, to one degree or another. I'll have to take a look more at it.

Well. I think I will have to play with this more before I can make a final judgement but as of now, again this looks cool and I can tell that a lot of time and effort went into it but truly, I'm not entirely sure of the point. The underlying premise seems to be "let's make developing simple web applications easier", but I'm not sure in the end how this does it. My experience tells me that most of the magical "let users build the applications they need without having to code" apps end up failing. And I think they fail because they are trying to fit a niche that doesn't really exist in my opinion. The reality is that the target users end up having to learn "a" language, even if it seems simple and really don't want to, and for more experienced developers the limitations of a home-grown API become frustrating rather quickly.

This niche between users and developers seems to be what Chrome Server is about, but I am not really too convinced that there are many people who fall into this category. I could be wrong. It's also possible that I am misunderstanding the point of this whole thing, I'll have to look into that as well.

Oh and my Chrome Server app can be found here.

http://chrome.maxstocker.com

It's not much to look at currently. I just set up my root page and a link to another page. As I have time to play with stuff it will probably improve.

Outsourcing IT : how much are you really saving?

Sun, 25 Jan 2009, 00:00:00 EST

While I have found myself pondering this before, some events from the past two weeks have really made me question the wisdom of companies hiring outsourced IT agencies to meet their general IT needs.

On paper, especially for small to medium sized businesses it seems like a great idea. Why spend money to hire a full time IT person when you can get 24/7 support from an agency supporting several clients for much less?

And yet...

I have been helping a client (who I have provided development services and am currently providing web hosting services for) with their email. The client already had email provided through an exchange server that is being managed by an outsourced IT company. In the course of my dealings with the client thus far I've set up some email forwarding for hosted domains to this exchange server.

Well now the client wants to add new emails and be able to synchronize calendars and contacts with the users of these new mailboxes. This of course requires making some changes on the managed exchange server. Additionally the client would like to be able to access the exchange server remotely via OWA (Outlook Web Access).

It should be noted at this point that (a) again the outsourced IT company has been managing the client's exchange services for some time and (b) I know that the outsourced IT company is managing an exchange server for another company that is currently running OWA and (c) the client is obviously not super-savvy technically but is far from hopeless as well, they certainly understand the basics.

To help facilitate the exchange updates the client asked me to contact the outsourced IT company and get them to answer some questions. The two questions being (1) how much will it cost to enable OWA and some new user accounts on the exchange server? and (2) about how long will this work take to be completed? For the second question it's not a matter of how many hours (which would be resolved by (1) anyway) but when exactly they could get around to performing this work, because it needs to be done soon.

On the afternoon Thursday January 15th I phoned the outsourced IT company to ask these questions. I was in a queue for a fairly lengthy amount of time and when I spoke to someone they essentially told me that the question was too technical for them and I should send an email in with the request instead. Not an overwhelmingly great start.

I composed an email and sent it on the morning of the 16th. Several hours later I received a reply. Unfortunately besides the odd delay the reply was worthless in that only answered questions that hadn't been asked, for example providing a link to webmail for the POP boxes associated with the client (but not OWA). I replied within 10 minutes by email and asked for "clarification" to get the original questions answered.

There was no reply until the morning of the 19th. At which point the support person told me that the client did not use POP but actually used exchange, as if this was some great revelation. This was to say the least puzzling since from the get go my questions had been about exchange...

I responded again within an hour to repeat the original questions. At the end of the day on Monday (19th) the support person responded again to basically restate what we already knew, that the client is on exchange and if we wanted work done we would need to get a quote from them. At this point I began to feel a bit discouraged. The IT company was now asking us on the 19th if we wanted a quote, a quote I had asked for by phone on the 15th and by email on the 16th. So even with the lengthy delays and inexplicable confusion on their end we were no closer to getting any answers to our original questions.

What happened next though made it all worse. At 8 am on Tuesday (20th) morning the IT company closed the ticket that had been opened with my original email saying that the matter had been resolved. I responded saying bluntly that the matter was not resolved and that none of my questions had been answered. I didn't say this but it point of fact the only thing that had been resolved to that point was the confusion at the support company end over what type of email services they were providing the client with!

The support person then responded to say that they had forwarded the request to the sales department for a quote and that someone would be in touch shortly. The support person also re-opened the ticket and said it would remain open until the quote was provided.

I responded to say thanks and that I hoped the quote would include all the information asked about, specifically mentioning the length of time to completion.

On Wednesday January 21st, despite the promise given the previous day by the support person the ticket was closed again. As of now, but even going back to end of business day Friday January 23rd the promised quote has still not materialized.

What the client (and I) have been asking for is not difficult. We're talking about something that at most will take a couple of hours of work. If there was an in house IT person there is a chance the needed changes would have been deployed on the 15th of January, and almost certainly no worse then the 16th. As it stands however, tomorrow (Monday) morning will be the 26th and we are no closer to a resolution. I am unsure how short of my and/or the clients getting hyper-aggressive with the company that there will be any resolution. And let's not forget that when and if this does actually get resolved the IT company will be charging premium (and I would say exorbitant) hourly rates for the work to be performed because somehow paying a monthly fee for services like managing an exchange server does not actually include any allotment for basic services like creating new accounts or enabling web based access.

So in the end what has the client gained by having outsourced their IT needs? A great cost in terms of both time and money as far as I can see and a frankly totally unreasonably long amount of time to get the issues resolved and needs met.

Now at best one might say, well maybe this is just an atypically bad experience but sadly I know that's not true. The other client I know who is using this same company's services quite often experiences lengthy service outages and delays and ridiculously long resolution times. It's fairly common for one or two people at the one client to have non-functioning workstations for two weeks out of every month for example.

It's hard to imagine how this is a savings...

AJAX to Servlet

Sat, 17 Jan 2009, 00:00:00 EST

An update of my last entry.

I created a Java Servlet for the back-end and updated the tic-tac-toe game to use it instead. The new servlet can be found here.

Silly AJAX game

Fri, 16 Jan 2009, 00:00:00 EST

Okay so I put up a new game. Tic-tac-toe which you can find here. I'm playing around with AJAX a bit. So you can sort of say this is proof-of-concept v1.0 for something else. I'm pretty happy with how it turned out at that end.

Also I suppose you can make your own implementations using the server script as a web-service of sorts. The back end can be found here. If anyone does want to play with it let me know and I can describe the "API" for you, but it's really pretty simple.

GNU Java is evil

Thu, 15 Jan 2009, 00:00:00 EST

If you are lucky enough to have found this blog entry and unlucky enough to have some form of GNU "Java" (GCJ) installed on your system here are two important steps to help you out.

Delete/uninstall GNU "Java"
Install a real version of Java, like one from Sun (available on many flavours) or failing that one from IBM

So now to the why.

GNU "Java" is the worst software project ever foisted on the public. The fact that it's open source and free doesn't mitigate that, and frankly just reflects badly on the open source community in general and GNU specifically. Which is a real shame because by and large open source (and GNU specifically) are really very good things. But GCJ is just such a disaster.

Here are a few of the things that make GNU Java bad.

Missing Core Libraries

GNU Java is missing support for among other things AWT and Swing although these are apparently in development. The GCJ site among other things essentially claims that there are too many APIs in Java etc. Fair enough I suppose but no AWT support? This is 2009. The "final" version of AWT was released with JDK 1.1 in February of 1997.

Twelve years is more than a little behind.

What core libraries are supported are mostly broken and incorrect and sometimes scarily so

Fundamentally basic parts of the Java API are not implemented properly in GCJ when they are implemented. For example threading doesn't work "quite right" in GCJ. And neither does support for Sockets. Both of which I have personally experienced. Even the GCJ site makes comments that programs will run better by disabling threading support for non-threading programs. A comment that is disturbing enough on it's own but gets worse when you stop to think about it, what kind of GC (garbage collection) support is there exactly?

The truly frightening stuff though are the broken ways in which other APIs are implemented. Like for example the reflection API which is noted by the GCJ site for having some security related problems (aka calls that should throw permission denied don't). Things that break a security model as a minor aside should be treated with at least a great deal of suspicion.

It's all pretty useless anyway

Since you can get working Java from Sun for Windows, Solaris and Linux. Apple provides it's own working versions. And IBM has working versions for several platforms including Linux and AIX. It makes one wonder what the point of GCJ actually is. BSD I suppose. But it's hard to conclude there is any real demand on a system not supported by a major vendor when a 12 year old project as this one is still in such poor shape.

The weasel wording used by the GCJ site to describe itself

Really the thing that makes GCJ progress from simply a bad project to an evil one is the whining, doublespeak and general weasel wording used on the GCJ site. To begin with they neglect to mention that GCJ is not compatible with "real" Java. For example Serialization and RMI don't work from GCI programs to real Java programs. I suspect because these sorts of comments would lead a person to conclude that GCJ is incompatible junk. Which of course it is.

But it's worse than that. As commented above there is a good deal of whining about how difficult it is to implement a changing API such as the whining done here

"I think it's important to stress that there is a big difference between Java and the many libraries which Java supports. Unfortunately, Sun's promise of "write once, run everywhere" assumes much more than a JVM: you also need the full set of JDK libraries. Considering that new Java APIs come out every week, it's going to be impossible to track everything. "

That comment sounds reasonable until you realize it's addressing the lack of AWT support! This isn't a new API, it's been around for 12 years now. So really the best thing you can say about that comment is that it's disingenuous.

And the same amount of spin applies to the versioning, aka what version of Java does GCJ actually support? Well on the front page of the GCJ site it says

"supports most of the 1.4 libraries plus some 1.5 additions"

So 1.4 and 1.5 are mentioned. (It's worth pointing out that Java is now on version 6, 1.5 has entered its EOL phase and 1.4 is officially unsupported so to begin with 1.4 and 1.5 are not exactly accomplishments to be proud of). However the statement is most of the 1.4 libraries which really means that at best what is supported is 1.3. So by their own admission GCJ is at the level of a Java version that was superseded in February of 2002, seven years behind!

But of course that impression is just so much hopefulness because it isn't even at that level. For one thing as noted no AWT support means that it isn't really at JDK 1.1 yet.

Okay, okay, it if was just all the GUI stuff we might be able to let that go (although it's pretty damning). At least the rest of the stuff works right? Well not exactly. The GNU Classpath project (a fundamental part of the GCI system) has this to say

"GNU Classpath 1.0 will be fully compatible with the 1.1 and 1.2 API specifications"

Which is another way of saying that it doesn't support Java 1.1 yet, never minding the GUI components.

All in all it's difficult to understand why GCJ hasn't been mothballed and it's impossible to understand why several Linux distributors continue to include it as part of their distribution. If you are inexperienced with Java and come to know it only through GCJ your experience would be pretty poor and worse the GCJ site is not honest about the shoddy state it's in and attempts to rubbish Sun to explain its own deficiencies.

A truly terrible product and a site bordering on telling outright lies make GCJ not just a useless or bad project but truly an evil one.

*******************

A previous discussion of some of the things discussed here can be found here. An example of GCJ making a basic task impossible can be found here. The weasel worded GCJ website can be found with Google, I refuse on principle to link to it directly.

Privacy starts with design

Sun, 11 Jan 2009, 00:00:00 EST

This really shouldn't still be an issue but it is, too many people are still adding requirements for privacy (and even security) as an afterthought add-on to applications and systems.

I recently had a client that had a database full of very sensitive personal information, and yes the client should have had this information, but the database was on a system that was open to the internet. I suggested putting the system on a private network instead, the response to that was

"But why?"

But this system has worse problems. Even the side on the private network is open to any number of users whose systems are not monitored for malware/spyware. Which means that realistically closing the system off the internet only closes one door, with another wide open.

The main problem in this case is that decisions were made with regard to the application and it's accessibility before any privacy or even basic security concerns were looked at.

The end result of all this is that short of scrapping the entire system and starting over I don't have a clue how to make it actually secure enough. And that will be costly. But the alternative is worse.

Un-Truth in Advertising

Fri, 09 Jan 2009, 00:00:00 EST

Yesterday I was perusing through a direct mail flyer from Dell Canada (This is the Dell Canada "Business" flyer for January 2009). I'm sort of on the lookout for a new computer, and especially a new (flat) screen, so I'm keeping my eye out for ads like this to see what's on offer.

Now while it's not uncommon to find a mistake or typo in a flyer of this length I was dismayed to find several errors including a fairly egregious one repeated several times.

The egregious error is one repeated on several pages and serves as a warning about a limitation in the operating system offered with the computer. The warning reads

"You cannot register a domain name with Windows Vista Home Basic. For businesses requiring a domain name, we recommend upgrading to Windows Vista Business"

And the warning appears with computers that are offered with Windows Vista Home Basic as their operating system. A $90 upgrade to Windows Vista Business is offered as one of the upgrades for the system.

The problem with the warning is that it is patently false. No operating system can prevent you from registering a domain name. All you need to register a domain name is a computer that can access the internet, even if some sort of insane restriction was built into IE any other web browser would be able to do it. You don't even need a web browser really.

So why would they say something that is 100% untrue? Because the marketing department at Dell apparently suffers from great confusion over basic technical terms.

You see there is a restriction with Windows Vista Home regarding Windows Domains, namely that you can't use a Windows Domain from Vista Home versions. The problem is that a Windows Domain and a domain name are not the same thing or even close to the same thing. And the term "registering a domain name" can only be used to describe, well, domain names. There is no "registration" type process for Windows Domains.

If they had only stuck with the term "domain" I wouldn't have such a problem with it, it would still be needlessly confusing but the repeated use of "domain name" along with the word "registration" makes the warning totally wrong and misleading. And repeating the same error three times is in my mind egregious.

On the plus side I don't believe that the error is the result of a malicious choice but rather ignorant incompetence. Later in the flyer there are some really silly mistakes like

"[...]Dell engineers in security such as NIC (Network Interface card[...]"

A NIC is a security feature now? This one really is just stupid more than anything else but it doesn't reflect very well on Dell.

My personal favourite though was this

"Along with processing power, expendability and availability features[...]"

Expendability? Really? From Merriam Webster

"ex·pend·able - more easily or economically replaced than rescued, salvaged, or protected"

So Dell is telling me that their systems are junk? It seems to me the really expendable item would be the marketing department, or at least the proof-readers at Dell Canada.

I have actually filed a complaint with Advertising Standards Canada regarding the domain warning error. I would have been happy to contact Dell directly about it but they have no way of giving feedback that doesn't relate to their website on their website. We'll see how this process goes.

RSS parsing with PHP

Tue, 06 Jan 2009, 00:00:00 EST

I've been looking for a bit for a decent RSS parser in PHP and I think I found one called Magpie. I've also been on the hunt for portlet style stuff with PHP as well.

One of the problems I find with PHP is that there is alot of stuff out there freely available but much of it either doesn't work at all, or doesn't work the way you need. I had the same issues when I was generating PDFs on the fly in PHP. It can be done and there are libraries but it tends to be grossly complex and fragile.

Well, we'll see how Magpie goes.

On Ubuntu

Sun, 21 Dec 2008, 00:00:00 EST

I have permanently given up on using Windows as my primary OS. Mainly because it's just so insecure. Having to run anti-virus and anti spy-ware programs 100% of the time is just a resource eater. And as a developer you'll find you also spend a great deal of time fighting with these services to get them to allow you to do the things you need.

For some time I had travelled the middle road and installed two different virus-spyware scanners that I ran regularly, one once per day and the other once per week. I was also very careful in making sure my system was updated with the latest patches from Microsoft.

But this past week, on Thursday I believe, my workstation was taken over by some sort of nasty trojan that fiddled with my internet settings and my router. The router was so bad that even the hardware reset didn't work and I had to junk it.

At any rate I can't live with a machine that can get breached so easily at any moment and I can't develop on one that requires significant resources to be eaten at all times by paranoid anti-spy/malware. So now I'm on Ubuntu.

I have played with Ubuntu a bit before and I'm glad to say that in my opinion it is getting better. Still not totally for the faint of heart though.

We'll see how it goes.

Facebook in decline?

Tue, 16 Dec 2008, 00:00:00 EST

I am noticing recently a number of negative new developments at Facebook. Starting with some "questionable" ads. For example one app in particular keeps randomly redirecting me to a pornographic site. That's new and not particularly very good.

But there are certainly more and more ads showing up in more places on the site as well.

I suspect any site that is ad revenue based must be struggling now. Just like the 2000/2001 era "correction" there seems to be a market correction in advertising rates and revenues going on, especially over the last few months. Historically it's not surprising that a fall off in ad rates/revenues lags behind a general economic fall off. It takes a few months at least for the effect of new (reduced) budgets to be seen in any advertising spending and online is no exception.

What is more curious to me is how low this will go and what effect this will have on the new batch of dot-coms like Facebook.

Myself I have never really believed that you can make a profitable do-com company from ad revenue alone unless you control some other parts of the system as well. For example Google will be fine because they are their own ad supplier. And Yahoo will continue to do okay because they have a well established portal with tons of user supplied content.

But Facebook? I don't know. Alot of the content is not actually provided by users but by application developers. So that slices into the pie.

We'll see what happens but the questionable ads by an application and the general low quality of many of the ads (small companies selling largely low cost items) would seem to indicate to me that the ad revenue streams are not doing overly well for Facebook at least.

Well I'm speechless (almost)

Mon, 15 Dec 2008, 00:00:00 EST

One of my current projects involves migrating data from one legacy format to another proprietary format for a CRM system. Well, migration is really the wrong word for it because it's more like synchronization since data will continue to come in the legacy format and the users will now use the new system at the same time. So, really to begin with this is less then an ideal setup.

The target CRM system is a third-party extension of a well-known CRM software suite and the whole thing is running on a well-known database server for a backend. It is worth noting at this point that this third-party extension is a popular solution for the target niche in question.

So that's fine.

But I have never ever seen such a horrific database design. Never. It's beyond appalling. The problems seem to begin and end with the third-party developers seeming to be completely unaware of what a relational database is and specifically why denormalized data is bad.

To step back for a moment, the extension basically adds the ability to have products associated with contacts. So imagine a list of products with fields like purchase date, product code, product name, sales transaction id, sales person id, quantity, etc. So far so good. But here is what the third party developers did...

Fields like purchasedate1, purchasedate2, purchasedate3, etc, productcode1, productcode2.

Ouch!

And it goes on and on for 7 "products". What happens if there are more than 7? Well you're screwed. And it's a pain to update and can't really be data-mined effectively and this list of bad things that happen with denormalization goes on.

All that is bad enough, but it actually gets much, much worse. For one thing the developers didn't feel that any consistency in naming would be helpful. So for example it's purchasedate_0, purchasedate_1, etc and productcode, productcode_2, 3productcode etc. And other variations in numbering/naming schemes that go a long ways to prevent any SQL code that isn't totally hard-coded.

And it gets worse. Remember how I said there were 7 products? Well there are only room for 7 but there are multiple types of products. There are three types of products actually and room for 7 of two of them and 5 of one and all the products have at least 15 fields to begin with. All of which means, this won't all fit in one table anymore, especially since the developers started by appending fields to the main contact table that already had about 100 fields.

So what we have now is the delight of...

CONTACT Table - many contact related fields, purchasedate_0
THIRDPARTY Table 1 - purchasedate_1, purchasedate_2, etc, productcode, productcode_2,etc
THIRDPARTY Table 2 - productcode_7,quantity_1

Are you seeing the full horror yet? One "record", that is one product entry actually spans three tables. Three! And for one product type it even spans 4!

I don't even know where to begin or what to say really (hence the title of this entry). The third-party developers should obviously not be in business and yet not only are they in business they seem to be reasonably successful at it. Which considering the obvious total lack of any clue of how a relational database works is a bit on the discouraging side.

Blog spammers blacklist

Thu, 11 Dec 2008, 00:00:00 EST

I am currently putting together a small site and list that will have blacklists of domains spammed as well as IP addresses used by spammers for spam blasting blog comments.

Along with the site these lists will be somehow downloadable, probably in CSV format, one list for the domains being spammed and the other being the IPs used.

I am doing this because it seems to me that, especially with the recent shutdown of several email spamming operations at the ISP level, the level of blog comment spam is rising fast. And to be honest the effects of such google bombing mean that besides the obvious annoyance factors people are probably getting stung in Google ratings in cases where such spam is prevalent.

Anyway I hope to have this subsite up over the weekend, we'll see how that goes.

On an unrelated sidenote I have only had one spam message posted successfully here, I think that math type questions are pretty good at blocking out bots so far. So pretty happy about that.

The number one (with a bullet) problem with Windows

Sun, 07 Dec 2008, 00:00:00 EST

When it comes to the never ending discussion about Windows vs anything else really it doesn't really pay, or make sense to be a "fan-boy" of either. For desktop usage there's really not much reasonable in arguing that Windows is not better. Users are comfortable with it, the support for apps like Outlook is fairly solid and even the administration tools and options have come a long way.

For servers on the other hand...

All the other touchpoints (ease of administration, security, stability and uptime, etc) aside here is where Windows loses big time. Where is the logging when things go wrong?

I mean I know there are event logs in Windows but they very rarely seem to actually be working when things go down the tubes.

That is just stupid. End of story.

A Windows 2003 Server that I help to manage (I am not the primary admin for it but fill-in and help in emergencies) is dying. Again. The services running on it, including IIS and SQL Server are actually continuing to run just fine. However. Explorer.exe seems to be dying. Logging in via RDC one can login just fine, the login screen goes away and the desktop colour comes up. And.

Nothing.

No taskbar. No icons. No error messages. No response to mouse clicks. Nothing. So it's dead. Because you can't login to shut it down or use remote shutdown on it... even though SQL Server Management studio has no problem connecting to it either. But there are some patches to deploy so someone will have to physically reboot this box to get it back.

What makes it all much much worse is that this is not the first time this has happened. Which raises the question of why. And the answer to that is who knows. Because Windows doesn't seem to have a clue that anything is even wrong at all. Certainly nothing has appeared in any logs yet.

And that in a nutshell is the problem with Windows in the server market. Sure linux error messages can be cryptic, and sure you might have to do some digging and searching to find the "right" log file and decrypt said message. But it's there. I have never had a problem with a service on Linux dying that I couldn't eventually trace and figure out but on Windows this just seems to be the case with puzzling crashes and lock-ups. You can guess, but you never really know.

Whatever faults in an OS or a service may or may not be present. Whatever problems there are in security, hardware, disk system etc. It is always better to know.

Changing of the guard

Fri, 05 Dec 2008, 00:00:00 EST

Earlier this week Ted Rogers the head of a Canadian telecommunications conglomerate passed away. The coverage of the story was, interesting, and I believe a bit of a testament to the value of owning lots of media outlets in telling positive tales in your obituary.

I am sure he was a fine person, I have no reason to suspect anything else, and there is enough evidence to suggest that he was as patriotic and charitable as the various obituaries suggested. But he was also routinely hailed as both a business and telecommunications visionary and that I do have a problem with.

There was an obit piece on a television station owned by Rodgers communications that certainly left one with the impression that he had personally invented high speed internet access, if not the entire internet! Ummm. In point of fact while the company was a pioneer of broadband internet access in Canada they were certainly plagued by difficulties, especially capacity and reliability as their network grew and they did a phenomenally poor job of customer education leading to their network being a haven for many years for spammers, bot-nets and amateur script-kiddie hacker hour.

As far as the business vision goes, there's no question he and his companies made alot of money certainly. The question of course then becomes how. Having a monopoly always helps but more there were plenty of ethically dubious actions taken in pursuit of the bottom line. I mean, to be honest, part of the legacy here is that legislation was passed here both provincially and federally making a billing scheme the company came up with illegal.

All that aside one story that didn't seem to be talked about much was what impact his death will have on the telecommunications industry in Canada. Probably significant actually. Ted Rogers was certainly a thorn in the side of Bell Canada and certainly some of Bell's non monopoly style abusive behaviour towards customers can be attributed to the level of competition and confrontation Rogers brought, and sometimes seemed to seek with Bell. I suspect that in the long term the company he founded will miss Ted Rogers drive and sometime cantankerousness.

It will if nothing else be interesting to see how the landscape for networks, connectivity and telecommunications shakes out over the next few years.

Well I'm addicted

Tue, 02 Dec 2008, 00:00:00 EST

To creating RSS feeds.

Maybe caffeine too.

But creating new RSS feeds for sure. I had done a few for a few minor things before but a few weeks back I created some for specific content for one client and since then have been busy creating them for all the other content that is naturally serializable (ha ha - this is a programming sort of joke that if you don't know then by the time I explain it you will want to hurt me so let's just leave it with the old eye-roll and move on).

But these feeds are pretty nifty and there is so much content that some sites have and can make available through RSS.

I suppose I'll get disenchanted eventually but for now I'm a kid in a candy-store.

Oh and if anyone feels like adding a "Welcome to 2004" comment on this I would point out that while I have known of RSS for some time it is only recently that I have seen it's usefulness.

Aaarrrrrrghhhhh

Sat, 29 Nov 2008, 00:00:00 EST

I know it's been said before but it would be nice if there was some cross-browser support in CSS that actually looked the same.

Mainly this is all IE's fault but all the same it's just such a mess. I really love the cranks who rant on about how you should never use HTML tables, "Use only CSS positioning", they cry, "people who use tables are [some pejorative]".

Yeah well here's the thing, I can tell that you've never had to make a site that looks the same in IE and Firefox, let alone anything else, that actually has dynamic content and features a design more complex then three colours and a gradient fill.

I mean that sort of site is fine, but it's not what most customers actually want. And those same people above when confronted with that sort of reality spout out about "graceful" failing. Here's a clue. If the page looks spectacular in Firefox but in IE, while the text is readable actually looks like complete crap that actually is not good enough. 'kay?

So the CSS only solution involves browser detection, and anywhere from 3 to 4 times the length of the development process to make stylesheets that work in each browser. And even then you still might be in some serious trouble.

And who exactly is supposed to pay for this lengthy development time? The customer? Because they couldn't care less if you use tables, style-sheets, or any other technology they haven't heard of. All they care is that it doesn't look like garbage in the browsers they use.

I'd love to use only style-sheets for positioning, instead just for colors, fonts and some margins. I'd love to. And if they actually worked even remotely the same on different browsers I would. But they simply don't.

More bang for the buck

Fri, 28 Nov 2008, 00:00:00 EST

If I had to pick the business or area of business most impacted by the presence of the Internet I would have to choose marketing.

Which seems a bit odd because you'd think I would have said something about IT but networking computers together (and all the issues that come along with that) is an IT issue that predates the Internet really so while it has certainly increased demand for experts it hasn't so much fundamentally changed what the job requirements are.

Marketing and advertising though have seen alot of changes, including some that are only being felt now. The obvious part about advertising on the Internet itself is one thing but there's much more to it.

For starters, the Internet was hailed from the get-go as the big breakthrough in user experience, connecting to users directly, responding to inquiries, suggestions, support requests, all-automagically and in "real time" was the wave of the future. Hasn't quite worked out that way for alot of reasons but one of the key points that was learned was that when it comes to service most people don't actually care.

This is actually a pretty stunning revelation. In some ways not entirely surprising but it does go against what years (decades) of asking people showed. If you ask people about quality of product and service vs price and what's important to them you'll get all sorts of answers, none of them very true. The reality is that a shockingly large amount of people make decisions based purely on lower price.

But this isn't the whole change either. The wealth of information about people's behaviour that has come from the Internet is after 15 years starting to make demands for such information from other industries.

For example after the success, and startling results, of some pilot projects, ratings for radio in Canada will now be moving to an electronic model. Previously a selected group of people were asked to record listening habits in a "diary" (using pen and paper) but now the system will work automatically and electronically.

The flaws with the pen-and-paper system were always known about but it was assumed that even if the numbers weren't entirely specifically accurate at least the general trends were.

Guess again!

The results of the pilot projects showed that people's actual listening habits were very different from what people say. As an example, historically radio audiences for live sporting events have been middling to poor. These numbers were backed up by the general concensus of opinion that only fans of the sport would want to listen to the game but most of them would prefer to watch on TV so radio was only used as a last resort by very few.

The electronic pilot project showed that to be entirely nonsense. Listenership to live sporting events is actually 3 to 4 times more than people had recorded. Nobody has any real guesses as to why this is, although I suspect it has something to do with people associating listening to game with a team or sport rather than a radio station but either way it's a good indicator of how much more or less valuable a property becomes based on having accurate information.

And I do think the Internet has alot to do with learning about such things. Because if there's one thing it's taught us about people it's that the decisions that people say they make are often very different from the decisions they actually make. And it's hard to pick an area of business more affected by that reality then marketing.

Throttling updated but not concluded

Tue, 25 Nov 2008, 00:00:00 EST

So last week the CRTC finally rules on the "packet shaping" or "throttling" being performed by Bell. At face the CRTC said that Bell wasn't breaking any laws but it's a decision that's left a fair degree of confusion in it's wake.

Bell claimed the decision as a victory but the CRTC then disagreed. Basically the CRTC spin is that while packet shaping may not be illegal it's not "correct" from an ethical or ideal perspective either.

I become more convinced with time that my idea of spinning off the network itself into a crown-corporation is a really good one. I think the bottom line concern, even for the smaller ISPs that filed the original complaint and are voicing outrage over the ruling, is not so much that throttling is happening at all but in the conflict of interest since Bell also acts as an ISP directly to customers. If the network is saturated who do you think will be the first group to be throttled? Bell's customers or someone elses? I think that's a question that we all know the answer too.

So now the next step of the process is moving on. The CRTC is collecting more information and will make some sort of ruling/plan supposedly in February. I say supposedly because they were two months behind (of a 6 month schedule) in issuing the ruling they have so far so it seems likely we could be waiting till next May to actually get the final result.

I'd like to add a point that if you read this and don't know what I am talking about or why it applies to you. Well. If you live in Canada this is pretty important stuff. The final rulings on this will affect alot of things including, the amount of competition for internet services in Canada, the price you pay, the quality of service and the responsibility for long-term upgrades and capacity planning. In many ways as far as Canada goes this ruling will go a long way to determining the future course of the internet and internet business in Canada. So yeah, it's important.

Improved RSS Feed

Thu, 20 Nov 2008, 00:00:00 EST

As a result of some learning done for other tasks I have improved my RSS feed here. Previously my feed was overly complex and some readers had some troubles with it. But that's all water under the bridge now... (he said hopefully).

With any luck facebook will soon pick up this post and I will then know that all is well.

Any less spam for you?

Wed, 19 Nov 2008, 00:00:00 EST

A big story last week in the world of the internet was the shut down of McColo Corp a shady hosting company responsible in part for a great deal of the spam on the internet.

In the days following the shutdown companies across the world reported a drop in spam activity of 50% to 80%!

It's the typical internet story to me at least. I think it's great it was shut down but what took so long? Rather than limiting bandwidth of ordinary customers why aren't ISPs held directly responsible for more of their clients dealings? Seems to me that simply enforcing accountability would clean up so much lawlessness and abuse.

The whole thing also provides more fodder for my long held belief that email protocols should be revamped entirely. SMTP is pathetically weak and worse most implementations are recklessly open out-of-the-box.

Canada Post on Strike

Mon, 17 Nov 2008, 00:00:00 EST

So in the midst of the economic meltdown "support" staff at Canada Post have decided to go in strike starting today.

2,100 workers now picketing across the country.

According to Canada Post the strike will not affect mail delivery. Even the head of the union involved claims that mail delivery will only be affected by the strike eventually.

Which all leaves me thinking two things

Is it possible to even imagine a business less doomed to extinction than Canada Post or post offices in general really?
I think I just found a way to cut costs at Canada Post. About 2,100 jobs worth of costs...

I mean in all honesty what in the world are these 2,100 people doing when they are working?

A new word

Thu, 13 Nov 2008, 00:00:00 EST

Unpossible.

Something that should be possible, and in fact technically is possible but for reasons best described as mysterious is made not possible.

For example: Why can't you have Notes in Facebook groups? Because I would like to import an RSS feed into a group that people could join and then view for updates...

Of course the RSS importing tool for Facebook notes doesn't really entirely work correctly but that's another issue unto itself.

From Possible to Practical

Wed, 12 Nov 2008, 00:00:00 EST

The gap between possible and practical sometimes yawns quite wide. So much so that often things which are possible are totally impractical for a variety of reasons.

But it is difficult for non-technical people to understand that this gapa exists or how bad it may be in a certain situation. Because most of my work as an independent contractor involves helping people meld disparate technologies into a funtioning whole I think I do encounter situations like these more than most.

I also think I am good at making possible into practical fairly regularly. But there's a down side to that as well which is that sometimes I get the feeling that some clients think I can solve any problem in a smooth way.

If there is any one piece of advice I could give it would be this. Make sure, by talking to someone independent, that the four different technologies you are planning on purchasing, will actually work together. Waiting to find out they don't after the fact is a little late. And the reality is that often vendors tell "versions of truth" that may well not be compatible with your particular reality.

Utterly Hopeless

Mon, 10 Nov 2008, 00:00:00 EST

So recently I have been working on some dynamic content fancy-nesses for a client. Basically what's happening is that a few RSS feeds are being read in and dumped into a database. Then the content is used for a variety of different tasks including some POH (Plain Old HTML, like it? I just made it up, sounds good I think), some RSS feeds and a few other things.

So all well and good. Other than the original RSS feeds have some compliance issues and some of the content is actually wrong but these can be worked around.

What I also was trying to do though is for some special updates that come in I wanted to update a Twitter account and have the program post updates like "I got X number of [type of content update]". Then twitterers could follow the account and get updates that way instead of just having to refresh an RSS feed which is a bit hit and miss.

But bloody twitter is so overloaded apparently that I can't make any of the simple profile changes I want. And I've been trying for three days now! Honestly. It's fairly pathetic.

I mean just throw up some AdSense blocks, take the revenue and buy more hardware guys. This isn't exactly rocket science here.