Friday, December 28, 2012

What's Wrong With LinkedIn Endorsements

For the longest time, I've been on LinkedIn and had no issues with it.  Just recently, I started getting lots of emails from LinkedIn saying that certain people are giving me "Endorsements".  At first, I thought that this was rather nice, until I looked into it a little more.  What I found was interesting for all the wrong reasons.  

Imagine I have a connection in LinkedIn that is a neighbour.  I've never worked with them professionally, and we actually work in totally different fields.  Let's say that they're an EMS worker whilst I'm a programmer.  I get an email from LinkedIn saying that this person has endorsed my "Objective-C" skills.

The first problem with this is I doubt very much that this person would know what Objective-C is?  The second problem with this is how would they know that I'm any good at it?  

What appears to be happening is some form of automated popup is asking people for a single click to agree to something, and "Bob's your uncle", you've just been endorsed.  So what does that make the endorsement?  

In short, the endorsement is worthless. If the people making the endorsements aren't even qualified to make them, then when trying to assess how good someone is at something just became impossible if you take these endorsements into account?

Thursday, December 27, 2012

Wet Apathy

I like to travel.  The biggest reason is it allows me to discover new things, though the major area for discovery is in the mundane.  Even if I don't particularly like a place (rare), I still learn something.

One of my favourite places is the Kansai prefecture of Japan.  This is the region that contains Kyoto, Osaka, Nara, etc.  I find it inspirational because the mundane things in life have often been addressed in ways that are the polar opposite to how things are done in the place where I spend most of my time - Toronto.

For instance, in Kyoto, the downtown centre has covered sidewalks - which means when it rains, you don't get wet when shopping.  In Toronto, you get the feeling that nobody wants you to shop in it's stores when it's wet - there's a feeling that they want to corral you into the shopping mall instead, or drive you into the PATH system (an underground tunnel system that whilst expansive, is largely geared towards office workers - you can't go there on a Saturday morning and expect to find everything open).

One of my biggest annoyances with Toronto's civil engineering is gutters, drainage, and water in general.  In Kyoto, much like Toronto there are ground waters to deal with when it's not raining.  So, in Kyoto they built little streams and bridges, and put tea houses on them and turned the water into features that everyone likes to enjoy.  

In Toronto, it's like the mantra is "Who needs water anyway?" as the creeks are all bricked up and buried, the views of Lake Ontario is repeatedly sold off to the highest bidding condo developers, and just to make sure nobody has any want to get to the water, they cordoned off the lake from the city with an elevated six lane highway.  

And so we come to gutters: Here's a gutter in Kansai…  

That's Koi swimming in it.  

In Toronto, the gutter is an afterthought - it's where you park your car, not a feature that was invented to handle water.  Nothing shows up this more than when it snows.  

People have been aware for quite some time now, that it snows in Canada.  When it snows, Toronto city laws stipulate that you have to shovel the sidewalk so that people can pass.  What generally happens is people shovel the snow towards the gutter - and the snowplough also pushes the snow off the road and towards the gutter.  

The net result is you get a snowbank.  Snowbanks are notorious for one reason: As a pedestrian crossing the street, you eventually have to navigate them.  Whilst it's not immediately obvious to those who don't live here, there's one major issue with them that everybody who has ever lived here knows about…  Slush.

The problem happens in two stages, both of which I find comical to watch, but infuriating to endure:
1.  After a significant snowfall, crossing the street is like something like a scene out of a WWI movie.  You first wait for the signal, then you climb several feet up the bank and go up "over the top" and dash across to the other side, trying to avoid obstacles like slippery buried rail tracks, potholes, and other people coming at you with briefcases and umbrellas.
2.  After several hours of this, the banks at these points have been mushed down to a grey puddle that normally expands out about three to ten feet in radius, and it usually just a little deeper than the height of whatever shoes you happen to wear that day.  This means you have two options to crossing the road: a) Leap out like a ballet dancer from Swan Lake, trying to land in as shallow an area as possible, or b) find a new un-mushed bit of snowbank and repeat the WWI "over the top" manoeuvre.  As I mentioned before, everyone here knows about this, and just like WWI, everyone spends their day with something resembling "trench foot" as a result.  

The question is what has anyone done about this?  

I can't see anything being done about it.  Nothing has changed between 1912 and 2012.  A simple solution would be to move the drains from their current positions (always far from the corners) to be nearer the corners where they are not buried under a snow bank but where the puddles are at the lowest point - where the slush gathers.  This is something that other countries have done - as seen here (the drain is circled in red).

I hate to sound like I'm always bashing things, but the sense of apathy I see that's evident in how things are done in Toronto just drives me bonkers because nobody is addressing this mundane stuff that affects a greater number of people than some of the "cooler" stuff that only affects the few.

Friday, December 21, 2012

How not to do digital advertising

Digital advertising mainly comes in three flavours:
  • Ad banners - used in web pages or in apps.
  • Digital audio sandwiched between real content.
  • Digital video, usually boxed into the corner of a screen, surrounded by other visual content.

Today, I'll touch on the last one of these.  Digital video advertising is supposed to have two major goals:
  1. It gets eyeballs to see the adverts.
  2. It does this in a cost effective way that at least breaks even, and hopefully makes a profit for the advertising network that owns the screens.

Every morning on my commute, I go through an underground concourse owned by Brookfield ( where I'm quickly subjected to a number of digital screens.  

Being a technology savvy person and having worked in digital advertising in the past, I tend to have a bit of a critical eye for digital screens, and these screens that I see everyday normally make me think of this:

Running these screens through my mental checklist, traditionally, it went like this:
  • Are the screens in a prominent spot?  Check.
  • Are the screens unaffected by glare? Fail. 
  • Are the screens in landscape mode?  Fail.
  • If in portrait mode, have they been polarized properly? Fail.
  • Do they look "modern" and eye catching? Fail.
  • Is there good use of the screen real-estate? Check.

As you can see, there's more failures than good marks.  Let me explain.

Digital screens are polarised in one direction.  You often notice that when using a laptop, you get better colours by adjusted the angle of the screen on the vertical axis.  When you rotate the screen, you now have vertical polarisation - this is now the same polarisation as glare coming off a body of water.  People who wear sunglasses often have them polarised in the same fashion, to stop glare from water and other horizontal from reaching their eyes.  The net result of this is during the summer months, your digital advertising has just been blocked - unless you used an appropriately polarised screen.

Modern look and feel is rather subjective.  I think we can all agree though that having screens still mocked up to look like iPhone 3GS's when the iPhone 5 is out is a little dated.  The point of digital advertising it should be very easy to update the interface.

Glare can be fixed in two ways:  First is positioning, or failing that, getting anti-glare technology (filters, matte screens, etc).  This will ensure that the viewer can actually see the advert.

And so it was when I noticed this morning that the screens had been updated.  No longer was I greeted by a flanking of iPhone 3GS looking screens. Instead I was greeted with this:

Now, they've gone and replaced a large portion of the screen with dead real-estate, and not only that, they're now showing landscape adverts in a portrait screen, leaving 2/3 of the rest of the screen blank. (The rest of what you see is not content, but glare)

I can only conclude that whoever is the decision maker of these screens doesn't know what they're doing.  This only serves to reinforce my original opinion, seen here:

Thursday, December 20, 2012

Dressing up the truth

Today in Canadian news, there was a bit of a media frenzy surrounding women being able to wear the Niqab in court, or not.

There's an irony to this:
1.  The court system seems quite happy to have people swearing on Bibles, Koran's and other religious texts as part of the oath taking process.
2.  The courts rely on belief as part of the testimony process:
- I believe I heard/saw…
- I believe that if I didn't hit him over the head, he would try to attack me, your honour...

So, to distill this down:
  • Religion is belief
  • Belief can be used for oath and testimony
  • In court, not all beliefs are equal.

This is why this Niqab issue came up in the first place:  
Someone believed they should be allowed to wear it, and someone else believed that this belief was invalid and therefore they should not be allowed to wear it. 

One belief, therefore, trumped another.

Due to this irony, you have to question whether the same system that is supposed to be finding facts based on the belief's of witness testimonies, that willingly discounted one set of beliefs, is discounting other beliefs as part of the judicial process?

Maybe they should replace judges with scientists; that way, it wouldn't matter if the truth was dressed in a Niqab, Jeans, or a tutu - it'd still be inclusive to the process.

Wednesday, December 19, 2012

When Process Hinders Progress

There's an old joke about a substance called Administrantium.  It runs like this:

The heaviest element known to science was recently discovered by university physicists. The element, tentatively named "Administratium," has no protons or electrons and thus has an atomic number of 0. However, it does have one neutron, 15 assistant neutrons, 70 vice neutrons, and 161 assistant vice neutrons. This gives it an atomic mass of 247. These 247 particles are held together in the nucleus by a force that involves the continuous exchange of meson-like particles called "morons." Since it has no electrons, Administratium, is inert.
However, it can be detected chemically as it impedes every reaction with which it comes in contact. According to discoverers, a minute amount of Administratium added to one reaction caused it to take over four days to complete. Without the Administratium, the reaction occurs in less than one second. Administratium has a half life of approximately three years, at which time it does not actually decay, but instead undergoes a reorganization in which assistant neutrons, vice neutrons and assistant vice neutrons exchange places. Studies seem to show that the atomic mass actually increases after each reorganization.
Research indicates that Administratium occurs naturally in the atmosphere. It tends to concentrate in certain locations such as governments, large corporations, and especially in universities. It can usually be found polluting the best appointed and best maintained buildings. Scientists warn that Administratium is known to be toxic and recommend plenty of alcoholic fluids followed by bed rest after even low levels of exposure.
It's with this old joke in the back of my head that I'm writing this post.
An article in the Star the other day ( showed some groundwater that is near-permanently seeping out of the road in my neighbourhood.  An investigation by City Hall found it's not a leaky pipe or sewer, and is probably attributable to groundwater.

This doesn't surprise me.  I live up a hill from this location and even at my higher elevation, I know that groundwater can rise to just 2 ft under my basement floor.  I know this because when we recently dug down in our basement and installed our backwater flow preventers to stop the city drains pushing up into our house again, the hole filled up with water whenever there was sustained periods of precipitation.  When it stopped raining, the hole would drain accordingly.

So, estimating the basement floor plus the depth of the hole, we know that the water table can be as little as 7ft below grade.

When the City responded that the issue in the Star was probably groundwater, it occurred to me that there's probably an acquifer running under the neighbourhood that means free water.  Given that the city charges me a water waste fee for water that evaporated out of the fish tank and never went down a drain, or that was soaked up by the lawn when we turn on the sprinklers, I wondered about offsetting this by just pumping up the water that's under the house.  

My thought process is that rather than pay for chlorinated water to be thrown on the lawn, a small pvc pipe with holes drilled in it that is inserted 7ft down should fill with water that can then be pumped onto the garden, stored in water butts, etc.   

So, I asked "Question: Given how much water is near the surface, are there any rules against pumping it up for private use?"

Now, this was promptly passed to 311 Toronto… and everyone knows I've dealt with them on a regular basis.  They know who I am, where I live, and probably have my photo on the office dart board because I'm such a pain in the arse.  Rather than just "yes, you can pump up your own water for private use", or point me to a published set of rules, they wanted to know my address… for the umpteenth time this year.

I'm still feeling quite raw from back in the summer with all this:  I couldn't report a basement flood because I wasn't actually phoning from the house (we were giving birth to twins, so was a bit busy), even though we'd had a stream of officials and visitors come to our property (basement flooding managers, water board engineers, city infrastructure managers, councillors, etc) and everyone knew the situation.  Our house was well documented, well known, and this flood was just a repeat of the same type of event, so the answers would be the same as before.  But, 311 Toronto was adamant that we call back another day from the house location - despite the fact that the answers to any questions would be the same regardless of where I was calling from.

Since that day, I've not bothered to report further basement floods. However, I'm acutely aware that whilst I'm not going through the motions because I'm annoyed at the rigid process and lack of common sense when it comes to recording a repeat event (same place, type of event, outcome, etc, just a different date), the downside of this is now the city and other planners and engineers are not getting the full data that helps solve the issue for other people.

The same thing happened with the waste department: I asked 311 Toronto for the line managers name of the twit that messed up our garbage pickups nearly every week.  Eventually, I cut 311 Toronto out of the picture and found the name of the person myself, emailed him directly and the garbage problems magically went away.  

The net result of this "process over progress" is highly ineffective.  For the basement flooding, you have to question why a person can't just enter their details in a form on the city website and record events that way.  You question why the same thing does not apply to the garbage service too.  Why do I have to email a physical human?

My conclusion is currently that the one thing that was supposed to make life easier for residents of the city has now turned into the biggest barrier to getting anything done with the city. 

Tuesday, December 18, 2012

A warped view of the Gardiner

Every now and then, stories that I've read about in the news will stay and mull about in my head.  Often, it'll be connected to something I've heard a lot about.  Today, I want to touch on Toronto's Gardiner Expressway.

For those who live in or near Toronto, it's an elevated highway that has some very big maintenance issues that have caused it to be in the news for a lot of the year.  Due to a lack of maintenance, chunks of concrete keep falling off of it, during what looks like periods of thermal contraction.  That's what the news has focused on.  That's what the residents of Toronto have thought was it's only major problem.

For those who've never been to Toronto, or heard of this expressway.  Let me paint a different picture of the same thing.

Where Toronto now stands, a huge glacier once stood.  This glacier was a number of times taller than the present CN Tower, and it was so heavy that it pushed the earth's crust down where it stood.  When the glacier receded, the ground started to recover by rising back up.  Even today, Toronto is still rising at a rate of 2mm per year.  Toronto also has a reclaimed waterline. This new land is made up from loose soil that was historically dug out of basements, foundations, and other items of the city's past infrastructure.  Sitting on top of this reclaimed shoreline is an elevated 6 lane highway.  The highway was built in the early 1960's, so it's basically 50 years old.  Doing the math, during this expressway's lifetime, Toronto has risen (50 years X 2mm) about 10cm. 

Common sense tells me this thing should be developing humps and dips from where the ground is warping the structure, and secondly, I'd question how it'd react if we ever had a serious earthquake in Toronto.  Would it survive?

The biggest question though in my mind is why is the media not covering this aspect of it's demise?  Is the obvious really not that obvious to them?

Thursday, December 13, 2012

Banking Spin and Your News

In Canada, we have something called The Royal Bank of Canada, also known as "RBC".  It is a Canadian bank, that much is true. It calls itself Royal, but has no royal ascent, warrants or royal charters, and the embedded phrase "Bank of Canada" doesn't mean it has has any bearing with the actual "Bank of Canada" either.  In short, the brand always leaves an impression to me personally that the whole thing is a bit "eyes + wool" - but that's branding for you.  Once you see past that facade, it's just another bank.

So, it was this morning as I was scanning through Twitter, I noticed something from CTV News that proclaimed that come 2013, the Canadian economy was going to get better - according to RBC.

This is not news.  Here's why:

1.  The Canadian economy is so fragile that the Bank of Canada (not the "Royal" one, but the actual one) set interest rates at 1% because the economy can't stand up on it's own.  
2.  The same bank has had to leave it there for what seems like a lifetime, because until something changes fundamentally, the economy is not going to recover.

So it is with this back drop of history that I turn your attention to this proclamation from the summer.

Yes, the same bank was singing from the same hymn sheet earlier in the year.  What has changed since? Not a lot - other than the dates.

I've said this before, but I'll say it again: A news release like this is not being done because the bank is being all philanthropic towards the general public.  It's corporate spin that does something to serve the bank's own purposes.  The media needs to stop automatically repeating corporate spin as "factual news" when (as seen here) there is a clear audit trail of inaccurate predictions that precede it.  As a member of the public, you should equally look at what the news is pumping into the media streams that you consume, and question it where necessary.  

If people had acted on what RBC had told them previously in the summer, such as borrowing more money, thinking we'd be out of this hole by now, it'd most likely have played out to be detrimental to the average person.  Q.E.D, when the public loses out to excessive borrowing, it's the lenders that profit from this loss.  As nobody holds the banks accountable to this type of thing, they're free to repeat it - and the news will gladly keep transmitting this tripe.

Now that I've framed the scenario: ask yourself what you think the point of that release was?

Wednesday, December 12, 2012

Mixed Messages

There's two ways of viewing government, regardless of whether you're looking at Federal, Provincial or Municipal versions of it:
1. It's a circus platform of elections, terms, and people scrambling to climb the political ladder of success.
2. It's a framework to support the societal needs of it's citizens.

I look at governments through the latter of these two lenses.  The needs of western society generally remains the same, regardless of whether you're looking at the 1970s, now, or in 2200 AD.  They are:
* Health
* Security
* Wealth
* Happiness

Breaking things down, government becomes a big machine with individual departments servicing the various needs of it's people.  When you look at it this way, it doesn't actually matter who the Prime Minister is, or who the Mayor is, or who the President is.  They come and go, but the underlying services remain in place.  (Disclaimer: I'm a Canadian Permanent Residence, but a British Citizen - so I still vote in the UK, not Canada).

You'd think by now that, especially at the municipal level, we'd have clear communication down pat.  Apparently, this is not so.

In Toronto, we have a system during the summer to communicate heat alerts to the public.  These are:
* Heat Alert
* Extreme Heat Alert

What happens is when we have what our grandparents would have called "a hot day" is the Heat Alert is issued.  The media goes into a frenzy telling the public that today the city has declared a heat alert, and then absolutely bugger-all happens.  Zip. Nada.  No extension to swimming pools, no places with air conditioning is opened to the public, etc.  Back in government, things are being evaluated with community partners, but that's not really a public thing - so the media continues to whip up the news about the alert whilst nothing observable has actually happened.

So what is the point of telling the public there's a heat alert?  In short, there is no point - other than to give news fodder to the media.

When an Extreme Heat Alert is issued, this is the point when stuff actually happens.  People can now get relief from a variety of sources, like pools, libraries, cooling centres, etc, which are opened up to help people from the oppressive effects of a really hot day.

This is the only time the public needs to know there is an alert.  A standard hot day that generates no relief to the public is just another hot day.  However, when there's a really hot day that forces the government to do something in reaction, well, that's when we should see a Heat Alert. 

For those inclined to read the City plan on heat alerts, you can find it here at the City website.

Today, I noted something else:  Most sane people are fully aware that giving animals at Christmas time is a generally accepted "bad idea".  We've all heard the "A pet is for life, not just for Christmas".  We've all heard from the animal shelters that get overwhelmed in the new year by rejected gift puppies and kittens.  The media regularly puts out articles to discourage this practice, too.

And so it's with this backdrop of a message that the City of Toronto put out this:

Seriously? What is the intended message here?

The city should try to NOT encourage people to grab a pet in the run-up to Christmas, but do any adoption drives in the new year when the other idiots that didn't get the message start returning unwanted gifts.

Whilst the elected officials come and go with their messages of "change" and "society", the people that remain should work out what the real message is… 

Tuesday, December 11, 2012

Man vs Machines

People who follow me on social media know that I have a penchant for learning about the world around me in order to understand it, as well as a desire to see corporations that affect my daily life be transparent and honest with me.  If something appears to be wrong, more often than not, I will reach out and try to initiate a fix to the situation.

It was a couple of weeks back when I was looking at a poster on Toronto's subway that I first noted an advert for Interac where the graphics of some local personality dressed as a superhero were relatively crisp and clear, yet the small print was more blurred than the view of out a bullet train window at speed.  

The first thought I had was that the small print was supposed to be the most important bit of the poster.  In theory the poster could have had anything in the world on it, but the small print should say things like the copyright info and other T's & C's that legalities would dictate.  Based on what I was observing, I concluded either Interac in putting out such a advert deemed the small print to be inconsequential, or they did look at it but just have very sloppy approval processes, or they don't want us to read it.  

Advertising is a means for an organization to show me, as a customer, the perception of that organization or their products that they want to me think.  My perception of Interac had now been inadvertently altered to reflect what the advertisement was showing me on the subway… albeit the wrong message.  It appeared to me now to be either sloppy or hiding something.

It is with that perception that I went forward a few weeks.

Every two weeks I move money between two banks to cover my mortgage. I usually do this around breakfast time, and this week I found myself running late for work because I had to wait for Interac's email to arrive, so that I may click the link and deposit the money into the destination bank.  It took over 20 minutes to arrive.  Having clicked the link and deposited the funds, the second email arrived in 10 seconds.

It then occurred to me that my perception of being able to instantly wire money from bank A to bank B was wholly inaccurate.

So I asked Interac on Twitter what the delay was?
Their answer: "Hi Jason - the first email involves coordination with your financial institution, security measures, etc."

Now, as a consumer, I know I can go to an ATM and get money in 2 minutes.  As a programmer, I know you can shift funds (using IFX/OFX) between institutions in seconds.  Often, the Virtual Vault is just an IBM RS6000 that sits behind an array of servers that do the same security procedures regardless of if you're coming in from their web-servers (Internet Banking) or ATMs.  It all ends up in the exact same place - the Virtual Vault.  

So, I said to Interact that this doesn't make sense. This time, the delay was explained thus:    
"Hi Jason, there is additional support from us to your financial institution. As a result there is a minimal processing time."

At this point, I'm feeling like I'm being fobbed off.  All I want to know is why the delay?  What was the reason for me being late to work?

So I asked them straight out if this delay was due to batching? (the process of gathering up a number of requests over time, then forwarding this in one lump).  Interac didn't respond.  Now I knew I was being ignored.  This only served to reinforce the "doesn't matter" / "sloppy" attitude that the advertising had previously instilled in me.

It then occurred to me that I can probably prove my point by manually going to a CIBC branch, withdrawing some physical cash, walk to a ScotiaBank branch and deposit it.  If I'm right, I can do this manual process faster than sending money electronically.

So this is what's going down today.  Having moved the money physically, an identical transaction will be done through Interac, and I will then compare times.

So I went to CIBC, withdrew the cash.  I left with a receipt that stated the time of the transaction.  I got to the ScotiaBank branch with the physical cash and there was an ATM issue with my card, so I queued up for the main desk.  Having deposited the money there, the receipt was not timestamped, so there's no proof of the duration of the experiment.  Go figure.

As a result, I will first resolve the card issue (and, of course, that couldn't be fixed for other reasons), then re-run the experiment.  Having proven my point, I will take this back to Interac and see what their defence is this time.

Monday, December 10, 2012

What do you consume at breakfast time?

Before I start, a quick disclaimer.

Disclaimer:  Although I have worked in Radio for many years in an on-and-off basis, whether it was being the lead programmer of iHeartRadio for iPhone, or being involved in programming one of the largest US broadcaster EOC's that took part in the November 2011 FEMA EAS test.  I currently do not work in Radio, and these views are my own and nobody elses.


It's with a lot of interest that I watch the media document the public backlash against the Australian radio DJ's that took part in the Royal baby prank hoax that ended with a nurse taking her life.  I'm not going to speculate on the nurse and her motives, because it would be just that… speculation.  Instead, I want to focus on the problem with radio.

Many people are not as lucky as I am to have lived in multiple countries and witness first hand how everyday life pans out.  My time in these places doesn't go to waste though, as I do pay attention and compare, analyse, and try to understand things.  

One place item that consistently disappoints is radio.  I also think I'm not the only one that's disappointed with it.  

Not too long ago, the BBC in the UK was shaken to it's foundation by a prank radio call.  In a nutshell, two overpaid idiots called up an old man and left lewd messages about his granddaughter on his answer phone.  Whilst the ensuing public uproar was going on, the basic response was "It was only supposed to be some lighthearted fun".  

This echoes of the childhood school bullies explaining to the teachers why Johnny has a bloodied nose, "It was only supposed to be some lighthearted fun", right?  The end result of that tasteless affair (known by many as "Sachsgate"), was a lot of people got fired, reshuffled and re-org'd, but the public was truly incensed.

Fast forward to the latest debacle coming out of the 2Day FM show, and we hear the same lame sentiment from Michael Christian (the male DJ involved) that we've heard ad nauseum: it was supposed to be "just a simple, harmless, fun call".

I fundamentally disagree, and think radio stations should wake up to the fact that in an age where success is a universally acknowledged phenomenon that is quickly forgotten whilst failure instantly turns you into an Internet meme that follows you around for years, belittling someone in public, for "entertainment" purposes is wrong.

You hear this prank call tripe coming from stations in every country.  It's the same format too:

1. Play whatever song the record company is paying the station's owners to keep in rotation.
2. Make some asinine jokes about something you read in the paper and have no idea about.
3. Invite a caller to tell a funny story that extrapolates on whatever just got poked fun at previously.
4. Make a prank call.
5. Go back to playing another song that a record company is paying the station's owners to keep in heavy rotation.

Sorry, but why is this acceptable?

First, you have the DJ's. Fire them. Any DJ should want to aspire to something more than that.
Second, you have the public.  What as a radio station are you delivering? (and do not answer "some lighthearted fun and entertainment", because we all know that's a lie).
Third, where's the regulation?  We stopped putting people in the stocks and publicly throwing vegetables at them centuries ago, but this hasn't filtered into breakfast radio yet.

Someone needs to get some testicular fortitude and put a stop to this endless cycle of tripe before another bit of "harmless lighthearted fun" causes untold problems for some other poor hapless person.

Monday, December 3, 2012

Little Tools - Big Tasks

When you think of government weather agencies, you normally think of millions of weather readings being taken throughout history, of people trying to work out what the next forecast will be based on past experiences and current conditions, and of computers. Big computers.

A problem I repeatedly hit is that in trying to prove something I've noticed for a while (a man-made weekly atmospheric pressure pattern that exists at night in both Toronto and to a lesser extent, Montreal), all this computing power and online data publishing did nothing to lessen my workload in gathering more data to explore it further.

Environment Canada has a historical data product CD that you can download, and this is great for some general purposes, but there's two drawbacks:
1. It was designed as a DOS product, so only works these days on Windows in a CMD shell.
2. It has daily totals, but not the hourly data that I need.

Ironically, the hourly data exists elsewhere on their site and you can click on a link and download the hourly data for each city at one month at a time (the records go back to 1840 according to the drop-down box), but that means manually downloading and processing 172 years of monthly data for 19 cities.  

That's approximately 39,000 files (172yrs x 12mnts x 19 cities).

So having manually done Toronto over a period of a few decades (that took like a week to save all the files and manually import into a spreadsheet and then massage and crunch the data), I decided that I needed some help speeding up this process.

As timing would have it, I'd recently taken delivery of a Raspberry Pi (see details of that here), so I had a little spare computer that would serve two purposes:
1. Teach me more about Linux (it runs a form of Debian) whilst allowing me to program it in straight C (a rarity as I spend 99% of my time in Objective-C).
2. It could go and get all the data I needed, and crunch it all down for me.

The fun thing here is when you look at the stats for the two systems - The Environment Canada Supercomputer that crunches the data, and my little Raspberry Pi that turns it into something I can use:

Processors:  936 vs 1.
Power: 275KW (Enough for 200 homes) vs 0.5W (enough for 5 USB ports)
Cooling: Enough refrigerant to cool 32 Avg homes on a hot day, vs none.  

As you can see, there are two very different systems in play here!

In the case of Environment Canada, speed is important.  It takes 40 minutes to compile a forecast that would take a standard PC 28 days to compile.  In my case, I just want a database of old data that I can query in it's entirety.  To download an entire city's history from the 1950's takes about two minutes.  To get the entire history from 1840 takes substantially longer - somewhere in the order of ten minutes or so.

I have one more tool to write, which takes these downloaded files and prepares them to be loaded into a database.  I will do this on the Raspberry Pi as well. 

You may ask why bother doing it on the Raspberry Pi?  The biggest impetus for this is if I can make this work reasonably fast on a board with limited specs and which only costs $35, then it will work like lightning on a full-blown modern computer that costs $1000.  

This is something that is largely lost in an era when processor speeds and core numbers, memory and access speeds are increasing so fast that you can write shoddy software that may run slow this year, but not worry as next years faster processors will compensate for that.

When I am done, I will likely license out the tools that were created, as I know I'm not the only person that needs historical weather data that can actually be queried on a modern computer.