Thursday, June 27, 2013

Gold and the current economy


It's no secret that I like gold and silver.  In the 15 years (almost) that I've been in Canada, I've watched the dollar devalue about 1/3rd, as what cost $1 in 1998 now costs $1.34.  Here's the Bank Of Canada's Inflation Calculator to corroborate this fact.  


As seen in the media recently, gold has dropped about 35% from an all time high.  But what does this mean?

An ounce of gold is largely a static item in terms of what you can barter it for.  Historically, one ounce of gold should get you a decent suit.  About 400 ounces should get you a decent house.  But gold is priced in dollars - and this goes up and down.  The two main factors supporting currencies like the US dollar are how much faith people have in it and how much of it there is.  

The latter of these two is pretty simple to explain:  In short, imagine that you invent a new currency - your own dollar, for instance.  If it takes all the currency in the world to purchase everything in the world, and you only printed 1 million notes, then obviously the world would be worth $1m of your new dollars.  But when you start create 500,000 new dollars the old dollars become worth less.  Now it takes $1.5m of your dollars to purchase everything in the world.   

Whilst this is heavily oversimplified and somewhat incorrect, it gets the jist across about the fluctuations caused by printing (or as it's know now, "Quantitative Easing" or "QE").  Right now, the US Government is pumping some $85bn a month of new dollars into their economy.  The problem is the banks aren't lending this money out much.  Why would they lend out $100 in today's money knowing that if you pay back $120 with interest, it's only going to be worth about $80 when they get it back?

This is where faith comes in.  The government has to forge a sense of faith in it's currency, so it just announced that it's going to stop printing new money in the not too distant future.  This means that the perceived value of the dollar goes up as it's no longer going to be liquidated every month, and things priced in this currency (such as gold) should go down.  But prices in dollars shouldn't go down too fast, as that's deflation and they really want inflation, instead.

Conclusion
The gold price falling is not what it first appears.  People move money out of currency into gold as a store of wealth when faith in the currency is failing. Basically, when money is scared, it leaves the currency mechanism.  When the price of gold collapses, it means a spike in the value of the dollar.  Thus, gold acts a bit like a barometer.

The real test for this is still in Europe.  Everyone knows of the PIIGS (Portugal, Italy, Ireland, Greece, Spain) economic issues.  Since joining the Euro, they can't print more money, so they have to borrow it from other nations.  In the old days, they'd print money like Zimbabwe recently did - and look what happened there.  It's highly likely that in the next six months we'll see Italy ask for a bailout.  

When that happens, pandemonium will take hold and it's really anyone's guess as to what happens then.

Monday, June 24, 2013

Canada's "1 in 100 year" flood planning information

Most people would have seen the images coming out of Calgary right now, where flooding has caused some of the worst problems in living memory.  Despite the disturbing pictures, I find some of the information around this to be a little troubling as well.

Take Calgary's flood planning information.  It points out on the map where the river is.  Then it points out where the flood plains are.  After you reconcile in your head the fact that so much of the city has been built on known flood plains, you then spot the "flood plain based on 1 in 100 year plain" note...

Given we know that the whether is getting worse, rainfall is getting higher and extremes are getting more extreme, why did Calgary base it's current information on past numbers and data, instead of the actually saying what's possible?  To me, this is a very thought jarring oversight.

It's not just Calgary though.  I've been in Toronto for 14 years and experienced numerous "once in a lifetime" events.
  • the 2003 blackout, 
  • two storms with “100 yr” rainfalls, 
  • snowfall big enough to call in the army, 
  • and the SARS breakout that shut down many essential services.  
As I joked before, Toronto's "lifetime" definition must be close to that of a hamster lifespan (5 events in 14 yrs = one "once in a lifetime" event every 2.8 years.)

Looking at the Toronto Flood Plain map, we see the same thinking as in Calgary...  

Here, they used the wording "the Regulatory Flood Plain is based on the regional storm, Hurricane Hazel, or the 100 year flood; whichever is greater."  

This "retroactive-looking" mentality flies in the face of the currently known facts where we are seeing greater rainfall in summer storms due to higher temperatures creating more warm air (which in turn holds more moisture and leads to more rainfall).  This increase is supposed to continue for many decades (see the City debate for the 2040 to 2049 decade as an example).

Conclusion
It appears that Canada's cities only tell you a part of the story.  This is likely because nobody would want to live anywhere that people knew would have issues, so the cities would rather keep the areas populated and then deal with the flood claims and issues afterwards.

My advice therefore is to take what the cities do tell you, then add a multiplier of 3 or 4 to the data.  So, if they say they base things on a 100yr event, you'll probably be closer if you multiply that by 4 so that event is likely every 25 years.  It'll likely be more informative for planning purposes than what we're being told "officially" right now.

Sunday, June 23, 2013

Canadian Weather Project - More Data

I have finally finished populating the day names in my weather database.  Now, I'm querying it.  But something odd has shown up...

Here are the fluctuations in the average annual air pressure for 1961 to 2000 in Vancouver.  Things largely don't change, then in 1985 there is a downwards trend until the mid 1990s when it recovers by snapping back in the opposite direction (positive).










Now here is whitehorse.  It has a slight tendancy to wobble under the 0 mark, but also starts to get worse in the mid 1980s, before snapping back up at the last moment.

\


Next we come to Calgary.  From this point and onward travelling East (the direction that the wind blows) a new trend emerges where the pressure is dropping year over year, and getting lower with each year.  That same point in the mid 1980s seems to be a key moment in time for this trend really taking off.



We see this same trend in other places to the east of Calgary too, as long as they're on a relatively high latitude (higher than Toronto).

Examples:

Halifax

Thunder Bay 

Here's Toronto...  Notice that Toronto has largely the same profile as Vancouver?  It's likely that because much of Toronto's air comes out the of the American Mid-West, rather than from Western Canada, it's escaped the bulk of whatever is causing the pattern in the more northerly locations.  Again, the mid-1980s is a key point, but it's not as affected as more northerly locations.



My current conclusion is this: 
What I am seeing from Calgary and directly east of it, is likely the Canadian Oil Sands pollution.  The drop seen everywhere including Vancouver and Toronto from the mid 1980s matches with when the oil sands started up properly around 1985/1986.  

The spike seen in places like Vancouver and Toronto corroborates this early hypothesis.  The oil prices were crashing in the mid-to-late 1990's and the oil sands had to reduce production accordingly, as extracting from the sand is rather more expensive than standard well production.  Of course, history tells us that prices would soon recover and soar not long after these charts finish...

Friday, June 21, 2013

Seeing Trends And Cycles


We are surrounded by natural cycles, seasons, repetition and patterns.  You don't have to look far to see them.
* Every 60 seconds is a new minute.
* Every 4 seasons is a new year.
* After every day comes a new night.

We are also surrounded by man-made cycles.  Using a tool like Google Trends, you can visualize them.

Above, for instance is the term "AIDS" in Google Trends.  What we can see here is that near the end of every year, there's a world AIDS Awareness Month.  We can also see that the general public is getting tired of hearing every year about AIDS Awareness Month…



Above is "iPhone".  You can clearly see that it's announced in spring and hits the shelves in autumn.  You can also tell that people are still eager to hear more.

You can also pick up the darker side of advertising through tools like this.  Every year, all over the world, the WHO has a "World Mental Health Day"… Except in Canada, one of the national carriers relies on the public's short memory spans to promote it's own mental health initiative, which is really nothing more than an advertising campaign for it's own services where it asks Canadians to use the most expensive services that it offers (i.e. SMS and Long Distance) in return for a paltry 5 cents going to mental health.  

Here's that chart for this.




In this day and age, it really pays to take notice of these trends and look at them using tools like Google Trends to find out what's really going on.  After all, when you have a bigger picture, you have an advantage.
  



Thursday, June 20, 2013

Politicians and their moral compass



Here's one paragraph from this morning's Guardian newspaper in the UK, regarding the fallout from from the photos of Saatchi clutching Nigella Lawson's throat.  

This one paragraph outlines everything that is wrong with many top politicians and their ability to do what's right when faced with the opportunity.





I really don't need to say anymore on this matter, do I?

Wednesday, June 19, 2013

RBC Economics' Provincial Forecast Versus Real Life Barometers


This morning, 680 News alerted me through Twitter, that RBC Economics have put out another forecast that everything is still improving.  I had a few things to say about this banking spin in December of 2012.

So, lets just take a quick look at some everyday economic barometers, to see what those point to.

TTC Subway Advertising
When times are good, the overhead advertising in the train cars is full of big brands.  When times are bad, it's half-full (exposing a lot of white, blank space) and what advertising it does have is full of government programs to get the economy out of a rut and further education adverts or credit counselling adverts for you to get yourself out of your own ruts.

Verdict:  
Given it's about 70% sold and full of government and college adverts, it doesn't indicate things are good.


Bank of Canada Behaviour
When times are good, interest rates are up and borrowing is encouraged.  When times are bad, interest rates are low, and the message is to deleverage and pay off debts.

Verdict:  Interest rates are still at 1%, and the resounding message right now is to reduce household debts.


Mens Casual Clothing Colours
When times are good, everything is bright, brash and confident. Think of 1950s attire, or 1980s outfits.  When times are not as good, we turn to bland colours that don't stick out as very confident.  Look around you and everyone's about as colourful as they were in the late 1970s.

Verdict:
We are back to a lot of the browns and greys again.  Even in the office, the bright red and yellow ties are out and more subdued blues and maroons are in.



Product Advertising
During good times, advertising is all about how wonderful the product is.  During the bad times it's all about how crappy the competition is.  For men, it's how bad another truck is against yours if you purchase this product.  For women, it's how the competition's skin cream leaves your skin feeling sticky, but if you purchase this brand, you'll not have the issues the competition will give you.

Verdict:
Adverts are scrappy right now.  The other banks don't stay open long enough.  Other beers are too fizzy or tasteless.  


Conclusion
Looking around me, I'd say things are not indicative of a recovery any time soon.




Tuesday, June 18, 2013

The case for better public weather data


This morning, I was pointed by Chris St. Clair (@cstclair1) to an article on the BBC News site.

The article pointed out (paraphrasing) that "the Jet Stream is one thing they will consider, though the Jet Stream's position is a symptom and not a cause".  This caught my attention because the Jet Stream is of the metrics that I've been looking for in historical data, but it's never recorded anywhere that I can find it at least.  The second thing is it's not just the Jet Stream that is missing from our weather data…

As they correctly pointed out in the article, it's a symptom and not a cause… and so are many other things.  Anyone following my escapade into the Canadian weather knows that the day of the week is a factor, and that is also a symptom (likely pollution) and not a cause, yet they don't record the day of the week either (in my case, that's a pain as I have to recalculate 27 million reading dates to work out the day of the week and put that next to each record).

Other things that are not recorded:
  • What is our distance from the sun?
  • What is our angle of tilt to the sun?
  • Is El Nino or La Nina in effect?
  • What phase of the sun cycle are we in?
  • Are we being hit by a CME during that hour?
  • What is the local tidal phase?
  • What is the local atmospheric tidal phase?
  • What are the AQI readings at that time?


All these are just as important as the time, the date, the temperature or the atmospheric pressure at that moment, mainly because they add to the bigger picture of what's going on.  

When I look at the weather data that the public can generally obtain for free from places like Environment Canada, it feels like I'm only looking at one piece of the puzzle at any one time.  What we really need is to build a bigger picture so we can see what lines up…  Correlation doesn't necessary imply causality, but it helps us to determine where the links are.

In short, we need better data to be made available to the public.

Monday, June 17, 2013

Society, Products and Safety

This past weekend, I was talking to someone who says they don't let their kids walk the 300m to school on their own. When I was a kid at 7yrs old, I would walk to school on my own, and my school was a mile or so away.   This got me thinking; Is it really that unsafe in this day and age, or were we more naive in the old days?  

Most people will agree now that "things are not like they used to be".  But what does that really mean?

Somehow, we are in general agreement in acknowledging that its less safe these days.  Does this mean that if the current trend continues, the future will continue to get very unsafe?


People appear to be resigned to the fact that if we continue this trend they will be more prepared for it, so they purchase bigger fences, bigger cars, more burglar and panic alarms for their cars, houses, and kids, and at the same time start tracking anything is not bolted down and moves, so they microchip the cat, track their phone and GPS track anything else including the elderly.

This would indicate that people have little faith in things getting better and becoming safer, and despite the rhetoric you hear, nobody has any intention of fixing it either.  The usual citation is "We live in modern times, so we need to take a fresh approach", then the police, politicians and everyone else just goes back to doing what they did before.

Some people would argue that the media has pumped society's head with irrational fears so we will buy their new products.  Just take the average car advert for example - is it saying the car is really safe, or is it saying society is bad, and if you don't buy this car to protect you and your loved ones, then you'll be blamed for anything that happens to anyone?

There's some blame to be put here in advertising, but there's equal blame on society for not identifying this and actually stating that maybe we should look at fixing things instead.

Friday, June 14, 2013

Metadata, dragnet approaches and the law.


Traditionally, Metadata is descriptive data that is added to data, in order to categorise and identify it. So a database of crime records might be tagged as "shooting" to start with, then extra descriptive data is added to indicate the type of firearms being used.  Twitter uses metadata where you have the original data (the tweet) and embedded in it are hashtags to help you categorize that tweet so others can find it.  In Facebook, you have status updates and photos that are data in themselves, but you can add in the names of people to both of these.

It also applies to objects.  For instance, your first name is data that was added after your birth to identify you within a family - and your surname is metadata to identify your family unit in society, but the government doesn't see this as enough, so they add new attributes about you - such as seen on your passport hair/eye colour, or tax brackets and social insurance numbers.

It is this metadata that is at the heart of the NSA program that has now blown into a multinational farce of trust.

As always the media largely is missing a few things in what it tells people - take the Verizon phone data as an example:  You have the phone records which tell the government who has been phoning people that are suspected terrorist, but these records also hold routing information for the call, which in turn gives away your location.  Now, in the "US vs Jones" case, last year, we saw that installing a GPS tracking device on a suspects car did constitute a search.  If that's the case, then so does this.

And this is where the lines get really blurry.  On the one hand, I'm all for helping to stop terrorism but on the other hand, I'm not for it at the expense of flouting the laws.  

You can argue that dragnet approaches nearly always end up like this scandal has, where the people it's supposed to protect are actually the ones that feel the most hurt by it.

You can equally argue that this is where too much metadata leads us every time, too.  You only have to look at the privacy issues and concerns that we regularly see around Google and Facebook to understand that at the root of both is the amount of metadata they have on you.  They have so much, that both Google and Facebook know about people not in their systems.

That is ultimately what these government programs are trying to achieve as well. 

The Upside-down Nature Of The News Media

The news these days is back to front... or upside-down... depending on how you want to look at it.  Either way, it's priorities are all wrong.

Someone does something really bad to other people, there's a hero or two in the story, and the media spends 5 minutes on the heroes whilst banging on for weeks about the perpetrator(s).

What does this accomplish?

It gives too much publicity and airtime to those who were seeking it in the first place.  Thus, bombers get their splinter group in the news, gang shooters get the infamy they wanted, and the attention and validation for the reason of the tragedy goes to the person behind it.  

What we should be doing is the opposite; Putting the media attention and validation on the heroes that the tragedies create for the majority of the time and less attention on those that caused the trouble in the first place.

It doesn't matter if you look at the air time given to the Boston Marathon Bombers compared to the heroes of that day, or Al Qaida compared to the heroes of 9/11 - every time, it's the perpetrators that get the bulk of the attention.

We need to turn the news upside down.  For everyone's benefit.

Thursday, June 13, 2013

Toronto Garbage Collection Issues

Not a season goes by without me having reason to complain about Toronto Waste and their garbage collection practices.  If they're not skipping houses, they are only doing one side of the street.  If they're not missing bags and bins, they're leaving them strewn all over the sidewalk.

It's the leaving things across the sidewalk that bothers me most.  Given the guy in the truck is doing everything using a remote control joystick, he should be able to put the bins back where he found them - and as a bonus, upright would be nice, too.



Since the old lady across the street died recently, I've one less reason to campaign for improved workmanship.  However, my significant other still has to push the twins up this street and it's a pain in the backside to have to do the city's job of moving the bins back, in order to get through.

Credit where credit is due, the directors and managers at City Hall do a good job of responding to this idiocy, but the guys on the street have no pride in their job, so any solution is temporary.  

In weeks, it's back to this:




We're long due the time to privatise this, so that there's a sense of accountability at street level and we don't have to put up with this, and get these guys off the job if they can't do it properly.



Wednesday, June 12, 2013

Apple and Media Bias

Few things annoy me more than one-sided reporting...  One thing I'm particularly getting sensitive about is the reporting on Apple.  

It's getting out of control.

Take this article from the Ottawa Citizen, that I'm going to disassemble for you, and point out the flaws in the article:

It starts with this tagline:
"Company offers nothing new regarding iPhone, iPad or any 'game changers'"

But to contradict that tagline, it contains these sentences:

  • "iOS 7 - unveiled Monday at the company's annual developer conference"
  • "A new version of the company's desktop operating system, OS X Mavericks, alongside refreshed laptop and desktop computers, was also revealed."
  • "It was enough for Apple's senior vice-president of worldwide marketing Phil Schiller to exclaim "Lack of innovation my ass" on stage, as he unveiled an update to the Mac Pro - redesigned and much smaller version of its desktop workstation aimed at high-end audio, video and creative professionals, and the company's first update in nearly two years."
  • "Also unveiled were next-generation MacBook Airs - Apple's thinnest laptop - with next-generation Intel chips meant to improve performance while extending battery life by hours."
  • "But the biggest applause came for Apple's newest version of iOS, which boasts a completely redesigned user interface spearheaded by the company's vice president of industrial design, Jonathan Ive."
  • "Some of the operating system's new features include updates to Siri, Apple's voice-activated personal assistant; improvements to core Messaging, Calendar and Mail apps; and a new quick-access settings menu called Control Centre that is invoked with an upward swipe from the bottom of the screen."
  • "The company also detailed its long-rumoured streaming music product, iTunes Radio."
So a radically designed new OS for the mobile platform, a new OS for the desktop, a new radio service, a new desktop computer that's radically designed, a new generation of laptops and a suite of new iWorks (not mentioned in the article) is enough to justify the tagline "Company offers nothing new regarding iPhone, iPad or any 'game changers'"
 
Time to start getting news from a better source by the looks of it.

Tuesday, June 11, 2013

100 Posts

It was last September that I was stuck in a hospital looking at a sick baby who'd been sleeping for days on end, and I was getting really bored.  I'd already explored every nook and cranny of the room, the equipment, asked every question of the nurses and doctors, and had even seen an exhibition on the history of the hospital.

I was really bored.  So I started writing.  Unlike the last time I was blogging, I don't really have a theme other than "things I come across in daily life".  I wasn't actually looking to promote the blog in the beginning.  It was just my mind rambling on about the things I've seen and experienced, and what I thought of it.  

Now, I'm at 100 posts.  

What has changed since the beginning is I do a few things to drive traffic to the blog, such as stumbler posts, tweets, etc.  To start with, this didn't have that much of a measurable effect, but now it does.  

The biggest reason is critical mass - after about 70 posts, Google started to direct more search request results to my posts.  Now I'm at 100 posts, about 1/3rd of my traffic comes from organic searches on Google, the other 2/3rd's is blog promotion (which takes all of 30 seconds a day).  

Naturally, I am keeping a spreadsheet of daily totals and such, because I like to measure the effect of more content (tweaks and changes to layout is largely unproductive).  My main strategy is to take the raw visits per day and generate three moving averages on a 10 day, 20 day and 30 day cycle.  As long as my daily visits is over the 30 day moving average, then I'm doing fine.  Currently, I'm gaining about 5% a week, which doesn't sound like much, but then again, this is compounding every week.

At some point, I will do a proper breakdown on what I've done and give some proper numbers, but for now, I'm just looking to keep posting new content as I've discovered that I'm effectively banking the reasons for people to visit the blog.


Apple iTunes Radio vs Traditional Broadcast and Streaming Radio



As you'll no doubt have heard, Apple has announced iTunes Radio.

I am a lifelong radio fan.  I prefer radio over TV as I can listen to music whilst doing something else, whereas TV requires me to pay attention to it, which means I can't look at, or concentrate on other things.

When I was a kid in the UK, I used to have many hobbies to keep me busy, and some of these revolved around radio.  
* Sunday evenings I would record the top 40 to a cassette to play for the rest of the week on my walkman as I walked to and from school, for instance.  
* I used to try and find alternative frequencies for the stations I liked so that on car trips when we went out of range of one transmitter and the music died, I already had the next transmitter frequency ready to retune the radio.
* I used to do DXing - which is long distance radio listening.  I would then write to the radio stations and send them a SINPO report.  This invariably got me free swag like tshirts and stickers from the likes of Radio Sweden, Luxembourg Radio, etc.

Then I grew up.

By my early 20s, one job I had involved streaming data to floating fish factories off the Falkland Islands.  This required knowledge of atmospheric skip, baud rates, YAGI antennae, and YAPP Packet Radios.  Basically, I'd aim from Kent in the UK south and a bit to the west, so I beamed over the western edge of Africa, sliced the Eastern edge of South America, and hit the ship's area to within a few hundred miles.  And then waited as we beamed data at "snail on valium" speeds of almost 300 bits per second (note we're not talking kbps, but bps!).

As an adult, my early memories of radio haven't left me. I've watched radio grow up too.  Seeing DAB and streaming radio is great in my mind.  RDS EON and such are wonderful additions to radio - if only stations would use it properly.

I still listen to radio on a regular basis.
* Every morning I listen to Capital FM out of London, UK.
* Every weekend I listen to Fresh FM out of Adelaide, Australia.

Later when the iPhone came out, I jumped on to this platform as I was getting bored of Windows Mobile.  It wasn't long till I inherited an extremely well known streaming radio app to look after, fix and expand.  I won't say who the customer is, but the app wasn't in a good state.  It didn't work very well, and only had 2 stars in the app store, despite being used by almost 8 million people.  

Now, I know a thing or two about Radio - and I definitely know more than any exec who judges his radio by looking at spreadsheets of listener numbers, earnings, and advertising pipelines.  First up, I actually spent a lot of time trying to use the product.

In the two years I spent working on that product, I did the things the execs wanted, like branding, pixel pushing, metrics, gloss and visual stuff, whilst simultaneously slipping in "freebie" items that I wanted and assumed other radio listeners wanted - like stabilizing the app so it doesn't crash every 30 minutes. I also added the ability to switch apps without the app crashing.  I added headphone support so you could pause the station when crossing the road.  I added the ability to fade out the station and resume when answering and hanging up phone calls (previously it just crashed).  I improved battery life.  I added support for docking on external base station speakers.  

Meanwhile, the execs were still flip-flopping over whether to lighten up the background, or darken it.

As you can see, I had a different outlook on what was important in a radio player.  By the time my stint at looking after this app was done, I'd stabilized the thing to run till your battery died, had 22 million users, and 4 stars in the App Store, 

This outlook is also what sets me on a constant collision course with most North American radio stations.  Radio here is generally in decline as its cookie-cuttered rubbish. The "Adult Contemporary" stations are the worst as you can be in any city in North America and pretty much hear the same 20 songs played over and over.  

Streaming radio to these broadcasters is generally a way to send this same rubbish to a mobile device.  Therefore, streaming radio to them is treated like a survival mechanism for the status quo.

Services like Pandora and Spotify have identified this radio as rubbish, so are trying to build an empire based on eating the broadcast guys lunch.  Thus, digital streaming to them is a tool to build their empires on.

Apple has an empire already.  Whilst it has no wish to get into radio broadcast, streaming is another tool to add to its arsenal. If people buy a song from their service, that's 30% commission that they don't have to give to Clear Channel or Pandora or Spotify.  

When you understand this fit for Apple compared to existing radio stations, you'll laugh at some of the comments coming from these other players.

In order to survive, radio stations need to provide a quality service.  This is why with streaming radio I listen to quality stations - but a new simpler radio player from Apple is also good, as it should sort the wheat from the chaff and make any cookie-cuttered stations that put out rubbish step up their game. 

Those that don't will disappear fast.


Tenacity, Progress, And Uncertainty

Those that know me well, know that I always have a to-do list as long as my arm.  If I've a few hours free, I'm likely to chip away at these projects.  Some projects are quick, and others take time.  One of the longest running projects I've had now is weather related - which started some 4 to 5 years ago and having discussed it with Environment Canada, I was left with more questions than I had to begin with.

Having spent a lot of time mulling over the problem, I finally got around to progressing it forward.  Unlike many of my projects where I go gangbusters on it for a week and I'm done, this one has been done at a much more leisurely pace.  I've allowed myself to deliberately take the most painful and difficult paths I could find, so I can say "I did that".

For instance, I could of just written a bit of script to go and download the weather data I needed.  But instead I chose to install a Debian squeeze on a Raspberry Pi, then code a custom utility in C using "nano" as my editor to grab the weather files.

The resulting code looked like this.

1:  /*  
2:      Name:         getcsv.c  
3:      Copyright:     (C) Jason Coulls, 2012  
4:      Purpose:     Downloads Environment Canada historical data to a Raspberry Pi,  
5:                   using the date ranges supplied for the specified station ID.  
6:  */  
7:    
8:  //#define CURL_STATICLIB  
9:  #define MAXPATH 255  
10:    
11:  #include <stdio.h>  
12:  #include <string.h>  
13:  #include <time.h>  
14:  #include <curl/curl.h>  
15:  #include <curl/easy.h>  
16:    
17:  size_t write_data(void *ptr, size_t size, size_t nmemb, FILE *stream) {      
18:    size_t written;  
19:    written = fwrite(ptr, size, nmemb, stream);  
20:    return written;  
21:  }  
22:    
23:  int main(int argc, char *argv[])  
24:  {  
25:    
26:      //Must have five arguments.  
27:      if( argc != 5){  
28:          //Let the user know how this should be done.  
29:          printf("Usage: getcsv [start mm/yyyy] [end mm/yyyy] [station id] [output dir]\n");  
30:          printf("The file name will be automatically named.\n");  
31:          return 99;    //Error 99 - Replace User and Press Any Key To Continue.  
32:      }  
33:    
34:      //Declare where we'll store our arguments.  
35:      char strStartDate[7];        //mm/yyyy  
36:      char strEndDate[7];        //mm/yyyy  
37:      char stationID[6];        //4-6 digits.  
38:      char strOutput[MAXPATH];    //The directory to save file in.  
39:    
40:      //Grab the supplied command line parameters.  
41:      strcpy(strStartDate, argv[1]);  
42:      strcpy(strEndDate, argv[2]);  
43:      strcpy(stationID, argv[3]);  
44:      strcpy(strOutput, argv[4]);  
45:    
46:      //Get ready to store the broken down dates.  
47:      int startYear, startMonth;  
48:      int endYear, endMonth;  
49:    
50:      //Break it down.  
51:      sscanf(strStartDate, "%d/%d", &startMonth, &startYear);  
52:      sscanf(strEndDate, "%d/%d", &endMonth, &endYear);  
53:    
54:      //Now we loop through each year between these dates.  
55:      int currentYear;  
56:      int currentMonth;  
57:      for(currentYear = startYear; currentYear <= endYear; currentYear++){  
58:    
59:          //Now we loop through each month between these dates.  
60:          for(currentMonth = startMonth; currentMonth <= endMonth; currentMonth++){  
61:    
62:              //Make this into a date.  
63:              struct tm * thisDate;  
64:              time_t now = time(NULL);  
65:              thisDate = gmtime(&now);  
66:              thisDate->tm_year = currentYear;  
67:              thisDate->tm_mon = currentMonth;  
68:              thisDate->tm_mday = 1;  
69:              thisDate->tm_hour = 1;  
70:              thisDate->tm_min = 1;  
71:              thisDate->tm_sec = 1;  
72:              thisDate->tm_isdst = 1;  
73:    
74:              printf("Processing: %d/%d\n", currentMonth, currentYear);  
75:    
76:              //Download the file to the destination.  
77:              //http://www.climate.weatheroffice.gc.ca/climateData/bulkdata_e.html?timeframe=1&Prov=XX&StationID=5097&Year=2012&Month=11&Day=&format=csv  
78:              char urlString[1024];  
79:              snprintf(urlString, sizeof(urlString), "http://www.climate.weatheroffice.gc.ca/climateData/bulkdata_e.html?timeframe=1&Prov=XX&StationID=%s&Year=%d&Month=%d&Day=&format=csv", stationID, currentYear, currentMonth);  
80:    
81:              //Having built the url for the month's data, now request it using CURL.  
82:              CURL *curl;  
83:              CURLcode res;  
84:    
85:              //Work out where we're saving this file.  
86:              FILE *filePointer;  
87:              char outputPath[MAXPATH];  
88:              snprintf(outputPath, sizeof(outputPath), "%s/%s-%d-%d.csv", strOutput, stationID, currentYear, currentMonth);  
89:              curl = curl_easy_init();  
90:              if(curl) {  
91:    
92:                  printf("fetching %s\n", outputPath);  
93:                  filePointer = fopen(outputPath,"wb");  
94:                  curl_easy_setopt(curl, CURLOPT_URL, urlString);  
95:                  curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);  
96:                  curl_easy_setopt(curl, CURLOPT_WRITEDATA, filePointer);  
97:    
98:                  /* Perform the request, res will get the return code */  
99:                  res = curl_easy_perform(curl);  
100:    
101:                  /* Check for errors */  
102:                  if(res != CURLE_OK){  
103:                      //Tell us what went wrong.  
104:                      fprintf(stderr, "curl_easy_perform() failed: %s\n",  
105:                      curl_easy_strerror(res));  
106:                  }  
107:    
108:                  /* always cleanup */  
109:                  curl_easy_cleanup(curl);  
110:                  fclose(filePointer);  
111:    
112:              }  
113:          }  
114:      }  
115:    
116:      return 0;  
117:  }  
118:    


As you can see, the code isn't rocket science, but doing it in nano means no debugger, and no help to see what went wrong.  As a result, I quickly got my raw C coding skills improved.  To continue the theme of scaling the most difficult way possible, I decided on compiling the thing using the GCC compiler on the command line.  

Seeing this come to life on the Raspberry Pi device was rather a big thing for me.  I like seeing things come to life - no matter how small.  

The next problem was I had to assemble the data into something that could be used.  For this, I set up a spare Mac Mini OS X Server with MySQL on it, and built a quick bit of PHP to populate it.  This took another two weeks to complete. (Partially due to two outages - one caused by our neighbour catching fire and knocking out the power, and one caused by one of the kids interfering with the laptop driving the PHP - and as I didn't know how far through the current city it was each time, I had to delete and reimport the current city).

As I move forward, once I've populated the 27million day names (Monday, Tuesday, etc) for the readings, the biggest issue is once I've worked out if the problem I've noticed with Toronto is across Canada (I know it's a similar problem in Montreal), what to do with that answer.

I've started running some "pre-flightcheck" runs of reports I've created whilst the rest of the cities are still imported.

Here we have some cities and their "Average kPa" over the past 5 decades.  Note the dip in the 1990s.



Next, we have the same data flattened out and compared with more cities.  The big anomoly here is Charlottetown PEI.  I'm not sure if that's Environment Canada's data yet, or a real anomaly...


So, as you can see, I'm starting to get some fruits from this labour.  I'm almost done importing, then it'll take a few days to populate the days.  Then, I can start the reports I have built running.

Then I have some questions to ask.

I know I need to check the following:
Is the weather pattern coast to coast?
Is the weather pattern recent, or has it always been there?
Is it getting more or less pronounced?

Then I'm sort of stuck again.

See, I'm not a meteorologist.  I'm just a guy that has spotted something man-made in the weather cycle and is trying to get to the bottom of the cause of what it is I'm seeing.

Fun times, eh?

Monday, June 10, 2013

What to expect at WWDC 2013


Today is a big day for those of us who work in or around the Apple ecosphere.  It's the World Wide Developers Conference (aka "WWDC", and also erroneously known as "The WWDC").  

This is one of several points in the year where Apple unveils new technologies like the next iOS version, but the reason this one is so important is because the programmers who make the apps you love are brought up to speed in the next few days, and that has a knock-on effect for both desktop, and mobile apps - as well as desktop/laptop hardware.  Also, this is the time where iCloud and "back end" stuff is unveiled.

Traditionally, Steve Jobs would do one of his "keynote" speeches here.  In fact, the software "Keynote" (Apple's version of PowerPoint) was largely created to allow Jobs to do these presentations.  These days, it's Tim Cook that does them - however, this year were likely to see more people up at the front of stage as the likes of Sir Jony Ive, Craig Federighi and Phil Schiller establish the prominence of their roles more.  

I'm not going to pontificate about what should or should not be unveiled - there's plenty of articles out there about that today - but I do feel it's important to point out the significance of the event.

* This is a very big event for programmers, but the media only focuses on the keynote presentation the next three days are largely ignored.
* The iOS updates are the biggest of the year.
* The hardware updates are likely to be the major portion for the year, excluding iPhone and iPad, which have their own event around September, ready for the Christmas rush.
* This year you won't see Scott Forstall, since he got fired over the maps unveiled at this conference last year.
* This is the first time we'll see what Sir Jony Ive has done to the user interface.  Bear in mind that unlike Jobs, Ive is not a fan of "skeuomorphic" UI features (i.e. leather toolbars, fake linen backgrounds, etc).  He's likely to go for something much cleaner (remember - Ive designed the iPhone, iPod, iPad, Unibody Mac's, etc - he's going to head in the "minimalist" direction.) 

Friday, June 7, 2013

The Missing Bit Of The NSA PRISM Story


Unless you really haven't been paying attention to the news this week, you'll be aware of the NSA's "PRISM" program.  On the one hand, there are a series of sources that say beyond Verizon handing over ALL call information, other companies are participating in the program, such as Facebook, Google, Microsoft, Skype, and so on.  On the other hand, there are now a number of statements from the same companies that categorically deny that they are participating.

Either way, there's a few things that are not coming up in the news stories because what we're seeing is very little "original thought" put into these stories - so it's just the same fodder being sent around and around in a loop.  

What's missing is this angle...

This program is supposed to help anti-terrorism efforts, and you'll probably remember that one of the biggest failures with 9/11 was the perceived failure to "connect the dots".  The basic response to this was "if we collect more dots, we might have better success at connecting them", which seems logical enough on the surface of the argument.

The need to harvest information was there, but not the technology to effectively harvest this information at a commercially acceptable price… 

...until three months later, when on December 2nd, Enron filed for bankruptcy.  

In the aftermath of the Enron scandal, we learned how Arthur Anderson had shredded much of the documentation, and deleted a lot of email correspondence.  This ultimately led to the Sarbanes-Oxley (also known as "SOX") rules that stipulated what you could or could not retain according to the law, and served as a backup so that when errant directors foul up a huge company, an audit trail of everything they've ever sent/recieved is still retrievable - even if the directors have hand fed their laptops and phones to a smouldering volcano to hide the evidence.  

In order to help with "SOX Compliance", a multitude of new technologies sprung up that could target every major system in existence, from sales, to customer relations, to emails and VOIP calls.  

Once the initial rush was over to become "SOX compliant", you had a lot of people sitting around in technology with useful knowledge and skills pertaining to the mountains that they had just climbed.  

Now, it doesn't take a genius to work out where that "know-how" went next.  But there's a problem with this approach:   Whilst the technology to harvest more "dots" has arrived, most people still aren't connecting them - which was the original problem to start off with.   

To put this problem in a way a five year old kid would understand:  Imagine you had problems harvesting all the fish in the oceans, collecting up all the oceans waters into a huge fish tank would would ultimately leave you with all the fish in the world - but you still have to apply the same methods of trawlers and fishing boats - just now they're in your huge fish tank too.

If anything, they're just going around in circles, and unlikely to garner much more info than they had to start with.

Thursday, June 6, 2013

Moving My Own Little Mountain


Last year, I returned to a question that has been bugging me for a number of years with Toronto's weather.  The problem was that in order to find the answer, I needed some data - and whilst the data is freely available to answer the question, it's just not in a form that I could easily query.  It was very obvious I was going to have to construct something pretty big at home, just so I had all the data in one place in order to query it.

So, back in the winter, I wrote a program for the RaspberryPi (http://www.raspberrypi.org/faqs) to grab the data I needed for 18 locations across canada, for every day on record, and put them somewhere for me to process at a later date…  Naturally, the Raspberry Pi did this job admirably.  OK, it was slow, but then again, for a $35 computer I wasn't expecting speed.  What the Raspberry Pi did do extremely well was allow me to teach myself a few things I didn't know before - like how to build using the GCC compiler on a linux distro that I'd installed from scratch.

What I ended up with was ~37,000 files to process.  This was a bit of a tall order especially as I was a new dad with twins, so the entire process had to be put on hold for a bit.

Now that it's almost summer the next year, I finally have some time to deal with this.  First, I had a look at the files.  They are CSV's with 17 lines of junk above the data that had to be stripped out, followed by another ~720 lines of data per month (one line for each hour, 24 hours a day, multiplied by 30 days).

So, I created a MySQL database with the corresponding fields for the stations and the readings.  Next, I created a simple PHP script to enumerate each folder, and for each file, strip out the header junk and for each line of data, break it into the correct fields and load these as a record into the readings table.  This makes it approximately 1.5m records per station. 

Time to load this is about 5 hours over wifi from my laptop to a Mac Mini server - again, speed isn't the concern here - rather that I have a way to get the files parsed and loaded with about 1 hour of programming time invested in a PHP script that I can easily repoint to new folders.

This will likely take a week or so (process one city whilst away at work, and one city through the night as I sleep) to get all 26m to 27m records in the system.  Then the next step is to augment the data.  The biggest item is to add the day of the week.  So, this needs to be done as well.  

If all goes well, I can get round to querying this data to get some answers by the end of the month!

Understanding the purpose of something

Yesterday, whilst I was on my soap-box on twitter ranting that we should have scientists instead of judges, after another silly move by a judge (this time in India, claiming that girls sometimes enjoyed rape - thankfully he quit later in the day), someone told me a "judge is knowledgeable on the law, and that is the topic of the case."

Needless to say, I disagreed.

The topic of a case is the event that needs to be inquired about, in order to find out the truth of what happened, whether it be a homicide, or fraud.  The event where someone got wronged is the topic of the case - the law is not the topic.

Once the truth about what happened during an event is determined, then it's just a case of checking to see if that action is against the law, and if so, what the punishment should be - or, if necessary, what correct the consequences of the action.

It's a slippery slope to think that "the law is the topic".  If you put the "law" cart before the "society" horse, it fails to protect those it was put in place for.

The icing on the cake here, was when I told this person (who said the above to me) that I already had a canned response to her side of the debate and presented a previous blog post as evidence, she didn't want to see the evidence and dismissed it.  

Oh, the irony...

Wednesday, June 5, 2013

Observations, Safety and Reasoning


People who follow me on twitter or this blog will know that I tend to find a lot of faults and issues in my daily life, and I raise many of these with the people responsible, so that they get fixed.  What people don't usually understand is why I find so many?  

In short, I pay more attention to my surroundings than the average person does and I question what is going on around me, and I often question why things are the way they are; Is something done a certain way because it's easier, or is it more profitable?  Would it be safer done a different way?

This sometimes leads to a sort of "snowball effect" because once I have spotted a few issues with something, I can often guess what the pattern of failure is and test the hypothesis.  Once I've proven something, I raise it as an issue, and generally the provider of the service or product improves as a result.  However, not everyone appreciates me pointing out ways to improve things.  When the issue is electronic, this is especially true.  

People like to think that their service providers and product vendors are doing everything they can to protect them - but the reality is often they're not.  They advertise that they are safe and are protecting you, but then they threaten to come down heavy on anyone that demonstrates any weakness in their systems.  They then threaten to escalate proceedings against anyone that repeatedly demonstrates any flaw they haven't fixed…. 

This leads to an interesting side effect:  When a large corporation has a security hole and people know that the hole exists, and more importantly the corporation has threatened legal action against people demonstrating further holes, the holes multiply because people know the holes are not being fixed and have already demonstrated that the corporation doesn't know where the holes are (because if they knew where they were, they'd close the holes), whilst still further holes can be pinpointed (which also can't be reported to the corporation for fear of legal action). 

Eventually, you can connect these dots and predict where the holes will be without having to use a computer at all, because whilst the corporation has put it's head in the sand, you've had a good chance to observe "the rest of the ostrich" without touching it.

Explaining Eddie Stobart to North Americans


If ever there was a cultural phenomenon that illustrated two ends of one spectrum between North American and British culture, it has to be trucks.

In Canada and the USA, the archetypal connotation of what constitutes the average trucker is some middle-aged butch guy with a paunch, possibly a beard, baseball cap and reflective shades - sitting in a cab that he has "personalized" in his own special way with junk on the dashboard, lights on the outside and some burgee or other flag plastered across the bunk area, so that he can sit there in his checkered shirt and white tshirt and feel right at home.

In the UK, however, the archetypal trucker is the Eddie Stobart driver.  To explain what happened here, you have to go back to the origins where a certain Mr Stobart had a small fleet of trucks and these were done out in a nice professional livery, and the drivers themselves had to be dressed nicely too.  Shirts and ties being the order of the day - though these days the ties are gone.  

The trucks themselves all have female names - with very few exceptions ("Optimus Prime" comes to mind as one and one train is called "Eddie the Engine").  This lead to people collecting and recording what trucks they'd spotted and eventually a fan club was born.  The drivers will also pose with kids for photos of the kids in their tractor units, and they have to remain professional and courteous at all times.

These days, Eddie Stobart trucks are a big part of British popular culture.  For example, here's their CD in the shops right now.... and that's not the first time they've been musically linked to the pop charts, either (Look up "I want to be an Eddie Stobart Driver" if you really must).

In addition, you can buy toy trucks for your kids, or collector edition models such as these.


This also extends into the home - so if you really want, among other things - you can buy the duvet for your kids room.


And all of this came about due to three things:  Being personable, dependable and professional.  

So, the next time you see a "tough" local trucker jumping out of his truck, spitting phlegm, swearing, and complaining about the traffic, you will now hopefully understand why his employers can't sell you a pair of adult sized fluffy truck slippers to commemorate this.




What you should agree on, is that this is branding and advertising done very, very well.

Tuesday, June 4, 2013

Bell Canada, Yellow Pages Group And Your Data


I'm now into month four of my ongoing issue with Bell Canada and trying to find out who else they gave my information to besides Yellow Pages, and something interesting just turned up during a double-check of Yellow Pages and their processes.

You may remember that Yellow Pages said they wouldn't remove my details because it'd been handed to them by Bell Canada, and therefore Bell Canada would have to request the removal of my details on my behalf…  After much ado, Bell Canada put in the request to Yellow Pages to remove my details.  Yellow Pages said they would comply and at first it looked like they'd actually followed through with this request.

So, about that removal process….

When Bell Canada requested that Yellow Pages removes your data from a YPG site, it appears that YPG only honour the removal from the specific site that you request - and not other sites that Yellow Pages Group owns which Bell has allowed information to propagate into.  

For instance, I asked Bell Canada to ask YPG to have me removed from Canada411.ca, and YPG left me in 411.ca, which they also have their name on.

This process is clearly broken.