Australia's got talent?

Mike Cannon-Brookes was on the radio. Slamming the government's drive to cut down on 457 visas. “We don't have the scale or scope of talent here”. For some reason his words were making me angry. Red hot under the collar, steam coming out of my ears type angry.

So I had to reflect. Was my anger because I'm not sure we actually have a shortage of software developers in Australia? Or that I think that employers in this industry are continually trying to hire people that they don't have to train? No. They might have been a contributing factor to my internal steam levels, but they weren't the cause.

I was getting annoyed because of the use of the word “talent”. To me it is such a nebulous term. What on earth do we mean when we use the word “talent” in the software industry?

I have a dread suspicion that it is an unconsciously used synonym to indicate “superstar” employee. That the real complaint here is not that we do not have enough developers, rather, that we do not enough superstar developers.

Trying to break this down a bit, first off, what is talent? In this article Scott Barry Kaufman, adjunct assistant professor of psychology at New York University, tells us that talent and practice are intertwined. And that 10 years of deliberate practice seems to be the general rule when it comes to developing deep domain specific knowledge.

Which leads me to wonder if the speaker at the developers breakfast I attended a few weeks ago, targeted the right thing when he suggested that we use the latest “hot” languages and tools to attract the “young talent, fresh out of university”. Surely we should be looking to attract the older and more staid developers, rather than avoiding them, if we are looking for deep domain specific knowledge?

Scott Barry Kaufman goes on in the article to tell us that talent can be see as the mix of characteristics that allows someone to either accelerate the acquisition of the deep domain specific knowledge, or to perform better than someone else who has a similar level of practice and learning.

He then notes that the interaction of genes and environment play a distinct role in helping us to develop by increasing our abilities according to our talents, and in turn those personal characteristics that we call talent. In the article he finishes by stating that “we shouldn't dismiss the seemingly untalented” and that we should in fact be “keeping the door open and instituting a dynamic talent development process where the only admission criterion is readiness for engagement”

Which is not what I feel we are doing in the software industry.

I have two dead canaries in this coal mine.

The first is Ted Neward. I've never met Ted Neward, and I have no idea what he is like as a person. However, I do know that not only does he write books and articles, he has also written what is to me one of the most insightful blog postings I've ever read in my career in software: The Vietnam Of Computer Science. I believe that the world would actually be a better place if this blog posting was required reading for software developers.

So if Ted Neward was interested in working for a company I was at, I would be delighted, as we could learn much from him. I would be trying to find out what he was like as a person, and seeing if he was a good fit for the people he was going to be working with. I certainly would not be getting him to write a programming test.

However, in this whacky industry: yes, he was required to write a programming test. And was rejected on that basis!

My second canary is someone I met socially, recently, right here in Melbourne, who has been unemployed since graduating two years ago. One of the desirable “young talent, fresh out of university”. Unable to find work, he has been getting by through tutoring and part time teaching at the university he graduated from. So. Degreed, good enough to be invited to tutor and teach, but not “talented” enough to hire, in a country where we are claiming to have a shortage that is so bad we need to bring in people from overseas on special visas.

What seems to be happening is that the software industry has adopted some sort of heard mentality as to how to hire people. Heard mentality in hiring is apparently normal. Not rational, but normal.

As ever, though, we've taken it to extremes by becoming increasingly set on using coding tests to find out if potential hires have “talent”. But really, how do you set a coding test for a mix of characteristics that will indicate that the person in front of you has the potential to be a better developer than someone with a similar level of experience?

The truth is that if we as an industry hire like this, a shortage of people passing our tests doesn't indicate a shortage of developers: it indicates a shortage of developers that companies are prepared to hire.

Some thoughts have been bothering me about all of this:

• If anyone is going to have to pass a test showing their ability at hiring time, then I would very much like it to be, say the brain surgeon who may be about to operate on me. Strangely enough, the medical profession doesn't seem to work this way. Why is that?
• Are we actually moving to a world where some newly graduated developers are not going to find work in the industry because they aren't “talented” enough? My second canary seems to indicate that this may indeed be the case.
• Is this the hoary old chestnut about developer productivity raising its head again?
• Why don't we regularly test existing employees? If recruitment tests are actually useful in determining “talent”, then it would be good to regularly check to see if our internal processes are helping our talented developers to further their acquisition of that deep domain knowledge?
• How many companies regularly review their recruitment process and measure its effectiveness?

Having pondered the Scott Barry Kaufman's article on talent, I've decided that if a developer can pass a simply “fizz buzz” test, then they have enough “talent” to hire. End of story.

The onus should actually be on the companies that hire developers to create environments that are conducive to developing “talent”. Rather than creating ever more stringent and time consuming new hire procedures, they should be “instituting a dynamic talent development process where the only admission criterion is readiness for engagement”.

Just about the only Australian software company I'm aware of that seems to do this, by design or accident, is Atlassian. The company founded and run by Mike Cannon-Brookes.
Go figure.

A click is not enough

In the Maldives a 15 year old girl has been sentenced to 8 months of house arrest and 100 lashes from a whip. Her crime? She confessed to having been raped. This is, to our western sensibilities, clearly outrageous. A travesty of justice.

Something must be done to help her.

In Uganda the government is close to passing a bill that will introduce the death penalty for homosexuality. Again, to us in the west, a clear breach of basic human rights.

Something must be done to stop this terrible law being passed!

The Rhinoceros stands on the very brink of extinction, the surviving few at risk of being killed and dismembered so that some Chinese and Vietnamese people can drink the powdered horn in the form of a tea, to cure their ailments. Whilst I can sympathise with the fear a cancer sufferer must have, the world stands on the brink of loosing a wonderful animal because of some unscientific belief system.

Something must be done!

I'm sure I'm not the only person in the world who is aware of these terrible injustices.

For they flood our email inboxes, crowd our Facebook pages and fill our twitter streams. Sent to us by our caring friends.

The truth is we can all take action about any perceived injustice right now! Thanks to modern tools, anyone can create and distribute an online petition. And invite their friends to join in the outrage by signing, cascading the anger on through the world.

Coca-Cola have brought and won a court case against deposits on containers right here in Australia, dealing recycling a tremendous blow.

There's a petition about that!

Monsanto are patenting genetic sequences: the building blocks of life, and charging people 'licensing fees' to grow crops that contain those genetic sequences.

There's a petition against that!

But what good are all these online petitions if all we ever do is simply sign them?

We've signed the petition: we've done something. Our consciences can rest easy! Now to plan our holiday to the Maldives, whilst munching on our genetically modified granola bar and sipping on our Coke.

Why fear online petition if the only effect it has is to stroke the signatories ego?

In fact, if the signatories now feel that they have acted on an outrage, on-line petitions might make wrong doing easier to get away with!

“They won't do anything: they've signed the petition. Their consciences are clean” I can hear the captains of industry and politics smirking.

Given the ease with which we can now create, sign and distribute on line petitions, I believe that we now have to do more when we become aware of an injustice.

You have to decide if you care enough about the injustice to take further action. If you don't care enough to anything more, then I believe you shouldn't sign the petition.

If you do care and sign, then, you have to work out what more you are going to do, and how you are going to let the parties know what you are doing.

The reason Coke gave for their lawsuit against the container deposit scheme was that “Australian families do not deserve to be slugged with yet another cost of living increase”

Knowing of the huge islands of plastic that is found in the oceans this justification made me really angry.

The Australian CEO of Coke received compensation of just under $8 million last year. Assuming that they make a 10% profit per can of Coke sold, and that a can sells for $1.20, that means that Coke Australia have to sell over 60 million cans right here in Australia just to cover their CEO's compensation alone.

Coke's global CEO got over $29 million in compensation in 2011.

If Coke really cares about the cost of living for Australian families they simply need to cut their CEO's remunerations to make a difference. Not only have I now signed several petitions against Coke's actions, I now am boycotting Coke's products where possible.

I let Coke know of my intentions by writing a message on their face book wall. If you care about the environment I strongly encourage you all to join me in these actions.

Online petitions need to show that not they only publicize wrongs, but that they also lead to behavioral changes by those that sign them.

Changes that will directly affect the targets of the petition.

If we don't give online petitions this added power, they are going to loose their impact on the world.

The only way we are going to change the world is if we change ourselves first.

A simple click is not enough.

Some thoughts now that I'm an Australian.

Australia.

An island, ringed with golden beaches, and girt by beautiful blue seas. To the north east the great barrier reef, verdant with colour and life.

A continent, the desert in the centre flat and ochre, Uluru, warn smooth and rounded by the hands of time punching through the dusty plains.

And more than just this natural beauty, a land full of iconic imagery.

Possessor of bountiful mineral resources. And wealth, one of the best performing economies in the world during these troubled times.

Australia, truly the lucky country!

Not nearly as lucky, is the country that I grew up in, South Africa.

Also blessed with golden beaches, iconic imagery and bountiful mineral resources, South Africa seems to have continually made choices that brought nothing but misfortune.

I have so clearly etched in my mind, memories of searchlights, shining, ironically, from an old english fort on the apex of a hill, their probing beams of light slicing through the night, sweeping slowly across the face of the township on the other side of the valley to our family home.

Every night the search lights used to dance.

If we took out field glasses and looked at where the oval disks of their light fell, we could sometimes see the security police, heavily armoured, moving slowly from door to door, looking for people. It was rumoured that the people that they were looking for were destined for summary execution - without trial.

But we didn't know.

Information was tightly controlled. All we knew was that from time to time people seen as enemies of the apartheid regime just disappeared.

Some people, unable to withstand the uncertainty of being seen as an enemy of the regime, fearful that they would be the next to vanish, would flee. Their trip to escape doubly dangerous, for if they were caught their attempt to flee was obviously an admission of guilt.

Those that succeeded became refugees, living in countries far away from their homes, families and friends.

But this country, Australia, I've learnt, was founded not just on the sweat of criminals, but also by the sweat of refugees.

I've learned about Australia's roots because I, as one of the scatterlings of Africa, was swotting up for my Australian citizenship exam.

We don't know why the Aboriginal came here, but they may very well have been the first refugees. 

Following the first fleet of convicts, there have regular waves of refugees. The first recorded being Lutherans, arriving in 1839, fleeing from religious prosecution in Prussia. Between 1933 and 1939 more than 7 000 Jewish Germans found sanctuary in Australia. After the second world war, the largest wave of refugees ever to come to Australia. Over 170 000 people, mainly from central Europe: countries like the Ukraine, Yugoslavia, Czechoslovakia and Poland.

Poland.

My father was a Polish refugee. That's how we landed up in South Africa.

He was a captain in the Polish army at the outbreak of the Second World War. Captured by the Russians, he somehow lost all sign of his rank.Which was really lucky, as the Russians took quite a lot of the Polish officers they captured to a forrest; Katyn, and shot them.

But my wily father not only managed to avoid the massacre by loosing his rank: he contrived to escape and made his way through occupied Europe, to England, where he joined, and fought for, the Free Polish Army.

At the end of the war, the British prevented him from marching in the victory celebrations and mandated that he was to be sent back to Poland.

Desperate not to return to a country under Russian control, he ran from embassy to embassy in London, looking for any country that would accept him as a refugee. The South African embassy franked his passport on the spot, and he was on the first ship he could find to Africa, a Polish refugee.

Had he been accepted on the spot by Australia, I would have been born here, and not in Africa. But I've made up for that now: I passed my citizenship test. I've got my letter from the minister, and attended the citizenship ceremony. I'm now a new minted a citizen of Australia!

I'm not the only person who wants to make this wonderful country their home.

I regularly hear terrible stories of Australian bound refugees on boats, drowning, because the overladen and unseaworthy rust buckets transporting them have foundered in the dangerous waters to the north.

I don't know why those “boat people” want to come to Australia.

Are they trying to come to Australia in order to build a better life for themselves and their children? If so, the risks they take are enormous.

Are they perhaps, like the enemies of apartheid, fleeing before the rough justice of their homeland state can be brought to bear?

Or is it because, like my father, they fear almost certain death for simply being who they are?

Whatever force is driving these refugees, it must be incredibly powerful.

Simply look at the risks they are taking with their families and themselves in order to try to get to a different place far on the other side of the world.

When I see these refugees being herded into detention centres, reminiscent of concentration camps, with children in tow, I feel deeply troubled.

For I learned during my citizenship studies that we Australians (I think I can say this now), we Australians hold a shared set of values.

• Peacefulness
• Respect for the dignity of the individual
• Freedom of speech and association
• Equality before the law
• Equality of opportunity

And that we share a strong spirit of egalitarianism that embraces tolerance, mutual respect, and a compassion for those in need. And that no one should be disadvantaged on the basis of their country of birth.

I've thought about these shared values that we Australians hold dear. And compared them to the values that were in play in the country that I grew up in. And to the countries that my father both fought and fled from in the second world war. And even to those of the countries that the boat people currently are fleeing from.

And I'm sure that its these shared Australian values that make the difference. That make this the place where we all want to be.

Knowing that, the way we currently treat refugees seems somehow un-Australian to me.

We Australians are legally obliged to vote.

So in this coming election I'm not going to vote emotionally. I'm not going to vote for the party I identify most with. I'm going to my research and vote for the party that most espouses our shared Australian values.

For I believe these shared values are what have truly made this the lucky country.

“For those who’ve come across the seas

We’ve boundless plains to share;

With courage let us all combine

To Advance Australia Fair”

Spatialite and Me

I had the misfortune to need to connect to a Spatialite enhanced SQLite database, and found myself in some sort of titanic struggle. I first tried using JDBC on OSX and got roundly defeated. For round two I tried with Python on Ubuntu.

For what it's worth (and because this should be documented somewhere, for those who follow) I believe that the following are the steps required to get Spatialite up and running in python on Ubuntu 13.04 (Raring Ringtail) and Ubuntu 12.04. I say 'I believe' because I haven't worked with this in any anger yet :(

Thoughts

If you bother to read through the following then you'll see that my problems were largely caused by the PyPi version of pyspatialite. I think that this software is fundamentally broken in several ways:
it assumes a layout and content of the Spatialite site it references, it does not notify the user clearly when those assumptions are broken, and it caches files and does not remove them if an error was encountered.

The net effect is that every time the Spatialite site is changed (what, no stable url's?) the PyPi download is likely to break and will need repair. And innocent users such as myself will waste a lot of time trying to work out what went wrong.

So if installing on Ubuntu simply use:

sudo apt-get install python-pyspatialite

The downside is that this package is only supported on Quantal and Raring Ringtail.

If you need some sort of automated deployment then clearly you are going to do a lot of work if you want to deploy to anything other than Quantal and Ringtail...

Ubuntu 13.04

I started out on Ubuntu 13.04 and decided that I would try to do some sort of self contained install so that I could make use of virtualenv.

Do not do what I did: 
pip install pyspatialite
For if you do you will be met with:
download_file = pattern.findall(download_page)[0]
IndexError: list index out of range

This is because it would appear that the installer is trying to fetch a file from the Internet: a file that does not exist. From my search on the Internet, this seems to be a regular occurrence with the different versions this software, and may be fixed only to reoccur on the next release :(

If you have the misfortune of making this mistake be aware that the installer has already downloaded files, and it will not try to get replacement copies without manual intervention. More on this in the section on installing on Ubuntu 12.04 below. For I think with a little bit of hacking you could fix this.

Be warned: if you get this error and then attempt to install an older version, you will be met with:
error: conflicting types for ‘spatialite_init’

Do not try to clear ~/.pip/cache – because that's not where the files are being cached :( A pip un-install will not work either, as the software was never installed :(

Having run into the above I simply switched to a new VM, discarding my 'broken' one, and in the new VM I ran:
pip install pyspatialite==2.6.2-spatialite.2.4.0-4

But don't you do this yet: first you need to run:
apt-get install python-dev

Or you will get the following error:
fatal error: Python.h: No such file or directory

Wait: don't run that initial install yet! You also need to run:
apt-get install libgoes-dev

Otherwise you will get:
error: geos_c.h: No such file or directory

And be still that twitching finger: Next you need to run:
apt-get install libproj-dev

Otherwise you will get:
error: proj_api.h: No such file or directory

Then still hold off: for next you need to do:
apt-get install libsqlite3-dev

For if you don't you will be met by a:
fatal error: sqlite3.h: No such file or directory

Then (yes, there's more) you need to
cd /usr/lib
sudo ln -s libgeos-3.3.3.so libgeos.so

For if you don't, you will be met by the somewhat mysterious:
/usr/bin/ld: cannot find -lgeos

This last one is happening because the libgeos-3.3.3 package installed with libgeos-dev forgets to create create this needed symlink (!) It used to, I know, because the Internet tells me so.

Then, finally, don't forget to do a:
apt-get install libspatialite-dev

Otherwise you might be met with a:
fatal error: spatialite.h: No such file or directory

Finally, finally you can run:
pip install pyspatialite==2.6.2-spatialite.2.4.0-4

Don't forget to use sudo if not running as root in all of the above…

Having got to this stage, you can then open up a python shell and try to connect to the database. I say 'try' because you are not yet done!

You need to:
gedit /usr/local/lib/python2.7/dis-packages/pyspatialite/dbapi2.py
And change line 50 to be:
version_info = tuple([int(x) if is instance(x, (int, long)) else x for x in version.split(".")])

This is because the version_info contains strings as well as integers, and the original, as written, thus dies at this point:
version_info = tuple([int(x) for x in version.split(".")])

This last step fills me with dread, as who knows how this string is actually used? How might this step bite later on? However, after all of the above, I could now open the two Sqlite databases I have with Spatialite data in python and execute queries against them.

As you can gather, I made each of the mistakes above, and had to resolve where I had gone wrong: a time consuming task. And one that has had me muttering dark words and wondering nasty thoughts about Spatialite.

The alternate process is to simply accept machine wide installation and to do the following:


sudo apt-get install python-pyspatialite

Which gave me my python bindings on more up to date versions of the software without any of the pain I went through doing the pip install.

However, having done all of this I discovered that I actually needed to run Spatialite version 3 on Ubuntu 12.04. So, getting the whole shebang set up on 12.04 became my next task.

Ubuntu 12.04

On 12.04, having been trained by my previous experiences, I simply ran:
sudo apt-get install python-dev
sudo apt-get install libgeos-dev
sudo apt-get install libproj-dev

Then I downloaded pyspatialite 3.0.1 from:
https://pypi.python.org/pypi/pyspatialite/3.0.1
and unpacked it into a directory off of my home directory.

Then I moved to this directory and ran
sudo python setup install
Which crashed and burned with the infamous error message:
download_file = pattern.findall(download_page)[0]
IndexError: list index out of range

I went to the line causing the error (line 97)
and replaced the "pattern.findall(download_page)[0]" with "libspatialite-amalgamation-3.0.1.zip" (the version of the file I wanted to install) I also edited the url in the following line to read: "http://www.gaia-gis.it/gaia-sins/libspatialite-sources/", because as it stood it was not a correct url.
I reran
sudo python setup install
and got the dreaded
error: conflicting types for ‘spatialite_init’
I simply deleted the amalgamation subdirectory into which the installer seems to download stuff into and reran:
sudo python setup install

And this time, it all seemed to work. Sweetness and light, and the trilling of angels ensued. More importantly, I was able to open and access the databases I wanted to work with.

It really shouldn't be this hard, should it? Also knowing all of this, I suspect I could get the pip install that I tried on 13.04 working. But life is too short.