From the Canyon Edge -- :-Dustin

Monday, February 20, 2012

Thoughts on Hiring Linux Hackers (in 2012)

I have interviewed hundreds of candidates and had the delight of hiring dozens of Linux and open source developers, engineers, and interns over the last 10 years -- at IBM, Canonical, and now Gazzang.  The most recent one signed his contract this morning, in fact!  It's quite a rush to bring new talent into a small team.

Linux jobs are actually hotter now than ever before!  The Wall Street Journal picked this up recently.  And while HostGator has been running giant billboards throughout Austin for at least 2 years now, which plainly asks, "Do you know Linux?  We're hiring!" -- I was impressed to see that they had the same billboard scaled up to 3-stories in height right in Times Square, New York.

Given that my own well being is so deeply invested in being an open source hacker, I selfishly love seeing the Linux and open source job market expanding so vibrantly.

From the interviewer's chair, however, my poking and prodding of a given candidate's Linux skills have changed a bit over those 10 years.  I'm often looking for the candidate's inquisitive nature.  I want to know how interested they really are in going down the rabbit hole.

  • 10 years ago, you had to know how to deploy and run a LAMP stack, and hack your way around Apache, MySQL, PostgreSQL, PHP, Perl, and Python.  You would shriek in horror at bad HTML and CSS and could really make a website sing with a little Javascript.
  • 9 years ago, I wanted to see someone who regularly compiled their own upstream kernel, maybe tweaked a few configuration options on or off just for fun.  Bonus points for each additional software package you compiled from source.  Gentoo users were shoe-ins.
  • 8 years ago, I wanted to talk to people who were sending and receiving PGP or GPG signed, encrypted email.  I was delighted by those who had at least 1024D keys!
  • 7 years ago, I found users who were willing and able to tweak their SELinux policies and AppArmor profiles absolutely intriguing.  If you were running SELinux in enforcing mode on a production system, well, damn, you probably got the job!
  • 6 years ago, I wanted someone who had built their own Beowulf cluster, for fun, over the weekend.  If not Beowulf, then some sort of cluster computing.  Maybe Condor, or MPICH.
  • 5 years ago, I'd structure some conversation around reinstalling dd-wrt or openwrt firmware on routers.  What serious hackers would run stock router firmware?!?
  • 4 years ago, I needed you to have experience with open source virtualization, such as KVMXen, and QEMU.  Oh, and surely you're running MythTV on a few computers around the house, right?
  • 3 years ago, it was all about developers who had Launchpad or Github accounts, had written some open source software and packaged it for Ubuntu or Fedora.  While your friends update one other over Facebook, you're pushing updates over git and bzr.
  • 2 years  ago, I was interested in people who had built or deployed their own cloud infrastructure using Eucalyptus or OpenStack.
  • And last year, it was all about the move from traditional configuration management to cloud-ready service orchestration; experience with Puppet/Chef/Juju were golden.
Nowadays?  Well, it's additive, to an extent.  Hopefully you have the LAMP stack and kernel compilations in your pocket, can send and receive signed/encrypted email.  No real hacker ever runs stock firmware on their router, surely you're using virtual machines and cloud computing on a daily basis, and hopefully you spend as much time on Launchpad/Github/StackExchange as Facebook/Twitter :-)

But you need to be on the cusp of what's next.  I'm hoping you've rooted your phone, jacked your bootloader, and installed a CyanogenMod of your choosing -- at least on your phone at least if not your tablet and e-Reader too!  Hopefully you've tried out this big data business and threw together a map-reduce Hadoop job or two, just for grins.  Clearly you'll have a strong, informed opinions on Unity vs. Gnome3, upstart vs. systemd, and the UEFI secure boot mess.

Oh, and big bonus points if you read my blog.  But you knew that already.  If you read my blog, you've seen this.  And this is what we'll talk about in our interview :-)



  1. aaa,, probably 2 years ago and 4 years ago, you were looking for the same thing.experience with KVM, Xen, and QEMU :)

  2. Erm - need a tea lady? ;o)

  3. Great piece, I just tweeted it.

    Which reminds me, i have a cool project im working on that I would love to get your thoughts on :)

  4. Great post, added a few of those to my set of interview questions

  5. >Hopefully you have the LAMP stack and kernel compilations in your >pocket,
    I showed my 11yr old how to do compile his own kernel.

    >can send and receive signed/encrypted email.
    This is a problem?
    >No real hacker ever runs stock firmware on their router,
    Done that.
    >surely you're using virtual machines
    My retired dad uses them to play Chessmaster on his KDE desktop.
    >and cloud computing on a daily basis,
    Nope, run my own servers for over 2 decades

    >much time on Launchpad/Github/StackExchange
    No (for both buntu and apple launchpad), yes, yes.

    No. No. Sorry, I meant NO and NO!

    >rooted your phone,
    >jacked your bootloader and installed a CyanogenMod
    Yup and yup

    >Hopefully you've tried out this big data business and threw together >a map-reduce Hadoop job or two
    Tried building a MapReduce job in java.

    >a strong, informed opinions on Unity vs. Gnome3,
    Sucks the big one and just plain sucks.

    >upstart vs. systemd,
    did SysV init system work.

    >the UEFI secure boot mess.
    Who didnt see this coming?

    > Emacs vs vi.
    Emacs because using a free version of vi
    is not a sin but a penance.
    oh wait, you didnt ask that!

    I can also 'code' HTML. Dont laugh, I was
    asked that same question 4 years ago.

    Unfortunately, even with all this experience Linux jobs are few and far between (I have no plans to move for a job with my wife being a teacher, having kids very close to two sets of grandparents,etc).

    I have 5-6 friends who have the same background as I do and only one found a Linux related job. He was willing to move away for it though.

    The jobs might be there but either not close by or low paying
    jobs or worse contract work with no benefits.

    So SAP is more or less where Im going to stay it seems for a while.
    Not thrilled by it, actually pretty much hate it but money is money.
    I used to love the idea of being a "Linux" guy but now Im passed

    It is cool however to see that big ass sign in NYC.

  6. Sticking with Ubuntu 10.04 after 12.04 AND 12.10 is released on your work desktop as business priorities are not primarily focused on the technological edge - and understanding that this makes perfect sense. That can be more impressive then running CyanogenMod on your phone.


Please do not use blog comments for support requests! Blog comments do not scale well to this effect.

Instead, please use Launchpad for Bugs and StackExchange for Questions.