ThankHN: A Thank-You Note to the HackerNews Community, from Ubuntu

A huge THANK YOU to the entire HackerNews community, from the Ubuntu community!  Holy smokes...wow...you are an amazing bunch!  Your feedback in the thread, "Ask HN: What do you want to see in Ubuntu 17.10?" is almost unbelievable!

We're truly humbled by your response.

I penned this thread, somewhat on a whim, from the Terminal 2 lounge at London Heathrow last Friday morning before flying home to Austin, Texas.  I clicked "submit", closed my laptop, and boarded an 11-hour flight, wondering if I'd be apologizing to my boss and colleagues later in the day, for such a cowboy approach to Product Management...

When finally I signed onto the in-flight WiFi some 2 hours later, I saw this post at the coveted top position of HackerNews page 1, with a whopping 181 comments (1.5 comments per minute) in the first two hours.  Impressively, it was only 6am on the US west coast by that point, so SFO/PDX/SEA weren't even awake yet.  I was blown away!

This thread is now among the most discussed thread ever in the history of HackerNews, with some 1115 comments and counting at the time of this blog post.

 2530 comments   3125 points     2016-06-24      UK votes to leave EU    dmmalam
 2215 comments   1817 points     2016-11-09      Donald Trump is the president-elect of the U.S. introvertmac
 1448 comments   1330 points     2016-05-31      Moving Forward on Basic Income  dwaxe
 1322 comments   1280 points     2016-10-18      Shame on Y Combinator   MattBearman
 1215 comments   1905 points     2015-06-26      Same-Sex Marriage Is a Right, Supreme Court Rules       imd23
 1214 comments   1630 points     2016-12-05      Tell HN: Political Detox Week – No politics on HN for one week  dang
 1121 comments   1876 points     2016-01-27      Request For Research: Basic Income      mattkrisiloff
*1115 comments   1333 points     2017-03-31      Ask HN: What do you want to see in Ubuntu 17.10?        dustinkirkland
 1090 comments   1493 points     2016-10-20      All Tesla Cars Being Produced Now Have Full Self-Driving Hardware       impish19
 1088 comments   2699 points     2017-03-07      CIA malware and hacking tools   randomname2
 1058 comments   1188 points     2014-03-16      Julie Ann Horvath Describes Sexism and Intimidation Behind Her GitHub Exit      dkasper
 1055 comments   2589 points     2017-02-28      Ask HN: Is S3 down?     iamdeedubs
 1046 comments   2123 points     2016-09-27      Making Humans a Multiplanetary Species [video]  tilt
 1030 comments   1558 points     2017-01-31      Welcome, ACLU   katm
 1013 comments   4107 points     2017-02-19      Reflecting on one very, very strange year at Uber       grey-area
 1008 comments   1990 points     2014-04-10      Drop Dropbox    PhilipA

Rest assured that I have read every single one, and many of my colleagues has followed closely along as well.

In fact, to read and process this thread, I first attempted to print it out -- but cancelled the job before it fully buffered, when I realized that it's 105 pages long!  Here's the PDF (1.6MB), if you're curious, or want to page through it on your e-reader.

So instead, I wrote the following Python script, using the HackerNews REST API, to download the thread from Google Firebase into a JSON document, and import into MongoDB, for item-by-item processing.  Actually, this script will work against any HackerNews thread, and it recursively grabs nested comments.  Next time you're asked to write a recursive function on a white board for a Google interview, hopefully you've remember this code!  :-)

$ cat ~/bin/hackernews.py
#!/usr/bin/python3

import json
import requests
import sys

#https://hacker-news.firebaseio.com/v0/item/14002821.json?print=pretty

def get_json_from_url(item):
        url = "https://hacker-news.firebaseio.com/v0/item/%s.json" % item
        data = json.loads(requests.get(url=url).text)
        #print(json.dumps(data, indent=4, sort_keys=True))
        if "kids" in data and len(data["kids"]) > 0:
                for k in data["kids"]:
                        data[k] = json.loads(get_json_from_url(k))
        return json.dumps(data)


data = json.loads(get_json_from_url(sys.argv[1]))
print(json.dumps(data, indent=4, sort_keys=False))

It takes 5+ minutes to run, so you can just download a snapshot of the JSON blob from here (768KB), or if you prefer to run it yourself...

$ hackernews.py 14002821 | tee 14002821.json

First some raw statistics...

• 1109 total comments
• 713 unique users contributed a comment
• 211 users contributed more than 1 comment
• 42 comments/replies contributed by dustinkirkland (that's me)
• 12 by vetinari
• 11 by JdeBP
• 9 by simosx and jnw2
• 438 top level comments
• 671 nested/replies
• 415 properly formatted uses of "Headline:"
• Thank you!  That was super useful in my processing of these!
• 519 mentions of Desktop
• 174 mentions of Server
• 69 + 64 mentions of Snaps and Core

I'll try to summarize a few of my key interpretations of the trends, having now processed the entire discussion.  Sincere apologies in advance if I've (a) misinterpreted a theme, (b) skipped your favorite them, or (c) conflated concepts.  If any of these are the case, well, please post your feedback in the HackerNews thread associated with this post :-)

First, grouped below are some of the Desktop themes, with some fuzzy, approximate "weighting" by the number of pertinent discussions/mentions/vehemence.

• Drop MIR/Unity for Wayland/Gnome (351 weight) [Beta available, 17.10]
• Release/GA Unity 8 (15 weight)
• Easily, the most heavily requested, major change in this thread was for Ubuntu to drop MIR/Unity in favor of Wayland/Gnome.  And that's exactly what Mark Shuttleworth announced in an Ubuntu Insights post here today.  There were a healthy handful of Unity 8 fans, calling for its GA, and more than a few HackerNews comments lamenting the end of Unity in this thread.
• Improve HiDPI, 4K, display scaling, multi-monitor (217 weight) [Beta available, 17.10]
• For the first time in a long time, I feel like a laggard in the technology space!  I own a dozen or so digital displays but not a single 4K or HiDPI monitor.  So while I can't yet directly relate, the HackerNews community is keen to see better support for multiple, high resolution monitors and world class display scaling.  And I suspect you're just a short year or so ahead of much of the rest of the world.
• Make track pad, touch gestures great (129 weight) [Beta available, 17.10]
• There's certainly an opportunity to improve the track pad and touch gestures in the Ubuntu Desktop "more Apple-like".
• Improve Bluetooth, WiFi, Wireless, Network Manager (97 weight) [Beta available, 17.10]
• This item captures some broad, general requests to make Bluetooth and Wireless more reliable in Ubuntu.  It's a little tough to capture an exact work item, but the relevant teams at Canonical have received the feedback.
• Better mouse settings, more options, scroll acceleration (89 weight) [Beta available, 17.10]
• Similar to the touch/track pad request, there was a collection of similar feedback suggesting better mouse settings out-of-the-box, and more fine grained options. 
• Better NVIDIA, GPU support (87 weight) [In-progress, 17.10]
• NVIDIA GPUs are extensively used in both Ubuntu Desktops and Servers, and the feedback here was largely around better driver availability, more reliable upgrades, CUDA package access.  For my part, I'm personally engaged with the high end GPU team at NVIDIA and we're actively working on a couple of initiatives to improve GPU support in Ubuntu (both Desktop and Server).
• Clean up Network Manager, easier VPN (71 weight) [Beta available, 17.10]
• There were several requests around both Network Manager, and a couple of excellent suggestions with respect to easier VPN configuration and connection.  Given the recent legislation in the USA, I for one am fully supportive of helping Ubuntu users do more than ever before to protect their security and privacy, and that may entail better VPN support.
• Easily customize, relocate the Unity launcher (53 weight) [Deprecated, 17.10]
• This thread made it abundantly clear that it's important to people to be able to move, hide, resize, and customize their launcher (Unity or Gnome).  I can certainly relate, as I personally prefer my launcher at the bottom of the screen.
• Add night mode, redshift, f.lux (42 weight)  [Beta available, 17.10]
• This request is one of the real gems of this whole exercise!  This seems like a nice, little, bite-sized feature, that we may be able include with minimal additional effort.  Great find.
• Make WINE and Windows apps work better (10 weight)
• If Microsoft can make Ubuntu on Windows work so well, why can't Canonical make Windows on Ubuntu work?  :-)  If it were only so easy...  For starters, the Windows Subsystem for Linux "simply" needs to implement a bunch of Linux syscalls, whose source is entirely available.  So there's that :-)  Anyway, this one is really going too be a tough one for us to move the needle on...
• Better accessibility for disabled users, children (9 weight)
• As a parent, and as a friend of many Ubuntu users with special needs, this is definitely a worthy cause.  We'll continue to try and push the envelop on accessibility in the Linux desktop.
• LDAP/ActiveDirectory integration out of the box (7 weight)
• This is actually a regular request of Canonical's corporate Ubuntu Desktop customers.  We're generally able to meet the needs of our enterprise customers around LDAP and ActiveDirectory authentication.  We'll look at what else we can do natively in the distro to improve this.
• Add support for voice commands (5 weight)
• Excellent suggestion.  We've grown so accustomed to "Okay Google...", "Alexa...", "Siri..."  How long until we can, "Hey you, Ubuntu..."  :-)

Grouped below are some themes, requests, and suggestions that generally apply to Ubuntu as an OS, or specifically as a cloud or server OS.

• Better, easier, safer, faster, rolling upgrades (153 weight)
• The ability to upgrade from one release of Ubuntu to the next has long been one of our most important features.  A variety of requests have identified a few ways that we should endeavor to improve: snapshots and rollbacks, A/B image based updates, delta diffs, easier with fewer questions, super safe rolling updates to new releases.  Several readers suggested killing off the non-LTS releases of Ubuntu and only releasing once a year, or every 2 years (which is the LTS status quo).  We're working on a number of these, with much of that effort focused on Ubuntu Core.  You'll see some major advances around this by Ubuntu 18.04 LTS.
• Official hardware that just-works, Nexus-of-Ubuntu (130 weight)
• This is perhaps my personal favorite suggestion of this entire thread -- for us to declare a "Nexus-of-each-Ubuntu-release", much like Google does for each major Android release.  Hypothetically, this would be an easily accessible, available, affordable hardware platform, perhaps designed in conjunction with an OEM, to work perfectly with Ubuntu out of the box.  That's a new concept.  We do have the Ubuntu Hardware Certification Programme, where we clearly list all hardware that's tested and known to work well with Ubuntu.  And we do work with major manufacturers on some fantastic desktops and laptops -- the Dell XPS and System76 both immediately come to mind.  But this suggestion is a step beyond that.  I'm set to speak to a few trusted partners about this idea in the coming weeks.
• Lighter, smaller, more minimal (113 weight) [Beta Available, 17.10]
• Add x-y-z-favorite-package to default install (105 weight)
• For every Ubuntu user that wants to remove stuff from Ubuntu, to make it smaller/faster/lighter/secure, I'll show you another user who wants to add something else to the default install :-)  This is a tricky one, and one that I'm always keen to keep an eye on.  We try very had to strike a delicate balance between minimal-but-usable.  When we have to err, we tend (usually, but not always) on the side of usability.  That's just the Ubuntu way.  That said, we're always evaluating our Ubuntu Server, Cloud, Container, and Docker images to insure that we minimize (or at least justify) any bloat.  We'll certainly take another hard look at the default package sets at both 17.10 and 18.04.  Thanks for bringing this up and we'll absolutely keep it in mind!
• More QA, testing, stability, general polish (99 weight) [In-progress, 17.10]
• The word "polish" is used a total of 24 times, with readers generally asking for more QA, more testing, more stability, and more "polish" to the Ubuntu experience.  This is a tough one to quantify.  That said, we have a strong commitment to quality, and CI/CD (continuous integration, continuous development) testing at Canonical.  As your Product Manager, I'll do my part to ensure that we invest more resources into Ubuntu quality.
• Fix /boot space, clean up old kernels (92 weight) [In-progress, 17.10]
• Ouch.  This is such an ugly, nasty problem.  It personally pissed me off so much, in 2010, that I created a script, "purge-old-kernels".  And it personally pissed me off again so much in 2014, that I jammed it into the Byobu package (which I also authored and maintain), for the sole reason to get it into Ubuntu.  That being said, that's the wrong approach.  I've spoken with Leann Ogasawara, the amazing manager and team lead for the Ubuntu kernel team, and she's committed to getting this problem solved once and for all in Ubuntu 17.10 -- and ideally getting those fixes backported to older releases of Ubuntu.
• ZFS supported as a root filesystem (84 weight)
• This was one of the more surprising requests I found here, and another real gem.  I know that we have quite a few ZFS fans in the Ubuntu community (of which, I'm certainly one) -- but I had no idea so many people want to see ZFS as a root filesystem option.  It makes sense to me -- integrity checking, compression, copy-on-write snapshots, clones.  In fact, we have some skunkworks engineering investigating the possibility.  Stay tuned...
• Improve power management, battery usage (73 weight)
• Longer batteries for laptops, lower energy bills for servers -- an important request.  We'll need to work closer with our hardware OEM/ODM partners to ensure that we're leveraging their latest and greatest energy conservation features, and work with upstream to ensure those changes are integrated into the Linux kernel and Gnome.
• Security hardening, grsecurity (72 weight)
• More security!  There were several requests for "extra security hardening" as an option, and the grsecurity kernel patch set.  So the grsecurity Linux kernel is a heavily modified, patched Linux kernel that adds a ton of additional security checks and features at the lowest level of the OS.  But the patch set is huge -- and it's not upstream in the Linux kernel.  It also only applies against the last LTS release of Ubuntu.  It would be difficult, though not necessarily impossible, to offer the grsecurity supported in the Ubuntu archive.  As for "extra security hardening", Canonical is working with IBM on a number of security certification initiatives, around FIPS, CIS Benchmarks, and DISA STIG documentation.  You'll see these becoming available throughout 2017.
• Dump Systemd (69 weight)
• Fun.  All the people fighting for Wayland/Gnome, and here's a vocal minority pitching a variety of other init systems besides Systemd :-)  So frankly, there's not much we can do about this one at this point.  We created, and developed, and maintained Upstart over the course of a decade -- but for various reasons, Red Hat, SUSE, Debian, and most of the rest of the Linux community chose Systemd.  We fought the good fight, but ultimately, we lost graciously, and migrated Ubuntu to Systemd.
• Root disk encryption, ext4 encryption, more crypto (47 weight) [In-progress, 17.10]
• The very first feature of Ubuntu, that I created when I started working for Canonical in 2008, was the Home Directory Encryption feature introduced in late 2008, so yes -- this feature has been near and dear to my heart!  But as one of the co-authors and co-maintainers of eCryptfs, we're putting our support behind EXT4 Encryption for the future of per-file encryption in Ubuntu.  Our good friends at Google (hi Mike, Ted, and co!) have created something super modern, efficient, and secure with EXT4 Encryption, and we hope to get there in Ubuntu over the next two releases.  Root disk encryption is still important, even more now than ever before, and I do hope we can do a bit better to make root disk encryption easier to enable in the Desktop installer.
• Fix suspend/resume (24 weight)
• These were a somewhat general set of bugs or issues around suspend/resume not working as well as it should.  If these are a closely grouped set of corner cases (e.g. multiple displays, particular hardware), then we should be able to shake these out with better QA, bug triage, and upstream fixes.  That said, I remember when suspend/resume never worked at all in Linux, so pardon me while I'm a little nostalgic about how far we've come :-)  Okay...now, yes, you're right.  We should do better.
• New server installer (19 weight) [Beta available, 17.10]
• Well aren't you in for a surprise :-)  There's a new server installer coming soon!  Stay tuned.
• Improve swap space management (12 weight)
• Another pet peeve of mine -- I feel you!  So I filed this blueprint in 2009, and I'm delighted to say that as of this month (8 years later), Ubuntu 17.04 (Zesty Zapus) will use swap files, rather than swap partitions, by default.  Now, there's a bit more to do -- we should make these a bit more dynamic, tune the swappiness sysctl, etc.  But this is a huge step in the right direction!
• Reproducible builds (7 weight)
• Ensuring that builds are reproducible is essential for the security and the integrity of our distribution.  We've been working with Debian upstream on this over the last few years, and will continue to do so.

Ladies and gentlemen, again, a most sincere "thank you", from the Ubuntu community to the HackerNews community.  We value openness -- open source code, open design, open feedback -- and this last week has been a real celebration of that openness for us.  We appreciate the work and effort you put into your comments, and we hope to continue our dialog throughout our future together, and most importantly, that Ubuntu continues to serve your needs and occasionally even exceed your expectations ;-)

Cheers,
:-Dustin