From the Canyon Edge -- :-Dustin
Showing posts with label python. Show all posts
Showing posts with label python. Show all posts

Monday, January 26, 2015

Introducing PetName libraries for Golang, Python, and Shell

Gratuitous picture of my pets, the day after we rescued them
The PetName libraries (Shell, Python, Golang) can generate infinite combinations of human readable UUIDs


Some Background

In March 2014, when I first started looking after MAAS as a product manager, I raised a minor feature request in Bug #1287224, noting that the random, 5-character hostnames that MAAS generates are not ideal. You can't read them or pronounce them or remember them easily. I'm talking about hostnames like: sldna, xwknd, hwrdz or wkrpb. From that perspective, they're not very friendly. Certainly not very Ubuntu.

We're not alone, in that respect. Amazon generates forgettable instance names like i-15a4417c, along with most virtual machine and container systems.


Meanwhile, there is a reasonably well-known concept -- Zooko's Triangle -- which says that names should be:
  • Human-meaningful: The quality of meaningfulness and memorability to the users of the naming system. Domain names and nicknaming are naming systems that are highly memorable
  • Decentralized: The lack of a centralized authority for determining the meaning of a name. Instead, measures such as a Web of trust are used.
  • Secure: The quality that there is one, unique and specific entity to which the name maps. For instance, domain names are unique because there is just one party able to prove that they are the owner of each domain name.
And, of course we know what XKCD has to say on a somewhat similar matter :-)

So I proposed a few different ways of automatically generating those names, modeled mostly after Ubuntu's beloved own code naming scheme -- Adjective Animal. To get the number of combinations high enough to model any reasonable MAAS user, though, we used Adjective Noun instead of Adjective Animal.

I collected a Adjective list and a Noun list from a blog run by moms, in the interest of having a nice, soft, friendly, non-offensive source of words.

For the most part, the feature served its purpose. We now get memorable, pronounceable names. However, we get a few odd balls in there from time to time. Most are humorous. But some combinations would prove, in fact, to be inappropriate, or perhaps even offensive to some people.

Accepting that, I started thinking about other solutions.

In the mean time, I realized that Docker had recently launched something similar, their NamesGenerator, which pairs an Adjective with a Famous Scientist's Last Name (except they have explicitly blacklisted boring_wozniak, because "Steve Wozniak is not boring", of course!).


Similarly, Github itself now also "suggests" random repo names.



I liked one part of the Docker approach better -- the use of proper names, rather than random nouns.

On the other hand, their approach is hard-coded into the Docker Golang source itself, and not usable or portable elsewhere, easily.

Moreover, there's only a few dozen Adjectives (57) and Names (76), yielding only about 4K combinations (4332) -- which is not nearly enough for MAAS's purposes, where we're shooting for 16M+, with minimal collisions (ie, covering a Class A network).

Introducing the PetName Libraries

I decided to scrap the Nouns list, and instead build a Names list. I started with Last Names (like Docker), but instead focused on First Names, and built a list of about 6,000 names from public census data.  I also built a new list of nearly 38,000 Adjectives.

The combination actually works pretty well! While smelly-Susan isn't particularly charming, it's certainly not an ad hominem attack targeted at any particular Susan! That 6,000 x 38,000 gives us well over 228 million unique combinations!

Moreover, I also thought about how I could actually make it infinitely extensible... The simple rules of English allow Adjectives to modify Nouns, while Adverbs can recursively modify other Adverbs or Adjectives.   How convenient!

So I built a word list of Adverbs (13,000) as well, and added support for specifying the "number" of words in a PetName.
  1. If you want 1, you get a random Name 
  2. If you want 2, you get a random Adjective followed by a Name 
  3. If you want 3 or more, you get N-2 Adverbs, an Adjective and a Name 
Oh, and the separator is now optional, and can be any character or string, with a default of a hyphen, "-".

In fact:
  • 2 words will generate over 221 million unique combinations, over 227 combinations
  • 3 words will generate over 2.8 trillion unique combinations, over 241 combinations (more than 32-bit space)
  • 4 words can generate over 255 combinations
  • 5 words can generate over 268 combinations (more than 64-bit space)
Interestingly, you need 10 words to cover 128-bit space!  So it's

unstoutly-clashingly-assentingly-overimpressibly-nonpermissibly-unfluently-chimerically-frolicly-irrational-wonda

versus

b9643037-4a79-412c-b7fc-80baa7233a31

Shell

So once the algorithm was spec'd out, I built and packaged a simple shell utility and text word lists, called petname, which are published at:
The packages are already in Ubuntu 15.04 (Vivid). On any other version of Ubuntu, you can use the PPA:

$ sudo apt-add-repository ppa:petname/ppa
$ sudo apt-get update

And:
$ sudo apt-get install petname
$ petname
itchy-Marvin
$ petname -w 3
listlessly-easygoing-Radia
$ petname -s ":" -w 5
onwardly:unflinchingly:debonairly:vibrant:Chandler

Python

That's only really useful from the command line, though. In MAAS, we'd want this in a native Python library. So it was really easy to create python-petname, source now published at:
The packages are already in Ubuntu 15.04 (Vivid). On any other version of Ubuntu, you can use the PPA:

$ sudo apt-add-repository ppa:python-petname/ppa
$ sudo apt-get update

And:
$ sudo apt-get install python-petname
$ python-petname
flaky-Megan
$ python-petname -w 4
mercifully-grimly-fruitful-Salma
$ python-petname -s "" -w 2
filthyLaurel

Using it in your own Python code looks as simple as this:

$ python
⟫⟫⟫ import petname
⟫⟫⟫ foo = petname.Generate(3, "_")
⟫⟫⟫ print(foo)
boomingly_tangible_Mikayla

Golang


In the way that NamesGenerator is useful to Docker, I though a Golang library might be useful for us in LXD (and perhaps even usable by Docker or others too), so I created:
Of course you can use "go get" to fetch the Golang package:

$ export GOPATH=$HOME/go
$ mkdir -p $GOPATH
$ export PATH=$PATH:$GOPATH/bin
$ go get github.com/dustinkirkland/golang-petname

And also, the packages are already in Ubuntu 15.04 (Vivid). On any other version of Ubuntu, you can use the PPA:

$ sudo apt-add-repository ppa:golang-petname/ppa
$ sudo apt-get update

And:
$ sudo apt-get install golang-petname
$ golang-petname
quarrelsome-Cullen
$ golang-petname -words=1
Vivian
$ golang-petname -separator="|" -words=10
snobbily|oracularly|contemptuously|discordantly|lachrymosely|afterwards|coquettishly|politely|elaborate|Samir

Using it in your own Golang code looks as simple as this:

package main
import (
        "fmt"
        "math/rand"
        "time"
        "github.com/dustinkirkland/golang-petname"
)
func main() {
        flag.Parse()
        rand.Seed(time.Now().UnixNano())
        fmt.Println(petname.Generate(2, ""))
}
Gratuitous picture of my pets, 7 years later.
Cheers,
happily-hacking-Dustin

Tuesday, February 5, 2013

ssh-import-id now supports Github!

tl;dr

As of ssh-import-id 3.0, you can now import SSH public keys from both Launchpad and Github using lp:$USER and gh:$USER like this:

$ ssh-import-id lp:kirkland gh:cmars
2013-02-05 17:54:15,638 INFO Authorized key ['4096', 'd3:dd:e4:72:25:18:f3:ea:93:10:1a:5b:9f:bc:ef:5e', 'kirkland@x220', '(RSA)']
2013-02-05 17:54:15,647 INFO Authorized key ['2048', '69:57:f9:b6:11:73:48:ae:11:10:b5:18:26:7c:15:9d', 'kirkland@mac', '(RSA)']
2013-02-05 17:54:22,125 INFO Authorized key ['2048', '84:df:01:9f:da:d3:ef:7d:a0:44:17:ff:ab:30:15:22', 'cmars@github/2114943', '(RSA)']
2013-02-05 17:54:22,134 INFO Authorized key ['2048', 'ab:6a:0c:99:09:49:0b:8f:2a:12:e2:f3:3d:c7:a9:79', 'cmars@github/3263683', '(RSA)']
2013-02-05 17:54:22,135 INFO Authorized [4] new SSH keys
This is now available in Ubuntu Raring 13.04, backported to all other supported Ubuntu releases in this PPA, in the upstream source tarballs, and now installable through pip from pypi!

Background

It's been almost 3 years now since I introduced ssh-import-id here on this blog.  I have a Google Alert setup to watch ssh-import-id and I'm delighted to see that it seems to be quite popular and heavily used!

As a brief reintroduction, ssh-import-id is similar to the ssh-copy-id command.  Whereas ssh-copy-id pushes your public key into a remote ~/.ssh/authorized_keys file, ssh-import-id pulls a public key into the local ~/.ssh/authorized_keys.  Especially in cloud instances, it's a great way to securely, easily, and conveniently retrieve and install your own SSH public key, or perhaps that of a friend or colleague.

When I initially wrote it, it was really just a simple shell script wrapper around wget, with some error checking, that would pull public keys over an SSL connection from Launchpad.net.  All of my network friends and colleagues had active, authenticated accounts at Launchpad.net, and everyone had to upload their public GPG keys and public SSH keys to Launchpad in order to get any work done.  This was really easy, since all keys are available as flat text at a very predictable URL pattern: https://launchpad.net/~%s/+sshkeys.

I have always wanted ssh-import-id to be able to pull keys from servers other than Launchpad.  The tool has long supported defining a $URL in your environment or in /etc/ssh/ssh_import_id at the system level.  There just aren't really any other good, authenticated SSH public key servers.

Github

A few days ago, my friend and Gazzang colleague Casey Marshall noticed that Github had actually recently added support to their API which exposes public SSH keys!  This was just awesome :-)  It would take a bit of effort to support, though, as the output format differs between Launchpad (raw text) and Github (JSON).

So this past Saturday on a beautiful evening in Austin, TX (when neither of us should really have been hacking), we both independently initiated our own implementation adding support for Github keys in ssh-import-id :-)  A bit of duplicated effort?  Yeah, oh well...  But we both took a similar approach: let's port this puppy from shell to Python so that we can take advantage of JSON parsing (our alternative was awk!).

Python

My approach was pretty elementary...  I basically implemented a line-by-line, function-by-function port from Shell to Python, since I knew, from a regression standpoint, this would be stable, solid code.  But Casey is undoubtedly the better programmer between the two of us :-)  He took a much more Pythonic approach, implementing each of the protocol handlers as sub commands.

Once we caught up with one another online around midnight Saturday night, we realized that we really duplicating efforts.  So we decided to team up on the problem!  Casey had a much more elegant design, complete with a setup.py and uploadable to pypi.python.org.  Meanwhile, I have maintained the source code and the package in Ubuntu for nearly 3 years and I understood the complex set of legacy compatibility I needed to preserve, as well as several years worth of gotchas and bugs-fixed.  So I took Casey's implementation, whole hog, and went to work on a bunch of little things to get it whipped into shape for upload to Ubuntu.

Portability

Given that Github is now supported in addition to Launchpad, there may actually be some interest in the tool beyond Ubuntu.  Non-Ubuntu users can now install ssh-import-id directly from pypi.python.org!

$ sudo pip install ssh-import-id
Downloading/unpacking ssh-import-id
  Running setup.py egg_info for package ssh-import-id
    
Requirement already satisfied (use --upgrade to upgrade): argparse in /usr/lib/python2.7 (from ssh-import-id)
Downloading/unpacking Requests >=1.1.0 (from ssh-import-id)
  Running setup.py egg_info for package Requests
    
Installing collected packages: ssh-import-id, Requests
  Running setup.py install for ssh-import-id
    
    changing mode of /usr/local/bin/ssh-import-id-lp to 775
    changing mode of /usr/local/bin/ssh-import-id to 775
    changing mode of /usr/local/bin/ssh-import-id-gh to 775
  Running setup.py install for Requests
    
Successfully installed ssh-import-id Requests
Cleaning up...

Other New Features

We've added a few other new features in the 3.x series...
  1. We now detect duplicate keys by their size/fingerprint/type, and avoid adding duplicates
  2. We also now support a -r|--remove option, which you can use to prune keys from ~/.ssh/authorized_keys file that were added by ssh-import-id
Please report bugs as you find them here!  And please use StackExchange for questions!  Enjoy ;-)

:-Dustin

Printfriendly