Going paperless means, "Email me a PDF", not "Email me your URL"

I love going paperless.  When it's done properly, anyway.

My tiny little Austin, Texas lawn service does it correctly!  Ashley emails me once a month, thanking me for my business, and attaches a PDF invoice.  And I send them an electronic check.

While Wells Fargo, the largest bank in the world, does not...  Wells Fargo sends me an email, politely informing me that it is now my responsibility to log into their customer service portal, click through 3 links and then download my latest statement.  I'm sorry, but that's crap.

The positive, "green" environmental impact, saving some paper, is lovely of course.  What I really appreciate is having my own digital archive of information, from bills to legal documents, especially when coupled with something like Google Docs, that I can use to search, anywhere, any time.

That said, I find it completely unacceptable when accounts offer/suggest/insist that you "go paperless" with them, but rather than emailing you an HTML or PDF snapshot, they instead send you a link to log into their website and view your paperless account information.

This is, frankly, a very insidious form of vendor lock-in that cloud computing has invited into our daily lives, and most of us haven't even noticed it yet.

The burden has shifted from your account managers pushing (mailing) information to you, and instead it now resides on you to pull that information from their archive regularly and archive it on your end.

Before...

After...

Do you see the difference?

What happens when you close that account?  What happens when that entity goes out of business?  Or gets acquired?  Merges with a different entity?   Automatically expires some information as too-old-to-be-archived-any-longer?  Rolls out some completely unnecessary changes to their website interface which requires Internet Explorer? Or deprecates the archive functionality entirely?

Or worse, what happens when unethical business practices affect your personal account information and your data gets modified under you?  It's pretty trivial for a sysadmin or rogue process to update some records in a database and comprehensively regenerate millions of reports...

Many people trusted Bernie Madoff with their money...

That's right -- you may no longer have accurate access to your own account information, if you didn't download, snapshot, and back it up yourself when it was originally published.

I learned this lesson the hard way.  I recently rolled over my 401(k) from a previous employer's plan manager.  And when I did so, they terminated my online account access.  I certainly understand why, as I no longer had any funds under management.  The consequence, though, is that I immediately lost access to several years of digital statements associated with my account.  Poof.



I'm sorry, but, "Yes, I want to go paperless" means, "Email me a PDF" not, "Email me a link to your damn website once a month".

p.s. And while you're implementing that, how about sending me something machine readable, in addition to a PDF?  Oh, and please sign and encrypt that email.  I know, I know...  Techmology is hard.  I'm asking way too much...  Arg.

Dustin

Ubuntu ARM Servers -- At last!!!

for Ubuntu Servers

I've long had a personal interest in the energy efficiency of the Ubuntu Server.  This interest has manifested in several ways.  From founding the PowerNap Project to using tiny netbooks and notebooks as servers, I'm just fascinated with the idea of making computing more energy efficient.

It wasn't so long ago, in December 2008 at UDS-Jaunty in Mountain View that I proposed just such a concept, and was nearly ridiculed out of the room.  (Surely no admin in his right mind would want enterprise infrastructure running on ARM processors!!! ...  Um, well, yeah, I do, actually....)  Just a little over two years ago, in July 2009, I longed for the day when Ubuntu ARM Servers might actually be a reality...

My friends, that day is here at last!  Ubuntu ARM Servers are now quite real!


The affable Chris Kenyon first introduced the world to Canonical's efforts in this space with his blog post, Ubuntu Server for ARM Processors a little over a week ago.  El Reg picked up the story quickly, and broke the news in Canonical ARMs Ubuntu for microserver wars.   And ZDNet wasn't far behind, running an article this week, Ubuntu Linux bets on the ARM server.  So the cat is now officially out of the bag -- Ubuntu Servers on ARM are here :-)

Looking for one?  This Geek.com article covers David Mandala's 42-core ARM cluster, based around TI Panda boards.  I also recently came across the ZT Systems R1801e Server, boasting 8 x Dual ARM Cortex A9 processors.  The most exciting part is that this is just the tip of the iceberg.  We've partnered with companies like Calxeda (here in Austin) and others to ensure that the ARM port of the Ubuntu Server will be the most attractive OS option for quite some time.

A huge round of kudos goes to the team of outstanding engineers at Canonical (and elsewhere) doing this work.  I'm sure I'm leaving off a ton of people (feel free to leave comments about who I've missed), but the work that's been most visible to me has been by:

• Michael Casadevall, Oliver Grawert, and David Mandala (leading and manning the ARM/Server/Toolchain bits)
• Serge Hallyn (hacking on LXC as a container solution on the virt-less platform)
• Chuck Short and Dave Walker (OpenStack, libvirt, qemu ARM support)
• Ben Howard (Cloud images for Ubuntu ARM)

So I'm looking forward to reducing my servers' energy footprint...are you?

:-Dustin

PowerNap Your Data Center! (LinuxCon 2011 Vancouver)

I was honored to speak at LinuxCon North America in beautiful Vancouver yesterday, about one of my favorite topics -- energy efficiency opportunities using Ubuntu Servers in the data center (something I've blogged before).

I'm pleased to share those slides with you today!  The talk is entitled PowerNap Your Data Center, and focused on Ubuntu's innovative PowerNap suite, from the system administrator's or data center manager's perspective.

We discussed the original, Cloud motivations for PowerNap, its evolution from the basic process monitoring and suspend/hibernate methods of PowerNap1, to our complete rewrite of PowerNap2 (thanks, Andres!) which added nearly a dozen monitors and the ubiquitously useful PowerSave mode.  PowerNap is now more useful and configurable than ever!

Flip through the presentation below, or download the PDF here.





Stay tuned for another PowerNap presentation I'm giving at Linux Plumbers next month in California.  That one should be a bit deeper dive into the technical implementation, and hopefully generate some plumbing layer discussion and improvement suggestions.

:-Dustin

A Statistical Analysis of Potential PowerNap Energy Savings

I was asked recently how much energy savings administrators might actually expect from an Ubuntu Enterprise Cloud, powered by Eucalyptus and using PowerNap.

I have blogged before about how much I enjoy mathematics and statistical analysis, and this is in fact another exciting, hard question!

The question was presented as the following hypothetical...

Given 10 quad-core servers in a cloud running 40 virtual machines. 20 virtual machines are simultaneously discarded. What power savings would you expect from PowerNap?

This is actually a rather complex combinatorics problem. I'll restate it as such:

Given 10 buckets, each of which can hold 0 - 4 items, how many ways can 20 items be distributed among those 10 buckets? Furthermore, what is the statistical distribution of empty buckets?

It's been 10 years since my university statistics classes, and my brain melted trying to derive the permutation formula. But a big thanks to my good friend Matt Dobson for solving that! If you're interested, you can view his combinatorics formula here.

It took me about an hour to hack a Python script that could empirically solve this problem by brute force, generating a comprehensive list of all of the permutations. You can now download and run /usr/bin/powernap_calculator provided by the powernap package in Karmic.

Here are the results running on the parameters above:

$ time powernap_calculator --hosts 10 --guests-per-host 4 --guests 20
Calculating...99%

In a cloud with [10] hosts, which can handle [4] guests-per-host, currently running [20] guests,
you may expect the following:

[ 5.2%] likely that [0/10] of your hosts would powernap, for a [0%] power savings
[ 27.5%] likely that [1/10] of your hosts would powernap, for a [10%] power savings
[ 42.5%] likely that [2/10] of your hosts would powernap, for a [20%] power savings
[ 21.8%] likely that [3/10] of your hosts would powernap, for a [30%] power savings
[ 2.9%] likely that [4/10] of your hosts would powernap, for a [40%] power savings
[ <1%] likely that [5/10] of your hosts would powernap, for a [50%] power savings

The overall expected value is [19.0%] power savings.

real 0m46.919s
user 0m46.227s
sys 0m0.276s

So at this snapshot in time, when your cloud suddenly dropped to 50% utilization, the expected value is a 19% power savings. Note that expected value is a very specific statistics term. Wikipedia says:

In probability theory and statistics, the expected value (or expectation value, or mathematical expectation, or mean, or first moment) of a random variable is the integral of the random variable with respect to its probability measure. For discrete random variables this is equivalent to the probability-weighted sum of the possible values, and for continuous random variables with a density function it is the probability density-weighted integral of the possible values.

Right, so we might expect about a 19% power savings for this random moment in time, when we simultaneously reduce our utilization by 50%.

However, if we observe the cloud over time, and with perhaps more realistic usage patterns, the distribution should be much better than random. VMs will come and go at more staggered intervals than simultaneous destruction of half the instances.

And you will have another tremendous factor working in your favor -- Eucalyptus' greedy scheduling algorithm. This ensures that each time a new VM is launched, it will land on a system that's already running. This is known as an annealing system -- one that's constantly correcting itself -- since under-utilized systems will automatically powernap, and new VMs will fill in the gaps on running hardware. Pretty cool, I think.

So I'm curious...

• What does (or would) your cloud look like?
• How many -h|--hosts, -p|--guests-per-host, and -g|--guests do you usually have?
• What does powernap_calculator say about your parameters?
• Post your results!
  

Note that the powernap_calculator algorithm is exponential, O((p+1)^h), so large values of (p, h) will take a very long time to compute! I'm totally open to code review of powernap_calculator and any performance enhancements ;-)

In my cloud, I have 8 dual-core hosts. I typically limit my usage to 2 guests-per-host. And I rarely run more than 4 VMs at a time.

$ powernap_calculator -h 8 -p 2 -g 4
In a cloud with [8] hosts, which can handle [2] guests-per-host, currently running [4] guests,
you may expect the following power savings:
[ 26.3%] likely that [4/8] of your hosts would powernap, for a [50%] power savings
[ 63.2%] likely that [5/8] of your hosts would powernap, for a [62%] power savings
[ 10.5%] likely that [6/8] of your hosts would powernap, for a [75%] power savings
The overall expected value is [60.5%] power savings.

My servers run at about 80 Watts fully loaded. My electricity is about $0.10 per kilowatt-hour. So a year's worth of electricity with all 8 servers running all of the time is 8 * .08 KW * 24 hr/day * 365 days/year * $0.10/KW-hr = $560/year.

I like the prospects of saving approximately 60.5% of that, or $339/year!

So how does the calculator work?

Let's use a slightly smaller example: hosts,h=4, guests-per-host,p=2, guests,g=3

Since each system can hold 0, 1, or 2 virtual machines, we're going to use a base-3 number (which is p+1). And we're going to generate all possible 4-digit base-3 numbers (which is (p+1)^h). In our case, that's 3⁴ or 81 scenarios to consider. For each of those scenarios, we convert the decimal integer to a list of each of the digits 0-2, and sum the list, throwing out any "invalid scenarios", where the sum does not add up to the target number of guests, 3, g. Our valid scenarios is actually much smaller, the following 16:

[2, 1, 0, 0]
[1, 2, 0, 0]
[2, 0, 1, 0]
[1, 1, 1, 0]
[0, 2, 1, 0]
[1, 0, 2, 0]
[0, 1, 2, 0]
[2, 0, 0, 1]
[1, 1, 0, 1]
[0, 2, 0, 1]
[1, 0, 1, 1]
[0, 1, 1, 1]
[0, 0, 2, 1]
[1, 0, 0, 2]
[0, 1, 0, 2]
[0, 0, 1, 2]

Now, of these, we are interested in how many 0's are in each row, as this indicates a host that has no guests, and can therefore be powernapped.

Our calculations yield: [0, 4, 12, 0, 0], or:

• 0 possible scenarios have no unused hosts
• 4 possible scenarios have 1 unused hosts
• 12 possible scenarios have 2 unused hosts
• 0 possible scenarios have 3 unused hosts
• 0 possible scenarios have 4 unused hosts

From this we can generate the following probabilities:

[ 25.0%] likely that [1/4] of your hosts would powernap, for a [25%] power savings
[ 75.0%] likely that [2/4] of your hosts would powernap, for a [50%] power savings

And weighting these probabilities, we can generate the expected value:

25%*.25 + 50%*.75 = 43.75%
The overall expected value is [43.8%] power savings.

Comments appreciated!

:-Dustin

PowerNap in the Podcasts

Howdy all-

If you don't already follow the Ubuntu Podcast and the Ubuntu UK Podcast, you really should! Both are community driven multimedia presentations that cover the news in the Linux and Ubuntu communities. These guys provide a tremendous service to the world of Ubuntu.

Nick Ali and Josh Chase discuss PowerNap in the Ubuntu Podcast Episode #31. They did a good job explaining the design, uses, and advantages of it. Give it a listen...

And the Ubuntu UK Podcast guys (Ciemon Dunville, Alan Pope, Tony Whitmore, Dave Walker) interviewed your humbled author in S02E09, The Dimensions of Time. We talked about the Ubuntu Server, PowerNap, KVM, and eCryptfs.

Enjoy!

:-Dustin

Slashdot (Power Capping) and What Ubuntu is Doing

I just read this article, referenced by Slashdot: Power Capping the Data Center.

And I thought I'd borrow this video (one of my favorites) from IBM:



I think that we in the Ubuntu Server Community are leading the charge in developing an energy-efficient server class Linux distribution.

It is my hope that by the 10.04 LTS release, the Ubuntu Server is widely recognized as the de facto Green Computing Server Platform.

I would like to highlight a few of the key ways we have improved the energy efficiency of the Ubuntu Server in 9.04.

9.04

• suspend/hibernate/resume
  
  • In Jaunty, we added pm-utils to the Server seed, and proved that Ubuntu Servers could suspend and hibernate, and be resumed remotely (requires support in the system's BIOS).
    
• cpu frequency scaling
  
  • We also enabled cpu frequency scaling on the Ubuntu Server, defaulting to the on-demand governor. This ensures that Ubuntu Servers run at minimum cpu frequency and power consumption when under-utilized.
    
• pwrkap
  
  • pwrkap is an open source project from the IBM Linux Technology Center which provides an energy use monitor and power capping enforcement tools (just what the Slashdot article is discussing!). We partnered with the LTC, working with Darrick Wong, to deliver this technology as a package in Ubuntu Universe.
    
• powerman
  
  • powerman is another new package in Jaunty. Here, we worked with Arnaud Quette of Eaton to provide in Ubuntu a tool for manipulating Power Distribution Units (PDUs) from a central location--useful for remote operation in data centers and cluster computing.
    
• Eucalyptus, EC2
  
  • Finally, we embarked on a tremendous effort to make the Ubuntu Server a better host and client in virtual and cloud computing environments. Cloud, grid, and utility computing, in a generic sense, provide far more scalable resources at the data center level. And being able to move computing efforts between your data center and someone else's (such as Amazon) also provides some interesting options on the power savings front.
    

These efforts continue in the development of Karmic 9.10.

9.10 (in progress)

• powernap
  
  • PowerNap is a new free software project from Canonical that acts as sort of a "screen saver" for servers. Ubuntu Servers running PowerNap can be configured to take a specific action (like suspending, or hibernating, or powering off) when deemed inactive (no keyboard mouse activity, and none of some list of processes running).
    
• powerwake
  
  • PowerWake is part of the PowerNap project, and provides a command line interface for remotely resuming power napping systems.
    
• Ubuntu Enterprise Cloud powered by Eucalyptus
  
  • Eucalyptus has been enhanced to leverage PowerNap and PowerWake, to maintain a far more energy efficient cloud. Eucalyptus uses PowerNap to suspend, hibernate, or power-off nodes that are not currently running any virtual machines. New VM requests are served from the available capacity of running systems. Eucalyptus will PowerWake sleeping systems only if load requires. With PowerNap, data centers will finally realize the energy savings promised by cloud computing.
    
• Condor
  
  • We are working on packaging Condor for Karmic. Condor is a system that "scavenges" otherwise unused computing cycles, leveraging them for a higher purpose. Think "protein folding" or "SETI@home", except in your data center, for your grid-capable applications. As such, Condor is less about saving power, but more about increasing utilization and efficiency or your computing resources. You could perhaps choose to PowerNap your under utilized hardware and save energy, or instead Condor your systems and task them to other work.
    
• mpich2
  
  • We have also synchronized the mpich2 package from Debian, thanks to some excellent work from a few Debian developers. mpich2 is an extremely important library for high performance, grid computing applications. Whereas Condor is intended for general purpose grid computing, mpich2 is used by developers and users of very specific applications. Like Condor, mpich2 is also about using available computing resources as efficiently as possible.
    

So what's next? I certainly hope to continue working on energy efficiency in the Ubuntu Server. I have a few ideas about what we could do in 10.04.

10.04 (future)

• Low Power Architectures
  
  • I have blogged a couple of times now (here, and here), about running the Ubuntu Server on Dell Mini's. These systems have Intel Atom processors, and run the lpia architecture. I would like to see us work more on this, and perhaps partner with the vendors on an Ubuntu Server product for these architectures.
  • Beyond that, ARM is a fascinating architecture, and will have a tremendous effect on the way we think about computing power. ARM based servers, with solid-state disks will soon run on fractions of a watt of power. Some people are excited about laptops that might have 24 hours of battery. I'm excited about Servers that have a 24 hour built-in battery backup, consume 1% of the power of their predecessors, and can fit in nooks and crannies in every room of your house.
    

:-Dustin

Introducing PowerNap!

Ladies and gentlemen, I'm pleased to announce an exciting, new open source project from Canonical for the Ubuntu Server -- PowerNap!

Mark's Karmic Koala announcement alluded to this work when he wrote:

A savvy Koala knows that the best way to conserve energy is to go
to sleep, and these days even servers can suspend and resume, so imagine
if we could make it possible to build a cloud computing facility that
drops its energy use virtually to zero by napping in the midday heat,
and waking up when there's work to be done. No need to drink at the
energy fountain when there's nothing going on. If we get all of this
right, our Koala will help take the edge off the bear market.

I have just uploaded PowerNap to Karmic, and we are well on our way to integrating the technology into the 9.10's Ubuntu Enterprise Cloud.

Actually, I spent last week in sunny Santa Barbara, California working with Dan Nurmi, of Eucalyptus Systems. We shot some amateur digital videos of Ubuntu Karmic Servers, PowerNap/PowerWake, Eucalyptus, and a Watt meter in action. I'll get those posted soon!

I'll go into much deeper technical detail on the design and implementation of PowerNap over the next few weeks in subsequent posts, but I'll give an overview here...

How Does It Work?

PowerNap operates sort of like a screen saver for servers. Besides watching the console and terminals for keyboard activity, it also watches the system's process table for activity.

An administrator defines a list of regular expressions describing some critical MONITORED_PROCESSES that should be running. When powernapd notices that all of the MONITORED_PROCESSES have been absent from the process table for some configurable ABSENT_SECONDS, powernapd emits a warning to all users of the system that it will run powernap, unless canceled within the next GRACE_SECONDS.

Sample Configuration

In the Ubuntu Enterprise Cloud case, the configuration file, /etc/powernap/config, might look something like:

MONITORED_PROCESSES = [ "^/usr/bin/kvm " ]
ABSENT_SECONDS = 300
GRACE_SECONDS = 60

Thus, if no instance of kvm runs for 5 minutes, then the system will emit a warning, and powernap after a 1 minute grace period.

PowerNap Now!

Alternatively, a system administrator can force the system to powernap immediately by either running /usr/sbin/powernap directly, or sending powernapd the "now" signal with: service powernap now. In fact, this is what the Eucalyptus Node Controller does, such that it can maintain state, and directly control its managed nodes.

What constitutes a powernap?

So, powernap will first check if /etc/powernap/action is executable, and if so, it will run that file. This will allow you, as an administrator, to run any arbitrary script or program of your design when powernapd determines that your server has become idle. Your script could send an email, for example.

echo "Inefficient server, wasting energy" | mail Al_Gore@example.com

But in the default case, powernap will check if your server supports suspend-to-ram, and if so, it will pm-suspend your system. Otherwise, it will suspend-to-disk, or power the system off, depending on the sleep states supported by your hardware.

Slick, huh? :-D

Beyond the Cloud

While PowerNap is bespoke for the Ubuntu Enterprise Cloud, I have implemented it in manner that I hope is generically useful.

In fact, I'm currently using PowerNap on my Mythbuntu front ends! My configuration currently looks something like:

MONITORED_PROCESSES = [ "mplayer", "vlc", "xine", "mythfrontend.real", "xmms", "gthumb" ]
ABSENT_SECONDS = 240
GRACE_SECONDS = 60

My hardware supports S3 suspend-to-ram, so this is great! If 4 minutes go by, where I'm not running any of my media players (mplayer, vlc, xine, mythfrontend.real, xmms), I'm given a 1 minute grace period, and then my system suspends. I have configured wake-on-usb and wake-on-lan in the BIOS, so I can resume the system in a couple of seconds either by tapping a key or sending a WoL magic packet.

But in the mean time, I've reduced the power consumption of 4 systems by 90%, for most of every day while I'm not directly using MythTV!

What about Waking Systems?

Which brings me to PowerNap's kid brother...PowerWake. /usr/bin/powerwake is another Python script. This script is designed to be a smarter, remote waking utility. Currently, it supports wake-on-lan, but it will eventually support other mechanisms, such as IPMI, and perhaps NUT.

With respect to wake-on-lan, it's "smarter" than some other wake-on-lan utilities because it uses a hierarchy of cache files, configuration files, and the current arp table, such that you can wake a system by MAC address, or IP address, or hostname. I find this far more convenient than trying to remember or look up MAC addresses. powerwake respects static configuration in /etc/ethers and maintains a dynamically learning cache of known MAC addresses in /var/cache/powerwake/ethers.

Interesting?

I'm eager to hear what other uses you might have for PowerNap and/or PowerWake for your data centers, basements, and living rooms!

Saving a few Watts,
:-Dustin