But his post today, Server Migration From Ubuntu 8.04 To Debian 5.0, makes me wonder..."wait a minute, huh?"
I should state up front that I'm a biased reader. I am employed by Canonical, on the Ubuntu Server Team. My daily job is to make the Ubuntu Server a rock-solid, secure, feature-ful, performant, enterprise-class operating system. So I can't even pretend that I'm not biased.
I do feel it's necessary, though, to challenge Aaron's opinion as stated in this blog entry. I very much believe that Ubuntu 8.04.2 LTS (aka Hardy Heron) is every bit as reliable and secure as the Debian Lenny release.
Aaron writes:
Another nice thing with Debian stable, is it releases when it’s ready. The Debian community has taken some flack for this, with 2-3 years at times between releases. However, Debian stable is the operating system that is high production quality. While most end users tend to run testing or unstable on their desktops or laptops, many prefer stable for their production server.This is what the Ubuntu LTS release is. These are only released every 2 years, as opposed to our non-LTS releases which happen every 6 months. The LTS development cycle is handled differently than other development cycles. Many features are bypassed in the interest of additional testing and bug fixing. Packages contained in an LTS release are supported for security updates for 5+ years; thus, we are more conservative about what is acceptable in an LTS release.
Ubuntu LTS is absolutely a stable and secure server operating system, ideally suited for production purposes. My production website (www.divitup.com) is running Ubuntu LTS 8.04.2 Server and will continue to do so until Ubuntu LTS 10.04 is released.
Aaron goes on to write:
Now, with that said, I personally have never had any problems with my LTS server, either Dapper 6.06 or Hardy 8.04. But do I want to risk it? Should I chance it? While nothing may ever happen that causes critical concern for me with an LTS release, I feel more comfortable putting my trust in Debian stable than I do Ubuntu LTS.Multiple years running Ubuntu LTS releases, and no problems. That's great! Me too.
So why spread Fear, Uncertainty and Doubt (tm) about the Ubuntu LTS Server on the planet.ubuntu.com aggregator, without concrete examples? That seems counterproductive. I don't get it...
I do not mean to attack Aaron personally. He's not the first person to tell me that they choose to run Debian on their server rather than Ubuntu. That's certainly their (your) right, and I'm all for choice.
If you know that there are concrete examples of things that we could do better on the Ubuntu Server, specific areas of improvement, particular cases where the Ubuntu Server did not meet your needs, by all means, we want this information! We're honestly trying to build the best enterprise class Linux server operating system on the market. We need to know from you where there is room for improvement.
:-Dustin
8.04.2, sure, but what about 8.04? Was 8.04-on-release-day as stable as Lenny-on-release-day? Not likely.
ReplyDeleteA lot of Ubuntu's testing doesn't happen til RC, when it's too late. Most of it happens after release.
Maybe for server you guys leave off new features (I wouldn't know), but the desktop team definitely doesn't. Remember the fiasco PulseAudio was on Hardy? It's greatly improved, but it was immature and poorly integrated when it released.
I am in strong disagreement that there is a stability equivalence. Simply put, Debian packages go through the experimental -> unstable -> testing -> stable process.
ReplyDeleteThis is a much greater testing base and time for regressions to appear then the Ubuntu process.
That said I think Hardy LTS is a great release.
Mackenzie-
ReplyDeleteThanks for the desktop example. Can you or anyone point to sample packages on the LTS Server? That were released partially baked?
:-Dustin
Steven-
ReplyDeleteI absolutely agree that there's a perception of far greater stability of Debian released packages.
But Ubuntu is built on top of Debian. We frequently apply fixes from Debian (and feed fixes back to them).
Can you give some examples of 6.06 LTS or 8.04 LTS Server packages where there's a demonstrable lack of stability?
I'll start the list, but I'm having trouble going much beyond the infamous openssh vulnerability...
:-Dustin
I dont have much to throw into the 'Which is more stable' arguement, but for my company, Ubuntu on the server was the best choice because of the support from Canonical and the stricter release cycles. Knowing when it will be released is a plus, even if I don't upgrade straight away.
ReplyDeletehttp://lwn.net/Articles/288726/
ReplyDeleteLike I said though, I love hardy and believe it is very stable, just that the Debian process is superior in terms of the amount of time a package has to stabilize and be tested.
Why do I like debian more then ubuntu? things like this do not happen to main debian mirrors:
ReplyDelete# apt-get install unzip
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
libecj-java defoma
Use 'apt-get autoremove' to remove them.
Suggested packages:
zip
The following NEW packages will be installed:
unzip
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 154kB of archives.
After this operation, 348kB of additional disk space will be used.
0% [Connecting to us.archive.ubuntu.com (91.189.88.45)]
Debian and Testing:
ReplyDeleteWe stop auto-syncing source packages from Debian unstable weeks before Feature Freeze (time to stabilise). Jaunty: DIF: Dec 25th, FF: Feb 19th. Also regarding the testing of various Debian 'releases': I'd say that most people just run unstable. Little testing of experimental or testing.
Mackenzie: I guess it's only fair to wait a few days after release before taking the plunge and upgrade. :-)
"...Can you or anyone point to sample packages on the LTS Server? That were released partially baked?..."
ReplyDeleteXen:, when 8.04 LTS was released a bug in the kernel causes networking to not work at all for virtual machines (see https://bugs.launchpad.net/ubuntu/+source/xen-3.2/+bug/204010 and https://bugs.launchpad.net/ubuntu/+source/linux/+bug/218126). Fortunately Hirano Takahito had patched the kernel and made the patched versions available.
... i know, i know, xen is not supported by Canonical, but is the best (only?) solution if your server doesn't have intel VT or AMD-V extentions
Regards
robbyt:
ReplyDeleteCan you confirm that this happened on an Ubuntu 8.04 LTS Server system?
I'd absolutely understand this happening on an under-development release (Jaunty). And while not necessarily excusable, I could perhaps even see this happening with an 8.10 release...
But the current subject is Ubuntu 8.04 LTS Server, and I'm having trouble understanding how your issue might occur there.
:-Dustin
Lorem-
ReplyDeleteThank you very much for your insightful post. I could see Xen being a problem area, if you're trying to build a stable server off of Ubuntu 8.04 LTS, and Xen is your "killer app".
As you correctly note, Xen is not a main package, and as such, it's not a feature that Canonical supported, actively developed, or even really tested for 8.04.
KVM is the virtualization mechanism that Canonical/Ubuntu have focused on delivering, and unfortunately for some people, this requires a particular CPU feature.
:-Dustin
Case #1: I like Ubuntu on server cause I don't have to disable unsecure SSLv2 on every service I install. I get that by default.
ReplyDeletere: ubuntu mirrors going down last night:
ReplyDeleteyes, it's a 8.04 system at linode. Not being able to get at ubuntu mirrors happens some times- Generally just have to flush the dns cache and try again...
anyhow my point wasn't to be snarky (hope you didn't think i was?) My point was more that Debian has some serious history, and it takes people a while to change perceptions about serious distributions.
Redhat had some big problems 8 years ago with rpm databases getting corrupt. i still meet people who think corrupt rpm dbs are common.
my point is, people run debian because debian has a long history of things "just working right" and people run ubuntu because they read about ubuntu on digg, and they want the latest stuff and they don't want to learn how to manually to install stuff like java and flash.
both are valid reasons, imho.
I think most people aren't that worried about the stability of Ubuntu server main packages vs. Debian. At least I know I'm not. If you view this in terms of Ubuntu main packages vs. Debian, I think you're missing the bigger picture (and you can stop reading now too, because I won't be talking about packages in Ubuntu main).
ReplyDeleteI had read that "Server Migration" post too, and it resonated with me because of a couple of experiences I've had with Ubuntu packages...
For example, I installed the Torque cluster packages at work for 8.04 LTS. And slowly discovered that they're completely broken and unusable. The GUI package is even missing the majority the TCL files that make up said GUI program! (Something that would have been caught by even the most basic of testing at packaging time.)
Installed munge (authentication) and slurm-llnl (another cluster program) later. Munge is broken on install. Slurm breaks after a reboot: the server won't start, and the queue of jobs pending is lost. (You can search for the bug reports if you want to know the gory details for these and Torque bugs too.)
Now I like Ubuntu server, but I can't help hearing this little voice in my head saying that if I had been running Debian on those machines, I wouldn't have had these problems. Say what you want about Debian, but it's unstable -> testing -> stable cycle ensures that packages that are broken (like the ones above) are weeded out before the release. I wasted many times what it would have taken me to install Torque from source trying to make the Ubuntu 8.04 LTS packages work. That's how I read the "Server Migration to Debian 5.0" post, personally.
> Can you give some examples of 6.06 LTS
ReplyDelete> or 8.04 LTS Server packages where
> there's a demonstrable lack of
> stability?
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/210672
Dustin,
ReplyDeleteLet me turn your question on its head.
What is the differentiator between Debian stable and Ubuntu Server edition?
I think part of the problem is that back when Ubuntu was introduced the discussion about how Ubuntu and Debian interacted was grounded in the Desktop.
Ubuntu was positioned to be the fast moving innovative Debian derivative doing things in a different way to be more focused on that desktop user experience.
You go back and you look at some of the attempts to document the Ubuntu/Debian relationship...to make peace..and there's no real discussion about Ubuntu as a server operating system that competes directly with Debian for rock-solid performance. All of that cultural friction discussion happened just before the Ubuntu server edition was introduced.
http://mako.cc/writing/to_fork_or_not_to_fork.html
There's no LTS or server editions mentioned in that lessons learned document.
The fact of the matter is, Ubuntu Server LTS encroaches deeply into traditional Debian usage scenarios in a way that the Ubuntu desktop does not. I don't think people have really gotten their heads around what that means in terms of how Ubuntu and Debian are going to compete for users. And make no mistake, Ubuntu LTS server edition more than anthing else on the market directly competes with Debian stable for users.
And I don't think I've ever seen anyone at Canonical articulate a rationale as to why Ubuntu server LTS is a better investment than an install of Debian stable. Even Canonical's own press materials make Debian's rock-solid performance the first reason to look at Ubuntu server.
http://www.ubuntu.com/products/WhatIsUbuntu/serveredition
The opening sentence on that page is an argument for users to use Debian stable for their servers. Making the Debian heritage your top bullet point... stresses how rock-solid Debian is for servers. You shouldn't be shocked that people find the heritage argument so compelling that they prefer Debian on servers.
The only thing on that page which differentiates Ubuntu server is the Canonical support. You should probably be prepared to see several, smaller independent Debian support companies touting their own support services for Debian 5 which in an effort to take paying customers away from Canonical, making the argument that paying for support of a Debian server system is better than Canonical based support of Ubuntu server.
Or maybe even bigger support companies..like HP.
http://h20219.www2.hp.com/services/cache/442406-0-0-0-121.html
http://h20219.www2.hp.com/services/cache/442407-0-0-0-121.html#fle
You've got a tough road ahead building Ubuntu server brand value in a market segment that already deeply values the Debian brand on servers. The work to get Ubuntu server CAPP/EAL certified is going to be your biggest payoff as it gets you into the doors or entities which can't run Debian stable due to non-technical policy requirements.
-jef
robbyt:
ReplyDeleteRegarding mirrors, I didn't experience that, but, sure, I agree, that sucks. Is this something that happens frequently? I don't really know, because I have a 500GB hard drive as a dedicated local mirror, that sync's daily at 6am.
Still, mirrors going down... Is that to be blamed on the Ubuntu Server? Would Ubuntu Desktops not suffer the same problem? Do Debian mirrors never go down?
Finally, I hope at least some people out there run Ubuntu because we make a great product, and not just because Digg says so...
:-Dustin
I am a Debian-specialised sysadmin (and an Ubuntu dev) in a data-center that has 3000+ servers running mostly Debian, and I keep pushing towards switching to Ubuntu.
ReplyDeleteNow our needs may not be the same as all data-centers. Our developers always want to use the latest technologies, and as a result, I spend 70% of my time backporting packages to Debian. Needless to say, a few days after Etch's release, I had already backported PHP, MySQL and Apache, since the versions were so old. So, for me, Debian stable is either too old, or it's not stable at all (since backporting recent stuff necessarily breaks the so-called stability of Debian). This is one big reason for me to switch our production to Ubuntu.
On top of that, I think it's also important to remember what Debian calls stable. Stable means "no critical bug known on the day of release" (unless I'm mistaken). With Etch, Debian has started to do something nice : they added the critical bugs from stable to the graph on http://bugs.debian.org/release-critical/. So now you can easily see that around the middle of its lifetime (that is, about a year ago), there were more critical bugs known in Etch than in Lenny! Just because all known critical bugs were fixed in Lenny on the 14th of Feb 2009 doesn't mean we won't discover more in the weeks/months that follow. Debian is only stable because we don't know major instabilities when it is released.
Now when comparing Debian and Ubuntu release cycles, I often compare them to an acrobat walking on a thread. Debian would be an acrobat making big steps very carefully, while Ubuntu would run on the thread. While Ubuntu's running steps might be a bit less stable than Debian careful big steps, they will both make it to the other side, and Ubuntu is actually going much faster.
"So why spread Fear, Uncertainty and Doubt (tm) about the Ubuntu LTS Server on the planet.ubuntu.com aggregator, without concrete examples? That seems counterproductive. I don't get it..."
ReplyDeleteHe said he was switching, that he had no problems, but he liked the Debian process better. That's not FUD.
"This is what the Ubuntu LTS release is. These are only released every 2 years, as opposed to our non-LTS releases which happen every 6 months. The LTS development cycle is handled differently than other development cycles."
Uh huh. And how long are you working on those LTS releases? Two years? Are these LTS releases a completely separate process that get as much time dedicated to them as the Debian releases?
My impression is that each version gets about six months. Maybe you learn from one version and incorporate some things into it from previous version, but the LTS is not a two year cycle. Rather it is just one of 4 versions that is given PERHAPS a bit more attention during its six month production.
As Steven Harms said, there is not a stability equivalence if this is the case. And like he said, I thought Hardy LTS was a nice release.
But I have learned enough with the most recent releases, despite not being LTS, to now watch Ubuntu with a wary and suspicious eye. For cryin' out loud, when I installed 8.10, the VERY NEXT DAY there were kernel updates. Those issues could not have been solved BEFORE the ISO burn?
I'll be giving Debian 5.0 a chance now. And if Ubuntu cleans up its act with fiascos like that, then I'll likely stay. But if not and if Debian serves my needs, it's no FUD to state I'll be moving along.
another reason why i choose ubuntu as primary server (same as debian) is Ubuntu LTS have much package and updated packages. for example, a last year ago i run debian server but i doesn't see php5 package include suhosin pacth. then i install ubuntu server i get it!
ReplyDeletefor overall performance, i think Ubuntu and Debian is same. i'm not interest about comparing debian and ubuntu. but i'm very interest if Ubuntu/Debian compared between RedHat/CentOS :D
# dustin
i thinnk ubuntu package must be upgraded again, for example apache. debian have 2.2.9 version but ubuntu (LTS) have 2.2.8 i think it's a good idea if Hardy upgraded apache webserver
and... sorry for my english. :D
@robbyt & chrish
ReplyDeleteI've been Debian user for a decade now and Ubuntu user since first version. I make my living with Ubuntu server. FWIW, I tend to use packages from main. If I need packages from Universe, I love the possibility to easily contribute any fixes trough MOTU program. You will agree that Debian's new maintainer process isn't that easy.
I've heard about problems with US archive. At some point, us.archive.ubuntu.com was even cname for uk.archive.ubuntu.com. Right, those problems exist or at least, have existed. BTW clearing cache indicates problems with DNS, not with an archive ;)
Now, about Debian experimental->unstable->testing->stable mantra. Well, not everything is peachy in Debian (not bashing Debian, problems happen everywhere to everybody). Maybe you guys aren't using Debian that long, but there was (iirc) Debian release with broken sources.list, there was a release that couldn't format ext2 on amd64, etc, etc.
You could also say that lots of packages go trough Debian experimental -> Debian unstable -> Ubuntu preFF -> Ubuntu postFF process.
Bottom line, Ubuntu server is based on Debian, it doesn't replace it. It adds additional value.
Being a software developer in a large bank I have to fight with pure conservatism each day.
ReplyDeleteYes, more testing is always good. However, shipping a system that consists of very old components that are not very well maintained anymore by their original authors offsets the advantage of more testing easily.
Test, but don't get too far behind the latest versions, especially if you rely on a large amount of open source packages.
In my opinion, ubuntu server strikes the right balance, and debian is falling so far behind that all the benefits of extra testing are undone by relying on less well supported packages.
Thanks for the reply. FYI, I was not spreading FUD. I find that a hit below the belt. I merely stated the facts in a clear an concise manner, all of which are fully documented.
ReplyDeleteOne point I thought of brining up in the post, but didn't, is the fact the an LTS release is just another 6-month release that Canonical is supporting for longer than the non-LTS releases. It isn't anything special. It doesn't go through any special type of testing the others don't. It doesn't have more "bug-free" packages than the others don't. It's just another 6-month release.
And, 8.04 wasn't exactly a stellar release like 6.06.
* Firefox Beta was shipped. While I understand Canonical not wanting to support an unsupported Mozilla browser for 3 or 5 years, you won't find beta software shipping in a Debian stable release.
* The integration with PulseAudio was a nightmare. Many have blamed this on the PulseAudio developer, but the truth is not the software itself, but it's horrible integration, which still is less than stellar.
* As already mentioned, there were pretty bad bugs with Xen and networking. While not officially supported by Canonical, KVM is hardly production quality as well. So what options do Ubuntu server users have?
* SELinux is horribly broken on 8.04, and I mean BAD. AppArmor is a mistake that Novell will eventually apologize for, and SELinux is your only savior, that is, if your operating system vendor can get it right. FWIW, it's still broken on 8.10, and I haven't seen any hope in 9.04.
8.04 wasn't a polished release, and this is because it's just another 6 month release like all the others, except Canonical is stamping on extra years of support. If it went through a bug testing process like Debian did from volitile to experimental to unstable to testing to stable, LTS releases would shine and rock hard, but they don't. They pull from unstable before feature freeze, hope the community will test the snot out of it, call it good, and stamp on "LTS". Even Red Hat does a better job with RHEL from Fedora.
Believe me, I want to see LTS succeed, but right now, as my post mentioned, I'm not convinced. It doesn't have the rigorous testing that goes into LTS like Debian does with stable. It's just not on par.
So, if you find fear in my post, then maybe you too should re-evaluate your current server setup. If you find uncertainty, study for yourself, and see that the facts in my post are dead on. If you are in doubt about Debian stable being more stable than Ubuntu LTS, try it for yourself. No, I didn't spread FUD, I just took off the fanboy glasses, evaluated what was important to me, and blogged about it. It was hard- I have solid respect for Ubuntu, I just don't think it's ready for the server platform quite yet.
Hi Aoirthoir-
ReplyDeleteFirst of all, thank you for your post. I appreciate the fact that you gave some concrete examples of why you may move on from Ubuntu to Debian (the 8.10 kernel update, for example).
However, I do consider the particular statement about having never had problems with Ubuntu LTS, but switching to Debian because it "feels better" to be quite clearly spreading "Fear, Uncertainty, Doubt."
"I personally have never had any problems with my LTS server, either Dapper 6.06 or Hardy 8.04. But do I want to risk it? Should I chance it?"
^^^ Fear
"While nothing may ever happen that causes critical concern for me with an LTS release,"
^^^ Uncertainty
"I feel more comfortable putting my trust in Debian stable than I do Ubuntu LTS."
^^^ Doubt
:-Dustin
Dustin,
ReplyDeletethen you are using BROAD definitions of those words.
First, he stated plainly that he had thus far had NO issues with Ubuntu except those he caused himself.
Next, should I risk it? That's not fear. That's a VALID question. It was a PERSONAL question to himself. I run Ubuntu servers in my office, INTERNAL for the staff. I asked myself those questions. I also ask should I risk changing to Debian? This is no fear of debian, it is a logical question of the ISSUES that **MIGHT** arise.
Fear would be stating that I had lost CRITICAL data. Or that he had. He made no such claim.
As far as your uncertainty, well that is the case in EVERY situation. While nothing may EVER happen to my boxes, it MIGHT. We're likely to never be fully certain about anything, thus there is always a DEGREE of uncertainty.
Doubt? THAT is your evidence for doubt? He said he is mor COMFORTABLE putting his trust in Debian stable. Right now I am more comfortable putting my trust in Ubuntu. I AM going to test Jaunty. (And probably stick a needle in my eye if it is as bad as the last two releases...). AND I am ALSO going to test Debian Stable. But even with the issues of the last two Ubuntu releases, I KNOW Ubuntu. So I am presently MORE COMFORTABLE with it. That's not spreading doubt about Debian.
I honestly think you Ubunteros need to relax a LOT. You're always so worried when anyone says anything less than glowing about your business, company, or OS that you cannot see any potential to listen to the criticism as being valid.
Ponce and myself BOTH pointed out that the LTS is really just a six month release. NOTHING at ALL special about it. It goes through NO MORE rigorous examination than the rest of the releases. Indeed if you started working on the next LTS at the release of the last LTS, then one might have MORE confidence in it.
But having LESS confidence is not spreading of FUD.
I'm just going to start making up crazy terms like that because it will enable people to have to THINK LESS, and just react.
"Fear, uncertainty and doubt (FUD) is a tactic of rhetoric and fallacy used in sales, marketing, public relations, politics and propaganda. FUD is generally a strategic attempt to influence public perception by disseminating negative information designed to undermine the credibility of their beliefs. An individual firm, for example, might use FUD to invite unfavorable opinions and speculation about a competitor's product; to increase the general estimation of switching costs among current customers; or to maintain leverage over a current business partner who could potentially become a rival. FUD techniques may be crude and simple. Alternatively they may be very subtle, employing an indirect approach." http://en.wikipedia.org/wiki/Fear,_uncertainty_and_doubt
ReplyDeleteNever have I used FUD in the post. I never undermined the credibility of Ubuntu as a server, I certainly have not spread false information regarding Ubuntu. Lastly, I'm not trying to persuade anyone about using Debian over Ubuntu. I merely blogged my experience in doing so. If my blog isn't a place where I can put my experiences, what is?
No, FUD was not used in that post.
Dustin I really like your post, and I really like Ubuntu. But I have awful problem (lack of stability) with Ubuntu or maybe with my hardware ). It locks up every few days/hours (randomly) without single error. No log, no panic nothing. Temperatures are fine. Actually I have described my problem in details here:
ReplyDeletehttp://ubuntuforums.org/showthread.php?t=1192506
And it seems to me very similar to this:
http://unix.derkeiler.com/Mailing-Lists/FreeBSD/current/2007-09/msg00851.html
Maybe hardware is my problem, but I just want OS to let me found what the problem is. Any help would be very very very much appreciated.
I'm ready to provide any additional information you may need.
Thank you very much
Valentin