Monday, January 18, 2010

35,000-Core Ubuntu Farm Renders Avatar


I just attended Paul Gunn's talk at LCA2010, entitled:
  • Challenges in Data Centre Growth (or, "You need how many processors to finish the movie???")
Paul is a Systems Administrator at Weta Digital, a Wellywood digital effects studio here in Wellington, New Zealand. Check out some of the feature films that Weta Digital has worked on, and I think you'll recognize a few. District9, Day the Earth Stood Still, Jumper, King Kong, Lord of the Rings, Fantastic Four, Eragon, X-Men, i-Robot. Wow!

It was a great talk, about the type of data center needed to render special effects in today's blockbuster movies. They have a 2 Petabyte disk array, 10gbps networking, and 35,000 cores (4,000+ HP blades) in their data center, and still it takes 48 hours to render some of their graphic sequences.

According to Paul, Ubuntu is at the core of all of this, running on all of the rendering nodes, and 90% of the desktops at Weta Digital. He notes that his farm (he calls it a "render wall") is in fact an Ubuntu Server farm, and not RHEL as he has seen reported in the media.

Here's a couple of articles on Weta Digital's data center and their work on Avatar:

:-Dustin

21 comments:

  1. Wow cool!

    They also did Avatar, which is on route to becoming to most successful movie of all time (just by $ without inflation, but still)

    COOL info. Will the video of the talk be available?

    ReplyDelete
  2. Tom-

    Yes, it was recorded. Stay tuned to the "Media" section of the LCA2010 website.

    :-Dustin

    ReplyDelete
  3. http://www.information-management.com/newsletters/avatar_data_processing-10016774-1.html

    This link is broken..you may want to fix it.... good post.

    ReplyDelete
  4. so the rumours it was all done on a couple of WinXP boxes running After Effects was a myth :)

    the mind boggles at that amount of computing power

    ReplyDelete
  5. Dustin... you da man.

    ReplyDelete
  6. Very cool info, Dustin. Thanks for reporting this.

    ReplyDelete
  7. I want a 35000 core Ubuntu cluster. (My birthday is in 3 weeks if anyone is wondering)

    ReplyDelete
  8. "Oh, sure, let me reproduce that, it's a known bug..."

    I wonder what internal vs. external resources they needed to build that or to get to a confidence level where they had a go to use it.

    ReplyDelete
  9. http://imagechan.com/img/7886/Orange%20Blue%20Contrast/

    ReplyDelete
  10. Try it with a PARDUS 2009.1 Cluster.....;-)

    ReplyDelete
  11. "They have a 2 Petabyte disk array..."

    I'd like some details on that, as I start building a 100 TB clustered storage solution, knowing how the 'big boys' do it may give me some clues of how to proceed.

    ReplyDelete
  12. I wonder how they came up with Ubuntu instead of RHEL. There are a lot more cluster tools on RHEL flavors...

    ReplyDelete
  13. I'm obviously in the absolute minority but I really disliked the Avatar movie. I thought it had no redeeming qualities. But, having a son who is entirely caught up with digital maneuvering and game playing and liked it, all I can say is "to each his own".

    ReplyDelete
  14. Wow, so cooL!

    Thanks for sharing some articles on Weta Digital's data center and their work on Avatar.

    ReplyDelete
  15. Avatar was an awesome film and the special effects where just mind blowing.

    ReplyDelete
  16. This is so cool. Avatar was just an amazing movie. The special effects were so incredible.

    ReplyDelete
  17. I didn't realize how much data storage is needed to film a movie like Avatar...

    2 Petabytes is literally beyond comprehension.

    1,024 gigabytes is 1 terabyte, and 1,024 terabytes is 1 petabyte.

    ReplyDelete
  18. That amount of computing power is mind boggling

    ReplyDelete
  19. super cool... I would have never imagined that they were running on Ubuntu. That is, interesting. data cleansing

    ReplyDelete

Please do not use blog comments for support requests! Blog comments do not scale well to this effect.

Instead, please use Launchpad for Bugs and StackExchange for Questions.
* bugs.launchpad.net
* stackexchange.com

Thanks,
:-Dustin