Monday, January 18, 2010

35,000-Core Ubuntu Farm Renders Avatar


I just attended Paul Gunn's talk at LCA2010, entitled:
  • Challenges in Data Centre Growth (or, "You need how many processors to finish the movie???")
Paul is a Systems Administrator at Weta Digital, a Wellywood digital effects studio here in Wellington, New Zealand. Check out some of the feature films that Weta Digital has worked on, and I think you'll recognize a few. District9, Day the Earth Stood Still, Jumper, King Kong, Lord of the Rings, Fantastic Four, Eragon, X-Men, i-Robot. Wow!

It was a great talk, about the type of data center needed to render special effects in today's blockbuster movies. They have a 2 Petabyte disk array, 10gbps networking, and 35,000 cores (4,000+ HP blades) in their data center, and still it takes 48 hours to render some of their graphic sequences.

According to Paul, Ubuntu is at the core of all of this, running on all of the rendering nodes, and 90% of the desktops at Weta Digital. He notes that his farm (he calls it a "render wall") is in fact an Ubuntu Server farm, and not RHEL as he has seen reported in the media.

Here's a couple of articles on Weta Digital's data center and their work on Avatar:

:-Dustin
at

21 comments:

  1. TomJanuary 18, 2010 at 5:31 PM

    Wow cool!

    They also did Avatar, which is on route to becoming to most successful movie of all time (just by $ without inflation, but still)

    COOL info. Will the video of the talk be available?

    ReplyDelete
  2. Dustin KirklandJanuary 18, 2010 at 6:45 PM

    Tom-

    Yes, it was recorded. Stay tuned to the "Media" section of the LCA2010 website.

    :-Dustin

    ReplyDelete
  3. ttk153January 18, 2010 at 9:33 PM

    http://www.information-management.com/newsletters/avatar_data_processing-10016774-1.html

    This link is broken..you may want to fix it.... good post.

    ReplyDelete
  4. AnonymousJanuary 18, 2010 at 10:10 PM

    so the rumours it was all done on a couple of WinXP boxes running After Effects was a myth :)

    the mind boggles at that amount of computing power

    ReplyDelete
  5. AnonymousJanuary 19, 2010 at 12:37 AM

    Dustin... you da man.

    ReplyDelete
  6. AnonymousJanuary 19, 2010 at 1:41 AM

    Very cool info, Dustin. Thanks for reporting this.

    ReplyDelete
  7. AnonymousJanuary 19, 2010 at 7:02 AM

    I want a 35000 core Ubuntu cluster. (My birthday is in 3 weeks if anyone is wondering)

    ReplyDelete
  8. AnonymousJanuary 19, 2010 at 8:36 AM

    "Oh, sure, let me reproduce that, it's a known bug..."

    I wonder what internal vs. external resources they needed to build that or to get to a confidence level where they had a go to use it.

    ReplyDelete
  9. AnonymousJanuary 22, 2010 at 5:15 AM

    http://imagechan.com/img/7886/Orange%20Blue%20Contrast/

    ReplyDelete
  10. Erick WesselsJanuary 22, 2010 at 8:39 AM

    Try it with a PARDUS 2009.1 Cluster.....;-)

    ReplyDelete
  11. fak3rMarch 4, 2010 at 3:51 PM

    "They have a 2 Petabyte disk array..."

    I'd like some details on that, as I start building a 100 TB clustered storage solution, knowing how the 'big boys' do it may give me some clues of how to proceed.

    ReplyDelete
  12. Carlos GomezMarch 5, 2010 at 10:59 AM

    I wonder how they came up with Ubuntu instead of RHEL. There are a lot more cluster tools on RHEL flavors...

    ReplyDelete
  13. clusterlösungenMay 9, 2010 at 4:36 PM

    Very good Job!

    ReplyDelete
  14. Alma's Hydroponic SystemAugust 4, 2010 at 10:43 AM

    I'm obviously in the absolute minority but I really disliked the Avatar movie. I thought it had no redeeming qualities. But, having a son who is entirely caught up with digital maneuvering and game playing and liked it, all I can say is "to each his own".

    ReplyDelete
  15. Toshii the Entrepreneur-To-BeAugust 14, 2010 at 4:29 AM

    Wow, so cooL!

    Thanks for sharing some articles on Weta Digital's data center and their work on Avatar.

    ReplyDelete
  16. CHSeptember 13, 2010 at 12:38 PM

    Avatar was an awesome film and the special effects where just mind blowing.

    ReplyDelete
  17. kinect for xbox 360 reviewOctober 17, 2010 at 7:36 PM

    This is so cool. Avatar was just an amazing movie. The special effects were so incredible.

    ReplyDelete
  18. August MenesesOctober 21, 2010 at 12:07 AM

    I have watched it again on my Samsung LN46A650 46-Inch 1080p 120Hz LCD HDTV with Red Touch of Color Review and it looks really awesome!

    ReplyDelete
  19. ShakiraJanuary 26, 2011 at 1:03 AM

    I didn't realize how much data storage is needed to film a movie like Avatar...

    2 Petabytes is literally beyond comprehension.

    1,024 gigabytes is 1 terabyte, and 1,024 terabytes is 1 petabyte.

    ReplyDelete
  20. pizza couponsFebruary 27, 2011 at 9:52 PM

    That amount of computing power is mind boggling

    ReplyDelete
  21. AnonymousAugust 6, 2012 at 3:11 PM

    super cool... I would have never imagined that they were running on Ubuntu. That is, interesting. data cleansing

    ReplyDelete

Please do not use blog comments for support requests! Blog comments do not scale well to this effect.

Instead, please use Launchpad for Bugs and StackExchange for Questions.
* bugs.launchpad.net
* stackexchange.com

Thanks,
:-Dustin