Engine Room makes a splash with Soul Surfer

In 2003, Hawaiian teen surfing star Bethany Hamilton lost her left arm to a shark attack. The story of her return to professional competition is now the subject of Soul Surfer, with actress AnnaSophia Robb playing Hamilton. To create scenes of Robb’s missing arm – around 500 shots – and other effects in the film, director Sean McNamara turned to LA-based visual effects house Engine Room. We talk to the studio’s founder and the film’s vfx supe Dan Schmit.
fxg: Five hundred arm removal shots seems pretty daunting. What were some of the ways you thought you would approach these, at least initially?
Schmit: When we first went into it, the assumption I think from the producers was that it was going to be all digital. It’d be essentially putting the stump on and creating that for every shot. It didn’t take long for us to play around and discover that a really good option was to work with an actual prosthetic in combination with the digital work. So AnnaSophia would often hold her arm behind her back and her shoulder would be pushed back, so we could put a little prosthetic in the front that looked like it was sitting in a normal position.

Before and after images of Engine Room's arm removal VFX

Schmit: Well first we actually did a screen test. We worked with a great make-up and effects guy, Mark Garbarino, to develop the prosthetic. We did a 35mm film shoot with John Leonetti, the DP, and with Sean the director of a body double with a green covered arm running around on the beach and jumping in the water. We brought that back to Engine Room to start working with it and develop our techniques, for (a) getting the green off the background and (b) how do we reconstruct the parts of her body once the stump is taken away? In the end we took those methods into our production.
fxg: So once you got to the shoot, can you describe a typical shoot set-up?
Schmit: The day would begin for AnnaSophia in the make-up chair having the prosthetic applied, which was about an hour and a half process. Mark would be there on set and would essentially cast a new silicone prosthetic for her every day. We had three different looks for the prosthetic because the stump changed through the movie as time goes on. At first, it’s got a bandage on it, then that comes off and the scar is really fresh and the arm swollen. Then for the later scenes in the movie, which are six months down the track, the stump is more toned and tanned. So we had that kind of continuity to keep track of. AnnaSophia wore either a green neoprene sock or we would paint on chromakey waterproof green paint that Mark developed.
fxg: How did you shoot the green arm scenes?
Schmit: We went in with a plan of saying, ‘We’re going to have, say, just three green arm shots in this particular scene,’ but as soon as we got into the movie we realized that the visual effects needs were never going to be the deciding factor in any scene. Whatever was shot was shot and we were going to have to deal with it. We talked about lots of close-ups and lots of lock-offs, but of course every shot was moving and lots of shots showing different angles. For every set-up we would have her perform and either myself or Michael Caplan, our visual effects producer, would be on set in Hawaii doing on-set supervision.
fxg: So what kind of specific challenges did having to remove the green arm and prosthetic bring?
Schmit: Well, the prosthetic was designed for AnnaSophia’s arm to be behind her back, but that was not always practical. We had to decide on a shot by shot basis what made the most sense. We couldn’t have where she was putting her arm inhibit her performance in any way. So often her arm was hanging right there in front of her and we would have to resolve that by reconstructing the whole side of her body.
One of the excellent techniques we developed to be able to do that was shooting an additional plate of the side of AnnaSophia’s body. After the end of every set-up we would have her raise her arm up and just rotate her torso around a little bit while the camera was still rolling, so we could see what her body looked like without her arm covering it. We just needed a few frames of that. We would also shoot a clean background.
fxg: How did split the work up back at Engine Room?
Schmit: The first thing was that we decided to split the work up amongst artists here and around the world. We abandoned the artist-based facility model several years ago at Engine Room. In our early years, we’d have 20 or 30 artists in-house. But we’ve realized it’s better for us to have a very high-end small team in-house and work with a lot of off-site people. So we had six in-house ‘rock star’ generalists, and about 18 off-site who were anywhere from across town to across the world. Most shots were done by a single artist or a very small team, and that lent itself well for being easily divisible.
fxg: Can you take me through how a typical green arm replacement shot was done, say for a surfing scene?
Schmit: We had some nice techniques for the water work. We discovered that if we were moving a green arm over a moving water background, rather than trying to patch water in from another part of the shot or a plate, we found we could do a lot of with stretching. We’d take the water that was next to the green arm and horizontally stretch it to create a clean background and fill in the areas behind the arm. We found that technique worked well for the sand beaches behind her as well. There was just something about the sand texture that meant you couldn’t tell it was being stretched as well.
We had figured that out in testing, so when we were shooting, we asked the water camera teams – some of the best surfing cinematographers in the world – that whenever they were framing a shot that had the surfer’s green arm on, we would ask them to please frame the shot so there would be a little extra space, so there would be something there for us to stretch.
fxg: How did you co-ordinate that work, given that it was going on in so many different places?
Schmit: We came up with an interesting analogue way of unifying the look. Initially we were getting a lot of good shots back, and each looked great, but they were all slightly different in terms of resolving some of the areas that the stump looked like from behind, for instance. We realized we needed to define for everybody what it needed to look like by coming up with a clear reference.
For that we started with a 3D model from a body scan of AnnaSophia – which was taken with her arm behind her back and the prosthetic on – and basically removed her stump and making a clean stump model in 3D in Maya. Then we had a 3D object print made as a polymer cast. The print was of Sofia’s shoulder and the stump and her arm gone, in life size. We then took that piece and gave it to Mark Garbarino and he cast it and made it out of silicone. It was painted in the same way he made it up on the set every day, but now it was complete. It was the stump from all sides.
So now we had this prop, and what we ended up doing was setting up a little photo studio at our place and then for any particular shot where we needed to see what the stump was going to look like from a particular angle, we did a little series of stills on a turntable. We’d light it to look like the lighting of the scene and then shoot a series of stills for the compositor. It could be 50 stills of the stump from all different angles, but as a photographic element. Then a lot of the work was done using those stills – morphing, time re-mapping them altogether into a seamless composite. It worked really well and allowed us to control the look of it and answer questions very quickly.
fxg: What was your toolset for the arm replacement work?
Schmit: All of our compositing and painting work was done in CS4 at Engine Room, but offsite it was a real combination of Flame, Nuke, Fusion. We also did some 3D work in Maya.
fxg: Can you talk about some of the other effects work you had to do for the film? Were there some face replacements for the surfing scenes?
Schmit: Yes, so there were three competition scenes in the movie, so three times you have these scenes where you’ve got six girls in the water, each with their own rashguards, surfboards, and the water unit – all the underwater camera guys with hydroflex camera housings, sound guys in inner tubes. And ADs with walkie talkies in zip-lock bags, floating around on little dingys.
And a lot of the surfing scenes were rewritten in the editing room with the material that had been shot. Part of that meant stealing shots from different competition scenes. The only problem was in say that scene the surfer was going left to right instead of right to left, and the surfboard’s blue instead of red, and the rashguard’s green instead of blue and it’s the wrong girl!
So we found ourselves doing a lot of shots like that where we were painting surfers out of the water or painting people in the water that needed to be there for continuity. It involved changing just about every aspect of the surfers from their bathing suit, to whether they were goofy foot or regular, and their faces. We had the most success using photographic elements we had shot of AnnaSophia against greenscreen, rather than anything from the 3D body scan.
fxg: What would have been the most challenging shot or shots in the film?
Schmit: Some of the arm replacement shots were very difficult, because AnnaSophia’s green arm was facing camera for a long, long time. But there were also some tricky shots that were surfing face replacement shots, and what was particularly interesting with these was that the real Bethany Hamilton did a lot of surfing in the film. Well, Bethany is now 21 or 22 years old and is basically an Olympic athlete, and AnnaSophia is 15.
So early on we were a bit weary of doing a face replacement on Bethany’s body because the scales of their bodies were very different. That was a real concern, but as the movie progressed and all the surfing footage was analyzed it was determined that the surfing that Bethany had done was the best, so it was just assumed we would figure it out! And we able to do some pretty impressive scaling in compositing to make those shots work.

Mr. X’s spy effects for Hanna

Joe Wright’s coming-of-age espionage thriller, Hanna, tells the story of a 16-year-old girl who has grown up in the Finnish wilderness and is now being pursued by the CIA. Mr. X Inc, the film’s sole visual effects vendor, completed around 200 shots for the film, and presented VFX sup. Brendan Taylor with the chance to collaborate closely with the director, production designer, DoP, stunt crew and other filmmakers on key sequences.

Original plate

Final composite

For Mr. X, production on Hanna moved swiftly. “We got the script in January of 2010, I think on a Wednesday,” recalls Taylor. “By Monday I was in Hamburg for a location scout!” At first glance, the studio’s shots seemed to be limited to driving composites, snow enhancements, muzzle flashes and gore enhancements. But after scouting the locations, it became clear to Taylor that Mr. X’s responsibilities were going to be much larger. “Hanna needed to go to all these locations across Europe,” he says, “and it needed to feel like a $100 million movie, but we had to do it at less than a third of the price.”
Production began shooting in March 2010 in northern Finland, an area that posed minus 35 degrees centigrade weather. The location shoot, however, was deemed important by the director, working with production designer Sarah Greenwood and DoP Alwin Kuchler, and Taylor, to show the harsh environment Hanna had been raised to survive in. It would also mean that digital additions of snow would be kept to a minimum. Additional scenes were filmed in Bavaria, Germany using masses of physical fake snow created using paper and water, with only minor digital touch-ups required. The same was essentially true for all of the film’s locations and sets. “It was really important to enhance the cinematographer’s and the production designer’s work,” remarks Taylor. “We hear that a lot from cinematographers and production designers – that when it gets into the visual effects realm, the images take on a whole life of their own. But because they did such a wonderful job, we just wanted to complement what they did.”

“for the reindeer… they are not very bright animals…we had considered going with a Steve Martin appliance”

Brendan Taylor

VFX Sup.

An early scene in Finland involves Hanna utilizing her archery skills to kill a reindeer, a sequence ultimately accomplished with a real animal and digital arrows. “I had never worked with reindeer before,” notes Taylor, “but they are not very bright animals. We had considered going with a ‘Steve Martin appliance’ (a reference to a 1970s Steve Martin arrow-through-the-head gag) but it was minus 35 at the time and there was no point in getting this giant arrow stuck onto the side of a big reindeer, who in the first place wasn’t doing what it was told. So we decided to do the arrow digitally, and concentrate on the performance of the reindeer on location.”

Shooting on a frozen lake. Photo by Brendan Taylor.

Mr. X also created a digital C-130 Hercules plane that flies over the snow-covered trees and Hanna’s head in another Finland scene. The plate for the shot, a low crane looking up at the actress as she follows the plane, was acquired on the first day of shooting. “It was actually a little nerve-wracking,” recalls Taylor. “Joe hasn’t really done that many visual effects films before and we hadn’t previs’d the shot, so we had to figure out how fast the airplane would be going over. Everyone’s nervous on the first day. But I was amazed at how quickly we got into a groove of being able to discuss the particulars of the shot. It’s the kind of collaboration that you really dream of.”
Further visual effects work for scenes in Finland by Mr. X also remained of the seamless variety. “There was a fight scene out on a frozen lake in the middle of nowhere and it needed to have pristine snow,” says Taylor. “But after you shoot the scene a number of times, say five takes, all the snow is trampled. They were shooting it very frenetically and hand-held, but I just said keep shooting and we’ll roto the people and smooth out the snow. In the end it was just done by compositing still photographs onto the snow and by using a variety of clean plates.”

Left to Right: Tim Dean (Libra Head Technician), Jorg Widmer ("A" Camera Operator) and Alwin Kuchler (Director of Photography) discuss the C-130 shot. Photo by Brendan Taylor.

That work, and the majority of location visual effects, was completed without the aid of detailed survey data or tracking markers. “In Finland we were only 60km from the Arctic Circle and another five kilometers to the border of Russia,” says Taylor. “So getting survey equipment up there wasn’t really practical or worth the cost. I debated whether or not to put tracking markers on the ground, but you don’t want to get into a situation where you’ve put all these great tracking markers down and then three quarters of your shots are removing those markers! It just meant we had to track to the shadow or footprint on the snow, and it really brought our shot count down.”
At one point in the film, Hanna is captured and taken to a bunker underneath the Moroccan desert. The underground scenes were filmed at the now disused Berlin Windkanal, a cavernous aerodynamic testing wind tunnel built for German aircraft after the First World War. Hanna escapes her holding cell and is pursued down the hallway before narrowly sliding to freedom under a giant closing steel door.

Original plate

Final shot

“The giant steel door didn’t exist in the wind tunnel, nor did the holding cell and area around it,” says Taylor. “So the art department built that area and we had Hanna run on the practical ground and then we digitally extended the entire tunnel around her. We ended up replacing the tunnel for continuity reasons. As well, we had two full-CG shots, and her POV down the tunnel as the door is closing.”
Chris MacLean modeled and lit the created environment, which was rendered in V-Ray, with compositing by Robert Greb and Greg Astles to match the location based on reference photos taken by Taylor. “Those shots were real monsters and took a long time and finessing to get them done,” he says. “I guess the most difficult thing about them was getting the texture and the material on the wall to feel like concrete. For a while, it just wasn’t reacting correctly and we wanted to match what Sarah had done with her production design and what Alwin had done – he had lit the environment with strobe lights and matching the intensity of those was a challenge.”
For a hotel room scene in which one of the characters, Lewis (John MacMillan), is killed, Taylor relied on some innovative on-set collaboration with the director and stunt coordinators in particular to help with the staging of a long stunt shot. Lewis walks down a large hotel room to talk to intelligence operative Marissa (Cate Blanchett), then hears a knock at the door, turns around, walks to the door, puts his eye up to the peep hole, before being suddenly shot and blown backwards.
The shot was conceived as a one take with the actual actor and not a stunt double, and without the benefit of a face replacement, since MacMillian had to deliver a line of dialogue and walk the length of the room before being violently killed. “Because he gets yanked back so hard,” says Taylor, “we needed to have him on a rig and I was really concerned that he walks all this way down and turns around and then you’ve got this stunt wire crossing in front of his blazer as it moves. I just didn’t want to get into the whole scenario of digitally re-creating his blazer or one of those monster paint outs that takes months.”


Above: Original plate (L) and final shot.
“So what we ended up doing,” continues Taylor, “was having the actor come down to camera and talk to Marissa with this pick point in his back. As he’s talking to Cate Blanchett we had a guy dressed in green crawl along the ground and meet him right where he gets hit, clip him in the back and then he gets yanked almost immediately. It was one of those shots where a little bit of shot design went a long way. Working with Joe and the stunt guys – over dinner actually – we were able to come up with a pretty effective solution.”
To help the assembly process, Mr. X also supplied editor Paul Tothill with an early effects shot to remove the comedic presence of the greenscreened person. “It was a very emotional scene, and Paul couldn’t cut it without cracking up,” laughs Taylor. “We needed to get Paul an effects shot without the extra guy in there, so we basically spent a day doing ‘green man removal’.”

Original container park plate

Final shot

Similar innovation was required for a scene towards the end of the film of a fist fight between Erik (Eric Bana) and Isaacs (Tom Hollander). “They shot the entire fight at 48 frames per second,” explains Taylor, “and I never would have known this, but a lot of the success in fake movie punches relies on shooting at 24 frames. When you start to slow it down, you really see the punches missing. Some of them looked fantastic but others you could feel were just a little bit too far away from the face. So what we ended up doing there was a completely 2D solution.”
Compositor Barb Benoit worked on the Flame to rotoscope the actor being hit and move them a little closer to the person delivering the punches. “We’d start with that, but it still wouldn’t feel like the punch was connecting,” says Taylor. “So in addition, Barb would do a subtle time warp just before the punch was connecting to slow it down. Then as the punch connects she’d speed it up. It doesn’t feel like a music video crazy-slow-then-fast time warp. It actually really makes the punch feel like it’s connecting. Just watching them sometimes in dailies we would go, “Oh, man!”. We only had to do that two or three times, but I was just really impressed with Barb’s wizardry for those shots.”
Mr. X carried out wire removals and set extensions for a scene in which Hanna is pursued by Isaacs’ henchmen atop several shipping containers. “This was beautifully choreographed by Jeff Imada,” says Taylor. “All of the scary as hell looking stuff is Saoirse! For our set extensions, it was really important to feel the damp murkiness of a cold Hamburg night. Matte painter Milan Schere handled that rather obscure direction perfectly.”
Other visual effects work by Mr. X included several matte paintings to help sell the multiple locations Hanna visits during the film. “In the movie, Hanna travels all through Europe but there wasn’t really the budget to do that for real,” says Taylor. “There’s one shot where they’re in a van driving through the desert. We shot the van in Morocco and then flew to Spain and took digital stills and some HD video on a Canon 5D. We created a digital matte painting and composited it into the plate and it looks really wonderful. That was Greg Astles again and Matt Schofield.”
For Taylor, some of the biggest – and most fun – challenges on Hanna included traveling to several countries and then ultimately dealing with 200 varying shots of differing complexities, some as one-offs, and working with an edit that remained in flux until near the end of production. The close interaction with the other creatives on the film, however, made it an incredibly enjoyable experience for the visual effects supervisor.

Above: Original plate (L) and final shot.
“The one thing that was really different about this film,” says Taylor, “was that the whole way through it really felt like a family. People were really looking out for each other and trying to find the best solution for the movie. I credit a lot of that to Joe. He was a real leader – we would get our shot lists in the morning and there would be a little inspirational note or a line from the movie just to keep everyone going. It helps you get through the freezing cold Hamburg night or being up at 3am in the morning at a container park, or the freezing cold Finland day!”

Your Highness: A little bit of Framestore magic

Your Highness is the medieval tale of two princes, brothers Thadeous and Fabious, and their attempt to rescue the later’s future bride, Belladonna, who has been kidnapped by the evil wizard Leezar. Joined by aide Courtney and warrior Isabel, the group encounter wizards, savages and strange creatures on their quest.
Framestore completed around 570 shots for the film, including a five-headed serpent, a flying mechanical bird, matte paintings, set extensions and much magic. In this interview with fxguide, Framestore co-founder and the film’s visual effects supervisor Mike McGee delves into how he collaborated closely with the Your Highness filmmakers on both a creative and technical level, and on unexpected CG solutions.
The VFX News: Can you tell me about how you got involved on the film?
Mike McGee: Danny McBride, who is both a writer on the film and the actor playing Thadeous, and the director, David Gordon Green, came over to Framestore to go through the script, but they said in advance if we had any ideas to throw them into the mix. So I talked through the script and discussed how we could achieve what was already in there, and as we were talking, I was then offering up moments where visual effects could add some real production value and put dollars on the screen.
The VFX News: You then took on the role of the overall visual effects supervisor. What did that involve initially?
McGee: Well, we filmed in Belfast for four months in the Summer of ’09. I went out a month and a half before filming so that I could be there for the whole pre-production process. I became involved in how much of the set we would build, how much would be topped up in CG, the locations where matte paintings could help to provide bigger vistas, and the bigger creature sequences in terms of what we could do practically and digitally.
The VFX News: How did you contribute to say the serpent creature sequence?
McGee: When I was first there, we had sequence where a creature would burst out of another body and it would be half human, half alien. But we decided there was something more we could do to make that scene have more production value. So they asked if I could come up with an idea for an alternative creature. So I sat down with the writer, the director and the production designer and this image popped into my head of a five-headed creature. I thought we could build one CG creature and multiply it five times. As I made the shape with my arm, I rested my elbow on the table and made my hand into the five heads. Everybody said, ‘That’s great – we’ll do a hand serpent!’ Then the writer said we could make it like a video game where the bad guys control the creature by putting his hand into a bowl of sick.
So this big bucket of sick became the game’s controller and this guy punches his fist into this bowl of yellow-looking sick, and as he moves his hand around inside the bucket, this monster bursts out of this arena out of the ground. And then this fight ensues. Our heroes pull out swords and then through various jumps and leaps they slice one by the one the heads off the creature. Each time they slice a head off, the guy loses one of his fingers in the bowl of sick. At the end of the shot, he pulls his hand out of the sick and he’s just left with his bird finger. It was quite bizarre movie-making!
The VFX News: That sounds great that as the VFX supe you were able to have that kind of contribution.
McGee: Actually after the idea of the hand serpent, the guys were like ‘This is great, you’re ideas are great – let’s keep collaborating,’ and I’d be invited to meetings about other parts of the script. It was really the most collaborative film I’ve ever worked on. At Framestore, we now have our own art department that works on projects, but we’ve also worked on films, even if we’re not doing the visual effects on those films. We designed the hand serpent and did all the concepts in-house to show the director what it would look like, but still worked closely with the production designer.
The VFX News: Was that serpent sequence something you also ‘boarded and previs’d?
McGee: We did all our own previs. Before that, the director actually said, ‘Go away, here’s the storyboard artist, you block this out and show it to me when you think you’ve got a good sequence’. So I briefed the storyboard artist, and then the director had a look and made a few changes. Then the storyboard artist went back to Framestore to get everything previs’d and the director then signed off on the previs. When we went to the set, we pretty much shot to the previs. We might change the lens occasionally. It involved a lot of stunts as well, so things changed when we realised what we could do – the stunt people were offering up some amazing new ideas.
The VFX News: How were those serpent scenes shot?
McGee: Before each set up, I would go to the individual actors, whether it was Natalie Portman playing Isabel or James Franco as Fabious, and say, ‘There’s the creature, there’s the guy with a fishing pole and that’s where the eyeline is for head number four – you’ll be chopping that off and then we’ll move to position number two where you’ll be attacked by head number one’. We had a fishing rod for rehearsals but then when we shot the scene they did filmed it with just the actors. If we needed to do stunts, we’d have to rig a jerk wire, then we’d bring in the stunt person. Natalie Portman had a big action sequence and she would run in and then we’d cut to a stunt double on a trampette so she could fire up into the air and do several somersaults before she came down.
fxg: The VFX News:- This is a comedy and it’s set in medieval times, but there are a lot of effects in the film. Did that mean you approached it any differently to other projects?
McGee: Well as you say the fact is that it’s a ‘medieval stoner comedy’ – which was the way we referred to it – with comedy being the key emphasis. If we were on set and something funny happened, then we would go with it. For example, we did a sequence with some magical CG fairies. There was a scripted scene that was two or three pages, and we were shooting this exterior nighttime scene, and they were struggling to make the scene funny, so David had some alternative lines written down and would then would encourage them to ad lib – quite often to the limit of what would be acceptable.
The VFX had to be able to follow that ad lib path. In one of the sequences, the guy caught a fairy that was fluttering around a pond. At first you think they’re fireflies and so he grabs it very gently. And then we go in for a close-up and he starts to pull the wings off this creature and it starts squirming in his hand. And then he rips the head off and then takes the body and crushes it in his hand and snorts it through a pipe. It’s meant to be a magic fairy and so he’s snorting magic dust which he gets high on. The shots changed from the storyboard because the way he acted it out was just very funny on the day, so we went with a different method. Although we weren’t really doing groundbreaking things from a CG point of view, the way we were capturing data and keeping up with the improvisation was one of the most challenging things. You don’t want to stop the flow of the comedy – it’s funniest when it keeps going and going and having the camera keep rolling. You can’t get your camera passes until the scene is finished, so we would normally wait until the end of a roll and then jump in and grab what we could.
The VFX News: What were some of the other major sequences Framestore worked on, the bird Simon for example?
McGee: The director wanted to pay homage to early fantasy sci-fi films and the original Harryhausen Clash of the Titans film. So on set they had an animatronic bird which was something the actors could perform with, and is the bird in many of the shots. But whenever that bird had to take off or fly or carry messages or help out in any of the action fight sequences, it had to be CG.
The VFX News: There’s a lot of magic in the film too – how was that created and what approach did you take to the electricity effect?
McGee: Well there are a lot of wizards that fire magic as a weapon. And the magic comes out in the shape of electricity from a staff or the palms of hands. The director wanted this to be an homage to magic from the past, so he didn’t want the effects to be super-glossy and Harry Potter-style. He wanted it to be a little more man-made, but still look like a modern visual effect. So we came up with this magic that fires from people’s hands and then we have a big action sequence at the end, where the wizard has taken his virgin bride that he has reared from a child, which he is going to impregnate with dragon seed – and then she’ll give birth to this dragon which he will then control and rule the kingdom. He has to impregnate her at the time of twin moons. When they eclipse he is able to impregnate her, so the film is moving to this climax of the twin moons. The ceiling of the observatory opens up to let the moonlight in and the moonlight sends down electricity which puts a forcefield around the bed that he’s strapped this virgin to.
The whole end sequence has magic everywhere coming off people’s hands, the walls, then there’s dust clouds and explosions. There’s lighting coming down though a crystal on the roof, focussing down on the bed that the virgin is lying on. And finally when the bad guy is killed in the climax of the film, his body disintegrates in a shower of embers and all these embers are sucked up by the electricity and a huge explosion happens that sends it up to the twin moons. So there we have a CG castle, with CG set extensions, magic effects which were particle renders heavily treated in 2D.
The VFX News: Were there any other shots in particular you wanted to talk about?
McGee: Well, there’s one sequence in a labyrinth where they have to go to collect a magical sword – a unicorn blade with a unicorn horn and it’s the only sword that can kill the bad guy. When they go into this labyrinth, the sword is protected by a minotaur – half bull and half man. It’s a guy in a suit in hooves, standing upright with a bull’s head. He chases our heroes around the labyrinth trying to protect the sword. At one point he manages to grab one of our heroes and pushes him against a wall and starts to dry hump him against the wall. So you’ve got this very funny sequence of this guy being worked on behind by this minotaur.
Basically, the minotaur in this rubber suit had an eight to ten inch flaccid penis as part of his costume. So whenever you see him running around, he has this penis that’s hanging down. But when he’s dry humping our guy, Natalie Portman runs around the corner and catches him. To distract him, she pulls some Peruvian pipes out of her pocket and plays these pipes and calms the creature down to distract him. As he turns to face us, the client said, ‘Hey, there’s something missing here guys – there’s no excitement on this minotaur!’.
So, we had to build in CG a fully erect minotaur’s penis, make a turntable, texture it and send it back for approval – which was the funny thing. We then had to paint out the existing flaccid penis, body track the erect one on. We did all that and then they said, ‘You know what guys, because Natalie is playing the pipes, that penis needs to animate from full excitement to semi-droop’. So we then got into conversations on how this penis is going to animate, and we then had to put an animator on it to make it go from erect to slightly droopy for the five seconds that shot lasts. So that was probably my most bizarre show and tell in how we made a 3D penis and how shiny it is and the surface scatter lighting on it.
The VFX News: I’m guessing you won’t be talking about that at Siggraph, then.
McGee: [Laughs]. People are always saying, ‘Can you do something we’ve never seen before?’ to visual effects artists. Well, I can honestly say with this movie I have done so many things that I’ll probably never do again!

Shining a Light on Bakery Relight

At last, a lighting tool specifically designed for artists in animation and VFX.

Visual development for the movie Ana. Courtesy of Lo Coloco.

One of the noteworthy launches this week at the 2011 NAB Show in Vegas is Bakery Relight, the industry's first interactive, all-in-one lighting and rendering suite of tools, from the French-based software firm, The Bakery, co-owned by Erwan Maigret and Arnauld Lamorlette (who received a 2011 Technical Achievement Academy Award for co-developing a global illumination solution).
Several years in development, Relight is touted as a technical solution to support the progressive and iterative process of lighters and shaders (a refinement properties and details) with resolution feedback within seconds. The proprietary tool set also enables superior matching of CG illumination with composited live-action elements, which makes it useful for both animation and VFX.
The brainchild of Maigret (a technical lead on Shrek the Third, Shrek 2 and Madagascar) and Lamorlette (head of FX on Shrek the Third and Shrek 2), Relight, solves an important industry need. "Arnauld and I both have artistic and technical backgrounds and understand the artists' needs, and we wanted to build a company that focuses on technology that every studio can use," Maigret suggests. "Relight introduces new artistic possibilities: it solves the lighting and rendering bottleneck found at nearly all studios.
"If you look at Pixar and DreamWorks, they have the time and the money and the technologies to be able to afford to work on the image. That is exactly what we are doing because there is no such thing on the market. Basically, the process is to work on the final image -- to change the color -- just like a painter.

More visual development from Ana. Courtesy of Lo Coloco.

"We talked to artists at studios and found out that interactive lighting was their biggest need because animation is about spending artists' time; rendering is about how much time it's going to take and what you can handle. And we decided to deliver this kind of software to market so that it can be used by many people and cut down on the time in the render farm and let artists do lighting that won't take hours. Our research tells us that companies that use Relight can improve turnaround times by up to 300%."

Templar project, courtesy of Dwarf Labs.

Relight is designed for both stereoscopic 3-D and flat productions and it's customized and optimized for a lighter's workflow. According to Maigret, when a lighter changes a parameter, only the strict minimum is recomputed. The results of complex computations are saved to disk so they need not be recomputed. Thus, the big technological advancement "is about the caching that we do: instead of doing very fast computations, going 100x faster like the GPU, we just do 1% of the computing and reuse it all the time."
That's because of a multithreaded implementation of state-of-the-art rendering algorithms, combined with a unique caching mechanism. Relight is comprised of an interactive lighting application, a rendering engine and a set of auxiliary tools, which include command line for batch processing. Interactive for visualizing the data generated at each step of the rendering process and conversion from third party formats (image, geometry, scene description).
The rendering engine is built around an open node-based architecture for shader definitions. The pool of shaders is extensible by design, so it is easy to add new lights, materials or any other shading operators. The engine architecture then allows for a highly customizable, yet seamless mix of different illumination and shading models in the same scene.
"Imagine you have to light a character with millions of hairs," Maigret continues. "When you need to change the color or shading of the hair, the new image is recomputed interactively. The geometric computations for the hair have already been performed. The same paradigm applies to material shading, global illumination, ray tracing, texturing and shadows. Lighters can choose to either increase their productivity or tweak their lighting to improve quality. Most of the time they can do both."
The development cycle of Relight is organized as a constant back-and-forth between development of new features and testing in production environment, to ensure a perfect balance between usability and performance.

Who's the Cat project, courtesy of Dwarf Labs.

VFX of Source Code

Modus handled pyrotechnics along with digital environments. Courtesy of Summit Ent.

Duncan Jones follows up his brilliant Moon debut with Source Code, a sci-fi thriller with Ground Hog Day overtones starring Jake Gyllenhaal as a soldier investigating a commuter train explosion by revisiting the incident in a continual eight-minute loop.
Louis Morin, the overall visual effects supervisor, hired six facilities to create more than 850 shots: Montreal-based Modus FX, the primary vendor, which contributed nearly 100 shots in-house, including complex digital crowds, CG trains, environments and dramatic pyrotechnics, and another 60 train station assets for the other vendors; Rodeo FX, which crafted most of the greenscreen windows for the train interiors; MPC Vancouver, which handled the big explosions and crash sequences; Fly Studio, which provided key transitions; Mr. X, which did additional window backgrounds; and Oblique FX, which created bomb interiors, a virtual stuntman and a slow-motion explosion sequence.
Seamless photorealism was vital. The shots featured hundreds of greenscreen windows showing passing landscapes or train station backdrops, and the use of CG in post allowed Jones and the actors to work without outside distractions during production. Set extensions on the train station were also crucial, especially since they decided to change the parking lot. That meant more than 65 additional shots, but these also provided logistical maneuverability as well as flexibility with the storytelling. Editorial, meanwhile, was done in tandem with VFX, allowing them to be ahead of the curve, so to speak.
Modus, which previously worked with Morin on Barney's Version, began early on Source Code. "We started doing the previs for the main action scenes --how the train would explode, and where this would take place -- so we were responsible for the feel and pacing of the key effects sequence," explains Yanick Wilisky, VP of production and vfx supervisor at Modus.

Lens distortion and reflections proved challenging.

However, once these shots were roughed out, Wilisky says they were handed over to MPC Vancouver for completion because Morin wanted a more experienced touch. Thus, Modus was able to do what it does best, focusing on exteriors of the train and the station, including reflective windows and the metallic surfaces of the train. Modus also created the CG commuter and cargo train models for the team at MPC, as well as shots depicting enormous traffic jams: lots of virtual crowds and cars for plates of the superhighway in Chicago, which were shot from a helicopter.
"The action takes place in Chicago but lots of it was filmed in Montreal, so we had to do CG replacements of the environments," Wilisky continues. "So we did a mix of Google Maps and other stuff to survey the area and then we went and took lots of pictures and from there we created an accident. And Duncan would say, 'Let's try another area of Chicago.' So we did three different areas until we figured out exactly the action. At the time we were working with Paul Hirsch, the editor, and he was working with us on a lot of versioning. And the final result is 1:1 with what we prevised.

"There was a real train station built in Montreal, but there were missing pieces, including the roof. And there was an empty field surrounding it, so we had to replace it with CG cars and CG parking and the city of Chicago way in the distance. We designed the whole suburb where the action takes place because Glenbrook in the film doesn't really exist so we had to rework and redesign completely. We imagined it being eight miles from downtown Chicago, and we knew the sun was facing northeast so we did it all accurately and up to scale.
"Everything in the train station is CG except for the door step. You don't want to throw the viewer off if they notice something that doesn't look right. For the movie to work, it had to be as transparent as possible. That was the biggest challenge. The other challenge involves the mayhem. The train is about to crash and everyone is running out of the city, so there's lots of traffic and people running around so we had to use Massive isolation for crowd and car behavior. And for the CG parking, everyone has a sense of what it should look like. We even looked at the existing vegetation in Chicago to make sure that it matched in the area that we were in."

The entire train station environment was appropriately Chicago-fied.

In terms of other tools, Modus used Softimage, Houdini, 3D Equalizer, Nuke and mental ray.
Lens distortion proved to be a challenge in matching CG elements with plates. So, to make sure the distortion values were identical, Modus used 3DEqualizer in concert with a special plug-in from La Maison.
In addition, so Modus had to create reflections and shadows of the station and the characters to mirror the movements of the train and the actors in the CG environment.
"We created three techniques," Wilisky explains. "One was a full CG character, such as Jake, that we used as a reflection; the second was a reflection pasted onto surfaces from different angles; and the third was a basic texture applied to a surface of people moving back and forth. All of them were positioned in space, accordingly, so we could get an accurate reflection of the action. And the compositors not only had the layers of all the different actors but also the scene that was already built to scale accurately with all the geometry. This gave them flexibility to move stuff away or create their own map if needed."
And what does Source Code mean to Modus? "Honestly, this one for us is the first time we were the main vendor for something that's more vfx-driven, unlike The American or Barney's Version," Wilisky concludes.

Stitching Together Sucker Punch

Ray Harryhausen would be proud of MPC's samurai. Courtesy of Warner Bros. Pictures.

Yes, there is a method to the madness behind Zack Snyder's Sucker Punch, the mind-bender about trying to escape from an insane asylum through the power of imagination. For her virtual rite of passage, Babydoll (Emily Browning) must defeat three massive samurai, a platoon of German zombies from World War I, a giant dragon and alien robots aboard a bullet train.
And when it came to dividing the four missions and 1,100 vfx shots, John DJ DesJardin, the overall visual effects supervisor, called on MPC Vancouver to tackle the Samurai sequence, Pixomondo to handle the World War I sequence, Animal Logic to create the Dragon sequence and Prime Focus to do the Bullet Train sequence.
"We extended what we learned from Watchmen as far as how we handled our digital characters and capturing shots on set and making the fights work right," confirms DesJardin. "It was a great chance for Zack, his long-time stunt coordinator Damon [Caro] and me to work closely.
"Damon and I came up with this idea called techvis with MPC. Zack could do action during rehearsal and we could quickly turn that into previs from what we captured with Damon and then we'd put that into our environment (a bullet train, a pagoda or a courtyard) and then Zack could set cameras in there. Then Zack would cut his scene before we even shot it in editorial, and Damon and I could take it apart and make these very complicated camera moves and fight moves that were pretty much maxed out in terms of what you could shoot for real and then identify with a lot of assurance what would be the CG stitching in between. We wanted to capitalize on what's been developed over the past several years to project actual footage onto geometry. We came to the conclusion early on that we could best help the girls in these fights if we took them to their limit and then use the CG to take them just a little further and then snap back into the range of what they could do so they're always grounded in something real."

Stitching together digital doubles with the real actress made the action more seamless and believable.

Thus, it was in the creative stitching from digital double to live actor that really made the action smooth and seamless. This came out of two months of R&D that DesJardin initiated to deal with the Bullet Train sequence, which then became the axis for the whole movie.
The sequence required Prime Focus (supervised by Bryan Hirota) to develop a CG helicopter and helipad, alien-like terrains, interior and exterior shots of a magnetic levitation train that gets destroyed in a dramatic style, hoards of armed robots, CG doubles for the three actors and a futuristic metropolis called "Bunny City" inspired by Alien. One two-and-a-half minute shot, in particular, required the animation and render of nearly 20 minutes of material so Snyder could "time warp" everything down to the final length.
As for the robots, Snyder wanted a blank mirror face plate but with a little transparency so you could see underneath, where there's a camera system for an eye and other hardware.
Meanwhile, MPC (under the supervision of Guillaume Rocheron) brought the three samurais to life, combining real samurai armor pieces and detail onto the original, very stylized body proportions. Next, they accurately built the Japanese Pagoda and its surrounding environment, based on the original artwork and set plans by production designer Rick Carter and his team so that each section and plank could be destroyed during the final fight against the machine gun samurai.

Pixomondo had ideal Red Baron experience and also created the Meka vehicle.

To handle the destruction work, MPC's software and vfx teams in Vancouver developed a new destruction system called "Kali" based on Pixelux's DMM, a finite elements solver usually used in the video game industry. This new approach allowed materials to flex and bend before breaking and the ability to define physical properties to realistically simulate wood, metal and stone breaking. It also gave the modelers the freedom to create assets without worrying about how they would break, eliminating the time consuming process of pre-cutting the geometry. MPC then implemented a full retiming solution into the pipeline, allowing artists to work on everything at 100fps and apply the final retime at render time to get accurate motion blur and animation interpolation. The compositing team then pieced together hundreds of CG layers for each shot.
Then, for the World War I sequence, in which Babydoll and her fellow inmates battle German storm troopers with steampowered mechanical faces on the battlefield and in the air, climaxing with the destruction of a giant Zeppelin, Pixomondo (under the supervision of Rainer Gombos) built and destroyed all aircraft, including a Meka, a futuristic armored endoskeleton. They used 3ds Max, Vray, Photoshop, Synthese, Nuke, FX Fume, Afterburner, Krakator, Thinking Particles and proprietary software.
"There were many meetings where Zack and Damon and I were talking Heavy Metal," DesJardin recalls, "and how that magazine, in particular, is responsible for a lot of us fanboys just thinking of these hybrid sci-fi/fantasy stories and environments: 'I want to have a WWI sequence where the girls go in and kick German ass. What if I have some zombies? Cool!'
"We knew Pixomondo's work from The Red Baron, so we figured they would be ideal at building the World War I environment and assets. The only thing we did was give them a face lift to give the scene a sci-fi, steampunk vibe. But Pixomondo made the giant Zeppelin. Rainer really tricked that thing out: there's detail that I never even dreamed of. He even pushed the steampunk guys that I asked for out on the strut near the engine to the nth degree."

Fire was a "Snap" for Animal Logic.

For the Dragon sequence, Animal Logic (supervised by Andy Brown) designed and built the entire volcanic environment and a medieval castle. In order to establish the battle between knights and orcs, they used a combination of motion capture, Massive software and hero animation in Softimage. A proprietary tool was written for viewing and animating Massive crowds in Maya that added speed and efficiency. The dragon fire was created using the proprietary "Snap," developed on Snyder's Legend of the Guardians, but on a much larger scale for Sucker Punch. The R&D department also wrote a custom hair sim tool called "Alfro," for its first foray into digital doubles with long hair, when Babydoll hangs suspended from the dragon.
In fact, the dragon was inspired by Dragonslayer, per DesJardin's suggestion."Zack actually liked one of the Harry Potter dragons and we looked at that first," DesJardin explains. "We fine tuned it to make it more like Dragonslayer: immense wings for cool flight, the neck is shorter but the spikes come off the face the same way. We also studied the motion blur that Phil Tippett achieved with his Go-Motion technique. We took that and ran with it because we knew we had to do the flight attack with the airplane."