VFX in Pirates of the Caribbean: On Stranger Tides.

The mermaids were enhanced by a new ILM facial system that decomposes expressions into individual shapes and a new application of Imocap. Images © Disney Enterprises Inc.

For Pirates of the Caribbean: On Stranger Tides, Industrial Light & Magic got to play in the water with mermaids, the Fountain of Youth and shrunken ships in bottles for 300 vfx shots. It was certainly a departure from the crustacean-like creatures and apocalyptic mayhem of the previous Pirates trilogy.
And for Ben Snow, ILM's visual effects supervisor, it was a nice change of pace from the hard surface challenges of the Iron Man franchise. The mermaids were especially different, appearing beautiful and human outside the water to entice and entrap the pirate victims and then menacing underwater with deadly fangs. But rather than going completely CG, they decided to apply a hybrid approach, in keeping with director Rob Marshall's glam aesthetic and desire to retain as much of the live-action performance as possible, particularly when it came to the hero mermaid, Syrena, played by Astrid Berges-Frisbey.
"The look of the mermaids was important," Snow says. "We conceived them as having an inner body that had all the scale texture on them and then an outer membrane that made them look human when they got out of the water. They evolved from being a little more human to a little more creature-like with vestigial gills, but we pulled back on that.

The fin was made more elegant by being proportionally larger than the actress' legs.

"It's a different performance capture challenge," he adds. "In this case, we had to take the scales and match them to the actual bodies and so it was a much more sheer transformation. We came up with new techniques and new tracking costume designs to help automate that: a nice smooth blend of an abdomen to a tail. Or put scales up on arms or faces. Briefly, you see transformations, so we had to match facial performances as well. So we came up with some facial tools."
 According to Tim Harrington, ILM's animation supervisor, the facial capture was driven by two Mova Contour sessions: one to create the facial animation rig and another performance session of the actress watching previs or the shot (ADR style) on a monitor.
"Astrid did about 80 different facial expressions and we have a new proprietary system at ILM where we can take a group of expressions and decompose them into all of the individual shapes that create our facial rig," Harrington explains. "We started by creating a 1:1 match of Astrid and it was one of the most accurate digital doubles ever done at ILM. On top of that, we wanted to either be able to do facial MoCap or to animate by hand the way we did Davy. There were some transformation shots where we were just going to copy the performance from the plate, so we needed to have both approaches and a system that could handle both [as a hybrid]. We had a new Imocap set up for her because she was going to be in water and basically nude. We applied markers using a tattoo stencil on her arm instead of the traditional bands of Velcro to capture her upper body. It would then go to animation and we would attach our mermaid to that and animate the tail [designed by Aaron McBride] using her legs as a basis. We came up with a big fin that was long and elegant that could be simulated."

ILM decided on a more dramatic approach to have the ships look like they're frozen in time at the moment of capture.

New tools for water and interaction with the mermaids were also created. "The idea of taking 30 creatures with all this streaming stuff (flowing hair and tendrils), and then running them through a full-on cloth sim was never going to happen," Snow suggests. "We worked out a way to bake all of these simulations into the cycle, and even some simplified simulation tools that the technical directors could run just so we didn't have to use our creature team to do first passes on all these things. And, likewise, we developed a bunch of little library splashes and then a means of plugging those in so that as the mermaids are attacking, they could automatically generate different types of splashes. It was really layers of tools on top of the simulation engines to make them run faster and easier to do.
ILM not only used its PhysBAM fluid sim engine, but it also applied the Plume GPU-accelerated technique developed with NVIDIA for smoke and fire on The Last Airbender. This was used for mist and for overall fast turnaround. ILM also used Houdini and Maya for additional sim help. "It was handy to have automated tools to help the artists and Rob Marshall to read the animation," Snow adds. "If the mermaid was moving in space, looking almost flat shaded, you could more easily buy the hardware render. But if we didn't put the splashes in quickly, it didn't look fast or powerful enough. The water interaction was key to selling the performance, and we learned a lot about getting characters to move through water convincingly. You had to have accurate-looking simulations right out of the box to show that the animation was performing well."
ILM collaborated with Scanline VFX for the triggering of the Fountain of Youth, beginning with a droplet of water running along a leaf, and then when water flows along the walls of the cave to make a portal. However, for a spectacular liquefying death resulting in a nasty skeleton, ILM scrambled to get some extra water sim by adding a whirling effect. There were some rendered surfaces, meta-surfaces for more of a glossy look, different layers of particle sim created with PhysBAM and the simulation tools and some practical water.

Macro photography was used for this surprise appearance by the monkey.

Finally, for the effect of shrinking ships in bottles (including the Black Pearl), this involved setting up simulations in San Francisco by Chris Foreman and then working with Mohen Leo and the Singapore team. They executed most of the shots.
"Initially, we just thought that the ships would be affected by the light in the room that they're in," Snow explains. "But then this idea came to have the ships look like they're frozen in time at the moment of capture: in battle or in a snowstorm in the Arctic or in a storm at sea. The art department made several bottles with real model ships inside. For the wider shots, we replaced our key ones of those and essentially added movement to some of the others. And then as we got closer, they became fully CG. We even played with macro photography on some shots.
"There were a couple of happy accidents where the compositors went over the top with a big explosion. I initially had them hold it back. But I showed Rob Marshall the early take and he requested that we go back to the bigger explosion."