SOFTIMAGE|XSI Case Study: TECMO - NINJA GAIDEN 2

By Yoshiyuki Watanabe

Ninja Gaiden is an enormously popular game title around the world, with many seeing it as the ultimate action game series. In the latest title released for the Xbox 360—Ninja Gaiden 2—players can test Ryu Hayabusa's astounding physical abilities and weapon skills in lightning fast combat, captured in full HD resolution. Ninja Gaiden 2 is full of thrilling action that can be enjoyed by everyone, from casual players to hardcore gamers.

Behind the development of this game is Team Ninja, a game pro-duction group at TECMO, LTD. The team said that their objective
for the new Ninja Gaiden 2 was to develop a game that was as enjoyable to look at as it was to play. In particular, their mission was to fully utilize the processing capabilities of the next-generation Xbox 360 to take real-time image display to the next level.

To create expressive animated characters, The creative team started working with real-time shaders
to add details that would improve the game resolution required for a next-generation console. But the main issue they faced was how to raise the quality of the facial animation that would accompany this added detail. It was at this time that one of the directors of Ninja Gaiden 2, Hiroaki Matsui, took a look at SOFTIMAGE|FACE ROBOT. The team felt that Face Robot provided the level of accuracy required for facial capture operations. The fact that it was designed with an export function for game runtime was another major reason why they decided to adopt Face Robot.

Facial Rig Setup with Face Robot

After Team Ninja decided to adopt Face Robot, staff from the Softimage Headquarters visited the Temco team several times to train the artists on the basic functionality. This enabled the team to smoothly incorporate Face Robot into their project workflow immediately after the training.

In Ninja Gaiden 2, Face Robot was used mainly for the facial animation of about 20 human characters that appeared in cinematic scenes. Face Robot was also used to create some in-game scenes featuring monsters with human faces. Facial animation for characters with monster faces was controlled with Magical VEngine, which is good at matching facial expressions to spoken dialogue using sound analysis.

Facial animation using Face Robot was performed by a team of three people, including Masahiro Nose, who was interviewed for this article. “First, we started the facial rig setup by creating a
model optimized for Face Robot,” said Mr. Nose.

The team then performed landmark picking and curve fitting on the completed model to make Face Robot recognize the shape and muscle movement of the face. Next, they ran the solver process to automatically generate the Face Robot rig. Technology this team had previously used, tended to produce linear facial movements for the shape control and bone control, but with Face Robot, the team was able to achieve smooth muscle movement using the mouth solver and facial solver controls. (When the control rig for the chin is moved, the muscles in the cheeks and around the
eyes also move smoothly.)

In most cases, it took less than 30 minutes to generate a simple rig in Face Robot. To add the desired facial expressions to the generated rig, the team then needed to perform adjustments—a process known as "tuning."

For example, envelope values can be adjusted, or areas painted, to define the strength and range of each controller. In the same way, the paint function can be used to define the wrinkle map and give life-like expression to wrinkles on character faces. In addition to these controls, Face Robot is equipped with many other adjustment functions that can bring subtle facial expressions to life, such as the shutting of eyelids, lip movement and the muscle movements around the chin. The team said that facial control was easiest for the older characters that had a wide range of expressions. The most painstaking work was creating the detailed facial expressions of the beautiful female characters. But once the team became skilled in the tuning process, the work went very smoothly.

Facial Capture Recording

While the Face Robot team was under way with the setup and tuning work, in August 2007, the capture data for facial animation was recorded and captured in a studio at Dynamo Pictures, Inc.

In some facial capture processes, the voice and facial performance are recorded at the same time. In this case, the directors wanted to use professional voice actors for maximum effect, so they recorded the voice parts prior to recording the facial capture data. The facial actors practiced their lines for the capture data recording session in advance, in order to match the voice actors’ performances as best they could. Since Team Ninja knew the game would be released globally, most of the lines were recorded in English, with appropriate facial motions. But since the game originates in Japan, they made sure that the facial expressions would also look natural in the Japanese edition.

To record the facial motion capture, eight Vicon cameras were arranged in a half-circle in the recording studio, and 35 markers were affixed to the faces of the actors. While reading the lines that were displayed on the monitor, they performed their facial expressions to match the pre-recorded voices. The captured data was then written in C3D format by Vicon for import into Face Robot.

Hiroaki Matsui giving performance directions based on the storyboard. Kenya Miki from Dynamo Pictures doing some editing work		Camera setup for recording Test performance by a Softimage staff member

In addition to the capture data, the positions of the actors’ bodies were also recorded for reference. The team said that this video reference was useful for setting targets in later animation work.

Mr. Matsui said that the advantage of data capture is that actions and expressions that are difficult to reproduce with manual animation can be performed by the actors on location during shooting. Facial animations were recorded one after another based on Mr. Matsui's performance directions, and C3D data was then generated from these recordings. About 330 capture data source types were recorded in all.

During the capture session, Team Ninja could preview materials immediately, on location, by importing C3D motions onto sample Face Robot models. Not only did this help ensure that there were no problems with the data, but it also boosted the morale of the actors. Seeing how their performances were being reflected in the data increased their enthusiasm for the performance. The fact that Face Robot could pick up their individual facial characteristics even in these rough
previews greatly fascinated the actors.

Powerful Retargeting Function

Once the Face Robot setup was completed and the C3D capture data obtained, the team's next step was to import the motion into Face Robot. For this, the retargeting technique in Face Robot was particularly useful. It goes without saying that the face shapes of the actors were different from the face shapes of the characters in the game. Corrections and adjustments were required for matching these different shapes, but with Face Robot, the user did not need to be aware of them. This is because the retargeting technique in Face Robot is based on an internal, automatic calculation, which allows motion to be applied in a one-click operation. Naturally, the user can use this retargeting technique to apply a single motion element to many differently-shaped face models.

After importing the C3D capture data, the team performed various manual animation adjustments to create the ideal facial expressions for the game characters.

First, they used a Face Robot function called "Calibrate" to adjust the performance of the entire motion element. Calibrate values could be varied to minimize face movements, or conversely, to exaggerate them. While the Calibrate function was used to adj-ust motion over the entire face, Adjust parameters were used for fine adjustments. The Adjust function can control motion varia-tions for each rig controller and can also vary the performance using keyframe animation.

The team used the Adjust function to control the timing of the characters’ blinking. First, they set the blinking parameters in the Adjust panel to zero and muted the captured movements. They then performed manual animation to make the blinking occur at the desired time.

Face Robot also makes it possible to add animation layers on top of a base animation created with motion capture. The layers are separate from the capture data so animators can use the capture data as a base, and add exaggerated movements and other variations on to of that. This kind of manual key frame animation work also has the advantage of enabling fine adjustments to be flexibly performed at the time of cut and paste operations.

The capture data can be flexibly adjusted with Adjust. Fine
adjustments can also be performed with key frame layer control.
>> View Movie

Further, plotting can be performed to convert the capture data into F-curve animation. The team plotted capture data to adjust the synchronization with the audio and to cut unnecessary frames.

In this way, different methods were used for different shots, and scenes were animated by combining capture data with manual animation. Although most projects use capture data as the base animation, we have heard of cases where the Face Robot controllers were animated manually right up to the final stages.

Face Robot's Game Export Function

Having been made more expressive, the Face Robot models were then output using the game export function. The game export function in Face Robot converts the motions from the facial solver to envelope weight values and Null bone movement and rotation values that can be used in the game engine. The first step is to use the paint function to define the Null bone positions for the game models. Then, Face Robot automatically calculates the optimum envelope value for the motion and reproduces the facial expression deformation using the Null bone movement and rotation.

Although the number varies from character to character, an average of 50 bones were used in the faces of the Ninja Gaiden 2 characters. The envelope weight values and Null bone animations were output to the console where the facial animation was reproduced.

Face Robot models with muscle movements derived from the solvers
and the Null bone control game export models can be checked.
>> View Movie

In addition to reproducing facial expressions with Null bones, another Face Robot feature is the ability to create subtle facial expressions with real-time shaders. In Face Robot animation, rich and varied expressions are created through solver calculations when the controllers are moved. During these controller movements, "stress maps", are calculated internally. Face Robot has a function that converts this information into numerical animation and uses it to perform texture blending. The team used this technique in Ninja Gaiden 2. For example, the wrinkles in the brow furrows of some characters were created by blending a normal map with a color map.

The team's workflow was as follows. First, they used Zbrush to create two normal maps for a given model. One normal map had wrinkles while the other did not. They also created two color maps with color variations. The textures of these maps were blended based on the stress value animations. However, subtle expressions could not be created just by blending the entire face. They achieved local blend control for the forehead and right eye by using Face Robot to automatically calculate blend animations corresponding to nine areas. Because the data for these areas were animation values for three color RGB values (R, G, B × 3 = 9 areas), they could use them in the game runtime. Of course, it was also easy for the team to redefine the wrinkle area by retouching the color information.

Blend control of nine areas		Facial expression using maps

Facial Animation Preview

In general, the team performed their animation work in Face Robot in a stabilized condition (while the faces were fixed). So, how did they connect faces to bodies to check the animation?  Instead of the method where the face and body rigs are imported into Face Robot at the same time, they performed all the checking work for the face-to-body connection animation on the console. The team used this console preview method to facilitate coordination with the game export workflow and the body motion supervisor.

First, the character movement completed by the body animator was output to the console. Next, facial animation from Face Robot was output to the same location, where both parts were combined. The rotation animation value for the neck that was recorded during capture was used to link the head to the body. The team said that because data from above the neck was output, problems such as gaps did not occur when connecting to the body.

However, the production team set the eye line (pupil movement) animation on the game console without creating it with a CG tool. They set the focal point movements of eyes on the console, and then defined this information in the structural data used for controlling event scene setup.

In addition to the console output checking method, simple motion checks such as those below can be performed by canceling the stabilized condition and using the Face Robot screen capture function.

Impressions of Using Face Robot

Masahiro Nose said that using Face Robot for facial animation presented him with many new learning options. He tested the limits of the program many times through trial and error, building up his know-how during the production process. He said that each time he retook a scene, he could see the quality getting better and better.

Team Ninja also used Face Robot for special applications. For example, the animation for the ninja mask of Ryu Hayabusa, the main character in the game. First, they used Face Robot to animate Ryu's head shape, including the imagined movement of his mouth. They then ran game export. Next, they transferred the envelope weight value to the Null bone that had grouped together the mask object. We have to admire the team for reproducing subtle mouth movements in the game, which would not have been possible had they directly animated the ninja mask. They also performed this envelope transfer control for other items such as eyelashes, facial hair and eye patches.

Using Face Robot for finely tuned facial expressions.
>> View Movie

Mr. Nose added, "When we used Face Robot and facial capture in parallel, we had to be careful when reproducing movements that produced a lot of motion around the lips. Usually, it is difficult to express plosive sounds in English words starting with the letters V and P. But we were surprised that with Face Robot, we could reproduce even sensuous lip movements just by importing the capture data, without the need for adjustments. Further, the natural linking of muscles with the facial solver allowed us to go beyond what is possible with manual animation."

When capture data is used, whether a character is angry or sad, the emotion can be understood even from hearing a simple sound such as "ah". Face Robot can accurately reproduce emotions. These expressions are difficult to achieve with a shape or rig base, and is one of the strengths of the capture data operation based around Face Robot.

Mr. Nose looked back over his time on Ninja Gaiden 2.
"During the project, everyone at Softimage, whether they were in Canada, the USA or Japan, all worked together to fix bugs and strengthen functions for us. We were very satisfied with the positive support that we received—a major benefit of the Face Robot product, which includes consulting. In this project, we were able to build up a wide range of operational know-how. We were very happy that we could try new things, and that these led to a successful completion to the game. We think that we can use our accumulated knowledge in future projects to take the quality to even higher levels."

SOFTIMAGE|XSI: The Main Tool

The 3D parts in Ninja Gaiden 2, other than those created with Face Robot, were developed using XSI. Kenichiro Nakajo, the chief supervisor who used XSI to create background CG in Ninja Gaiden 2 has been a prolific user of Softimage products—even since the early days of SOFTIMAGE|3D, running on the platform.

The background production process started with the concepts from Mr. Matsui's planning group. The level design team then designed the maps, based on which the CG team started to build the 3D data. Events such as battles were then imported onto the backgrounds, and if there were no problems, further details were added. In Ninja Gaiden 2, wide-area maps were created in much more detail; for example, there are some maps which once passed through cannot be passed through again. In addition to wide-area maps, other next-generation elements such as HDR, tone mapping and normal maps were required. This meant that volumes and data size were greatly increased. Even though the data size increased by 60% to 70% compared with the last title, work with XSI was very easy to perform. On the game console, image processing is amazingly fast even when many dense characters appear at the same time on a heavy background.

These results were made possible through the skills and efforts not only of the designers, but also the programmers. The solidarity and cooperative spirit that exists between designers and programmers as they work together in an atmosphere of mutual respect is one of the hallmarks of Team Ninja.

For designers to create data that can be processed at high speed, it is important to delete unnecessary information. The team used XSI's polygon reduction to produce LOD models of large backgrounds. Even for independent objects, they performed optimization through repeated merge
and split operations to improve the imaging performance. The XSI non-destructive environment allows this merging and splitting to be performed any number of times while maintaining the object's UV information, which makes it perfect for repeated trial and error.

Mr. Nakajo said, "The background is very important for establishing the mood of a game. Even though Ninja Gaiden 2 is full of nonstop action, and the player hardly has time to look at the background, we created a design that would convey the mood of the game powerfully and coherently. We would like players to have an extraordinary experience while playing the game, and hope that the atmosphere contributes to this."

Naoya Okamoto, was active on a number of different fronts in Ninja Gaiden 2, including script production, rig setup, enhancing the environment for the motion group, and acting as a bridge
between different departments.

Mr. Okamoto said that while the rig structure for the central character, Ryu Hayabusa, was relatively simple—because they used existing data and wanted to minimize processing loads—creating many of the other characters were quite challenging. For example, they introduced an expression for controlling arms so that when an arm was raised, it was done in a natural way that moved the shoulder. This allowed animators to create natural-looking animation without having to think about the shoulder movement.

Another challenge the team faced in the rig setup was the high number of creatures with tails and tentacles. For these, they performed rig setup by creating script that automatically incorporated Spline IK. Mr. Okamoto said that the script environment in XSI was more than satisfactory. It was easy for beginners to customize based on the log, while advanced users could choose from a wide variety of languages to perform advanced control techniques. In motion creation there are a lot of simple and repetitive operations, so it was effective for them to use scripts to automate these processes.

Team Ninja created a work environment where they could preview the XSI animation on a console with only a few clicks. For testing body action animation, previews had to be output to the console many times. For this reason, they created an environment where previews could be quickly performed

The workflow for previewing was as follows. When the team ran the tool, the items required for preview output were all automatically identified. The process was so easy that the only action required of the operator was to select the output destination. Linked to this preview function, the latest data was transferred automatically to the master data server. Excel files were also updated with the data, for information management. These environment enhancements also facilitated coordination with other departments.

Regarding his work in the motion group, Mr. Okamoto had this to say. "The most important thing for Ninja Gaiden 2 was to create animation that could properly communicate nuances such as weight and show fatigue in the characters. To create these animations, XSI has such a wide range of excellent choices that sometimes we did not know which one to use. I would very much like to study more about the functions that we couldn't use in this project and use them next time."

Finally, XSI is used as the main tool for 3D content production at all TECMO game production departments, including Team Ninja. It is easy to imagine that having a unified tool eliminates the work involved in converting data, and avoids the need to develop plug-ins that are compatible with multiple software types.

Mr. Matsui wrapped up the interview by saying, "Using the same tool has the major benefit of cultivating the same know-how and skills across different work groups, while making it easy to share data. Being able to move staff between sections or add staff to projects during critical stages makes full use of our production resources and creates a more dynamic workplace. I have supervised motion design in the past, and I have been familiar with Softimage products ever since the days of SI3D. Because we have grown in parallel, even now as a director I know that I can trust Softimage tools such as Face Robot and XSI. In this latest Ninja Gaiden 2 project, I think that the hopes and expectations that our development staff had in these products as 3D development tools were fully satisfied."

The Team Ninja staff that we interviewed

TECMO, LTD.

Game Production: Team Ninja (from right)
Supervisors for the Ninja Gaiden 2 project

Hiroaki Matsui: Supervisor for art, direction
Masahiro Nose: Supervisor for character CG production and facial animation using Face Robot
Naoya Okamoto: Supervisor for environment enhancements related to motion, such as script
creation and rig setup
Kenichiro Nakajo: Supervisor for background CG production

Visit the TECMO website
Visit the official Ninja Gaiden 2 website

© TECMO, LTD. Team Ninja 2008