Tutorial: Soft AO Cel Shading

Soft AO Cel Shading Title
In this tutorial you will discover how to set up an interesting NPR style in Blender Internal.

We’ll need the Line Squiggle node (which we created in that tutorial) and the EdgeNode, both contained in the Sharkigator Node Collection (Download).


I will use a toy helicopter model in this tutorial, which I made a while ago, to demonstrate the technique.
It is also included in the Sharkigator Node Collection in the folder “Setups”, file “HelicopterLite.blend”.
You can Link/Append the group “Helicopter” to you project, if you want to use it.

Note: If you want to use the helicopter model, you’ll have to scale it up, because it’s pretty small.
E.g. press S 50 Enter to scale up 50 times.

Of course you can also use your own model.

So, start with a clean fresh scene with your model in it.

I set up a background plane (to get nice occlusion effects) and a camera, it should look something like this now:

Fig. 1 | Full size.

Scene Settings

We’ll use Ambient Occlusion only.
You could also do this in Cycles, but we want to use the EdgeNode, which works best in Blender Internal, where we can enable Full Sample Anti-Aliasing, to get smooth edges.
(Though you could combine both if you wanted to…)

So, in the properties panel in the World Settings (little globe icon) check the Ambient Occlusion checkbox.
Down in the Gather settings you can adjust the settings, if you want to. (Just play with them.)

For this tutorial’s featured image, I used Adaptive QMC sampling and 16 samples.
It is important, that the resulting image is as noise free as possible, or you will get strong dithering in the compositing.
So you will have to increase the sample count if necessary.

Fig. 2


Now we’ll need materials.
I used one single solid colored material, but you could also assign multiple materials. (See faaaar below in the ideas section.)

So select an object (e.g. the background plane) and create a new material and give it a name. (“AO” in this example)
You can just leave it at the default settings.

Now go to the Render Layers.
Set our newly created material as an Override Material.

Also enable the Normal and AO passes and click the little camera button next to the AO pass checkbox (exclude from combined).
We’ll need the Normal pass for the edge detection and the AO pass for the compositing.

Fig. 3


Now we’re ready to jump into the compositor.
Press F12 to render, so we’ve got something to work with.

In the Compositor, add Filter>Filter node and set the type to Soften. Connect it to the AO input.
Also add a Converter>ColorRamp node, connect it’s input to the Soften node’s output and connect it’s output to the Composite Output.

The Soften node will blur the AO pass slightly, so it’s smoother.
The ColorRamp will work like a “Diffuse Ramp” (if you’re familiar with NPR shading).

It should look like this:

Fig. 4 | Full size.


Now you can start adjusting the ColorRamp.
Add color stops, and move them around until you like the result.

By pulling two color stops close together, you will get harder looking color gradients.

Below, you can see the color ramp I used for the render.

Fig. 5 | Full size.


By now, it should look something like this:

Fig. 6 | Full size.


Now we want to add some edges on top.
For that we’ll use the EdgeNode.
Go ahead and append it in.

To add the EdgeNode to the NodeTree, press Shift+A, then Group>EdgeNode.
Connect the Z and Normal input. Then tweak the Distance and Normal Enhance input, until you’re happy with the edge detection.

You can now also add the LineSquiggle Node, to make the lines a bit squiggly.
Use a Filter>Dilate/Erode node, with the mode set to “Distance”, to control the line thickness.

To give the edges a color, add a Converter>Set Alpha node, and plug the edges mask into the Alpha input.
You can use the Image input of the Set Alpha node to set a line color (e.g. a dark brown).

Use a Color>Alpha Over node to combine the edges with the rest of the image.
The edges go into the lower input.
Don’t forget to check “Convert Premul“.

Now, the node setup looks like this:

Fig. 7 | Full size.


Now we’re almost done.
To make it look even better, you can overlay a texture. (For example one from my Paper Texture Vol. 01. I used “Watercolor006.jpg”)

Add a Color>Mix node and set it to “Overlay”.
Plug the output of the Alpha Over node into it’s first input.
Now add a Input>Image node and use it to load your overlay texture. Scale it to the correct resolution using a Distort>Scale node with the mode set to “Render Size” and the frame method set to “Crop”.
Plug the output of the scale node into the second input of the Overlay node.
Now you can control the influence of the texture with the Factor slider of the overlay node.


Finally, you can go into the Color Management (Scene Tab in the Properties Panel) to adjust the colors of your image.
In this case, I made it slightly orange and increased the contrast.

Now for the final render you can increase the samples of your Ambient Occlusion and turn on Full Sample in the Anti-Aliasing settings (Render tab in the Properties Panel).
The result looks like this:

Fig. 8 | Full size.


Wondering how you could possibly improve this now?
There’s a few things you can do now:

  • Do a paint over:
    That’s what I did. I like to use this technique for basic shading and color “by hand” afterwards.
  • Use more sophisticated lighting:
    You could use some lamps and use the Combined pass (we didn’t use it at all for this tutorial) to get more control over your lighting.
  • Assign (colored) materials:
    Every object can get it’s own material and color with it. You can get the colors (and even the textures, if you assigned ones) using the Color pass.

One thought on “Tutorial: Soft AO Cel Shading

  1. Hey man!

    This looks sweet! Will it work with animation as well?

    Again, great, great job!!!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s