Bake-API: new approach for cage

There is a new option to select whether you want to use cage or not.
When not using cage the results will be more similar with Blender
Internal, where the inwards rays (trying to hit the highpoly objects)
don't always come from smooth normals. So if the active object has sharp
edges and an EdgeSplit modifier you get bad corners.

This is useful, however, to bake to planes without the need of adding
extra loops around the edges.

When cage is "on" the user can decide on setting a cage extrusion or to
pick a Custom Cage object. The cage extrusion option works in a
duplicated copy of the active object with EdgeSplit modifiers removed to
inforce smooth normals. The custom cage option takes an object with the
same number of faces as the active object (and the same face ordering).

The custom cage now controls the direction and the origin of the
rays casted to the highpoly objects. The direction is a ray from the
point in the cage mesh to the equivalent point to the base mesh. That
means the face normals are entirely ignored when using a cage object.

For developers:
When using an object cage the ray is calculated from the cage mesh to
the base mesh. It uses the barycentric coordinate from the base mesh UV,
so we expect both meshes to have the same primitive ids (which won't be
the case if the cage gets edited in a destructive way).

That fixes T40023 (giving the expected result when 'use_cage' is false).

Thanks for Andy Davies (metalliandy) for the consulting with normal
baking workflow and extensive testing. His 'stress-test' file will be
added later to our svn tests folder. (The file itself is not public yet
since he still has to add testing notes to it).

Many thanks for the reviewers.

More on cages:
http://wiki.polycount.com/NormalMap/#Working_with_Cages

Reviewers: campbellbarton, sergey

CC: adriano, metalliandy, brecht, malkavian

Differential Revision: https://developer.blender.org/D547

1 files changed, 3 insertions, 2 deletions

diff --git a/source/blender/makesdna/DNA_scene_types.h b/source/blender/makesdna/DNA_scene_types.h
index ceb938c3af2..c95cdb5c1f8 100644
--- a/source/blender/makesdna/DNA_scene_types.h
+++ b/source/blender/makesdna/DNA_scene_types.h
@@ -1442,8 +1442,9 @@ enum {
 #define R_BAKE_LORES_MESH	32
 #define R_BAKE_VCOL			64
 #define R_BAKE_USERSCALE	128
-#define R_BAKE_SPLIT_MAT	256
-#define R_BAKE_AUTO_NAME	512
+#define R_BAKE_CAGE			256
+#define R_BAKE_SPLIT_MAT	512
+#define R_BAKE_AUTO_NAME	1024
 
 /* bake_normal_space */
 #define R_BAKE_SPACE_CAMERA	 0