Bake-API: new approach for cage

There is a new option to select whether you want to use cage or not.
When not using cage the results will be more similar with Blender
Internal, where the inwards rays (trying to hit the highpoly objects)
don't always come from smooth normals. So if the active object has sharp
edges and an EdgeSplit modifier you get bad corners.

This is useful, however, to bake to planes without the need of adding
extra loops around the edges.

When cage is "on" the user can decide on setting a cage extrusion or to
pick a Custom Cage object. The cage extrusion option works in a
duplicated copy of the active object with EdgeSplit modifiers removed to
inforce smooth normals. The custom cage option takes an object with the
same number of faces as the active object (and the same face ordering).

The custom cage now controls the direction and the origin of the
rays casted to the highpoly objects. The direction is a ray from the
point in the cage mesh to the equivalent point to the base mesh. That
means the face normals are entirely ignored when using a cage object.

For developers:
When using an object cage the ray is calculated from the cage mesh to
the base mesh. It uses the barycentric coordinate from the base mesh UV,
so we expect both meshes to have the same primitive ids (which won't be
the case if the cage gets edited in a destructive way).

That fixes T40023 (giving the expected result when 'use_cage' is false).

Thanks for Andy Davies (metalliandy) for the consulting with normal
baking workflow and extensive testing. His 'stress-test' file will be
added later to our svn tests folder. (The file itself is not public yet
since he still has to add testing notes to it).

Many thanks for the reviewers.

More on cages:
http://wiki.polycount.com/NormalMap/#Working_with_Cages

Reviewers: campbellbarton, sergey

CC: adriano, metalliandy, brecht, malkavian

Differential Revision: https://developer.blender.org/D547

1 files changed, 10 insertions, 2 deletions

diff --git a/source/blender/makesrna/intern/rna_scene.c b/source/blender/makesrna/intern/rna_scene.c
index e272d93a1e7..92c4d3dd204 100644
--- a/source/blender/makesrna/intern/rna_scene.c
+++ b/source/blender/makesrna/intern/rna_scene.c
@@ -3218,8 +3218,10 @@ static void rna_def_bake_data(BlenderRNA *brna)
 	RNA_def_struct_nested(brna, srna, "RenderSettings");
 	RNA_def_struct_ui_text(srna, "Bake Data", "Bake data for a Scene datablock");
 
-	prop = RNA_def_property(srna, "cage", PROP_STRING, PROP_NONE);
-	RNA_def_property_ui_text(prop, "Cage", "Object to use as cage");
+	prop = RNA_def_property(srna, "custom_cage", PROP_STRING, PROP_NONE);
+	RNA_def_property_string_sdna(prop, NULL, "cage");
+	RNA_def_property_ui_text(prop, "Custom Cage", "Object to use as cage "
+	                         "instead of calculating the cage from the active object with cage extrusion");
 	RNA_def_property_update(prop, NC_SCENE | ND_RENDER_OPTIONS, NULL);
 
 	prop = RNA_def_property(srna, "filepath", PROP_STRING, PROP_FILEPATH);
@@ -3309,6 +3311,12 @@ static void rna_def_bake_data(BlenderRNA *brna)
 	RNA_def_property_ui_text(prop, "Automatic Name",
 	                         "Automatically name the output file with the pass type (external only)");
 	RNA_def_property_update(prop, NC_SCENE | ND_RENDER_OPTIONS, NULL);
+
+	prop = RNA_def_property(srna, "use_cage", PROP_BOOLEAN, PROP_NONE);
+	RNA_def_property_boolean_sdna(prop, NULL, "flag", R_BAKE_CAGE);
+	RNA_def_property_ui_text(prop, "Cage",
+	                         "Cast rays to active object from a cage");
+	RNA_def_property_update(prop, NC_SCENE | ND_RENDER_OPTIONS, NULL);
 }
 
 static void rna_def_scene_game_data(BlenderRNA *brna)