Collections and groups unification

OVERVIEW

* In 2.7 terminology, all layers and groups are now collection datablocks.
* These collections are nestable, linkable, instanceable, overrideable, ..
  which opens up new ways to set up scenes and link + override data.
* Viewport/render visibility and selectability are now a part of the collection
  and shared across all view layers and linkable.
* View layers define which subset of the scene collection hierarchy is excluded
  for each. For many workflows one view layer can be used, these are more of an
  advanced feature now.

OUTLINER

* The outliner now has a "View Layer" display mode instead of "Collections",
  which can display the collections and/or objects in the view layer.
* In this display mode, collections can be excluded with the right click menu.
  These will then be greyed out and their objects will be excluded.
* To view collections not linked to any scene, the "Blender File" display mode
  can be used, with the new filtering option to just see Colleciton datablocks.
* The outliner right click menus for collections and objects were reorganized.
* Drag and drop still needs to be improved. Like before, dragging the icon or
  text gives different results, we'll unify this later.

LINKING AND OVERRIDES

* Collections can now be linked into the scene without creating an instance,
  with the link/append operator or from the collections view in the outliner.
* Collections can get static overrides with the right click menu in the outliner,
  but this is rather unreliable and not clearly communicated at the moment.
* We still need to improve the make override operator to turn collection instances
  into collections with overrides directly in the scene.

PERFORMANCE

* We tried to make performance not worse than before and improve it in some
  cases. The main thing that's still a bit slower is multiple scenes, we have to
  change the layer syncing to only updated affected scenes.
* Collections keep a list of their parent collections for faster incremental
  updates in syncing and caching.
* View layer bases are now in a object -> base hash to avoid quadratic time
  lookups internally and in API functions like visible_get().

VERSIONING

* Compatibility with 2.7 files should be improved due to the new visibility
  controls. Of course users may not want to set up their scenes differently
  now to avoid having separate layers and groups.
* Compatibility with 2.8 is mostly there, and was tested on Eevee demo and Hero
  files. There's a few things which are know to be not quite compatible, like
  nested layer collections inside groups.
* The versioning code for 2.8 files is quite complicated, and isolated behind
  #ifdef so it can be removed at the end of the release cycle.

KNOWN ISSUES

* The G-key group operators in the 3D viewport were left mostly as is, they
  need to be modified still to fit better.
* Same for the groups panel in the object properties. This needs to be updated
  still, or perhaps replaced by something better.
* Collections must all have a unique name. Less restrictive namespacing is to
  be done later, we'll have to see how important this is as all objects within
  the collections must also have a unique name anyway.
* Full scene copy and delete scene are exactly doing the right thing yet.

Differential Revision: https://developer.blender.org/D3383

https://code.blender.org/2018/05/collections-and-groups/

11 files changed, 74 insertions, 59 deletions

diff --git a/source/blender/depsgraph/DEG_depsgraph.h b/source/blender/depsgraph/DEG_depsgraph.h
index 604034bb0a7..0a0e7ee638b 100644
--- a/source/blender/depsgraph/DEG_depsgraph.h
+++ b/source/blender/depsgraph/DEG_depsgraph.h
@@ -262,11 +262,9 @@ void DEG_debug_print_eval_parent_typed(struct Depsgraph *depsgraph,
                                        const char *function_name,
                                        const char *object_name,
                                        const void *object_address,
-                                       const char *object_type,
                                        const char *parent_comment,
                                        const char *parent_name,
-                                       const void *parent_address,
-                                       const char *parent_type);
+                                       const void *parent_address);
 
 void DEG_debug_print_eval_time(struct Depsgraph *depsgraph,
                                const char* function_name,
diff --git a/source/blender/depsgraph/DEG_depsgraph_build.h b/source/blender/depsgraph/DEG_depsgraph_build.h
index 3460cbf7c91..202e8ef3cf0 100644
--- a/source/blender/depsgraph/DEG_depsgraph_build.h
+++ b/source/blender/depsgraph/DEG_depsgraph_build.h
@@ -42,7 +42,7 @@ struct Depsgraph;
 
 struct CacheFile;
 struct EffectorWeights;
-struct Group;
+struct Collection;
 struct Main;
 struct ModifierData;
 struct Object;
@@ -159,7 +159,7 @@ typedef bool (*DEG_CollobjFilterFunction)(struct Object *obj, struct ModifierDat
 void DEG_add_collision_relations(struct DepsNodeHandle *handle,
                                  struct Scene *scene,
                                  struct Object *object,
-                                 struct Group *group,
+                                 struct Collection *collection,
                                  unsigned int modifier_type,
                                  DEG_CollobjFilterFunction fn,
                                  bool dupli,
diff --git a/source/blender/depsgraph/intern/builder/deg_builder_nodes.cc b/source/blender/depsgraph/intern/builder/deg_builder_nodes.cc
index 31a0e74b569..98881ba3e57 100644
--- a/source/blender/depsgraph/intern/builder/deg_builder_nodes.cc
+++ b/source/blender/depsgraph/intern/builder/deg_builder_nodes.cc
@@ -71,12 +71,12 @@ extern "C" {
 #include "BKE_action.h"
 #include "BKE_armature.h"
 #include "BKE_animsys.h"
+#include "BKE_collection.h"
 #include "BKE_constraint.h"
 #include "BKE_curve.h"
 #include "BKE_effect.h"
 #include "BKE_fcurve.h"
 #include "BKE_idcode.h"
-#include "BKE_group.h"
 #include "BKE_key.h"
 #include "BKE_lattice.h"
 #include "BKE_library.h"
@@ -388,7 +388,7 @@ void DepsgraphNodeBuilder::build_id(ID* id) {
 	}
 	switch (GS(id->name)) {
 		case ID_GR:
-			build_group((Group *)id);
+			build_collection((Collection *)id);
 			break;
 		case ID_OB:
 			build_object(-1, (Object *)id, DEG_ID_LINKED_INDIRECTLY);
@@ -419,29 +419,21 @@ void DepsgraphNodeBuilder::build_id(ID* id) {
 	}
 }
 
-void DepsgraphNodeBuilder::build_group(Group *group)
+void DepsgraphNodeBuilder::build_collection(Collection *collection)
 {
-	if (built_map_.checkIsBuiltAndTag(group)) {
+	if (built_map_.checkIsBuiltAndTag(collection)) {
 		return;
 	}
-	/* Build group objects. */
-	LISTBASE_FOREACH (Base *, base, &group->view_layer->object_bases) {
-		build_object(-1, base->object, DEG_ID_LINKED_INDIRECTLY);
+	/* Build collection objects. */
+	LISTBASE_FOREACH (CollectionObject *, cob, &collection->gobject) {
+		build_object(-1, cob->ob, DEG_ID_LINKED_INDIRECTLY);
 	}
-	/* Operation to evaluate the whole view layer.
-	 *
-	 * NOTE: We re-use DONE opcode even though the function does everything.
-	 * This way we wouldn't need to worry about possible relations from DONE,
-	 * regardless whether it's a group or scene or something else.
-	 */
-	add_id_node(&group->id);
-	Group *group_cow = get_cow_datablock(group);
-	add_operation_node(&group->id,
-	                   DEG_NODE_TYPE_LAYER_COLLECTIONS,
-	                   function_bind(BKE_group_eval_view_layers,
-	                                 _1,
-	                                 group_cow),
-	                   DEG_OPCODE_VIEW_LAYER_EVAL);
+	/* Build child collections. */
+	LISTBASE_FOREACH (CollectionChild *, child, &collection->children) {
+		build_collection(child->collection);
+	}
+
+	add_id_node(&collection->id);
 }
 
 void DepsgraphNodeBuilder::build_object(int base_index,
@@ -514,7 +506,7 @@ void DepsgraphNodeBuilder::build_object(int base_index,
 	}
 	/* Object dupligroup. */
 	if (object->dup_group != NULL) {
-		build_group(object->dup_group);
+		build_collection(object->dup_group);
 	}
 }
 
@@ -848,7 +840,8 @@ void DepsgraphNodeBuilder::build_rigidbody(Scene *scene)
 
 	/* objects - simulation participants */
 	if (rbw->group) {
-		LISTBASE_FOREACH (Base *, base, &rbw->group->view_layer->object_bases) {
+		const ListBase group_objects = BKE_collection_object_cache_get(rbw->group);
+		LISTBASE_FOREACH (Base *, base, &group_objects) {
 			Object *object = base->object;
 
 			if (!object || (object->type != OB_MESH))
@@ -922,7 +915,7 @@ void DepsgraphNodeBuilder::build_particles(Object *object)
 				break;
 			case PART_DRAW_GR:
 				if (part->dup_group != NULL) {
-					build_group(part->dup_group);
+					build_collection(part->dup_group);
 				}
 				break;
 		}
diff --git a/source/blender/depsgraph/intern/builder/deg_builder_nodes.h b/source/blender/depsgraph/intern/builder/deg_builder_nodes.h
index 0ed3f5e334f..488f4f273b9 100644
--- a/source/blender/depsgraph/intern/builder/deg_builder_nodes.h
+++ b/source/blender/depsgraph/intern/builder/deg_builder_nodes.h
@@ -43,7 +43,7 @@ struct GHash;
 struct ID;
 struct Image;
 struct FCurve;
-struct Group;
+struct Collection;
 struct Key;
 struct LayerCollection;
 struct Main;
@@ -158,10 +158,11 @@ struct DepsgraphNodeBuilder {
 	                                       int name_tag = -1);
 
 	void build_id(ID* id);
+	void build_layer_collections(ListBase *lb);
 	void build_view_layer(Scene *scene,
 	                       ViewLayer *view_layer,
 	                       eDepsNode_LinkedState_Type linked_state);
-	void build_group(Group *group);
+	void build_collection(Collection *collection);
 	void build_object(int base_index,
 	                  Object *object,
 	                  eDepsNode_LinkedState_Type linked_state);
diff --git a/source/blender/depsgraph/intern/builder/deg_builder_nodes_view_layer.cc b/source/blender/depsgraph/intern/builder/deg_builder_nodes_view_layer.cc
index ac3970ed3ab..a396aecd3a8 100644
--- a/source/blender/depsgraph/intern/builder/deg_builder_nodes_view_layer.cc
+++ b/source/blender/depsgraph/intern/builder/deg_builder_nodes_view_layer.cc
@@ -65,6 +65,14 @@ extern "C" {
 
 namespace DEG {
 
+void DepsgraphNodeBuilder::build_layer_collections(ListBase *lb)
+{
+	for (LayerCollection *lc = (LayerCollection *)lb->first; lc; lc = lc->next) {
+		build_collection(lc->collection);
+		build_layer_collections(&lc->layer_collections);
+	}
+}
+
 void DepsgraphNodeBuilder::build_view_layer(
         Scene *scene,
         ViewLayer *view_layer,
@@ -100,6 +108,7 @@ void DepsgraphNodeBuilder::build_view_layer(
 		base->object->select_color = select_color++;
 		++base_index;
 	}
+	build_layer_collections(&view_layer->layer_collections);
 	if (scene->camera != NULL) {
 		build_object(-1, scene->camera, DEG_ID_LINKED_INDIRECTLY);
 	}
@@ -140,7 +149,7 @@ void DepsgraphNodeBuilder::build_view_layer(
 	                   DEG_NODE_TYPE_LAYER_COLLECTIONS,
 	                   function_bind(BKE_layer_eval_view_layer_indexed,
 	                                 _1,
-	                                 &scene_cow->id,
+	                                 scene_cow,
 	                                 view_layer_index_),
 	                   DEG_OPCODE_VIEW_LAYER_EVAL);
 	/* Parameters evaluation for scene relations mainly. */
diff --git a/source/blender/depsgraph/intern/builder/deg_builder_relations.cc b/source/blender/depsgraph/intern/builder/deg_builder_relations.cc
index e1eae851ec6..edb2969fa63 100644
--- a/source/blender/depsgraph/intern/builder/deg_builder_relations.cc
+++ b/source/blender/depsgraph/intern/builder/deg_builder_relations.cc
@@ -72,12 +72,12 @@ extern "C" {
 #include "BKE_action.h"
 #include "BKE_armature.h"
 #include "BKE_animsys.h"
+#include "BKE_collection.h"
 #include "BKE_constraint.h"
 #include "BKE_curve.h"
 #include "BKE_effect.h"
 #include "BKE_collision.h"
 #include "BKE_fcurve.h"
-#include "BKE_group.h"
 #include "BKE_key.h"
 #include "BKE_library.h"
 #include "BKE_main.h"
@@ -296,14 +296,14 @@ void DepsgraphRelationBuilder::add_collision_relations(
         const OperationKey &key,
         Scene *scene,
         Object *object,
-        Group *group,
+        Collection *collection,
         bool dupli,
         const char *name)
 {
 	unsigned int numcollobj;
 	Object **collobjs = get_collisionobjects_ext(scene,
 	                                             object,
-	                                             group,
+	                                             collection,
 	                                             &numcollobj,
 	                                             eModifierType_Collision,
 	                                             dupli);
@@ -397,7 +397,7 @@ void DepsgraphRelationBuilder::build_id(ID *id)
 	}
 	switch (GS(id->name)) {
 		case ID_GR:
-			build_group(NULL, (Group *)id);
+			build_collection(NULL, (Collection *)id);
 			break;
 		case ID_OB:
 			build_object(NULL, (Object *)id);
@@ -425,19 +425,23 @@ void DepsgraphRelationBuilder::build_id(ID *id)
 	}
 }
 
-void DepsgraphRelationBuilder::build_group(Object *object, Group *group)
+void DepsgraphRelationBuilder::build_collection(Object *object, Collection *collection)
 {
-	const bool group_done = built_map_.checkIsBuiltAndTag(group);
+	const bool group_done = built_map_.checkIsBuiltAndTag(collection);
 	OperationKey object_local_transform_key(object != NULL ? &object->id : NULL,
 	                                        DEG_NODE_TYPE_TRANSFORM,
 	                                        DEG_OPCODE_TRANSFORM_LOCAL);
 	if (!group_done) {
-		LISTBASE_FOREACH (Base *, base, &group->view_layer->object_bases) {
-			build_object(NULL, base->object);
+		LISTBASE_FOREACH (CollectionObject *, cob, &collection->gobject) {
+			build_object(NULL, cob->ob);
+		}
+		LISTBASE_FOREACH (CollectionChild *, child, &collection->children) {
+			build_collection(NULL, child->collection);
 		}
 	}
 	if (object != NULL) {
-		LISTBASE_FOREACH (Base *, base, &group->view_layer->object_bases) {
+		const ListBase group_objects = BKE_collection_object_cache_get(collection);
+		LISTBASE_FOREACH (Base *, base, &group_objects) {
 			ComponentKey dupli_transform_key(&base->object->id, DEG_NODE_TYPE_TRANSFORM);
 			add_relation(dupli_transform_key, object_local_transform_key, "Dupligroup");
 		}
@@ -549,7 +553,7 @@ void DepsgraphRelationBuilder::build_object(Base *base, Object *object)
 	}
 	/* Object dupligroup. */
 	if (object->dup_group != NULL) {
-		build_group(object, object->dup_group);
+		build_collection(object, object->dup_group);
 	}
 }
 
@@ -1353,7 +1357,8 @@ void DepsgraphRelationBuilder::build_rigidbody(Scene *scene)
 
 	/* objects - simulation participants */
 	if (rbw->group) {
-		LISTBASE_FOREACH (Base *, base, &rbw->group->view_layer->object_bases) {
+		const ListBase group_objects = BKE_collection_object_cache_get(rbw->group);
+		LISTBASE_FOREACH (Base *, base, &group_objects) {
 			Object *object = base->object;
 			if (object == NULL || object->type != OB_MESH) {
 				continue;
@@ -1407,7 +1412,8 @@ void DepsgraphRelationBuilder::build_rigidbody(Scene *scene)
 
 	/* constraints */
 	if (rbw->constraints) {
-		LISTBASE_FOREACH (Base *, base, &rbw->constraints->view_layer->object_bases) {
+		const ListBase constraint_objects = BKE_collection_object_cache_get(rbw->constraints);
+		LISTBASE_FOREACH (Base *, base, &constraint_objects) {
 			Object *object = base->object;
 			if (object == NULL || !object->rigidbody_constraint) {
 				continue;
@@ -1529,8 +1535,8 @@ void DepsgraphRelationBuilder::build_particles(Object *object)
 				break;
 			case PART_DRAW_GR:
 				if (part->dup_group != NULL) {
-					build_group(NULL, part->dup_group);
-					LISTBASE_FOREACH (GroupObject *, go, &part->dup_group->gobject) {
+					build_collection(NULL, part->dup_group);
+					LISTBASE_FOREACH (CollectionObject *, go, &part->dup_group->gobject) {
 						build_particles_visualization_object(object,
 						                                     psys,
 						                                     go->ob);
diff --git a/source/blender/depsgraph/intern/builder/deg_builder_relations.h b/source/blender/depsgraph/intern/builder/deg_builder_relations.h
index 9e45d01fd79..fec30160622 100644
--- a/source/blender/depsgraph/intern/builder/deg_builder_relations.h
+++ b/source/blender/depsgraph/intern/builder/deg_builder_relations.h
@@ -55,7 +55,7 @@ struct ListBase;
 struct GHash;
 struct ID;
 struct FCurve;
-struct Group;
+struct Collection;
 struct Key;
 struct LayerCollection;
 struct Main;
@@ -195,8 +195,9 @@ struct DepsgraphRelationBuilder
 	                                       bool check_unique = false);
 
 	void build_id(ID* id);
+	void build_layer_collections(ListBase *lb);
 	void build_view_layer(Scene *scene, ViewLayer *view_layer);
-	void build_group(Object *object, Group *group);
+	void build_collection(Object *object, Collection *collection);
 	void build_object(Base *base, Object *object);
 	void build_object_flags(Base *base, Object *object);
 	void build_object_data(Object *object);
@@ -259,7 +260,7 @@ struct DepsgraphRelationBuilder
 	void add_collision_relations(const OperationKey &key,
 	                             Scene *scene,
 	                             Object *object,
-	                             Group *group,
+	                             Collection *collection,
 	                             bool dupli,
 	                             const char *name);
 	void add_forcefield_relations(const OperationKey &key,
diff --git a/source/blender/depsgraph/intern/builder/deg_builder_relations_view_layer.cc b/source/blender/depsgraph/intern/builder/deg_builder_relations_view_layer.cc
index 99295a733fc..f575be9f659 100644
--- a/source/blender/depsgraph/intern/builder/deg_builder_relations_view_layer.cc
+++ b/source/blender/depsgraph/intern/builder/deg_builder_relations_view_layer.cc
@@ -69,6 +69,14 @@ extern "C" {
 
 namespace DEG {
 
+void DepsgraphRelationBuilder::build_layer_collections(ListBase *lb)
+{
+	for (LayerCollection *lc = (LayerCollection *)lb->first; lc; lc = lc->next) {
+		build_collection(NULL, lc->collection);
+		build_layer_collections(&lc->layer_collections);
+	}
+}
+
 void DepsgraphRelationBuilder::build_view_layer(Scene *scene, ViewLayer *view_layer)
 {
 	/* Setup currently building context. */
@@ -81,6 +89,9 @@ void DepsgraphRelationBuilder::build_view_layer(Scene *scene, ViewLayer *view_la
 	LISTBASE_FOREACH (Base *, base, &view_layer->object_bases) {
 		build_object(base, base->object);
 	}
+
+	build_layer_collections(&view_layer->layer_collections);
+
 	if (scene->camera != NULL) {
 		build_object(NULL, scene->camera);
 	}
diff --git a/source/blender/depsgraph/intern/depsgraph.cc b/source/blender/depsgraph/intern/depsgraph.cc
index f25be922db9..4307ac94390 100644
--- a/source/blender/depsgraph/intern/depsgraph.cc
+++ b/source/blender/depsgraph/intern/depsgraph.cc
@@ -696,30 +696,26 @@ void DEG_debug_print_eval_parent_typed(struct Depsgraph *depsgraph,
                                        const char *function_name,
                                        const char *object_name,
                                        const void *object_address,
-                                       const char *object_type,
                                        const char *parent_comment,
                                        const char *parent_name,
-                                       const void *parent_address,
-                                       const char *parent_type)
+                                       const void *parent_address)
 {
 	if ((DEG_debug_flags_get(depsgraph) & G_DEBUG_DEPSGRAPH_EVAL) == 0) {
 		return;
 	}
 	fprintf(stdout,
-	        "%s%s on %s %s(%p)%s [%s] %s %s %s(%p)%s %s\n",
+	        "%s%s on %s %s(%p) [%s] %s %s %s(%p)%s\n",
 	        depsgraph_name_for_logging(depsgraph).c_str(),
 	        function_name,
 	        object_name,
 	        DEG::deg_color_for_pointer(object_address).c_str(),
 	        object_address,
-	        object_type,
 	        DEG::deg_color_end().c_str(),
 	        parent_comment,
 	        parent_name,
 	        DEG::deg_color_for_pointer(parent_address).c_str(),
 	        parent_address,
-	        DEG::deg_color_end().c_str(),
-	        parent_type);
+	        DEG::deg_color_end().c_str());
 	fflush(stdout);
 }
 
diff --git a/source/blender/depsgraph/intern/depsgraph_build.cc b/source/blender/depsgraph/intern/depsgraph_build.cc
index 18f1d6edbaf..84b9bad17c9 100644
--- a/source/blender/depsgraph/intern/depsgraph_build.cc
+++ b/source/blender/depsgraph/intern/depsgraph_build.cc
@@ -328,14 +328,14 @@ void DEG_relations_tag_update(Main *bmain)
 void DEG_add_collision_relations(DepsNodeHandle *handle,
                                  Scene *scene,
                                  Object *object,
-                                 Group *group,
+                                 Collection *collection,
                                  unsigned int modifier_type,
                                  DEG_CollobjFilterFunction fn,
                                  bool dupli,
                                  const char *name)
 {
 	unsigned int numcollobj;
-	Object **collobjs = get_collisionobjects_ext(scene, object, group, &numcollobj, modifier_type, dupli);
+	Object **collobjs = get_collisionobjects_ext(scene, object, collection, &numcollobj, modifier_type, dupli);
 
 	for (unsigned int i = 0; i < numcollobj; i++) {
 		Object *ob1 = collobjs[i];
diff --git a/source/blender/depsgraph/intern/depsgraph_intern.h b/source/blender/depsgraph/intern/depsgraph_intern.h
index c048a7c1b93..526cecde457 100644
--- a/source/blender/depsgraph/intern/depsgraph_intern.h
+++ b/source/blender/depsgraph/intern/depsgraph_intern.h
@@ -49,7 +49,7 @@ extern "C" {
 #include "DEG_depsgraph_debug.h"
 
 struct DEGEditorUpdateContext;
-struct Group;
+struct Collection;
 struct Main;
 struct Scene;