Orange: more noodle updates!

**** NEW: Group Nodes

Node trees usually become messy and confusing quickly, so we need
not only a way to collapse Nodes into single 'groups', but also a
way to re-use that data to create libraries of effects.

This has been done by making a new Library data type, the NodeTree.
Everything that has been grouped is stored here, and available for
re-use, appending or linking. These NodeTrees are fully generic,
i.e. can store shader trees, composit trees, and so on. The 'type'
value as stored in the NodeTree will keep track of internal type
definitions and execute/drawing callbacks. Needless to say, re-using
shader trees in a composit tree is a bit useless, and will be
prevented in the browsing code. :)

So; any NodeTree can become a "Goup Node" inside in a NodeTree. This
Group Node then works just like any Node.
To prevent the current code to become too complex, I've disabled
the possibility to insert Groups inside of Groups. That might be
enabled later, but is a real nasty piece of code to get OK.

Since Group Nodes are a dynamic Node type, a lot of work has been
done to ensure Node definitions can be dynamic too, but still allow
to be stored in files, and allow to be verified for type-definition
changes on reloading. This system needs a little bit maturing still,
so the Python gurus should better wait a little bit! (Also for me to
write the definite API docs for it).

What works now:

- Press CTRL+G to create a new Group. The grouping code checks for
impossible selections (like an unselected node between selected nodes).
Everthing that's selected then gets removed from the current tree, and
inserted in a new NodeTree library data block. A Group Node then is
added which links to this new NodeTree.

- Press ALT+G to ungroup. This will not delete the NodeTree library
data, but just duplicate the Group into the current tree.

- Press TAB, or click on the NodeTree icon to edit Groups. Note that
NodeTrees are instances, so editing one Group will also change the
other users.
This also means that when removing nodes in a Group (or hiding sockets
or changing internal links) this is immediately corrected for all users
of this Group, also in other Materials.

- While editing Groups, only the internal Nodes can be edited. A single
click outside of the Group boundary will close this 'edit mode'.

What needs to be done:

- SHIFT+A menu in toolbox style, also including a list of Groups
- Enable the single-user button in the Group Node
- Displaying all (visible) internal group UI elements in the Node Panel
- Enable Library linking and prevent editing of Groups then.


**** NEW: Socket Visibility control

Node types will be generated with a lot of possible inputs or outputs,
and drawing all sockets all the time isn't very useful then.

A new option in the Node header ('plus' icon) allows to either hide all
unused sockets (first keypress) or to reveil them (when there are hidden
sockets, the icon displays black, otherwise it's blended).

Hidden sockets in Nodes also are not exported to a Group, so this way
you can control what options (in/outputs) exactly are available.

To be done:
- a way to hide individual sockets, like with a RMB click on it.

**** NEW: Nodes now render!

This is still quite primitive, more on a level to replace the (now
obsolete and disabled) Material Layers.

What needs to be done:

- make the "Geometry" node work properly, also for AA textures
- make the Texture Node work (does very little at the moment)
- give Material Nodes all inputs as needed (like Map-to Panel)
- find a way to export more data from a Material Node, like the
  shadow value, or light intensity only, etc

Very important also to separate from the Material Buttons the
"global" options, like "Ztransp" or "Wire" or "Halo". These can not
be set for each Material-Node individually.

Also note that the Preview Render (Buttons window) now renders a bit
differently. This was a horrid piece of antique code, using a totally
incompatible way of rendering. Target is to fully re-use internal
render code for previews.

OK... that's it mostly. Now test!

1 files changed, 627 insertions, 41 deletions

diff --git a/source/blender/blenkernel/intern/node.c b/source/blender/blenkernel/intern/node.c
index b267ebb2a11..ab4d755c868 100644
--- a/source/blender/blenkernel/intern/node.c
+++ b/source/blender/blenkernel/intern/node.c
@@ -32,8 +32,12 @@
 
 #include "DNA_ID.h"
 #include "DNA_node_types.h"
+#include "DNA_material_types.h"
 
 #include "BKE_blender.h"
+#include "BKE_global.h"
+#include "BKE_library.h"
+#include "BKE_main.h"
 #include "BKE_node.h"
 #include "BKE_texture.h"
 #include "BKE_utildefines.h"
@@ -43,16 +47,27 @@
 
 #include "MEM_guardedalloc.h"
 
-/* ************** Type stuff ********** */
+/* not very important, but the stack solver likes to know a maximum */
+#define MAX_SOCKET	64
 
-static bNodeType *nodeGetType(bNodeTree *ntree, int type)
+#pragma mark /* ************** Type stuff **********  */
+
+static bNodeType *node_get_type(bNodeTree *ntree, int type, bNodeTree *ngroup)
 {
-	bNodeType **typedefs= ntree->alltypes;
-	
-	while( *typedefs && (*typedefs)->type!=type)
-		typedefs++;
-	
-	return *typedefs;
+	if(type==NODE_GROUP) {
+		if(ngroup && GS(ngroup->id.name)==ID_NT) {
+			return ngroup->owntype;
+		}
+		return NULL;
+	}
+	else {
+		bNodeType **typedefs= ntree->alltypes;
+		
+		while( *typedefs && (*typedefs)->type!=type)
+			typedefs++;
+		
+		return *typedefs;
+	}
 }
 
 void ntreeInitTypes(bNodeTree *ntree)
@@ -68,7 +83,7 @@ void ntreeInitTypes(bNodeTree *ntree)
 	
 	for(node= ntree->nodes.first; node; node= next) {
 		next= node->next;
-		node->typeinfo= nodeGetType(ntree, node->type);
+		node->typeinfo= node_get_type(ntree, node->type, (bNodeTree *)node->id);
 		if(node->typeinfo==NULL) {
 			printf("Error: Node type %s doesn't exist anymore, removed\n", node->name);
 			nodeFreeNode(ntree, node);
@@ -88,6 +103,9 @@ static bNodeSocket *node_add_socket_type(ListBase *lb, bNodeSocketType *stype)
 	else sock->limit= stype->limit;
 	sock->type= stype->type;
 	
+	sock->to_index= stype->own_index;
+	sock->tosock= stype->internsock;
+	
 	sock->ns.vec[0]= stype->val1;
 	sock->ns.vec[1]= stype->val2;
 	sock->ns.vec[2]= stype->val3;
@@ -119,14 +137,19 @@ static bNodeSocket *verify_socket(ListBase *lb, bNodeSocketType *stype)
 	bNodeSocket *sock;
 	
 	for(sock= lb->first; sock; sock= sock->next) {
-		if(strncmp(sock->name, stype->name, NODE_MAXSTR)==0)
-			break;
+		/* both indices are zero for non-groups, otherwise it's a unique index */
+		if(sock->to_index==stype->own_index)
+			if(strncmp(sock->name, stype->name, NODE_MAXSTR)==0)
+				break;
 	}
 	if(sock) {
 		sock->type= stype->type;		/* in future, read this from tydefs! */
 		if(stype->limit==0) sock->limit= 0xFFF;
 		else sock->limit= stype->limit;
+		sock->tosock= stype->internsock;
+		
 		BLI_remlink(lb, sock);
+		
 		return sock;
 	}
 	else {
@@ -162,37 +185,482 @@ static void verify_socket_list(bNodeTree *ntree, ListBase *lb, bNodeSocketType *
 	}
 }
 
+void nodeVerifyType(bNodeTree *ntree, bNode *node)
+{
+	bNodeType *ntype= node->typeinfo;
+	
+	if(ntype) {
+		/* might add some other verify stuff here */
+		
+		verify_socket_list(ntree, &node->inputs, ntype->inputs);
+		verify_socket_list(ntree, &node->outputs, ntype->outputs);
+	}
+}
+
 void ntreeVerifyTypes(bNodeTree *ntree)
 {
 	bNode *node;
-	bNodeType *ntype;
 	
 	if((ntree->init & NTREE_TYPE_INIT)==0)
 		ntreeInitTypes(ntree);
 	
 	/* check inputs and outputs, and remove or insert them */
+	for(node= ntree->nodes.first; node; node= node->next)
+		nodeVerifyType(ntree, node);
+	
+}
+
+#pragma mark /* ************** Group stuff ********** */
+
+/* prototype */
+static void node_group_exec_func(void *data, bNode *node, bNodeStack **in, bNodeStack **out);
+
+bNodeType node_group_typeinfo= {
+	/* type code   */	NODE_GROUP,
+	/* name        */	"Group",
+	/* width+range */	120, 60, 200,
+	/* class+opts  */	NODE_CLASS_GROUP, NODE_OPTIONS,
+	/* input sock  */	NULL,
+	/* output sock */	NULL,
+	/* storage     */	"",
+	/* execfunc    */	node_group_exec_func,
+	
+};
+
+/* tag internal sockets */
+static void group_tag_internal_sockets(bNodeTree *ngroup)
+{
+	bNode *node;
+	bNodeSocket *sock;
+	bNodeLink *link;
+	
+	/* clear intern tag, but check already for hidden sockets */
+	for(node= ngroup->nodes.first; node; node= node->next) {
+		for(sock= node->inputs.first; sock; sock= sock->next)
+			sock->intern= sock->flag & SOCK_HIDDEN;
+		for(sock= node->outputs.first; sock; sock= sock->next)
+			sock->intern= sock->flag & SOCK_HIDDEN;
+	}
+	/* set tag */
+	for(link= ngroup->links.first; link; link= link->next) {
+		link->fromsock->intern= 1;
+		link->tosock->intern= 1;
+	}
+	
+	/* remove link pointer to external links (only happens on create group) */
+	for(node= ngroup->nodes.first; node; node= node->next) {
+		for(sock= node->inputs.first; sock; sock= sock->next)
+			if(sock->intern==0)
+				sock->link= NULL;
+	}
+
+	/* set all intern sockets to own_index zero, makes sure that later use won't mixup */
+	for(node= ngroup->nodes.first; node; node= node->next) {
+		for(sock= node->inputs.first; sock; sock= sock->next)
+			if(sock->intern)
+				sock->own_index= 0;
+		for(sock= node->outputs.first; sock; sock= sock->next)
+			if(sock->intern)
+				sock->own_index= 0;
+	}
+}
+
+/* after editing group, new sockets are zero */
+/* this routine ensures unique identifiers for zero sockets that are exposed */
+static void group_verify_own_indices(bNodeTree *ngroup)
+{
+	bNode *node;
+	bNodeSocket *sock;
+	
+	for(node= ngroup->nodes.first; node; node= node->next) {
+		for(sock= node->inputs.first; sock; sock= sock->next)
+			if(sock->own_index==0 && sock->intern==0)
+				sock->own_index= ++(ngroup->cur_index);
+		for(sock= node->outputs.first; sock; sock= sock->next)
+			if(sock->own_index==0 && sock->intern==0)
+				sock->own_index= ++(ngroup->cur_index);
+	}
+	printf("internal index %d\n", ngroup->cur_index);
+}
+
+
+/* nodetrees can be used as groups, so we need typeinfo structs generated */
+void ntreeMakeOwnType(bNodeTree *ngroup)
+{
+	bNode *node;
+	bNodeSocket *sock;
+	int totin= 0, totout=0, a;
+
+	/* tags socket when internal linked */
+	group_tag_internal_sockets(ngroup);
+	
+	/* ensure all sockets have own unique id */
+	group_verify_own_indices(ngroup);
+	
+	/* counting stats */
+	for(node= ngroup->nodes.first; node; node= node->next) {
+		if(node->type==NODE_GROUP)
+			break;
+		for(sock= node->inputs.first; sock; sock= sock->next)
+			if(sock->intern==0) 
+				totin++;
+		for(sock= node->outputs.first; sock; sock= sock->next)
+			if(sock->intern==0) 
+				totout++;
+	}
+	/* debug: nodetrees in nodetrees not handled yet */
+	if(node) {
+		printf("group in group, not supported yet\n");
+		return;
+	}
+	
+	/* free own type struct */
+	if(ngroup->owntype) {
+		if(ngroup->owntype->inputs)
+			MEM_freeN(ngroup->owntype->inputs);
+		if(ngroup->owntype->outputs)
+			MEM_freeN(ngroup->owntype->outputs);
+		MEM_freeN(ngroup->owntype);
+	}
+	
+	/* make own type struct */
+	ngroup->owntype= MEM_mallocN(sizeof(bNodeType), "group type");
+	*ngroup->owntype= node_group_typeinfo;
+	
+	/* input type arrays */
+	if(totin) {
+		bNodeSocketType *stype;
+		bNodeSocketType *inputs= MEM_mallocN(sizeof(bNodeSocketType)*(totin+1), "bNodeSocketType");
+		a= 0;
+		
+		for(node= ngroup->nodes.first; node; node= node->next) {
+			/* nodes are presumed fully verified, stype and socket list are in sync */
+			stype= node->typeinfo->inputs;
+			for(sock= node->inputs.first; sock; sock= sock->next, stype++) {
+				if(sock->intern==0) {
+					/* debug only print */
+					if(stype==NULL || stype->type==-1) printf("group verification error %s\n", ngroup->id.name);
+					
+					inputs[a]= *stype;
+					inputs[a].own_index= sock->own_index;
+					inputs[a].internsock= sock;	
+					a++;
+				}
+			}
+		}
+		inputs[a].type= -1;	/* terminator code */
+		ngroup->owntype->inputs= inputs;
+	}	
+	
+	/* output type arrays */
+	if(totout) {
+		bNodeSocketType *stype;
+		bNodeSocketType *outputs= MEM_mallocN(sizeof(bNodeSocketType)*(totout+1), "bNodeSocketType");
+		a= 0;
+		
+		for(node= ngroup->nodes.first; node; node= node->next) {
+			/* nodes are presumed fully verified, stype and socket list are in sync */
+			stype= node->typeinfo->outputs;
+			for(sock= node->outputs.first; sock; sock= sock->next, stype++) {
+				if(sock->intern==0) {
+					/* debug only print */
+					if(stype==NULL || stype->type==-1) printf("group verification error %s\n", ngroup->id.name);
+					
+					outputs[a]= *stype;
+					outputs[a].own_index= sock->own_index;
+					outputs[a].internsock= sock;	
+					a++;
+				}
+			}
+		}
+		outputs[a].type= -1;	/* terminator code */
+		ngroup->owntype->outputs= outputs;
+	}
+	
+	/* voila, the nodetree has the full definition for generating group-node instances! */
+}
+
+
+static bNodeSocket *groupnode_find_tosock(bNode *gnode, int index)
+{
+	bNodeSocket *sock;
+	
+	for(sock= gnode->inputs.first; sock; sock= sock->next)
+		if(sock->to_index==index)
+			return sock;
+	return NULL;
+}
+
+static bNodeSocket *groupnode_find_fromsock(bNode *gnode, int index)
+{
+	bNodeSocket *sock;
+	
+	for(sock= gnode->outputs.first; sock; sock= sock->next)
+		if(sock->to_index==index)
+			return sock;
+	return NULL;
+}
+
+bNode *nodeMakeGroupFromSelected(bNodeTree *ntree)
+{
+	bNodeLink *link, *linkn;
+	bNode *node, *gnode, *nextn;
+	bNodeTree *ngroup;
+	float min[2], max[2];
+	int totnode=0;
+	
+	INIT_MINMAX2(min, max);
+	
+	/* is there something to group? also do some clearing */
 	for(node= ntree->nodes.first; node; node= node->next) {
-		ntype= node->typeinfo;
-		if(ntype) {
-			/* might add some other verify stuff here */
-			
-			verify_socket_list(ntree, &node->inputs, ntype->inputs);
-			verify_socket_list(ntree, &node->outputs, ntype->outputs);
+		if(node->flag & NODE_SELECT) {
+			/* no groups in groups */
+			if(node->type==NODE_GROUP)
+				return NULL;
+			DO_MINMAX2( (&node->locx), min, max);
+			totnode++;
+		}
+		node->done= 0;
+	}
+	if(totnode==0) return NULL;
+	
+	/* check if all connections are OK, no unselected node has both
+		inputs and outputs to a selection */
+	for(link= ntree->links.first; link; link= link->next) {
+		if(link->fromnode->flag & NODE_SELECT)
+			link->tonode->done |= 1;
+		if(link->tonode->flag & NODE_SELECT)
+			link->fromnode->done |= 2;
+	}	
+	
+	for(node= ntree->nodes.first; node; node= node->next) {
+		if((node->flag & NODE_SELECT)==0)
+			if(node->done==3)
+				break;
+	}
+	if(node) 
+		return NULL;
+	
+	/* OK! new nodetree */
+	ngroup= alloc_libblock(&G.main->nodetree, ID_NT, "NodeGroup");
+	ngroup->type= ntree->type;
+	ngroup->alltypes= ntree->alltypes;
+	
+	/* move nodes over */
+	for(node= ntree->nodes.first; node; node= nextn) {
+		nextn= node->next;
+		if(node->flag & NODE_SELECT) {
+			BLI_remlink(&ntree->nodes, node);
+			BLI_addtail(&ngroup->nodes, node);
+			node->locx-= 0.5f*(min[0]+max[0]);
+			node->locy-= 0.5f*(min[1]+max[1]);
+		}
+	}
+
+	/* move links over */
+	for(link= ntree->links.first; link; link= linkn) {
+		linkn= link->next;
+		if(link->fromnode->flag & link->tonode->flag & NODE_SELECT) {
+			BLI_remlink(&ntree->links, link);
+			BLI_addtail(&ngroup->links, link);
 		}
 	}
+	
+	/* now we can make own group typeinfo */
+	ntreeMakeOwnType(ngroup);
+	
+	/* make group node */
+	gnode= nodeAddNodeType(ntree, NODE_GROUP, ngroup);
+	gnode->locx= 0.5f*(min[0]+max[0]);
+	gnode->locy= 0.5f*(min[1]+max[1]);
+	
+	/* relink external sockets */
+	for(link= ntree->links.first; link; link= link->next) {
+		if(link->tonode->flag & NODE_SELECT) {
+			link->tonode= gnode;
+			link->tosock= groupnode_find_tosock(gnode, link->tosock->own_index);
+			if(link->tosock==NULL) printf("Bad!\n");
+		}
+		else if(link->fromnode->flag & NODE_SELECT) {
+			link->fromnode= gnode;
+			link->fromsock= groupnode_find_fromsock(gnode, link->fromsock->own_index);
+			if(link->fromsock==NULL) printf("Bad!\n");
+		}
+	}
+	
+	return gnode;
 }
 
+/* note: ungroup: group_indices zero! */
 
+/* here's a nasty little one, need to check users... */
+/* should become callbackable... */
+void nodeVerifyGroup(bNodeTree *ngroup)
+{
+	Material *ma;
+	
+	/* group changed, so we rebuild the type definition */
+	ntreeMakeOwnType(ngroup);
+	
+	if(ngroup->type==NTREE_SHADER) {
+		for(ma= G.main->mat.first; ma; ma= ma->id.next) {
+			if(ma->nodetree) {
+				bNode *node;
+				
+				/* find if group is in tree */
+				for(node= ma->nodetree->nodes.first; node; node= node->next)
+					if(node->id == (ID *)ngroup)
+						break;
+				
+				if(node) {
+					/* set all type pointers OK */
+					ntreeInitTypes(ma->nodetree);
+					
+					for(node= ma->nodetree->nodes.first; node; node= node->next)
+						if(node->id == (ID *)ngroup)
+							nodeVerifyType(ma->nodetree, node);
+				}
+			}
+		}
+	}
+}
 
-/* ************** Add stuff ********** */
+/* also to check all users... */
+/* should become callbackable... */
+void nodeGroupSocketUseFlags(bNodeTree *ngroup)
+{
+	bNode *node;
+	bNodeSocket *sock;
+	Material *ma;
 
-/* not very important, but the stack solver likes to know a maximum */
-#define MAX_SOCKET	64
+	/* clear flags */
+	for(node= ngroup->nodes.first; node; node= node->next) {
+		for(sock= node->inputs.first; sock; sock= sock->next)
+			sock->flag &= ~SOCK_IN_USE;
+		for(sock= node->outputs.first; sock; sock= sock->next)
+			sock->flag &= ~SOCK_IN_USE;
+	}
+	
+	/* tag all thats in use */
+	if(ngroup->type==NTREE_SHADER) {
+		for(ma= G.main->mat.first; ma; ma= ma->id.next) {
+			if(ma->nodetree) {
+				for(node= ma->nodetree->nodes.first; node; node= node->next) {
+					if(node->id==(ID *)ngroup) {
+						for(sock= node->inputs.first; sock; sock= sock->next)
+							if(sock->link)
+								if(sock->tosock) 
+									sock->tosock->flag |= SOCK_IN_USE;
+						for(sock= node->outputs.first; sock; sock= sock->next)
+							if(nodeCountSocketLinks(ma->nodetree, sock))
+								if(sock->tosock) 
+									sock->tosock->flag |= SOCK_IN_USE;
+					}
+				}
+			}
+		}
+	}
+}
+
+static void find_node_with_socket(bNodeTree *ntree, bNodeSocket *sock, bNode **nodep, int *sockindex)
+{
+	bNode *node;
+	bNodeSocket *tsock;
+	int index;
+	
+	for(node= ntree->nodes.first; node; node= node->next) {
+		for(index=0, tsock= node->inputs.first; tsock; tsock= tsock->next, index++)
+			if(tsock==sock)
+				break;
+		if(tsock)
+			break;
+		for(index=0, tsock= node->outputs.first; tsock; tsock= tsock->next, index++)
+			if(tsock==sock)
+				break;
+		if(tsock)
+			break;
+	}
+	if(node) {
+		*nodep= node;
+		*sockindex= index;
+	}
+	else {
+		*nodep= NULL;
+	}
+}
 
-bNode *nodeAddNodeType(bNodeTree *ntree, int type)
+/* returns 1 if its OK */
+int nodeGroupUnGroup(bNodeTree *ntree, bNode *gnode)
+{
+	bNodeLink *link, *linkn;
+	bNode *node, *nextn;
+	bNodeTree *ngroup, *wgroup;
+	int index;
+	
+	ngroup= (bNodeTree *)gnode->id;
+	if(ngroup==NULL) return 0;
+	
+	/* clear new pointers, set in copytree */
+	for(node= ntree->nodes.first; node; node= node->next)
+		node->new= NULL;
+
+	wgroup= ntreeCopyTree(ngroup, 0);
+	
+	/* add the nodes into the ntree */
+	for(node= wgroup->nodes.first; node; node= nextn) {
+		nextn= node->next;
+		BLI_remlink(&wgroup->nodes, node);
+		BLI_addtail(&ntree->nodes, node);
+		node->locx+= gnode->locx;
+		node->locy+= gnode->locy;
+		node->flag |= NODE_SELECT;
+	}
+	/* and the internal links */
+	for(link= wgroup->links.first; link; link= linkn) {
+		linkn= link->next;
+		BLI_remlink(&wgroup->links, link);
+		BLI_addtail(&ntree->links, link);
+	}
+
+	/* restore links to and from the gnode */
+	for(link= ntree->links.first; link; link= link->next) {
+		if(link->tonode==gnode) {
+			/* link->tosock->tosock is on the node we look for */
+			find_node_with_socket(ngroup, link->tosock->tosock, &nextn, &index);
+			if(nextn==NULL) printf("wrong stuff!\n");
+			else if(nextn->new==NULL) printf("wrong stuff too!\n");
+			else {
+				link->tonode= nextn->new;
+				link->tosock= BLI_findlink(&link->tonode->inputs, index);
+			}
+		}
+		else if(link->fromnode==gnode) {
+			/* link->fromsock->tosock is on the node we look for */
+			find_node_with_socket(ngroup, link->fromsock->tosock, &nextn, &index);
+			if(nextn==NULL) printf("1 wrong stuff!\n");
+			else if(nextn->new==NULL) printf("1 wrong stuff too!\n");
+			else {
+				link->fromnode= nextn->new;
+				link->fromsock= BLI_findlink(&link->fromnode->outputs, index);
+			}
+		}
+	}
+	
+	/* remove the gnode & work tree */
+	ntreeFreeTree(wgroup);
+	MEM_freeN(wgroup);
+	
+	nodeFreeNode(ntree, gnode);
+	
+	return 1;
+}
+
+#pragma mark /* ************** Add stuff ********** */
+
+bNode *nodeAddNodeType(bNodeTree *ntree, int type, bNodeTree *ngroup)
 {
 	bNode *node;
-	bNodeType *ntype= nodeGetType(ntree, type);
+	bNodeType *ntype= node_get_type(ntree, type, ngroup);
 	bNodeSocketType *stype;
 	
 	node= MEM_callocN(sizeof(bNode), "new node");
@@ -205,6 +673,9 @@ bNode *nodeAddNodeType(bNodeTree *ntree, int type)
 	node->width= ntype->width;
 	node->miniwidth= 15.0f;		/* small value only, allows print of first chars */
 	
+	if(type==NODE_GROUP)
+		node->id= (ID *)ngroup;
+	
 	if(ntype->inputs) {
 		stype= ntype->inputs;
 		while(stype->type != -1) {
@@ -236,12 +707,19 @@ bNode *nodeAddNodeType(bNodeTree *ntree, int type)
 bNode *nodeCopyNode(struct bNodeTree *ntree, struct bNode *node)
 {
 	bNode *nnode= MEM_callocN(sizeof(bNode), "dupli node");
-	
+	bNodeSocket *sock;
+
 	*nnode= *node;
 	BLI_addtail(&ntree->nodes, nnode);
 	
 	duplicatelist(&nnode->inputs, &node->inputs);
+	for(sock= nnode->inputs.first; sock; sock= sock->next)
+		sock->own_index= 0;
+	
 	duplicatelist(&nnode->outputs, &node->outputs);
+	for(sock= nnode->outputs.first; sock; sock= sock->next)
+		sock->own_index= 0;
+	
 	if(nnode->id)
 		nnode->id->us++;
 	
@@ -286,7 +764,7 @@ bNodeTree *ntreeAddTree(int type)
 	return ntree;
 }
 
-/* ************** Free stuff ********** */
+#pragma mark /* ************** Free stuff ********** */
 
 /* goes over entire tree */
 static void node_unlink_node(bNodeTree *ntree, bNode *node)
@@ -340,6 +818,7 @@ void nodeFreeNode(bNodeTree *ntree, bNode *node)
 	MEM_freeN(node);
 }
 
+/* do not free ntree itself here, free_libblock calls this function too */
 void ntreeFreeTree(bNodeTree *ntree)
 {
 	bNode *node, *next;
@@ -350,7 +829,13 @@ void ntreeFreeTree(bNodeTree *ntree)
 	}
 	BLI_freelistN(&ntree->links);
 	
-	MEM_freeN(ntree);
+	if(ntree->owntype) {
+		if(ntree->owntype->inputs)
+			MEM_freeN(ntree->owntype->inputs);
+		if(ntree->owntype->outputs)
+			MEM_freeN(ntree->owntype->outputs);
+		MEM_freeN(ntree->owntype);
+	}
 }
 
 bNodeTree *ntreeCopyTree(bNodeTree *ntree, int internal_select)
@@ -404,10 +889,21 @@ bNodeTree *ntreeCopyTree(bNodeTree *ntree, int internal_select)
 			nlink->tosock= BLI_findlink(&link->tonode->new->inputs, a);
 		}
 	}
+	
+	/* own type definition for group usage */
+	if(internal_select==0) {
+		if(ntree->owntype) {
+			newtree->owntype= MEM_dupallocN(ntree->owntype);
+			if(ntree->owntype->inputs)
+				newtree->owntype->inputs= MEM_dupallocN(ntree->owntype->inputs);
+			if(ntree->owntype->outputs)
+				newtree->owntype->outputs= MEM_dupallocN(ntree->owntype->outputs);
+		}
+	}	
 	return newtree;
 }
 
-/* ************ find stuff *************** */
+#pragma mark /* ************ find stuff *************** */
 
 bNodeLink *nodeFindLink(bNodeTree *ntree, bNodeSocket *from, bNodeSocket *to)
 {
@@ -492,7 +988,7 @@ void nodeSetActive(bNodeTree *ntree, bNode *node)
 		node->flag |= NODE_ACTIVE_ID;
 }
 
-/* ************** dependency stuff *********** */
+#pragma mark /* ************** dependency stuff *********** */
 
 /* node is guaranteed to be not checked before */
 static int node_recurs_check(bNode *node, bNode ***nsort, int level)
@@ -592,7 +1088,7 @@ void ntreeSolveOrder(bNodeTree *ntree)
 	
 }
 
-/* *************** preview *********** */
+#pragma mark /* *************** preview *********** */
 
 /* if node->preview, then we assume the rect to exist */
 
@@ -612,6 +1108,7 @@ static void nodeInitPreview(bNode *node, int xsize, int ysize)
 	
 	if(node->preview==NULL) {
 		node->preview= MEM_callocN(sizeof(bNodePreview), "node preview");
+		printf("added preview %s\n", node->name);
 	}
 	if(node->preview->rect==NULL) {
 		node->preview->rect= MEM_callocN(4*xsize + xsize*ysize*sizeof(float)*4, "node preview rect");
@@ -620,6 +1117,16 @@ static void nodeInitPreview(bNode *node, int xsize, int ysize)
 	}
 }
 
+void ntreeInitPreview(bNodeTree *ntree, int xsize, int ysize)
+{
+	bNode *node;
+	
+	for(node= ntree->nodes.first; node; node= node->next) {
+		if(node->typeinfo->flag & NODE_PREVIEW)	/* hrms, check for closed nodes? */
+			nodeInitPreview(node, xsize, ysize);
+	}		
+}
+
 void nodeAddToPreview(bNode *node, float *col, int x, int y)
 {
 	bNodePreview *preview= node->preview;
@@ -637,27 +1144,95 @@ void nodeAddToPreview(bNode *node, float *col, int x, int y)
 
 
 
-/* ******************* executing ************* */
+#pragma mark /* ******************* executing ************* */
+
+/* see notes at ntreeBeginExecTree */
+static void group_node_get_stack(bNode *node, bNodeStack *stack, bNodeStack **in, bNodeStack **out, bNodeStack **gin, bNodeStack **gout)
+{
+	bNodeSocket *sock;
+	
+	/* build pointer stack */
+	for(sock= node->inputs.first; sock; sock= sock->next) {
+		if(sock->intern) {
+			/* yep, intern can have link or is hidden socket */
+			if(sock->link)
+				*(in++)= stack + sock->link->fromsock->stack_index;
+			else
+				*(in++)= &sock->ns;
+		}
+		else
+			*(in++)= gin[sock->stack_index_ext];
+	}
+	
+	for(sock= node->outputs.first; sock; sock= sock->next) {
+		if(sock->intern)
+			*(out++)= stack + sock->stack_index;
+		else
+			*(out++)= gout[sock->stack_index_ext];
+	}
+}
+
+static void node_group_exec_func(void *data, bNode *gnode, bNodeStack **in, bNodeStack **out)
+{
+	bNode *node;
+	bNodeTree *ntree= (bNodeTree *)gnode->id;
+	bNodeStack *nsin[MAX_SOCKET];	/* arbitrary... watch this */
+	bNodeStack *nsout[MAX_SOCKET];	/* arbitrary... watch this */
+	
+	if(ntree==NULL) return;
+	
+	if(ntree->init & NTREE_EXEC_INIT) {
+		for(node= ntree->nodes.first; node; node= node->next) {
+			if(node->typeinfo->execfunc) {
+				group_node_get_stack(node, ntree->stack, nsin, nsout, in, out);
+				node->typeinfo->execfunc(data, node, nsin, nsout);
+			}
+		}
+	}
+}
 
+/* stack indices make sure all nodes only write in allocated data, for making it thread safe */
+/* per tree (and per group) unique indices are created */
+/* the index_ext we need to be able to map from groups to the group-node own stack */
 
-void ntreeBeginExecTree(bNodeTree *ntree, int xsize, int ysize)
+void ntreeBeginExecTree(bNodeTree *ntree)
 {
 	bNode *node;
 	bNodeSocket *sock;
-	int index= 0;
+	int index= 0, index_in= 0, index_out= 0;
 	
 	if((ntree->init & NTREE_TYPE_INIT)==0)
 		ntreeInitTypes(ntree);
+	if(ntree->init & NTREE_EXEC_INIT)
+		return;
 	
 	/* create indices for stack, check preview */
 	for(node= ntree->nodes.first; node; node= node->next) {
+		
+		for(sock= node->inputs.first; sock; sock= sock->next) {
+			if(sock->intern==0)
+				sock->stack_index_ext= index_in++;
+		}
+		
 		for(sock= node->outputs.first; sock; sock= sock->next) {
 			sock->stack_index= index++;
+			if(sock->intern==0)
+				sock->stack_index_ext= index_out++;
 		}
 		
-		if(node->typeinfo->flag & NODE_PREVIEW)	/* hrms, check for closed nodes? */
-			nodeInitPreview(node, xsize, ysize);
-		
+		if(node->type==NODE_GROUP) {
+			if(node->id) {
+				
+				ntreeBeginExecTree((bNodeTree *)node->id);
+				
+				/* copy internal data from internal nodes to own input sockets */
+				for(sock= node->inputs.first; sock; sock= sock->next) {
+					if(sock->tosock) {
+						sock->ns= sock->tosock->ns;
+					}
+				}
+			}
+		}
 	}
 	if(index) {
 		bNodeStack *ns;
@@ -672,12 +1247,24 @@ void ntreeBeginExecTree(bNodeTree *ntree, int xsize, int ysize)
 
 void ntreeEndExecTree(bNodeTree *ntree)
 {
-	if(ntree->stack) {
-		MEM_freeN(ntree->stack);
-		ntree->stack= NULL;
-	}
+	bNode *node;
+	
+	if(ntree->init & NTREE_EXEC_INIT) {
+		if(ntree->stack) {
+			MEM_freeN(ntree->stack);
+			ntree->stack= NULL;
+		}
 
-	ntree->init &= ~NTREE_EXEC_INIT;
+		ntree->init &= ~NTREE_EXEC_INIT;
+		
+		for(node= ntree->nodes.first; node; node= node->next) {
+			if(node->type==NODE_GROUP) {
+				if(node->id) {
+					ntreeEndExecTree((bNodeTree *)node->id);
+				}
+			}
+		}
+	}
 }
 
 static void node_get_stack(bNode *node, bNodeStack *stack, bNodeStack **in, bNodeStack **out)
@@ -706,7 +1293,6 @@ void ntreeExecTree(bNodeTree *ntree)
 	
 	/* only when initialized */
 	if(ntree->init & NTREE_EXEC_INIT) {
-	
 		for(node= ntree->nodes.first; node; node= node->next) {
 			if(node->typeinfo->execfunc) {
 				node_get_stack(node, ntree->stack, nsin, nsout);