Fix T39517,

Issue here is that "show diffuse" option does not respect its intended
purpose which is to be used only for masking.

There are a couple of caveats here:

Dyntopo and multires -always- have mask data enabled, and thus as soon
as one goes to dyntopo mode or adds a multires modifier he would get the
default grey color instead.

Matcaps would break when nodes asked for a diffuse material color (this
was broken before too). Solved by adding global material state for when
matcaps are enabled. Also matcaps don't always played well with VBOs
off.

Added a few more missing updates for mask operators to notify
show_diffuse property as changed. This was also needed on rebuilding
dyntopo pbvh.

Also make zero mask color duller again after artist feedback.

1 files changed, 16 insertions, 1 deletions

diff --git a/source/blender/blenkernel/intern/pbvh.c b/source/blender/blenkernel/intern/pbvh.c
index 40974d7d4f8..c595ee56633 100644
--- a/source/blender/blenkernel/intern/pbvh.c
+++ b/source/blender/blenkernel/intern/pbvh.c
@@ -1925,5 +1925,20 @@ void pbvh_vertex_iter_init(PBVH *bvh, PBVHNode *node,
 
 void pbvh_show_diffuse_color_set(PBVH *bvh, bool show_diffuse_color)
 {
-	bvh->show_diffuse_color = show_diffuse_color;
+	bool has_mask = false;
+
+	switch (bvh->type) {
+		case PBVH_GRIDS:
+			has_mask = (bvh->gridkey.has_mask != 0);
+			break;
+		case PBVH_FACES:
+			has_mask = (bvh->vdata && CustomData_get_layer(bvh->vdata,
+			                                CD_PAINT_MASK));
+			break;
+		case PBVH_BMESH:
+			has_mask = (bvh->bm && (CustomData_get_offset(&bvh->bm->vdata, CD_PAINT_MASK) != -1));
+			break;
+	}
+
+	bvh->show_diffuse_color = !has_mask || show_diffuse_color;
 }