Mesh: concave quad support

Previously quads always split along first-third vertices.
This is still the default, to avoid flickering with animated deformation
however concave quads that would create two opposing triangles now use
second-fourth split.

Reported as T53999 although this issue has been known limitation
for a long time.

1 files changed, 12 insertions, 0 deletions

diff --git a/source/blender/blenlib/intern/math_geom.c b/source/blender/blenlib/intern/math_geom.c
index d3080e5530f..a3d850f9551 100644
--- a/source/blender/blenlib/intern/math_geom.c
+++ b/source/blender/blenlib/intern/math_geom.c
@@ -4903,6 +4903,18 @@ int is_quad_flip_v3(const float v1[3], const float v2[3], const float v3[3], con
 	return ret;
 }
 
+bool is_quad_flip_v3_first_third_fast(const float v1[3], const float v2[3], const float v3[3], const float v4[3])
+{
+	float d_12[3], d_13[3], d_14[3];
+	float cross_a[3], cross_b[3];
+	sub_v3_v3v3(d_12, v2, v1);
+	sub_v3_v3v3(d_13, v3, v1);
+	sub_v3_v3v3(d_14, v4, v1);
+	cross_v3_v3v3(cross_a, d_12, d_13);
+	cross_v3_v3v3(cross_b, d_14, d_13);
+	return dot_v3v3(cross_a, cross_b) > 0.0f;
+}
+
 /**
  * Return the value which the distance between points will need to be scaled by,
  * to define a handle, given both points are on a perfect circle.