Limit Rotation: explicitly orthogonalize the matrix before processing.

Add a call to orthogonalize the matrix before processing for the
same reasons as D8915, and an early exit in case no limits are
enabled for a bit of extra efficiency.

Since the constraint goes through Euler decomposition, it would
in fact remove shear even before this change, but the resulting
rotation won't make much sense.

This change allows using the constraint without any enabled limits
purely for the purpose of efficiently removing shear.

Differential Revision: https://developer.blender.org/D9626

1 files changed, 10 insertions, 0 deletions

diff --git a/source/blender/blenkernel/intern/constraint.c b/source/blender/blenkernel/intern/constraint.c
index 766d0f5dd2e..826c79c3764 100644
--- a/source/blender/blenkernel/intern/constraint.c
+++ b/source/blender/blenkernel/intern/constraint.c
@@ -1635,6 +1635,16 @@ static void rotlimit_evaluate(bConstraint *con, bConstraintOb *cob, ListBase *UN
   float eul[3];
   float size[3];
 
+  /* This constraint is based on euler rotation math, which doesn't work well with shear.
+   * The Y axis is chosen as the main one because constraints are most commonly used on bones.
+   * This also allows using the constraint to simply remove shear. */
+  orthogonalize_m4_stable(cob->matrix, 1, false);
+
+  /* Only do the complex processing if some limits are actually enabled. */
+  if (!(data->flag & (LIMIT_XROT | LIMIT_YROT | LIMIT_ZROT))) {
+    return;
+  }
+
   /* Select the Euler rotation order, defaulting to the owner value. */
   short rot_order = cob->rotOrder;