Fix weird Swing+Twist decomposition with noncanonical quaternions.

It turns out that after the fix to T83196 (rB814b2787cadd) the matrix
to quaternion conversion can produce noncanonical results in large
areas of the rotation space, when previously this was limited to
way smaller areas. This in turn causes Swing+Twist math to produce
angles beyond 180 degrees, e.g. outputting a -120..240 range.

This fixes both issues, ensuring that conversion outputs a canonical
result, and decomposition canonifies its input.

This was reported in chat by @jpbouza.

1 files changed, 16 insertions, 1 deletions

diff --git a/source/blender/blenlib/intern/math_rotation.c b/source/blender/blenlib/intern/math_rotation.c
index a0ee16bee76..19828e69638 100644
--- a/source/blender/blenlib/intern/math_rotation.c
+++ b/source/blender/blenlib/intern/math_rotation.c
@@ -372,6 +372,11 @@ void mat3_normalized_to_quat(float q[4], const float mat[3][3])
       q[1] = (mat[2][0] + mat[0][2]) * s;
       q[2] = (mat[2][1] + mat[1][2]) * s;
     }
+
+    /* Make sure w is nonnegative for a canonical result. */
+    if (q[0] < 0) {
+      negate_v4(q);
+    }
   }
 
   normalize_qt(q);
@@ -556,10 +561,20 @@ void rotation_between_quats_to_quat(float q[4], const float q1[4], const float q
  * \param r_twist: if not NULL, receives the twist quaternion.
  * \returns twist angle.
  */
-float quat_split_swing_and_twist(const float q[4], int axis, float r_swing[4], float r_twist[4])
+float quat_split_swing_and_twist(const float q_in[4], int axis, float r_swing[4], float r_twist[4])
 {
   BLI_assert(axis >= 0 && axis <= 2);
 
+  /* The calculation requires a canonical quaternion. */
+  float q[4];
+
+  if (q_in[0] < 0) {
+    negate_v4_v4(q, q_in);
+  }
+  else {
+    copy_v4_v4(q, q_in);
+  }
+
   /* Half-twist angle can be computed directly. */
   float t = atan2f(q[axis + 1], q[0]);