Fix T40271: recalculation of the bone roll does not work correctly.

Check that up_axis is not aligned with bone was wrong in at least two aspects
(not working against negative alignement case, and since ages it seems,
using Z axis when bones are along Y axis...).

Also optimized a bit here, better to have a normalized version of vec_roll_to_mat3(),
since it needs normalized vector anyway, and we have to normalize it for the tests
before calling it anyway (so now, we only do that twice in Transform code, instead
of three times).

And we can perform aling test *before* calling vec_roll_to_mat3!

1 files changed, 3 insertions, 8 deletions

diff --git a/source/blender/editors/armature/armature_edit.c b/source/blender/editors/armature/armature_edit.c
index fffdb2fe9c0..f71cfd52175 100644
--- a/source/blender/editors/armature/armature_edit.c
+++ b/source/blender/editors/armature/armature_edit.c
@@ -209,17 +209,12 @@ float ED_rollBoneToVector(EditBone *bone, const float align_axis[3], const bool
 
 	sub_v3_v3v3(nor, bone->tail, bone->head);
 
-	/* if tail == head! */
-	if (is_zero_v3(nor)) {
+	/* If tail == head or the bone is aligned with the axis... */
+	if (normalize_v3(nor) <= FLT_EPSILON || (fabsf(dot_v3v3(align_axis, nor)) >= (1.0f - FLT_EPSILON))) {
 		return roll;
 	}
 
-	vec_roll_to_mat3(nor, 0.0f, mat);
-
-	/* check the bone isn't aligned with the axis */
-	if (dot_v3v3(align_axis, mat[2]) >= (1.0f - FLT_EPSILON)) {
-		return roll;
-	}
+	vec_roll_to_mat3_normalized(nor, 0.0f, mat);
 
 	/* project the new_up_axis along the normal */
 	project_v3_v3v3(vec, align_axis, nor);