Dirty fix for memory corruption in the rigid body API.

Problem happens when removing a rigid body reference in a constraint,
and then jumping to the start frame right away. This will cause a full
rebuild of the rigid body world. However, the btRigidBodys are removed
before the constraints, and this leaves dangling pointers in the
btTypedConstraints, which causes corruption when deleting those
constraints later.

Fix for now is to explicitly delete constraints in advance when
rebuilding, while they still have valid btRigidBody pointers.

Ultimately the whole memory management and ownership of Bullet data
needs redesign. This is already happening in the particles_refactor
branch and could be ported to master separately:
https://developer.blender.org/diffusion/B/browse/particles_refactor/source/blender/blenkernel/intern/rigidbody.c

1 files changed, 21 insertions, 0 deletions

diff --git a/source/blender/blenkernel/intern/rigidbody.c b/source/blender/blenkernel/intern/rigidbody.c
index 8e5b67404ac..86128994092 100644
--- a/source/blender/blenkernel/intern/rigidbody.c
+++ b/source/blender/blenkernel/intern/rigidbody.c
@@ -1105,6 +1105,26 @@ static void rigidbody_update_simulation(Scene *scene, RigidBodyWorld *rbw, bool
 		BKE_rigidbody_validate_sim_world(scene, rbw, true);
 	rigidbody_update_sim_world(scene, rbw);
 
+	/* XXX TODO For rebuild: remove all constraints first.
+	 * Otherwise we can end up deleting objects that are still
+	 * referenced by constraints, corrupting bullet's internal list.
+	 * 
+	 * Memory management needs redesign here, this is just a dirty workaround.
+	 */
+	if (rebuild && rbw->constraints) {
+		for (go = rbw->constraints->gobject.first; go; go = go->next) {
+			Object *ob = go->ob;
+			if (ob) {
+				RigidBodyCon *rbc = ob->rigidbody_constraint;
+				if (rbc && rbc->physics_constraint) {
+					RB_dworld_remove_constraint(rbw->physics_world, rbc->physics_constraint);
+					RB_constraint_delete(rbc->physics_constraint);
+					rbc->physics_constraint = NULL;
+				}
+			}
+		}
+	}
+
 	/* update objects */
 	for (go = rbw->group->gobject.first; go; go = go->next) {
 		Object *ob = go->ob;
@@ -1150,6 +1170,7 @@ static void rigidbody_update_simulation(Scene *scene, RigidBodyWorld *rbw, bool
 			rigidbody_update_sim_ob(scene, rbw, ob, rbo);
 		}
 	}
+	
 	/* update constraints */
 	if (rbw->constraints == NULL) /* no constraints, move on */
 		return;