Fix T38918: Boolean modifier crashes when using specific topology

There were loads of issues in the code still which are mow likely fixed:

- Hole resolver hook had memory leak -- it didn't free face with holes
  when triangulating it.

- Original edge mapping didn't work correct. old code related on the fact
  that loop order is not changing when constructing the MeshSet class, but
  in fact it does change.

  Currently used edge map for this because it was easiest way to do it now,
  but after the release we're to change it. Main reason is that face mapping
  is not correct as well (and it was never correct actually). So we'll need
  to construct Mesh structures by our own to be sure we're using correct
  original index mapping.

- Carve might produce faces with ears, which is forbidden in Blender.
  it wasn't an issue in old integration because triangulation will remove
  the ears. So for now simply added ears removing back as a hook.

  But actual reason of the ears is to be investigated really.

  This hook will only work for NGons, quads are assumed not be able to
  have ears. So this additional hook shouldn't slow down things much.

- Carve's hole resolver produces duplicated faces in some cases. Still not
  sure what is the reason of this. Code here is not so much straightforward,
  this is to be investigated later.

  For now solved the issue as own hole resolver which checks for duplicated
  faces after the hole resolving.

  The additional checks here will only run if the mesh actually have hole
  and wouldn't introduce slowdown for faces which doesn't have holes.

- Made it so if edge user triangulation gets a split (for example, in cases
  when this edge intersects with the second operand) it wouldn't be dissolved.

  This prevents cases of crappy topology after dissolving in several cases.

- Edge dissolver didn't check for whether edge is a non-manifold. We couldn't
  really dissolve open manifold edges.

  The bad thing about this is that mesh triangulation might produce non-manifold
  edges and they wouldn't be dissolved. Not worst case in the world, but would
  be nice to have it solved somehow.

- Exporting mesh form Carve to Blender might have produced duplicated edges
  in cases when several non-manifold faces shared the edge. This is also fixed
  now.

- Mesh triangulation might have produced duplicated faces, which is really bad.
  Fixed by keeping a track on which faces we've created and skipping adding new
  triangle if we already have one.

This all might introduce some slowdown, but we're too close to the release now,
so would rather have it slower but robust. After the release we might look into
ways to speed things up.

1 files changed, 131 insertions, 4 deletions

diff --git a/extern/carve/carve-util.h b/extern/carve/carve-util.h
index 07743de2d16..324c6ae5b47 100644
--- a/extern/carve/carve-util.h
+++ b/extern/carve/carve-util.h
@@ -31,9 +31,40 @@
 #include <carve/geom3d.hpp>
 #include <carve/interpolator.hpp>
 #include <carve/mesh.hpp>
+#include <carve/triangulator.hpp>
 
 #include "carve-capi.h"
 
+struct TriIdxCompare {
+	bool operator() (const carve::triangulate::tri_idx &left,
+	                 const carve::triangulate::tri_idx &right) {
+		if (left.a < right.a) {
+			return true;
+		}
+		else if (left.a > right.a) {
+			return false;
+		}
+
+		if (left.b < right.b) {
+			return true;
+		}
+		else if (left.b > right.b) {
+			return false;
+		}
+
+		if (left.c < right.c) {
+			return true;
+		}
+		else if (left.c > right.c) {
+			return false;
+		}
+
+		return false;
+	}
+};
+
+typedef std::set<carve::triangulate::tri_idx, TriIdxCompare> TrianglesStorage;
+
 void carve_getRescaleMinMax(const carve::mesh::MeshSet<3> *left,
                             const carve::mesh::MeshSet<3> *right,
                             carve::geom3d::Vector *min,
@@ -50,15 +81,12 @@ bool carve_checkPolyPlanarAndGetNormal(const std::vector<carve::geom3d::Vector>
 
 int carve_triangulatePoly(struct ImportMeshData *import_data,
                           CarveMeshImporter *mesh_importer,
-                          int poly_index,
-                          int start_loop_index,
                           const std::vector<carve::geom3d::Vector> &vertices,
                           const int verts_per_poly,
                           const int *verts_of_poly,
                           const carve::math::Matrix3 &axis_matrix,
                           std::vector<int> *face_indices,
-                          std::vector<int> *orig_loop_index_map,
-                          std::vector<int> *orig_poly_index_map);
+                          TrianglesStorage *triangles_storage);
 
 namespace carve {
 	namespace interpolate {
@@ -116,6 +144,105 @@ namespace carve {
 			}
 		};
 
+		template<typename attr_t>
+		class SimpleFaceEdgeAttr : public Interpolator {
+		public:
+			typedef std::pair<const meshset_t::face_t *, unsigned> key_t;
+
+		protected:
+			typedef std::pair<const meshset_t::vertex_t *, const meshset_t::vertex_t *> vpair_t;
+
+			struct key_hash {
+				size_t operator()(const key_t &v) const {
+					return size_t(v.first) ^ size_t(v.second);
+				}
+			};
+			struct vpair_hash {
+				size_t operator()(const vpair_t &v) const {
+					return size_t(v.first) ^ size_t(v.second);
+				}
+			};
+
+			typedef std::unordered_map<key_t, attr_t, key_hash> attrmap_t;
+			typedef std::unordered_map<vpair_t, key_t, vpair_hash> edgedivmap_t;
+
+			attrmap_t attrs;
+
+			struct Hook : public Interpolator::Hook {
+			public:
+				virtual unsigned hookBits() const {
+					return carve::csg::CSG::Hooks::PROCESS_OUTPUT_FACE_BIT;
+				}
+				Hook(Interpolator *_interpolator, const carve::csg::CSG &_csg) : Interpolator::Hook(_interpolator, _csg) {
+				}
+				virtual ~Hook() {
+				}
+			};
+
+			virtual Interpolator::Hook *makeHook(carve::csg::CSG &csg) {
+				return new Hook(this, csg);
+			}
+
+			virtual void processOutputFace(const carve::csg::CSG &csg,
+			                               std::vector<carve::mesh::MeshSet<3>::face_t *> &new_faces,
+			                               const meshset_t::face_t *orig_face,
+			                               bool flipped) {
+				edgedivmap_t undiv;
+
+				for (meshset_t::face_t::const_edge_iter_t e = orig_face->begin(); e != orig_face->end(); ++e) {
+					key_t k(orig_face, e.idx());
+					typename attrmap_t::const_iterator attr_i = attrs.find(k);
+					if (attr_i == attrs.end()) {
+						continue;
+					} else {
+						undiv[vpair_t(e->v1(), e->v2())] = k;
+					}
+				}
+
+				for (size_t fnum = 0; fnum < new_faces.size(); ++fnum) {
+					const carve::mesh::MeshSet<3>::face_t *new_face = new_faces[fnum];
+					for (meshset_t::face_t::const_edge_iter_t e = new_face->begin(); e != new_face->end(); ++e) {
+						key_t k(new_face, e.idx());
+
+						vpair_t vp;
+						if (!flipped) {
+							vp = vpair_t(e->v1(), e->v2());
+						} else {
+							vp = vpair_t(e->v2(), e->v1());
+						}
+						typename edgedivmap_t::const_iterator vp_i;
+						if ((vp_i = undiv.find(vp)) != undiv.end()) {
+							attrs[k] = attrs[vp_i->second];
+						}
+					}
+				}
+			}
+
+		public:
+
+			bool hasAttribute(const meshset_t::face_t *f, unsigned e) {
+				return attrs.find(std::make_pair(f, e)) != attrs.end();
+			}
+
+			attr_t getAttribute(const meshset_t::face_t *f, unsigned e, const attr_t &def = attr_t()) {
+				typename attrmap_t::const_iterator fv = attrs.find(std::make_pair(f, e));
+				if (fv != attrs.end()) {
+					return (*fv).second;
+				}
+				return def;
+			}
+
+			void setAttribute(const meshset_t::face_t *f, unsigned e, const attr_t &attr) {
+				attrs[std::make_pair(f, e)] = attr;
+			}
+
+			SimpleFaceEdgeAttr() : Interpolator() {
+			}
+
+			virtual ~SimpleFaceEdgeAttr() {
+			}
+		};
+
 	}  // namespace interpolate
 }  // namespace carve