OBJ: further optimize, cleanup and harden the new C++ importer

Continued improvements to the new C++ based OBJ importer.

Performance: about 2x faster.
- Rungholt.obj (several meshes, 263MB file): Windows 12.7s -> 5.9s, Mac 7.7s -> 3.1s.
- Blender 3.0 splash (24k meshes, 2.4GB file): Windows 97.3s -> 53.6s, Mac 137.3s -> 80.0s.
- "Windows" is VS2022, AMD Ryzen 5950X (32 threads), "Mac" is Xcode/clang 13, M1Max (10 threads).
- Slightly reduced memory usage during import as well.

The performance gains are a combination of several things:
- Replacing `std::stof` / `std::stoi` with C++17 `from_chars`.
- Stop reading input file char-by-char using `std::getline`, and instead read in 64kb chunks, and parse from there (taking care of possibly handling lines split mid-way due to chunk boundaries).
- Removing abstractions for splitting a line by some char,
- Avoid tiny memory allocations: instead of storing a vector of polygon corners in each face, store all the corners in one big array, and per-face only store indices "where do corners start, and how many". Likewise, don't store full string names of material/group names for each face; only store indices into overall material/group names arrays.
- Stop always doing mesh validation, which is slow. Do it just like the Alembic importer does: only do validation if found some invalid faces during import, or if requested by the user via an import setting checkbox (which defaults to off).
- Stop doing "collection sync" for each object being added; instead do the collection sync right after creating all the objects.

Cleanup / Robustness:

This reworking of parser (see "removing abstractions" point above) means that all the functions that were in `parser_string_utils` file are gone, and replaced with different set of functions. However they are not OBJ specific, so as pointed out during review of the previous differential, they are now in `source/blender/io/common` library.

Added gtest coverage for said functions as well; something that was only indirectly covered by obj tests previously.

Rework of some bits of parsing made the parser actually better able to deal with invalid syntax. E.g. previously, if a face corner were a `/123` string, it would have incorrectly treated that as a vertex index (since it would get "hey that's one number" after splitting a string by a slash), instead of properly marking it as invalid syntax.

Added gtest coverage for .mtl parsing; something that was not covered by any tests at all previously.

Reviewed By: Howard Trickey
Differential Revision: https://developer.blender.org/D14586

2 files changed, 174 insertions, 1 deletions

diff --git a/source/blender/io/wavefront_obj/tests/obj_importer_tests.cc b/source/blender/io/wavefront_obj/tests/obj_importer_tests.cc
index 611e0cbb209..3d34fb6f9c6 100644
--- a/source/blender/io/wavefront_obj/tests/obj_importer_tests.cc
+++ b/source/blender/io/wavefront_obj/tests/obj_importer_tests.cc
@@ -60,7 +60,8 @@ class obj_importer_test : public BlendfileLoadingBaseTest {
 
     std::string obj_path = blender::tests::flags_test_asset_dir() + "/io_tests/obj/" + path;
     strncpy(params.filepath, obj_path.c_str(), FILE_MAX - 1);
-    importer_main(bfile->main, bfile->curscene, bfile->cur_view_layer, params);
+    const size_t read_buffer_size = 650;
+    importer_main(bfile->main, bfile->curscene, bfile->cur_view_layer, params, read_buffer_size);
 
     depsgraph_create(DAG_EVAL_VIEWPORT);
 
diff --git a/source/blender/io/wavefront_obj/tests/obj_mtl_parser_tests.cc b/source/blender/io/wavefront_obj/tests/obj_mtl_parser_tests.cc
new file mode 100644
index 00000000000..176d32a0be4
--- /dev/null
+++ b/source/blender/io/wavefront_obj/tests/obj_mtl_parser_tests.cc
@@ -0,0 +1,172 @@
+/* SPDX-License-Identifier: Apache-2.0 */
+
+#include <gtest/gtest.h>
+
+#include "testing/testing.h"
+
+#include "obj_import_file_reader.hh"
+
+namespace blender::io::obj {
+
+class obj_mtl_parser_test : public testing::Test {
+ public:
+  void check(const char *file, const MTLMaterial *expect, size_t expect_count)
+  {
+    std::string obj_dir = blender::tests::flags_test_asset_dir() + "/io_tests/obj/";
+    MTLParser parser(file, obj_dir + "dummy.obj");
+    Map<std::string, std::unique_ptr<MTLMaterial>> materials;
+    parser.parse_and_store(materials);
+
+    for (int i = 0; i < expect_count; ++i) {
+      const MTLMaterial &exp = expect[i];
+      if (!materials.contains(exp.name)) {
+        fprintf(stderr, "Material '%s' was expected in parsed result\n", exp.name.c_str());
+        ADD_FAILURE();
+        continue;
+      }
+      const MTLMaterial &got = *materials.lookup(exp.name);
+      const float tol = 0.0001f;
+      EXPECT_V3_NEAR(exp.Ka, got.Ka, tol);
+      EXPECT_V3_NEAR(exp.Kd, got.Kd, tol);
+      EXPECT_V3_NEAR(exp.Ks, got.Ks, tol);
+      EXPECT_V3_NEAR(exp.Ke, got.Ke, tol);
+      EXPECT_NEAR(exp.Ns, got.Ns, tol);
+      EXPECT_NEAR(exp.Ni, got.Ni, tol);
+      EXPECT_NEAR(exp.d, got.d, tol);
+      EXPECT_NEAR(exp.map_Bump_strength, got.map_Bump_strength, tol);
+      EXPECT_EQ(exp.illum, got.illum);
+      for (const auto &it : exp.texture_maps.items()) {
+        const tex_map_XX &exp_tex = it.value;
+        const tex_map_XX &got_tex = got.texture_maps.lookup(it.key);
+        EXPECT_STREQ(exp_tex.image_path.c_str(), got_tex.image_path.c_str());
+        EXPECT_V3_NEAR(exp_tex.translation, got_tex.translation, tol);
+        EXPECT_V3_NEAR(exp_tex.scale, got_tex.scale, tol);
+        EXPECT_EQ(exp_tex.projection_type, got_tex.projection_type);
+      }
+    }
+    EXPECT_EQ(materials.size(), expect_count);
+  }
+};
+
+TEST_F(obj_mtl_parser_test, cube)
+{
+  MTLMaterial mat;
+  mat.name = "red";
+  mat.Ka = {0.2f, 0.2f, 0.2f};
+  mat.Kd = {1, 0, 0};
+  check("cube.mtl", &mat, 1);
+}
+
+TEST_F(obj_mtl_parser_test, all_objects)
+{
+  MTLMaterial mat[7];
+  for (auto &m : mat) {
+    m.Ka = {1, 1, 1};
+    m.Ks = {0.5f, 0.5f, 0.5f};
+    m.Ke = {0, 0, 0};
+    m.Ns = 250;
+    m.Ni = 1;
+    m.d = 1;
+    m.illum = 2;
+  }
+  mat[0].name = "Blue";
+  mat[0].Kd = {0, 0, 1};
+  mat[1].name = "BlueDark";
+  mat[1].Kd = {0, 0, 0.5f};
+  mat[2].name = "Green";
+  mat[2].Kd = {0, 1, 0};
+  mat[3].name = "GreenDark";
+  mat[3].Kd = {0, 0.5f, 0};
+  mat[4].name = "Material";
+  mat[4].Kd = {0.8f, 0.8f, 0.8f};
+  mat[5].name = "Red";
+  mat[5].Kd = {1, 0, 0};
+  mat[6].name = "RedDark";
+  mat[6].Kd = {0.5f, 0, 0};
+  check("all_objects.mtl", mat, ARRAY_SIZE(mat));
+}
+
+TEST_F(obj_mtl_parser_test, materials)
+{
+  MTLMaterial mat[5];
+  mat[0].name = "no_textures_red";
+  mat[0].Ka = {0.3f, 0.3f, 0.3f};
+  mat[0].Kd = {0.8f, 0.3f, 0.1f};
+  mat[0].Ns = 5.624998f;
+
+  mat[1].name = "four_maps";
+  mat[1].Ka = {1, 1, 1};
+  mat[1].Kd = {0.8f, 0.8f, 0.8f};
+  mat[1].Ks = {0.5f, 0.5f, 0.5f};
+  mat[1].Ke = {0, 0, 0};
+  mat[1].Ns = 1000;
+  mat[1].Ni = 1.45f;
+  mat[1].d = 1;
+  mat[1].illum = 2;
+  mat[1].map_Bump_strength = 1;
+  {
+    tex_map_XX &kd = mat[1].tex_map_of_type(eMTLSyntaxElement::map_Kd);
+    kd.image_path = "texture.png";
+    tex_map_XX &ns = mat[1].tex_map_of_type(eMTLSyntaxElement::map_Ns);
+    ns.image_path = "sometexture_Roughness.png";
+    tex_map_XX &refl = mat[1].tex_map_of_type(eMTLSyntaxElement::map_refl);
+    refl.image_path = "sometexture_Metallic.png";
+    tex_map_XX &bump = mat[1].tex_map_of_type(eMTLSyntaxElement::map_Bump);
+    bump.image_path = "sometexture_Normal.png";
+  }
+
+  mat[2].name = "Clay";
+  mat[2].Ka = {1, 1, 1};
+  mat[2].Kd = {0.8f, 0.682657f, 0.536371f};
+  mat[2].Ks = {0.5f, 0.5f, 0.5f};
+  mat[2].Ke = {0, 0, 0};
+  mat[2].Ns = 440.924042f;
+  mat[2].Ni = 1.45f;
+  mat[2].d = 1;
+  mat[2].illum = 2;
+
+  mat[3].name = "Hat";
+  mat[3].Ka = {1, 1, 1};
+  mat[3].Kd = {0.8f, 0.8f, 0.8f};
+  mat[3].Ks = {0.5f, 0.5f, 0.5f};
+  mat[3].Ns = 800;
+  mat[3].map_Bump_strength = 0.5f;
+  {
+    tex_map_XX &kd = mat[3].tex_map_of_type(eMTLSyntaxElement::map_Kd);
+    kd.image_path = "someHatTexture_BaseColor.jpg";
+    tex_map_XX &ns = mat[3].tex_map_of_type(eMTLSyntaxElement::map_Ns);
+    ns.image_path = "someHatTexture_Roughness.jpg";
+    tex_map_XX &refl = mat[3].tex_map_of_type(eMTLSyntaxElement::map_refl);
+    refl.image_path = "someHatTexture_Metalness.jpg";
+    tex_map_XX &bump = mat[3].tex_map_of_type(eMTLSyntaxElement::map_Bump);
+    bump.image_path = "someHatTexture_Normal.jpg";
+  }
+
+  mat[4].name = "Parser_Test";
+  mat[4].Ka = {0.1f, 0.2f, 0.3f};
+  mat[4].Kd = {0.4f, 0.5f, 0.6f};
+  mat[4].Ks = {0.7f, 0.8f, 0.9f};
+  mat[4].illum = 6;
+  mat[4].Ns = 15.5;
+  mat[4].Ni = 1.5;
+  mat[4].d = 0.5;
+  mat[4].map_Bump_strength = 0.1f;
+  {
+    tex_map_XX &kd = mat[4].tex_map_of_type(eMTLSyntaxElement::map_Kd);
+    kd.image_path = "sometex_d.png";
+    tex_map_XX &ns = mat[4].tex_map_of_type(eMTLSyntaxElement::map_Ns);
+    ns.image_path = "sometex_ns.psd";
+    tex_map_XX &refl = mat[4].tex_map_of_type(eMTLSyntaxElement::map_refl);
+    refl.image_path = "clouds.tiff";
+    refl.scale = {1.5f, 2.5f, 3.5f};
+    refl.translation = {4.5f, 5.5f, 6.5f};
+    refl.projection_type = SHD_PROJ_SPHERE;
+    tex_map_XX &bump = mat[4].tex_map_of_type(eMTLSyntaxElement::map_Bump);
+    bump.image_path = "somebump.tga";
+    bump.scale = {3, 4, 5};
+  }
+
+  check("materials.mtl", mat, ARRAY_SIZE(mat));
+}
+
+}  // namespace blender::io::obj