OBJ: further optimize, cleanup and harden the new C++ importer

Continued improvements to the new C++ based OBJ importer.

Performance: about 2x faster.
- Rungholt.obj (several meshes, 263MB file): Windows 12.7s -> 5.9s, Mac 7.7s -> 3.1s.
- Blender 3.0 splash (24k meshes, 2.4GB file): Windows 97.3s -> 53.6s, Mac 137.3s -> 80.0s.
- "Windows" is VS2022, AMD Ryzen 5950X (32 threads), "Mac" is Xcode/clang 13, M1Max (10 threads).
- Slightly reduced memory usage during import as well.

The performance gains are a combination of several things:
- Replacing `std::stof` / `std::stoi` with C++17 `from_chars`.
- Stop reading input file char-by-char using `std::getline`, and instead read in 64kb chunks, and parse from there (taking care of possibly handling lines split mid-way due to chunk boundaries).
- Removing abstractions for splitting a line by some char,
- Avoid tiny memory allocations: instead of storing a vector of polygon corners in each face, store all the corners in one big array, and per-face only store indices "where do corners start, and how many". Likewise, don't store full string names of material/group names for each face; only store indices into overall material/group names arrays.
- Stop always doing mesh validation, which is slow. Do it just like the Alembic importer does: only do validation if found some invalid faces during import, or if requested by the user via an import setting checkbox (which defaults to off).
- Stop doing "collection sync" for each object being added; instead do the collection sync right after creating all the objects.

Cleanup / Robustness:

This reworking of parser (see "removing abstractions" point above) means that all the functions that were in `parser_string_utils` file are gone, and replaced with different set of functions. However they are not OBJ specific, so as pointed out during review of the previous differential, they are now in `source/blender/io/common` library.

Added gtest coverage for said functions as well; something that was only indirectly covered by obj tests previously.

Rework of some bits of parsing made the parser actually better able to deal with invalid syntax. E.g. previously, if a face corner were a `/123` string, it would have incorrectly treated that as a vertex index (since it would get "hey that's one number" after splitting a string by a slash), instead of properly marking it as invalid syntax.

Added gtest coverage for .mtl parsing; something that was not covered by any tests at all previously.

Reviewed By: Howard Trickey
Differential Revision: https://developer.blender.org/D14586

1 files changed, 5 insertions, 109 deletions

diff --git a/source/blender/io/wavefront_obj/importer/obj_import_file_reader.hh b/source/blender/io/wavefront_obj/importer/obj_import_file_reader.hh
index 24d026d75e5..8093417fcda 100644
--- a/source/blender/io/wavefront_obj/importer/obj_import_file_reader.hh
+++ b/source/blender/io/wavefront_obj/importer/obj_import_file_reader.hh
@@ -18,14 +18,16 @@ namespace blender::io::obj {
 class OBJParser {
  private:
   const OBJImportParams &import_params_;
-  blender::fstream obj_file_;
+  FILE *obj_file_;
   Vector<std::string> mtl_libraries_;
+  size_t read_buffer_size_;
 
  public:
   /**
    * Open OBJ file at the path given in import parameters.
    */
-  OBJParser(const OBJImportParams &import_params);
+  OBJParser(const OBJImportParams &import_params, size_t read_buffer_size);
+  ~OBJParser();
 
   /**
    * Read the OBJ file line by line and create OBJ Geometry instances. Also store all the vertex
@@ -39,111 +41,6 @@ class OBJParser {
   Span<std::string> mtl_libraries() const;
 };
 
-enum class eOBJLineKey {
-  V,
-  VN,
-  VT,
-  F,
-  L,
-  CSTYPE,
-  DEG,
-  CURV,
-  PARM,
-  O,
-  G,
-  S,
-  USEMTL,
-  MTLLIB,
-  COMMENT
-};
-
-constexpr eOBJLineKey line_key_str_to_enum(const std::string_view key_str)
-{
-  if (key_str == "v" || key_str == "V") {
-    return eOBJLineKey::V;
-  }
-  if (key_str == "vn" || key_str == "VN") {
-    return eOBJLineKey::VN;
-  }
-  if (key_str == "vt" || key_str == "VT") {
-    return eOBJLineKey::VT;
-  }
-  if (key_str == "f" || key_str == "F") {
-    return eOBJLineKey::F;
-  }
-  if (key_str == "l" || key_str == "L") {
-    return eOBJLineKey::L;
-  }
-  if (key_str == "cstype" || key_str == "CSTYPE") {
-    return eOBJLineKey::CSTYPE;
-  }
-  if (key_str == "deg" || key_str == "DEG") {
-    return eOBJLineKey::DEG;
-  }
-  if (key_str == "curv" || key_str == "CURV") {
-    return eOBJLineKey::CURV;
-  }
-  if (key_str == "parm" || key_str == "PARM") {
-    return eOBJLineKey::PARM;
-  }
-  if (key_str == "o" || key_str == "O") {
-    return eOBJLineKey::O;
-  }
-  if (key_str == "g" || key_str == "G") {
-    return eOBJLineKey::G;
-  }
-  if (key_str == "s" || key_str == "S") {
-    return eOBJLineKey::S;
-  }
-  if (key_str == "usemtl" || key_str == "USEMTL") {
-    return eOBJLineKey::USEMTL;
-  }
-  if (key_str == "mtllib" || key_str == "MTLLIB") {
-    return eOBJLineKey::MTLLIB;
-  }
-  if (key_str == "#") {
-    return eOBJLineKey::COMMENT;
-  }
-  return eOBJLineKey::COMMENT;
-}
-
-/**
- * All texture map options with number of arguments they accept.
- */
-class TextureMapOptions {
- private:
-  Map<const std::string, int> tex_map_options;
-
- public:
-  TextureMapOptions()
-  {
-    tex_map_options.add_new("-blendu", 1);
-    tex_map_options.add_new("-blendv", 1);
-    tex_map_options.add_new("-boost", 1);
-    tex_map_options.add_new("-mm", 2);
-    tex_map_options.add_new("-o", 3);
-    tex_map_options.add_new("-s", 3);
-    tex_map_options.add_new("-t", 3);
-    tex_map_options.add_new("-texres", 1);
-    tex_map_options.add_new("-clamp", 1);
-    tex_map_options.add_new("-bm", 1);
-    tex_map_options.add_new("-imfchan", 1);
-  }
-
-  /**
-   * All valid option strings.
-   */
-  Map<const std::string, int>::KeyIterator all_options() const
-  {
-    return tex_map_options.keys();
-  }
-
-  int number_of_args(StringRef option) const
-  {
-    return tex_map_options.lookup_as(std::string(option));
-  }
-};
-
 class MTLParser {
  private:
   char mtl_file_path_[FILE_MAX];
@@ -151,7 +48,6 @@ class MTLParser {
    * Directory in which the MTL file is found.
    */
   char mtl_dir_path_[FILE_MAX];
-  blender::fstream mtl_file_;
 
  public:
   /**
@@ -162,6 +58,6 @@ class MTLParser {
   /**
    * Read MTL file(s) and add MTLMaterial instances to the given Map reference.
    */
-  void parse_and_store(Map<std::string, std::unique_ptr<MTLMaterial>> &r_mtl_materials);
+  void parse_and_store(Map<std::string, std::unique_ptr<MTLMaterial>> &r_materials);
 };
 }  // namespace blender::io::obj