diff options
| author | Aras Pranckevicius <aras@nesnausk.org> | 2022-04-17 22:07:43 +0300 |
|---|---|---|
| committer | Aras Pranckevicius <aras@nesnausk.org> | 2022-04-17 22:07:43 +0300 |
| commit | 213cd39b6db387bd88f12589fd50ff0e6563cf56 (patch) | |
| tree | d32f81c22469c10e97c6644c681379614f42e8d5 | |
| parent | a3eb4027c2383827b9f5beed709c54c53c7d6d20 (diff) | |
OBJ: further optimize, cleanup and harden the new C++ importer
Continued improvements to the new C++ based OBJ importer.
Performance: about 2x faster.
- Rungholt.obj (several meshes, 263MB file): Windows 12.7s -> 5.9s, Mac 7.7s -> 3.1s.
- Blender 3.0 splash (24k meshes, 2.4GB file): Windows 97.3s -> 53.6s, Mac 137.3s -> 80.0s.
- "Windows" is VS2022, AMD Ryzen 5950X (32 threads), "Mac" is Xcode/clang 13, M1Max (10 threads).
- Slightly reduced memory usage during import as well.
The performance gains are a combination of several things:
- Replacing `std::stof` / `std::stoi` with C++17 `from_chars`.
- Stop reading input file char-by-char using `std::getline`, and instead read in 64kb chunks, and parse from there (taking care of possibly handling lines split mid-way due to chunk boundaries).
- Removing abstractions for splitting a line by some char,
- Avoid tiny memory allocations: instead of storing a vector of polygon corners in each face, store all the corners in one big array, and per-face only store indices "where do corners start, and how many". Likewise, don't store full string names of material/group names for each face; only store indices into overall material/group names arrays.
- Stop always doing mesh validation, which is slow. Do it just like the Alembic importer does: only do validation if found some invalid faces during import, or if requested by the user via an import setting checkbox (which defaults to off).
- Stop doing "collection sync" for each object being added; instead do the collection sync right after creating all the objects.
Cleanup / Robustness:
This reworking of parser (see "removing abstractions" point above) means that all the functions that were in `parser_string_utils` file are gone, and replaced with different set of functions. However they are not OBJ specific, so as pointed out during review of the previous differential, they are now in `source/blender/io/common` library.
Added gtest coverage for said functions as well; something that was only indirectly covered by obj tests previously.
Rework of some bits of parsing made the parser actually better able to deal with invalid syntax. E.g. previously, if a face corner were a `/123` string, it would have incorrectly treated that as a vertex index (since it would get "hey that's one number" after splitting a string by a slash), instead of properly marking it as invalid syntax.
Added gtest coverage for .mtl parsing; something that was not covered by any tests at all previously.
Reviewed By: Howard Trickey
Differential Revision: https://developer.blender.org/D14586
24 files changed, 4251 insertions, 802 deletions
diff --git a/extern/fast_float/LICENSE-MIT b/extern/fast_float/LICENSE-MIT new file mode 100644 index 00000000000..2fb2a37ad7f --- /dev/null +++ b/extern/fast_float/LICENSE-MIT @@ -0,0 +1,27 @@ +MIT License + +Copyright (c) 2021 The fast_float authors + +Permission is hereby granted, free of charge, to any +person obtaining a copy of this software and associated +documentation files (the "Software"), to deal in the +Software without restriction, including without +limitation the rights to use, copy, modify, merge, +publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software +is furnished to do so, subject to the following +conditions: + +The above copyright notice and this permission notice +shall be included in all copies or substantial portions +of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF +ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED +TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A +PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT +SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR +IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +DEALINGS IN THE SOFTWARE. diff --git a/extern/fast_float/README.blender b/extern/fast_float/README.blender new file mode 100644 index 00000000000..a584a0511ee --- /dev/null +++ b/extern/fast_float/README.blender @@ -0,0 +1,7 @@ +Project: fast_float +URL: https://github.com/fastfloat/fast_float +License: MIT +Upstream version: 3.4.0 (b7f9d6c) +Local modifications: + +- Took only the fast_float.h header and the license/readme files diff --git a/extern/fast_float/README.md b/extern/fast_float/README.md new file mode 100644 index 00000000000..1e1c06d0a3e --- /dev/null +++ b/extern/fast_float/README.md @@ -0,0 +1,218 @@ +## fast_float number parsing library: 4x faster than strtod + +/badge.svg) +/badge.svg) + + + +[](https://github.com/fastfloat/fast_float/actions/workflows/vs16-ci.yml) + +The fast_float library provides fast header-only implementations for the C++ from_chars +functions for `float` and `double` types. These functions convert ASCII strings representing +decimal values (e.g., `1.3e10`) into binary types. We provide exact rounding (including +round to even). In our experience, these `fast_float` functions many times faster than comparable number-parsing functions from existing C++ standard libraries. + +Specifically, `fast_float` provides the following two functions with a C++17-like syntax (the library itself only requires C++11): + +```C++ +from_chars_result from_chars(const char* first, const char* last, float& value, ...); +from_chars_result from_chars(const char* first, const char* last, double& value, ...); +``` + +The return type (`from_chars_result`) is defined as the struct: +```C++ +struct from_chars_result { + const char* ptr; + std::errc ec; +}; +``` + +It parses the character sequence [first,last) for a number. It parses floating-point numbers expecting +a locale-independent format equivalent to the C++17 from_chars function. +The resulting floating-point value is the closest floating-point values (using either float or double), +using the "round to even" convention for values that would otherwise fall right in-between two values. +That is, we provide exact parsing according to the IEEE standard. + + +Given a successful parse, the pointer (`ptr`) in the returned value is set to point right after the +parsed number, and the `value` referenced is set to the parsed value. In case of error, the returned +`ec` contains a representative error, otherwise the default (`std::errc()`) value is stored. + +The implementation does not throw and does not allocate memory (e.g., with `new` or `malloc`). + +It will parse infinity and nan values. + +Example: + +``` C++ +#include "fast_float/fast_float.h" +#include <iostream> + +int main() { + const std::string input = "3.1416 xyz "; + double result; + auto answer = fast_float::from_chars(input.data(), input.data()+input.size(), result); + if(answer.ec != std::errc()) { std::cerr << "parsing failure\n"; return EXIT_FAILURE; } + std::cout << "parsed the number " << result << std::endl; + return EXIT_SUCCESS; +} +``` + + +Like the C++17 standard, the `fast_float::from_chars` functions take an optional last argument of +the type `fast_float::chars_format`. It is a bitset value: we check whether +`fmt & fast_float::chars_format::fixed` and `fmt & fast_float::chars_format::scientific` are set +to determine whether we allow the fixed point and scientific notation respectively. +The default is `fast_float::chars_format::general` which allows both `fixed` and `scientific`. + +The library seeks to follow the C++17 (see [20.19.3](http://eel.is/c++draft/charconv.from.chars).(7.1)) specification. +* The `from_chars` function does not skip leading white-space characters. +* [A leading `+` sign](https://en.cppreference.com/w/cpp/utility/from_chars) is forbidden. +* It is generally impossible to represent a decimal value exactly as binary floating-point number (`float` and `double` types). We seek the nearest value. We round to an even mantissa when we are in-between two binary floating-point numbers. + +Furthermore, we have the following restrictions: +* We only support `float` and `double` types at this time. +* We only support the decimal format: we do not support hexadecimal strings. +* For values that are either very large or very small (e.g., `1e9999`), we represent it using the infinity or negative infinity value. + +We support Visual Studio, macOS, Linux, freeBSD. We support big and little endian. We support 32-bit and 64-bit systems. + + + +## Using commas as decimal separator + + +The C++ standard stipulate that `from_chars` has to be locale-independent. In +particular, the decimal separator has to be the period (`.`). However, +some users still want to use the `fast_float` library with in a locale-dependent +manner. Using a separate function called `from_chars_advanced`, we allow the users +to pass a `parse_options` instance which contains a custom decimal separator (e.g., +the comma). You may use it as follows. + +```C++ +#include "fast_float/fast_float.h" +#include <iostream> + +int main() { + const std::string input = "3,1416 xyz "; + double result; + fast_float::parse_options options{fast_float::chars_format::general, ','}; + auto answer = fast_float::from_chars_advanced(input.data(), input.data()+input.size(), result, options); + if((answer.ec != std::errc()) || ((result != 3.1416))) { std::cerr << "parsing failure\n"; return EXIT_FAILURE; } + std::cout << "parsed the number " << result << std::endl; + return EXIT_SUCCESS; +} +``` + + +## Reference + +- Daniel Lemire, [Number Parsing at a Gigabyte per Second](https://arxiv.org/abs/2101.11408), Software: Pratice and Experience 51 (8), 2021. + +## Other programming languages + +- [There is an R binding](https://github.com/eddelbuettel/rcppfastfloat) called `rcppfastfloat`. +- [There is a Rust port of the fast_float library](https://github.com/aldanor/fast-float-rust/) called `fast-float-rust`. +- [There is a Java port of the fast_float library](https://github.com/wrandelshofer/FastDoubleParser) called `FastDoubleParser`. +- [There is a C# port of the fast_float library](https://github.com/CarlVerret/csFastFloat) called `csFastFloat`. + + +## Relation With Other Work + +The fastfloat algorithm is part of the [LLVM standard libraries](https://github.com/llvm/llvm-project/commit/87c016078ad72c46505461e4ff8bfa04819fe7ba). + +The fast_float library provides a performance similar to that of the [fast_double_parser](https://github.com/lemire/fast_double_parser) library but using an updated algorithm reworked from the ground up, and while offering an API more in line with the expectations of C++ programmers. The fast_double_parser library is part of the [Microsoft LightGBM machine-learning framework](https://github.com/microsoft/LightGBM). + +## Users + +The fast_float library is used by [Apache Arrow](https://github.com/apache/arrow/pull/8494) where it multiplied the number parsing speed by two or three times. It is also used by [Yandex ClickHouse](https://github.com/ClickHouse/ClickHouse) and by [Google Jsonnet](https://github.com/google/jsonnet). + + +## How fast is it? + +It can parse random floating-point numbers at a speed of 1 GB/s on some systems. We find that it is often twice as fast as the best available competitor, and many times faster than many standard-library implementations. + +<img src="http://lemire.me/blog/wp-content/uploads/2020/11/fastfloat_speed.png" width="400"> + +``` +$ ./build/benchmarks/benchmark +# parsing random integers in the range [0,1) +volume = 2.09808 MB +netlib : 271.18 MB/s (+/- 1.2 %) 12.93 Mfloat/s +doubleconversion : 225.35 MB/s (+/- 1.2 %) 10.74 Mfloat/s +strtod : 190.94 MB/s (+/- 1.6 %) 9.10 Mfloat/s +abseil : 430.45 MB/s (+/- 2.2 %) 20.52 Mfloat/s +fastfloat : 1042.38 MB/s (+/- 9.9 %) 49.68 Mfloat/s +``` + +See https://github.com/lemire/simple_fastfloat_benchmark for our benchmarking code. + + +## Video + +[](http://www.youtube.com/watch?v=AVXgvlMeIm4)<br /> + +## Using as a CMake dependency + +This library is header-only by design. The CMake file provides the `fast_float` target +which is merely a pointer to the `include` directory. + +If you drop the `fast_float` repository in your CMake project, you should be able to use +it in this manner: + +```cmake +add_subdirectory(fast_float) +target_link_libraries(myprogram PUBLIC fast_float) +``` + +Or you may want to retrieve the dependency automatically if you have a sufficiently recent version of CMake (3.11 or better at least): + +```cmake +FetchContent_Declare( + fast_float + GIT_REPOSITORY https://github.com/lemire/fast_float.git + GIT_TAG tags/v1.1.2 + GIT_SHALLOW TRUE) + +FetchContent_MakeAvailable(fast_float) +target_link_libraries(myprogram PUBLIC fast_float) + +``` + +You should change the `GIT_TAG` line so that you recover the version you wish to use. + +## Using as single header + +The script `script/amalgamate.py` may be used to generate a single header +version of the library if so desired. +Just run the script from the root directory of this repository. +You can customize the license type and output file if desired as described in +the command line help. + +You may directly download automatically generated single-header files: + +https://github.com/fastfloat/fast_float/releases/download/v1.1.2/fast_float.h + +## Credit + +Though this work is inspired by many different people, this work benefited especially from exchanges with +Michael Eisel, who motivated the original research with his key insights, and with Nigel Tao who provided +invaluable feedback. Rémy Oudompheng first implemented a fast path we use in the case of long digits. + +The library includes code adapted from Google Wuffs (written by Nigel Tao) which was originally published +under the Apache 2.0 license. + +## License + +<sup> +Licensed under either of <a href="LICENSE-APACHE">Apache License, Version +2.0</a> or <a href="LICENSE-MIT">MIT license</a> at your option. +</sup> + +<br> + +<sub> +Unless you explicitly state otherwise, any contribution intentionally submitted +for inclusion in this repository by you, as defined in the Apache-2.0 license, +shall be dual licensed as above, without any additional terms or conditions. +</sub> diff --git a/extern/fast_float/fast_float.h b/extern/fast_float/fast_float.h new file mode 100644 index 00000000000..2482dfdbd6c --- /dev/null +++ b/extern/fast_float/fast_float.h @@ -0,0 +1,2979 @@ +// fast_float by Daniel Lemire +// fast_float by João Paulo Magalhaes +// +// with contributions from Eugene Golushkov +// with contributions from Maksim Kita +// with contributions from Marcin Wojdyr +// with contributions from Neal Richardson +// with contributions from Tim Paine +// with contributions from Fabio Pellacini +// +// Licensed under the Apache License, Version 2.0, or the +// MIT License at your option. This file may not be copied, +// modified, or distributed except according to those terms. +// +// MIT License Notice +// +// MIT License +// +// Copyright (c) 2021 The fast_float authors +// +// Permission is hereby granted, free of charge, to any +// person obtaining a copy of this software and associated +// documentation files (the "Software"), to deal in the +// Software without restriction, including without +// limitation the rights to use, copy, modify, merge, +// publish, distribute, sublicense, and/or sell copies of +// the Software, and to permit persons to whom the Software +// is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice +// shall be included in all copies or substantial portions +// of the Software. +// +// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF +// ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED +// TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A +// PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT +// SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY +// CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +// OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR +// IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +// DEALINGS IN THE SOFTWARE. +// +// Apache License (Version 2.0) Notice +// +// Copyright 2021 The fast_float authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// + +#ifndef FASTFLOAT_FAST_FLOAT_H +#define FASTFLOAT_FAST_FLOAT_H + +#include <system_error> + +namespace fast_float { +enum chars_format { + scientific = 1<<0, + fixed = 1<<2, + hex = 1<<3, + general = fixed | scientific +}; + + +struct from_chars_result { + const char *ptr; + std::errc ec; +}; + +struct parse_options { + constexpr explicit parse_options(chars_format fmt = chars_format::general, + char dot = '.') + : format(fmt), decimal_point(dot) {} + + /** Which number formats are accepted */ + chars_format format; + /** The character used as decimal point */ + char decimal_point; +}; + +/** + * This function parses the character sequence [first,last) for a number. It parses floating-point numbers expecting + * a locale-indepent format equivalent to what is used by std::strtod in the default ("C") locale. + * The resulting floating-point value is the closest floating-point values (using either float or double), + * using the "round to even" convention for values that would otherwise fall right in-between two values. + * That is, we provide exact parsing according to the IEEE standard. + * + * Given a successful parse, the pointer (`ptr`) in the returned value is set to point right after the + * parsed number, and the `value` referenced is set to the parsed value. In case of error, the returned + * `ec` contains a representative error, otherwise the default (`std::errc()`) value is stored. + * + * The implementation does not throw and does not allocate memory (e.g., with `new` or `malloc`). + * + * Like the C++17 standard, the `fast_float::from_chars` functions take an optional last argument of + * the type `fast_float::chars_format`. It is a bitset value: we check whether + * `fmt & fast_float::chars_format::fixed` and `fmt & fast_float::chars_format::scientific` are set + * to determine whether we allowe the fixed point and scientific notation respectively. + * The default is `fast_float::chars_format::general` which allows both `fixed` and `scientific`. + */ +template<typename T> +from_chars_result from_chars(const char *first, const char *last, + T &value, chars_format fmt = chars_format::general) noexcept; + +/** + * Like from_chars, but accepts an `options` argument to govern number parsing. + */ +template<typename T> +from_chars_result from_chars_advanced(const char *first, const char *last, + T &value, parse_options options) noexcept; + +} +#endif // FASTFLOAT_FAST_FLOAT_H + +#ifndef FASTFLOAT_FLOAT_COMMON_H +#define FASTFLOAT_FLOAT_COMMON_H + +#include <cfloat> +#include <cstdint> +#include <cassert> +#include <cstring> +#include <type_traits> + +#if (defined(__x86_64) || defined(__x86_64__) || defined(_M_X64) \ + || defined(__amd64) || defined(__aarch64__) || defined(_M_ARM64) \ + || defined(__MINGW64__) \ + || defined(__s390x__) \ + || (defined(__ppc64__) || defined(__PPC64__) || defined(__ppc64le__) || defined(__PPC64LE__)) \ + || defined(__EMSCRIPTEN__)) +#define FASTFLOAT_64BIT +#elif (defined(__i386) || defined(__i386__) || defined(_M_IX86) \ + || defined(__arm__) || defined(_M_ARM) \ + || defined(__MINGW32__)) +#define FASTFLOAT_32BIT +#else + // Need to check incrementally, since SIZE_MAX is a size_t, avoid overflow. + // We can never tell the register width, but the SIZE_MAX is a good approximation. + // UINTPTR_MAX and INTPTR_MAX are optional, so avoid them for max portability. + #if SIZE_MAX == 0xffff + #error Unknown platform (16-bit, unsupported) + #elif SIZE_MAX == 0xffffffff + #define FASTFLOAT_32BIT + #elif SIZE_MAX == 0xffffffffffffffff + #define FASTFLOAT_64BIT + #else + #error Unknown platform (not 32-bit, not 64-bit?) + #endif +#endif + +#if ((defined(_WIN32) || defined(_WIN64)) && !defined(__clang__)) +#include <intrin.h> +#endif + +#if defined(_MSC_VER) && !defined(__clang__) +#define FASTFLOAT_VISUAL_STUDIO 1 +#endif + +#ifdef _WIN32 +#define FASTFLOAT_IS_BIG_ENDIAN 0 +#else +#if defined(__APPLE__) || defined(__FreeBSD__) +#include <machine/endian.h> +#elif defined(sun) || defined(__sun) +#include <sys/byteorder.h> +#else +#include <endian.h> +#endif +# +#ifndef __BYTE_ORDER__ +// safe choice +#define FASTFLOAT_IS_BIG_ENDIAN 0 +#endif +# +#ifndef __ORDER_LITTLE_ENDIAN__ +// safe choice +#define FASTFLOAT_IS_BIG_ENDIAN 0 +#endif +# +#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__ +#define FASTFLOAT_IS_BIG_ENDIAN 0 +#else +#define FASTFLOAT_IS_BIG_ENDIAN 1 +#endif +#endif + +#ifdef FASTFLOAT_VISUAL_STUDIO +#define fastfloat_really_inline __forceinline +#else +#define fastfloat_really_inline inline __attribute__((always_inline)) +#endif + +#ifndef FASTFLOAT_ASSERT +#define FASTFLOAT_ASSERT(x) { if (!(x)) abort(); } +#endif + +#ifndef FASTFLOAT_DEBUG_ASSERT +#include <cassert> +#define FASTFLOAT_DEBUG_ASSERT(x) assert(x) +#endif + +// rust style `try!()` macro, or `?` operator +#define FASTFLOAT_TRY(x) { if (!(x)) return false; } + +namespace fast_float { + +// Compares two ASCII strings in a case insensitive manner. +inline bool fastfloat_strncasecmp(const char *input1, const char *input2, + size_t length) { + char running_diff{0}; + for (size_t i = 0; i < length; i++) { + running_diff |= (input1[i] ^ input2[i]); + } + return (running_diff == 0) || (running_diff == 32); +} + +#ifndef FLT_EVAL_METHOD +#error "FLT_EVAL_METHOD should be defined, please include cfloat." +#endif + +// a pointer and a length to a contiguous block of memory +template <typename T> +struct span { + const T* ptr; + size_t length; + span(const T* _ptr, size_t _length) : ptr(_ptr), length(_length) {} + span() : ptr(nullptr), length(0) {} + + constexpr size_t len() const noexcept { + return length; + } + + const T& operator[](size_t index) const noexcept { + FASTFLOAT_DEBUG_ASSERT(index < length); + return ptr[index]; + } +}; + +struct value128 { + uint64_t low; + uint64_t high; + value128(uint64_t _low, uint64_t _high) : low(_low), high(_high) {} + value128() : low(0), high(0) {} +}; + +/* result might be undefined when input_num is zero */ +fastfloat_really_inline int leading_zeroes(uint64_t input_num) { + assert(input_num > 0); +#ifdef FASTFLOAT_VISUAL_STUDIO + #if defined(_M_X64) || defined(_M_ARM64) + unsigned long leading_zero = 0; + // Search the mask data from most significant bit (MSB) + // to least significant bit (LSB) for a set bit (1). + _BitScanReverse64(&leading_zero, input_num); + return (int)(63 - leading_zero); + #else + int last_bit = 0; + if(input_num & uint64_t(0xffffffff00000000)) input_num >>= 32, last_bit |= 32; + if(input_num & uint64_t( 0xffff0000)) input_num >>= 16, last_bit |= 16; + if(input_num & uint64_t( 0xff00)) input_num >>= 8, last_bit |= 8; + if(input_num & uint64_t( 0xf0)) input_num >>= 4, last_bit |= 4; + if(input_num & uint64_t( 0xc)) input_num >>= 2, last_bit |= 2; + if(input_num & uint64_t( 0x2)) input_num >>= 1, last_bit |= 1; + return 63 - last_bit; + #endif +#else + return __builtin_clzll(input_num); +#endif +} + +#ifdef FASTFLOAT_32BIT + +// slow emulation routine for 32-bit +fastfloat_really_inline uint64_t emulu(uint32_t x, uint32_t y) { + return x * (uint64_t)y; +} + +// slow emulation routine for 32-bit +#if !defined(__MINGW64__) +fastfloat_really_inline uint64_t _umul128(uint64_t ab, uint64_t cd, + uint64_t *hi) { + uint64_t ad = emulu((uint32_t)(ab >> 32), (uint32_t)cd); + uint64_t bd = emulu((uint32_t)ab, (uint32_t)cd); + uint64_t adbc = ad + emulu((uint32_t)ab, (uint32_t)(cd >> 32)); + uint64_t adbc_carry = !!(adbc < ad); + uint64_t lo = bd + (adbc << 32); + *hi = emulu((uint32_t)(ab >> 32), (uint32_t)(cd >> 32)) + (adbc >> 32) + + (adbc_carry << 32) + !!(lo < bd); + return lo; +} +#endif // !__MINGW64__ + +#endif // FASTFLOAT_32BIT + + +// compute 64-bit a*b +fastfloat_really_inline value128 full_multiplication(uint64_t a, + uint64_t b) { + value128 answer; +#ifdef _M_ARM64 + // ARM64 has native support for 64-bit multiplications, no need to emulate + answer.high = __umulh(a, b); + answer.low = a * b; +#elif defined(FASTFLOAT_32BIT) || (defined(_WIN64) && !defined(__clang__)) + answer.low = _umul128(a, b, &answer.high); // _umul128 not available on ARM64 +#elif defined(FASTFLOAT_64BIT) + __uint128_t r = ((__uint128_t)a) * b; + answer.low = uint64_t(r); + answer.high = uint64_t(r >> 64); +#else + #error Not implemented +#endif + return answer; +} + +struct adjusted_mantissa { + uint64_t mantissa{0}; + int32_t power2{0}; // a negative value indicates an invalid result + adjusted_mantissa() = default; + bool operator==(const adjusted_mantissa &o) const { + return mantissa == o.mantissa && power2 == o.power2; + } + bool operator!=(const adjusted_mantissa &o) const { + return mantissa != o.mantissa || power2 != o.power2; + } +}; + +// Bias so we can get the real exponent with an invalid adjusted_mantissa. +constexpr static int32_t invalid_am_bias = -0x8000; + +constexpr static double powers_of_ten_double[] = { + 1e0, 1e1, 1e2, 1e3, 1e4, 1e5, 1e6, 1e7, 1e8, 1e9, 1e10, 1e11, + 1e12, 1e13, 1e14, 1e15, 1e16, 1e17, 1e18, 1e19, 1e20, 1e21, 1e22}; +constexpr static float powers_of_ten_float[] = {1e0, 1e1, 1e2, 1e3, 1e4, 1e5, + 1e6, 1e7, 1e8, 1e9, 1e10}; + +template <typename T> struct binary_format { + using equiv_uint = typename std::conditional<sizeof(T) == 4, uint32_t, uint64_t>::type; + + static inline constexpr int mantissa_explicit_bits(); + static inline constexpr int minimum_exponent(); + static inline constexpr int infinite_power(); + static inline constexpr int sign_index(); + static inline constexpr int min_exponent_fast_path(); + static inline constexpr int max_exponent_fast_path(); + static inline constexpr int max_exponent_round_to_even(); + static inline constexpr int min_exponent_round_to_even(); + static inline constexpr uint64_t max_mantissa_fast_path(); + static inline constexpr int largest_power_of_ten(); + static inline constexpr int smallest_power_of_ten(); + static inline constexpr T exact_power_of_ten(int64_t power); + static inline constexpr size_t max_digits(); + static inline constexpr equiv_uint exponent_mask(); + static inline constexpr equiv_uint mantissa_mask(); + static inline constexpr equiv_uint hidden_bit_mask(); +}; + +template <> inline constexpr int binary_format<double>::mantissa_explicit_bits() { + return 52; +} +template <> inline constexpr int binary_format<float>::mantissa_explicit_bits() { + return 23; +} + +template <> inline constexpr int binary_format<double>::max_exponent_round_to_even() { + return 23; +} + +template <> inline constexpr int binary_format<float>::max_exponent_round_to_even() { + return 10; +} + +template <> inline constexpr int binary_format<double>::min_exponent_round_to_even() { + return -4; +} + +template <> inline constexpr int binary_format<float>::min_exponent_round_to_even() { + return -17; +} + +template <> inline constexpr int binary_format<double>::minimum_exponent() { + return -1023; +} +template <> inline constexpr int binary_format<float>::minimum_exponent() { + return -127; +} + +template <> inline constexpr int binary_format<double>::infinite_power() { + return 0x7FF; +} +template <> inline constexpr int binary_format<float>::infinite_power() { + return 0xFF; +} + +template <> inline constexpr int binary_format<double>::sign_index() { return 63; } +template <> inline constexpr int binary_format<float>::sign_index() { return 31; } + +template <> inline constexpr int binary_format<double>::min_exponent_fast_path() { +#if (FLT_EVAL_METHOD != 1) && (FLT_EVAL_METHOD != 0) + return 0; +#else + return -22; +#endif +} +template <> inline constexpr int binary_format<float>::min_exponent_fast_path() { +#if (FLT_EVAL_METHOD != 1) && (FLT_EVAL_METHOD != 0) + return 0; +#else + return -10; +#endif +} + +template <> inline constexpr int binary_format<double>::max_exponent_fast_path() { + return 22; +} +template <> inline constexpr int binary_format<float>::max_exponent_fast_path() { + return 10; +} + +template <> inline constexpr uint64_t binary_format<double>::max_mantissa_fast_path() { + return uint64_t(2) << mantissa_explicit_bits(); +} +template <> inline constexpr uint64_t binary_format<float>::max_mantissa_fast_path() { + return uint64_t(2) << mantissa_explicit_bits(); +} + +template <> +inline constexpr double binary_format<double>::exact_power_of_ten(int64_t power) { + return powers_of_ten_double[power]; +} +template <> +inline constexpr float binary_format<float>::exact_power_of_ten(int64_t power) { + + return powers_of_ten_float[power]; +} + + +template <> +inline constexpr int binary_format<double>::largest_power_of_ten() { + return 308; +} +template <> +inline constexpr int binary_format<float>::largest_power_of_ten() { + return 38; +} + +template <> +inline constexpr int binary_format<double>::smallest_power_of_ten() { + return -342; +} +template <> +inline constexpr int binary_format<float>::smallest_power_of_ten() { + return -65; +} + +template <> inline constexpr size_t binary_format<double>::max_digits() { + return 769; +} +template <> inline constexpr size_t binary_format<float>::max_digits() { + return 114; +} + +template <> inline constexpr binary_format<float>::equiv_uint + binary_format<float>::exponent_mask() { + return 0x7F800000; +} +template <> inline constexpr binary_format<double>::equiv_uint + binary_format<double>::exponent_mask() { + return 0x7FF0000000000000; +} + +template <> inline constexpr binary_format<float>::equiv_uint + binary_format<float>::mantissa_mask() { + return 0x007FFFFF; +} +template <> inline constexpr binary_format<double>::equiv_uint + binary_format<double>::mantissa_mask() { + return 0x000FFFFFFFFFFFFF; +} + +template <> inline constexpr binary_format<float>::equiv_uint + binary_format<float>::hidden_bit_mask() { + return 0x00800000; +} +template <> inline constexpr binary_format<double>::equiv_uint + binary_format<double>::hidden_bit_mask() { + return 0x0010000000000000; +} + +template<typename T> +fastfloat_really_inline void to_float(bool negative, adjusted_mantissa am, T &value) { + uint64_t word = am.mantissa; + word |= uint64_t(am.power2) << binary_format<T>::mantissa_explicit_bits(); + word = negative + ? word | (uint64_t(1) << binary_format<T>::sign_index()) : word; +#if FASTFLOAT_IS_BIG_ENDIAN == 1 + if (std::is_same<T, float>::value) { + ::memcpy(&value, (char *)&word + 4, sizeof(T)); // extract value at offset 4-7 if float on big-endian + } else { + ::memcpy(&value, &word, sizeof(T)); + } +#else + // For little-endian systems: + ::memcpy(&value, &word, sizeof(T)); +#endif +} + +} // namespace fast_float + +#endif + +#ifndef FASTFLOAT_ASCII_NUMBER_H +#define FASTFLOAT_ASCII_NUMBER_H + +#include <cctype> +#include <cstdint> +#include <cstring> +#include <iterator> + + +namespace fast_float { + +// Next function can be micro-optimized, but compilers are entirely +// able to optimize it well. +fastfloat_really_inline bool is_integer(char c) noexcept { return c >= '0' && c <= '9'; } + +fastfloat_really_inline uint64_t byteswap(uint64_t val) { + return (val & 0xFF00000000000000) >> 56 + | (val & 0x00FF000000000000) >> 40 + | (val & 0x0000FF0000000000) >> 24 + | (val & 0x000000FF00000000) >> 8 + | (val & 0x00000000FF000000) << 8 + | (val & 0x0000000000FF0000) << 24 + | (val & 0x000000000000FF00) << 40 + | (val & 0x00000000000000FF) << 56; +} + +fastfloat_really_inline uint64_t read_u64(const char *chars) { + uint64_t val; + ::memcpy(&val, chars, sizeof(uint64_t)); +#if FASTFLOAT_IS_BIG_ENDIAN == 1 + // Need to read as-if the number was in little-endian order. + val = byteswap(val); +#endif + return val; +} + +fastfloat_really_inline void write_u64(uint8_t *chars, uint64_t val) { +#if FASTFLOAT_IS_BIG_ENDIAN == 1 + // Need to read as-if the number was in little-endian order. + val = byteswap(val); +#endif + ::memcpy(chars, &val, sizeof(uint64_t)); +} + +// credit @aqrit +fastfloat_really_inline uint32_t parse_eight_digits_unrolled(uint64_t val) { + const uint64_t mask = 0x000000FF000000FF; + const uint64_t mul1 = 0x000F424000000064; // 100 + (1000000ULL << 32) + const uint64_t mul2 = 0x0000271000000001; // 1 + (10000ULL << 32) + val -= 0x3030303030303030; + val = (val * 10) + (val >> 8); // val = (val * 2561) >> 8; + val = (((val & mask) * mul1) + (((val >> 16) & mask) * mul2)) >> 32; + return uint32_t(val); +} + +fastfloat_really_inline uint32_t parse_eight_digits_unrolled(const char *chars) noexcept { + return parse_eight_digits_unrolled(read_u64(chars)); +} + +// credit @aqrit +fastfloat_really_inline bool is_made_of_eight_digits_fast(uint64_t val) noexcept { + return !((((val + 0x4646464646464646) | (val - 0x3030303030303030)) & + 0x8080808080808080)); +} + +fastfloat_really_inline bool is_made_of_eight_digits_fast(const char *chars) noexcept { + return is_made_of_eight_digits_fast(read_u64(chars)); +} + +typedef span<const char> byte_span; + +struct parsed_number_string { + int64_t exponent{0}; + uint64_t mantissa{0}; + const char *lastmatch{nullptr}; + bool negative{false}; + bool valid{false}; + bool too_many_digits{false}; + // contains the range of the significant digits + byte_span integer{}; // non-nullable + byte_span fraction{}; // nullable +}; + +// Assuming that you use no more than 19 digits, this will +// parse an ASCII string. +fastfloat_really_inline +parsed_number_string parse_number_string(const char *p, const char *pend, parse_options options) noexcept { + const chars_format fmt = options.format; + const char decimal_point = options.decimal_point; + + parsed_number_string answer; + answer.valid = false; + answer.too_many_digits = false; + answer.negative = (*p == '-'); + if (*p == '-') { // C++17 20.19.3.(7.1) explicitly forbids '+' sign here + ++p; + if (p == pend) { + return answer; + } + if (!is_integer(*p) && (*p != decimal_point)) { // a sign must be followed by an integer or the dot + return answer; + } + } + const char *const start_digits = p; + + uint64_t i = 0; // an unsigned int avoids signed overflows (which are bad) + + while ((std::distance(p, pend) >= 8) && is_made_of_eight_digits_fast(p)) { + i = i * 100000000 + parse_eight_digits_unrolled(p); // in rare cases, this will overflow, but that's ok + p += 8; + } + while ((p != pend) && is_integer(*p)) { + // a multiplication by 10 is cheaper than an arbitrary integer + // multiplication + i = 10 * i + + uint64_t(*p - '0'); // might overflow, we will handle the overflow later + ++p; + } + const char *const end_of_integer_part = p; + int64_t digit_count = int64_t(end_of_integer_part - start_digits); + answer.integer = byte_span(start_digits, size_t(digit_count)); + int64_t exponent = 0; + if ((p != pend) && (*p == decimal_point)) { + ++p; + const char* before = p; + // can occur at most twice without overflowing, but let it occur more, since + // for integers with many digits, digit parsing is the primary bottleneck. + while ((std::distance(p, pend) >= 8) && is_made_of_eight_digits_fast(p)) { + i = i * 100000000 + parse_eight_digits_unrolled(p); // in rare cases, this will overflow, but that's ok + p += 8; + } + while ((p != pend) && is_integer(*p)) { + uint8_t digit = uint8_t(*p - '0'); + ++p; + i = i * 10 + digit; // in rare cases, this will overflow, but that's ok + } + exponent = before - p; + answer.fraction = byte_span(before, size_t(p - before)); + digit_count -= exponent; + } + // we must have encountered at least one integer! + if (digit_count == 0) { + return answer; + } + int64_t exp_number = 0; // explicit exponential part + if ((fmt & chars_format::scientific) && (p != pend) && (('e' == *p) || ('E' == *p))) { + const char * location_of_e = p; + ++p; + bool neg_exp = false; + if ((p != pend) && ('-' == *p)) { + neg_exp = true; + ++p; + } else if ((p != pend) && ('+' == *p)) { // '+' on exponent is allowed by C++17 20.19.3.(7.1) + ++p; + } + if ((p == pend) || !is_integer(*p)) { + if(!(fmt & chars_format::fixed)) { + // We are in error. + return answer; + } + // Otherwise, we will be ignoring the 'e'. + p = location_of_e; + } else { + while ((p != pend) && is_integer(*p)) { + uint8_t digit = uint8_t(*p - '0'); + if (exp_number < 0x10000000) { + exp_number = 10 * exp_number + digit; + } + ++p; + } + if(neg_exp) { exp_number = - exp_number; } + exponent += exp_number; + } + } else { + // If it scientific and not fixed, we have to bail out. + if((fmt & chars_format::scientific) && !(fmt & chars_format::fixed)) { return answer; } + } + answer.lastmatch = p; + answer.valid = true; + + // If we frequently had to deal with long strings of digits, + // we could extend our code by using a 128-bit integer instead + // of a 64-bit integer. However, this is uncommon. + // + // We can deal with up to 19 digits. + if (digit_count > 19) { // this is uncommon + // It is possible that the integer had an overflow. + // We have to handle the case where we have 0.0000somenumber. + // We need to be mindful of the case where we only have zeroes... + // E.g., 0.000000000...000. + const char *start = start_digits; + while ((start != pend) && (*start == '0' || *start == decimal_point)) { + if(*start == '0') { digit_count --; } + start++; + } + if (digit_count > 19) { + answer.too_many_digits = true; + // Let us start again, this time, avoiding overflows. + // We don't need to check if is_integer, since we use the + // pre-tokenized spans from above. + i = 0; + p = answer.integer.ptr; + const char* int_end = p + answer.integer.len(); + const uint64_t minimal_nineteen_digit_integer{1000000000000000000}; + while((i < minimal_nineteen_digit_integer) && (p != int_end)) { + i = i * 10 + uint64_t(*p - '0'); + ++p; + } + if (i >= minimal_nineteen_digit_integer) { // We have a big integers + exponent = end_of_integer_part - p + exp_number; + } else { // We have a value with a fractional component. + p = answer.fraction.ptr; + const char* frac_end = p + answer.fraction.len(); + while((i < minimal_nineteen_digit_integer) && (p != frac_end)) { + i = i * 10 + uint64_t(*p - '0'); + ++p; + } + exponent = answer.fraction.ptr - p + exp_number; + } + // We have now corrected both exponent and i, to a truncated value + } + } + answer.exponent = exponent; + answer.mantissa = i; + return answer; +} + +} // namespace fast_float + +#endif + +#ifndef FASTFLOAT_FAST_TABLE_H +#define FASTFLOAT_FAST_TABLE_H + +#include <cstdint> + +namespace fast_float { + +/** + * When mapping numbers from decimal to binary, + * we go from w * 10^q to m * 2^p but we have + * 10^q = 5^q * 2^q, so effectively + * we are trying to match + * w * 2^q * 5^q to m * 2^p. Thus the powers of two + * are not a concern since they can be represented + * exactly using the binary notation, only the powers of five + * affect the binary significand. + */ + +/** + * The smallest non-zero float (binary64) is 2^−1074. + * We take as input numbers of the form w x 10^q where w < 2^64. + * We have that w * 10^-343 < 2^(64-344) 5^-343 < 2^-1076. + * However, we have that + * (2^64-1) * 10^-342 = (2^64-1) * 2^-342 * 5^-342 > 2^−1074. + * Thus it is possible for a number of the form w * 10^-342 where + * w is a 64-bit value to be a non-zero floating-point number. + ********* + * Any number of form w * 10^309 where w>= 1 is going to be + * infinite in binary64 so we never need to worry about powers + * of 5 greater than 308. + */ +template <class unused = void> +struct powers_template { + +constexpr static int smallest_power_of_five = binary_format<double>::smallest_power_of_ten(); +constexpr static int largest_power_of_five = binary_format<double>::largest_power_of_ten(); +constexpr static int number_of_entries = 2 * (largest_power_of_five - smallest_power_of_five + 1); +// Powers of five from 5^-342 all the way to 5^308 rounded toward one. +static const uint64_t power_of_five_128[number_of_entries]; +}; + +template <class unused> +const uint64_t powers_template<unused>::power_of_five_128[number_of_entries] = { + 0xeef453d6923bd65a,0x113faa2906a13b3f, + 0x9558b4661b6565f8,0x4ac7ca59a424c507, + 0xbaaee17fa23ebf76,0x5d79bcf00d2df649, + 0xe95a99df8ace6f53,0xf4d82c2c107973dc, + 0x91d8a02bb6c10594,0x79071b9b8a4be869, + 0xb64ec836a47146f9,0x9748e2826cdee284, + 0xe3e27a444d8d98b7,0xfd1b1b2308169b25, + 0x8e6d8c6ab0787f72,0xfe30f0f5e50e20f7, + 0xb208ef855c969f4f,0xbdbd2d335e51a935, + 0xde8b2b66b3bc4723,0xad2c788035e61382, + 0x8b16fb203055ac76,0x4c3bcb5021afcc31, + 0xaddcb9e83c6b1793,0xdf4abe242a1bbf3d, + 0xd953e8624b85dd78,0xd71d6dad34a2af0d, + 0x87d4713d6f33aa6b,0x8672648c40e5ad68, + 0xa9c98d8ccb009506,0x680efdaf511f18c2, + 0xd43bf0effdc0ba48,0x212bd1b2566def2, + 0x84a57695fe98746d,0x14bb630f7604b57, + 0xa5ced43b7e3e9188,0x419ea3bd35385e2d, + 0xcf42894a5dce35ea,0x52064cac828675b9, + 0x818995ce7aa0e1b2,0x7343efebd1940993, + 0xa1ebfb4219491a1f,0x1014ebe6c5f90bf8, + 0xca66fa129f9b60a6,0xd41a26e077774ef6, + 0xfd00b897478238d0,0x8920b098955522b4, + 0x9e20735e8cb16382,0x55b46e5f5d5535b0, + 0xc5a890362fddbc62,0xeb2189f734aa831d, + 0xf712b443bbd52b7b,0xa5e9ec7501d523e4, + 0x9a6bb0aa55653b2d,0x47b233c92125366e, + 0xc1069cd4eabe89f8,0x999ec0bb696e840a, + 0xf148440a256e2c76,0xc00670ea43ca250d, + 0x96cd2a865764dbca,0x380406926a5e5728, + 0xbc807527ed3e12bc,0xc605083704f5ecf2, + 0xeba09271e88d976b,0xf7864a44c633682e, + 0x93445b8731587ea3,0x7ab3ee6afbe0211d, + 0xb8157268fdae9e4c,0x5960ea05bad82964, + 0xe61acf033d1a45df,0x6fb92487298e33bd, + 0x8fd0c16206306bab,0xa5d3b6d479f8e056, + 0xb3c4f1ba87bc8696,0x8f48a4899877186c, + 0xe0b62e2929aba83c,0x331acdabfe94de87, + 0x8c71dcd9ba0b4925,0x9ff0c08b7f1d0b14, + 0xaf8e5410288e1b6f,0x7ecf0ae5ee44dd9, + 0xdb71e91432b1a24a,0xc9e82cd9f69d6150, + 0x892731ac9faf056e,0xbe311c083a225cd2, + 0xab70fe17c79ac6ca,0x6dbd630a48aaf406, + 0xd64d3d9db981787d,0x92cbbccdad5b108, + 0x85f0468293f0eb4e,0x25bbf56008c58ea5, + 0xa76c582338ed2621,0xaf2af2b80af6f24e, + 0xd1476e2c07286faa,0x1af5af660db4aee1, + 0x82cca4db847945ca,0x50d98d9fc890ed4d, + 0xa37fce126597973c,0xe50ff107bab528a0, + 0xcc5fc196fefd7d0c,0x1e53ed49a96272c8, + 0xff77b1fcbebcdc4f,0x25e8e89c13bb0f7a, + 0x9faacf3df73609b1,0x77b191618c54e9ac, + 0xc795830d75038c1d,0xd59df5b9ef6a2417, + 0xf97ae3d0d2446f25,0x4b0573286b44ad1d, + 0x9becce62836ac577,0x4ee367f9430aec32, + 0xc2e801fb244576d5,0x229c41f793cda73f, + 0xf3a20279ed56d48a,0x6b43527578c1110f, + 0x9845418c345644d6,0x830a13896b78aaa9, + 0xbe5691ef416bd60c,0x23cc986bc656d553, + 0xedec366b11c6cb8f,0x2cbfbe86b7ec8aa8, + 0x94b3a202eb1c3f39,0x7bf7d71432f3d6a9, + 0xb9e08a83a5e34f07,0xdaf5ccd93fb0cc53, + 0xe858ad248f5c22c9,0xd1b3400f8f9cff68, + 0x91376c36d99995be,0x23100809b9c21fa1, + 0xb58547448ffffb2d,0xabd40a0c2832a78a, + 0xe2e69915b3fff9f9,0x16c90c8f323f516c, + 0x8dd01fad907ffc3b,0xae3da7d97f6792e3, + 0xb1442798f49ffb4a,0x99cd11cfdf41779c, + 0xdd95317f31c7fa1d,0x40405643d711d583, + 0x8a7d3eef7f1cfc52,0x482835ea666b2572, + 0xad1c8eab5ee43b66,0xda3243650005eecf, + 0xd863b256369d4a40,0x90bed43e40076a82, + 0x873e4f75e2224e68,0x5a7744a6e804a291, + 0xa90de3535aaae202,0x711515d0a205cb36, + 0xd3515c2831559a83,0xd5a5b44ca873e03, + 0x8412d9991ed58091,0xe858790afe9486c2, + 0xa5178fff668ae0b6,0x626e974dbe39a872, + 0xce5d73ff402d98e3,0xfb0a3d212dc8128f, + 0x80fa687f881c7f8e,0x7ce66634bc9d0b99, + 0xa139029f6a239f72,0x1c1fffc1ebc44e80, + 0xc987434744ac874e,0xa327ffb266b56220, + 0xfbe9141915d7a922,0x4bf1ff9f0062baa8, + 0x9d71ac8fada6c9b5,0x6f773fc3603db4a9, + 0xc4ce17b399107c22,0xcb550fb4384d21d3, + 0xf6019da07f549b2b,0x7e2a53a146606a48, + 0x99c102844f94e0fb,0x2eda7444cbfc426d, + 0xc0314325637a1939,0xfa911155fefb5308, + 0xf03d93eebc589f88,0x793555ab7eba27ca, + 0x96267c7535b763b5,0x4bc1558b2f3458de, + 0xbbb01b9283253ca2,0x9eb1aaedfb016f16, + 0xea9c227723ee8bcb,0x465e15a979c1cadc, + 0x92a1958a7675175f,0xbfacd89ec191ec9, + 0xb749faed14125d36,0xcef980ec671f667b, + 0xe51c79a85916f484,0x82b7e12780e7401a, + 0x8f31cc0937ae58d2,0xd1b2ecb8b0908810, + 0xb2fe3f0b8599ef07,0x861fa7e6dcb4aa15, + 0xdfbdcece67006ac9,0x67a791e093e1d49a, + 0x8bd6a141006042bd,0xe0c8bb2c5c6d24e0, + 0xaecc49914078536d,0x58fae9f773886e18, + 0xda7f5bf590966848,0xaf39a475506a899e, + 0x888f99797a5e012d,0x6d8406c952429603, + 0xaab37fd7d8f58178,0xc8e5087ba6d33b83, + 0xd5605fcdcf32e1d6,0xfb1e4a9a90880a64, + 0x855c3be0a17fcd26,0x5cf2eea09a55067f, + 0xa6b34ad8c9dfc06f,0xf42faa48c0ea481e, + 0xd0601d8efc57b08b,0xf13b94daf124da26, + 0x823c12795db6ce57,0x76c53d08d6b70858, + 0xa2cb1717b52481ed,0x54768c4b0c64ca6e, + 0xcb7ddcdda26da268,0xa9942f5dcf7dfd09, + 0xfe5d54150b090b02,0xd3f93b35435d7c4c, + 0x9efa548d26e5a6e1,0xc47bc5014a1a6daf, + 0xc6b8e9b0709f109a,0x359ab6419ca1091b, + 0xf867241c8cc6d4c0,0xc30163d203c94b62, + 0x9b407691d7fc44f8,0x79e0de63425dcf1d, + 0xc21094364dfb5636,0x985915fc12f542e4, + 0xf294b943e17a2bc4,0x3e6f5b7b17b2939d, + 0x979cf3ca6cec5b5a,0xa705992ceecf9c42, + 0xbd8430bd08277231,0x50c6ff782a838353, + 0xece53cec4a314ebd,0xa4f8bf5635246428, + 0x940f4613ae5ed136,0x871b7795e136be99, + 0xb913179899f68584,0x28e2557b59846e3f, + 0xe757dd7ec07426e5,0x331aeada2fe589cf, + 0x9096ea6f3848984f,0x3ff0d2c85def7621, + 0xb4bca50b065abe63,0xfed077a756b53a9, + 0xe1ebce4dc7f16dfb,0xd3e8495912c62894, + 0x8d3360f09cf6e4bd,0x64712dd7abbbd95c, + 0xb080392cc4349dec,0xbd8d794d96aacfb3, + 0xdca04777f541c567,0xecf0d7a0fc5583a0, + 0x89e42caaf9491b60,0xf41686c49db57244, + 0xac5d37d5b79b6239,0x311c2875c522ced5, + 0xd77485cb25823ac7,0x7d633293366b828b, + 0x86a8d39ef77164bc,0xae5dff9c02033197, + 0xa8530886b54dbdeb,0xd9f57f830283fdfc, + 0xd267caa862a12d66,0xd072df63c324fd7b, + 0x8380dea93da4bc60,0x4247cb9e59f71e6d, + 0xa46116538d0deb78,0x52d9be85f074e608, + 0xcd795be870516656,0x67902e276c921f8b, + 0x806bd9714632dff6,0xba1cd8a3db53b6, + 0xa086cfcd97bf97f3,0x80e8a40eccd228a4, + 0xc8a883c0fdaf7df0,0x6122cd128006b2cd, + 0xfad2a4b13d1b5d6c,0x796b805720085f81, + 0x9cc3a6eec6311a63,0xcbe3303674053bb0, + 0xc3f490aa77bd60fc,0xbedbfc4411068a9c, + 0xf4f1b4d515acb93b,0xee92fb5515482d44, + 0x991711052d8bf3c5,0x751bdd152d4d1c4a, + 0xbf5cd54678eef0b6,0xd262d45a78a0635d, + 0xef340a98172aace4,0x86fb897116c87c34, + 0x9580869f0e7aac0e,0xd45d35e6ae3d4da0, + 0xbae0a846d2195712,0x8974836059cca109, + 0xe998d258869facd7,0x2bd1a438703fc94b, + 0x91ff83775423cc06,0x7b6306a34627ddcf, + 0xb67f6455292cbf08,0x1a3bc84c17b1d542, + 0xe41f3d6a7377eeca,0x20caba5f1d9e4a93, + 0x8e938662882af53e,0x547eb47b7282ee9c, + 0xb23867fb2a35b28d,0xe99e619a4f23aa43, + 0xdec681f9f4c31f31,0x6405fa00e2ec94d4, + 0x8b3c113c38f9f37e,0xde83bc408dd3dd04, + 0xae0b158b4738705e,0x9624ab50b148d445, + 0xd98ddaee19068c76,0x3badd624dd9b0957, + 0x87f8a8d4cfa417c9,0xe54ca5d70a80e5d6, + 0xa9f6d30a038d1dbc,0x5e9fcf4ccd211f4c, + 0xd47487cc8470652b,0x7647c3200069671f, + 0x84c8d4dfd2c63f3b,0x29ecd9f40041e073, + 0xa5fb0a17c777cf09,0xf468107100525890, + 0xcf79cc9db955c2cc,0x7182148d4066eeb4, + 0x81ac1fe293d599bf,0xc6f14cd848405530, + 0xa21727db38cb002f,0xb8ada00e5a506a7c, + 0xca9cf1d206fdc03b,0xa6d90811f0e4851c, + 0xfd442e4688bd304a,0x908f4a166d1da663, + 0x9e4a9cec15763e2e,0x9a598e4e043287fe, + 0xc5dd44271ad3cdba,0x40eff1e1853f29fd, + 0xf7549530e188c128,0xd12bee59e68ef47c, + 0x9a94dd3e8cf578b9,0x82bb74f8301958ce, + 0xc13a148e3032d6e7,0xe36a52363c1faf01, + 0xf18899b1bc3f8ca1,0xdc44e6c3cb279ac1, + 0x96f5600f15a7b7e5,0x29ab103a5ef8c0b9, + 0xbcb2b812db11a5de,0x7415d448f6b6f0e7, + 0xebdf661791d60f56,0x111b495b3464ad21, + 0x936b9fcebb25c995,0xcab10dd900beec34, + 0xb84687c269ef3bfb,0x3d5d514f40eea742, + 0xe65829b3046b0afa,0xcb4a5a3112a5112, + 0x8ff71a0fe2c2e6dc,0x47f0e785eaba72ab, + 0xb3f4e093db73a093,0x59ed216765690f56, + 0xe0f218b8d25088b8,0x306869c13ec3532c, + 0x8c974f7383725573,0x1e414218c73a13fb, + 0xafbd2350644eeacf,0xe5d1929ef90898fa, + 0xdbac6c247d62a583,0xdf45f746b74abf39, + 0x894bc396ce5da772,0x6b8bba8c328eb783, + 0xab9eb47c81f5114f,0x66ea92f3f326564, + 0xd686619ba27255a2,0xc80a537b0efefebd, + 0x8613fd0145877585,0xbd06742ce95f5f36, + 0xa798fc4196e952e7,0x2c48113823b73704, + 0xd17f3b51fca3a7a0,0xf75a15862ca504c5, + 0x82ef85133de648c4,0x9a984d73dbe722fb, + 0xa3ab66580d5fdaf5,0xc13e60d0d2e0ebba, + 0xcc963fee10b7d1b3,0x318df905079926a8, + 0xffbbcfe994e5c61f,0xfdf17746497f7052, + 0x9fd561f1fd0f9bd3,0xfeb6ea8bedefa633, + 0xc7caba6e7c5382c8,0xfe64a52ee96b8fc0, + 0xf9bd690a1b68637b,0x3dfdce7aa3c673b0, + 0x9c1661a651213e2d,0x6bea10ca65c084e, + 0xc31bfa0fe5698db8,0x486e494fcff30a62, + 0xf3e2f893dec3f126,0x5a89dba3c3efccfa, + 0x986ddb5c6b3a76b7,0xf89629465a75e01c, + 0xbe89523386091465,0xf6bbb397f1135823, + 0xee2ba6c0678b597f,0x746aa07ded582e2c, + 0x94db483840b717ef,0xa8c2a44eb4571cdc, + 0xba121a4650e4ddeb,0x92f34d62616ce413, + 0xe896a0d7e51e1566,0x77b020baf9c81d17, + 0x915e2486ef32cd60,0xace1474dc1d122e, + 0xb5b5ada8aaff80b8,0xd819992132456ba, + 0xe3231912d5bf60e6,0x10e1fff697ed6c69, + 0x8df5efabc5979c8f,0xca8d3ffa1ef463c1, + 0xb1736b96b6fd83b3,0xbd308ff8a6b17cb2, + 0xddd0467c64bce4a0,0xac7cb3f6d05ddbde, + 0x8aa22c0dbef60ee4,0x6bcdf07a423aa96b, + 0xad4ab7112eb3929d,0x86c16c98d2c953c6, + 0xd89d64d57a607744,0xe871c7bf077ba8b7, + 0x87625f056c7c4a8b,0x11471cd764ad4972, + 0xa93af6c6c79b5d2d,0xd598e40d3dd89bcf, + 0xd389b47879823479,0x4aff1d108d4ec2c3, + 0x843610cb4bf160cb,0xcedf722a585139ba, + 0xa54394fe1eedb8fe,0xc2974eb4ee658828, + 0xce947a3da6a9273e,0x733d226229feea32, + 0x811ccc668829b887,0x806357d5a3f525f, + 0xa163ff802a3426a8,0xca07c2dcb0cf26f7, + 0xc9bcff6034c13052,0xfc89b393dd02f0b5, + 0xfc2c3f3841f17c67,0xbbac2078d443ace2, + 0x9d9ba7832936edc0,0xd54b944b84aa4c0d, + 0xc5029163f384a931,0xa9e795e65d4df11, + 0xf64335bcf065d37d,0x4d4617b5ff4a16d5, + 0x99ea0196163fa42e,0x504bced1bf8e4e45, + 0xc06481fb9bcf8d39,0xe45ec2862f71e1d6, + 0xf07da27a82c37088,0x5d767327bb4e5a4c, + 0x964e858c91ba2655,0x3a6a07f8d510f86f, + 0xbbe226efb628afea,0x890489f70a55368b, + 0xeadab0aba3b2dbe5,0x2b45ac74ccea842e, + 0x92c8ae6b464fc96f,0x3b0b8bc90012929d, + 0xb77ada0617e3bbcb,0x9ce6ebb40173744, + 0xe55990879ddcaabd,0xcc420a6a101d0515, + 0x8f57fa54c2a9eab6,0x9fa946824a12232d, + 0xb32df8e9f3546564,0x47939822dc96abf9, + 0xdff9772470297ebd,0x59787e2b93bc56f7, + 0x8bfbea76c619ef36,0x57eb4edb3c55b65a, + 0xaefae51477a06b03,0xede622920b6b23f1, + 0xdab99e59958885c4,0xe95fab368e45eced, + 0x88b402f7fd75539b,0x11dbcb0218ebb414, + 0xaae103b5fcd2a881,0xd652bdc29f26a119, + 0xd59944a37c0752a2,0x4be76d3346f0495f, + 0x857fcae62d8493a5,0x6f70a4400c562ddb, + 0xa6dfbd9fb8e5b88e,0xcb4ccd500f6bb952, + 0xd097ad07a71f26b2,0x7e2000a41346a7a7, + 0x825ecc24c873782f,0x8ed400668c0c28c8, + 0xa2f67f2dfa90563b,0x728900802f0f32fa, + 0xcbb41ef979346bca,0x4f2b40a03ad2ffb9, + 0xfea126b7d78186bc,0xe2f610c84987bfa8, + 0x9f24b832e6b0f436,0xdd9ca7d2df4d7c9, + 0xc6ede63fa05d3143,0x91503d1c79720dbb, + 0xf8a95fcf88747d94,0x75a44c6397ce912a, + 0x9b69dbe1b548ce7c,0xc986afbe3ee11aba, + 0xc24452da229b021b,0xfbe85badce996168, + 0xf2d56790ab41c2a2,0xfae27299423fb9c3, + 0x97c560ba6b0919a5,0xdccd879fc967d41a, + 0xbdb6b8e905cb600f,0x5400e987bbc1c920, + 0xed246723473e3813,0x290123e9aab23b68, + 0x9436c0760c86e30b,0xf9a0b6720aaf6521, + 0xb94470938fa89bce,0xf808e40e8d5b3e69, + 0xe7958cb87392c2c2,0xb60b1d1230b20e04, + 0x90bd77f3483bb9b9,0xb1c6f22b5e6f48c2, + 0xb4ecd5f01a4aa828,0x1e38aeb6360b1af3, + 0xe2280b6c20dd5232,0x25c6da63c38de1b0, + 0x8d590723948a535f,0x579c487e5a38ad0e, + 0xb0af48ec79ace837,0x2d835a9df0c6d851, + 0xdcdb1b2798182244,0xf8e431456cf88e65, + 0x8a08f0f8bf0f156b,0x1b8e9ecb641b58ff, + 0xac8b2d36eed2dac5,0xe272467e3d222f3f, + 0xd7adf884aa879177,0x5b0ed81dcc6abb0f, + 0x86ccbb52ea94baea,0x98e947129fc2b4e9, + 0xa87fea27a539e9a5,0x3f2398d747b36224, + 0xd29fe4b18e88640e,0x8eec7f0d19a03aad, + 0x83a3eeeef9153e89,0x1953cf68300424ac, + 0xa48ceaaab75a8e2b,0x5fa8c3423c052dd7, + 0xcdb02555653131b6,0x3792f412cb06794d, + 0x808e17555f3ebf11,0xe2bbd88bbee40bd0, + 0xa0b19d2ab70e6ed6,0x5b6aceaeae9d0ec4, + 0xc8de047564d20a8b,0xf245825a5a445275, + 0xfb158592be068d2e,0xeed6e2f0f0d56712, + 0x9ced737bb6c4183d,0x55464dd69685606b, + 0xc428d05aa4751e4c,0xaa97e14c3c26b886, + 0xf53304714d9265df,0xd53dd99f4b3066a8, + 0x993fe2c6d07b7fab,0xe546a8038efe4029, + 0xbf8fdb78849a5f96,0xde98520472bdd033, + 0xef73d256a5c0f77c,0x963e66858f6d4440, + 0x95a8637627989aad,0xdde7001379a44aa8, + 0xbb127c53b17ec159,0x5560c018580d5d52, + 0xe9d71b689dde71af,0xaab8f01e6e10b4a6, + 0x9226712162ab070d,0xcab3961304ca70e8, + 0xb6b00d69bb55c8d1,0x3d607b97c5fd0d22, + 0xe45c10c42a2b3b05,0x8cb89a7db77c506a, + 0x8eb98a7a9a5b04e3,0x77f3608e92adb242, + 0xb267ed1940f1c61c,0x55f038b237591ed3, + 0xdf01e85f912e37a3,0x6b6c46dec52f6688, + 0x8b61313bbabce2c6,0x2323ac4b3b3da015, + 0xae397d8aa96c1b77,0xabec975e0a0d081a, + 0xd9c7dced53c72255,0x96e7bd358c904a21, + 0x881cea14545c7575,0x7e50d64177da2e54, + 0xaa242499697392d2,0xdde50bd1d5d0b9e9, + 0xd4ad2dbfc3d07787,0x955e4ec64b44e864, + 0x84ec3c97da624ab4,0xbd5af13bef0b113e, + 0xa6274bbdd0fadd61,0xecb1ad8aeacdd58e, + 0xcfb11ead453994ba,0x67de18eda5814af2, + 0x81ceb32c4b43fcf4,0x80eacf948770ced7, + 0xa2425ff75e14fc31,0xa1258379a94d028d, + 0xcad2f7f5359a3b3e,0x96ee45813a04330, + 0xfd87b5f28300ca0d,0x8bca9d6e188853fc, + 0x9e74d1b791e07e48,0x775ea264cf55347e, + 0xc612062576589dda,0x95364afe032a819e, + 0xf79687aed3eec551,0x3a83ddbd83f52205, + 0x9abe14cd44753b52,0xc4926a9672793543, + 0xc16d9a0095928a27,0x75b7053c0f178294, + 0xf1c90080baf72cb1,0x5324c68b12dd6339, + 0x971da05074da7bee,0xd3f6fc16ebca5e04, + 0xbce5086492111aea,0x88f4bb1ca6bcf585, + 0xec1e4a7db69561a5,0x2b31e9e3d06c32e6, + 0x9392ee8e921d5d07,0x3aff322e62439fd0, + 0xb877aa3236a4b449,0x9befeb9fad487c3, + 0xe69594bec44de15b,0x4c2ebe687989a9b4, + 0x901d7cf73ab0acd9,0xf9d37014bf60a11, + 0xb424dc35095cd80f,0x538484c19ef38c95, + 0xe12e13424bb40e13,0x2865a5f206b06fba, + 0x8cbccc096f5088cb,0xf93f87b7442e45d4, + 0xafebff0bcb24aafe,0xf78f69a51539d749, + 0xdbe6fecebdedd5be,0xb573440e5a884d1c, + 0x89705f4136b4a597,0x31680a88f8953031, + 0xabcc77118461cefc,0xfdc20d2b36ba7c3e, + 0xd6bf94d5e57a42bc,0x3d32907604691b4d, + 0x8637bd05af6c69b5,0xa63f9a49c2c1b110, + 0xa7c5ac471b478423,0xfcf80dc33721d54, + 0xd1b71758e219652b,0xd3c36113404ea4a9, + 0x83126e978d4fdf3b,0x645a1cac083126ea, + 0xa3d70a3d70a3d70a,0x3d70a3d70a3d70a4, + 0xcccccccccccccccc,0xcccccccccccccccd, + 0x8000000000000000,0x0, + 0xa000000000000000,0x0, + 0xc800000000000000,0x0, + 0xfa00000000000000,0x0, + 0x9c40000000000000,0x0, + 0xc350000000000000,0x0, + 0xf424000000000000,0x0, + 0x9896800000000000,0x0, + 0xbebc200000000000,0x0, + 0xee6b280000000000,0x0, + 0x9502f90000000000,0x0, + 0xba43b74000000000,0x0, + 0xe8d4a51000000000,0x0, + 0x9184e72a00000000,0x0, + 0xb5e620f480000000,0x0, + 0xe35fa931a0000000,0x0, + 0x8e1bc9bf04000000,0x0, + 0xb1a2bc2ec5000000,0x0, + 0xde0b6b3a76400000,0x0, + 0x8ac7230489e80000,0x0, + 0xad78ebc5ac620000,0x0, + 0xd8d726b7177a8000,0x0, + 0x878678326eac9000,0x0, + 0xa968163f0a57b400,0x0, + 0xd3c21bcecceda100,0x0, + 0x84595161401484a0,0x0, + 0xa56fa5b99019a5c8,0x0, + 0xcecb8f27f4200f3a,0x0, + 0x813f3978f8940984,0x4000000000000000, + 0xa18f07d736b90be5,0x5000000000000000, + 0xc9f2c9cd04674ede,0xa400000000000000, + 0xfc6f7c4045812296,0x4d00000000000000, + 0x9dc5ada82b70b59d,0xf020000000000000, + 0xc5371912364ce305,0x6c28000000000000, + 0xf684df56c3e01bc6,0xc732000000000000, + 0x9a130b963a6c115c,0x3c7f400000000000, + 0xc097ce7bc90715b3,0x4b9f100000000000, + 0xf0bdc21abb48db20,0x1e86d40000000000, + 0x96769950b50d88f4,0x1314448000000000, + 0xbc143fa4e250eb31,0x17d955a000000000, + 0xeb194f8e1ae525fd,0x5dcfab0800000000, + 0x92efd1b8d0cf37be,0x5aa1cae500000000, + 0xb7abc627050305ad,0xf14a3d9e40000000, + 0xe596b7b0c643c719,0x6d9ccd05d0000000, + 0x8f7e32ce7bea5c6f,0xe4820023a2000000, + 0xb35dbf821ae4f38b,0xdda2802c8a800000, + 0xe0352f62a19e306e,0xd50b2037ad200000, + 0x8c213d9da502de45,0x4526f422cc340000, + 0xaf298d050e4395d6,0x9670b12b7f410000, + 0xdaf3f04651d47b4c,0x3c0cdd765f114000, + 0x88d8762bf324cd0f,0xa5880a69fb6ac800, + 0xab0e93b6efee0053,0x8eea0d047a457a00, + 0xd5d238a4abe98068,0x72a4904598d6d880, + 0x85a36366eb71f041,0x47a6da2b7f864750, + 0xa70c3c40a64e6c51,0x999090b65f67d924, + 0xd0cf4b50cfe20765,0xfff4b4e3f741cf6d, + 0x82818f1281ed449f,0xbff8f10e7a8921a4, + 0xa321f2d7226895c7,0xaff72d52192b6a0d, + 0xcbea6f8ceb02bb39,0x9bf4f8a69f764490, + 0xfee50b7025c36a08,0x2f236d04753d5b4, + 0x9f4f2726179a2245,0x1d762422c946590, + 0xc722f0ef9d80aad6,0x424d3ad2b7b97ef5, + 0xf8ebad2b84e0d58b,0xd2e0898765a7deb2, + 0x9b934c3b330c8577,0x63cc55f49f88eb2f, + 0xc2781f49ffcfa6d5,0x3cbf6b71c76b25fb, + 0xf316271c7fc3908a,0x8bef464e3945ef7a, + 0x97edd871cfda3a56,0x97758bf0e3cbb5ac, + 0xbde94e8e43d0c8ec,0x3d52eeed1cbea317, + 0xed63a231d4c4fb27,0x4ca7aaa863ee4bdd, + 0x945e455f24fb1cf8,0x8fe8caa93e74ef6a, + 0xb975d6b6ee39e436,0xb3e2fd538e122b44, + 0xe7d34c64a9c85d44,0x60dbbca87196b616, + 0x90e40fbeea1d3a4a,0xbc8955e946fe31cd, + 0xb51d13aea4a488dd,0x6babab6398bdbe41, + 0xe264589a4dcdab14,0xc696963c7eed2dd1, + 0x8d7eb76070a08aec,0xfc1e1de5cf543ca2, + 0xb0de65388cc8ada8,0x3b25a55f43294bcb, + 0xdd15fe86affad912,0x49ef0eb713f39ebe, + 0x8a2dbf142dfcc7ab,0x6e3569326c784337, + 0xacb92ed9397bf996,0x49c2c37f07965404, + 0xd7e77a8f87daf7fb,0xdc33745ec97be906, + 0x86f0ac99b4e8dafd,0x69a028bb3ded71a3, + 0xa8acd7c0222311bc,0xc40832ea0d68ce0c, + 0xd2d80db02aabd62b,0xf50a3fa490c30190, + 0x83c7088e1aab65db,0x792667c6da79e0fa, + 0xa4b8cab1a1563f52,0x577001b891185938, + 0xcde6fd5e09abcf26,0xed4c0226b55e6f86, + 0x80b05e5ac60b6178,0x544f8158315b05b4, + 0xa0dc75f1778e39d6,0x696361ae3db1c721, + 0xc913936dd571c84c,0x3bc3a19cd1e38e9, + 0xfb5878494ace3a5f,0x4ab48a04065c723, + 0x9d174b2dcec0e47b,0x62eb0d64283f9c76, + 0xc45d1df942711d9a,0x3ba5d0bd324f8394, + 0xf5746577930d6500,0xca8f44ec7ee36479, + 0x9968bf6abbe85f20,0x7e998b13cf4e1ecb, + 0xbfc2ef456ae276e8,0x9e3fedd8c321a67e, + 0xefb3ab16c59b14a2,0xc5cfe94ef3ea101e, + 0x95d04aee3b80ece5,0xbba1f1d158724a12, + 0xbb445da9ca61281f,0x2a8a6e45ae8edc97, + 0xea1575143cf97226,0xf52d09d71a3293bd, + 0x924d692ca61be758,0x593c2626705f9c56, + 0xb6e0c377cfa2e12e,0x6f8b2fb00c77836c, + 0xe498f455c38b997a,0xb6dfb9c0f956447, + 0x8edf98b59a373fec,0x4724bd4189bd5eac, + 0xb2977ee300c50fe7,0x58edec91ec2cb657, + 0xdf3d5e9bc0f653e1,0x2f2967b66737e3ed, + 0x8b865b215899f46c,0xbd79e0d20082ee74, + 0xae67f1e9aec07187,0xecd8590680a3aa11, + 0xda01ee641a708de9,0xe80e6f4820cc9495, + 0x884134fe908658b2,0x3109058d147fdcdd, + 0xaa51823e34a7eede,0xbd4b46f0599fd415, + 0xd4e5e2cdc1d1ea96,0x6c9e18ac7007c91a, + 0x850fadc09923329e,0x3e2cf6bc604ddb0, + 0xa6539930bf6bff45,0x84db8346b786151c, + 0xcfe87f7cef46ff16,0xe612641865679a63, + 0x81f14fae158c5f6e,0x4fcb7e8f3f60c07e, + 0xa26da3999aef7749,0xe3be5e330f38f09d, + 0xcb090c8001ab551c,0x5cadf5bfd3072cc5, + 0xfdcb4fa002162a63,0x73d9732fc7c8f7f6, + 0x9e9f11c4014dda7e,0x2867e7fddcdd9afa, + 0xc646d63501a1511d,0xb281e1fd541501b8, + 0xf7d88bc24209a565,0x1f225a7ca91a4226, + 0x9ae757596946075f,0x3375788de9b06958, + 0xc1a12d2fc3978937,0x52d6b1641c83ae, + 0xf209787bb47d6b84,0xc0678c5dbd23a49a, + 0x9745eb4d50ce6332,0xf840b7ba963646e0, + 0xbd176620a501fbff,0xb650e5a93bc3d898, + 0xec5d3fa8ce427aff,0xa3e51f138ab4cebe, + 0x93ba47c980e98cdf,0xc66f336c36b10137, + 0xb8a8d9bbe123f017,0xb80b0047445d4184, + 0xe6d3102ad96cec1d,0xa60dc059157491e5, + 0x9043ea1ac7e41392,0x87c89837ad68db2f, + 0xb454e4a179dd1877,0x29babe4598c311fb, + 0xe16a1dc9d8545e94,0xf4296dd6fef3d67a, + 0x8ce2529e2734bb1d,0x1899e4a65f58660c, + 0xb01ae745b101e9e4,0x5ec05dcff72e7f8f, + 0xdc21a1171d42645d,0x76707543f4fa1f73, + 0x899504ae72497eba,0x6a06494a791c53a8, + 0xabfa45da0edbde69,0x487db9d17636892, + 0xd6f8d7509292d603,0x45a9d2845d3c42b6, + 0x865b86925b9bc5c2,0xb8a2392ba45a9b2, + 0xa7f26836f282b732,0x8e6cac7768d7141e, + 0xd1ef0244af2364ff,0x3207d795430cd926, + 0x8335616aed761f1f,0x7f44e6bd49e807b8, + 0xa402b9c5a8d3a6e7,0x5f16206c9c6209a6, + 0xcd036837130890a1,0x36dba887c37a8c0f, + 0x802221226be55a64,0xc2494954da2c9789, + 0xa02aa96b06deb0fd,0xf2db9baa10b7bd6c, + 0xc83553c5c8965d3d,0x6f92829494e5acc7, + 0xfa42a8b73abbf48c,0xcb772339ba1f17f9, + 0x9c69a97284b578d7,0xff2a760414536efb, + 0xc38413cf25e2d70d,0xfef5138519684aba, + 0xf46518c2ef5b8cd1,0x7eb258665fc25d69, + 0x98bf2f79d5993802,0xef2f773ffbd97a61, + 0xbeeefb584aff8603,0xaafb550ffacfd8fa, + 0xeeaaba2e5dbf6784,0x95ba2a53f983cf38, + 0x952ab45cfa97a0b2,0xdd945a747bf26183, + 0xba756174393d88df,0x94f971119aeef9e4, + 0xe912b9d1478ceb17,0x7a37cd5601aab85d, + 0x91abb422ccb812ee,0xac62e055c10ab33a, + 0xb616a12b7fe617aa,0x577b986b314d6009, + 0xe39c49765fdf9d94,0xed5a7e85fda0b80b, + 0x8e41ade9fbebc27d,0x14588f13be847307, + 0xb1d219647ae6b31c,0x596eb2d8ae258fc8, + 0xde469fbd99a05fe3,0x6fca5f8ed9aef3bb, + 0x8aec23d680043bee,0x25de7bb9480d5854, + 0xada72ccc20054ae9,0xaf561aa79a10ae6a, + 0xd910f7ff28069da4,0x1b2ba1518094da04, + 0x87aa9aff79042286,0x90fb44d2f05d0842, + 0xa99541bf57452b28,0x353a1607ac744a53, + 0xd3fa922f2d1675f2,0x42889b8997915ce8, + 0x847c9b5d7c2e09b7,0x69956135febada11, + 0xa59bc234db398c25,0x43fab9837e699095, + 0xcf02b2c21207ef2e,0x94f967e45e03f4bb, + 0x8161afb94b44f57d,0x1d1be0eebac278f5, + 0xa1ba1ba79e1632dc,0x6462d92a69731732, + 0xca28a291859bbf93,0x7d7b8f7503cfdcfe, + 0xfcb2cb35e702af78,0x5cda735244c3d43e, + 0x9defbf01b061adab,0x3a0888136afa64a7, + 0xc56baec21c7a1916,0x88aaa1845b8fdd0, + 0xf6c69a72a3989f5b,0x8aad549e57273d45, + 0x9a3c2087a63f6399,0x36ac54e2f678864b, + 0xc0cb28a98fcf3c7f,0x84576a1bb416a7dd, + 0xf0fdf2d3f3c30b9f,0x656d44a2a11c51d5, + 0x969eb7c47859e743,0x9f644ae5a4b1b325, + 0xbc4665b596706114,0x873d5d9f0dde1fee, + 0xeb57ff22fc0c7959,0xa90cb506d155a7ea, + 0x9316ff75dd87cbd8,0x9a7f12442d588f2, + 0xb7dcbf5354e9bece,0xc11ed6d538aeb2f, + 0xe5d3ef282a242e81,0x8f1668c8a86da5fa, + 0x8fa475791a569d10,0xf96e017d694487bc, + 0xb38d92d760ec4455,0x37c981dcc395a9ac, + 0xe070f78d3927556a,0x85bbe253f47b1417, + 0x8c469ab843b89562,0x93956d7478ccec8e, + 0xaf58416654a6babb,0x387ac8d1970027b2, + 0xdb2e51bfe9d0696a,0x6997b05fcc0319e, + 0x88fcf317f22241e2,0x441fece3bdf81f03, + 0xab3c2fddeeaad25a,0xd527e81cad7626c3, + 0xd60b3bd56a5586f1,0x8a71e223d8d3b074, + 0x85c7056562757456,0xf6872d5667844e49, + 0xa738c6bebb12d16c,0xb428f8ac016561db, + 0xd106f86e69d785c7,0xe13336d701beba52, + 0x82a45b450226b39c,0xecc0024661173473, + 0xa34d721642b06084,0x27f002d7f95d0190, + 0xcc20ce9bd35c78a5,0x31ec038df7b441f4, + 0xff290242c83396ce,0x7e67047175a15271, + 0x9f79a169bd203e41,0xf0062c6e984d386, + 0xc75809c42c684dd1,0x52c07b78a3e60868, + 0xf92e0c3537826145,0xa7709a56ccdf8a82, + 0x9bbcc7a142b17ccb,0x88a66076400bb691, + 0xc2abf989935ddbfe,0x6acff893d00ea435, + 0xf356f7ebf83552fe,0x583f6b8c4124d43, + 0x98165af37b2153de,0xc3727a337a8b704a, + 0xbe1bf1b059e9a8d6,0x744f18c0592e4c5c, + 0xeda2ee1c7064130c,0x1162def06f79df73, + 0x9485d4d1c63e8be7,0x8addcb5645ac2ba8, + 0xb9a74a0637ce2ee1,0x6d953e2bd7173692, + 0xe8111c87c5c1ba99,0xc8fa8db6ccdd0437, + 0x910ab1d4db9914a0,0x1d9c9892400a22a2, + 0xb54d5e4a127f59c8,0x2503beb6d00cab4b, + 0xe2a0b5dc971f303a,0x2e44ae64840fd61d, + 0x8da471a9de737e24,0x5ceaecfed289e5d2, + 0xb10d8e1456105dad,0x7425a83e872c5f47, + 0xdd50f1996b947518,0xd12f124e28f77719, + 0x8a5296ffe33cc92f,0x82bd6b70d99aaa6f, + 0xace73cbfdc0bfb7b,0x636cc64d1001550b, + 0xd8210befd30efa5a,0x3c47f7e05401aa4e, + 0x8714a775e3e95c78,0x65acfaec34810a71, + 0xa8d9d1535ce3b396,0x7f1839a741a14d0d, + 0xd31045a8341ca07c,0x1ede48111209a050, + 0x83ea2b892091e44d,0x934aed0aab460432, + 0xa4e4b66b68b65d60,0xf81da84d5617853f, + 0xce1de40642e3f4b9,0x36251260ab9d668e, + 0x80d2ae83e9ce78f3,0xc1d72b7c6b426019, + 0xa1075a24e4421730,0xb24cf65b8612f81f, + 0xc94930ae1d529cfc,0xdee033f26797b627, + 0xfb9b7cd9a4a7443c,0x169840ef017da3b1, + 0x9d412e0806e88aa5,0x8e1f289560ee864e, + 0xc491798a08a2ad4e,0xf1a6f2bab92a27e2, + 0xf5b5d7ec8acb58a2,0xae10af696774b1db, + 0x9991a6f3d6bf1765,0xacca6da1e0a8ef29, + 0xbff610b0cc6edd3f,0x17fd090a58d32af3, + 0xeff394dcff8a948e,0xddfc4b4cef07f5b0, + 0x95f83d0a1fb69cd9,0x4abdaf101564f98e, + 0xbb764c4ca7a4440f,0x9d6d1ad41abe37f1, + 0xea53df5fd18d5513,0x84c86189216dc5ed, + 0x92746b9be2f8552c,0x32fd3cf5b4e49bb4, + 0xb7118682dbb66a77,0x3fbc8c33221dc2a1, + 0xe4d5e82392a40515,0xfabaf3feaa5334a, + 0x8f05b1163ba6832d,0x29cb4d87f2a7400e, + 0xb2c71d5bca9023f8,0x743e20e9ef511012, + 0xdf78e4b2bd342cf6,0x914da9246b255416, + 0x8bab8eefb6409c1a,0x1ad089b6c2f7548e, + 0xae9672aba3d0c320,0xa184ac2473b529b1, + 0xda3c0f568cc4f3e8,0xc9e5d72d90a2741e, + 0x8865899617fb1871,0x7e2fa67c7a658892, + 0xaa7eebfb9df9de8d,0xddbb901b98feeab7, + 0xd51ea6fa85785631,0x552a74227f3ea565, + 0x8533285c936b35de,0xd53a88958f87275f, + 0xa67ff273b8460356,0x8a892abaf368f137, + 0xd01fef10a657842c,0x2d2b7569b0432d85, + 0x8213f56a67f6b29b,0x9c3b29620e29fc73, + 0xa298f2c501f45f42,0x8349f3ba91b47b8f, + 0xcb3f2f7642717713,0x241c70a936219a73, + 0xfe0efb53d30dd4d7,0xed238cd383aa0110, + 0x9ec95d1463e8a506,0xf4363804324a40aa, + 0xc67bb4597ce2ce48,0xb143c6053edcd0d5, + 0xf81aa16fdc1b81da,0xdd94b7868e94050a, + 0x9b10a4e5e9913128,0xca7cf2b4191c8326, + 0xc1d4ce1f63f57d72,0xfd1c2f611f63a3f0, + 0xf24a01a73cf2dccf,0xbc633b39673c8cec, + 0x976e41088617ca01,0xd5be0503e085d813, + 0xbd49d14aa79dbc82,0x4b2d8644d8a74e18, + 0xec9c459d51852ba2,0xddf8e7d60ed1219e, + 0x93e1ab8252f33b45,0xcabb90e5c942b503, + 0xb8da1662e7b00a17,0x3d6a751f3b936243, + 0xe7109bfba19c0c9d,0xcc512670a783ad4, + 0x906a617d450187e2,0x27fb2b80668b24c5, + 0xb484f9dc9641e9da,0xb1f9f660802dedf6, + 0xe1a63853bbd26451,0x5e7873f8a0396973, + 0x8d07e33455637eb2,0xdb0b487b6423e1e8, + 0xb049dc016abc5e5f,0x91ce1a9a3d2cda62, + 0xdc5c5301c56b75f7,0x7641a140cc7810fb, + 0x89b9b3e11b6329ba,0xa9e904c87fcb0a9d, + 0xac2820d9623bf429,0x546345fa9fbdcd44, + 0xd732290fbacaf133,0xa97c177947ad4095, + 0x867f59a9d4bed6c0,0x49ed8eabcccc485d, + 0xa81f301449ee8c70,0x5c68f256bfff5a74, + 0xd226fc195c6a2f8c,0x73832eec6fff3111, + 0x83585d8fd9c25db7,0xc831fd53c5ff7eab, + 0xa42e74f3d032f525,0xba3e7ca8b77f5e55, + 0xcd3a1230c43fb26f,0x28ce1bd2e55f35eb, + 0x80444b5e7aa7cf85,0x7980d163cf5b81b3, + 0xa0555e361951c366,0xd7e105bcc332621f, + 0xc86ab5c39fa63440,0x8dd9472bf3fefaa7, + 0xfa856334878fc150,0xb14f98f6f0feb951, + 0x9c935e00d4b9d8d2,0x6ed1bf9a569f33d3, + 0xc3b8358109e84f07,0xa862f80ec4700c8, + 0xf4a642e14c6262c8,0xcd27bb612758c0fa, + 0x98e7e9cccfbd7dbd,0x8038d51cb897789c, + 0xbf21e44003acdd2c,0xe0470a63e6bd56c3, + 0xeeea5d5004981478,0x1858ccfce06cac74, + 0x95527a5202df0ccb,0xf37801e0c43ebc8, + 0xbaa718e68396cffd,0xd30560258f54e6ba, + 0xe950df20247c83fd,0x47c6b82ef32a2069, + 0x91d28b7416cdd27e,0x4cdc331d57fa5441, + 0xb6472e511c81471d,0xe0133fe4adf8e952, + 0xe3d8f9e563a198e5,0x58180fddd97723a6, + 0x8e679c2f5e44ff8f,0x570f09eaa7ea7648,}; +using powers = powers_template<>; + +} + +#endif + +#ifndef FASTFLOAT_DECIMAL_TO_BINARY_H +#define FASTFLOAT_DECIMAL_TO_BINARY_H + +#include <cfloat> +#include <cinttypes> +#include <cmath> +#include <cstdint> +#include <cstdlib> +#include <cstring> + +namespace fast_float { + +// This will compute or rather approximate w * 5**q and return a pair of 64-bit words approximating +// the result, with the "high" part corresponding to the most significant bits and the +// low part corresponding to the least significant bits. +// +template <int bit_precision> +fastfloat_really_inline +value128 compute_product_approximation(int64_t q, uint64_t w) { + const int index = 2 * int(q - powers::smallest_power_of_five); + // For small values of q, e.g., q in [0,27], the answer is always exact because + // The line value128 firstproduct = full_multiplication(w, power_of_five_128[index]); + // gives the exact answer. + value128 firstproduct = full_multiplication(w, powers::power_of_five_128[index]); + static_assert((bit_precision >= 0) && (bit_precision <= 64), " precision should be in (0,64]"); + constexpr uint64_t precision_mask = (bit_precision < 64) ? + (uint64_t(0xFFFFFFFFFFFFFFFF) >> bit_precision) + : uint64_t(0xFFFFFFFFFFFFFFFF); + if((firstproduct.high & precision_mask) == precision_mask) { // could further guard with (lower + w < lower) + // regarding the second product, we only need secondproduct.high, but our expectation is that the compiler will optimize this extra work away if needed. + value128 secondproduct = full_multiplication(w, powers::power_of_five_128[index + 1]); + firstproduct.low += secondproduct.high; + if(secondproduct.high > firstproduct.low) { + firstproduct.high++; + } + } + return firstproduct; +} + +namespace detail { +/** + * For q in (0,350), we have that + * f = (((152170 + 65536) * q ) >> 16); + * is equal to + * floor(p) + q + * where + * p = log(5**q)/log(2) = q * log(5)/log(2) + * + * For negative values of q in (-400,0), we have that + * f = (((152170 + 65536) * q ) >> 16); + * is equal to + * -ceil(p) + q + * where + * p = log(5**-q)/log(2) = -q * log(5)/log(2) + */ + constexpr fastfloat_really_inline int32_t power(int32_t q) noexcept { + return (((152170 + 65536) * q) >> 16) + 63; + } +} // namespace detail + +// create an adjusted mantissa, biased by the invalid power2 +// for significant digits already multiplied by 10 ** q. +template <typename binary> +fastfloat_really_inline +adjusted_mantissa compute_error_scaled(int64_t q, uint64_t w, int lz) noexcept { + int hilz = int(w >> 63) ^ 1; + adjusted_mantissa answer; + answer.mantissa = w << hilz; + int bias = binary::mantissa_explicit_bits() - binary::minimum_exponent(); + answer.power2 = int32_t(detail::power(int32_t(q)) + bias - hilz - lz - 62 + invalid_am_bias); + return answer; +} + +// w * 10 ** q, without rounding the representation up. +// the power2 in the exponent will be adjusted by invalid_am_bias. +template <typename binary> +fastfloat_really_inline +adjusted_mantissa compute_error(int64_t q, uint64_t w) noexcept { + int lz = leading_zeroes(w); + w <<= lz; + value128 product = compute_product_approximation<binary::mantissa_explicit_bits() + 3>(q, w); + return compute_error_scaled<binary>(q, product.high, lz); +} + +// w * 10 ** q +// The returned value should be a valid ieee64 number that simply need to be packed. +// However, in some very rare cases, the computation will fail. In such cases, we +// return an adjusted_mantissa with a negative power of 2: the caller should recompute +// in such cases. +template <typename binary> +fastfloat_really_inline +adjusted_mantissa compute_float(int64_t q, uint64_t w) noexcept { + adjusted_mantissa answer; + if ((w == 0) || (q < binary::smallest_power_of_ten())) { + answer.power2 = 0; + answer.mantissa = 0; + // result should be zero + return answer; + } + if (q > binary::largest_power_of_ten()) { + // we want to get infinity: + answer.power2 = binary::infinite_power(); + answer.mantissa = 0; + return answer; + } + // At this point in time q is in [powers::smallest_power_of_five, powers::largest_power_of_five]. + + // We want the most significant bit of i to be 1. Shift if needed. + int lz = leading_zeroes(w); + w <<= lz; + + // The required precision is binary::mantissa_explicit_bits() + 3 because + // 1. We need the implicit bit + // 2. We need an extra bit for rounding purposes + // 3. We might lose a bit due to the "upperbit" routine (result too small, requiring a shift) + + value128 product = compute_product_approximation<binary::mantissa_explicit_bits() + 3>(q, w); + if(product.low == 0xFFFFFFFFFFFFFFFF) { // could guard it further + // In some very rare cases, this could happen, in which case we might need a more accurate + // computation that what we can provide cheaply. This is very, very unlikely. + // + const bool inside_safe_exponent = (q >= -27) && (q <= 55); // always good because 5**q <2**128 when q>=0, + // and otherwise, for q<0, we have 5**-q<2**64 and the 128-bit reciprocal allows for exact computation. + if(!inside_safe_exponent) { + return compute_error_scaled<binary>(q, product.high, lz); + } + } + // The "compute_product_approximation" function can be slightly slower than a branchless approach: + // value128 product = compute_product(q, w); + // but in practice, we can win big with the compute_product_approximation if its additional branch + // is easily predicted. Which is best is data specific. + int upperbit = int(product.high >> 63); + + answer.mantissa = product.high >> (upperbit + 64 - binary::mantissa_explicit_bits() - 3); + + answer.power2 = int32_t(detail::power(int32_t(q)) + upperbit - lz - binary::minimum_exponent()); + if (answer.power2 <= 0) { // we have a subnormal? + // Here have that answer.power2 <= 0 so -answer.power2 >= 0 + if(-answer.power2 + 1 >= 64) { // if we have more than 64 bits below the minimum exponent, you have a zero for sure. + answer.power2 = 0; + answer.mantissa = 0; + // result should be zero + return answer; + } + // next line is safe because -answer.power2 + 1 < 64 + answer.mantissa >>= -answer.power2 + 1; + // Thankfully, we can't have both "round-to-even" and subnormals because + // "round-to-even" only occurs for powers close to 0. + answer.mantissa += (answer.mantissa & 1); // round up + answer.mantissa >>= 1; + // There is a weird scenario where we don't have a subnormal but just. + // Suppose we start with 2.2250738585072013e-308, we end up + // with 0x3fffffffffffff x 2^-1023-53 which is technically subnormal + // whereas 0x40000000000000 x 2^-1023-53 is normal. Now, we need to round + // up 0x3fffffffffffff x 2^-1023-53 and once we do, we are no longer + // subnormal, but we can only know this after rounding. + // So we only declare a subnormal if we are smaller than the threshold. + answer.power2 = (answer.mantissa < (uint64_t(1) << binary::mantissa_explicit_bits())) ? 0 : 1; + return answer; + } + + // usually, we round *up*, but if we fall right in between and and we have an + // even basis, we need to round down + // We are only concerned with the cases where 5**q fits in single 64-bit word. + if ((product.low <= 1) && (q >= binary::min_exponent_round_to_even()) && (q <= binary::max_exponent_round_to_even()) && + ((answer.mantissa & 3) == 1) ) { // we may fall between two floats! + // To be in-between two floats we need that in doing + // answer.mantissa = product.high >> (upperbit + 64 - binary::mantissa_explicit_bits() - 3); + // ... we dropped out only zeroes. But if this happened, then we can go back!!! + if((answer.mantissa << (upperbit + 64 - binary::mantissa_explicit_bits() - 3)) == product.high) { + answer.mantissa &= ~uint64_t(1); // flip it so that we do not round up + } + } + + answer.mantissa += (answer.mantissa & 1); // round up + answer.mantissa >>= 1; + if (answer.mantissa >= (uint64_t(2) << binary::mantissa_explicit_bits())) { + answer.mantissa = (uint64_t(1) << binary::mantissa_explicit_bits()); + answer.power2++; // undo previous addition + } + + answer.mantissa &= ~(uint64_t(1) << binary::mantissa_explicit_bits()); + if (answer.power2 >= binary::infinite_power()) { // infinity + answer.power2 = binary::infinite_power(); + answer.mantissa = 0; + } + return answer; +} + +} // namespace fast_float + +#endif + +#ifndef FASTFLOAT_BIGINT_H +#define FASTFLOAT_BIGINT_H + +#include <algorithm> +#include <cstdint> +#include <climits> +#include <cstring> + + +namespace fast_float { + +// the limb width: we want efficient multiplication of double the bits in +// limb, or for 64-bit limbs, at least 64-bit multiplication where we can +// extract the high and low parts efficiently. this is every 64-bit +// architecture except for sparc, which emulates 128-bit multiplication. +// we might have platforms where `CHAR_BIT` is not 8, so let's avoid +// doing `8 * sizeof(limb)`. +#if defined(FASTFLOAT_64BIT) && !defined(__sparc) +#define FASTFLOAT_64BIT_LIMB +typedef uint64_t limb; +constexpr size_t limb_bits = 64; +#else +#define FASTFLOAT_32BIT_LIMB +typedef uint32_t limb; +constexpr size_t limb_bits = 32; +#endif + +typedef span<limb> limb_span; + +// number of bits in a bigint. this needs to be at least the number +// of bits required to store the largest bigint, which is +// `log2(10**(digits + max_exp))`, or `log2(10**(767 + 342))`, or +// ~3600 bits, so we round to 4000. +constexpr size_t bigint_bits = 4000; +constexpr size_t bigint_limbs = bigint_bits / limb_bits; + +// vector-like type that is allocated on the stack. the entire +// buffer is pre-allocated, and only the length changes. +template <uint16_t size> +struct stackvec { + limb data[size]; + // we never need more than 150 limbs + uint16_t length{0}; + + stackvec() = default; + stackvec(const stackvec &) = delete; + stackvec &operator=(const stackvec &) = delete; + stackvec(stackvec &&) = delete; + stackvec &operator=(stackvec &&other) = delete; + + // create stack vector from existing limb span. + stackvec(limb_span s) { + FASTFLOAT_ASSERT(try_extend(s)); + } + + limb& operator[](size_t index) noexcept { + FASTFLOAT_DEBUG_ASSERT(index < length); + return data[index]; + } + const limb& operator[](size_t index) const noexcept { + FASTFLOAT_DEBUG_ASSERT(index < length); + return data[index]; + } + // index from the end of the container + const limb& rindex(size_t index) const noexcept { + FASTFLOAT_DEBUG_ASSERT(index < length); + size_t rindex = length - index - 1; + return data[rindex]; + } + + // set the length, without bounds checking. + void set_len(size_t len) noexcept { + length = uint16_t(len); + } + constexpr size_t len() const noexcept { + return length; + } + constexpr bool is_empty() const noexcept { + return length == 0; + } + constexpr size_t capacity() const noexcept { + return size; + } + // append item to vector, without bounds checking + void push_unchecked(limb value) noexcept { + data[length] = value; + length++; + } + // append item to vector, returning if item was added + bool try_push(limb value) noexcept { + if (len() < capacity()) { + push_unchecked(value); + return true; + } else { + return false; + } + } + // add items to the vector, from a span, without bounds checking + void extend_unchecked(limb_span s) noexcept { + limb* ptr = data + length; + ::memcpy((void*)ptr, (const void*)s.ptr, sizeof(limb) * s.len()); + set_len(len() + s.len()); + } + // try to add items to the vector, returning if items were added + bool try_extend(limb_span s) noexcept { + if (len() + s.len() <= capacity()) { + extend_unchecked(s); + return true; + } else { + return false; + } + } + // resize the vector, without bounds checking + // if the new size is longer than the vector, assign value to each + // appended item. + void resize_unchecked(size_t new_len, limb value) noexcept { + if (new_len > len()) { + size_t count = new_len - len(); + limb* first = data + len(); + limb* last = first + count; + ::std::fill(first, last, value); + set_len(new_len); + } else { + set_len(new_len); + } + } + // try to resize the vector, returning if the vector was resized. + bool try_resize(size_t new_len, limb value) noexcept { + if (new_len > capacity()) { + return false; + } else { + resize_unchecked(new_len, value); + return true; + } + } + // check if any limbs are non-zero after the given index. + // this needs to be done in reverse order, since the index + // is relative to the most significant limbs. + bool nonzero(size_t index) const noexcept { + while (index < len()) { + if (rindex(index) != 0) { + return true; + } + index++; + } + return false; + } + // normalize the big integer, so most-significant zero limbs are removed. + void normalize() noexcept { + while (len() > 0 && rindex(0) == 0) { + length--; + } + } +}; + +fastfloat_really_inline +uint64_t empty_hi64(bool& truncated) noexcept { + truncated = false; + return 0; +} + +fastfloat_really_inline +uint64_t uint64_hi64(uint64_t r0, bool& truncated) noexcept { + truncated = false; + int shl = leading_zeroes(r0); + return r0 << shl; +} + +fastfloat_really_inline +uint64_t uint64_hi64(uint64_t r0, uint64_t r1, bool& truncated) noexcept { + int shl = leading_zeroes(r0); + if (shl == 0) { + truncated = r1 != 0; + return r0; + } else { + int shr = 64 - shl; + truncated = (r1 << shl) != 0; + return (r0 << shl) | (r1 >> shr); + } +} + +fastfloat_really_inline +uint64_t uint32_hi64(uint32_t r0, bool& truncated) noexcept { + return uint64_hi64(r0, truncated); +} + +fastfloat_really_inline +uint64_t uint32_hi64(uint32_t r0, uint32_t r1, bool& truncated) noexcept { + uint64_t x0 = r0; + uint64_t x1 = r1; + return uint64_hi64((x0 << 32) | x1, truncated); +} + +fastfloat_really_inline +uint64_t uint32_hi64(uint32_t r0, uint32_t r1, uint32_t r2, bool& truncated) noexcept { + uint64_t x0 = r0; + uint64_t x1 = r1; + uint64_t x2 = r2; + return uint64_hi64(x0, (x1 << 32) | x2, truncated); +} + +// add two small integers, checking for overflow. +// we want an efficient operation. for msvc, where +// we don't have built-in intrinsics, this is still +// pretty fast. +fastfloat_really_inline +limb scalar_add(limb x, limb y, bool& overflow) noexcept { + limb z; + +// gcc and clang +#if defined(__has_builtin) + #if __has_builtin(__builtin_add_overflow) + overflow = __builtin_add_overflow(x, y, &z); + return z; + #endif +#endif + + // generic, this still optimizes correctly on MSVC. + z = x + y; + overflow = z < x; + return z; +} + +// multiply two small integers, getting both the high and low bits. +fastfloat_really_inline +limb scalar_mul(limb x, limb y, limb& carry) noexcept { +#ifdef FASTFLOAT_64BIT_LIMB + #if defined(__SIZEOF_INT128__) + // GCC and clang both define it as an extension. + __uint128_t z = __uint128_t(x) * __uint128_t(y) + __uint128_t(carry); + carry = limb(z >> limb_bits); + return limb(z); + #else + // fallback, no native 128-bit integer multiplication with carry. + // on msvc, this optimizes identically, somehow. + value128 z = full_multiplication(x, y); + bool overflow; + z.low = scalar_add(z.low, carry, overflow); + z.high += uint64_t(overflow); // cannot overflow + carry = z.high; + return z.low; + #endif +#else + uint64_t z = uint64_t(x) * uint64_t(y) + uint64_t(carry); + carry = limb(z >> limb_bits); + return limb(z); +#endif +} + +// add scalar value to bigint starting from offset. +// used in grade school multiplication +template <uint16_t size> +inline bool small_add_from(stackvec<size>& vec, limb y, size_t start) noexcept { + size_t index = start; + limb carry = y; + bool overflow; + while (carry != 0 && index < vec.len()) { + vec[index] = scalar_add(vec[index], carry, overflow); + carry = limb(overflow); + index += 1; + } + if (carry != 0) { + FASTFLOAT_TRY(vec.try_push(carry)); + } + return true; +} + +// add scalar value to bigint. +template <uint16_t size> +fastfloat_really_inline bool small_add(stackvec<size>& vec, limb y) noexcept { + return small_add_from(vec, y, 0); +} + +// multiply bigint by scalar value. +template <uint16_t size> +inline bool small_mul(stackvec<size>& vec, limb y) noexcept { + limb carry = 0; + for (size_t index = 0; index < vec.len(); index++) { + vec[index] = scalar_mul(vec[index], y, carry); + } + if (carry != 0) { + FASTFLOAT_TRY(vec.try_push(carry)); + } + return true; +} + +// add bigint to bigint starting from index. +// used in grade school multiplication +template <uint16_t size> +bool large_add_from(stackvec<size>& x, limb_span y, size_t start) noexcept { + // the effective x buffer is from `xstart..x.len()`, so exit early + // if we can't get that current range. + if (x.len() < start || y.len() > x.len() - start) { + FASTFLOAT_TRY(x.try_resize(y.len() + start, 0)); + } + + bool carry = false; + for (size_t index = 0; index < y.len(); index++) { + limb xi = x[index + start]; + limb yi = y[index]; + bool c1 = false; + bool c2 = false; + xi = scalar_add(xi, yi, c1); + if (carry) { + xi = scalar_add(xi, 1, c2); + } + x[index + start] = xi; + carry = c1 | c2; + } + + // handle overflow + if (carry) { + FASTFLOAT_TRY(small_add_from(x, 1, y.len() + start)); + } + return true; +} + +// add bigint to bigint. +template <uint16_t size> +fastfloat_really_inline bool large_add_from(stackvec<size>& x, limb_span y) noexcept { + return large_add_from(x, y, 0); +} + +// grade-school multiplication algorithm +template <uint16_t size> +bool long_mul(stackvec<size>& x, limb_span y) noexcept { + limb_span xs = limb_span(x.data, x.len()); + stackvec<size> z(xs); + limb_span zs = limb_span(z.data, z.len()); + + if (y.len() != 0) { + limb y0 = y[0]; + FASTFLOAT_TRY(small_mul(x, y0)); + for (size_t index = 1; index < y.len(); index++) { + limb yi = y[index]; + stackvec<size> zi; + if (yi != 0) { + // re-use the same buffer throughout + zi.set_len(0); + FASTFLOAT_TRY(zi.try_extend(zs)); + FASTFLOAT_TRY(small_mul(zi, yi)); + limb_span zis = limb_span(zi.data, zi.len()); + FASTFLOAT_TRY(large_add_from(x, zis, index)); + } + } + } + + x.normalize(); + return true; +} + +// grade-school multiplication algorithm +template <uint16_t size> +bool large_mul(stackvec<size>& x, limb_span y) noexcept { + if (y.len() == 1) { + FASTFLOAT_TRY(small_mul(x, y[0])); + } else { + FASTFLOAT_TRY(long_mul(x, y)); + } + return true; +} + +// big integer type. implements a small subset of big integer +// arithmetic, using simple algorithms since asymptotically +// faster algorithms are slower for a small number of limbs. +// all operations assume the big-integer is normalized. +struct bigint { + // storage of the limbs, in little-endian order. + stackvec<bigint_limbs> vec; + + bigint(): vec() {} + bigint(const bigint &) = delete; + bigint &operator=(const bigint &) = delete; + bigint(bigint &&) = delete; + bigint &operator=(bigint &&other) = delete; + + bigint(uint64_t value): vec() { +#ifdef FASTFLOAT_64BIT_LIMB + vec.push_unchecked(value); +#else + vec.push_unchecked(uint32_t(value)); + vec.push_unchecked(uint32_t(value >> 32)); +#endif + vec.normalize(); + } + + // get the high 64 bits from the vector, and if bits were truncated. + // this is to get the significant digits for the float. + uint64_t hi64(bool& truncated) const noexcept { +#ifdef FASTFLOAT_64BIT_LIMB + if (vec.len() == 0) { + return empty_hi64(truncated); + } else if (vec.len() == 1) { + return uint64_hi64(vec.rindex(0), truncated); + } else { + uint64_t result = uint64_hi64(vec.rindex(0), vec.rindex(1), truncated); + truncated |= vec.nonzero(2); + return result; + } +#else + if (vec.len() == 0) { + return empty_hi64(truncated); + } else if (vec.len() == 1) { + return uint32_hi64(vec.rindex(0), truncated); + } else if (vec.len() == 2) { + return uint32_hi64(vec.rindex(0), vec.rindex(1), truncated); + } else { + uint64_t result = uint32_hi64(vec.rindex(0), vec.rindex(1), vec.rindex(2), truncated); + truncated |= vec.nonzero(3); + return result; + } +#endif + } + + // compare two big integers, returning the large value. + // assumes both are normalized. if the return value is + // negative, other is larger, if the return value is + // positive, this is larger, otherwise they are equal. + // the limbs are stored in little-endian order, so we + // must compare the limbs in ever order. + int compare(const bigint& other) const noexcept { + if (vec.len() > other.vec.len()) { + return 1; + } else if (vec.len() < other.vec.len()) { + return -1; + } else { + for (size_t index = vec.len(); index > 0; index--) { + limb xi = vec[index - 1]; + limb yi = other.vec[index - 1]; + if (xi > yi) { + return 1; + } else if (xi < yi) { + return -1; + } + } + return 0; + } + } + + // shift left each limb n bits, carrying over to the new limb + // returns true if we were able to shift all the digits. + bool shl_bits(size_t n) noexcept { + // Internally, for each item, we shift left by n, and add the previous + // right shifted limb-bits. + // For example, we transform (for u8) shifted left 2, to: + // b10100100 b01000010 + // b10 b10010001 b00001000 + FASTFLOAT_DEBUG_ASSERT(n != 0); + FASTFLOAT_DEBUG_ASSERT(n < sizeof(limb) * 8); + + size_t shl = n; + size_t shr = limb_bits - shl; + limb prev = 0; + for (size_t index = 0; index < vec.len(); index++) { + limb xi = vec[index]; + vec[index] = (xi << shl) | (prev >> shr); + prev = xi; + } + + limb carry = prev >> shr; + if (carry != 0) { + return vec.try_push(carry); + } + return true; + } + + // move the limbs left by `n` limbs. + bool shl_limbs(size_t n) noexcept { + FASTFLOAT_DEBUG_ASSERT(n != 0); + if (n + vec.len() > vec.capacity()) { + return false; + } else if (!vec.is_empty()) { + // move limbs + limb* dst = vec.data + n; + const limb* src = vec.data; + ::memmove(dst, src, sizeof(limb) * vec.len()); + // fill in empty limbs + limb* first = vec.data; + limb* last = first + n; + ::std::fill(first, last, 0); + vec.set_len(n + vec.len()); + return true; + } else { + return true; + } + } + + // move the limbs left by `n` bits. + bool shl(size_t n) noexcept { + size_t rem = n % limb_bits; + size_t div = n / limb_bits; + if (rem != 0) { + FASTFLOAT_TRY(shl_bits(rem)); + } + if (div != 0) { + FASTFLOAT_TRY(shl_limbs(div)); + } + return true; + } + + // get the number of leading zeros in the bigint. + int ctlz() const noexcept { + if (vec.is_empty()) { + return 0; + } else { +#ifdef FASTFLOAT_64BIT_LIMB + return leading_zeroes(vec.rindex(0)); +#else + // no use defining a specialized leading_zeroes for a 32-bit type. + uint64_t r0 = vec.rindex(0); + return leading_zeroes(r0 << 32); +#endif + } + } + + // get the number of bits in the bigint. + int bit_length() const noexcept { + int lz = ctlz(); + return int(limb_bits * vec.len()) - lz; + } + + bool mul(limb y) noexcept { + return small_mul(vec, y); + } + + bool add(limb y) noexcept { + return small_add(vec, y); + } + + // multiply as if by 2 raised to a power. + bool pow2(uint32_t exp) noexcept { + return shl(exp); + } + + // multiply as if by 5 raised to a power. + bool pow5(uint32_t exp) noexcept { + // multiply by a power of 5 + static constexpr uint32_t large_step = 135; + static constexpr uint64_t small_power_of_5[] = { + 1UL, 5UL, 25UL, 125UL, 625UL, 3125UL, 15625UL, 78125UL, 390625UL, + 1953125UL, 9765625UL, 48828125UL, 244140625UL, 1220703125UL, + 6103515625UL, 30517578125UL, 152587890625UL, 762939453125UL, + 3814697265625UL, 19073486328125UL, 95367431640625UL, 476837158203125UL, + 2384185791015625UL, 11920928955078125UL, 59604644775390625UL, + 298023223876953125UL, 1490116119384765625UL, 7450580596923828125UL, + }; +#ifdef FASTFLOAT_64BIT_LIMB + constexpr static limb large_power_of_5[] = { + 1414648277510068013UL, 9180637584431281687UL, 4539964771860779200UL, + 10482974169319127550UL, 198276706040285095UL}; +#else + constexpr static limb large_power_of_5[] = { + 4279965485U, 329373468U, 4020270615U, 2137533757U, 4287402176U, + 1057042919U, 1071430142U, 2440757623U, 381945767U, 46164893U}; +#endif + size_t large_length = sizeof(large_power_of_5) / sizeof(limb); + limb_span large = limb_span(large_power_of_5, large_length); + while (exp >= large_step) { + FASTFLOAT_TRY(large_mul(vec, large)); + exp -= large_step; + } +#ifdef FASTFLOAT_64BIT_LIMB + uint32_t small_step = 27; + limb max_native = 7450580596923828125UL; +#else + uint32_t small_step = 13; + limb max_native = 1220703125U; +#endif + while (exp >= small_step) { + FASTFLOAT_TRY(small_mul(vec, max_native)); + exp -= small_step; + } + if (exp != 0) { + FASTFLOAT_TRY(small_mul(vec, limb(small_power_of_5[exp]))); + } + + return true; + } + + // multiply as if by 10 raised to a power. + bool pow10(uint32_t exp) noexcept { + FASTFLOAT_TRY(pow5(exp)); + return pow2(exp); + } +}; + +} // namespace fast_float + +#endif + +#ifndef FASTFLOAT_ASCII_NUMBER_H +#define FASTFLOAT_ASCII_NUMBER_H + +#include <cctype> +#include <cstdint> +#include <cstring> +#include <iterator> + + +namespace fast_float { + +// Next function can be micro-optimized, but compilers are entirely +// able to optimize it well. +fastfloat_really_inline bool is_integer(char c) noexcept { return c >= '0' && c <= '9'; } + +fastfloat_really_inline uint64_t byteswap(uint64_t val) { + return (val & 0xFF00000000000000) >> 56 + | (val & 0x00FF000000000000) >> 40 + | (val & 0x0000FF0000000000) >> 24 + | (val & 0x000000FF00000000) >> 8 + | (val & 0x00000000FF000000) << 8 + | (val & 0x0000000000FF0000) << 24 + | (val & 0x000000000000FF00) << 40 + | (val & 0x00000000000000FF) << 56; +} + +fastfloat_really_inline uint64_t read_u64(const char *chars) { + uint64_t val; + ::memcpy(&val, chars, sizeof(uint64_t)); +#if FASTFLOAT_IS_BIG_ENDIAN == 1 + // Need to read as-if the number was in little-endian order. + val = byteswap(val); +#endif + return val; +} + +fastfloat_really_inline void write_u64(uint8_t *chars, uint64_t val) { +#if FASTFLOAT_IS_BIG_ENDIAN == 1 + // Need to read as-if the number was in little-endian order. + val = byteswap(val); +#endif + ::memcpy(chars, &val, sizeof(uint64_t)); +} + +// credit @aqrit +fastfloat_really_inline uint32_t parse_eight_digits_unrolled(uint64_t val) { + const uint64_t mask = 0x000000FF000000FF; + const uint64_t mul1 = 0x000F424000000064; // 100 + (1000000ULL << 32) + const uint64_t mul2 = 0x0000271000000001; // 1 + (10000ULL << 32) + val -= 0x3030303030303030; + val = (val * 10) + (val >> 8); // val = (val * 2561) >> 8; + val = (((val & mask) * mul1) + (((val >> 16) & mask) * mul2)) >> 32; + return uint32_t(val); +} + +fastfloat_really_inline uint32_t parse_eight_digits_unrolled(const char *chars) noexcept { + return parse_eight_digits_unrolled(read_u64(chars)); +} + +// credit @aqrit +fastfloat_really_inline bool is_made_of_eight_digits_fast(uint64_t val) noexcept { + return !((((val + 0x4646464646464646) | (val - 0x3030303030303030)) & + 0x8080808080808080)); +} + +fastfloat_really_inline bool is_made_of_eight_digits_fast(const char *chars) noexcept { + return is_made_of_eight_digits_fast(read_u64(chars)); +} + +typedef span<const char> byte_span; + +struct parsed_number_string { + int64_t exponent{0}; + uint64_t mantissa{0}; + const char *lastmatch{nullptr}; + bool negative{false}; + bool valid{false}; + bool too_many_digits{false}; + // contains the range of the significant digits + byte_span integer{}; // non-nullable + byte_span fraction{}; // nullable +}; + +// Assuming that you use no more than 19 digits, this will +// parse an ASCII string. +fastfloat_really_inline +parsed_number_string parse_number_string(const char *p, const char *pend, parse_options options) noexcept { + const chars_format fmt = options.format; + const char decimal_point = options.decimal_point; + + parsed_number_string answer; + answer.valid = false; + answer.too_many_digits = false; + answer.negative = (*p == '-'); + if (*p == '-') { // C++17 20.19.3.(7.1) explicitly forbids '+' sign here + ++p; + if (p == pend) { + return answer; + } + if (!is_integer(*p) && (*p != decimal_point)) { // a sign must be followed by an integer or the dot + return answer; + } + } + const char *const start_digits = p; + + uint64_t i = 0; // an unsigned int avoids signed overflows (which are bad) + + while ((std::distance(p, pend) >= 8) && is_made_of_eight_digits_fast(p)) { + i = i * 100000000 + parse_eight_digits_unrolled(p); // in rare cases, this will overflow, but that's ok + p += 8; + } + while ((p != pend) && is_integer(*p)) { + // a multiplication by 10 is cheaper than an arbitrary integer + // multiplication + i = 10 * i + + uint64_t(*p - '0'); // might overflow, we will handle the overflow later + ++p; + } + const char *const end_of_integer_part = p; + int64_t digit_count = int64_t(end_of_integer_part - start_digits); + answer.integer = byte_span(start_digits, size_t(digit_count)); + int64_t exponent = 0; + if ((p != pend) && (*p == decimal_point)) { + ++p; + const char* before = p; + // can occur at most twice without overflowing, but let it occur more, since + // for integers with many digits, digit parsing is the primary bottleneck. + while ((std::distance(p, pend) >= 8) && is_made_of_eight_digits_fast(p)) { + i = i * 100000000 + parse_eight_digits_unrolled(p); // in rare cases, this will overflow, but that's ok + p += 8; + } + while ((p != pend) && is_integer(*p)) { + uint8_t digit = uint8_t(*p - '0'); + ++p; + i = i * 10 + digit; // in rare cases, this will overflow, but that's ok + } + exponent = before - p; + answer.fraction = byte_span(before, size_t(p - before)); + digit_count -= exponent; + } + // we must have encountered at least one integer! + if (digit_count == 0) { + return answer; + } + int64_t exp_number = 0; // explicit exponential part + if ((fmt & chars_format::scientific) && (p != pend) && (('e' == *p) || ('E' == *p))) { + const char * location_of_e = p; + ++p; + bool neg_exp = false; + if ((p != pend) && ('-' == *p)) { + neg_exp = true; + ++p; + } else if ((p != pend) && ('+' == *p)) { // '+' on exponent is allowed by C++17 20.19.3.(7.1) + ++p; + } + if ((p == pend) || !is_integer(*p)) { + if(!(fmt & chars_format::fixed)) { + // We are in error. + return answer; + } + // Otherwise, we will be ignoring the 'e'. + p = location_of_e; + } else { + while ((p != pend) && is_integer(*p)) { + uint8_t digit = uint8_t(*p - '0'); + if (exp_number < 0x10000000) { + exp_number = 10 * exp_number + digit; + } + ++p; + } + if(neg_exp) { exp_number = - exp_number; } + exponent += exp_number; + } + } else { + // If it scientific and not fixed, we have to bail out. + if((fmt & chars_format::scientific) && !(fmt & chars_format::fixed)) { return answer; } + } + answer.lastmatch = p; + answer.valid = true; + + // If we frequently had to deal with long strings of digits, + // we could extend our code by using a 128-bit integer instead + // of a 64-bit integer. However, this is uncommon. + // + // We can deal with up to 19 digits. + if (digit_count > 19) { // this is uncommon + // It is possible that the integer had an overflow. + // We have to handle the case where we have 0.0000somenumber. + // We need to be mindful of the case where we only have zeroes... + // E.g., 0.000000000...000. + const char *start = start_digits; + while ((start != pend) && (*start == '0' || *start == decimal_point)) { + if(*start == '0') { digit_count --; } + start++; + } + if (digit_count > 19) { + answer.too_many_digits = true; + // Let us start again, this time, avoiding overflows. + // We don't need to check if is_integer, since we use the + // pre-tokenized spans from above. + i = 0; + p = answer.integer.ptr; + const char* int_end = p + answer.integer.len(); + const uint64_t minimal_nineteen_digit_integer{1000000000000000000}; + while((i < minimal_nineteen_digit_integer) && (p != int_end)) { + i = i * 10 + uint64_t(*p - '0'); + ++p; + } + if (i >= minimal_nineteen_digit_integer) { // We have a big integers + exponent = end_of_integer_part - p + exp_number; + } else { // We have a value with a fractional component. + p = answer.fraction.ptr; + const char* frac_end = p + answer.fraction.len(); + while((i < minimal_nineteen_digit_integer) && (p != frac_end)) { + i = i * 10 + uint64_t(*p - '0'); + ++p; + } + exponent = answer.fraction.ptr - p + exp_number; + } + // We have now corrected both exponent and i, to a truncated value + } + } + answer.exponent = exponent; + answer.mantissa = i; + return answer; +} + +} // namespace fast_float + +#endif + +#ifndef FASTFLOAT_DIGIT_COMPARISON_H +#define FASTFLOAT_DIGIT_COMPARISON_H + +#include <algorithm> +#include <cstdint> +#include <cstring> +#include <iterator> + + +namespace fast_float { + +// 1e0 to 1e19 +constexpr static uint64_t powers_of_ten_uint64[] = { + 1UL, 10UL, 100UL, 1000UL, 10000UL, 100000UL, 1000000UL, 10000000UL, 100000000UL, + 1000000000UL, 10000000000UL, 100000000000UL, 1000000000000UL, 10000000000000UL, + 100000000000000UL, 1000000000000000UL, 10000000000000000UL, 100000000000000000UL, + 1000000000000000000UL, 10000000000000000000UL}; + +// calculate the exponent, in scientific notation, of the number. +// this algorithm is not even close to optimized, but it has no practical +// effect on performance: in order to have a faster algorithm, we'd need +// to slow down performance for faster algorithms, and this is still fast. +fastfloat_really_inline int32_t scientific_exponent(parsed_number_string& num) noexcept { + uint64_t mantissa = num.mantissa; + int32_t exponent = int32_t(num.exponent); + while (mantissa >= 10000) { + mantissa /= 10000; + exponent += 4; + } + while (mantissa >= 100) { + mantissa /= 100; + exponent += 2; + } + while (mantissa >= 10) { + mantissa /= 10; + exponent += 1; + } + return exponent; +} + +// this converts a native floating-point number to an extended-precision float. +template <typename T> +fastfloat_really_inline adjusted_mantissa to_extended(T value) noexcept { + using equiv_uint = typename binary_format<T>::equiv_uint; + constexpr equiv_uint exponent_mask = binary_format<T>::exponent_mask(); + constexpr equiv_uint mantissa_mask = binary_format<T>::mantissa_mask(); + constexpr equiv_uint hidden_bit_mask = binary_format<T>::hidden_bit_mask(); + + adjusted_mantissa am; + int32_t bias = binary_format<T>::mantissa_explicit_bits() - binary_format<T>::minimum_exponent(); + equiv_uint bits; + ::memcpy(&bits, &value, sizeof(T)); + if ((bits & exponent_mask) == 0) { + // denormal + am.power2 = 1 - bias; + am.mantissa = bits & mantissa_mask; + } else { + // normal + am.power2 = int32_t((bits & exponent_mask) >> binary_format<T>::mantissa_explicit_bits()); + am.power2 -= bias; + am.mantissa = (bits & mantissa_mask) | hidden_bit_mask; + } + + return am; +} + +// get the extended precision value of the halfway point between b and b+u. +// we are given a native float that represents b, so we need to adjust it +// halfway between b and b+u. +template <typename T> +fastfloat_really_inline adjusted_mantissa to_extended_halfway(T value) noexcept { + adjusted_mantissa am = to_extended(value); + am.mantissa <<= 1; + am.mantissa += 1; + am.power2 -= 1; + return am; +} + +// round an extended-precision float to the nearest machine float. +template <typename T, typename callback> +fastfloat_really_inline void round(adjusted_mantissa& am, callback cb) noexcept { + int32_t mantissa_shift = 64 - binary_format<T>::mantissa_explicit_bits() - 1; + if (-am.power2 >= mantissa_shift) { + // have a denormal float + int32_t shift = -am.power2 + 1; + cb(am, std::min(shift, 64)); + // check for round-up: if rounding-nearest carried us to the hidden bit. + am.power2 = (am.mantissa < (uint64_t(1) << binary_format<T>::mantissa_explicit_bits())) ? 0 : 1; + return; + } + + // have a normal float, use the default shift. + cb(am, mantissa_shift); + + // check for carry + if (am.mantissa >= (uint64_t(2) << binary_format<T>::mantissa_explicit_bits())) { + am.mantissa = (uint64_t(1) << binary_format<T>::mantissa_explicit_bits()); + am.power2++; + } + + // check for infinite: we could have carried to an infinite power + am.mantissa &= ~(uint64_t(1) << binary_format<T>::mantissa_explicit_bits()); + if (am.power2 >= binary_format<T>::infinite_power()) { + am.power2 = binary_format<T>::infinite_power(); + am.mantissa = 0; + } +} + +template <typename callback> +fastfloat_really_inline +void round_nearest_tie_even(adjusted_mantissa& am, int32_t shift, callback cb) noexcept { + uint64_t mask; + uint64_t halfway; + if (shift == 64) { + mask = UINT64_MAX; + } else { + mask = (uint64_t(1) << shift) - 1; + } + if (shift == 0) { + halfway = 0; + } else { + halfway = uint64_t(1) << (shift - 1); + } + uint64_t truncated_bits = am.mantissa & mask; + uint64_t is_above = truncated_bits > halfway; + uint64_t is_halfway = truncated_bits == halfway; + + // shift digits into position + if (shift == 64) { + am.mantissa = 0; + } else { + am.mantissa >>= shift; + } + am.power2 += shift; + + bool is_odd = (am.mantissa & 1) == 1; + am.mantissa += uint64_t(cb(is_odd, is_halfway, is_above)); +} + +fastfloat_really_inline void round_down(adjusted_mantissa& am, int32_t shift) noexcept { + if (shift == 64) { + am.mantissa = 0; + } else { + am.mantissa >>= shift; + } + am.power2 += shift; +} + +fastfloat_really_inline void skip_zeros(const char*& first, const char* last) noexcept { + uint64_t val; + while (std::distance(first, last) >= 8) { + ::memcpy(&val, first, sizeof(uint64_t)); + if (val != 0x3030303030303030) { + break; + } + first += 8; + } + while (first != last) { + if (*first != '0') { + break; + } + first++; + } +} + +// determine if any non-zero digits were truncated. +// all characters must be valid digits. +fastfloat_really_inline bool is_truncated(const char* first, const char* last) noexcept { + // do 8-bit optimizations, can just compare to 8 literal 0s. + uint64_t val; + while (std::distance(first, last) >= 8) { + ::memcpy(&val, first, sizeof(uint64_t)); + if (val != 0x3030303030303030) { + return true; + } + first += 8; + } + while (first != last) { + if (*first != '0') { + return true; + } + first++; + } + return false; +} + +fastfloat_really_inline bool is_truncated(byte_span s) noexcept { + return is_truncated(s.ptr, s.ptr + s.len()); +} + +fastfloat_really_inline +void parse_eight_digits(const char*& p, limb& value, size_t& counter, size_t& count) noexcept { + value = value * 100000000 + parse_eight_digits_unrolled(p); + p += 8; + counter += 8; + count += 8; +} + +fastfloat_really_inline +void parse_one_digit(const char*& p, limb& value, size_t& counter, size_t& count) noexcept { + value = value * 10 + limb(*p - '0'); + p++; + counter++; + count++; +} + +fastfloat_really_inline +void add_native(bigint& big, limb power, limb value) noexcept { + big.mul(power); + big.add(value); +} + +fastfloat_really_inline void round_up_bigint(bigint& big, size_t& count) noexcept { + // need to round-up the digits, but need to avoid rounding + // ....9999 to ...10000, which could cause a false halfway point. + add_native(big, 10, 1); + count++; +} + +// parse the significant digits into a big integer +inline void parse_mantissa(bigint& result, parsed_number_string& num, size_t max_digits, size_t& digits) noexcept { + // try to minimize the number of big integer and scalar multiplication. + // therefore, try to parse 8 digits at a time, and multiply by the largest + // scalar value (9 or 19 digits) for each step. + size_t counter = 0; + digits = 0; + limb value = 0; +#ifdef FASTFLOAT_64BIT_LIMB + size_t step = 19; +#else + size_t step = 9; +#endif + + // process all integer digits. + const char* p = num.integer.ptr; + const char* pend = p + num.integer.len(); + skip_zeros(p, pend); + // process all digits, in increments of step per loop + while (p != pend) { + while ((std::distance(p, pend) >= 8) && (step - counter >= 8) && (max_digits - digits >= 8)) { + parse_eight_digits(p, value, counter, digits); + } + while (counter < step && p != pend && digits < max_digits) { + parse_one_digit(p, value, counter, digits); + } + if (digits == max_digits) { + // add the temporary value, then check if we've truncated any digits + add_native(result, limb(powers_of_ten_uint64[counter]), value); + bool truncated = is_truncated(p, pend); + if (num.fraction.ptr != nullptr) { + truncated |= is_truncated(num.fraction); + } + if (truncated) { + round_up_bigint(result, digits); + } + return; + } else { + add_native(result, limb(powers_of_ten_uint64[counter]), value); + counter = 0; + value = 0; + } + } + + // add our fraction digits, if they're available. + if (num.fraction.ptr != nullptr) { + p = num.fraction.ptr; + pend = p + num.fraction.len(); + if (digits == 0) { + skip_zeros(p, pend); + } + // process all digits, in increments of step per loop + while (p != pend) { + while ((std::distance(p, pend) >= 8) && (step - counter >= 8) && (max_digits - digits >= 8)) { + parse_eight_digits(p, value, counter, digits); + } + while (counter < step && p != pend && digits < max_digits) { + parse_one_digit(p, value, counter, digits); + } + if (digits == max_digits) { + // add the temporary value, then check if we've truncated any digits + add_native(result, limb(powers_of_ten_uint64[counter]), value); + bool truncated = is_truncated(p, pend); + if (truncated) { + round_up_bigint(result, digits); + } + return; + } else { + add_native(result, limb(powers_of_ten_uint64[counter]), value); + counter = 0; + value = 0; + } + } + } + + if (counter != 0) { + add_native(result, limb(powers_of_ten_uint64[counter]), value); + } +} + +template <typename T> +inline adjusted_mantissa positive_digit_comp(bigint& bigmant, int32_t exponent) noexcept { + FASTFLOAT_ASSERT(bigmant.pow10(uint32_t(exponent))); + adjusted_mantissa answer; + bool truncated; + answer.mantissa = bigmant.hi64(truncated); + int bias = binary_format<T>::mantissa_explicit_bits() - binary_format<T>::minimum_exponent(); + answer.power2 = bigmant.bit_length() - 64 + bias; + + round<T>(answer, [truncated](adjusted_mantissa& a, int32_t shift) { + round_nearest_tie_even(a, shift, [truncated](bool is_odd, bool is_halfway, bool is_above) -> bool { + return is_above || (is_halfway && truncated) || (is_odd && is_halfway); + }); + }); + + return answer; +} + +// the scaling here is quite simple: we have, for the real digits `m * 10^e`, +// and for the theoretical digits `n * 2^f`. Since `e` is always negative, +// to scale them identically, we do `n * 2^f * 5^-f`, so we now have `m * 2^e`. +// we then need to scale by `2^(f- e)`, and then the two significant digits +// are of the same magnitude. +template <typename T> +inline adjusted_mantissa negative_digit_comp(bigint& bigmant, adjusted_mantissa am, int32_t exponent) noexcept { + bigint& real_digits = bigmant; + int32_t real_exp = exponent; + + // get the value of `b`, rounded down, and get a bigint representation of b+h + adjusted_mantissa am_b = am; + // gcc7 buf: use a lambda to remove the noexcept qualifier bug with -Wnoexcept-type. + round<T>(am_b, [](adjusted_mantissa&a, int32_t shift) { round_down(a, shift); }); + T b; + to_float(false, am_b, b); + adjusted_mantissa theor = to_extended_halfway(b); + bigint theor_digits(theor.mantissa); + int32_t theor_exp = theor.power2; + + // scale real digits and theor digits to be same power. + int32_t pow2_exp = theor_exp - real_exp; + uint32_t pow5_exp = uint32_t(-real_exp); + if (pow5_exp != 0) { + FASTFLOAT_ASSERT(theor_digits.pow5(pow5_exp)); + } + if (pow2_exp > 0) { + FASTFLOAT_ASSERT(theor_digits.pow2(uint32_t(pow2_exp))); + } else if (pow2_exp < 0) { + FASTFLOAT_ASSERT(real_digits.pow2(uint32_t(-pow2_exp))); + } + + // compare digits, and use it to director rounding + int ord = real_digits.compare(theor_digits); + adjusted_mantissa answer = am; + round<T>(answer, [ord](adjusted_mantissa& a, int32_t shift) { + round_nearest_tie_even(a, shift, [ord](bool is_odd, bool _, bool __) -> bool { + (void)_; // not needed, since we've done our comparison + (void)__; // not needed, since we've done our comparison + if (ord > 0) { + return true; + } else if (ord < 0) { + return false; + } else { + return is_odd; + } + }); + }); + + return answer; +} + +// parse the significant digits as a big integer to unambiguously round the +// the significant digits. here, we are trying to determine how to round +// an extended float representation close to `b+h`, halfway between `b` +// (the float rounded-down) and `b+u`, the next positive float. this +// algorithm is always correct, and uses one of two approaches. when +// the exponent is positive relative to the significant digits (such as +// 1234), we create a big-integer representation, get the high 64-bits, +// determine if any lower bits are truncated, and use that to direct +// rounding. in case of a negative exponent relative to the significant +// digits (such as 1.2345), we create a theoretical representation of +// `b` as a big-integer type, scaled to the same binary exponent as +// the actual digits. we then compare the big integer representations +// of both, and use that to direct rounding. +template <typename T> +inline adjusted_mantissa digit_comp(parsed_number_string& num, adjusted_mantissa am) noexcept { + // remove the invalid exponent bias + am.power2 -= invalid_am_bias; + + int32_t sci_exp = scientific_exponent(num); + size_t max_digits = binary_format<T>::max_digits(); + size_t digits = 0; + bigint bigmant; + parse_mantissa(bigmant, num, max_digits, digits); + // can't underflow, since digits is at most max_digits. + int32_t exponent = sci_exp + 1 - int32_t(digits); + if (exponent >= 0) { + return positive_digit_comp<T>(bigmant, exponent); + } else { + return negative_digit_comp<T>(bigmant, am, exponent); + } +} + +} // namespace fast_float + +#endif + +#ifndef FASTFLOAT_PARSE_NUMBER_H +#define FASTFLOAT_PARSE_NUMBER_H + + +#include <cmath> +#include <cstring> +#include <limits> +#include <system_error> + +namespace fast_float { + + +namespace detail { +/** + * Special case +inf, -inf, nan, infinity, -infinity. + * The case comparisons could be made much faster given that we know that the + * strings a null-free and fixed. + **/ +template <typename T> +from_chars_result parse_infnan(const char *first, const char *last, T &value) noexcept { + from_chars_result answer; + answer.ptr = first; + answer.ec = std::errc(); // be optimistic + bool minusSign = false; + if (*first == '-') { // assume first < last, so dereference without checks; C++17 20.19.3.(7.1) explicitly forbids '+' here + minusSign = true; + ++first; + } + if (last - first >= 3) { + if (fastfloat_strncasecmp(first, "nan", 3)) { + answer.ptr = (first += 3); + value = minusSign ? -std::numeric_limits<T>::quiet_NaN() : std::numeric_limits<T>::quiet_NaN(); + // Check for possible nan(n-char-seq-opt), C++17 20.19.3.7, C11 7.20.1.3.3. At least MSVC produces nan(ind) and nan(snan). + if(first != last && *first == '(') { + for(const char* ptr = first + 1; ptr != last; ++ptr) { + if (*ptr == ')') { + answer.ptr = ptr + 1; // valid nan(n-char-seq-opt) + break; + } + else if(!(('a' <= *ptr && *ptr <= 'z') || ('A' <= *ptr && *ptr <= 'Z') || ('0' <= *ptr && *ptr <= '9') || *ptr == '_')) + break; // forbidden char, not nan(n-char-seq-opt) + } + } + return answer; + } + if (fastfloat_strncasecmp(first, "inf", 3)) { + if ((last - first >= 8) && fastfloat_strncasecmp(first + 3, "inity", 5)) { + answer.ptr = first + 8; + } else { + answer.ptr = first + 3; + } + value = minusSign ? -std::numeric_limits<T>::infinity() : std::numeric_limits<T>::infinity(); + return answer; + } + } + answer.ec = std::errc::invalid_argument; + return answer; +} + +} // namespace detail + +template<typename T> +from_chars_result from_chars(const char *first, const char *last, + T &value, chars_format fmt /*= chars_format::general*/) noexcept { + return from_chars_advanced(first, last, value, parse_options{fmt}); +} + +template<typename T> +from_chars_result from_chars_advanced(const char *first, const char *last, + T &value, parse_options options) noexcept { + + static_assert (std::is_same<T, double>::value || std::is_same<T, float>::value, "only float and double are supported"); + + + from_chars_result answer; + if (first == last) { + answer.ec = std::errc::invalid_argument; + answer.ptr = first; + return answer; + } + parsed_number_string pns = parse_number_string(first, last, options); + if (!pns.valid) { + return detail::parse_infnan(first, last, value); + } + answer.ec = std::errc(); // be optimistic + answer.ptr = pns.lastmatch; + // Next is Clinger's fast path. + if (binary_format<T>::min_exponent_fast_path() <= pns.exponent && pns.exponent <= binary_format<T>::max_exponent_fast_path() && pns.mantissa <=binary_format<T>::max_mantissa_fast_path() && !pns.too_many_digits) { + value = T(pns.mantissa); + if (pns.exponent < 0) { value = value / binary_format<T>::exact_power_of_ten(-pns.exponent); } + else { value = value * binary_format<T>::exact_power_of_ten(pns.exponent); } + if (pns.negative) { value = -value; } + return answer; + } + adjusted_mantissa am = compute_float<binary_format<T>>(pns.exponent, pns.mantissa); + if(pns.too_many_digits && am.power2 >= 0) { + if(am != compute_float<binary_format<T>>(pns.exponent, pns.mantissa + 1)) { + am = compute_error<binary_format<T>>(pns.exponent, pns.mantissa); + } + } + // If we called compute_float<binary_format<T>>(pns.exponent, pns.mantissa) and we have an invalid power (am.power2 < 0), + // then we need to go the long way around again. This is very uncommon. + if(am.power2 < 0) { am = digit_comp<T>(pns, am); } + to_float(pns.negative, am, value); + return answer; +} + +} // namespace fast_float + +#endif + diff --git a/source/blender/editors/io/io_obj.c b/source/blender/editors/io/io_obj.c index 97f1e08fdff..beed4abd52b 100644 --- a/source/blender/editors/io/io_obj.c +++ b/source/blender/editors/io/io_obj.c @@ -378,6 +378,7 @@ static int wm_obj_import_exec(bContext *C, wmOperator *op) import_params.clamp_size = RNA_float_get(op->ptr, "clamp_size"); import_params.forward_axis = RNA_enum_get(op->ptr, "forward_axis"); import_params.up_axis = RNA_enum_get(op->ptr, "up_axis"); + import_params.validate_meshes = RNA_boolean_get(op->ptr, "validate_meshes"); OBJ_import(C, &import_params); @@ -388,8 +389,8 @@ static void ui_obj_import_settings(uiLayout *layout, PointerRNA *imfptr) { uiLayoutSetPropSep(layout, true); uiLayoutSetPropDecorate(layout, false); - uiLayout *box = uiLayoutBox(layout); + uiLayout *box = uiLayoutBox(layout); uiItemL(box, IFACE_("Transform"), ICON_OBJECT_DATA); uiLayout *col = uiLayoutColumn(box, false); uiLayout *sub = uiLayoutColumn(col, false); @@ -397,6 +398,11 @@ static void ui_obj_import_settings(uiLayout *layout, PointerRNA *imfptr) sub = uiLayoutColumn(col, false); uiItemR(sub, imfptr, "forward_axis", 0, IFACE_("Axis Forward"), ICON_NONE); uiItemR(sub, imfptr, "up_axis", 0, IFACE_("Up"), ICON_NONE); + + box = uiLayoutBox(layout); + uiItemL(box, IFACE_("Options"), ICON_EXPORT); + col = uiLayoutColumn(box, false); + uiItemR(col, imfptr, "validate_meshes", 0, NULL, ICON_NONE); } static void wm_obj_import_draw(bContext *C, wmOperator *op) @@ -442,4 +448,9 @@ void WM_OT_obj_import(struct wmOperatorType *ot) "Forward Axis", ""); RNA_def_enum(ot->srna, "up_axis", io_obj_transform_axis_up, OBJ_AXIS_Y_UP, "Up Axis", ""); + RNA_def_boolean(ot->srna, + "validate_meshes", + false, + "Validate Meshes", + "Check imported mesh objects for invalid data (slow)"); } diff --git a/source/blender/io/common/CMakeLists.txt b/source/blender/io/common/CMakeLists.txt index 02bd5b2b81f..b1add38bf01 100644 --- a/source/blender/io/common/CMakeLists.txt +++ b/source/blender/io/common/CMakeLists.txt @@ -7,6 +7,8 @@ set(INC ../../blenlib ../../depsgraph ../../makesdna + ../../../../intern/guardedalloc + ../../../../extern/fast_float ) set(INC_SYS @@ -17,9 +19,11 @@ set(SRC intern/dupli_parent_finder.cc intern/dupli_persistent_id.cc intern/object_identifier.cc + intern/string_utils.cc IO_abstract_hierarchy_iterator.h IO_dupli_persistent_id.hh + IO_string_utils.hh IO_types.h intern/dupli_parent_finder.hh ) @@ -38,6 +42,7 @@ if(WITH_GTESTS) intern/abstract_hierarchy_iterator_test.cc intern/hierarchy_context_order_test.cc intern/object_identifier_test.cc + intern/string_utils_test.cc ) set(TEST_INC ../../blenloader diff --git a/source/blender/io/common/IO_string_utils.hh b/source/blender/io/common/IO_string_utils.hh new file mode 100644 index 00000000000..25f1f01c6ed --- /dev/null +++ b/source/blender/io/common/IO_string_utils.hh @@ -0,0 +1,69 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ + +#pragma once + +#include "BLI_string_ref.hh" + +/* + * Various text parsing utilities commonly used by text-based input formats. + */ + +namespace blender::io { + +/** + * Fetches next line from an input string buffer. + * + * The returned line will not have '\n' characters at the end; + * the `buffer` is modified to contain remaining text without + * the input line. + * + * Note that backslash (\) character is treated as a line + * continuation, similar to OBJ file format or a C preprocessor. + */ +StringRef read_next_line(StringRef &buffer); + +/** + * Drop leading white-space from a StringRef. + * Note that backslash character is considered white-space. + */ +StringRef drop_whitespace(StringRef str); + +/** + * Drop leading non-white-space from a StringRef. + * Note that backslash character is considered white-space. + */ +StringRef drop_non_whitespace(StringRef str); + +/** + * Parse an integer from an input string. + * The parsed result is stored in `dst`. The function skips + * leading white-space unless `skip_space=false`. If the + * number can't be parsed (invalid syntax, out of range), + * `fallback` value is stored instead. + * + * Returns the remainder of the input string after parsing. + */ +StringRef parse_int(StringRef str, int fallback, int &dst, bool skip_space = true); + +/** + * Parse a float from an input string. + * The parsed result is stored in `dst`. The function skips + * leading white-space unless `skip_space=false`. If the + * number can't be parsed (invalid syntax, out of range), + * `fallback` value is stored instead. + * + * Returns the remainder of the input string after parsing. + */ +StringRef parse_float(StringRef str, float fallback, float &dst, bool skip_space = true); + +/** + * Parse a number of white-space separated floats from an input string. + * The parsed `count` numbers are stored in `dst`. If a + * number can't be parsed (invalid syntax, out of range), + * `fallback` value is stored instead. + * + * Returns the remainder of the input string after parsing. + */ +StringRef parse_floats(StringRef str, float fallback, float *dst, int count); + +} // namespace blender::io diff --git a/source/blender/io/common/intern/string_utils.cc b/source/blender/io/common/intern/string_utils.cc new file mode 100644 index 00000000000..01107b0866e --- /dev/null +++ b/source/blender/io/common/intern/string_utils.cc @@ -0,0 +1,99 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ + +#include "IO_string_utils.hh" + +/* Note: we could use C++17 <charconv> from_chars to parse + * floats, but even if some compilers claim full support, + * their standard libraries are not quite there yet. + * LLVM/libc++ only has a float parser since LLVM 14, + * and gcc/libstdc++ since 11.1. So until at least these are + * the mininum spec, use an external library. */ +#include "fast_float.h" +#include <charconv> + +namespace blender::io { + +StringRef read_next_line(StringRef &buffer) +{ + const char *start = buffer.begin(); + const char *end = buffer.end(); + size_t len = 0; + char prev = 0; + const char *ptr = start; + while (ptr < end) { + char c = *ptr++; + if (c == '\n' && prev != '\\') { + break; + } + prev = c; + ++len; + } + + buffer = StringRef(ptr, end); + return StringRef(start, len); +} + +static bool is_whitespace(char c) +{ + return c <= ' ' || c == '\\'; +} + +StringRef drop_whitespace(StringRef str) +{ + while (!str.is_empty() && is_whitespace(str[0])) { + str = str.drop_prefix(1); + } + return str; +} + +StringRef drop_non_whitespace(StringRef str) +{ + while (!str.is_empty() && !is_whitespace(str[0])) { + str = str.drop_prefix(1); + } + return str; +} + +static StringRef drop_plus(StringRef str) +{ + if (!str.is_empty() && str[0] == '+') { + str = str.drop_prefix(1); + } + return str; +} + +StringRef parse_float(StringRef str, float fallback, float &dst, bool skip_space) +{ + if (skip_space) { + str = drop_whitespace(str); + } + str = drop_plus(str); + fast_float::from_chars_result res = fast_float::from_chars(str.begin(), str.end(), dst); + if (res.ec == std::errc::invalid_argument || res.ec == std::errc::result_out_of_range) { + dst = fallback; + } + return StringRef(res.ptr, str.end()); +} + +StringRef parse_floats(StringRef str, float fallback, float *dst, int count) +{ + for (int i = 0; i < count; ++i) { + str = parse_float(str, fallback, dst[i]); + } + return str; +} + +StringRef parse_int(StringRef str, int fallback, int &dst, bool skip_space) +{ + if (skip_space) { + str = drop_whitespace(str); + } + str = drop_plus(str); + std::from_chars_result res = std::from_chars(str.begin(), str.end(), dst); + if (res.ec == std::errc::invalid_argument || res.ec == std::errc::result_out_of_range) { + dst = fallback; + } + return StringRef(res.ptr, str.end()); +} + +} // namespace blender::io diff --git a/source/blender/io/common/intern/string_utils_test.cc b/source/blender/io/common/intern/string_utils_test.cc new file mode 100644 index 00000000000..a78bd7ab8a3 --- /dev/null +++ b/source/blender/io/common/intern/string_utils_test.cc @@ -0,0 +1,118 @@ +/* SPDX-License-Identifier: Apache-2.0 */ + +#include "IO_string_utils.hh" + +#include "testing/testing.h" + +namespace blender::io { + +#define EXPECT_STRREF_EQ(str1, str2) EXPECT_STREQ(str1, std::string(str2).c_str()) + +TEST(string_utils, read_next_line) +{ + std::string str = "abc\n \n\nline with \\\ncontinuation\nCRLF ending:\r\na"; + StringRef s = str; + EXPECT_STRREF_EQ("abc", read_next_line(s)); + EXPECT_STRREF_EQ(" ", read_next_line(s)); + EXPECT_STRREF_EQ("", read_next_line(s)); + EXPECT_STRREF_EQ("line with \\\ncontinuation", read_next_line(s)); + EXPECT_STRREF_EQ("CRLF ending:\r", read_next_line(s)); + EXPECT_STRREF_EQ("a", read_next_line(s)); + EXPECT_TRUE(s.is_empty()); +} + +TEST(string_utils, drop_whitespace) +{ + /* Empty */ + EXPECT_STRREF_EQ("", drop_whitespace("")); + /* Only whitespace */ + EXPECT_STRREF_EQ("", drop_whitespace(" ")); + EXPECT_STRREF_EQ("", drop_whitespace(" ")); + EXPECT_STRREF_EQ("", drop_whitespace(" \t\n\r ")); + /* Drops leading whitespace */ + EXPECT_STRREF_EQ("a", drop_whitespace(" a")); + EXPECT_STRREF_EQ("a b", drop_whitespace(" a b")); + EXPECT_STRREF_EQ("a b ", drop_whitespace(" a b ")); + /* No leading whitespace */ + EXPECT_STRREF_EQ("c", drop_whitespace("c")); + /* Case with backslash, should be treated as whitespace */ + EXPECT_STRREF_EQ("d", drop_whitespace(" \\ d")); +} + +TEST(string_utils, parse_int_valid) +{ + std::string str = "1 -10 \t 1234 1234567890 +7 123a"; + StringRef s = str; + int val; + s = parse_int(s, 0, val); + EXPECT_EQ(1, val); + s = parse_int(s, 0, val); + EXPECT_EQ(-10, val); + s = parse_int(s, 0, val); + EXPECT_EQ(1234, val); + s = parse_int(s, 0, val); + EXPECT_EQ(1234567890, val); + s = parse_int(s, 0, val); + EXPECT_EQ(7, val); + s = parse_int(s, 0, val); + EXPECT_EQ(123, val); + EXPECT_STRREF_EQ("a", s); +} + +TEST(string_utils, parse_int_invalid) +{ + int val; + /* Invalid syntax */ + EXPECT_STRREF_EQ("--123", parse_int("--123", -1, val)); + EXPECT_EQ(val, -1); + EXPECT_STRREF_EQ("foobar", parse_int("foobar", -2, val)); + EXPECT_EQ(val, -2); + /* Out of integer range */ + EXPECT_STRREF_EQ(" a", parse_int("1234567890123 a", -3, val)); + EXPECT_EQ(val, -3); + /* Has leading white-space when we don't expect it */ + EXPECT_STRREF_EQ(" 1", parse_int(" 1", -4, val, false)); + EXPECT_EQ(val, -4); +} + +TEST(string_utils, parse_float_valid) +{ + std::string str = "1 -10 123.5 -17.125 0.1 1e6 50.0e-1"; + StringRef s = str; + float val; + s = parse_float(s, 0, val); + EXPECT_EQ(1.0f, val); + s = parse_float(s, 0, val); + EXPECT_EQ(-10.0f, val); + s = parse_float(s, 0, val); + EXPECT_EQ(123.5f, val); + s = parse_float(s, 0, val); + EXPECT_EQ(-17.125f, val); + s = parse_float(s, 0, val); + EXPECT_EQ(0.1f, val); + s = parse_float(s, 0, val); + EXPECT_EQ(1.0e6f, val); + s = parse_float(s, 0, val); + EXPECT_EQ(5.0f, val); + EXPECT_TRUE(s.is_empty()); +} + +TEST(string_utils, parse_float_invalid) +{ + float val; + /* Invalid syntax */ + EXPECT_STRREF_EQ("_0", parse_float("_0", -1.0f, val)); + EXPECT_EQ(val, -1.0f); + EXPECT_STRREF_EQ("..5", parse_float("..5", -2.0f, val)); + EXPECT_EQ(val, -2.0f); + /* Out of float range. Current float parser (fast_float) + * clamps out of range numbers to +/- infinity, so this + * one gets a +inf instead of fallback -3.0. */ + EXPECT_STRREF_EQ(" a", parse_float("9.0e500 a", -3.0f, val)); + EXPECT_EQ(val, std::numeric_limits<float>::infinity()); + /* Has leading white-space when we don't expect it */ + EXPECT_STRREF_EQ(" 1", parse_float(" 1", -4.0f, val, false)); + EXPECT_EQ(val, -4.0f); +} + +} // namespace blender::io diff --git a/source/blender/io/wavefront_obj/CMakeLists.txt b/source/blender/io/wavefront_obj/CMakeLists.txt index 9cdd96ee7be..67cec000778 100644 --- a/source/blender/io/wavefront_obj/CMakeLists.txt +++ b/source/blender/io/wavefront_obj/CMakeLists.txt @@ -4,6 +4,7 @@ set(INC . ./exporter ./importer + ../common ../../blenkernel ../../blenlib ../../bmesh @@ -35,7 +36,6 @@ set(SRC importer/obj_import_mtl.cc importer/obj_import_nurbs.cc importer/obj_importer.cc - importer/parser_string_utils.cc IO_wavefront_obj.h exporter/obj_export_file_writer.hh @@ -51,11 +51,11 @@ set(SRC importer/obj_import_nurbs.hh importer/obj_import_objects.hh importer/obj_importer.hh - importer/parser_string_utils.hh ) set(LIB bf_blenkernel + bf_io_common ) if(WITH_TBB) @@ -70,6 +70,7 @@ if(WITH_GTESTS) set(TEST_SRC tests/obj_exporter_tests.cc tests/obj_importer_tests.cc + tests/obj_mtl_parser_tests.cc tests/obj_exporter_tests.hh ) diff --git a/source/blender/io/wavefront_obj/IO_wavefront_obj.h b/source/blender/io/wavefront_obj/IO_wavefront_obj.h index b06ccbf8702..8b71ec750c0 100644 --- a/source/blender/io/wavefront_obj/IO_wavefront_obj.h +++ b/source/blender/io/wavefront_obj/IO_wavefront_obj.h @@ -84,6 +84,7 @@ struct OBJImportParams { float clamp_size; eTransformAxisForward forward_axis; eTransformAxisUp up_axis; + bool validate_meshes; }; /** diff --git a/source/blender/io/wavefront_obj/importer/obj_import_file_reader.cc b/source/blender/io/wavefront_obj/importer/obj_import_file_reader.cc index d3d4e1ba860..184810b9802 100644 --- a/source/blender/io/wavefront_obj/importer/obj_import_file_reader.cc +++ b/source/blender/io/wavefront_obj/importer/obj_import_file_reader.cc @@ -8,7 +8,7 @@ #include "BLI_string_ref.hh" #include "BLI_vector.hh" -#include "parser_string_utils.hh" +#include "IO_string_utils.hh" #include "obj_import_file_reader.hh" @@ -34,7 +34,8 @@ static Geometry *create_geometry(Geometry *const prev_geometry, Geometry *g = r_all_geometries.last().get(); g->geom_type_ = new_type; g->geometry_name_ = name.is_empty() ? "New object" : name; - r_offset.set_index_offset(global_vertices.vertices.size()); + g->vertex_start_ = global_vertices.vertices.size(); + r_offset.set_index_offset(g->vertex_start_); return g; }; @@ -66,48 +67,40 @@ static Geometry *create_geometry(Geometry *const prev_geometry, } static void geom_add_vertex(Geometry *geom, - const StringRef rest_line, + const StringRef line, GlobalVertices &r_global_vertices) { - float3 curr_vert; - Vector<StringRef> str_vert_split; - split_by_char(rest_line, ' ', str_vert_split); - copy_string_to_float(str_vert_split, FLT_MAX, {curr_vert, 3}); - r_global_vertices.vertices.append(curr_vert); - geom->vertex_indices_.append(r_global_vertices.vertices.size() - 1); + float3 vert; + parse_floats(line, FLT_MAX, vert, 3); + r_global_vertices.vertices.append(vert); + geom->vertex_count_++; } static void geom_add_vertex_normal(Geometry *geom, - const StringRef rest_line, + const StringRef line, GlobalVertices &r_global_vertices) { - float3 curr_vert_normal; - Vector<StringRef> str_vert_normal_split; - split_by_char(rest_line, ' ', str_vert_normal_split); - copy_string_to_float(str_vert_normal_split, FLT_MAX, {curr_vert_normal, 3}); - r_global_vertices.vertex_normals.append(curr_vert_normal); + float3 normal; + parse_floats(line, FLT_MAX, normal, 3); + r_global_vertices.vertex_normals.append(normal); geom->has_vertex_normals_ = true; } -static void geom_add_uv_vertex(const StringRef rest_line, GlobalVertices &r_global_vertices) +static void geom_add_uv_vertex(const StringRef line, GlobalVertices &r_global_vertices) { - float2 curr_uv_vert; - Vector<StringRef> str_uv_vert_split; - split_by_char(rest_line, ' ', str_uv_vert_split); - copy_string_to_float(str_uv_vert_split, FLT_MAX, {curr_uv_vert, 2}); - r_global_vertices.uv_vertices.append(curr_uv_vert); + float2 uv; + parse_floats(line, FLT_MAX, uv, 2); + r_global_vertices.uv_vertices.append(uv); } static void geom_add_edge(Geometry *geom, - const StringRef rest_line, + StringRef line, const VertexIndexOffset &offsets, GlobalVertices &r_global_vertices) { - int edge_v1 = -1, edge_v2 = -1; - Vector<StringRef> str_edge_split; - split_by_char(rest_line, ' ', str_edge_split); - copy_string_to_int(str_edge_split[0], -1, edge_v1); - copy_string_to_int(str_edge_split[1], -1, edge_v2); + int edge_v1, edge_v2; + line = parse_int(line, -1, edge_v1); + line = parse_int(line, -1, edge_v2); /* Always keep stored indices non-negative and zero-based. */ edge_v1 += edge_v1 < 0 ? r_global_vertices.vertices.size() : -offsets.get_index_offset() - 1; edge_v2 += edge_v2 < 0 ? r_global_vertices.vertices.size() : -offsets.get_index_offset() - 1; @@ -116,78 +109,45 @@ static void geom_add_edge(Geometry *geom, } static void geom_add_polygon(Geometry *geom, - const StringRef rest_line, + StringRef line, const GlobalVertices &global_vertices, const VertexIndexOffset &offsets, - const StringRef state_material_name, - const StringRef state_object_group, - const bool state_shaded_smooth) + const int material_index, + const int group_index, + const bool shaded_smooth) { PolyElem curr_face; - curr_face.shaded_smooth = state_shaded_smooth; - if (!state_material_name.is_empty()) { - curr_face.material_name = state_material_name; - } - if (!state_object_group.is_empty()) { - curr_face.vertex_group = state_object_group; - /* Yes it repeats several times, but another if-check will not reduce steps either. */ + curr_face.shaded_smooth = shaded_smooth; + curr_face.material_index = material_index; + if (group_index >= 0) { + curr_face.vertex_group_index = group_index; geom->use_vertex_groups_ = true; } + const int orig_corners_size = geom->face_corners_.size(); + curr_face.start_index_ = orig_corners_size; + bool face_valid = true; - Vector<StringRef> str_corners_split; - split_by_char(rest_line, ' ', str_corners_split); - for (StringRef str_corner : str_corners_split) { + while (!line.is_empty() && face_valid) { PolyCorner corner; - const size_t n_slash = std::count(str_corner.begin(), str_corner.end(), '/'); bool got_uv = false, got_normal = false; - if (n_slash == 0) { - /* Case: "f v1 v2 v3". */ - copy_string_to_int(str_corner, INT32_MAX, corner.vert_index); - } - else if (n_slash == 1) { - /* Case: "f v1/vt1 v2/vt2 v3/vt3". */ - Vector<StringRef> vert_uv_split; - split_by_char(str_corner, '/', vert_uv_split); - if (vert_uv_split.size() != 1 && vert_uv_split.size() != 2) { - fprintf(stderr, "Invalid face syntax '%s', ignoring\n", std::string(str_corner).c_str()); - face_valid = false; + /* Parse vertex index. */ + line = parse_int(line, INT32_MAX, corner.vert_index, false); + face_valid &= corner.vert_index != INT32_MAX; + if (!line.is_empty() && line[0] == '/') { + /* Parse UV index. */ + line = line.drop_prefix(1); + if (!line.is_empty() && line[0] != '/') { + line = parse_int(line, INT32_MAX, corner.uv_vert_index, false); + got_uv = corner.uv_vert_index != INT32_MAX; } - else { - copy_string_to_int(vert_uv_split[0], INT32_MAX, corner.vert_index); - if (vert_uv_split.size() == 2) { - copy_string_to_int(vert_uv_split[1], INT32_MAX, corner.uv_vert_index); - got_uv = corner.uv_vert_index != INT32_MAX; - } + /* Parse normal index. */ + if (!line.is_empty() && line[0] == '/') { + line = line.drop_prefix(1); + line = parse_int(line, INT32_MAX, corner.vertex_normal_index, false); + got_normal = corner.uv_vert_index != INT32_MAX; } } - else if (n_slash == 2) { - /* Case: "f v1//vn1 v2//vn2 v3//vn3". */ - /* Case: "f v1/vt1/vn1 v2/vt2/vn2 v3/vt3/vn3". */ - Vector<StringRef> vert_uv_normal_split; - split_by_char(str_corner, '/', vert_uv_normal_split); - if (vert_uv_normal_split.size() != 2 && vert_uv_normal_split.size() != 3) { - fprintf(stderr, "Invalid face syntax '%s', ignoring\n", std::string(str_corner).c_str()); - face_valid = false; - } - else { - copy_string_to_int(vert_uv_normal_split[0], INT32_MAX, corner.vert_index); - if (vert_uv_normal_split.size() == 3) { - copy_string_to_int(vert_uv_normal_split[1], INT32_MAX, corner.uv_vert_index); - got_uv = corner.uv_vert_index != INT32_MAX; - copy_string_to_int(vert_uv_normal_split[2], INT32_MAX, corner.vertex_normal_index); - got_normal = corner.vertex_normal_index != INT32_MAX; - } - else { - copy_string_to_int(vert_uv_normal_split[1], INT32_MAX, corner.vertex_normal_index); - got_normal = corner.vertex_normal_index != INT32_MAX; - } - } - } - else { - fprintf(stderr, "Invalid face syntax '%s', ignoring\n", std::string(str_corner).c_str()); - face_valid = false; - } /* Always keep stored indices non-negative and zero-based. */ corner.vert_index += corner.vert_index < 0 ? global_vertices.vertices.size() : -offsets.get_index_offset() - 1; @@ -221,19 +181,28 @@ static void geom_add_polygon(Geometry *geom, face_valid = false; } } - curr_face.face_corners.append(corner); + geom->face_corners_.append(corner); + curr_face.corner_count_++; + + /* Skip whitespace to get to the next face corner. */ + line = drop_whitespace(line); } if (face_valid) { geom->face_elements_.append(curr_face); - geom->total_loops_ += curr_face.face_corners.size(); + geom->total_loops_ += curr_face.corner_count_; + } + else { + /* Remove just-added corners for the invalid face. */ + geom->face_corners_.resize(orig_corners_size); + geom->has_invalid_polys_ = true; } } static Geometry *geom_set_curve_type(Geometry *geom, const StringRef rest_line, const GlobalVertices &global_vertices, - const StringRef state_object_group, + const StringRef group_name, VertexIndexOffset &r_offsets, Vector<std::unique_ptr<Geometry>> &r_all_geometries) { @@ -242,254 +211,409 @@ static Geometry *geom_set_curve_type(Geometry *geom, return geom; } geom = create_geometry( - geom, GEOM_CURVE, state_object_group, global_vertices, r_all_geometries, r_offsets); - geom->nurbs_element_.group_ = state_object_group; + geom, GEOM_CURVE, group_name, global_vertices, r_all_geometries, r_offsets); + geom->nurbs_element_.group_ = group_name; return geom; } -static void geom_set_curve_degree(Geometry *geom, const StringRef rest_line) +static void geom_set_curve_degree(Geometry *geom, const StringRef line) { - copy_string_to_int(rest_line, 3, geom->nurbs_element_.degree); + parse_int(line, 3, geom->nurbs_element_.degree); } static void geom_add_curve_vertex_indices(Geometry *geom, - const StringRef rest_line, + StringRef line, const GlobalVertices &global_vertices) { - Vector<StringRef> str_curv_split; - split_by_char(rest_line, ' ', str_curv_split); - /* Remove "0.0" and "1.0" from the strings. They are hardcoded. */ - str_curv_split.remove(0); - str_curv_split.remove(0); - geom->nurbs_element_.curv_indices.resize(str_curv_split.size()); - copy_string_to_int(str_curv_split, INT32_MAX, geom->nurbs_element_.curv_indices); - for (int &curv_index : geom->nurbs_element_.curv_indices) { + /* Curve lines always have "0.0" and "1.0", skip over them. */ + float dummy[2]; + line = parse_floats(line, 0, dummy, 2); + /* Parse indices. */ + while (!line.is_empty()) { + int index; + line = parse_int(line, INT32_MAX, index); + if (index == INT32_MAX) { + return; + } /* Always keep stored indices non-negative and zero-based. */ - curv_index += curv_index < 0 ? global_vertices.vertices.size() : -1; + index += index < 0 ? global_vertices.vertices.size() : -1; + geom->nurbs_element_.curv_indices.append(index); } } -static void geom_add_curve_parameters(Geometry *geom, const StringRef rest_line) +static void geom_add_curve_parameters(Geometry *geom, StringRef line) { - Vector<StringRef> str_parm_split; - split_by_char(rest_line, ' ', str_parm_split); - if (str_parm_split[0] != "u" && str_parm_split[0] != "v") { - std::cerr << "Surfaces are not supported:'" << str_parm_split[0] << "'" << std::endl; + line = drop_whitespace(line); + if (line.is_empty()) { + std::cerr << "Invalid OBJ curve parm line: '" << line << "'" << std::endl; + return; + } + if (line[0] != 'u') { + std::cerr << "OBJ curve surfaces are not supported: '" << line[0] << "'" << std::endl; return; } - str_parm_split.remove(0); - geom->nurbs_element_.parm.resize(str_parm_split.size()); - copy_string_to_float(str_parm_split, FLT_MAX, geom->nurbs_element_.parm); + line = line.drop_prefix(1); + + while (!line.is_empty()) { + float val; + line = parse_float(line, FLT_MAX, val); + if (val != FLT_MAX) { + geom->nurbs_element_.parm.append(val); + } + else { + std::cerr << "OBJ curve parm line has invalid number" << std::endl; + return; + } + } } -static void geom_update_object_group(const StringRef rest_line, std::string &r_state_object_group) +static void geom_update_group(const StringRef rest_line, std::string &r_group_name) { if (rest_line.find("off") != string::npos || rest_line.find("null") != string::npos || rest_line.find("default") != string::npos) { /* Set group for future elements like faces or curves to empty. */ - r_state_object_group = ""; + r_group_name = ""; return; } - r_state_object_group = rest_line; -} - -static void geom_update_polygon_material(Geometry *geom, - const StringRef rest_line, - std::string &r_state_material_name) -{ - /* Materials may repeat if faces are written without sorting. */ - geom->material_names_.add(string(rest_line)); - r_state_material_name = rest_line; + r_group_name = rest_line; } -static void geom_update_smooth_group(const StringRef rest_line, bool &r_state_shaded_smooth) +static void geom_update_smooth_group(StringRef line, bool &r_state_shaded_smooth) { + line = drop_whitespace(line); /* Some implementations use "0" and "null" too, in addition to "off". */ - if (rest_line != "0" && rest_line.find("off") == StringRef::not_found && - rest_line.find("null") == StringRef::not_found) { - int smooth = 0; - copy_string_to_int(rest_line, 0, smooth); - r_state_shaded_smooth = smooth != 0; - } - else { - /* The OBJ file explicitly set shading to off. */ + if (line == "0" || line.startswith("off") || line.startswith("null")) { r_state_shaded_smooth = false; + return; } + + int smooth = 0; + parse_int(line, 0, smooth); + r_state_shaded_smooth = smooth != 0; } -OBJParser::OBJParser(const OBJImportParams &import_params) : import_params_(import_params) +OBJParser::OBJParser(const OBJImportParams &import_params, size_t read_buffer_size = 64 * 1024) + : import_params_(import_params), read_buffer_size_(read_buffer_size) { - obj_file_.open(import_params_.filepath); - if (!obj_file_.good()) { + obj_file_ = BLI_fopen(import_params_.filepath, "rb"); + if (!obj_file_) { fprintf(stderr, "Cannot read from OBJ file:'%s'.\n", import_params_.filepath); return; } } +OBJParser::~OBJParser() +{ + if (obj_file_) { + fclose(obj_file_); + } +} + void OBJParser::parse(Vector<std::unique_ptr<Geometry>> &r_all_geometries, GlobalVertices &r_global_vertices) { - if (!obj_file_.good()) { + if (!obj_file_) { return; } - string line; - /* Store vertex coordinates that belong to other Geometry instances. */ VertexIndexOffset offsets; - /* Non owning raw pointer to a Geometry. To be updated while creating a new Geometry. */ Geometry *curr_geom = create_geometry( nullptr, GEOM_MESH, "", r_global_vertices, r_all_geometries, offsets); - /* State-setting variables: if set, they remain the same for the remaining + /* State variables: once set, they remain the same for the remaining * elements in the object. */ bool state_shaded_smooth = false; - string state_object_group; + string state_group_name; + int state_group_index = -1; string state_material_name; + int state_material_index = -1; + + /* Read the input file in chunks. We need up to twice the possible chunk size, + * to possibly store remainder of the previous input line that got broken mid-chunk. */ + Array<char> buffer(read_buffer_size_ * 2); + + size_t buffer_offset = 0; + size_t line_number = 0; + while (true) { + /* Read a chunk of input from the file. */ + size_t bytes_read = fread(buffer.data() + buffer_offset, 1, read_buffer_size_, obj_file_); + if (bytes_read == 0 && buffer_offset == 0) { + break; /* No more data to read. */ + } - while (std::getline(obj_file_, line)) { - /* Keep reading new lines if the last character is `\`. */ - /* Another way is to make a getline wrapper and use it in the while condition. */ - read_next_line(obj_file_, line); + /* Ensure buffer ends in a newline. */ + if (bytes_read < read_buffer_size_) { + if (bytes_read == 0 || buffer[buffer_offset + bytes_read - 1] != '\n') { + buffer[buffer_offset + bytes_read] = '\n'; + bytes_read++; + } + } - StringRef line_key, rest_line; - split_line_key_rest(line, line_key, rest_line); - if (line.empty() || rest_line.is_empty()) { - continue; + size_t buffer_end = buffer_offset + bytes_read; + if (buffer_end == 0) { + break; } - switch (line_key_str_to_enum(line_key)) { - case eOBJLineKey::V: { - geom_add_vertex(curr_geom, rest_line, r_global_vertices); - break; - } - case eOBJLineKey::VN: { - geom_add_vertex_normal(curr_geom, rest_line, r_global_vertices); - break; + + /* Find last newline. */ + size_t last_nl = buffer_end; + while (last_nl > 0) { + --last_nl; + if (buffer[last_nl] == '\n') { + if (last_nl < 1 || buffer[last_nl - 1] != '\\') + break; } - case eOBJLineKey::VT: { - geom_add_uv_vertex(rest_line, r_global_vertices); - break; + } + if (buffer[last_nl] != '\n') { + /* Whole line did not fit into our read buffer. Warn and exit. */ + fprintf(stderr, + "OBJ file contains a line #%zu that is too long (max. length %zu)\n", + line_number, + read_buffer_size_); + break; + } + ++last_nl; + + /* Parse the buffer (until last newline) that we have so far, + * line by line. */ + StringRef buffer_str{buffer.data(), (int64_t)last_nl}; + while (!buffer_str.is_empty()) { + StringRef line = read_next_line(buffer_str); + ++line_number; + if (line.is_empty()) + continue; + /* Most common things that start with 'v': vertices, normals, UVs. */ + if (line[0] == 'v') { + if (line.startswith("v ")) { + line = line.drop_prefix(2); + geom_add_vertex(curr_geom, line, r_global_vertices); + } + else if (line.startswith("vn ")) { + line = line.drop_prefix(3); + geom_add_vertex_normal(curr_geom, line, r_global_vertices); + } + else if (line.startswith("vt ")) { + line = line.drop_prefix(3); + geom_add_uv_vertex(line, r_global_vertices); + } } - case eOBJLineKey::F: { + /* Faces. */ + else if (line.startswith("f ")) { + line = line.drop_prefix(2); geom_add_polygon(curr_geom, - rest_line, + line, r_global_vertices, offsets, - state_material_name, - state_material_name, + state_material_index, + state_group_index, /* TODO was wrongly material name! */ state_shaded_smooth); - break; - } - case eOBJLineKey::L: { - geom_add_edge(curr_geom, rest_line, offsets, r_global_vertices); - break; - } - case eOBJLineKey::CSTYPE: { - curr_geom = geom_set_curve_type(curr_geom, - rest_line, - r_global_vertices, - state_object_group, - offsets, - r_all_geometries); - break; - } - case eOBJLineKey::DEG: { - geom_set_curve_degree(curr_geom, rest_line); - break; - } - case eOBJLineKey::CURV: { - geom_add_curve_vertex_indices(curr_geom, rest_line, r_global_vertices); - break; - } - case eOBJLineKey::PARM: { - geom_add_curve_parameters(curr_geom, rest_line); - break; - } - case eOBJLineKey::O: { + } + /* Faces. */ + else if (line.startswith("l ")) { + line = line.drop_prefix(2); + geom_add_edge(curr_geom, line, offsets, r_global_vertices); + } + /* Objects. */ + else if (line.startswith("o ")) { + line = line.drop_prefix(2); state_shaded_smooth = false; - state_object_group = ""; + state_group_name = ""; state_material_name = ""; curr_geom = create_geometry( - curr_geom, GEOM_MESH, rest_line, r_global_vertices, r_all_geometries, offsets); - break; - } - case eOBJLineKey::G: { - geom_update_object_group(rest_line, state_object_group); - break; - } - case eOBJLineKey::S: { - geom_update_smooth_group(rest_line, state_shaded_smooth); - break; - } - case eOBJLineKey::USEMTL: { - geom_update_polygon_material(curr_geom, rest_line, state_material_name); - break; - } - case eOBJLineKey::MTLLIB: { - mtl_libraries_.append(string(rest_line)); - break; - } - case eOBJLineKey::COMMENT: - break; - default: - std::cout << "Element not recognised: '" << line_key << "'" << std::endl; - break; + curr_geom, GEOM_MESH, line, r_global_vertices, r_all_geometries, offsets); + } + /* Groups. */ + else if (line.startswith("g ")) { + line = line.drop_prefix(2); + geom_update_group(line, state_group_name); + int new_index = curr_geom->group_indices_.size(); + state_group_index = curr_geom->group_indices_.lookup_or_add(state_group_name, new_index); + if (new_index == state_group_index) { + curr_geom->group_order_.append(state_group_name); + } + } + /* Smoothing groups. */ + else if (line.startswith("s ")) { + line = line.drop_prefix(2); + geom_update_smooth_group(line, state_shaded_smooth); + } + /* Materials and their libraries. */ + else if (line.startswith("usemtl ")) { + line = line.drop_prefix(7); + state_material_name = line; + int new_mat_index = curr_geom->material_indices_.size(); + state_material_index = curr_geom->material_indices_.lookup_or_add(state_material_name, + new_mat_index); + if (new_mat_index == state_material_index) { + curr_geom->material_order_.append(state_material_name); + } + } + else if (line.startswith("mtllib ")) { + line = line.drop_prefix(7); + mtl_libraries_.append(string(line)); + } + /* Comments. */ + else if (line.startswith("#")) { + /* Nothing to do. */ + } + /* Curve related things. */ + else if (line.startswith("cstype ")) { + line = line.drop_prefix(7); + curr_geom = geom_set_curve_type( + curr_geom, line, r_global_vertices, state_group_name, offsets, r_all_geometries); + } + else if (line.startswith("deg ")) { + line = line.drop_prefix(4); + geom_set_curve_degree(curr_geom, line); + } + else if (line.startswith("curv ")) { + line = line.drop_prefix(5); + geom_add_curve_vertex_indices(curr_geom, line, r_global_vertices); + } + else if (line.startswith("parm ")) { + line = line.drop_prefix(5); + geom_add_curve_parameters(curr_geom, line); + } + else if (line.startswith("end")) { + /* End of curve definition, nothing else to do. */ + } + else { + std::cout << "OBJ element not recognised: '" << line << "'" << std::endl; + } } + + /* We might have a line that was cut in the middle by the previous buffer; + * copy it over for next chunk reading. */ + size_t left_size = buffer_end - last_nl; + memmove(buffer.data(), buffer.data() + last_nl, left_size); + buffer_offset = left_size; } } -/** - * Skip all texture map options and get the filepath from a "map_" line. - */ -static StringRef skip_unsupported_options(StringRef line) +static eMTLSyntaxElement mtl_line_start_to_enum(StringRef &line) { - TextureMapOptions map_options; - StringRef last_option; - int64_t last_option_pos = 0; - - /* Find the last texture map option. */ - for (StringRef option : map_options.all_options()) { - const int64_t pos{line.find(option)}; - /* Equality (>=) takes care of finding an option in the beginning of the line. Avoid messing - * with signed-unsigned int comparison. */ - if (pos != StringRef::not_found && pos >= last_option_pos) { - last_option = option; - last_option_pos = pos; - } + if (line.startswith("map_Kd")) { + line = line.drop_prefix(6); + return eMTLSyntaxElement::map_Kd; } - - if (last_option.is_empty()) { - /* No option found, line is the filepath */ - return line; + if (line.startswith("map_Ks")) { + line = line.drop_prefix(6); + return eMTLSyntaxElement::map_Ks; + } + if (line.startswith("map_Ns")) { + line = line.drop_prefix(6); + return eMTLSyntaxElement::map_Ns; + } + if (line.startswith("map_d")) { + line = line.drop_prefix(5); + return eMTLSyntaxElement::map_d; } + if (line.startswith("refl")) { + line = line.drop_prefix(4); + return eMTLSyntaxElement::map_refl; + } + if (line.startswith("map_refl")) { + line = line.drop_prefix(8); + return eMTLSyntaxElement::map_refl; + } + if (line.startswith("map_Ke")) { + line = line.drop_prefix(6); + return eMTLSyntaxElement::map_Ke; + } + if (line.startswith("bump")) { + line = line.drop_prefix(4); + return eMTLSyntaxElement::map_Bump; + } + if (line.startswith("map_Bump") || line.startswith("map_bump")) { + line = line.drop_prefix(8); + return eMTLSyntaxElement::map_Bump; + } + return eMTLSyntaxElement::string; +} - /* Remove up to start of the last option + size of the last option + space after it. */ - line = line.drop_prefix(last_option_pos + last_option.size() + 1); - for (int i = 0; i < map_options.number_of_args(last_option); i++) { - const int64_t pos_space{line.find_first_of(' ')}; - if (pos_space != StringRef::not_found) { - BLI_assert(pos_space + 1 < line.size()); - line = line.drop_prefix(pos_space + 1); +static const std::pair<const char *, int> unsupported_texture_options[] = { + {"-blendu ", 1}, + {"-blendv ", 1}, + {"-boost ", 1}, + {"-cc ", 1}, + {"-clamp ", 1}, + {"-imfchan ", 1}, + {"-mm ", 2}, + {"-t ", 3}, + {"-texres ", 1}, +}; + +static bool parse_texture_option(StringRef &line, MTLMaterial *material, tex_map_XX &tex_map) +{ + line = drop_whitespace(line); + if (line.startswith("-o ")) { + line = line.drop_prefix(3); + line = parse_floats(line, 0.0f, tex_map.translation, 3); + return true; + } + if (line.startswith("-s ")) { + line = line.drop_prefix(3); + line = parse_floats(line, 1.0f, tex_map.scale, 3); + return true; + } + if (line.startswith("-bm ")) { + line = line.drop_prefix(4); + line = parse_float(line, 0.0f, material->map_Bump_strength); + return true; + } + if (line.startswith("-type ")) { + line = line.drop_prefix(6); + line = drop_whitespace(line); + /* Only sphere is supported. */ + tex_map.projection_type = SHD_PROJ_SPHERE; + if (!line.startswith("sphere")) { + std::cerr << "OBJ import: only sphere MTL projection type is supported: '" << line << "'" + << std::endl; + } + line = drop_non_whitespace(line); + return true; + } + /* Check for unsupported options and skip them. */ + for (const auto &opt : unsupported_texture_options) { + if (line.startswith(opt.first)) { + /* Drop the option name. */ + line = line.drop_known_prefix(opt.first); + /* Drop the arguments. */ + for (int i = 0; i < opt.second; ++i) { + line = drop_whitespace(line); + line = drop_non_whitespace(line); + } + return true; } } - return line; + return false; } -/** - * Fix incoming texture map line keys for variations due to other exporters. - */ -static string fix_bad_map_keys(StringRef map_key) +static void parse_texture_map(StringRef line, MTLMaterial *material, const char *mtl_dir_path) { - string new_map_key(map_key); - if (map_key == "refl") { - new_map_key = "map_refl"; + bool is_map = line.startswith("map_"); + bool is_refl = line.startswith("refl"); + bool is_bump = line.startswith("bump"); + if (!is_map && !is_refl && !is_bump) { + return; + } + eMTLSyntaxElement key = mtl_line_start_to_enum(line); + if (key == eMTLSyntaxElement::string || !material->texture_maps.contains(key)) { + /* No supported texture map found. */ + std::cerr << "OBJ import: MTL texture map type not supported: '" << line << "'" << std::endl; + return; } - if (map_key.find("bump") != StringRef::not_found) { - /* Handles both "bump" and "map_Bump" */ - new_map_key = "map_Bump"; + tex_map_XX &tex_map = material->texture_maps.lookup(key); + tex_map.mtl_dir_path = mtl_dir_path; + + /* Parse texture map options. */ + while (parse_texture_option(line, material, tex_map)) { } - return new_map_key; + + /* What remains is the image path. */ + line = line.trim(); + tex_map.image_path = line; } Span<std::string> OBJParser::mtl_libraries() const @@ -503,125 +627,72 @@ MTLParser::MTLParser(StringRef mtl_library, StringRefNull obj_filepath) BLI_split_dir_part(obj_filepath.data(), obj_file_dir, FILE_MAXDIR); BLI_path_join(mtl_file_path_, FILE_MAX, obj_file_dir, mtl_library.data(), NULL); BLI_split_dir_part(mtl_file_path_, mtl_dir_path_, FILE_MAXDIR); - mtl_file_.open(mtl_file_path_); - if (!mtl_file_.good()) { - fprintf(stderr, "Cannot read from MTL file:'%s'\n", mtl_file_path_); - return; - } } -void MTLParser::parse_and_store(Map<string, std::unique_ptr<MTLMaterial>> &r_mtl_materials) +void MTLParser::parse_and_store(Map<string, std::unique_ptr<MTLMaterial>> &r_materials) { - if (!mtl_file_.good()) { + size_t buffer_len; + void *buffer = BLI_file_read_text_as_mem(mtl_file_path_, 0, &buffer_len); + if (buffer == nullptr) { + fprintf(stderr, "OBJ import: cannot read from MTL file: '%s'\n", mtl_file_path_); return; } - string line; - MTLMaterial *current_mtlmaterial = nullptr; + MTLMaterial *material = nullptr; - while (std::getline(mtl_file_, line)) { - StringRef line_key, rest_line; - split_line_key_rest(line, line_key, rest_line); - if (line.empty() || rest_line.is_empty()) { + StringRef buffer_str{(const char *)buffer, (int64_t)buffer_len}; + while (!buffer_str.is_empty()) { + StringRef line = read_next_line(buffer_str); + if (line.is_empty()) continue; - } - /* Fix lower case/ incomplete texture map identifiers. */ - const string fixed_key = fix_bad_map_keys(line_key); - line_key = fixed_key; - - if (line_key == "newmtl") { - if (r_mtl_materials.remove_as(rest_line)) { - std::cerr << "Duplicate material found:'" << rest_line + if (line.startswith("newmtl ")) { + line = line.drop_prefix(7); + if (r_materials.remove_as(line)) { + std::cerr << "Duplicate material found:'" << line << "', using the last encountered Material definition." << std::endl; } - current_mtlmaterial = - r_mtl_materials.lookup_or_add(string(rest_line), std::make_unique<MTLMaterial>()).get(); - } - else if (line_key == "Ns") { - copy_string_to_float(rest_line, 324.0f, current_mtlmaterial->Ns); - } - else if (line_key == "Ka") { - Vector<StringRef> str_ka_split; - split_by_char(rest_line, ' ', str_ka_split); - copy_string_to_float(str_ka_split, 0.0f, {current_mtlmaterial->Ka, 3}); - } - else if (line_key == "Kd") { - Vector<StringRef> str_kd_split; - split_by_char(rest_line, ' ', str_kd_split); - copy_string_to_float(str_kd_split, 0.8f, {current_mtlmaterial->Kd, 3}); - } - else if (line_key == "Ks") { - Vector<StringRef> str_ks_split; - split_by_char(rest_line, ' ', str_ks_split); - copy_string_to_float(str_ks_split, 0.5f, {current_mtlmaterial->Ks, 3}); - } - else if (line_key == "Ke") { - Vector<StringRef> str_ke_split; - split_by_char(rest_line, ' ', str_ke_split); - copy_string_to_float(str_ke_split, 0.0f, {current_mtlmaterial->Ke, 3}); - } - else if (line_key == "Ni") { - copy_string_to_float(rest_line, 1.45f, current_mtlmaterial->Ni); + material = r_materials.lookup_or_add(string(line), std::make_unique<MTLMaterial>()).get(); } - else if (line_key == "d") { - copy_string_to_float(rest_line, 1.0f, current_mtlmaterial->d); - } - else if (line_key == "illum") { - copy_string_to_int(rest_line, 2, current_mtlmaterial->illum); - } - - /* Parse image textures. */ - else if (line_key.find("map_") != StringRef::not_found) { - /* TODO(@howardt): fix this. */ - eMTLSyntaxElement line_key_enum = mtl_line_key_str_to_enum(line_key); - if (line_key_enum == eMTLSyntaxElement::string || - !current_mtlmaterial->texture_maps.contains_as(line_key_enum)) { - /* No supported texture map found. */ - std::cerr << "Texture map type not supported:'" << line_key << "'" << std::endl; - continue; + else if (material != nullptr) { + if (line.startswith("Ns ")) { + line = line.drop_prefix(3); + parse_float(line, 324.0f, material->Ns); } - tex_map_XX &tex_map = current_mtlmaterial->texture_maps.lookup(line_key_enum); - Vector<StringRef> str_map_xx_split; - split_by_char(rest_line, ' ', str_map_xx_split); - - /* TODO(@ankitm): use `skip_unsupported_options` for parsing these options too? */ - const int64_t pos_o{str_map_xx_split.first_index_of_try("-o")}; - if (pos_o != -1 && pos_o + 3 < str_map_xx_split.size()) { - copy_string_to_float({str_map_xx_split[pos_o + 1], - str_map_xx_split[pos_o + 2], - str_map_xx_split[pos_o + 3]}, - 0.0f, - {tex_map.translation, 3}); - } - const int64_t pos_s{str_map_xx_split.first_index_of_try("-s")}; - if (pos_s != -1 && pos_s + 3 < str_map_xx_split.size()) { - copy_string_to_float({str_map_xx_split[pos_s + 1], - str_map_xx_split[pos_s + 2], - str_map_xx_split[pos_s + 3]}, - 1.0f, - {tex_map.scale, 3}); - } - /* Only specific to Normal Map node. */ - const int64_t pos_bm{str_map_xx_split.first_index_of_try("-bm")}; - if (pos_bm != -1 && pos_bm + 1 < str_map_xx_split.size()) { - copy_string_to_float( - str_map_xx_split[pos_bm + 1], 0.0f, current_mtlmaterial->map_Bump_strength); - } - const int64_t pos_projection{str_map_xx_split.first_index_of_try("-type")}; - if (pos_projection != -1 && pos_projection + 1 < str_map_xx_split.size()) { - /* Only Sphere is supported, so whatever the type is, set it to Sphere. */ - tex_map.projection_type = SHD_PROJ_SPHERE; - if (str_map_xx_split[pos_projection + 1] != "sphere") { - std::cerr << "Using projection type 'sphere', not:'" - << str_map_xx_split[pos_projection + 1] << "'." << std::endl; - } + else if (line.startswith("Ka ")) { + line = line.drop_prefix(3); + parse_floats(line, 0.0f, material->Ka, 3); + } + else if (line.startswith("Kd ")) { + line = line.drop_prefix(3); + parse_floats(line, 0.8f, material->Kd, 3); + } + else if (line.startswith("Ks ")) { + line = line.drop_prefix(3); + parse_floats(line, 0.5f, material->Ks, 3); + } + else if (line.startswith("Ke ")) { + line = line.drop_prefix(3); + parse_floats(line, 0.0f, material->Ke, 3); + } + else if (line.startswith("Ni ")) { + line = line.drop_prefix(3); + parse_float(line, 1.45f, material->Ni); + } + else if (line.startswith("d ")) { + line = line.drop_prefix(2); + parse_float(line, 1.0f, material->d); + } + else if (line.startswith("illum ")) { + line = line.drop_prefix(6); + parse_int(line, 2, material->illum); + } + else { + parse_texture_map(line, material, mtl_dir_path_); } - - /* Skip all unsupported options and arguments. */ - tex_map.image_path = string(skip_unsupported_options(rest_line)); - tex_map.mtl_dir_path = mtl_dir_path_; } } + + MEM_freeN(buffer); } } // namespace blender::io::obj diff --git a/source/blender/io/wavefront_obj/importer/obj_import_file_reader.hh b/source/blender/io/wavefront_obj/importer/obj_import_file_reader.hh index 24d026d75e5..8093417fcda 100644 --- a/source/blender/io/wavefront_obj/importer/obj_import_file_reader.hh +++ b/source/blender/io/wavefront_obj/importer/obj_import_file_reader.hh @@ -18,14 +18,16 @@ namespace blender::io::obj { class OBJParser { private: const OBJImportParams &import_params_; - blender::fstream obj_file_; + FILE *obj_file_; Vector<std::string> mtl_libraries_; + size_t read_buffer_size_; public: /** * Open OBJ file at the path given in import parameters. */ - OBJParser(const OBJImportParams &import_params); + OBJParser(const OBJImportParams &import_params, size_t read_buffer_size); + ~OBJParser(); /** * Read the OBJ file line by line and create OBJ Geometry instances. Also store all the vertex @@ -39,111 +41,6 @@ class OBJParser { Span<std::string> mtl_libraries() const; }; -enum class eOBJLineKey { - V, - VN, - VT, - F, - L, - CSTYPE, - DEG, - CURV, - PARM, - O, - G, - S, - USEMTL, - MTLLIB, - COMMENT -}; - -constexpr eOBJLineKey line_key_str_to_enum(const std::string_view key_str) -{ - if (key_str == "v" || key_str == "V") { - return eOBJLineKey::V; - } - if (key_str == "vn" || key_str == "VN") { - return eOBJLineKey::VN; - } - if (key_str == "vt" || key_str == "VT") { - return eOBJLineKey::VT; - } - if (key_str == "f" || key_str == "F") { - return eOBJLineKey::F; - } - if (key_str == "l" || key_str == "L") { - return eOBJLineKey::L; - } - if (key_str == "cstype" || key_str == "CSTYPE") { - return eOBJLineKey::CSTYPE; - } - if (key_str == "deg" || key_str == "DEG") { - return eOBJLineKey::DEG; - } - if (key_str == "curv" || key_str == "CURV") { - return eOBJLineKey::CURV; - } - if (key_str == "parm" || key_str == "PARM") { - return eOBJLineKey::PARM; - } - if (key_str == "o" || key_str == "O") { - return eOBJLineKey::O; - } - if (key_str == "g" || key_str == "G") { - return eOBJLineKey::G; - } - if (key_str == "s" || key_str == "S") { - return eOBJLineKey::S; - } - if (key_str == "usemtl" || key_str == "USEMTL") { - return eOBJLineKey::USEMTL; - } - if (key_str == "mtllib" || key_str == "MTLLIB") { - return eOBJLineKey::MTLLIB; - } - if (key_str == "#") { - return eOBJLineKey::COMMENT; - } - return eOBJLineKey::COMMENT; -} - -/** - * All texture map options with number of arguments they accept. - */ -class TextureMapOptions { - private: - Map<const std::string, int> tex_map_options; - - public: - TextureMapOptions() - { - tex_map_options.add_new("-blendu", 1); - tex_map_options.add_new("-blendv", 1); - tex_map_options.add_new("-boost", 1); - tex_map_options.add_new("-mm", 2); - tex_map_options.add_new("-o", 3); - tex_map_options.add_new("-s", 3); - tex_map_options.add_new("-t", 3); - tex_map_options.add_new("-texres", 1); - tex_map_options.add_new("-clamp", 1); - tex_map_options.add_new("-bm", 1); - tex_map_options.add_new("-imfchan", 1); - } - - /** - * All valid option strings. - */ - Map<const std::string, int>::KeyIterator all_options() const - { - return tex_map_options.keys(); - } - - int number_of_args(StringRef option) const - { - return tex_map_options.lookup_as(std::string(option)); - } -}; - class MTLParser { private: char mtl_file_path_[FILE_MAX]; @@ -151,7 +48,6 @@ class MTLParser { * Directory in which the MTL file is found. */ char mtl_dir_path_[FILE_MAX]; - blender::fstream mtl_file_; public: /** @@ -162,6 +58,6 @@ class MTLParser { /** * Read MTL file(s) and add MTLMaterial instances to the given Map reference. */ - void parse_and_store(Map<std::string, std::unique_ptr<MTLMaterial>> &r_mtl_materials); + void parse_and_store(Map<std::string, std::unique_ptr<MTLMaterial>> &r_materials); }; } // namespace blender::io::obj diff --git a/source/blender/io/wavefront_obj/importer/obj_import_mesh.cc b/source/blender/io/wavefront_obj/importer/obj_import_mesh.cc index 55b2873a3de..01a2d63927e 100644 --- a/source/blender/io/wavefront_obj/importer/obj_import_mesh.cc +++ b/source/blender/io/wavefront_obj/importer/obj_import_mesh.cc @@ -18,6 +18,7 @@ #include "BLI_math_vector.h" #include "BLI_set.hh" +#include "IO_wavefront_obj.h" #include "importer_mesh_utils.hh" #include "obj_import_mesh.hh" @@ -35,7 +36,7 @@ Object *MeshFromGeometry::create_mesh( } fixup_invalid_faces(); - const int64_t tot_verts_object{mesh_geometry_.vertex_indices_.size()}; + const int64_t tot_verts_object{mesh_geometry_.vertex_count_}; /* Total explicitly imported edges, not the ones belonging the polygons to be created. */ const int64_t tot_edges{mesh_geometry_.edges_.size()}; const int64_t tot_face_elems{mesh_geometry_.face_elements_.size()}; @@ -52,11 +53,13 @@ Object *MeshFromGeometry::create_mesh( create_normals(mesh); create_materials(bmain, materials, created_materials, obj); - bool verbose_validate = false; + if (import_params.validate_meshes || mesh_geometry_.has_invalid_polys_) { + bool verbose_validate = false; #ifdef DEBUG - verbose_validate = true; + verbose_validate = true; #endif - BKE_mesh_validate(mesh, verbose_validate, false); + BKE_mesh_validate(mesh, verbose_validate, false); + } transform_object(obj, import_params); /* FIXME: after 2.80; `mesh->flag` isn't copied by #BKE_mesh_nomain_to_mesh() */ @@ -73,9 +76,9 @@ void MeshFromGeometry::fixup_invalid_faces() for (int64_t face_idx = 0; face_idx < mesh_geometry_.face_elements_.size(); ++face_idx) { const PolyElem &curr_face = mesh_geometry_.face_elements_[face_idx]; - if (curr_face.face_corners.size() < 3) { + if (curr_face.corner_count_ < 3) { /* Skip and remove faces that have fewer than 3 corners. */ - mesh_geometry_.total_loops_ -= curr_face.face_corners.size(); + mesh_geometry_.total_loops_ -= curr_face.corner_count_; mesh_geometry_.face_elements_.remove_and_reorder(face_idx); continue; } @@ -84,12 +87,14 @@ void MeshFromGeometry::fixup_invalid_faces() * basically whether it has duplicate vertex indices. */ bool valid = true; Set<int, 8> used_verts; - for (const PolyCorner &corner : curr_face.face_corners) { - if (used_verts.contains(corner.vert_index)) { + for (int i = 0; i < curr_face.corner_count_; ++i) { + int corner_idx = curr_face.start_index_ + i; + int vertex_idx = mesh_geometry_.face_corners_[corner_idx].vert_index; + if (used_verts.contains(vertex_idx)) { valid = false; break; } - used_verts.add(corner.vert_index); + used_verts.add(vertex_idx); } if (valid) { continue; @@ -100,20 +105,22 @@ void MeshFromGeometry::fixup_invalid_faces() Vector<int, 8> face_verts; Vector<int, 8> face_uvs; Vector<int, 8> face_normals; - face_verts.reserve(curr_face.face_corners.size()); - face_uvs.reserve(curr_face.face_corners.size()); - face_normals.reserve(curr_face.face_corners.size()); - for (const PolyCorner &corner : curr_face.face_corners) { + face_verts.reserve(curr_face.corner_count_); + face_uvs.reserve(curr_face.corner_count_); + face_normals.reserve(curr_face.corner_count_); + for (int i = 0; i < curr_face.corner_count_; ++i) { + int corner_idx = curr_face.start_index_ + i; + const PolyCorner &corner = mesh_geometry_.face_corners_[corner_idx]; face_verts.append(corner.vert_index); face_normals.append(corner.vertex_normal_index); face_uvs.append(corner.uv_vert_index); } - std::string face_vertex_group = curr_face.vertex_group; - std::string face_material_name = curr_face.material_name; + int face_vertex_group = curr_face.vertex_group_index; + int face_material = curr_face.material_index; bool face_shaded_smooth = curr_face.shaded_smooth; /* Remove the invalid face. */ - mesh_geometry_.total_loops_ -= curr_face.face_corners.size(); + mesh_geometry_.total_loops_ -= curr_face.corner_count_; mesh_geometry_.face_elements_.remove_and_reorder(face_idx); Vector<Vector<int>> new_faces = fixup_invalid_polygon(global_vertices_.vertices, face_verts); @@ -124,13 +131,14 @@ void MeshFromGeometry::fixup_invalid_faces() continue; } PolyElem new_face{}; - new_face.vertex_group = face_vertex_group; - new_face.material_name = face_material_name; + new_face.vertex_group_index = face_vertex_group; + new_face.material_index = face_material; new_face.shaded_smooth = face_shaded_smooth; - new_face.face_corners.reserve(face.size()); + new_face.start_index_ = mesh_geometry_.face_corners_.size(); + new_face.corner_count_ = face.size(); for (int idx : face) { BLI_assert(idx >= 0 && idx < face_verts.size()); - new_face.face_corners.append({face_verts[idx], face_uvs[idx], face_normals[idx]}); + mesh_geometry_.face_corners_.append({face_verts[idx], face_uvs[idx], face_normals[idx]}); } mesh_geometry_.face_elements_.append(new_face); mesh_geometry_.total_loops_ += face.size(); @@ -140,13 +148,14 @@ void MeshFromGeometry::fixup_invalid_faces() void MeshFromGeometry::create_vertices(Mesh *mesh) { - const int64_t tot_verts_object{mesh_geometry_.vertex_indices_.size()}; + const int tot_verts_object{mesh_geometry_.vertex_count_}; for (int i = 0; i < tot_verts_object; ++i) { - if (mesh_geometry_.vertex_indices_[i] < global_vertices_.vertices.size()) { - copy_v3_v3(mesh->mvert[i].co, global_vertices_.vertices[mesh_geometry_.vertex_indices_[i]]); + int vi = mesh_geometry_.vertex_start_ + i; + if (vi < global_vertices_.vertices.size()) { + copy_v3_v3(mesh->mvert[i].co, global_vertices_.vertices[vi]); } else { - std::cerr << "Vertex index:" << mesh_geometry_.vertex_indices_[i] + std::cerr << "Vertex index:" << vi << " larger than total vertices:" << global_vertices_.vertices.size() << " ." << std::endl; } @@ -158,7 +167,7 @@ void MeshFromGeometry::create_polys_loops(Object *obj, Mesh *mesh) /* Will not be used if vertex groups are not imported. */ mesh->dvert = nullptr; float weight = 0.0f; - const int64_t total_verts = mesh_geometry_.vertex_indices_.size(); + const int64_t total_verts = mesh_geometry_.vertex_count_; if (total_verts && mesh_geometry_.use_vertex_groups_) { mesh->dvert = static_cast<MDeformVert *>( CustomData_add_layer(&mesh->vdata, CD_MDEFORMVERT, CD_CALLOC, nullptr, total_verts)); @@ -168,34 +177,32 @@ void MeshFromGeometry::create_polys_loops(Object *obj, Mesh *mesh) UNUSED_VARS(weight); } - /* Do not remove elements from the VectorSet since order of insertion is required. - * StringRef is fine since per-face deform group name outlives the VectorSet. */ - VectorSet<StringRef> group_names; const int64_t tot_face_elems{mesh->totpoly}; int tot_loop_idx = 0; for (int poly_idx = 0; poly_idx < tot_face_elems; ++poly_idx) { const PolyElem &curr_face = mesh_geometry_.face_elements_[poly_idx]; - if (curr_face.face_corners.size() < 3) { + if (curr_face.corner_count_ < 3) { /* Don't add single vertex face, or edges. */ std::cerr << "Face with less than 3 vertices found, skipping." << std::endl; continue; } MPoly &mpoly = mesh->mpoly[poly_idx]; - mpoly.totloop = curr_face.face_corners.size(); + mpoly.totloop = curr_face.corner_count_; mpoly.loopstart = tot_loop_idx; if (curr_face.shaded_smooth) { mpoly.flag |= ME_SMOOTH; } - mpoly.mat_nr = mesh_geometry_.material_names_.index_of_try(curr_face.material_name); + mpoly.mat_nr = curr_face.material_index; /* Importing obj files without any materials would result in negative indices, which is not * supported. */ if (mpoly.mat_nr < 0) { mpoly.mat_nr = 0; } - for (const PolyCorner &curr_corner : curr_face.face_corners) { + for (int idx = 0; idx < curr_face.corner_count_; ++idx) { + const PolyCorner &curr_corner = mesh_geometry_.face_corners_[curr_face.start_index_ + idx]; MLoop &mloop = mesh->mloop[tot_loop_idx]; tot_loop_idx++; mloop.v = curr_corner.vert_index; @@ -212,23 +219,17 @@ void MeshFromGeometry::create_polys_loops(Object *obj, Mesh *mesh) MEM_callocN(sizeof(MDeformWeight), "OBJ Import Deform Weight")); } /* Every vertex in a face is assigned the same deform group. */ - int64_t pos_name{group_names.index_of_try(curr_face.vertex_group)}; - if (pos_name == -1) { - group_names.add_new(curr_face.vertex_group); - pos_name = group_names.size() - 1; - } - BLI_assert(pos_name >= 0); + int group_idx = curr_face.vertex_group_index; /* Deform group number (def_nr) must behave like an index into the names' list. */ - *(def_vert.dw) = {static_cast<unsigned int>(pos_name), weight}; + *(def_vert.dw) = {static_cast<unsigned int>(group_idx), weight}; } } if (!mesh->dvert) { return; } - /* Add deform group(s) to the object's defbase. */ - for (StringRef name : group_names) { - /* Adding groups in this order assumes that def_nr is an index into the names' list. */ + /* Add deform group names. */ + for (const std::string &name : mesh_geometry_.group_order_) { BKE_object_defgroup_add_name(obj, name.data()); } } @@ -236,7 +237,7 @@ void MeshFromGeometry::create_polys_loops(Object *obj, Mesh *mesh) void MeshFromGeometry::create_edges(Mesh *mesh) { const int64_t tot_edges{mesh_geometry_.edges_.size()}; - const int64_t total_verts{mesh_geometry_.vertex_indices_.size()}; + const int64_t total_verts{mesh_geometry_.vertex_count_}; UNUSED_VARS_NDEBUG(total_verts); for (int i = 0; i < tot_edges; ++i) { const MEdge &src_edge = mesh_geometry_.edges_[i]; @@ -263,7 +264,8 @@ void MeshFromGeometry::create_uv_verts(Mesh *mesh) int tot_loop_idx = 0; for (const PolyElem &curr_face : mesh_geometry_.face_elements_) { - for (const PolyCorner &curr_corner : curr_face.face_corners) { + for (int idx = 0; idx < curr_face.corner_count_; ++idx) { + const PolyCorner &curr_corner = mesh_geometry_.face_corners_[curr_face.start_index_ + idx]; if (curr_corner.uv_vert_index >= 0 && curr_corner.uv_vert_index < global_vertices_.uv_vertices.size()) { const float2 &mluv_src = global_vertices_.uv_vertices[curr_corner.uv_vert_index]; @@ -317,7 +319,7 @@ void MeshFromGeometry::create_materials( Map<std::string, Material *> &created_materials, Object *obj) { - for (const std::string &name : mesh_geometry_.material_names_) { + for (const std::string &name : mesh_geometry_.material_order_) { Material *mat = get_or_create_material(bmain, name, materials, created_materials); if (mat == nullptr) { continue; @@ -340,7 +342,8 @@ void MeshFromGeometry::create_normals(Mesh *mesh) MEM_malloc_arrayN(mesh_geometry_.total_loops_, sizeof(float[3]), __func__)); int tot_loop_idx = 0; for (const PolyElem &curr_face : mesh_geometry_.face_elements_) { - for (const PolyCorner &curr_corner : curr_face.face_corners) { + for (int idx = 0; idx < curr_face.corner_count_; ++idx) { + const PolyCorner &curr_corner = mesh_geometry_.face_corners_[curr_face.start_index_ + idx]; int n_index = curr_corner.vertex_normal_index; float3 normal(0, 0, 0); if (n_index >= 0) { diff --git a/source/blender/io/wavefront_obj/importer/obj_import_mesh.hh b/source/blender/io/wavefront_obj/importer/obj_import_mesh.hh index 2a838215421..7cc7ed25ad1 100644 --- a/source/blender/io/wavefront_obj/importer/obj_import_mesh.hh +++ b/source/blender/io/wavefront_obj/importer/obj_import_mesh.hh @@ -45,11 +45,8 @@ class MeshFromGeometry : NonMovable, NonCopyable { void fixup_invalid_faces(); void create_vertices(Mesh *mesh); /** - * Create polygons for the Mesh, set smooth shading flag, deform group name, - * assigned material also. - * - * It must receive all polygons to be added to the mesh. - * Remove holes from polygons before * calling this. + * Create polygons for the Mesh, set smooth shading flags, deform group names, + * Materials. */ void create_polys_loops(Object *obj, Mesh *mesh); /** diff --git a/source/blender/io/wavefront_obj/importer/obj_import_mtl.cc b/source/blender/io/wavefront_obj/importer/obj_import_mtl.cc index 88a8c07e325..f2a8941e8a7 100644 --- a/source/blender/io/wavefront_obj/importer/obj_import_mtl.cc +++ b/source/blender/io/wavefront_obj/importer/obj_import_mtl.cc @@ -13,12 +13,13 @@ #include "DNA_material_types.h" #include "DNA_node_types.h" +#include "IO_string_utils.hh" + #include "NOD_shader.h" /* TODO: move eMTLSyntaxElement out of following file into a more neutral place */ #include "obj_export_io.hh" #include "obj_import_mtl.hh" -#include "parser_string_utils.hh" namespace blender::io::obj { @@ -96,13 +97,16 @@ static bool load_texture_image(Main *bmain, const tex_map_XX &tex_map, bNode *r_ return true; } /* Try removing quotes. */ - std::string no_quote_path{replace_all_occurences(tex_path, "\"", "")}; + std::string no_quote_path{tex_path}; + auto end_pos = std::remove(no_quote_path.begin(), no_quote_path.end(), '"'); + no_quote_path.erase(end_pos, no_quote_path.end()); if (no_quote_path != tex_path && load_texture_image_at_path(bmain, tex_map, r_node, no_quote_path)) { return true; } /* Try replacing underscores with spaces. */ - std::string no_underscore_path{replace_all_occurences(no_quote_path, "_", " ")}; + std::string no_underscore_path{no_quote_path}; + std::replace(no_underscore_path.begin(), no_underscore_path.end(), '_', ' '); if (no_underscore_path != no_quote_path && no_underscore_path != tex_path && load_texture_image_at_path(bmain, tex_map, r_node, no_underscore_path)) { return true; diff --git a/source/blender/io/wavefront_obj/importer/obj_import_mtl.hh b/source/blender/io/wavefront_obj/importer/obj_import_mtl.hh index 4b7827b2035..74bc9f21bc4 100644 --- a/source/blender/io/wavefront_obj/importer/obj_import_mtl.hh +++ b/source/blender/io/wavefront_obj/importer/obj_import_mtl.hh @@ -90,29 +90,4 @@ class ShaderNodetreeWrap { void add_image_textures(Main *bmain, Material *mat); }; -constexpr eMTLSyntaxElement mtl_line_key_str_to_enum(const std::string_view key_str) -{ - if (key_str == "map_Kd") { - return eMTLSyntaxElement::map_Kd; - } - if (key_str == "map_Ks") { - return eMTLSyntaxElement::map_Ks; - } - if (key_str == "map_Ns") { - return eMTLSyntaxElement::map_Ns; - } - if (key_str == "map_d") { - return eMTLSyntaxElement::map_d; - } - if (key_str == "refl" || key_str == "map_refl") { - return eMTLSyntaxElement::map_refl; - } - if (key_str == "map_Ke") { - return eMTLSyntaxElement::map_Ke; - } - if (key_str == "map_Bump" || key_str == "bump") { - return eMTLSyntaxElement::map_Bump; - } - return eMTLSyntaxElement::string; -} } // namespace blender::io::obj diff --git a/source/blender/io/wavefront_obj/importer/obj_import_objects.hh b/source/blender/io/wavefront_obj/importer/obj_import_objects.hh index c6ce7d3c434..b67ba46af03 100644 --- a/source/blender/io/wavefront_obj/importer/obj_import_objects.hh +++ b/source/blender/io/wavefront_obj/importer/obj_import_objects.hh @@ -8,6 +8,7 @@ #include "BKE_lib_id.h" +#include "BLI_map.hh" #include "BLI_math_vec_types.hh" #include "BLI_vector.hh" #include "BLI_vector_set.hh" @@ -61,10 +62,11 @@ struct PolyCorner { }; struct PolyElem { - std::string vertex_group; - std::string material_name; + int vertex_group_index = -1; + int material_index = -1; bool shaded_smooth = false; - Vector<PolyCorner> face_corners; + int start_index_ = 0; + int corner_count_ = 0; }; /** @@ -93,15 +95,20 @@ enum eGeometryType { struct Geometry { eGeometryType geom_type_ = GEOM_MESH; std::string geometry_name_; - VectorSet<std::string> material_names_; - /** - * Indices in the vector range from zero to total vertices in a geometry. - * Values range from zero to total coordinates in the global list. - */ - Vector<int> vertex_indices_; + Map<std::string, int> group_indices_; + Vector<std::string> group_order_; + Map<std::string, int> material_indices_; + Vector<std::string> material_order_; + + int vertex_start_ = 0; + int vertex_count_ = 0; /** Edges written in the file in addition to (or even without polygon) elements. */ Vector<MEdge> edges_; + + Vector<PolyCorner> face_corners_; Vector<PolyElem> face_elements_; + + bool has_invalid_polys_ = false; bool has_vertex_normals_ = false; bool use_vertex_groups_ = false; NurbsElement nurbs_element_; diff --git a/source/blender/io/wavefront_obj/importer/obj_importer.cc b/source/blender/io/wavefront_obj/importer/obj_importer.cc index 631ddcc5cf4..c21d2d9583c 100644 --- a/source/blender/io/wavefront_obj/importer/obj_importer.cc +++ b/source/blender/io/wavefront_obj/importer/obj_importer.cc @@ -42,6 +42,12 @@ static void geometry_to_blender_objects( BKE_view_layer_base_deselect_all(view_layer); LayerCollection *lc = BKE_layer_collection_get_active(view_layer); + /* Don't do collection syncs for each object, will do once after the loop. */ + BKE_layer_collection_resync_forbid(); + + /* Create all the objects. */ + Vector<Object *> objects; + objects.reserve(all_geometries.size()); for (const std::unique_ptr<Geometry> &geometry : all_geometries) { Object *obj = nullptr; if (geometry->geom_type_ == GEOM_MESH) { @@ -54,17 +60,25 @@ static void geometry_to_blender_objects( } if (obj != nullptr) { BKE_collection_object_add(bmain, lc->collection, obj); - Base *base = BKE_view_layer_base_find(view_layer, obj); - /* TODO: is setting active needed? */ - BKE_view_layer_base_select_and_set_active(view_layer, base); - - DEG_id_tag_update(&lc->collection->id, ID_RECALC_COPY_ON_WRITE); - DEG_id_tag_update_ex(bmain, - &obj->id, - ID_RECALC_TRANSFORM | ID_RECALC_GEOMETRY | ID_RECALC_ANIMATION | - ID_RECALC_BASE_FLAGS); + objects.append(obj); } } + + /* Sync the collection after all objects are created. */ + BKE_layer_collection_resync_allow(); + BKE_main_collection_sync(bmain); + + /* After collection sync, select objects in the view layer and do DEG updates. */ + for (Object *obj : objects) { + Base *base = BKE_view_layer_base_find(view_layer, obj); + BKE_view_layer_base_select_and_set_active(view_layer, base); + + DEG_id_tag_update(&lc->collection->id, ID_RECALC_COPY_ON_WRITE); + int flags = ID_RECALC_TRANSFORM | ID_RECALC_GEOMETRY | ID_RECALC_ANIMATION | + ID_RECALC_BASE_FLAGS; + DEG_id_tag_update_ex(bmain, &obj->id, flags); + } + DEG_id_tag_update(&scene->id, ID_RECALC_BASE_FLAGS); DEG_relations_tag_update(bmain); } @@ -81,7 +95,8 @@ void importer_main(bContext *C, const OBJImportParams &import_params) void importer_main(Main *bmain, Scene *scene, ViewLayer *view_layer, - const OBJImportParams &import_params) + const OBJImportParams &import_params, + size_t read_buffer_size) { /* List of Geometry instances to be parsed from OBJ file. */ Vector<std::unique_ptr<Geometry>> all_geometries; @@ -91,7 +106,7 @@ void importer_main(Main *bmain, Map<std::string, std::unique_ptr<MTLMaterial>> materials; Map<std::string, Material *> created_materials; - OBJParser obj_parser{import_params}; + OBJParser obj_parser{import_params, read_buffer_size}; obj_parser.parse(all_geometries, global_vertices); for (StringRef mtl_library : obj_parser.mtl_libraries()) { diff --git a/source/blender/io/wavefront_obj/importer/obj_importer.hh b/source/blender/io/wavefront_obj/importer/obj_importer.hh index fd83117ebc6..35f401d7cb0 100644 --- a/source/blender/io/wavefront_obj/importer/obj_importer.hh +++ b/source/blender/io/wavefront_obj/importer/obj_importer.hh @@ -17,6 +17,7 @@ void importer_main(bContext *C, const OBJImportParams &import_params); void importer_main(Main *bmain, Scene *scene, ViewLayer *view_layer, - const OBJImportParams &import_params); + const OBJImportParams &import_params, + size_t read_buffer_size = 64 * 1024); } // namespace blender::io::obj diff --git a/source/blender/io/wavefront_obj/importer/parser_string_utils.cc b/source/blender/io/wavefront_obj/importer/parser_string_utils.cc deleted file mode 100644 index 6671a86f5ee..00000000000 --- a/source/blender/io/wavefront_obj/importer/parser_string_utils.cc +++ /dev/null @@ -1,174 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ - -#include <fstream> -#include <iostream> -#include <sstream> - -#include "BLI_math_vec_types.hh" -#include "BLI_span.hh" -#include "BLI_string_ref.hh" -#include "BLI_vector.hh" - -#include "parser_string_utils.hh" - -/* Note: these OBJ parser helper functions are planned to get fairly large - * changes "soon", so don't read too much into current implementation... */ - -namespace blender::io::obj { -using std::string; - -void read_next_line(std::fstream &file, string &r_line) -{ - std::string new_line; - while (file.good() && !r_line.empty() && r_line.back() == '\\') { - new_line.clear(); - const bool ok = static_cast<bool>(std::getline(file, new_line)); - /* Remove the last backslash character. */ - r_line.pop_back(); - r_line.append(new_line); - if (!ok || new_line.empty()) { - return; - } - } -} - -void split_line_key_rest(const StringRef line, StringRef &r_line_key, StringRef &r_rest_line) -{ - if (line.is_empty()) { - return; - } - - const int64_t pos_split{line.find_first_of(' ')}; - if (pos_split == StringRef::not_found) { - /* Use the first character if no space is found in the line. It's usually a comment like: - * #This is a comment. */ - r_line_key = line.substr(0, 1); - } - else { - r_line_key = line.substr(0, pos_split); - } - - /* Eat the delimiter also using "+ 1". */ - r_rest_line = line.drop_prefix(r_line_key.size() + 1); - if (r_rest_line.is_empty()) { - return; - } - - /* Remove any leading spaces, trailing spaces & \r character, if any. */ - const int64_t leading_space{r_rest_line.find_first_not_of(' ')}; - if (leading_space != StringRef::not_found) { - r_rest_line = r_rest_line.drop_prefix(leading_space); - } - - /* Another way is to do a test run before the actual parsing to find the newline - * character and use it in the getline. */ - const int64_t carriage_return{r_rest_line.find_first_of('\r')}; - if (carriage_return != StringRef::not_found) { - r_rest_line = r_rest_line.substr(0, carriage_return + 1); - } - - const int64_t trailing_space{r_rest_line.find_last_not_of(' ')}; - if (trailing_space != StringRef::not_found) { - /* The position is of a character that is not ' ', so count of characters is position + 1. */ - r_rest_line = r_rest_line.substr(0, trailing_space + 1); - } -} - -void split_by_char(StringRef in_string, const char delimiter, Vector<StringRef> &r_out_list) -{ - r_out_list.clear(); - - while (!in_string.is_empty()) { - const int64_t pos_delim{in_string.find_first_of(delimiter)}; - const int64_t word_len = pos_delim == StringRef::not_found ? in_string.size() : pos_delim; - - StringRef word{in_string.data(), word_len}; - if (!word.is_empty() && !(word == " " && !(word[0] == '\0'))) { - r_out_list.append(word); - } - if (pos_delim == StringRef::not_found) { - return; - } - /* Skip the word already stored. */ - in_string = in_string.drop_prefix(word_len); - /* Skip all delimiters. */ - const int64_t pos_non_delim = in_string.find_first_not_of(delimiter); - if (pos_non_delim == StringRef::not_found) { - return; - } - in_string = in_string.drop_prefix(std::min(pos_non_delim, in_string.size())); - } -} - -void copy_string_to_float(StringRef src, const float fallback_value, float &r_dst) -{ - try { - r_dst = std::stof(string(src)); - } - catch (const std::invalid_argument &inv_arg) { - std::cerr << "Bad conversion to float:'" << inv_arg.what() << "':'" << src << "'" << std::endl; - r_dst = fallback_value; - } - catch (const std::out_of_range &out_of_range) { - std::cerr << "Out of range for float:'" << out_of_range.what() << ":'" << src << "'" - << std::endl; - r_dst = fallback_value; - } -} - -void copy_string_to_float(Span<StringRef> src, - const float fallback_value, - MutableSpan<float> r_dst) -{ - for (int i = 0; i < r_dst.size(); ++i) { - if (i < src.size()) { - copy_string_to_float(src[i], fallback_value, r_dst[i]); - } - else { - r_dst[i] = fallback_value; - } - } -} - -void copy_string_to_int(StringRef src, const int fallback_value, int &r_dst) -{ - try { - r_dst = std::stoi(string(src)); - } - catch (const std::invalid_argument &inv_arg) { - std::cerr << "Bad conversion to int:'" << inv_arg.what() << "':'" << src << "'" << std::endl; - r_dst = fallback_value; - } - catch (const std::out_of_range &out_of_range) { - std::cerr << "Out of range for int:'" << out_of_range.what() << ":'" << src << "'" - << std::endl; - r_dst = fallback_value; - } -} - -void copy_string_to_int(Span<StringRef> src, const int fallback_value, MutableSpan<int> r_dst) -{ - for (int i = 0; i < r_dst.size(); ++i) { - if (i < src.size()) { - copy_string_to_int(src[i], fallback_value, r_dst[i]); - } - else { - r_dst[i] = fallback_value; - } - } -} - -std::string replace_all_occurences(StringRef original, StringRef to_remove, StringRef to_add) -{ - std::string clean{original}; - while (true) { - const std::string::size_type pos = clean.find(to_remove); - if (pos == std::string::npos) { - break; - } - clean.replace(pos, to_add.size(), to_add); - } - return clean; -} - -} // namespace blender::io::obj diff --git a/source/blender/io/wavefront_obj/importer/parser_string_utils.hh b/source/blender/io/wavefront_obj/importer/parser_string_utils.hh deleted file mode 100644 index 62cfbebccf3..00000000000 --- a/source/blender/io/wavefront_obj/importer/parser_string_utils.hh +++ /dev/null @@ -1,54 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ - -namespace blender::io::obj { - -/* Note: these OBJ parser helper functions are planned to get fairly large - * changes "soon", so don't read too much into current implementation... */ - -/** - * Store multiple lines separated by an escaped newline character: `\\n`. - * Use this before doing any parse operations on the read string. - */ -void read_next_line(std::fstream &file, std::string &r_line); -/** - * Split a line string into the first word (key) and the rest of the line. - * Also remove leading & trailing spaces as well as `\r` carriage return - * character if present. - */ -void split_line_key_rest(StringRef line, StringRef &r_line_key, StringRef &r_rest_line); -/** - * Split the given string by the delimiter and fill the given vector. - * If an intermediate string is empty, or space or null character, it is not appended to the - * vector. - */ -void split_by_char(StringRef in_string, const char delimiter, Vector<StringRef> &r_out_list); -/** - * Convert the given string to float and assign it to the destination value. - * - * If the string cannot be converted to a float, the fallback value is used. - */ -void copy_string_to_float(StringRef src, const float fallback_value, float &r_dst); -/** - * Convert all members of the Span of strings to floats and assign them to the float - * array members. Usually used for values like coordinates. - * - * If a string cannot be converted to a float, the fallback value is used. - */ -void copy_string_to_float(Span<StringRef> src, - const float fallback_value, - MutableSpan<float> r_dst); -/** - * Convert the given string to int and assign it to the destination value. - * - * If the string cannot be converted to an integer, the fallback value is used. - */ -void copy_string_to_int(StringRef src, const int fallback_value, int &r_dst); -/** - * Convert the given strings to ints and fill the destination int buffer. - * - * If a string cannot be converted to an integer, the fallback value is used. - */ -void copy_string_to_int(Span<StringRef> src, const int fallback_value, MutableSpan<int> r_dst); -std::string replace_all_occurences(StringRef original, StringRef to_remove, StringRef to_add); - -} // namespace blender::io::obj diff --git a/source/blender/io/wavefront_obj/tests/obj_importer_tests.cc b/source/blender/io/wavefront_obj/tests/obj_importer_tests.cc index 611e0cbb209..3d34fb6f9c6 100644 --- a/source/blender/io/wavefront_obj/tests/obj_importer_tests.cc +++ b/source/blender/io/wavefront_obj/tests/obj_importer_tests.cc @@ -60,7 +60,8 @@ class obj_importer_test : public BlendfileLoadingBaseTest { std::string obj_path = blender::tests::flags_test_asset_dir() + "/io_tests/obj/" + path; strncpy(params.filepath, obj_path.c_str(), FILE_MAX - 1); - importer_main(bfile->main, bfile->curscene, bfile->cur_view_layer, params); + const size_t read_buffer_size = 650; + importer_main(bfile->main, bfile->curscene, bfile->cur_view_layer, params, read_buffer_size); depsgraph_create(DAG_EVAL_VIEWPORT); diff --git a/source/blender/io/wavefront_obj/tests/obj_mtl_parser_tests.cc b/source/blender/io/wavefront_obj/tests/obj_mtl_parser_tests.cc new file mode 100644 index 00000000000..176d32a0be4 --- /dev/null +++ b/source/blender/io/wavefront_obj/tests/obj_mtl_parser_tests.cc @@ -0,0 +1,172 @@ +/* SPDX-License-Identifier: Apache-2.0 */ + +#include <gtest/gtest.h> + +#include "testing/testing.h" + +#include "obj_import_file_reader.hh" + +namespace blender::io::obj { + +class obj_mtl_parser_test : public testing::Test { + public: + void check(const char *file, const MTLMaterial *expect, size_t expect_count) + { + std::string obj_dir = blender::tests::flags_test_asset_dir() + "/io_tests/obj/"; + MTLParser parser(file, obj_dir + "dummy.obj"); + Map<std::string, std::unique_ptr<MTLMaterial>> materials; + parser.parse_and_store(materials); + + for (int i = 0; i < expect_count; ++i) { + const MTLMaterial &exp = expect[i]; + if (!materials.contains(exp.name)) { + fprintf(stderr, "Material '%s' was expected in parsed result\n", exp.name.c_str()); + ADD_FAILURE(); + continue; + } + const MTLMaterial &got = *materials.lookup(exp.name); + const float tol = 0.0001f; + EXPECT_V3_NEAR(exp.Ka, got.Ka, tol); + EXPECT_V3_NEAR(exp.Kd, got.Kd, tol); + EXPECT_V3_NEAR(exp.Ks, got.Ks, tol); + EXPECT_V3_NEAR(exp.Ke, got.Ke, tol); + EXPECT_NEAR(exp.Ns, got.Ns, tol); + EXPECT_NEAR(exp.Ni, got.Ni, tol); + EXPECT_NEAR(exp.d, got.d, tol); + EXPECT_NEAR(exp.map_Bump_strength, got.map_Bump_strength, tol); + EXPECT_EQ(exp.illum, got.illum); + for (const auto &it : exp.texture_maps.items()) { + const tex_map_XX &exp_tex = it.value; + const tex_map_XX &got_tex = got.texture_maps.lookup(it.key); + EXPECT_STREQ(exp_tex.image_path.c_str(), got_tex.image_path.c_str()); + EXPECT_V3_NEAR(exp_tex.translation, got_tex.translation, tol); + EXPECT_V3_NEAR(exp_tex.scale, got_tex.scale, tol); + EXPECT_EQ(exp_tex.projection_type, got_tex.projection_type); + } + } + EXPECT_EQ(materials.size(), expect_count); + } +}; + +TEST_F(obj_mtl_parser_test, cube) +{ + MTLMaterial mat; + mat.name = "red"; + mat.Ka = {0.2f, 0.2f, 0.2f}; + mat.Kd = {1, 0, 0}; + check("cube.mtl", &mat, 1); +} + +TEST_F(obj_mtl_parser_test, all_objects) +{ + MTLMaterial mat[7]; + for (auto &m : mat) { + m.Ka = {1, 1, 1}; + m.Ks = {0.5f, 0.5f, 0.5f}; + m.Ke = {0, 0, 0}; + m.Ns = 250; + m.Ni = 1; + m.d = 1; + m.illum = 2; + } + mat[0].name = "Blue"; + mat[0].Kd = {0, 0, 1}; + mat[1].name = "BlueDark"; + mat[1].Kd = {0, 0, 0.5f}; + mat[2].name = "Green"; + mat[2].Kd = {0, 1, 0}; + mat[3].name = "GreenDark"; + mat[3].Kd = {0, 0.5f, 0}; + mat[4].name = "Material"; + mat[4].Kd = {0.8f, 0.8f, 0.8f}; + mat[5].name = "Red"; + mat[5].Kd = {1, 0, 0}; + mat[6].name = "RedDark"; + mat[6].Kd = {0.5f, 0, 0}; + check("all_objects.mtl", mat, ARRAY_SIZE(mat)); +} + +TEST_F(obj_mtl_parser_test, materials) +{ + MTLMaterial mat[5]; + mat[0].name = "no_textures_red"; + mat[0].Ka = {0.3f, 0.3f, 0.3f}; + mat[0].Kd = {0.8f, 0.3f, 0.1f}; + mat[0].Ns = 5.624998f; + + mat[1].name = "four_maps"; + mat[1].Ka = {1, 1, 1}; + mat[1].Kd = {0.8f, 0.8f, 0.8f}; + mat[1].Ks = {0.5f, 0.5f, 0.5f}; + mat[1].Ke = {0, 0, 0}; + mat[1].Ns = 1000; + mat[1].Ni = 1.45f; + mat[1].d = 1; + mat[1].illum = 2; + mat[1].map_Bump_strength = 1; + { + tex_map_XX &kd = mat[1].tex_map_of_type(eMTLSyntaxElement::map_Kd); + kd.image_path = "texture.png"; + tex_map_XX &ns = mat[1].tex_map_of_type(eMTLSyntaxElement::map_Ns); + ns.image_path = "sometexture_Roughness.png"; + tex_map_XX &refl = mat[1].tex_map_of_type(eMTLSyntaxElement::map_refl); + refl.image_path = "sometexture_Metallic.png"; + tex_map_XX &bump = mat[1].tex_map_of_type(eMTLSyntaxElement::map_Bump); + bump.image_path = "sometexture_Normal.png"; + } + + mat[2].name = "Clay"; + mat[2].Ka = {1, 1, 1}; + mat[2].Kd = {0.8f, 0.682657f, 0.536371f}; + mat[2].Ks = {0.5f, 0.5f, 0.5f}; + mat[2].Ke = {0, 0, 0}; + mat[2].Ns = 440.924042f; + mat[2].Ni = 1.45f; + mat[2].d = 1; + mat[2].illum = 2; + + mat[3].name = "Hat"; + mat[3].Ka = {1, 1, 1}; + mat[3].Kd = {0.8f, 0.8f, 0.8f}; + mat[3].Ks = {0.5f, 0.5f, 0.5f}; + mat[3].Ns = 800; + mat[3].map_Bump_strength = 0.5f; + { + tex_map_XX &kd = mat[3].tex_map_of_type(eMTLSyntaxElement::map_Kd); + kd.image_path = "someHatTexture_BaseColor.jpg"; + tex_map_XX &ns = mat[3].tex_map_of_type(eMTLSyntaxElement::map_Ns); + ns.image_path = "someHatTexture_Roughness.jpg"; + tex_map_XX &refl = mat[3].tex_map_of_type(eMTLSyntaxElement::map_refl); + refl.image_path = "someHatTexture_Metalness.jpg"; + tex_map_XX &bump = mat[3].tex_map_of_type(eMTLSyntaxElement::map_Bump); + bump.image_path = "someHatTexture_Normal.jpg"; + } + + mat[4].name = "Parser_Test"; + mat[4].Ka = {0.1f, 0.2f, 0.3f}; + mat[4].Kd = {0.4f, 0.5f, 0.6f}; + mat[4].Ks = {0.7f, 0.8f, 0.9f}; + mat[4].illum = 6; + mat[4].Ns = 15.5; + mat[4].Ni = 1.5; + mat[4].d = 0.5; + mat[4].map_Bump_strength = 0.1f; + { + tex_map_XX &kd = mat[4].tex_map_of_type(eMTLSyntaxElement::map_Kd); + kd.image_path = "sometex_d.png"; + tex_map_XX &ns = mat[4].tex_map_of_type(eMTLSyntaxElement::map_Ns); + ns.image_path = "sometex_ns.psd"; + tex_map_XX &refl = mat[4].tex_map_of_type(eMTLSyntaxElement::map_refl); + refl.image_path = "clouds.tiff"; + refl.scale = {1.5f, 2.5f, 3.5f}; + refl.translation = {4.5f, 5.5f, 6.5f}; + refl.projection_type = SHD_PROJ_SPHERE; + tex_map_XX &bump = mat[4].tex_map_of_type(eMTLSyntaxElement::map_Bump); + bump.image_path = "somebump.tga"; + bump.scale = {3, 4, 5}; + } + + check("materials.mtl", mat, ARRAY_SIZE(mat)); +} + +} // namespace blender::io::obj |