diff options
| author | Taku Kudo <taku910@users.noreply.github.com> | 2020-06-28 06:32:15 +0300 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2020-06-28 06:32:15 +0300 |
| commit | 72ccd5f13b6e2a2de6e0dd879905a912171d1022 (patch) | |
| tree | c53732d6956b62077cce85a861d6eb21ec8688ed | |
| parent | be169e2ac98ef3ecb343ffcab3628a60372a2ee2 (diff) | |
| parent | a4b65ba893df19c3df7b1423e46668630f783d1b (diff) | |
Merge pull request #515 from mingruimingrui/bugfix/fix-trainer-spec-split-digit
[Bugfix] Add split_digits to SentencePieceTrainer spec parser
| -rw-r--r-- | src/spec_parser.h | 1 |
1 files changed, 1 insertions, 0 deletions
diff --git a/src/spec_parser.h b/src/spec_parser.h index 729e036..6dd054b 100644 --- a/src/spec_parser.h +++ b/src/spec_parser.h @@ -207,6 +207,7 @@ util::Status SentencePieceTrainer::SetProtoField(const std::string &name, PARSE_BOOL(split_by_unicode_script); PARSE_BOOL(split_by_number); PARSE_BOOL(split_by_whitespace); + PARSE_BOOL(split_digits); PARSE_BOOL(treat_whitespace_as_suffix); PARSE_REPEATED_STRING(control_symbols); PARSE_REPEATED_STRING(user_defined_symbols); |