Welcome to mirror list, hosted at ThFree Co, Russian Federation.

github.com/marian-nmt/sentencepiece.git - Unnamed repository; edit this file 'description' to name the repository.
summaryrefslogtreecommitdiff
path: root/src/sentencepiece_processor.cc
AgeCommit message (Expand)Author
2021-08-31Fix Surface String to Token Mappings for Case Encoding (#12)Rohit Jain
2021-06-16Enables --encode_unicode_case option for case-aware sentence piece (#10)Marcin Junczys-Dowmunt
2020-10-13support big-endian architectureTaku Kudo
2020-10-13support big-endian architectureTaku Kudo
2020-10-13merges internal changes to githubTaku Kudo
2020-06-01Port absl::flat_hash_mapTaku Kudo
2020-05-200.1.91 pre-releaseTaku Kudo
2020-05-17added interface to read from iterator/write to io bufferTaku Kudo
2020-05-07Initial release of 0.19. Merged internal sentencepiece.Taku Kudo
2019-01-09added --treat_whitespace_as_suffix option to make _ be a suffix of word.Taku Kudo
2019-01-08added (Encode|Decode)AsSerializedProto interface so Python module can get ful...Taku Kudo
2018-12-08support to change the piece of unk/bos/eos/padTaku Kudo
2018-11-13Fixed minor bugsTaku Kudo
2018-11-13Do not set UNUSED type to UNK/CONTROL/USER_DEFINED tokensTaku Kudo
2018-11-08stop normalization for user_defined_symbolsTaku Kudo
2018-09-16performance tuningsrTaku Kudo
2018-09-02Added self testing feature.Taku Kudo
2018-09-01Added is_binary flag to filesystem API.Taku Kudo
2018-09-01Introduced new Filesystem API phase 2.Taku Kudo
2018-09-01Introduced new Filesystem API.Taku Kudo
2018-08-14Added make_py_wheel_mac.pyTaku Kudo
2018-08-03Added JoinPath and StrCatTaku Kudo
2018-08-01Minor fixes for windowsTaku Kudo
2018-07-26Added --unk_surface option to allow user to change unknown surface string.Taku Kudo
2018-07-12Added new API to get bos/eos/unk/pad idsTaku Kudo
2018-06-22Add LoadFromSerialiedProtoTaku Kudo
2018-06-18Introduced minimum string_wrapper to remove extra string copyTaku Kudo
2018-06-18Uses abs::string_view instead of StringPieceTaku Kudo
2018-06-17Minor fixesTaku Kudo
2018-06-16Minor fixes.Taku Kudo
2018-06-11Support an empty normalziation and other minor fixesTaku Kudo
2018-06-06Support vocab restriction feature in BPE model.Taku Kudo
2018-06-06Support vocab restriction featureTaku Kudo
2018-06-04Minor style fixesTaku Kudo
2018-05-06CHECK to Status migration for Trainer.Taku Kudo
2018-04-30Reimplement Trainer with Proto reflectionTaku Kudo
2018-04-28Uses util::Status to propagate error messagesTaku Kudo
2018-04-09Support to change ids of <unk>, <s>, </s>Taku Kudo
2018-02-28Add Sample/NBestEncodeTaku Kudo
2017-07-23use pointer instead of not-const referenceresec
2017-07-20Impl std::istream variant to Load and LoadOrDieresec
2017-03-07Initialize repositoryTaku Kudo
    generated by cgit v1.2.3 (git 2.25.1) at 2024-07-05 15:34:03 +0300