diff options
| author | Taku Kudo <taku910@users.noreply.github.com> | 2020-10-17 06:28:36 +0300 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2020-10-17 06:28:36 +0300 |
| commit | 496f22507529d6c4e2935a5967fd4fb4e53ebd47 (patch) | |
| tree | 75ed571eae158d56238d02f7ba5e9595f7271508 | |
| parent | 0b5bd12205364c064a93c77d425ac7e6f8e41df3 (diff) | |
| parent | d796421cbaa4b8f57f8005bb3c1a1ad4173d68d8 (diff) | |
Merge pull request #556 from equivalence1/fix_readme_sil_symbol
Fix space symbol in code snippet: _ -> ▁
| -rw-r--r-- | README.md | 2 |
1 files changed, 1 insertions, 1 deletions
@@ -84,7 +84,7 @@ Then, this text is segmented into small pieces, for example: Since the whitespace is preserved in the segmented text, we can detokenize the text without any ambiguities. ``` - detokenized = ''.join(pieces).replace('_', ' ') + detokenized = ''.join(pieces).replace('▁', ' ') ``` This feature makes it possible to perform detokenization without relying on language-specific resources. |