diff options
| author | Dmitrii Kravchenko <equivalence1@gmail.com> | 2020-10-15 14:51:35 +0300 |
|---|---|---|
| committer | Dmitrii Kravchenko <equivalence1@gmail.com> | 2020-10-15 14:51:35 +0300 |
| commit | d796421cbaa4b8f57f8005bb3c1a1ad4173d68d8 (patch) | |
| tree | 75ed571eae158d56238d02f7ba5e9595f7271508 | |
| parent | 0b5bd12205364c064a93c77d425ac7e6f8e41df3 (diff) | |
Fix SIL symbol in code snippet: _ -> _
| -rw-r--r-- | README.md | 2 |
1 files changed, 1 insertions, 1 deletions
@@ -84,7 +84,7 @@ Then, this text is segmented into small pieces, for example: Since the whitespace is preserved in the segmented text, we can detokenize the text without any ambiguities. ``` - detokenized = ''.join(pieces).replace('_', ' ') + detokenized = ''.join(pieces).replace('▁', ' ') ``` This feature makes it possible to perform detokenization without relying on language-specific resources. |