diff options
| author | alvations <alvations@gmail.com> | 2019-11-25 05:10:40 +0300 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2019-11-25 05:10:40 +0300 |
| commit | f6d7adde154fb047b47192a70452ec7597a1da5c (patch) | |
| tree | 3a7a285840d4fa5cca816afaa38bfae3af4a7860 | |
| parent | 74d54b54c32dfebcbdc5f4447ce755e3e0232655 (diff) | |
Single quotes should be escaped as single quotes.alvations-patch-normalization
| -rwxr-xr-x | scripts/tokenizer/normalize-punctuation.perl | 4 |
1 files changed, 2 insertions, 2 deletions
diff --git a/scripts/tokenizer/normalize-punctuation.perl b/scripts/tokenizer/normalize-punctuation.perl index 7dab7543a..c56220322 100755 --- a/scripts/tokenizer/normalize-punctuation.perl +++ b/scripts/tokenizer/normalize-punctuation.perl @@ -42,8 +42,8 @@ while(<STDIN>) { s/´/\'/g; s/([a-z])‘([a-z])/$1\'$2/gi; s/([a-z])’([a-z])/$1\'$2/gi; - s/‘/\"/g; - s/‚/\"/g; + s/‘/\'/g; + s/‚/\'/g; s/’/\"/g; s/''/\"/g; s/´´/\"/g; |