diff options
| author | Joel Barry <joelb@isi.edu> | 2019-02-27 18:17:29 +0300 |
|---|---|---|
| committer | Joel Barry <joelb@isi.edu> | 2019-02-27 18:17:29 +0300 |
| commit | fdb7384d3d401cc9d6ece34a038e1b585712c512 (patch) | |
| tree | 8ae1f1cef67b1fd16873b9962c987748c1ef8adf | |
| parent | 49b388ac79107d96b70255d59ce556e37d6b89ad (diff) | |
Fix non-ASCII lowercasing
| -rwxr-xr-x | scripts/generic/multi-bleu-detok.perl | 3 |
1 files changed, 3 insertions, 0 deletions
diff --git a/scripts/generic/multi-bleu-detok.perl b/scripts/generic/multi-bleu-detok.perl index d2ef60c90..e1f69501e 100755 --- a/scripts/generic/multi-bleu-detok.perl +++ b/scripts/generic/multi-bleu-detok.perl @@ -14,6 +14,9 @@ use warnings; use strict; +binmode(STDIN, ":utf8"); +use open ':encoding(UTF-8)'; + my $lowercase = 0; if ($ARGV[0] eq "-lc") { $lowercase = 1; |