diff options
| author | Marcin Junczys-Dowmunt <marcinjd@microsoft.com> | 2020-11-10 18:31:21 +0300 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2020-11-10 18:31:21 +0300 |
| commit | 38824c04b5717a889a8e8e9bdd023d82ca1303de (patch) | |
| tree | d6a9822c3177833768b976c9491d4306e4f871a9 /tests/sentencepiece/test_create_spm_vocabs.sh | |
| parent | a30d3ffbebbe5e5587e6eabbdc6b7963c70bc87d (diff) | |
| parent | 8acd64aa1e579339db629e605ab46759c4268d50 (diff) | |
Merge pull request #68 from marian-nmt/pr-mjd-updatespm
Update tests with SentencePiece
Diffstat (limited to 'tests/sentencepiece/test_create_spm_vocabs.sh')
| -rw-r--r-- | tests/sentencepiece/test_create_spm_vocabs.sh | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/tests/sentencepiece/test_create_spm_vocabs.sh b/tests/sentencepiece/test_create_spm_vocabs.sh index 2781964..61e2c04 100644 --- a/tests/sentencepiece/test_create_spm_vocabs.sh +++ b/tests/sentencepiece/test_create_spm_vocabs.sh @@ -35,7 +35,7 @@ grep -q "Loading SentencePiece vocabulary .*vocab.en.spm" vocabs.log grep -q "Loading SentencePiece vocabulary .*vocab.de.spm" vocabs.log # Extract a textual vocabulary and compare with the expected output -LC_ALL=C $MRT_MARIAN/spm_export_vocab --model vocabs/vocab.en.spm | head -n 3980 | sort > vocabs.en.out +LC_ALL=C $MRT_MARIAN/spm_export_vocab --model vocabs/vocab.en.spm | sort > vocabs.en.out LC_ALL=C $MRT_MARIAN/spm_export_vocab --model vocabs/vocab.de.spm | sort > vocabs.de.out $MRT_TOOLS/diff-nums.py vocabs.en.out vocabs.en.expected -o vocabs.en.diff |