Welcome to
mirror list
, hosted at
ThFree Co
, Russian Federation.
/scripts/tokenizer/
/scripts/tokenizer/
../
basic-protected-patterns
deescape-special-chars-PTB.perl
deescape-special-chars.perl
delete-long-words.perl
detokenizer.perl
escape-special-chars.perl
lowercase.perl
normalize-punctuation.perl
pre-tok-clean.perl
pre-tokenizer.perl
pre_tokenize_cleaning.py
remove-non-printing-char.perl
replace-unicode-punctuation.perl
tokenizer.perl
tokenizer_PTB.perl