Mode | Name | Size | |
---|---|---|---|
-rw-r--r-- | basic-protected-patterns | 178 | logplain |
-rwxr-xr-x | deescape-special-chars-PTB.perl | 662 | logplain |
-rwxr-xr-x | deescape-special-chars.perl | 631 | logplain |
-rwxr-xr-x | delete-long-words.perl | 211 | logplain |
-rwxr-xr-x | detokenizer.perl | 12473 | logplain |
-rwxr-xr-x | escape-special-chars.perl | 749 | logplain |
-rwxr-xr-x | lowercase.perl | 285 | logplain |
-rwxr-xr-x | normalize-punctuation.perl | 1912 | logplain |
-rwxr-xr-x | pre-tok-clean.perl | 1428 | logplain |
-rwxr-xr-x | pre-tokenizer.perl | 967 | logplain |
-rw-r--r-- | pre_tokenize_cleaning.py | 2964 | logplain |
-rwxr-xr-x | remove-non-printing-char.perl | 451 | logplain |
-rwxr-xr-x | replace-unicode-punctuation.perl | 774 | logplain |
-rwxr-xr-x | tokenizer.perl | 16820 | logplain |
-rwxr-xr-x | tokenizer_PTB.perl | 12321 | logplain |