Welcome to mirror list, hosted at ThFree Co, Russian Federation.

github.com/moses-smt/mosesdecoder.git - Unnamed repository; edit this file 'description' to name the repository.
summaryrefslogtreecommitdiff
path: root/scripts/tokenizer
AgeCommit message (Collapse)Author
2012-05-30more escaping in tokenizer; wrapper for berkeley parser (german)phikoehn
2012-05-26minor fixesphikoehn
2012-04-12minor fixesphikoehn
2012-03-23scripts to escape moses specific charactersphikoehn
2012-03-23bug fix to filter hierarchicalphikoehn
2012-03-20lotsa minor changes: mostly bug fixes, tokenizer now esacapes special Moses ↵phikoehn
characters (|<>&)
2012-01-18Option to disable buffering (from Tom Hoar)Barry Haddow
2011-11-30reminder of language codesHieu Hoang
2011-08-08when detokenizing, remove whitespace between a pair of CJK ↵bgottesman
(Chinese/Japanese/Korean) words This gets the Chinese and Japanese tests working, so remove the failure expectation. git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4134 1f5c12ca-751b-0410-a591-d2e778427230
2011-08-08fix a detokenization bug that was preventing the removal of the whitespace ↵bgottesman
following a contracted French or Italian article/pronoun (e.g. "l' immigration") when the contraction was the second-last word in the segment remove the expectation of failure on the corresponding unit test git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4133 1f5c12ca-751b-0410-a591-d2e778427230
2011-06-16Add a space before double punctuation signs in Frenchrafpayen
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4016 1f5c12ca-751b-0410-a591-d2e778427230
2011-02-16added files from Tom Hoarphkoehn
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3881 1f5c12ca-751b-0410-a591-d2e778427230
2011-02-14mark web/bin/detokenizer.perl as outdatedbojar
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3880 1f5c12ca-751b-0410-a591-d2e778427230
2011-02-14Czech detokenizationbojar
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3879 1f5c12ca-751b-0410-a591-d2e778427230
2010-12-08Non-breaking prefixes for Dutchbhaddow
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3764 1f5c12ca-751b-0410-a591-d2e778427230
2010-10-19Added nonbreaking_prefix.sv for Swedishsarst
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3630 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-15new prefix files for tokenizerphkoehn
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3467 1f5c12ca-751b-0410-a591-d2e778427230
2010-08-02add lowercaserhieuhoang1972
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3380 1f5c12ca-751b-0410-a591-d2e778427230
2010-05-05official release of experiment.perlphkoehn
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3224 1f5c12ca-751b-0410-a591-d2e778427230
    generated by cgit v1.2.3 (git 2.25.1) at 2024-09-05 23:21:34 +0300