| Age | Commit message (Collapse) | Author |
|---|
|
|
|
Don't load all references, read them line by line.
Corpora with millions of sentences can now be evaluated without consuming gigabytes of RAM.
|
|
|
|
Mostly signed/unsigned comparisons and reordered member
initializations; also a few unused variables.
There are more, but if I chip away at them for a while, who knows, it
may catch on and warnings may eventually become socially stigmatizing.
:)
|
|
metrics need getReferenceLength (for background smoothing) to work with kbmira
|
|
|
|
|
|
|
|
|
|
As in Nakov et al (Coling 2012)
|
|
Fixes inefficient updating N-gram counts.
NOTE: Using '--binary' option (this option is not enabled by default yet)
for saving outputs would lead to significant speed up.
|
|
For storing the word vocabulary used in computation of
BLEU scores. This change will reduce the running time
of extractor about 2-3 seconds (9% reduction).
|
|
By replacing std::map with boost::unordered_map.
Runtime of extractor on 100-best lists of 2679 sentences:
Before:
real 0m35.314s
user 0m34.030s
sys 0m1.280s
Ater:
real 0m26.729s
user 0m25.420s
sys 0m1.310s
|
|
Using directives should be put into the implementation files.
|
|
Compiles, not tested.
Conflicts:
Jamroot
OnDiskPt/PhraseNode.h
OnDiskPt/TargetPhrase.cpp
OnDiskPt/TargetPhrase.h
OnDiskPt/TargetPhraseCollection.cpp
mert/BleuScorer.cpp
mert/Data.cpp
mert/FeatureData.cpp
moses-chart-cmd/src/Main.cpp
moses/src/AlignmentInfo.h
moses/src/ChartManager.cpp
moses/src/LM/Ken.cpp
moses/src/LM/Ken.h
moses/src/LMList.h
moses/src/LexicalReordering.h
moses/src/PhraseDictionaryTree.h
moses/src/ScoreIndexManager.h
moses/src/StaticData.h
moses/src/TargetPhrase.h
moses/src/Word.cpp
scripts/ems/experiment.meta
scripts/ems/experiment.perl
scripts/training/train-model.perl
|
|
|
|
must double check with barry or lexi
|
|
|
|
and fear translations, add variations to 1slack formulation
|
|
Regression tests all check out, and kbmira seems to work fine
on a Hansard French->English task.
HypPackEnumerator class may be of interest to pro.cpp and future
optimizers, as it abstracts a lot of the boilerplate involved in
enumerating multiple k-best lists.
MiraWeightVector is not really mira-specific - just a weight vector
that enables efficient averaging. Could be useful to a perceptron
as well. Same goes for MiraFeatureVector.
Interaction with sparse features is written, but untested.
|
|
|
|
|
|
evaluator
|
|
|
|
|
|
- Move a definition of sentenceLevelBleuPlusOne() from pro.cpp
to BleuScorer.cpp.
- Add check for the length of an input vector.
|
|
- Make BleuScorer::setReferenceFiles() more testable by
adding OpenReference() and OpenReferenceStream().
|
|
- Add Vocabulary factory and the unit test.
- Remove Scorer::ClearVocabulary().
|
|
- Introduce the namespace to avoid naming collisions. The class name
is used in KenLM.
- Add the unit test.
|
|
To add unit tests.
|
|
This commit also includes
- Fix typo.
- Fix indentations.
- Add 'const' to Scorer::applyFactors().
|
|
- Add an unit test for Reference.
- Move functions to calculate the reference length from
BleuScorer to Reference.
|
|
|
|
|
|
The reason is that we want to add the unit test.
|
|
To avoid defining the similar variables twice to calculate
document-wise BLEU and sentence-wise BLEU scores.
|
|
example:
Use --factor "0|2" to use only first and third factor from nbest list and from reference.
If you use interpolated scorer, separate records with comma (e.g. --factor "0|2,1").
|
|
|
|
Consider using constants the result if it is possible.
|
|
Use std::ostream instead of directly using std::cerr.
|
|
|
|
- Remove unnecessary conversions.
- Add 'const' to local variables.
|
|
|
|
|
|
|
|
Instead of using typedefs inside a class only,
it might be better to create a private class to do same things.
|
|
|
|
The reason is that the type is used as internal purpose.
|
|
|
|
|
