sentence-bleu less greedy regarding memory

Don't load all references, read them line by line.
Corpora with millions of sentences can now be evaluated without consuming gigabytes of RAM.

1 files changed, 5 insertions, 0 deletions

diff --git a/mert/Reference.h b/mert/Reference.h
index 2c12f2ed7..a7878f3e7 100644
--- a/mert/Reference.h
+++ b/mert/Reference.h
@@ -59,6 +59,11 @@ public:
   int CalcClosest(std::size_t length) const;
   int CalcShortest() const;
 
+  void clear() {
+    m_length.clear();
+    m_counts->clear();
+  }
+
 private:
   NgramCounts* m_counts;