mert/TODO


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

- check that mert-moses.pl emits devset score after every iteration
  - correctly for whichever metric we are optimizing
  - even when using --pairwise-ranked (PRO)
    - this may make use of 'evaluator', soon to be added by Matous Machacek

- check that --pairwise-ranked is compatible with all optimization metrics

- Replace the standard rand() currently used in MERT and PRO with better
  random generators such as Boost's random generators (e.g., boost::mt19937).
  - create a Random class to hide the details, i.e., how to generate
    random numbers, which allows us to use custom random generators more
    easily.

  Pros:
  - In MERT, you might want to use the random restarting technique to avoid
    local optima.
  - PRO uses a sampling technique to choose candidate translation pairs
    from N-best lists, which means the choice of random generators seems to
    be important.

  Cons:
  - This change will require us to re-create the truth results for regression
    testing related to MERT and PRO because the new random generator will
    generate different numbers from the current generator does.