results++

2 files changed, 72 insertions, 13 deletions

diff --git a/blog/_posts/2016-05-11-nce.md b/blog/_posts/2016-05-11-nce.md
index f1ff0bd..310c63f 100644
--- a/blog/_posts/2016-05-11-nce.md
+++ b/blog/_posts/2016-05-11-nce.md
@@ -3,9 +3,7 @@ layout: post
 title: Noise Contrastive Estimation
 comments: True
 author: nicholas-leonard
-excerpt: Noise contrastive estimation is used
-to train a multi-GPU recurrent neural network language model 
-on the Google billion words dataset.
+excerpt: Noise contrastive estimation is used to train a multi-GPU recurrent neural network language model on the Google billion words dataset.
 picture: https://raw.githubusercontent.com/torch/torch.github.io/master/blog/_posts/images/output_52iFki.gif
 ---
 
@@ -400,6 +398,7 @@ nn.Serial @ nn.Sequential {
 
 To use about one third less memory, you can set momentum of 0. 
 
+<a name='nce.eval'></a>
 ### Evaluation script
 
 The evaluation script can be used to measure perplexity on the test set or sample independent sentences.
@@ -577,23 +576,83 @@ Training runs at about 3800 words/second.
 
 ### Generating Sentences
 
-Here are 8 sentences sampled independently from the 4-layer LSTM with a `temperature` or 0.7:
+Here are some sentences sampled independently from the 4-layer LSTM with a `temperature` or 0.7:
 
 ```xml
-<S> The company said its net profit rose to $ 289 million , or 96 cents per share , in the three months ended on March 31 compared with $ 173 million , or $ 0.68 a share , a year ago . </S>
-<S> But I 've been a bit disappointed with our performance , " said Wenger . </S>
-<S> The first is an even bigger problem . </S>
-<S> The next big thing for him is he will be able to tell the world he is thinking about his future . </S>
-<S> The new rules have been added to the legislation so that they don 't have to be approved for public use . </S>
-<S> The Pentagon 's top counter-terrorism official , who has been in charge of a new system of intelligence collection and inspection , wrote in an e-mail message that while the new system could be easily implemented , it remains an option . </S>
-<S> " I was trying to get a glass of water . </S>
-<S> Later he was driven to a nearby house where he was later found to be severely ill . </S>
+<S> The first , for a lot of reasons , is the " Asian Glory " : an American military outpost in the middle of an Iranian desert . </S>
+<S> But the first new stage of the project will be a new <UNK> tunnel linking the new terminal with the new terminal at the airport . </S>
+<S> The White House said Bush would also sign a memorandum of understanding with Iraq , which will allow the Americans to take part in the poll . </S>
+<S> The folks who have campaigned for his nomination know that he is in a fight for survival . </S>
+<S> The three survivors , including a woman whose name was withheld and not authorized to speak , were buried Saturday in a makeshift cemetery in the town and seven people were killed in the town of Eldoret , which lies around a dozen miles ( 40 kilometers ) southwest of Kathmandu . </S>
+<S> The art of the garden was created by pouring water over a small brick wall and revealing that an older , more polished design was leading to the creation of a new house in the district . </S>
+<S> She added : " The club has not made any concession to the club 's fans and was not notified of the fact they had reached an agreement with the club . </S>
+<S> The Times has learnt that the former officer who fired the fatal shots must have known about the fatal carnage . </S>
+<S> Obama supporters say they 're worried about the impact of the healthcare and energy policies of Congress . </S>
+<S> Not to mention the painful changes to the way that women are treated in the workplace . </S>
+<S> The dollar stood at 14.38 yen ( <UNK> ) and <UNK> Swiss francs ( <UNK> ) . </S>
+<S> The current , the more intractable <UNK> , the <UNK> and the <UNK> about a lot of priorities . </S>
+<S> The job , which could possibly be completed in 2011 , needs to be approved in a new compact between the two companies . </S>
+<S> " The most important thing for me is to get back to the top , " he said . </S>
+<S> It was a one-year ban and the right to a penalty . </S>
+<S> The government of president Michelle Bachelet has promised to maintain a " strong and systematic " military presence in key areas and to tackle any issue of violence , including kidnappings . </S>
+<S> The six were scheduled to return to Washington on Wednesday . </S>
+<S> " It 's a ... mistake , " he said . </S>
+<S> The government 's offensive against the rebels and insurgents has been criticized by the United Nations and UN agencies . </S>
+<S> " Our <UNK> model is not much different from many of its competitors , " said Richard Bangs , CEO of the National Center for Science in the Public Interest in Chicago . </S>
+<S> He is now a large part of a group of young people who are spending less time studying and work in the city . </S>
+<S> He said he was confident that while he and his wife would have been comfortable working with him , he would be able to get them to do so . </S>
+<S> The summer 's financial meltdown is the worst in decades . </S>
+<S> It was a good night for Stuart Broad , who took the ball to Ravi Bopara at short leg to leave England on 88 for five at lunch . </S>
+<S> And even for those who worked for them , almost everything was at risk . </S>
+<S> The new strategy is all part of a stepped-up war against Taliban and al-Qaida militants in northwest Pakistan . </S>
+<S> The governor 's office says the proposal is based on a vision of an outsider in the town who wants to preserve the state 's image . </S>
+<S> " The fact that there is no evidence to support the claim made by the government is entirely convincing and that Dr Mohamed will have to be detained for a further two years , " he said . </S>
+<S> The country 's tiny nuclear power plants were the first to use nuclear technology , and the first such reactors in the world . </S>
+<S> " What is also important about this is that we can go back to the way we worked and work and fight , " he says . </S>
+<S> And while he has been the star of " The Wire " and " The Office , " Mr. Murphy has been a careful , intelligent , engaging competitor for years . </S>
+<S> On our return to the water , we found a large abandoned house . </S>
+<S> The national average for a gallon of regular gas was $ 5.99 for the week ending Jan . </S>
+<S> The vote was a rare early start for the contest , which was held after a partial recount in 26 percent of the vote . </S>
+<S> The first one was a show of force by a few , but the second was an attempt to show that the country was serious about peace . </S>
+<S> It was a little more than half an hour after the first reports of a shooting . </S>
+<S> The central bank is expected to cut interest rates further by purchasing more than $ 100 billion of commercial paper and Treasuries this week . </S>
+<S> Easy , it 's said , to have a child with autism . </S>
+<S> He said : " I am very disappointed with the outcome because the board has not committed itself . </S>
+<S> " There is a great deal of tension between us , " said Mr C. </S>
+<S> The odds that the Fed will keep its benchmark interest rate unchanged are at least half as much as they were at the end of 2008 . </S>
+<S> For them , investors have come to see that : a ) the government will maintain a stake in banks and ( 2 ) the threat of financial regulation and supervision ; and ( 3 ) it will not be able to raise enough capital from the private sector to support the economy . </S>
+<S> The court heard he had been drinking and drank alcohol at the time of the attack . </S>
+<S> " The whole thing is quite a bit more intense . </S>
+<S> This is a very important project and one that we are working closely with . </S>
+<S> " We are confident that in this economy and in the current economy , we will continue to grow , " said John Lipsky , who chaired the IMF 's board of governors for several weeks . </S>
+<S> The researchers said they found no differences among how men drank and whether they were obese . </S>
+<S> Even though there are many brands that have low voice and no connection to the Internet , the iPhone is a great deal for consumers . </S>
+<S> The £ 7m project is a new project for the city of Milton Keynes and aims to launch a new challenge for the British Government . </S>
+<S> But he was not without sympathy for his father . </S>
 ``` 
 
+The syntax seems quite reasonable, especially when comparing it to the previous results obtained from the [single-GPU 2x250 LSTM](#nce.eval). 
+However, in some cases, the semantics, i.e. the meaning of the words, is not so good. 
+For example, the sentence `<S> Easy , it 's said , to have a child with autism . </S>` would make more sense, to me at least, by replacing `Easy` with `Not easy`.
+But then again, this sentence as nice semantics: `<S> The government of president Michelle Bachelet has promised to maintain a " strong and systematic " military presence in key areas and to tackle any issue of violence , including kidnappings . </S>`.
+[Michelle Bachelet](https://en.wikipedia.org/wiki/Michelle_Bachelet) was actually a president of Chile.
+In her earlier life, she was also [kidnapped by military men](https://www.theguardian.com/world/2005/nov/22/chile.gender), so it kind of makes sense that she would be strong on the issue of kidnappings.
+Here is an example of some weird semantics : `<S> Even though there are many brands that have low voice and no connection to the Internet , the iPhone is a great deal for consumers . </S>`
+The first part about `load voice` doesn't mean anything to me. 
+And I fail to see how there being `many brands that have no connection to the Internet` relates to `the iPhone is a great deal for consumers`.
+But of course, all these sentences are generated independently, so the LM needs to learn to generate a meaning on the fly.
+This is hard as there is no context to the sentence.
 
 ### Learning Curves
 
+The following figure outlines the learning curves for the above model. 
+The figure plots the NCE training and validation error for the model, which is the error output but the `NCEModule`.
+Test set error isn't plotted as doing so for any epoch requires about 3 hours because test set inference uses `Linear` + `SoftMax` with `batchsize=1`.
 
+![LSTM NCE Learning curves](images/LSTM-NCE-curve.png)
+
+As you can see, most of the learning is done in the first epochs. 
+Nevertheless, the training and validation error are consistently reduced training progresses.
 
 <a name='nce.ref'></a>
 ## References
@@ -605,4 +664,4 @@ Here are 8 sentences sampled independently from the 4-layer LSTM with a `tempera
 5. *A Graves, A Mohamed, G Hinton*, [Speech Recognition with Deep Recurrent Neural Networks](http://arxiv.org/pdf/1303.5778.pdf)
 6. *K Greff, RK Srivastava, J Koutník*, [LSTM: A Search Space Odyssey](http://arxiv.org/pdf/1503.04069)
 7. *C Chelba, T Mikolov, M Schuster, Q Ge, T Brants, P Koehn, T Robinson*, [One billion word benchmark for measuring progress in statistical language modeling](http://arxiv.org/pdf/1312.3005)
-8. *A Graves*, [Generating Sequences With Recurrent Neural Networks](http://arxiv.org/pdf/1308.0850v5.pdf)
+8. *A Graves*, [Generating Sequences With Recurrent Neural Networks, table 1](http://arxiv.org/pdf/1308.0850v5.pdf)
diff --git a/blog/_posts/images/LSTM-NCE-curve.png b/blog/_posts/images/LSTM-NCE-curve.png
new file mode 100644
index 0000000..447cc56
--- /dev/null
+++ b/blog/_posts/images/LSTM-NCE-curve.png