stream: Fix unshift() race conditions

Fix #5272

The consumption of a readable stream is a dance with 3 partners.

1. The specific stream Author (A)
2. The Stream Base class (B), and
3. The Consumer of the stream (C)

When B calls the _read() method that A implements, it sets a 'reading'
flag, so that parallel calls to _read() can be avoided.  When A calls
stream.push(), B knows that it's safe to start calling _read() again.

If the consumer C is some kind of parser that wants in some cases to
pass the source stream off to some other party, but not before "putting
back" some bit of previously consumed data (as in the case of Node's
websocket http upgrade implementation).  So, stream.unshift() will
generally *never* be called by A, but *only* called by C.

Prior to this patch, stream.unshift() *also* unset the state.reading
flag, meaning that C could indicate the end of a read, and B would
dutifully fire off another _read() call to A.  This is inappropriate.
In the case of fs streams, and other variably-laggy streams that don't
tolerate overlapped _read() calls, this causes big problems.

Also, calling stream.shift() after the 'end' event did not raise any
kind of error, but would cause very strange behavior indeed.  Calling it
after the EOF chunk was seen, but before the 'end' event was fired would
also cause weird behavior, and could lead to data being lost, since it
would not emit another 'readable' event.

This change makes it so that:

1. stream.unshift() does *not* set state.reading = false
2. stream.unshift() is allowed up until the 'end' event.
3. unshifting onto a EOF-encountered and zero-length (but not yet
end-emitted) stream will defer the 'end' event until the new data is
consumed.
4. pushing onto a EOF-encountered stream is now an error.

So, if you read(), you have that single tick to safely unshift() data
back into the stream, even if the null chunk was pushed, and the length
was 0.

1 files changed, 15 insertions, 4 deletions

diff --git a/doc/api/stream.markdown b/doc/api/stream.markdown
index 2c9fc1c6c4c..ce7656b8d70 100644
--- a/doc/api/stream.markdown
+++ b/doc/api/stream.markdown
@@ -184,20 +184,31 @@ stream._read = function(n) {
 * `chunk` {Buffer | null | String} Chunk of data to unshift onto the read queue
 * return {Boolean} Whether or not more pushes should be performed
 
+Note: **This function should usually be called by Readable consumers,
+NOT by implementors of Readable subclasses.**  It does not indicate
+the end of a `_read()` transaction in the way that
+`readable.push(chunk)` does.  If you find that you have to call
+`this.unshift(chunk)` in your Readable class, then there's a good
+chance you ought to be using the
+[stream.Transform](#stream_class_stream_transform) class instead.
+
 This is the corollary of `readable.push(chunk)`.  Rather than putting
 the data at the *end* of the read queue, it puts it at the *front* of
 the read queue.
 
-This is useful in certain use-cases where a stream is being consumed
-by a parser, which needs to "un-consume" some data that it has
-optimistically pulled out of the source.
+This is useful in certain cases where a stream is being consumed by a
+parser, which needs to "un-consume" some data that it has
+optimistically pulled out of the source, so that the stream can be
+passed on to some other party.
 
 ```javascript
 // A parser for a simple data protocol.
 // The "header" is a JSON object, followed by 2 \n characters, and
 // then a message body.
 //
-// Note: This can be done more simply as a Transform stream.  See below.
+// NOTE: This can be done more simply as a Transform stream!
+// Using Readable directly for this is sub-optimal.  See the
+// alternative example below under the Transform section.
 
 function SimpleProtocol(source, options) {
   if (!(this instanceof SimpleProtocol))