avcodec/hevc_mp4toannexb_bsf: Check NAL size against available input

The hevc_mp4toannexb bsf does not explicitly check whether a NAL unit
is so big that it extends beyond the end of the input packet; it does so
only implicitly by using the checked version of the bytestream2 API.
But this has downsides compared to real checks: It can lead to huge
allocations (up to 2GiB) even when the input packet is just a few bytes.
And furthermore it leads to uninitialized data being output.
So add a check to error out early if it happens.

Also check directly whether there is enough data for the length field.

Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
(cherry picked from commit ea1b71e82f5a1752d59d3bfb9704092a79eba6b5)
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>

1 files changed, 5 insertions, 1 deletions

diff --git a/libavcodec/hevc_mp4toannexb_bsf.c b/libavcodec/hevc_mp4toannexb_bsf.c
index 30f733d775..477d86d9fd 100644
--- a/libavcodec/hevc_mp4toannexb_bsf.c
+++ b/libavcodec/hevc_mp4toannexb_bsf.c
@@ -141,10 +141,14 @@ static int hevc_mp4toannexb_filter(AVBSFContext *ctx, AVPacket *out)
         int      nalu_type;
         int is_irap, add_extradata, extra_size, prev_size;
 
+        if (bytestream2_get_bytes_left(&gb) < s->length_size) {
+            ret = AVERROR_INVALIDDATA;
+            goto fail;
+        }
         for (i = 0; i < s->length_size; i++)
             nalu_size = (nalu_size << 8) | bytestream2_get_byte(&gb);
 
-        if (nalu_size < 2) {
+        if (nalu_size < 2 || nalu_size > bytestream2_get_bytes_left(&gb)) {
             ret = AVERROR_INVALIDDATA;
             goto fail;
         }