asfdec_o: reject size > INT64_MAX in asf_read_unknown

Both avio_skip and detect_unknown_subobject use int64_t for the size
parameter.

This fixes a segmentation fault due to infinite recursion.

Reviewed-by: Alexandra Hájková <alexandra.khirnova@gmail.com>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>

1 files changed, 3 insertions, 0 deletions

diff --git a/libavformat/asfdec_o.c b/libavformat/asfdec_o.c
index 79b9ee43cc..02809bb435 100644
--- a/libavformat/asfdec_o.c
+++ b/libavformat/asfdec_o.c
@@ -178,6 +178,9 @@ static int asf_read_unknown(AVFormatContext *s, const GUIDParseTable *g)
     uint64_t size   = avio_rl64(pb);
     int ret;
 
+    if (size > INT64_MAX)
+        return AVERROR_INVALIDDATA;
+
     if (asf->is_header)
         asf->unknown_size = size;
     asf->is_header = 0;