From f531193690d91399dd99ae1bf61af311d9734f32 Mon Sep 17 00:00:00 2001 From: Matthew Einhorn Date: Wed, 24 Aug 2011 20:14:03 -0400 Subject: Fixes avpicture_layout to not write past buffer end. avpicture_get_size() returns the size of buffer required for avpicture_layout. For pseudo-paletted formats (gray8...) this size does not include the palette. However, avpicture_layout doesn't know this and still writes the palette. Consequently, avpicture_layout writes passed the length of the buffer. This fixes it by fixing avpicture_layout so that it doesn't write the palette for these formats. Signed-off-by: Matthew Einhorn Signed-off-by: Michael Niedermayer (cherry picked from commit e662b263d9c500270a8f1dc7e1b81b51d5bdfd4e) Signed-off-by: Michael Niedermayer --- libavcodec/imgconvert.c | 10 ++++++++++ 1 file changed, 10 insertions(+) (limited to 'libavcodec') diff --git a/libavcodec/imgconvert.c b/libavcodec/imgconvert.c index 9aa584fa5c..04c58ca278 100644 --- a/libavcodec/imgconvert.c +++ b/libavcodec/imgconvert.c @@ -470,6 +470,16 @@ int avpicture_layout(const AVPicture* src, enum PixelFormat pix_fmt, int width, } } + switch (pix_fmt) { + case PIX_FMT_RGB8: + case PIX_FMT_BGR8: + case PIX_FMT_RGB4_BYTE: + case PIX_FMT_BGR4_BYTE: + case PIX_FMT_GRAY8: + // do not include palette for these pseudo-paletted formats + return size; + } + if (desc->flags & PIX_FMT_PAL) memcpy((unsigned char *)(((size_t)dest + 3) & ~3), src->data[1], 256 * 4); -- cgit v1.2.3