diff options
| author | Andreas Rheinhardt <andreas.rheinhardt@outlook.com> | 2022-04-15 17:05:20 +0300 |
|---|---|---|
| committer | Andreas Rheinhardt <andreas.rheinhardt@outlook.com> | 2022-04-28 02:28:03 +0300 |
| commit | e4a26a64a875d4376cd78f77905a58c4f0238d85 (patch) | |
| tree | be7dfaaba486ad7e1ff9802308ab5a2f6e5cad36 /libavcodec/pgxdec.c | |
| parent | 67f796050519a3a3f1d52c6b054d3c80b8980cd2 (diff) | |
avcodec/pgxdec: Fix issue with negative linesizes
The PGX decoder accesses the lines via code like
(PIXEL*)frame->data[0] + i*frame->linesize[0]/sizeof(PIXEL)
where PIXEL is a macro parameter. This code has issues with negative
linesizes, because the type of sizeof(PIXEL) is size_t, so
that on common systems i*linesize/sizeof(PIXEL) will
always be an unsigned type that is very large in case linesize is
negative. This happens to work*, but it is undefined behaviour
and e.g. leads to "src/libavcodec/pgxdec.c:114:1: runtime error:
addition of unsigned offset to 0x7efe9c2b7040 overflowed to 0x7efe9c2b6040"
errors from UBSAN.
Fix this by using (PIXEL*)(frame->data[0] + i*frame->linesize[0]).
This is allowed because linesize has to be suitably aligned.
*: Converting a negative int to size_t works by adding SIZE_MAX + 1
to the number, so that the result is off by (SIZE_MAX + 1) /
sizeof(PIXEL). Converting the pointer arithmetic (performed on PIXELs)
back to ordinary pointers is tantamount to multiplying by sizeof(PIXEL),
so that the result is off by SIZE_MAX + 1; but SIZE_MAX + 1 == 0
for the underlying pointers.
Reviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
Diffstat (limited to 'libavcodec/pgxdec.c')
| -rw-r--r-- | libavcodec/pgxdec.c | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/libavcodec/pgxdec.c b/libavcodec/pgxdec.c index c9ada5afb5..30895b51ee 100644 --- a/libavcodec/pgxdec.c +++ b/libavcodec/pgxdec.c @@ -97,7 +97,7 @@ error: { \ int i, j; \ for (i = 0; i < height; i++) { \ - PIXEL *line = (PIXEL*)frame->data[0] + i*frame->linesize[0]/sizeof(PIXEL); \ + PIXEL *line = (PIXEL*)(frame->data[0] + i * frame->linesize[0]); \ for (j = 0; j < width; j++) { \ unsigned val; \ if (sign) \ |