diff options
| author | John Brooks <john.brooks@bluecherry.net> | 2012-01-04 10:27:35 +0400 |
|---|---|---|
| committer | Michael Niedermayer <michaelni@gmx.at> | 2012-01-05 05:23:02 +0400 |
| commit | fc9c5ad9ea70479d09a5d37c800665746ff4710d (patch) | |
| tree | 0a837994d088cb332d408715deff284ab6bbf852 | |
| parent | 68f121269694d61c137f046ea8f51ce717e57d7c (diff) | |
rv34: fix invalid memory access for small video dimensions
For small video dimensions calculations of the upper bound for pixel
access may result in negative value. Using an unsigned comparison
works only if the greater operand is non-negative. This is fixed by
doing edge emulation explicitly for such conditions.
Signed-off-by: Kostya Shishkov <kostya.shishkov@gmail.com>
(cherry picked from commit aacf6b3a2fd8bc8603e3deaa6e612ea03cf08707)
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
| -rw-r--r-- | libavcodec/rv34.c | 5 |
1 files changed, 3 insertions, 2 deletions
diff --git a/libavcodec/rv34.c b/libavcodec/rv34.c index 15a5c5430f..1e05906a6f 100644 --- a/libavcodec/rv34.c +++ b/libavcodec/rv34.c @@ -690,8 +690,9 @@ static inline void rv34_mc(RV34DecContext *r, const int block_type, srcY += src_y * s->linesize + src_x; srcU += uvsrc_y * s->uvlinesize + uvsrc_x; srcV += uvsrc_y * s->uvlinesize + uvsrc_x; - if( (unsigned)(src_x - !!lx*2) > s->h_edge_pos - !!lx*2 - (width <<3) - 4 - || (unsigned)(src_y - !!ly*2) > s->v_edge_pos - !!ly*2 - (height<<3) - 4){ + if(s->h_edge_pos - (width << 3) < 6 || s->v_edge_pos - (height << 3) < 6 || + (unsigned)(src_x - !!lx*2) > s->h_edge_pos - !!lx*2 - (width <<3) - 4 || + (unsigned)(src_y - !!ly*2) > s->v_edge_pos - !!ly*2 - (height<<3) - 4) { uint8_t *uvbuf = s->edge_emu_buffer + 22 * s->linesize; srcY -= 2 + 2*s->linesize; |