diff options
| author | Edward Hervey <edward@centricular.com> | 2017-11-24 14:05:26 +0300 |
|---|---|---|
| committer | Edward Hervey <bilboed@bilboed.com> | 2017-12-01 11:48:39 +0300 |
| commit | 23ee27987d8b1ca623b8daa9040ef4e5396f80c4 (patch) | |
| tree | 4f0a4cde2574c35fccda985ef156b45cef3318a6 | |
| parent | e33d740474b0ce21a17faa04d1b5d347ac9f789c (diff) | |
gstutils: Fix linear regression comparision
The check for dropping precision was wrong when sxx and syy were negative.
if they are negative then "G_MAXINT64 - val" would always overflow
The check was meant to use G_MININT64 (like in the loop contained just
after).
| -rw-r--r-- | gst/gstutils.c | 4 |
1 files changed, 2 insertions, 2 deletions
diff --git a/gst/gstutils.c b/gst/gstutils.c index 79193ccff3..909eab6c4b 100644 --- a/gst/gstutils.c +++ b/gst/gstutils.c @@ -4304,7 +4304,7 @@ gst_calculate_linear_regression (const GstClockTime * xy, tmp /= 4; } while (G_MAXINT64 - sxx <= tmp); break; - } else if (G_UNLIKELY (tmp < 0 && sxx < 0 && (G_MAXINT64 - sxx >= tmp))) { + } else if (G_UNLIKELY (tmp < 0 && sxx < 0 && (G_MININT64 - sxx >= tmp))) { do { /* Drop some precision and restart */ pshift++; @@ -4323,7 +4323,7 @@ gst_calculate_linear_regression (const GstClockTime * xy, tmp /= 4; } while (G_MAXINT64 - syy <= tmp); break; - } else if (G_UNLIKELY (tmp < 0 && syy < 0 && (G_MAXINT64 - syy >= tmp))) { + } else if (G_UNLIKELY (tmp < 0 && syy < 0 && (G_MININT64 - syy >= tmp))) { do { pshift++; syy /= 4; |