stepcompress: Fix rounding error when interval=0

In the rare case where the step generation code finds an interval of
zero, it was possible for an integer division of a negative number to
cause an incorrect result.  Fix that by using the idiv_up() and
idiv_down() helpers.

Signed-off-by: Kevin O'Connor <kevin@koconnor.net>
This commit is contained in:
Kevin O'Connor 2020-01-25 14:15:57 -05:00
parent 502e83725b
commit 5ca132eafa
1 changed files with 2 additions and 2 deletions

View File

@ -115,9 +115,9 @@ compress_bisect_add(struct stepcompress *sc)
int32_t nextaddfactor = nextcount*(nextcount-1)/2;
int32_t c = add*nextaddfactor;
if (nextmininterval*nextcount < nextpoint.minp - c)
nextmininterval = DIV_ROUND_UP(nextpoint.minp - c, nextcount);
nextmininterval = idiv_up(nextpoint.minp - c, nextcount);
if (nextmaxinterval*nextcount > nextpoint.maxp - c)
nextmaxinterval = (nextpoint.maxp - c) / nextcount;
nextmaxinterval = idiv_down(nextpoint.maxp - c, nextcount);
if (nextmininterval > nextmaxinterval)
break;
interval = nextmaxinterval;