Fix GH-16257 imagescale underflow on RGB channels.

backport of https://github.com/libgd/libgd/commit/948bb0a5c2010a24227e4b44a90e8b8aa9bda8ce

close GH-16257