Fix T96518: Gpencil Fill freezes when use invert and click inside areas

When using inverted filling and click inside a closed area and not outside as is expected, the algorithm to detect the contour to fill is unable to find the filling shape and try to fill outside of the valid index.

The infinite loop was adding more memory for each loop and the process continued while there was system resources and finally crashed the system.

As the tool in negative mode is designed to fill all areas when you click outside of any shape, now the algorithm check if the outline is not working as expected and cancels the filling process.

1 files changed, 9 insertions, 0 deletions

diff --git a/source/blender/editors/gpencil/gpencil_fill.c b/source/blender/editors/gpencil/gpencil_fill.c
index 069493025dc..45a2247c65e 100644
--- a/source/blender/editors/gpencil/gpencil_fill.c
+++ b/source/blender/editors/gpencil/gpencil_fill.c
@@ -1387,6 +1387,15 @@ static void gpencil_get_outline_points(tGPDfill *tgpf, const bool dilate)
       current_check_co[1] = boundary_co[1] + offset[offset_idx][1];
 
       int image_idx = ibuf->x * current_check_co[1] + current_check_co[0];
+      /* Check if the index is inside the image. If the index is outside is
+       * because the algorithm is unable to find the outline of the figure. This is
+       * possible for negative filling when click inside a figure instead of
+       * clicking outside.
+       * If the index is out of range, finish the filling. */
+      if (image_idx > imagesize - 1) {
+        start_found = false;
+        break;
+      }
       get_pixel(ibuf, image_idx, rgba);
 
       /* find next boundary pixel */