Overlay Engine: Refactor & Cleanup

This is the unification of all overlays into one overlay engine as described in T65347.

I went over all the code making it more future proof with less hacks and removing old / not relevent parts.

Goals / Acheivements:
- Remove internal shader usage (only drw shaders)
- Remove viewportSize and viewportSizeInv and put them in gloabl ubo
- Fixed some drawing issues: Missing probe option and Missing Alt+B clipping of some shader
- Remove old (legacy) shaders dependancy (not using view UBO).
- Less shader variation (less compilation time at first load and less patching needed for vulkan)
- removed some geom shaders when I could
- Remove static e_data (except shaders storage where it is OK)
- Clear the way to fix some anoying limitations (dithered transparency, background image compositing etc...)
- Wireframe drawing now uses the same batching capabilities as workbench & eevee (indirect drawing).
- Reduced complexity, removed ~3000 Lines of code in draw (also removed a lot of unused shader in GPU).
- Post AA to avoid complexity and cost of MSAA.

Remaining issues:
- ~~Armature edits, overlay toggles, (... others?) are not refreshing viewport after AA is complete~~
- FXAA is not the best for wires, maybe investigate SMAA
- Maybe do something more temporally stable for AA.
- ~~Paint overlays are not working with AA.~~
- ~~infront objects are difficult to select.~~
- ~~the infront wires sometimes goes through they solid counterpart (missing clear maybe?) (toggle overlays on-off when using infront+wireframe overlay in solid shading)~~

Note: I made some decision to change slightly the appearance of some objects to simplify their drawing. Namely the empty arrows end (which is now hollow/wire) and distance points of the cameras/spots being done by lines.

Reviewed By: jbakker

Differential Revision: https://developer.blender.org/D6296

1 files changed, 88 insertions, 0 deletions

diff --git a/source/blender/draw/engines/overlay/shaders/edit_mesh_geom.glsl b/source/blender/draw/engines/overlay/shaders/edit_mesh_geom.glsl
new file mode 100644
index 00000000000..92252bbd223
--- /dev/null
+++ b/source/blender/draw/engines/overlay/shaders/edit_mesh_geom.glsl
@@ -0,0 +1,88 @@
+
+layout(lines) in;
+layout(triangle_strip, max_vertices = 4) out;
+
+in vec4 finalColor[2];
+in vec4 finalColorOuter[2];
+in int selectOveride[2];
+
+flat out vec4 finalColorOuter_f;
+out vec4 finalColor_f;
+noperspective out float edgeCoord_f;
+
+void do_vertex(vec4 color, vec4 pos, float coord, vec2 offset)
+{
+  finalColor_f = color;
+  edgeCoord_f = coord;
+  gl_Position = pos;
+  /* Multiply offset by 2 because gl_Position range is [-1..1]. */
+  gl_Position.xy += offset * 2.0 * pos.w;
+  /* Correct but fails due to an AMD compiler bug, see: T62792.
+   * Do inline instead. */
+#if 0
+  world_clip_planes_set_clip_distance(gl_in[i].gl_ClipDistance);
+#endif
+  EmitVertex();
+}
+
+void main()
+{
+  vec2 ss_pos[2];
+
+  /* Clip line against near plane to avoid deformed lines. */
+  vec4 pos0 = gl_in[0].gl_Position;
+  vec4 pos1 = gl_in[1].gl_Position;
+  vec2 pz_ndc = vec2(pos0.z / pos0.w, pos1.z / pos1.w);
+  bvec2 clipped = lessThan(pz_ndc, vec2(-1.0));
+  if (all(clipped)) {
+    /* Totally clipped. */
+    return;
+  }
+
+  vec4 pos01 = pos0 - pos1;
+  float ofs = abs((pz_ndc.y + 1.0) / (pz_ndc.x - pz_ndc.y));
+  if (clipped.y) {
+    pos1 += pos01 * ofs;
+  }
+  else if (clipped.x) {
+    pos0 -= pos01 * (1.0 - ofs);
+  }
+
+  ss_pos[0] = pos0.xy / pos0.w;
+  ss_pos[1] = pos1.xy / pos1.w;
+
+  vec2 line = ss_pos[0] - ss_pos[1];
+  line = abs(line) * sizeViewport.xy;
+
+  finalColorOuter_f = finalColorOuter[0];
+  float half_size = sizeEdge;
+  /* Enlarge edge for flag display. */
+  half_size += (finalColorOuter_f.a > 0.0) ? max(sizeEdge, 1.0) : 0.0;
+
+#ifdef USE_SMOOTH_WIRE
+  /* Add 1 px for AA */
+  half_size += 0.5;
+#endif
+
+  vec3 edge_ofs = vec3(half_size * sizeViewportInv.xy, 0.0);
+
+  bool horizontal = line.x > line.y;
+  edge_ofs = (horizontal) ? edge_ofs.zyz : edge_ofs.xzz;
+
+#ifdef USE_WORLD_CLIP_PLANES
+  /* Due to an AMD glitch, this line was moved out of the `do_vertex`
+   * function (see T62792). */
+  world_clip_planes_set_clip_distance(gl_in[0].gl_ClipDistance);
+#endif
+  do_vertex(finalColor[0], pos0, half_size, edge_ofs.xy);
+  do_vertex(finalColor[0], pos0, -half_size, -edge_ofs.xy);
+
+#ifdef USE_WORLD_CLIP_PLANES
+  world_clip_planes_set_clip_distance(gl_in[1].gl_ClipDistance);
+#endif
+  vec4 final_color = (selectOveride[0] == 0) ? finalColor[1] : finalColor[0];
+  do_vertex(final_color, pos1, half_size, edge_ofs.xy);
+  do_vertex(final_color, pos1, -half_size, -edge_ofs.xy);
+
+  EndPrimitive();
+}