Fix #36226, Select Linked works not in touch with Prefs.

When setting keymap properties to values equalling the RNA default, they will
get "unset" and automatic operator behavior is used. There is no way to
explicitly set the default value as a user.

1) To allow distinguishing uninitialized (not set) properties in the keymap
items, a few changes to the RNA struct comparison function are needed: Instead
of allowing only strict/non-strict comparison of 2 properties A and B in a
struct, this now has 3 modes:
* STRICT: compare only the actual property values (same as 'strict' before)
* UNSET_MATCH_ANY: if either A or B is unset, consider them a match (same as
non-strict before)
* UNSET_MATCH_NONE: if one property is set and the other not, consider them a
mismatch.

The new UNSET_MATCH_NONE mode is useful for keymaps, because it allows keeping
user-defined property values in the keymap even if they match the default
property value (see wm_keymap_diff function in wm_keymap.c)

2) A new operator is added for unsetting ID properties in the RMB context menu
and in user preferences next to keymap properties. This only works on ID
properties and deletes the ID property storage, so that the default value is used.

In the user preferences for keymaps the properties are shown in an inactive
layout to indicate that the default value is used (which some operators such as
the "select linked" op from the report use to trigger automatic behavior). When
the user sets a property it gets set and stays that way until explicitly "unset"
using the new operator.

1 files changed, 8 insertions, 2 deletions

diff --git a/source/blender/makesrna/RNA_access.h b/source/blender/makesrna/RNA_access.h
index d18fe7629fd..b4b79059f93 100644
--- a/source/blender/makesrna/RNA_access.h
+++ b/source/blender/makesrna/RNA_access.h
@@ -1119,8 +1119,14 @@ void _RNA_warning(const char *format, ...) ATTR_PRINTF_FORMAT(1, 2);
 /* Equals test (skips pointers and collections)
  * is_strict false assumes uninitialized properties are equal */
 
-bool RNA_property_equals(struct PointerRNA *a, struct PointerRNA *b, struct PropertyRNA *prop, bool is_strict);
-bool RNA_struct_equals(struct PointerRNA *a, struct PointerRNA *b, bool is_strict);
+typedef enum eRNAEqualsMode {
+	RNA_EQ_STRICT,          /* set/unset ignored */
+	RNA_EQ_UNSET_MATCH_ANY, /* unset property matches anything */
+	RNA_EQ_UNSET_MATCH_NONE /* unset property never matches set property */
+} eRNAEqualsMode;
+
+bool RNA_property_equals(struct PointerRNA *a, struct PointerRNA *b, struct PropertyRNA *prop, eRNAEqualsMode mode);
+bool RNA_struct_equals(struct PointerRNA *a, struct PointerRNA *b, eRNAEqualsMode mode);
 
 #ifdef __cplusplus
 }