avfilter/formats: Factor checking for mergeability out of ff_merge_*

The callers of the ff_merge_*() functions fall into two categories with
quite different needs:

One caller is can_merge_formats() which only wants to test for mergeability
without it merging anything. In order to do so, it duplicates the lists
it intends to test and resets their owners so that they are not modified
by ff_merge_*(). It also means that it needs to receive the merged list
(and not only an int containing whether the lists are mergeable) to
properly free it.

The other callers want the lists to be actually merged. But given the
fact that ff_merge_*() automatically updates the owners of the lists,
they only want the information whether the merge succeeded or not; they
don't want a link to the new list.

Therefore this commit splits these functions in two: ff_merge_*() for
the latter callers and ff_can_merge_*() for the former.
ff_merge_*() doesn't need to return a pointer to the combined list at all
and hence these functions have been modified to return an int, which
allows to distinguish between incompability and memory allocation failures.

ff_can_merge_*() meanwhile doesn't modify its arguments at all obviating
the need for copies. This in turn implies that there is no reason to
return a pointer to the new list, as nothing needs to be freed. These
functions therefore return an int as well. This allowed to completely
remove can_merge_formats() in avfiltergraph.c.

Notice that no ff_can_merge_channel_layouts() has been created, because
there is currently no caller for this. It could be added if needed.

Reviewed-by: Nicolas George <george@nsup.org>
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>

1 files changed, 24 insertions, 20 deletions

diff --git a/libavfilter/formats.h b/libavfilter/formats.h
index ffe7a12d53..dd0cbca6d5 100644
--- a/libavfilter/formats.h
+++ b/libavfilter/formats.h
@@ -110,17 +110,32 @@ typedef struct AVFilterChannelLayouts {
                            (int)((l) & 0x7FFFFFFF) : 0)
 
 /**
- * Return a channel layouts/samplerates list which contains the intersection of
- * the layouts/samplerates of a and b. Also, all the references of a, all the
- * references of b, and a and b themselves will be deallocated.
+ * Check the formats/samplerates lists for compatibility for merging
+ * without actually merging.
  *
- * If a and b do not share any common elements, neither is modified, and NULL
- * is returned.
+ * @return 1 if they are compatible, 0 if not.
  */
-AVFilterChannelLayouts *ff_merge_channel_layouts(AVFilterChannelLayouts *a,
-                                                 AVFilterChannelLayouts *b);
-AVFilterFormats *ff_merge_samplerates(AVFilterFormats *a,
-                                      AVFilterFormats *b);
+int ff_can_merge_formats(const AVFilterFormats *a, const AVFilterFormats *b,
+                         enum AVMediaType type);
+int ff_can_merge_samplerates(const AVFilterFormats *a, const AVFilterFormats *b);
+
+/**
+ * Merge the formats/channel layouts/samplerates lists if they are compatible
+ * and update all the references of a and b to point to the combined list and
+ * free the old lists as needed. The combined list usually contains the
+ * intersection of the lists of a and b.
+ *
+ * Both a and b must have owners (i.e. refcount > 0) for these functions.
+ *
+ * @return 1 if merging succeeded, 0 if a and b are incompatible
+ *         and negative AVERROR code on failure.
+ *         a and b are unmodified if 0 is returned.
+ */
+int ff_merge_channel_layouts(AVFilterChannelLayouts *a,
+                             AVFilterChannelLayouts *b);
+int ff_merge_formats(AVFilterFormats *a, AVFilterFormats *b,
+                     enum AVMediaType type);
+int ff_merge_samplerates(AVFilterFormats *a, AVFilterFormats *b);
 
 /**
  * Construct an empty AVFilterChannelLayouts/AVFilterFormats struct --
@@ -239,17 +254,6 @@ av_warn_unused_result
 AVFilterFormats *ff_planar_sample_fmts(void);
 
 /**
- * Return a format list which contains the intersection of the formats of
- * a and b. Also, all the references of a, all the references of b, and
- * a and b themselves will be deallocated.
- *
- * If a and b do not share any common formats, neither is modified, and NULL
- * is returned.
- */
-AVFilterFormats *ff_merge_formats(AVFilterFormats *a, AVFilterFormats *b,
-                                  enum AVMediaType type);
-
-/**
  * Add *ref as a new reference to formats.
  * That is the pointers will point like in the ascii art below:
  *   ________