From 2c72ed740f302bf51e6cd0824b8e7cbde1b1e8c2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Date: Fri, 9 Feb 2018 20:32:11 +0000 Subject: git fetch doc: add a new section to explain the ins & outs of pruning MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a new section to canonically explain how remote reference pruning works, and how users should be careful about using it in conjunction with tag refspecs in particular. A subsequent commit will update the git-remote documentation to refer to this section, and details the motivation for writing this in the first place. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano --- Documentation/git-fetch.txt | 49 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) (limited to 'Documentation') diff --git a/Documentation/git-fetch.txt b/Documentation/git-fetch.txt index b153aefa68..e94bcfb8c3 100644 --- a/Documentation/git-fetch.txt +++ b/Documentation/git-fetch.txt @@ -99,6 +99,55 @@ The latter use of the `remote..fetch` values can be overridden by giving the `--refmap=` parameter(s) on the command line. +PRUNING +------- + +Git has a default disposition of keeping data unless it's explicitly +thrown away; this extends to holding onto local references to branches +on remotes that have themselves deleted those branches. + +If left to accumulate, these stale references might make performance +worse on big and busy repos that have a lot of branch churn, and +e.g. make the output of commands like `git branch -a --contains +` needlessly verbose, as well as impacting anything else +that'll work with the complete set of known references. + +These remote-tracking references can be deleted as a one-off with +either of: + +------------------------------------------------ +# While fetching +$ git fetch --prune + +# Only prune, don't fetch +$ git remote prune +------------------------------------------------ + +To prune references as part of your normal workflow without needing to +remember to run that, set `fetch.prune` globally, or +`remote..prune` per-remote in the config. See +linkgit:git-config[1]. + +Here's where things get tricky and more specific. The pruning feature +doesn't actually care about branches, instead it'll prune local <-> +remote-references as a function of the refspec of the remote (see +`` and <> above). + +Therefore if the refspec for the remote includes +e.g. `refs/tags/*:refs/tags/*`, or you manually run e.g. `git fetch +--prune "refs/tags/*:refs/tags/*"` it won't be stale remote +tracking branches that are deleted, but any local tag that doesn't +exist on the remote. + +This might not be what you expect, i.e. you want to prune remote +``, but also explicitly fetch tags from it, so when you fetch +from it you delete all your local tags, most of which may not have +come from the `` remote in the first place. + +So be careful when using this with a refspec like +`refs/tags/*:refs/tags/*`, or any other refspec which might map +references from multiple remotes to the same local namespace. + OUTPUT ------ -- cgit v1.2.3 From d0e07472faa938c83e9b1f76567f16614672668c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Date: Fri, 9 Feb 2018 20:32:12 +0000 Subject: git remote doc: correct dangerous lies about what prune does MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The "git remote prune " command uses the same machinery as "git fetch --prune", and shares all the same caveats, but its documentation has suggested that it'll just "delete stale remote-tracking branches under ". This isn't true, and hasn't been true since at least v1.8.5.6 (the oldest version I could be bothered to test). E.g. if "refs/tags/*:refs/tags/*" is explicitly set in the refspec of the remote, it'll delete all local tags doesn't know about. Instead, briefly give the reader just enough of a hint that this option might constitute a shotgun aimed at their foot, and point them to the new PRUNING section in the git-fetch documentation which explains all the nuances of what this facility does. See "[BUG] git remote prune removes local tags, depending on fetch config" (CACi5S_39wNrbfjLfn0xhCY+uewtFN2YmnAcRc86z6pjUTjWPHQ@mail.gmail.com) by Michael Giuffrida for the initial report. Reported-by: Michael Giuffrida Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano --- Documentation/git-remote.txt | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) (limited to 'Documentation') diff --git a/Documentation/git-remote.txt b/Documentation/git-remote.txt index 577b969c1b..4feddc0293 100644 --- a/Documentation/git-remote.txt +++ b/Documentation/git-remote.txt @@ -172,10 +172,14 @@ With `-n` option, the remote heads are not queried first with 'prune':: -Deletes all stale remote-tracking branches under . -These stale branches have already been removed from the remote repository -referenced by , but are still locally available in -"remotes/". +Deletes stale references associated with . By default, stale +remote-tracking branches under are deleted, but depending on +global configuration and the configuration of the remote we might even +prune local tags that haven't been pushed there. Equivalent to `git +fetch --prune `, except that no new references will be fetched. ++ +See the PRUNING section of linkgit:git-fetch[1] for what it'll prune +depending on various configuration. + With `--dry-run` option, report what branches will be pruned, but do not actually prune them. @@ -189,7 +193,7 @@ remotes.default is not defined, all remotes which do not have the configuration parameter remote..skipDefaultUpdate set to true will be updated. (See linkgit:git-config[1]). + -With `--prune` option, prune all the remotes that are updated. +With `--prune` option, run pruning against all the remotes that are updated. DISCUSSION -- cgit v1.2.3 From 627a129b46530773369586f7982925fd41dc7227 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Date: Fri, 9 Feb 2018 20:32:13 +0000 Subject: git-fetch & config doc: link to the new PRUNING section MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Amend the documentation for fetch.prune, fetch..prune and --prune to link to the recently added PRUNING section. I'd have liked to link directly to it with "<>" from fetch-options.txt, since it's included in git-fetch.txt (git-pull.txt also includes it, but doesn't include that option). However making a reference across files yields this error: [...]/Documentation/git-fetch.xml:226: element xref: validity error : IDREF attribute linkend references an unknown ID "PRUNING" Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano --- Documentation/config.txt | 6 +++++- Documentation/fetch-options.txt | 3 +++ 2 files changed, 8 insertions(+), 1 deletion(-) (limited to 'Documentation') diff --git a/Documentation/config.txt b/Documentation/config.txt index 0e25b2c92b..0f27af5760 100644 --- a/Documentation/config.txt +++ b/Documentation/config.txt @@ -1398,7 +1398,8 @@ fetch.unpackLimit:: fetch.prune:: If true, fetch will automatically behave as if the `--prune` - option was given on the command line. See also `remote..prune`. + option was given on the command line. See also `remote..prune` + and the PRUNING section of linkgit:git-fetch[1]. fetch.output:: Control how ref update status is printed. Valid values are @@ -2944,6 +2945,9 @@ remote..prune:: remove any remote-tracking references that no longer exist on the remote (as if the `--prune` option was given on the command line). Overrides `fetch.prune` settings, if any. ++ +See also `remote..prune` and the PRUNING section of +linkgit:git-fetch[1]. remotes.:: The list of remotes which are fetched by "git remote update diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt index fb6bebbc61..9f5c85ad96 100644 --- a/Documentation/fetch-options.txt +++ b/Documentation/fetch-options.txt @@ -74,6 +74,9 @@ ifndef::git-pull[] line or in the remote configuration, for example if the remote was cloned with the --mirror option), then they are also subject to pruning. ++ +See the PRUNING section below for more details. + endif::git-pull[] ifndef::git-pull[] -- cgit v1.2.3 From 97716d217c1ea00adfc64e4f6bb85c1236d661ff Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Date: Fri, 9 Feb 2018 20:32:15 +0000 Subject: fetch: add a --prune-tags option and fetch.pruneTags config MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a --prune-tags option to git-fetch, along with fetch.pruneTags config option and a -P shorthand (-p is --prune). This allows for doing any of: git fetch -p -P git fetch --prune --prune-tags git fetch -p -P origin git fetch --prune --prune-tags origin Or simply: git config fetch.prune true && git config fetch.pruneTags true && git fetch Instead of the much more verbose: git fetch --prune origin 'refs/tags/*:refs/tags/*' '+refs/heads/*:refs/remotes/origin/*' Before this feature it was painful to support the use-case of pulling from a repo which is having both its branches *and* tags deleted regularly, and have our local references to reflect upstream. At work we create deployment tags in the repo for each rollout, and there's *lots* of those, so they're archived within weeks for performance reasons. Without this change it's hard to centrally configure such repos in /etc/gitconfig (on servers that are only used for working with them). You need to set fetch.prune=true globally, and then for each repo: git -C {} config --replace-all remote.origin.fetch "refs/tags/*:refs/tags/*" "^\+*refs/tags/\*:refs/tags/\*$" Now I can simply set fetch.pruneTags=true in /etc/gitconfig as well, and users running "git pull" will automatically get the pruning semantics I want. Even though "git remote" has corresponding "prune" and "update --prune" subcommands I'm intentionally not adding a corresponding prune-tags or "update --prune --prune-tags" mode to that command. It's advertised (as noted in my recent "git remote doc: correct dangerous lies about what prune does") as only modifying remote tracking references, whereas any --prune-tags option is always going to modify what from the user's perspective is a local copy of the tag, since there's no such thing as a remote tracking tag. Ideally add_prune_tags_to_fetch_refspec() would be something that would use ALLOC_GROW() to grow the 'fetch` member of the 'remote' struct. Instead I'm realloc-ing remote->fetch and adding the tag_refspec to the end. The reason is that parse_{fetch,push}_refspec which allocate the refspec (ultimately remote->fetch) struct are called many places that don't have access to a 'remote' struct. It would be hard to change all their callsites to be amenable to carry around the bookkeeping variables required for dynamic allocation. All the other callers of the API first incrementally construct the string version of the refspec in remote->fetch_refspec via add_fetch_refspec(), before finally calling parse_fetch_refspec() via some variation of remote_get(). It's less of a pain to deal with the one special case that needs to modify already constructed refspecs than to chase down and change all the other callsites. The API I'm adding is intentionally not generalized because if we add more of these we'd probably want to re-visit how this is done. See my "Re: [BUG] git remote prune removes local tags, depending on fetch config" (87po6ahx87.fsf@evledraar.gmail.com; https://public-inbox.org/git/87po6ahx87.fsf@evledraar.gmail.com/) for more background info. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano --- Documentation/config.txt | 14 ++++++++++++ Documentation/fetch-options.txt | 14 +++++++++++- Documentation/git-fetch.txt | 47 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 74 insertions(+), 1 deletion(-) (limited to 'Documentation') diff --git a/Documentation/config.txt b/Documentation/config.txt index 0f27af5760..e254bfd531 100644 --- a/Documentation/config.txt +++ b/Documentation/config.txt @@ -1401,6 +1401,14 @@ fetch.prune:: option was given on the command line. See also `remote..prune` and the PRUNING section of linkgit:git-fetch[1]. +fetch.pruneTags:: + If true, fetch will automatically behave as if the + `refs/tags/*:refs/tags/*` refspec was provided when pruning, + if not set already. This allows for setting both this option + and `fetch.prune` to maintain a 1=1 mapping to upstream + refs. See also `remote..pruneTags` and the PRUNING + section of linkgit:git-fetch[1]. + fetch.output:: Control how ref update status is printed. Valid values are `full` and `compact`. Default value is `full`. See section @@ -2945,6 +2953,12 @@ remote..prune:: remove any remote-tracking references that no longer exist on the remote (as if the `--prune` option was given on the command line). Overrides `fetch.prune` settings, if any. + +remote..pruneTags:: + When set to true, fetching from this remote by default will also + remove any local tags that no longer exist on the remote if pruning + is activated in general via `remote..prune`, `fetch.prune` or + `--prune`. Overrides `fetch.pruneTags` settings, if any. + See also `remote..prune` and the PRUNING section of linkgit:git-fetch[1]. diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt index 9f5c85ad96..8631e365f4 100644 --- a/Documentation/fetch-options.txt +++ b/Documentation/fetch-options.txt @@ -73,7 +73,19 @@ ifndef::git-pull[] are fetched due to an explicit refspec (either on the command line or in the remote configuration, for example if the remote was cloned with the --mirror option), then they are also - subject to pruning. + subject to pruning. Supplying `--prune-tags` is a shorthand for + providing the tag refspec. ++ +See the PRUNING section below for more details. + +-P:: +--prune-tags:: + Before fetching, remove any local tags that no longer exist on + the remote if `--prune` is enabled. This option should be used + more carefully, unlike `--prune` it will remove any local + references (local tags) that have been created. This option is + a shorthand for providing the explicit tag refspec along with + `--prune`, see the discussion about that in its documentation. + See the PRUNING section below for more details. diff --git a/Documentation/git-fetch.txt b/Documentation/git-fetch.txt index e94bcfb8c3..af12310f75 100644 --- a/Documentation/git-fetch.txt +++ b/Documentation/git-fetch.txt @@ -148,6 +148,53 @@ So be careful when using this with a refspec like `refs/tags/*:refs/tags/*`, or any other refspec which might map references from multiple remotes to the same local namespace. +Since keeping up-to-date with both branches and tags on the remote is +a common use-case the `--prune-tags` option can be supplied along with +`--prune` to prune local tags that don't exist on the remote, and +force-update those tags that differ. Tag pruning can also be enabled +with `fetch.pruneTags` or `remote..pruneTags` in the config. See +linkgit:git-config[1]. + +The `--prune-tags` option is equivalent to having +`refs/tags/*:refs/tags/*` declared in the refspecs of the remote. This +can lead to some seemingly strange interactions: + +------------------------------------------------ +# These both fetch tags +$ git fetch --no-tags origin 'refs/tags/*:refs/tags/*' +$ git fetch --no-tags --prune-tags origin +------------------------------------------------ + +The reason it doesn't error out when provided without `--prune` or its +config versions is for flexibility of the configured versions, and to +maintain a 1=1 mapping between what the command line flags do, and +what the configuration versions do. + +It's reasonable to e.g. configure `fetch.pruneTags=true` in +`~/.gitconfig` to have tags pruned whenever `git fetch --prune` is +run, without making every invocation of `git fetch` without `--prune` +an error. + +Another special case of `--prune-tags` is that +`refs/tags/*:refs/tags/*` will not be implicitly provided if an URL is +being fetched. I.e.: + +------------------------------------------------ +$ git fetch --prune --prune-tags +------------------------------------------------ + +Will prune no tags, as opposed to: + +------------------------------------------------ +$ git fetch origin --prune --prune-tags +------------------------------------------------ + +To prune tags given a URL supply the refspec explicitly: + +------------------------------------------------ +$ git fetch --prune 'refs/tags/*:refs/tags/*' +------------------------------------------------ + OUTPUT ------ -- cgit v1.2.3 From 6317972cff9b4df7a6cc666b08be7133ba81617c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Date: Fri, 9 Feb 2018 20:32:16 +0000 Subject: fetch: make the --prune-tags work with MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Make the new --prune-tags option work properly when git-fetch is invoked with a parameter instead of a parameter. This change is split off from the introduction of --prune-tags due to the relative complexity of munging the incoming argv, which is easier to review as a separate change. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano --- Documentation/git-fetch.txt | 21 ++++++--------------- 1 file changed, 6 insertions(+), 15 deletions(-) (limited to 'Documentation') diff --git a/Documentation/git-fetch.txt b/Documentation/git-fetch.txt index af12310f75..e319935597 100644 --- a/Documentation/git-fetch.txt +++ b/Documentation/git-fetch.txt @@ -175,24 +175,15 @@ It's reasonable to e.g. configure `fetch.pruneTags=true` in run, without making every invocation of `git fetch` without `--prune` an error. -Another special case of `--prune-tags` is that -`refs/tags/*:refs/tags/*` will not be implicitly provided if an URL is -being fetched. I.e.: - ------------------------------------------------- -$ git fetch --prune --prune-tags ------------------------------------------------- - -Will prune no tags, as opposed to: +Pruning tags with `--prune-tags` also works when fetching a URL +instead of a named remote. These will all prune tags not found on +origin: ------------------------------------------------ $ git fetch origin --prune --prune-tags ------------------------------------------------- - -To prune tags given a URL supply the refspec explicitly: - ------------------------------------------------- -$ git fetch --prune 'refs/tags/*:refs/tags/*' +$ git fetch origin --prune 'refs/tags/*:refs/tags/*' +$ git fetch --prune --prune-tags +$ git fetch --prune 'refs/tags/*:refs/tags/*' ------------------------------------------------ OUTPUT -- cgit v1.2.3