From e86bbcf987faad8e487beb814351a404fa80957f Mon Sep 17 00:00:00 2001 From: Elijah Newren Date: Tue, 17 Sep 2019 09:35:01 -0700 Subject: clean: disambiguate the definition of -d The -d flag pre-dated git-clean's ability to have paths specified. As such, the default for git-clean was to only remove untracked files in the current directory, and -d existed to allow it to recurse into subdirectories. The interaction of paths and the -d option appears to not have been carefully considered, as evidenced by numerous bugs and a dearth of tests covering such pairings in the testsuite. The definition turns out to be important, so let's look at some of the various ways one could interpret the -d option: A) Without -d, only look in subdirectories which contain tracked files under them; with -d, also look in subdirectories which are untracked for files to clean. B) Without specified paths from the user for us to delete, we need to have some kind of default, so...without -d, only look in subdirectories which contain tracked files under them; with -d, also look in subdirectories which are untracked for files to clean. The important distinction here is that choice B says that the presence or absence of '-d' is irrelevant if paths are specified. The logic behind option B is that if a user explicitly asked us to clean a specified pathspec, then we should clean anything that matches that pathspec. Some examples may clarify. Should git clean -f untracked_dir/file remove untracked_dir/file or not? It seems crazy not to, but a strict reading of option A says it shouldn't be removed. How about git clean -f untracked_dir/file1 tracked_dir/file2 or git clean -f untracked_dir_1/file1 untracked_dir_2/file2 ? Should it remove either or both of these files? Should it require multiple runs to remove both the files listed? (If this sounds like a crazy question to even ask, see the commit message of "t7300: Add some testcases showing failure to clean specified pathspecs" added earlier in this patch series.) What if -ffd were used instead of -f -- should that allow these to be removed? Should it take multiple invocations with -ffd? What if a glob (such as '*tracked*') were used instead of spelling out the directory names? What if the filenames involved globs, such as git clean -f '*.o' or git clean -f '*/*.o' ? The current documentation actually suggests a definition that is slightly different than choice A, and the implementation prior to this series provided something radically different than either choices A or B. (The implementation, though, was clearly just buggy). There may be other choices as well. However, for almost any given choice of definition for -d that I can think of, some of the examples above will appear buggy to the user. The only case that doesn't have negative surprises is choice B: treat a user-specified path as a request to clean all untracked files which match that path specification, including recursing into any untracked directories. Change the documentation and basic implementation to use this definition. There were two regression tests that indirectly depended on the current implementation, but neither was about subdirectory handling. These two tests were introduced in commit 5b7570cfb41c ("git-clean: add tests for relative path", 2008-03-07) which was solely created to add coverage for the changes in commit fb328947c8e ("git-clean: correct printing relative path", 2008-03-07). Both tests specified a directory that happened to have an untracked subdirectory, but both were only checking that the resulting printout of a file that was removed was shown with a relative path. Update these tests appropriately. Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano --- builtin/clean.c | 8 ++++++++ 1 file changed, 8 insertions(+) (limited to 'builtin/clean.c') diff --git a/builtin/clean.c b/builtin/clean.c index d5579da716..68d70e41c0 100644 --- a/builtin/clean.c +++ b/builtin/clean.c @@ -949,6 +949,14 @@ int cmd_clean(int argc, const char **argv, const char *prefix) dir.flags |= DIR_SHOW_OTHER_DIRECTORIES; + if (argc) { + /* + * Remaining args implies pathspecs specified, and we should + * recurse within those. + */ + remove_directories = 1; + } + if (remove_directories) dir.flags |= DIR_SHOW_IGNORED_TOO | DIR_KEEP_UNTRACKED_CONTENTS; -- cgit v1.2.3 From 09487f2cbad36d5da42e204c9e0d201e8607acb8 Mon Sep 17 00:00:00 2001 From: Elijah Newren Date: Tue, 17 Sep 2019 09:35:02 -0700 Subject: clean: avoid removing untracked files in a nested git repository Users expect files in a nested git repository to be left alone unless sufficiently forced (with two -f's). Unfortunately, in certain circumstances, git would delete both tracked (and possibly dirty) files and untracked files within a nested repository. To explain how this happens, let's contrast a couple cases. First, take the following example setup (which assumes we are already within a git repo): git init nested cd nested >tracked git add tracked git commit -m init >untracked cd .. In this setup, everything works as expected; running 'git clean -fd' will result in fill_directory() returning the following paths: nested/ nested/tracked nested/untracked and then correct_untracked_entries() would notice this can be compressed to nested/ and then since "nested/" is a directory, we would call remove_dirs("nested/", ...), which would check is_nonbare_repository_dir() and then decide to skip it. However, if someone also creates an ignored file: >nested/ignored then running 'git clean -fd' would result in fill_directory() returning the same paths: nested/ nested/tracked nested/untracked but correct_untracked_entries() will notice that we had ignored entries under nested/ and thus simplify this list to nested/tracked nested/untracked Since these are not directories, we do not call remove_dirs() which was the only place that had the is_nonbare_repository_dir() safety check -- resulting in us deleting both the untracked file and the tracked (and possibly dirty) file. One possible fix for this issue would be walking the parent directories of each path and checking if they represent nonbare repositories, but that would be wasteful. Even if we added caching of some sort, it's still a waste because we should have been able to check that "nested/" represented a nonbare repository before even descending into it in the first place. Add a DIR_SKIP_NESTED_GIT flag to dir_struct.flags and use it to prevent fill_directory() and friends from descending into nested git repos. With this change, we also modify two regression tests added in commit 91479b9c72f1 ("t7300: add tests to document behavior of clean and nested git", 2015-06-15). That commit, nor its series, nor the six previous iterations of that series on the mailing list discussed why those tests coded the expectation they did. In fact, it appears their purpose was simply to test _existing_ behavior to make sure that the performance changes didn't change the behavior. However, these two tests directly contradicted the manpage's claims that two -f's were required to delete files/directories under a nested git repository. While one could argue that the user gave an explicit path which matched files/directories that were within a nested repository, there's a slippery slope that becomes very difficult for users to understand once you go down that route (e.g. what if they specified "git clean -f -d '*.c'"?) It would also be hard to explain what the exact behavior was; avoid such problems by making it really simple. Also, clean up some grammar errors describing this functionality in the git-clean manpage. Finally, there are still a couple bugs with -ffd not cleaning out enough (e.g. missing the nested .git) and with -ffdX possibly cleaning out the wrong files (paying attention to outer .gitignore instead of inner). This patch does not address these cases at all (and does not change the behavior relative to those flags), it only fixes the handling when given a single -f. See https://public-inbox.org/git/20190905212043.GC32087@szeder.dev/ for more discussion of the -ffd[X?] bugs. Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano --- builtin/clean.c | 2 ++ 1 file changed, 2 insertions(+) (limited to 'builtin/clean.c') diff --git a/builtin/clean.c b/builtin/clean.c index 68d70e41c0..3a7a63ae71 100644 --- a/builtin/clean.c +++ b/builtin/clean.c @@ -946,6 +946,8 @@ int cmd_clean(int argc, const char **argv, const char *prefix) if (force > 1) rm_flags = 0; + else + dir.flags |= DIR_SKIP_NESTED_GIT; dir.flags |= DIR_SHOW_OTHER_DIRECTORIES; -- cgit v1.2.3 From ca8b5390db928014af9aadc73c876198501bfb35 Mon Sep 17 00:00:00 2001 From: Elijah Newren Date: Tue, 17 Sep 2019 09:35:03 -0700 Subject: clean: rewrap overly long line Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano --- builtin/clean.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) (limited to 'builtin/clean.c') diff --git a/builtin/clean.c b/builtin/clean.c index 3a7a63ae71..6030842f3a 100644 --- a/builtin/clean.c +++ b/builtin/clean.c @@ -158,7 +158,8 @@ static int remove_dirs(struct strbuf *path, const char *prefix, int force_flag, *dir_gone = 1; - if ((force_flag & REMOVE_DIR_KEEP_NESTED_GIT) && is_nonbare_repository_dir(path)) { + if ((force_flag & REMOVE_DIR_KEEP_NESTED_GIT) && + is_nonbare_repository_dir(path)) { if (!quiet) { quote_path_relative(path->buf, prefix, "ed); printf(dry_run ? _(msg_would_skip_git_dir) : _(msg_skip_git_dir), -- cgit v1.2.3 From 902b90cf42bc3c05f9fd124edac42ef1fc4ffafa Mon Sep 17 00:00:00 2001 From: Elijah Newren Date: Tue, 17 Sep 2019 09:35:04 -0700 Subject: clean: fix theoretical path corruption cmd_clean() had the following code structure: struct strbuf abs_path = STRBUF_INIT; for_each_string_list_item(item, &del_list) { strbuf_addstr(&abs_path, prefix); strbuf_addstr(&abs_path, item->string); PROCESS(&abs_path); strbuf_reset(&abs_path); } where I've elided a bunch of unnecessary details and PROCESS(&abs_path) represents a big chunk of code rather than an actual function call. One piece of PROCESS was: if (lstat(abs_path.buf, &st)) continue; which would cause the strbuf_reset() to be missed -- meaning that the next path to be handled would have two paths concatenated. This path used to use die_errno() instead of continue prior to commit 396049e5fb62 ("git-clean: refactor git-clean into two phases", 2013-06-25), but my understanding of how correct_untracked_entries() works is that it will prevent both dir/ and dir/file from being in the list to clean so this should be dead code and the die_errno() should be safe. But I hesitate to remove it since I am not certain. However, we can fix both this bug and possible similar future bugs by simply moving the strbuf_reset(&abs_path) to the beginning of the loop. It'll result in N calls to strbuf_reset() instead of N-1, but that's a small price to pay to avoid sneaky bugs like this. Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano --- builtin/clean.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'builtin/clean.c') diff --git a/builtin/clean.c b/builtin/clean.c index 6030842f3a..4cf2399f59 100644 --- a/builtin/clean.c +++ b/builtin/clean.c @@ -1018,6 +1018,7 @@ int cmd_clean(int argc, const char **argv, const char *prefix) for_each_string_list_item(item, &del_list) { struct stat st; + strbuf_reset(&abs_path); if (prefix) strbuf_addstr(&abs_path, prefix); @@ -1051,7 +1052,6 @@ int cmd_clean(int argc, const char **argv, const char *prefix) printf(dry_run ? _(msg_would_remove) : _(msg_remove), qname); } } - strbuf_reset(&abs_path); } strbuf_release(&abs_path); -- cgit v1.2.3