Welcome to mirror list, hosted at ThFree Co, Russian Federation.

git.kernel.org/pub/scm/git/git.git - Unnamed repository; edit this file 'description' to name the repository.
summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorShaoxuan Yuan <shaoxuan.yuan02@gmail.com>2022-09-23 07:18:42 +0300
committerJunio C Hamano <gitster@pobox.com>2022-09-23 19:41:27 +0300
commit7cae7627c459ae5d0cd8f4c1ff76601e6c952bf3 (patch)
treebc43aa75cb2a561092f2e518bedecf0d3bc881d7 /t/t1092-sparse-checkout-compatibility.sh
parent07ee72db0e97b5c233f8ada0abb412248c2f1c6f (diff)
builtin/grep.c: integrate with sparse index
Turn on sparse index and remove ensure_full_index(). Before this patch, `git-grep` utilizes the ensure_full_index() method to expand the index and search all the entries. Because this method requires walking all the trees and constructing the index, it is the slow part within the whole command. To achieve better performance, this patch uses grep_tree() to search the sparse directory entries and get rid of the ensure_full_index() method. Why grep_tree() is a better choice over ensure_full_index()? 1) grep_tree() is as correct as ensure_full_index(). grep_tree() looks into every sparse-directory entry (represented by a tree) recursively when looping over the index, and the result of doing so matches the result of expanding the index. 2) grep_tree() utilizes pathspecs to limit the scope of searching. ensure_full_index() always expands the index, which means it will always walk all the trees and blobs in the repo without caring if the user only wants a subset of the content, i.e. using a pathspec. On the other hand, grep_tree() will only search the contents that match the pathspec, and thus possibly walking fewer trees. 3) grep_tree() does not construct and copy back a new index, while ensure_full_index() does. This also saves some time. ---------------- Performance test - Summary: p2000 tests demonstrate a ~71% execution time reduction for `git grep --cached bogus -- "f2/f1/f1/*"` using tree-walking logic. However, notice that this result varies depending on the pathspec given. See below "Command used for testing" for more details. Test HEAD~ HEAD ------------------------------------------------------- 2000.78: git grep ... (full-v3) 0.35 0.39 (≈) 2000.79: git grep ... (full-v4) 0.36 0.30 (≈) 2000.80: git grep ... (sparse-v3) 0.88 0.23 (-73.8%) 2000.81: git grep ... (sparse-v4) 0.83 0.26 (-68.6%) - Command used for testing: git grep --cached bogus -- "f2/f1/f1/*" The reason for specifying a pathspec is that, if we don't specify a pathspec, then grep_tree() will walk all the trees and blobs to find the pattern, and the time consumed doing so is not too different from using the original ensure_full_index() method, which also spends most of the time walking trees. However, when a pathspec is specified, this latest logic will only walk the area of trees enclosed by the pathspec, and the time consumed is reasonably a lot less. Generally speaking, because the performance gain is acheived by walking less trees, which are specified by the pathspec, the HEAD time v.s. HEAD~ time in sparse-v[3|4], should be proportional to "pathspec enclosed area" v.s. "all area", respectively. Namely, the wider the <pathspec> is encompassing, the less the performance difference between HEAD~ and HEAD, and vice versa. That is, if we don't specify a pathspec, the performance difference [1] is indistinguishable: both methods walk all the trees and take generally same amount of time (even with the index construction time included for ensure_full_index()). [1] Performance test result without pathspec (hence walking all trees): Command used: git grep --cached bogus Test HEAD~ HEAD --------------------------------------------------- 2000.78: git grep ... (full-v3) 6.17 5.19 (≈) 2000.79: git grep ... (full-v4) 6.19 5.46 (≈) 2000.80: git grep ... (sparse-v3) 6.57 6.44 (≈) 2000.81: git grep ... (sparse-v4) 6.65 6.28 (≈) -------------------------- NEEDSWORK about submodules There are a few NEEDSWORKs that belong to improvements beyond this topic. See the NEEDSWORK in builtin/grep.c::grep_submodule() for more context. The other two NEEDSWORKs in t1092 are also relative. Suggested-by: Derrick Stolee <derrickstolee@github.com> Helped-by: Derrick Stolee <derrickstolee@github.com> Helped-by: Victoria Dye <vdye@github.com> Helped-by: Elijah Newren <newren@gmail.com> Signed-off-by: Shaoxuan Yuan <shaoxuan.yuan02@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Diffstat (limited to 't/t1092-sparse-checkout-compatibility.sh')
-rwxr-xr-xt/t1092-sparse-checkout-compatibility.sh72
1 files changed, 72 insertions, 0 deletions
diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh
index a6a14c8a21..37b828ba5e 100755
--- a/t/t1092-sparse-checkout-compatibility.sh
+++ b/t/t1092-sparse-checkout-compatibility.sh
@@ -162,6 +162,19 @@ init_repos () {
git -C sparse-index sparse-checkout set deep
}
+init_repos_as_submodules () {
+ git reset --hard &&
+ init_repos &&
+ git submodule add ./full-checkout &&
+ git submodule add ./sparse-checkout &&
+ git submodule add ./sparse-index &&
+
+ git submodule status >actual &&
+ grep full-checkout actual &&
+ grep sparse-checkout actual &&
+ grep sparse-index actual
+}
+
run_on_sparse () {
(
cd sparse-checkout &&
@@ -1972,4 +1985,63 @@ test_expect_success 'sparse index is not expanded: rm' '
ensure_not_expanded rm -r deep
'
+test_expect_success 'grep with and --cached' '
+ init_repos &&
+
+ test_all_match git grep --cached a &&
+ test_all_match git grep --cached a -- "folder1/*"
+'
+
+test_expect_success 'grep is not expanded' '
+ init_repos &&
+
+ ensure_not_expanded grep a &&
+ ensure_not_expanded grep a -- deep/* &&
+
+ # All files within the folder1/* pathspec are sparse,
+ # so this command does not find any matches
+ ensure_not_expanded ! grep a -- folder1/* &&
+
+ # test out-of-cone pathspec with or without wildcard
+ ensure_not_expanded grep --cached a -- "folder1/a" &&
+ ensure_not_expanded grep --cached a -- "folder1/*" &&
+
+ # test in-cone pathspec with or without wildcard
+ ensure_not_expanded grep --cached a -- "deep/a" &&
+ ensure_not_expanded grep --cached a -- "deep/*"
+'
+
+# NEEDSWORK: when running `grep` in the superproject with --recurse-submodules,
+# Git expands the index of the submodules unexpectedly. Even though `grep`
+# builtin is marked as "command_requires_full_index = 0", this config is only
+# useful for the superproject. Namely, the submodules have their own configs,
+# which are _not_ populated by the one-time sparse-index feature switch.
+test_expect_failure 'grep within submodules is not expanded' '
+ init_repos_as_submodules &&
+
+ # do not use ensure_not_expanded() here, becasue `grep` should be
+ # run in the superproject, not in "./sparse-index"
+ GIT_TRACE2_EVENT="$(pwd)/trace2.txt" \
+ git grep --cached --recurse-submodules a -- "*/folder1/*" &&
+ test_region ! index ensure_full_index trace2.txt
+'
+
+# NEEDSWORK: this test is not actually testing the code. The design purpose
+# of this test is to verify the grep result when the submodules are using a
+# sparse-index. Namely, we want "folder1/" as a tree (a sparse directory); but
+# because of the index expansion, we are now grepping the "folder1/a" blob.
+# Because of the problem stated above 'grep within submodules is not expanded',
+# we don't have the ideal test environment yet.
+test_expect_success 'grep sparse directory within submodules' '
+ init_repos_as_submodules &&
+
+ cat >expect <<-\EOF &&
+ full-checkout/folder1/a:a
+ sparse-checkout/folder1/a:a
+ sparse-index/folder1/a:a
+ EOF
+ git grep --cached --recurse-submodules a -- "*/folder1/*" >actual &&
+ test_cmp actual expect
+'
+
test_done