diff options
author | Junio C Hamano <gitster@pobox.com> | 2008-02-01 12:50:53 +0300 |
---|---|---|
committer | Junio C Hamano <gitster@pobox.com> | 2008-02-02 07:49:34 +0300 |
commit | 41ac414ea2bef81af94474cbef25a38868b4788e (patch) | |
tree | e9c598e65753ab473eefc6fcc5899714d8085a2f /t/test-lib.sh | |
parent | 6ce8e44a1eeaa07325f1304f6f392f35f54d29c7 (diff) |
Sane use of test_expect_failure
Originally, test_expect_failure was designed to be the opposite
of test_expect_success, but this was a bad decision. Most tests
run a series of commands that leads to the single command that
needs to be tested, like this:
test_expect_{success,failure} 'test title' '
setup1 &&
setup2 &&
setup3 &&
what is to be tested
'
And expecting a failure exit from the whole sequence misses the
point of writing tests. Your setup$N that are supposed to
succeed may have failed without even reaching what you are
trying to test. The only valid use of test_expect_failure is to
check a trivial single command that is expected to fail, which
is a minority in tests of Porcelain-ish commands.
This large-ish patch rewrites all uses of test_expect_failure to
use test_expect_success and rewrites the condition of what is
tested, like this:
test_expect_success 'test title' '
setup1 &&
setup2 &&
setup3 &&
! this command should fail
'
test_expect_failure is redefined to serve as a reminder that
that test *should* succeed but due to a known breakage in git it
currently does not pass. So if git-foo command should create a
file 'bar' but you discovered a bug that it doesn't, you can
write a test like this:
test_expect_failure 'git-foo should create bar' '
rm -f bar &&
git foo &&
test -f bar
'
This construct acts similar to test_expect_success, but instead
of reporting "ok/FAIL" like test_expect_success does, the
outcome is reported as "FIXED/still broken".
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Diffstat (limited to 't/test-lib.sh')
-rw-r--r-- | t/test-lib.sh | 30 |
1 files changed, 26 insertions, 4 deletions
diff --git a/t/test-lib.sh b/t/test-lib.sh index 142540e1b1..9a3c0b4cad 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -139,6 +139,8 @@ fi test_failure=0 test_count=0 +test_fixed=0 +test_broken=0 trap 'echo >&5 "FATAL: Unexpected exit with code $?"; exit 1' exit @@ -171,6 +173,17 @@ test_failure_ () { test "$immediate" = "" || { trap - exit; exit 1; } } +test_known_broken_ok_ () { + test_count=$(expr "$test_count" + 1) + test_fixed=$(($test_fixed+1)) + say_color "" " FIXED $test_count: $@" +} + +test_known_broken_failure_ () { + test_count=$(expr "$test_count" + 1) + test_broken=$(($test_broken+1)) + say_color skip " still broken $test_count: $@" +} test_debug () { test "$debug" = "" || eval "$1" @@ -211,13 +224,13 @@ test_expect_failure () { error "bug in the test script: not 2 parameters to test-expect-failure" if ! test_skip "$@" then - say >&3 "expecting failure: $2" + say >&3 "checking known breakage: $2" test_run_ "$2" - if [ "$?" = 0 -a "$eval_ret" != 0 -a "$eval_ret" -lt 129 ] + if [ "$?" = 0 -a "$eval_ret" = 0 ] then - test_ok_ "$1" + test_known_broken_ok_ "$1" else - test_failure_ "$@" + test_known_broken_failure_ "$1" fi fi echo >&3 "" @@ -274,6 +287,15 @@ test_create_repo () { test_done () { trap - exit + + if test "$test_fixed" != 0 + then + say_color pass "fixed $test_fixed known breakage(s)" + fi + if test "$test_broken" != 0 + then + say_color error "still have $test_broken known breakage(s)" + fi case "$test_failure" in 0) # We could: |