Skip to content

Commit d6388c2

Browse files
committed
Check the specific Windows GIX_TEST_IGNORE_ARCHIVES=1 failures
This modifies the `test-fixtures-windows` job that tests on Windows with `GIX_TEST_IGNORE_ARCHIVES=1` so that, instead of checking that no more than 14 failures occur, it checks that the failing tests are exactly those that are documented in GitoxideLabs#1358 as expected to fail. The initial check that no tests have *error* status is preserved, with only stylistic changes, and kept separate from the subsequent logic so that the output is clearer. The new steps are no longer conditional on `nextest` having exited with a failure status, since (a) that was probably unnecessary before and definitely unnecessary now, (b) at last for now, the comparison is precise, so it would be strange to pass if the diff were to have changes on *all* lines, and (c) this makes it slightly less likely that GitoxideLabs#1358 will accidentally stay open even once fixed. The current approach is to actually retrieve the list of tests expected to fail on Windows with `GIX_TEST_IGNORE_ARCHIVES=1` from the GitoxideLabs#1358 issue body. This has the advantage that it automatically keeps up to date with changes made to that issue description, but this is of course not the only possible approach for populating the expected value. Two changes should be made before this is ready: - As noted in the "FIXME" comment, the job should currently fail becuase the performance test reported to fail in GitoxideLabs#1358 is not being filtered out from the expected failures list. It's left in as of this commit, to verify that the job is capable of failing. (After that, the performance test should either be filtered out or removed from the list in GitoxideLabs#1358, but the former approach is currently preferable because I have not done diverse enough testing to check if the failure on my main Windows system is due to that system being too slow rather than a performance bug.) - The scratchwork file should be removed once no longer needed.
1 parent cf0c7ee commit d6388c2

File tree

2 files changed

+77
-8
lines changed

2 files changed

+77
-8
lines changed

.github/workflows/ci.yml

+34-8
Original file line numberDiff line numberDiff line change
@@ -95,15 +95,41 @@ jobs:
9595
GIX_TEST_IGNORE_ARCHIVES: 1
9696
run: cargo nextest --profile=with-xml run --workspace --no-fail-fast
9797
continue-on-error: true
98-
- name: Check how many tests failed
99-
if: steps.nextest.outcome == 'failure'
100-
env:
101-
# See https://github.com/GitoxideLabs/gitoxide/issues/1358.
102-
EXPECTED_FAILURE_COUNT: 14
98+
- name: Check for errors
99+
run: |
100+
[xml]$junit_xml = Get-Content -Path 'target/nextest/with-xml/junit.xml'
101+
if ($junit_xml.testsuites.errors -ne 0) { exit 1 }
102+
- name: Collect actual failures
103+
run: |
104+
[xml]$junit_xml = Get-Content 'target/nextest/with-xml/junit.xml'
105+
106+
$actual_failures = $junit_xml.SelectNodes("//testcase[failure]") |
107+
ForEach-Object { "$($_.classname) $($_.name)" } |
108+
Sort-Object
109+
110+
Write-Output $actual_failures
111+
Set-Content -Path 'actual-failures.txt' -Value $actual_failures
112+
- name: Collect expected failures
113+
run: |
114+
$issue = 1358 # https://github.com/GitoxideLabs/gitoxide/issues/1358
115+
116+
$match_info = gh issue --repo GitoxideLabs/gitoxide view $issue --json body --jq .body |
117+
Out-String |
118+
Select-String -Pattern '(?s)```text\r?\n(.*?)```'
119+
120+
# FIXME: Check that the diff can fail, then filter out performance tests in Where-Object.
121+
$expected_failures = $match_info.Matches.Groups[1].Value -split "`n" |
122+
Where-Object { $_ -match '^\s*FAIL \[' } |
123+
ForEach-Object { $_ -replace '^\s*FAIL \[\s*\d+\.\d+s\]\s*', '' -replace '\s+$', '' } |
124+
Sort-Object
125+
126+
Write-Output $expected_failures
127+
Set-Content -Path 'expected-failures.txt' -Value $expected_failures
128+
- name: Compare expected and actual failures
103129
run: |
104-
[xml]$junit = Get-Content -Path 'target/nextest/with-xml/junit.xml'
105-
if ($junit.testsuites.errors -ne 0) { exit 1 }
106-
if ($junit.testsuites.failures -gt $env:EXPECTED_FAILURE_COUNT) { exit 1 }
130+
# Fail the check if there are any differences, even unexpectedly passing tests, so they can be
131+
# investigated. (If this check is made blocking for PRs, this exact check may need to be changed.)
132+
git --no-pager diff --no-index --exit-code -U1000000 -- expected-failures.txt actual-failures.txt
107133
108134
test-32bit:
109135
runs-on: ubuntu-latest

scratchwork.ps1

+43
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# Omit from all script steps, becuase GHA prepends it.
2+
$ErrorActionPreference = 'stop'
3+
4+
5+
Write-Output '====== Output of scratchwork for "Collect actual failures" step: ======'
6+
7+
[xml]$junit_xml = Get-Content 'target/nextest/with-xml/junit.xml'
8+
9+
$actual_failures = $junit_xml.SelectNodes("//testcase[failure]") |
10+
ForEach-Object { "$($_.classname) $($_.name)" } |
11+
Sort-Object
12+
13+
Write-Output $actual_failures
14+
Set-Content -Path 'actual-failures.txt' -Value $actual_failures
15+
16+
17+
Write-Output '====== Output of scratchwork for "Collect expected failures" step: ======'
18+
19+
$issue = 1358 # https://github.com/GitoxideLabs/gitoxide/issues/1358
20+
21+
$match_info = gh issue --repo GitoxideLabs/gitoxide view $issue --json body --jq .body |
22+
Out-String |
23+
Select-String -Pattern '(?s)```text\r?\n(.*?)```'
24+
25+
# FIXME: Check that the diff can fail, then filter out performance tests in Where-Object.
26+
$expected_failures = $match_info.Matches.Groups[1].Value -split "`n" |
27+
Where-Object { $_ -match '^\s*FAIL \[' } |
28+
ForEach-Object { $_ -replace '^\s*FAIL \[\s*\d+\.\d+s\]\s*', '' -replace '\s+$', '' } |
29+
Sort-Object
30+
31+
Write-Output $expected_failures
32+
Set-Content -Path 'expected-failures.txt' -Value $expected_failures
33+
34+
35+
Write-Output '====== Output of scratchwork for "Compare expected and actual failures" step: ======'
36+
37+
# Fail the check if there are any differences, even unexpectedly passing tests, so they can be
38+
# investigated. (If this check is made blocking for PRs, this exact check may need to be changed.)
39+
git --no-pager diff --no-index --exit-code -U1000000 -- expected-failures.txt actual-failures.txt
40+
41+
42+
# Omit from script steps, because GHA appends it.
43+
if ((Test-Path -LiteralPath variable:\LASTEXITCODE)) { exit $LASTEXITCODE }

0 commit comments

Comments
 (0)