Skip to content
2026-10-04 full-chain

Curvine full-chain daily test report - 2026-10-04

Quality conclusion

Executive summary

Caution

Release decision: NO-GO. Pipeline FAIL; ran 7 profiles, 2 passed, 4 failed.

Quality gates

GateCriterionActualVerdict
Full-chain resultAll required profiles passed2/7 passedFAIL
Failure attributionFailures classified10 root-cause groupsPASS
Resource cleanupAll profile cleanups succeeded7/7PASS

Conclusion

This full-chain run did not pass. Failed profiles require resolution before release.

Test results

Profile summary

ProfilePreflightResultDurationClassCleanup
fastPASSPASS12m 14spassedPASSED
integrationPASSFAIL5m 18sharness_failurePASSED
dailyPASSFAIL9m 30sunknown_failurePASSED
fusePASSFAIL4m 32sunknown_failurePASSED
ltpPASSFAIL37m 42sunknown_failurePASSED
csiPASSPASS0m 36spassedPASSED
perf-benchmarkNOT_RECORDEDFAILED2m 42sfailedPASSED

LTP

  • Status: COMPLETED_WITH_FAILURES
  • Suites completed: 7
  • Suites remaining: 0
  • Stats: 1112 passed / 17 real failed / 141 skipped / 0 report-consistency errors
SuiteStatusPassedReal failedSkippedReport errorsReturn code
fs_perms_simplepassed180000
fsxpassed10000
fs_bindpassed10000
smoketestpassed120100
iopassed20000
fs-jfsfailed1017201
syscalls-jfsfailed1068013801

Performance

Note

Gate policy: report only, non-blocking for the full-chain result. Mark yellow/red vs baseline for human follow-up.

  • Status: failed
  • Gate mode: report_only

Metadata performance (this run)

ITEMVALUEAVG COSTP50 msP95 msP99 msMAX msSAMPLESERRORSBaseline VALUEVALUE deltaBaseline P99 msP99 deltaStatus
Create file9765.40 ops/s4.10 ms/op8.198.198.19175.36200000021668.74 ops/s-54.9%4.09+100.3%fail
Stat file21463.62 ops/s1.86 ms/op2.052.054.094.89200000063487.48 ops/s-66.2%2.05+99.8%fail
Open file21500.95 ops/s1.86 ms/op2.052.054.094.57200000063113.20 ops/s-65.9%2.05+99.8%fail
Rename file16828.16 ops/s2.38 ms/op4.094.094.095.44200000030314.48 ops/s-44.5%4.09+0.1%fail
Delete file16913.45 ops/s2.36 ms/op4.094.094.095.12200000031231.33 ops/s-45.8%4.09+0.1%fail

FIO read/write (this run)

ITEMSPEED GiB/sIOPSAVG COSTP50 msP95 msP99 msMAX msSAMPLESERRORSBaseline SPEED GiB/sSPEED deltaBaseline P99 msP99 deltaStatus
Sequential write 64KB1.6827547.719.07 ms/op9.1110.5511.4717.4126214401.71-1.7%10.94+4.8%pass
Sequential read 64KB2.1234652.216.90 ms/op6.3911.7314.4835.4826214402.28-7.2%12.78+13.3%pass
Random write 64KB1.6927678.608.96 ms/op8.459.7611.21568.5826214401.77-4.6%10.29+8.9%pass
Random read 64KB1.0617391.6313.86 ms/op13.7017.1719.2753.6226214401.12-5.2%19.53-1.3%pass
Sequential write 256KB2.6210726.0222.42 ms/op23.9932.9040.1150.026553602.73-4.1%36.44+10.1%pass
Sequential read 256KB2.6810986.7621.61 ms/op23.9927.9229.7550.556553603.11-13.8%26.08+14.1%pass
Random write 256KB2.409850.5924.21 ms/op22.4129.2332.90739.686553602.51-4.2%30.28+8.6%pass
Random read 256KB2.319456.8525.73 ms/op25.5631.3334.3448.196553602.37-2.6%34.340.0%pass
Sequential write 1MB3.173246.9370.65 ms/op69.73145.75183.50403.891638403.45-8.1%191.89-4.4%pass
Sequential read 1MB2.152206.30113.99 ms/op113.77128.45139.46212.101638402.72-20.8%108.53+28.5%fail
Random write 1MB2.852918.9479.93 ms/op76.02111.67354.42698.601638403.09-7.7%229.64+54.3%fail
Random read 1MB2.222270.82107.69 ms/op108.53122.16132.64198.921638402.15+3.1%135.27-1.9%pass

Failures and attribution

Failure analysis

g-daily-rustc-recursion-compile

  • Class: unknown_failure
  • Failure layer: build
  • Root-cause confidence: medium
  • Root cause: Rust compiler query recursion limit is exceeded while type-checking async test code in curvine-tests fallback_read_test and write_cache_test at 032c6dc5. Whether test refactor or toolchain limits is primary remains unproven.
  • Recommendation: Bisect when daily compilation last passed; add recursion_limit attributes or refactor async helpers in curvine-tests, or adjust CI toolchain settings if environment-specific.

g-integration-compile-missing-summary

  • Class: harness_failure
  • Failure layer: build
  • Root-cause confidence: high
  • Root cause: Integration profile compilation failed on curvine-tests recursion-limit overflow; orchestration exit 40 reflects missing test_summary.json artifact, not a separate product runtime defect.
  • Recommendation: Fix curvine-tests compilation (same build blocker as daily) so integration emits exactly one new test_summary.json after successful cargo test –no-run.

g-ltp-ftest-eio-worker-flush

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: high
  • Root cause: curvine-fuse read and readv paths return Input/output error errno=5 for ftest workloads on fs-jfs. Issue #1701 documents the defect; attributing it to any specific merged PR without bisect on 032c6dc5 would be speculative.
  • Recommendation: Investigate curvine-fuse read/readv errno=5 on fs-jfs ftest workloads under GitHub issue #1701; add regression coverage after root cause is confirmed by bisect.

g-ltp-lftest-eio-worker-flush

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: high
  • Root cause: Likely the same worker Running-frame flush regression as ftest read EIO; write-path symptom on lftest01.
  • Recommendation: Address alongside worker flush regression in issue #1701.

g-ltp-stream-eio-worker-flush

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: medium
  • Root cause: stream04 reports direct fread Input/output error; stream01/03/05 failures follow in the same fs-jfs run. Likely related to curvine-fuse I/O errors tracked under issue #1701.
  • Recommendation: Re-test stream cases after worker flush fix in issue #1701.

g-ltp-writetest-eio-worker-flush

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: medium
  • Root cause: Likely verify failure caused by incomplete writes from the worker flush regression.
  • Recommendation: Re-test writetest after worker flush fix in issue #1701.

g-ltp-inode-eio-worker-flush

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: high
  • Root cause: Likely the same worker flush regression manifesting on inode stress writes.
  • Recommendation: Re-test inode02 after worker flush fix in issue #1701.

g-ltp-fs-di-eio-worker-flush

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: medium
  • Root cause: Likely unable to create large test files because of the worker flush regression returning EIO on writes.
  • Recommendation: Re-test fs_di after worker flush fix in issue #1701.

g-ltp-fs-fill-eio-worker-flush

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: high
  • Root cause: Likely the same worker flush regression blocking loop-device file creation on curvine-fuse.
  • Recommendation: Re-test fs_fill after worker flush fix in issue #1701.

g-fuse-git-clone-eio-worker-flush

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: medium
  • Root cause: Git clone to curvine-fuse fails with index.lock write Input/output error while fio passed. Likely related to curvine-fuse write-path instability tracked under issue #1701.
  • Recommendation: Re-test git clone on curvine-fuse after worker flush fix in issue #1701.

Failed case summary

CaseSuite / PackageStatusKey errorRoot group
integration profileintegration / integrationFAILEDprofile failedg-integration-compile-missing-summary
daily profiledaily / dailyFAILEDprofile failedg-daily-rustc-recursion-compile
fusefuse / fuseFAILED“status”: “failed”g-fuse-git-clone-eio-worker-flush
inode02fs-jfs / ltpFAILEDinode02 1 TFAIL : inode02.c:851: Test failedg-ltp-inode-eio-worker-flush
stream01fs-jfs / ltpFAILEDstream01 1 TFAIL : stream01.c:92: bad contents in stream011.34403 / stream01 2 TFAIL : stream01.c:106: bad contents in stream012.34403 / stream01 3 TFAIL : stream01.c:115: Test failed.g-ltp-stream-eio-worker-flush
stream03fs-jfs / ltpFAILEDstream03 1 TFAIL : stream03.c:164: strlen(junk)=26: file pointer descrepancy 7 (pos=0) / stream03 2 TFAIL : stream03.c:175: Test failed in block0. / stream03 3 TFAIL : stream03.c:278: strlen(junk)=26: file pointer descrepancy 7 (opos=0)g-ltp-stream-eio-worker-flush
stream04fs-jfs / ltpFAILEDstream04 1 TFAIL : stream04.c:104: fread failed: Input/output errorg-ltp-stream-eio-worker-flush
stream05fs-jfs / ltpFAILEDstream05 2 TFAIL : stream05.c:115: read did not read right number / stream05 3 TFAIL : stream05.c:119: read returned bad values / stream05 4 TFAIL : stream05.c:125: Test failed in block1.g-ltp-stream-eio-worker-flush
ftest01fs-jfs / ltpFAILEDftest01 1 TFAIL : ftest01.c:343: Test[0]: read fail at ca800, errno = 5. / ftest01 1 TFAIL : ftest01.c:343: Test[1]: read fail at 80800, errno = 5. / ftest01 1 TFAIL : ftest01.c:343: Test[2]: read fail at a7000, errno = 5.g-ltp-ftest-eio-worker-flush
ftest02fs-jfs / ltpFAILEDftest02 1 TBROK : ftest02.c:413: Test[0]: error 5 on read / ftest02 2 TBROK : ftest02.c:413: Remaining cases broken / ftest02 1 TBROK : ftest02.c:413: Test[3]: error 5 on readg-ltp-ftest-eio-worker-flush
ftest03fs-jfs / ltpFAILEDftest03 1 TFAIL : ftest03.c:404: Test[0]: readv fail at 8f800, errno = 5. / ftest03 1 TFAIL : ftest03.c:404: Test[1]: readv fail at d8800, errno = 5. / ftest03 1 TFAIL : ftest03.c:404: Test[2]: readv fail at 87000, errno = 5.g-ltp-ftest-eio-worker-flush
ftest04fs-jfs / ltpFAILEDftest04 1 TFAIL : ftest04.c:327: Test[1]: readv fail at 7d800, errno = 5. / ftest04 1 TFAIL : ftest04.c:327: Test[2]: readv fail at b2800, errno = 5. / ftest04 1 TFAIL : ftest04.c:327: Test[0]: readv fail at 41000, errno = 5.g-ltp-ftest-eio-worker-flush
ftest05fs-jfs / ltpFAILEDftest05 1 TFAIL : ftest05.c:337: Test[0]: read fail at 46800: errno=EIO(5): Input/output error / ftest05 1 TFAIL : ftest05.c:337: Test[1]: read fail at 3e800: errno=EIO(5): Input/output error / ftest05 1 TFAIL : ftest05.c:337: Test[2]: read fail at e1800: errno=EIO(5): Input/output errorg-ltp-ftest-eio-worker-flush
ftest06fs-jfs / ltpFAILEDftest06 1 TFAIL : ftest06.c:431: Test[0]: error 5 on read / ftest06 1 TFAIL : ftest06.c:431: Test[1]: error 5 on read / ftest06 1 TFAIL : ftest06.c:431: Test[4]: error 5 on readg-ltp-ftest-eio-worker-flush
ftest07fs-jfs / ltpFAILEDftest07 1 TFAIL : ftest07.c:399: Test[0]: readv fail at 7800, errno = 5. / ftest07 1 TFAIL : ftest07.c:399: Test[1]: readv fail at 28800, errno = 5. / ftest07 1 TFAIL : ftest07.c:399: Test[2]: readv fail at 49800, errno = 5.g-ltp-ftest-eio-worker-flush
ftest08fs-jfs / ltpFAILEDftest08 1 TFAIL : ftest08.c:340: Test[3]: readv fail at 15800x, errno = 5. / ftest08 1 TFAIL : ftest08.c:340: Test[0]: readv fail at f5000x, errno = 5. / ftest08 1 TFAIL : ftest08.c:340: Test[1]: readv fail at 3f000x, errno = 5.g-ltp-ftest-eio-worker-flush
lftest01fs-jfs / ltpFAILEDlftest.c:55: TFAIL: write() failed: EIO (5)g-ltp-lftest-eio-worker-flush
writetest01fs-jfs / ltpFAILEDwritetest 2 TFAIL : writetest.c:253: Verify: Failureg-ltp-writetest-eio-worker-flush
fs_difs-jfs / ltpFAILEDfs_di 10 TFAIL : ltpapicmd.c:188: Test Failed: Could not create testfile of size 30Mbg-ltp-fs-di-eio-worker-flush
fs_fillfs-jfs / ltpFAILEDtst_device.c:335: TBROK: Failed to acquire deviceg-ltp-fs-fill-eio-worker-flush
writetestfs-jfs / ltpTFAILwritetest.c:253: Verify: Failureg-ltp-writetest-eio-worker-flush

Failed case reconciliation

ProfileReported failuresSource failuresDelta
integration110
daily101
fuse110
ltp18180

Common root-cause groups

g-daily-rustc-recursion-compile

  • Model class: unknown_failure; confidence: medium
  • Recommendation: Bisect when daily compilation last passed; add recursion_limit attributes or refactor async helpers in curvine-tests, or adjust CI toolchain settings if environment-specific.

g-integration-compile-missing-summary

  • Model class: harness_failure; confidence: high
  • Recommendation: Fix curvine-tests compilation (same build blocker as daily) so integration emits exactly one new test_summary.json after successful cargo test –no-run.

g-ltp-ftest-eio-worker-flush

  • Model class: pre_existing_failure; confidence: high
  • Recommendation: Investigate curvine-fuse read/readv errno=5 on fs-jfs ftest workloads under GitHub issue #1701; add regression coverage after root cause is confirmed by bisect.

g-ltp-lftest-eio-worker-flush

  • Model class: pre_existing_failure; confidence: high
  • Recommendation: Address alongside worker flush regression in issue #1701.

g-ltp-stream-eio-worker-flush

  • Model class: pre_existing_failure; confidence: medium
  • Recommendation: Re-test stream cases after worker flush fix in issue #1701.

g-ltp-writetest-eio-worker-flush

  • Model class: pre_existing_failure; confidence: medium
  • Recommendation: Re-test writetest after worker flush fix in issue #1701.

g-ltp-inode-eio-worker-flush

  • Model class: pre_existing_failure; confidence: high
  • Recommendation: Re-test inode02 after worker flush fix in issue #1701.

g-ltp-fs-di-eio-worker-flush

  • Model class: pre_existing_failure; confidence: medium
  • Recommendation: Re-test fs_di after worker flush fix in issue #1701.

g-ltp-fs-fill-eio-worker-flush

  • Model class: pre_existing_failure; confidence: high
  • Recommendation: Re-test fs_fill after worker flush fix in issue #1701.

g-fuse-git-clone-eio-worker-flush

  • Model class: pre_existing_failure; confidence: medium
  • Recommendation: Re-test git clone on curvine-fuse after worker flush fix in issue #1701.

Follow-up

Defects and fixes

  • GitHub Issue: 1701 (open, reused)

Risks

  • A subset of green profiles does not override a full-chain NO-GO.
  • Report-only performance regressions still require human follow-up.

Next actions

  • Bisect when daily compilation last passed; add recursion_limit attributes or refactor async helpers in curvine-tests, or adjust CI toolchain settings if environment-specific.
  • Fix curvine-tests compilation (same build blocker as daily) so integration emits exactly one new test_summary.json after successful cargo test –no-run.
  • Investigate curvine-fuse read/readv errno=5 on fs-jfs ftest workloads under GitHub issue #1701; add regression coverage after root cause is confirmed by bisect.
  • Address alongside worker flush regression in issue #1701.
  • Re-test stream cases after worker flush fix in issue #1701.
  • Re-test writetest after worker flush fix in issue #1701.
  • Re-test inode02 after worker flush fix in issue #1701.
  • Re-test fs_di after worker flush fix in issue #1701.
  • Re-test fs_fill after worker flush fix in issue #1701.
  • Re-test git clone on curvine-fuse after worker flush fix in issue #1701.
  • Rerun the full chain after fixes and require all release gates to pass.

Publication note

This public report contains only sanitized test outcomes. Private execution evidence remains internal.