Skip to content
2026-09-30 full-chain

Curvine full-chain daily test report - 2026-09-30

Quality conclusion

Executive summary

Caution

Release decision: NO-GO. Pipeline FAIL; ran 7 profiles, 2 passed, 4 failed.

Quality gates

GateCriterionActualVerdict
Full-chain resultAll required profiles passed2/7 passedFAIL
Failure attributionFailures classified10 root-cause groupsPASS
Resource cleanupAll profile cleanups succeeded7/7PASS

Conclusion

This full-chain run did not pass. Failed profiles require resolution before release.

Test results

Profile summary

ProfilePreflightResultDurationClassCleanup
fastPASSPASS12m 39spassedPASSED
integrationPASSFAIL5m 15sharness_failurePASSED
dailyPASSFAIL9m 28sunknown_failurePASSED
fusePASSFAIL4m 35sunknown_failurePASSED
ltpPASSFAIL37m 58sunknown_failurePASSED
csiPASSPASS0m 36spassedPASSED
perf-benchmarkNOT_RECORDEDFAILED6m 31sfailedPASSED

LTP

  • Status: COMPLETED_WITH_FAILURES
  • Suites completed: 7
  • Suites remaining: 0
  • Stats: 1112 passed / 17 real failed / 141 skipped / 0 report-consistency errors
SuiteStatusPassedReal failedSkippedReport errorsReturn code
fs_perms_simplepassed180000
fsxpassed10000
fs_bindpassed10000
smoketestpassed120100
iopassed20000
fs-jfsfailed1017201
syscalls-jfsfailed1068013801

Performance

Note

Gate policy: report only, non-blocking for the full-chain result. Mark yellow/red vs baseline for human follow-up.

  • Status: failed
  • Gate mode: report_only

Metadata performance (this run)

ITEMVALUEAVG COSTP50 msP95 msP99 msMAX msSAMPLESERRORSBaseline VALUEVALUE deltaBaseline P99 msP99 deltaStatus
Create file9777.27 ops/s4.09 ms/op6.756.756.756.75200000021668.74 ops/s-54.9%4.09+65.0%fail
Stat file21571.14 ops/s1.85 ms/op2.052.054.094.32200000063487.48 ops/s-66.0%2.05+99.8%fail
Open file22135.24 ops/s1.81 ms/op2.052.054.094.27200000063113.20 ops/s-64.9%2.05+99.8%fail
Rename file16869.38 ops/s2.37 ms/op4.094.094.095.00200000030314.48 ops/s-44.4%4.09+0.1%fail
Delete file16046.80 ops/s2.49 ms/op4.094.094.095.59200000031231.33 ops/s-48.6%4.09+0.1%fail

FIO read/write (this run)

ITEMSPEED GiB/sIOPSAVG COSTP50 msP95 msP99 msMAX msSAMPLESERRORSBaseline SPEED GiB/sSPEED deltaBaseline P99 msP99 deltaStatus
Sequential write 64KB1.7127991.888.96 ms/op8.9810.4211.2117.7726214401.71-0.1%10.94+2.4%pass
Sequential read 64KB2.1134492.636.93 ms/op6.4611.8614.3526.5426214402.28-7.7%12.78+12.3%pass
Random write 64KB1.7228121.008.82 ms/op8.459.7611.21472.6626214401.77-3.0%10.29+8.9%pass
Random read 64KB1.1018076.4013.37 ms/op13.3016.5818.7437.9426214401.12-1.5%19.53-4.0%pass
Sequential write 256KB2.8811795.5420.54 ms/op22.4131.0635.3951.136553602.73+5.5%36.44-2.9%pass
Sequential read 256KB2.379690.3725.07 ms/op26.6129.7531.3348.486553603.11-23.9%26.08+20.1%fail
Random write 256KB2.5210325.5122.76 ms/op21.1028.1831.85706.726553602.51+0.4%30.28+5.2%pass
Random read 256KB2.449984.1624.74 ms/op24.5129.7532.9046.816553602.37+2.8%34.34-4.2%pass
Sequential write 1MB3.213286.6668.72 ms/op60.03162.53214.96339.061638403.45-7.0%191.89+12.0%pass
Sequential read 1MB2.222269.88109.86 ms/op112.72121.11125.30216.551638402.72-18.5%108.53+15.5%degraded
Random write 1MB2.943011.2175.00 ms/op69.73112.72421.53794.871638403.09-4.8%229.64+83.6%fail
Random read 1MB2.122173.81112.27 ms/op113.77126.35131.60191.601638402.15-1.3%135.27-2.7%pass

Failures and attribution

Failure analysis

g-daily-rustc-recursion-compile

  • Class: unknown_failure
  • Failure layer: build
  • Root-cause confidence: medium
  • Root cause: Rust compiler query recursion limit is exceeded while type-checking async test code in curvine-tests fallback_read_test and write_cache_test at 032c6dc5. Whether test refactor or toolchain limits is primary remains unproven.
  • Recommendation: Bisect when daily compilation last passed; add recursion_limit attributes or refactor async helpers in curvine-tests, or adjust CI toolchain settings if environment-specific.

g-integration-compile-missing-summary

  • Class: harness_failure
  • Failure layer: build
  • Root-cause confidence: high
  • Root cause: Integration profile compilation failed on curvine-tests recursion-limit overflow; orchestration exit 40 reflects missing test_summary.json artifact, not a separate product runtime defect.
  • Recommendation: Fix curvine-tests compilation (same build blocker as daily) so integration emits exactly one new test_summary.json after successful cargo test –no-run.

g-ltp-ftest-eio-worker-flush

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: high
  • Root cause: curvine-fuse read and readv paths return Input/output error errno=5 for ftest workloads on fs-jfs. Issue #1701 documents the defect; attributing it to any specific merged PR without bisect on 032c6dc5 would be speculative.
  • Recommendation: Investigate curvine-fuse read/readv errno=5 on fs-jfs ftest workloads under GitHub issue #1701; add regression coverage after root cause is confirmed by bisect.

g-ltp-lftest-eio-worker-flush

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: high
  • Root cause: Likely the same worker Running-frame flush regression as ftest read EIO; write-path symptom on lftest01.
  • Recommendation: Address alongside worker flush regression in issue #1701.

g-ltp-stream-eio-worker-flush

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: medium
  • Root cause: stream04 reports direct fread Input/output error; stream01/03/05 failures follow in the same fs-jfs run. Likely related to curvine-fuse I/O errors tracked under issue #1701.
  • Recommendation: Re-test stream cases after worker flush fix in issue #1701.

g-ltp-writetest-eio-worker-flush

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: medium
  • Root cause: Likely verify failure caused by incomplete writes from the worker flush regression.
  • Recommendation: Re-test writetest after worker flush fix in issue #1701.

g-ltp-inode-eio-worker-flush

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: high
  • Root cause: Likely the same worker flush regression manifesting on inode stress writes.
  • Recommendation: Re-test inode02 after worker flush fix in issue #1701.

g-ltp-fs-di-eio-worker-flush

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: medium
  • Root cause: Likely unable to create large test files because of the worker flush regression returning EIO on writes.
  • Recommendation: Re-test fs_di after worker flush fix in issue #1701.

g-ltp-fs-fill-eio-worker-flush

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: high
  • Root cause: Likely the same worker flush regression blocking loop-device file creation on curvine-fuse.
  • Recommendation: Re-test fs_fill after worker flush fix in issue #1701.

g-fuse-git-clone-eio-worker-flush

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: medium
  • Root cause: Git clone to curvine-fuse fails with index.lock write Input/output error while most other fuse sub-tests passed. Likely related to curvine-fuse write-path instability tracked under issue #1701.
  • Recommendation: Re-test git clone on curvine-fuse after worker flush fix in issue #1701.

Failed case summary

CaseSuite / PackageStatusKey errorRoot group
integration profileintegration / integrationFAILEDprofile failedg-integration-compile-missing-summary
daily profiledaily / dailyFAILEDprofile failedg-daily-rustc-recursion-compile
fusefuse / fuseFAILED“status”: “failed”g-fuse-git-clone-eio-worker-flush
inode02fs-jfs / ltpFAILEDinode02 1 TFAIL : inode02.c:851: Test failedg-ltp-inode-eio-worker-flush
stream01fs-jfs / ltpFAILEDstream01 1 TFAIL : stream01.c:92: bad contents in stream011.34409 / stream01 2 TFAIL : stream01.c:106: bad contents in stream012.34409 / stream01 3 TFAIL : stream01.c:115: Test failed.g-ltp-stream-eio-worker-flush
stream03fs-jfs / ltpFAILEDstream03 1 TFAIL : stream03.c:164: strlen(junk)=26: file pointer descrepancy 7 (pos=0) / stream03 2 TFAIL : stream03.c:175: Test failed in block0. / stream03 3 TFAIL : stream03.c:278: strlen(junk)=26: file pointer descrepancy 7 (opos=0)g-ltp-stream-eio-worker-flush
stream04fs-jfs / ltpFAILEDstream04 1 TFAIL : stream04.c:104: fread failed: Input/output errorg-ltp-stream-eio-worker-flush
stream05fs-jfs / ltpFAILEDstream05 2 TFAIL : stream05.c:115: read did not read right number / stream05 3 TFAIL : stream05.c:119: read returned bad values / stream05 4 TFAIL : stream05.c:125: Test failed in block1.g-ltp-stream-eio-worker-flush
ftest01fs-jfs / ltpFAILEDftest01 1 TFAIL : ftest01.c:343: Test[0]: read fail at 4800, errno = 5. / ftest01 1 TFAIL : ftest01.c:343: Test[1]: read fail at 9c800, errno = 5. / ftest01 1 TFAIL : ftest01.c:343: Test[2]: read fail at b000, errno = 5.g-ltp-ftest-eio-worker-flush
ftest02fs-jfs / ltpFAILEDftest02 1 TBROK : ftest02.c:413: Test[3]: error 5 on read / ftest02 2 TBROK : ftest02.c:413: Remaining cases broken / ftest02 1 TBROK : ftest02.c:413: Test[0]: error 5 on readg-ltp-ftest-eio-worker-flush
ftest03fs-jfs / ltpFAILEDftest03 1 TFAIL : ftest03.c:404: Test[0]: readv fail at ce000, errno = 5. / ftest03 1 TFAIL : ftest03.c:404: Test[1]: readv fail at 29000, errno = 5. / ftest03 1 TFAIL : ftest03.c:404: Test[2]: readv fail at e8000, errno = 5.g-ltp-ftest-eio-worker-flush
ftest04fs-jfs / ltpFAILEDftest04 1 TFAIL : ftest04.c:327: Test[2]: readv fail at fb000, errno = 5. / ftest04 1 TFAIL : ftest04.c:327: Test[1]: readv fail at a800, errno = 5. / ftest04 1 TFAIL : ftest04.c:327: Test[3]: readv fail at 65800, errno = 5.g-ltp-ftest-eio-worker-flush
ftest05fs-jfs / ltpFAILEDftest05 1 TFAIL : ftest05.c:337: Test[0]: read fail at 48800: errno=EIO(5): Input/output error / ftest05 1 TFAIL : ftest05.c:337: Test[1]: read fail at 39000: errno=EIO(5): Input/output error / ftest05 1 TFAIL : ftest05.c:337: Test[2]: read fail at 2c000: errno=EIO(5): Input/output errorg-ltp-ftest-eio-worker-flush
ftest06fs-jfs / ltpFAILEDftest06 1 TFAIL : ftest06.c:431: Test[1]: error 5 on read / ftest06 1 TFAIL : ftest06.c:431: Test[2]: error 5 on read / ftest06 1 TFAIL : ftest06.c:431: Test[0]: error 5 on readg-ltp-ftest-eio-worker-flush
ftest07fs-jfs / ltpFAILEDftest07 1 TFAIL : ftest07.c:399: Test[0]: readv fail at d7800, errno = 5. / ftest07 1 TFAIL : ftest07.c:399: Test[1]: readv fail at 49000, errno = 5. / ftest07 1 TFAIL : ftest07.c:399: Test[2]: readv fail at 87000, errno = 5.g-ltp-ftest-eio-worker-flush
ftest08fs-jfs / ltpFAILEDftest08 1 TFAIL : ftest08.c:340: Test[1]: readv fail at 4b800x, errno = 5. / ftest08 1 TFAIL : ftest08.c:340: Test[2]: readv fail at c4000x, errno = 5. / ftest08 1 TFAIL : ftest08.c:340: Test[3]: readv fail at e2800x, errno = 5.g-ltp-ftest-eio-worker-flush
lftest01fs-jfs / ltpFAILEDlftest.c:55: TFAIL: write() failed: EIO (5)g-ltp-lftest-eio-worker-flush
writetest01fs-jfs / ltpFAILEDwritetest 2 TFAIL : writetest.c:253: Verify: Failureg-ltp-writetest-eio-worker-flush
fs_difs-jfs / ltpFAILEDfs_di 10 TFAIL : ltpapicmd.c:188: Test Failed: Could not create testfile of size 30Mbg-ltp-fs-di-eio-worker-flush
fs_fillfs-jfs / ltpFAILEDtst_device.c:335: TBROK: Failed to acquire deviceg-ltp-fs-fill-eio-worker-flush
writetestfs-jfs / ltpTFAILwritetest.c:253: Verify: Failureg-ltp-writetest-eio-worker-flush

Failed case reconciliation

ProfileReported failuresSource failuresDelta
integration110
daily101
fuse110
ltp18180

Common root-cause groups

g-daily-rustc-recursion-compile

  • Model class: unknown_failure; confidence: medium
  • Recommendation: Bisect when daily compilation last passed; add recursion_limit attributes or refactor async helpers in curvine-tests, or adjust CI toolchain settings if environment-specific.

g-integration-compile-missing-summary

  • Model class: harness_failure; confidence: high
  • Recommendation: Fix curvine-tests compilation (same build blocker as daily) so integration emits exactly one new test_summary.json after successful cargo test –no-run.

g-ltp-ftest-eio-worker-flush

  • Model class: pre_existing_failure; confidence: high
  • Recommendation: Investigate curvine-fuse read/readv errno=5 on fs-jfs ftest workloads under GitHub issue #1701; add regression coverage after root cause is confirmed by bisect.

g-ltp-lftest-eio-worker-flush

  • Model class: pre_existing_failure; confidence: high
  • Recommendation: Address alongside worker flush regression in issue #1701.

g-ltp-stream-eio-worker-flush

  • Model class: pre_existing_failure; confidence: medium
  • Recommendation: Re-test stream cases after worker flush fix in issue #1701.

g-ltp-writetest-eio-worker-flush

  • Model class: pre_existing_failure; confidence: medium
  • Recommendation: Re-test writetest after worker flush fix in issue #1701.

g-ltp-inode-eio-worker-flush

  • Model class: pre_existing_failure; confidence: high
  • Recommendation: Re-test inode02 after worker flush fix in issue #1701.

g-ltp-fs-di-eio-worker-flush

  • Model class: pre_existing_failure; confidence: medium
  • Recommendation: Re-test fs_di after worker flush fix in issue #1701.

g-ltp-fs-fill-eio-worker-flush

  • Model class: pre_existing_failure; confidence: high
  • Recommendation: Re-test fs_fill after worker flush fix in issue #1701.

g-fuse-git-clone-eio-worker-flush

  • Model class: pre_existing_failure; confidence: medium
  • Recommendation: Re-test git clone on curvine-fuse after worker flush fix in issue #1701.

Follow-up

Defects and fixes

  • GitHub Issue: 1701 (open, reused)

Risks

  • A subset of green profiles does not override a full-chain NO-GO.
  • Report-only performance regressions still require human follow-up.

Next actions

  • Bisect when daily compilation last passed; add recursion_limit attributes or refactor async helpers in curvine-tests, or adjust CI toolchain settings if environment-specific.
  • Fix curvine-tests compilation (same build blocker as daily) so integration emits exactly one new test_summary.json after successful cargo test –no-run.
  • Investigate curvine-fuse read/readv errno=5 on fs-jfs ftest workloads under GitHub issue #1701; add regression coverage after root cause is confirmed by bisect.
  • Address alongside worker flush regression in issue #1701.
  • Re-test stream cases after worker flush fix in issue #1701.
  • Re-test writetest after worker flush fix in issue #1701.
  • Re-test inode02 after worker flush fix in issue #1701.
  • Re-test fs_di after worker flush fix in issue #1701.
  • Re-test fs_fill after worker flush fix in issue #1701.
  • Re-test git clone on curvine-fuse after worker flush fix in issue #1701.
  • Rerun the full chain after fixes and require all release gates to pass.

Publication note

This public report contains only sanitized test outcomes. Private execution evidence remains internal.