Skip to content
2026-09-20 full-chain

Curvine full-chain daily test report - 2026-09-20

Quality conclusion

Executive summary

Caution

Release decision: NO-GO. Pipeline FAIL; ran 7 profiles, 3 passed, 3 failed.

Quality gates

GateCriterionActualVerdict
Full-chain resultAll required profiles passed3/7 passedFAIL
Failure attributionFailures classified8 root-cause groupsPASS
Resource cleanupAll profile cleanups succeeded7/7PASS

Conclusion

This full-chain run did not pass. Failed profiles require resolution before release.

Test results

Profile summary

ProfilePreflightResultDurationClassCleanup
fastPASSPASS12m 16spassedPASSED
integrationPASSFAIL5m 13sharness_failurePASSED
dailyPASSFAIL9m 18sunknown_failurePASSED
fusePASSPASS5m 51spassedPASSED
ltpPASSFAIL37m 31sunknown_failurePASSED
csiPASSPASS0m 37spassedPASSED
perf-benchmarkNOT_RECORDEDFAILED5m 38sfailedPASSED

LTP

  • Status: COMPLETED_WITH_FAILURES
  • Suites completed: 7
  • Suites remaining: 0
  • Stats: 1112 passed / 17 real failed / 141 skipped / 0 report-consistency errors
SuiteStatusPassedReal failedSkippedReport errorsReturn code
fs_perms_simplepassed180000
fsxpassed10000
fs_bindpassed10000
smoketestpassed120100
iopassed20000
fs-jfsfailed1017201
syscalls-jfsfailed1068013801

Performance

Note

Gate policy: report only, non-blocking for the full-chain result. Mark yellow/red vs baseline for human follow-up.

  • Status: failed
  • Gate mode: report_only

Metadata performance (this run)

ITEMVALUEAVG COSTP50 msP95 msP99 msMAX msSAMPLESERRORSBaseline VALUEVALUE deltaBaseline P99 msP99 deltaStatus
Create file24303.01 ops/s1.64 ms/op2.054.094.094.52200000021668.74 ops/s+12.2%4.09+0.1%pass
Stat file71507.28 ops/s0.56 ms/op1.021.021.022.78200000063487.48 ops/s+12.6%2.05-50.1%pass
Open file69299.76 ops/s0.57 ms/op1.021.021.022.93200000063113.20 ops/s+9.8%2.05-50.1%pass
Rename file33230.68 ops/s1.19 ms/op2.053.713.713.71200000030314.48 ops/s+9.6%4.09-9.2%pass
Delete file33997.68 ops/s1.17 ms/op2.054.094.094.94200000031231.33 ops/s+8.9%4.09+0.1%pass

FIO read/write (this run)

ITEMSPEED GiB/sIOPSAVG COSTP50 msP95 msP99 msMAX msSAMPLESERRORSBaseline SPEED GiB/sSPEED deltaBaseline P99 msP99 deltaStatus
Sequential write 64KB1.7027801.899.01 ms/op8.9810.4211.2118.1426214401.71-0.8%10.94+2.4%pass
Sequential read 64KB2.3638618.746.25 ms/op5.8710.4212.7824.5526214402.28+3.4%12.780.0%pass
Random write 64KB1.7528671.558.67 ms/op8.459.7610.81290.9126214401.77-1.1%10.29+5.1%pass
Random read 64KB1.0917897.4513.53 ms/op13.4316.7119.0142.4626214401.12-2.5%19.53-2.7%pass
Sequential write 256KB2.6810962.8621.86 ms/op23.2032.9037.4946.976553602.73-2.0%36.44+2.9%pass
Sequential read 256KB2.6510837.7721.87 ms/op23.9927.6629.2348.946553603.11-14.9%26.08+12.1%pass
Random write 256KB2.6010656.2622.90 ms/op22.1527.6630.54462.946553602.51+3.7%30.28+0.9%pass
Random read 256KB2.359643.3225.07 ms/op25.0330.2833.8251.296553602.37-0.7%34.34-1.5%pass
Sequential write 1MB3.073140.5071.86 ms/op67.63158.33219.15348.271638403.45-11.1%191.89+14.2%pass
Sequential read 1MB2.432492.6296.48 ms/op105.38113.77117.96210.851638402.72-10.5%108.53+8.7%pass
Random write 1MB2.852921.5479.98 ms/op74.97122.16291.50497.011638403.09-7.7%229.64+26.9%fail
Random read 1MB2.202253.34110.32 ms/op111.67123.21129.50178.981638402.15+2.3%135.27-4.3%pass

Failures and attribution

Failure analysis

g-rustc-recursion-compile

  • Class: unknown_failure
  • Failure layer: build
  • Root-cause confidence: medium
  • Root cause: Rust compiler query recursion limit is exceeded while type-checking async test code in fallback_read_test and write_cache_test. Whether increased type complexity in curvine-tests or rustc 1.98 limits is primary remains unproven.
  • Recommendation: Bisect commits between last passing daily compilation SHA and current main; add recursion_limit attributes or refactor async test helpers if product-side, or adjust toolchain settings if environment-specific.

g-ltp-ftest-eio-worker-flush

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: high
  • Root cause: The worker Running-frame flush change in write_handler.rs (PR #1694) likely regressed read/readv durability on curvine-fuse for ftest workloads. Failure persists on de580be3; narrowed bisect against LTP-clean SHA 04d988e3 is still required.
  • Recommendation: Review flush-only Running frame handling in curvine-worker/src/worker/handler/write_handler.rs from PR #1694; add ftest read/readv regression coverage on curvine-fuse.

g-ltp-lftest-eio-worker-flush

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: high
  • Root cause: Likely the same worker Running-frame flush regression as ftest read EIO; write-path symptom on lftest01.
  • Recommendation: Address alongside PR #1694 write_handler flush regression in issue #1701.

g-ltp-stream-eio-worker-flush

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: medium
  • Root cause: Likely cascade from the worker flush regression affecting stream I/O on curvine-fuse.
  • Recommendation: Re-test stream cases after worker flush fix in issue #1701.

g-ltp-writetest-eio-worker-flush

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: medium
  • Root cause: Likely verify failure caused by incomplete writes from the worker flush regression.
  • Recommendation: Re-test writetest after worker flush fix in issue #1701.

g-ltp-inode-eio-worker-flush

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: high
  • Root cause: Likely the same worker flush regression manifesting on inode stress writes.
  • Recommendation: Re-test inode02 after worker flush fix in issue #1701.

g-ltp-fs-di-eio-worker-flush

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: medium
  • Root cause: Likely unable to create large test files because of the worker flush regression returning EIO on writes.
  • Recommendation: Re-test fs_di after worker flush fix in issue #1701.

g-ltp-fs-fill-eio-worker-flush

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: high
  • Root cause: Likely the same worker flush regression blocking loop-device file creation on curvine-fuse.
  • Recommendation: Re-test fs_fill after worker flush fix in issue #1701.

Failed case summary

CaseSuite / PackageStatusKey errorRoot group
integration profileintegration / integrationFAILEDprofile failedg-rustc-recursion-compile
daily profiledaily / dailyFAILEDprofile failedg-rustc-recursion-compile
inode02fs-jfs / ltpFAILEDinode02 1 TFAIL : inode02.c:851: Test failedg-ltp-inode-eio-worker-flush
stream01fs-jfs / ltpFAILEDstream01 1 TFAIL : stream01.c:92: bad contents in stream011.34412 / stream01 2 TFAIL : stream01.c:106: bad contents in stream012.34412 / stream01 3 TFAIL : stream01.c:115: Test failed.g-ltp-stream-eio-worker-flush
stream03fs-jfs / ltpFAILEDstream03 1 TFAIL : stream03.c:164: strlen(junk)=26: file pointer descrepancy 7 (pos=0) / stream03 2 TFAIL : stream03.c:175: Test failed in block0. / stream03 3 TFAIL : stream03.c:278: strlen(junk)=26: file pointer descrepancy 7 (opos=0)g-ltp-stream-eio-worker-flush
stream04fs-jfs / ltpFAILEDstream04 1 TFAIL : stream04.c:104: fread failed: Input/output errorg-ltp-stream-eio-worker-flush
stream05fs-jfs / ltpFAILEDstream05 2 TFAIL : stream05.c:115: read did not read right number / stream05 3 TFAIL : stream05.c:119: read returned bad values / stream05 4 TFAIL : stream05.c:125: Test failed in block1.g-ltp-stream-eio-worker-flush
ftest01fs-jfs / ltpFAILEDftest01 1 TFAIL : ftest01.c:343: Test[0]: read fail at 9800, errno = 5. / ftest01 1 TFAIL : ftest01.c:343: Test[1]: read fail at f4800, errno = 5. / ftest01 1 TFAIL : ftest01.c:343: Test[2]: read fail at 6a800, errno = 5.g-ltp-ftest-eio-worker-flush
ftest02fs-jfs / ltpFAILEDftest02 1 TBROK : ftest02.c:413: Test[0]: error 5 on read / ftest02 2 TBROK : ftest02.c:413: Remaining cases broken / ftest02 1 TBROK : ftest02.c:413: Test[3]: error 5 on readg-ltp-ftest-eio-worker-flush
ftest03fs-jfs / ltpFAILEDftest03 1 TFAIL : ftest03.c:404: Test[0]: readv fail at b6000, errno = 5. / ftest03 1 TFAIL : ftest03.c:404: Test[1]: readv fail at 8e000, errno = 5. / ftest03 1 TFAIL : ftest03.c:404: Test[2]: readv fail at a7000, errno = 5.g-ltp-ftest-eio-worker-flush
ftest04fs-jfs / ltpFAILEDftest04 1 TFAIL : ftest04.c:327: Test[0]: readv fail at 64000, errno = 5. / ftest04 1 TFAIL : ftest04.c:327: Test[2]: readv fail at 3f800, errno = 5. / ftest04 1 TFAIL : ftest04.c:327: Test[3]: readv fail at a4000, errno = 5.g-ltp-ftest-eio-worker-flush
ftest05fs-jfs / ltpFAILEDftest05 1 TFAIL : ftest05.c:337: Test[0]: read fail at 86000: errno=EIO(5): Input/output error / ftest05 1 TFAIL : ftest05.c:337: Test[1]: read fail at 26800: errno=EIO(5): Input/output error / ftest05 1 TFAIL : ftest05.c:337: Test[2]: read fail at 68800: errno=EIO(5): Input/output errorg-ltp-ftest-eio-worker-flush
ftest06fs-jfs / ltpFAILEDftest06 1 TFAIL : ftest06.c:431: Test[2]: error 5 on read / ftest06 1 TFAIL : ftest06.c:431: Test[0]: error 5 on read / ftest06 1 TFAIL : ftest06.c:431: Test[1]: error 5 on readg-ltp-ftest-eio-worker-flush
ftest07fs-jfs / ltpFAILEDftest07 1 TFAIL : ftest07.c:399: Test[0]: readv fail at e3000, errno = 5. / ftest07 1 TFAIL : ftest07.c:399: Test[1]: readv fail at 88800, errno = 5. / ftest07 1 TFAIL : ftest07.c:399: Test[2]: readv fail at dc000, errno = 5.g-ltp-ftest-eio-worker-flush
ftest08fs-jfs / ltpFAILEDftest08 1 TFAIL : ftest08.c:430: Test[0]: writev fail at e1000x xfr -1, errno = 5. / ftest08 1 TFAIL : ftest08.c:340: Test[2]: readv fail at 35800x, errno = 5. / ftest08 1 TFAIL : ftest08.c:340: Test[3]: readv fail at 83800x, errno = 5.g-ltp-ftest-eio-worker-flush
lftest01fs-jfs / ltpFAILEDlftest.c:55: TFAIL: write() failed: EIO (5)g-ltp-lftest-eio-worker-flush
writetest01fs-jfs / ltpFAILEDwritetest 2 TFAIL : writetest.c:253: Verify: Failureg-ltp-writetest-eio-worker-flush
fs_difs-jfs / ltpFAILEDfs_di 10 TFAIL : ltpapicmd.c:188: Test Failed: Could not create testfile of size 30Mbg-ltp-fs-di-eio-worker-flush
fs_fillfs-jfs / ltpFAILEDtst_device.c:335: TBROK: Failed to acquire deviceg-ltp-fs-fill-eio-worker-flush
writetestfs-jfs / ltpTFAILwritetest.c:253: Verify: Failureg-ltp-writetest-eio-worker-flush

Failed case reconciliation

ProfileReported failuresSource failuresDelta
integration110
daily101
ltp18180

Common root-cause groups

g-rustc-recursion-compile

  • Model class: unknown_failure; confidence: medium
  • Recommendation: Bisect commits between last passing daily compilation SHA and current main; add recursion_limit attributes or refactor async test helpers if product-side, or adjust toolchain settings if environment-specific.

g-ltp-ftest-eio-worker-flush

  • Model class: pre_existing_failure; confidence: high
  • Recommendation: Review flush-only Running frame handling in curvine-worker/src/worker/handler/write_handler.rs from PR #1694; add ftest read/readv regression coverage on curvine-fuse.

g-ltp-lftest-eio-worker-flush

  • Model class: pre_existing_failure; confidence: high
  • Recommendation: Address alongside PR #1694 write_handler flush regression in issue #1701.

g-ltp-stream-eio-worker-flush

  • Model class: pre_existing_failure; confidence: medium
  • Recommendation: Re-test stream cases after worker flush fix in issue #1701.

g-ltp-writetest-eio-worker-flush

  • Model class: pre_existing_failure; confidence: medium
  • Recommendation: Re-test writetest after worker flush fix in issue #1701.

g-ltp-inode-eio-worker-flush

  • Model class: pre_existing_failure; confidence: high
  • Recommendation: Re-test inode02 after worker flush fix in issue #1701.

g-ltp-fs-di-eio-worker-flush

  • Model class: pre_existing_failure; confidence: medium
  • Recommendation: Re-test fs_di after worker flush fix in issue #1701.

g-ltp-fs-fill-eio-worker-flush

  • Model class: pre_existing_failure; confidence: high
  • Recommendation: Re-test fs_fill after worker flush fix in issue #1701.

Follow-up

Defects and fixes

  • GitHub Issue: 1701 (open, reused)

Risks

  • A subset of green profiles does not override a full-chain NO-GO.
  • Report-only performance regressions still require human follow-up.

Next actions

  • Bisect commits between last passing daily compilation SHA and current main; add recursion_limit attributes or refactor async test helpers if product-side, or adjust toolchain settings if environment-specific.
  • Review flush-only Running frame handling in curvine-worker/src/worker/handler/write_handler.rs from PR #1694; add ftest read/readv regression coverage on curvine-fuse.
  • Address alongside PR #1694 write_handler flush regression in issue #1701.
  • Re-test stream cases after worker flush fix in issue #1701.
  • Re-test writetest after worker flush fix in issue #1701.
  • Re-test inode02 after worker flush fix in issue #1701.
  • Re-test fs_di after worker flush fix in issue #1701.
  • Re-test fs_fill after worker flush fix in issue #1701.
  • Rerun the full chain after fixes and require all release gates to pass.

Publication note

This public report contains only sanitized test outcomes. Private execution evidence remains internal.