Skip to content
2026-09-22 full-chain

Curvine full-chain daily test report - 2026-09-22

Quality conclusion

Executive summary

Caution

Release decision: NO-GO. Pipeline FAIL; ran 7 profiles, 2 passed, 4 failed.

Quality gates

GateCriterionActualVerdict
Full-chain resultAll required profiles passed2/7 passedFAIL
Failure attributionFailures classified9 root-cause groupsPASS
Resource cleanupAll profile cleanups succeeded7/7PASS

Conclusion

This full-chain run did not pass. Failed profiles require resolution before release.

Test results

Profile summary

ProfilePreflightResultDurationClassCleanup
fastPASSPASS12m 14spassedPASSED
integrationPASSFAIL5m 09sharness_failurePASSED
dailyPASSFAIL9m 19sunknown_failurePASSED
fusePASSFAIL4m 22sunknown_failurePASSED
ltpPASSFAIL38m 16sunknown_failurePASSED
csiPASSPASS0m 34spassedPASSED
perf-benchmarkNOT_RECORDEDFAILED2m 05sfailedPASSED

LTP

  • Status: COMPLETED_WITH_FAILURES
  • Suites completed: 7
  • Suites remaining: 0
  • Stats: 1112 passed / 17 real failed / 141 skipped / 0 report-consistency errors
SuiteStatusPassedReal failedSkippedReport errorsReturn code
fs_perms_simplepassed180000
fsxpassed10000
fs_bindpassed10000
smoketestpassed120100
iopassed20000
fs-jfsfailed1017201
syscalls-jfsfailed1068013801

Performance

Note

Gate policy: report only, non-blocking for the full-chain result. Mark yellow/red vs baseline for human follow-up.

  • Status: failed
  • Gate mode: report_only

Metadata performance (this run)

ITEMVALUEAVG COSTP50 msP95 msP99 msMAX msSAMPLESERRORSBaseline VALUEVALUE deltaBaseline P99 msP99 deltaStatus
Create file23039.81 ops/s1.73 ms/op2.054.094.09166.50200000021668.74 ops/s+6.3%4.09+0.1%pass
Stat file70291.07 ops/s0.57 ms/op1.021.021.022.64200000063487.48 ops/s+10.7%2.05-50.1%pass
Open file70153.61 ops/s0.57 ms/op1.021.021.021.70200000063113.20 ops/s+11.2%2.05-50.1%pass
Rename file33258.20 ops/s1.19 ms/op2.054.094.094.39200000030314.48 ops/s+9.7%4.09+0.1%pass
Delete file32202.40 ops/s1.23 ms/op2.054.094.095.46200000031231.33 ops/s+3.1%4.09+0.1%pass

FIO read/write (this run)

ITEMSPEED GiB/sIOPSAVG COSTP50 msP95 msP99 msMAX msSAMPLESERRORSBaseline SPEED GiB/sSPEED deltaBaseline P99 msP99 deltaStatus
Sequential write 64KB1.6927719.579.01 ms/op8.9810.4211.2118.4126214401.71-1.1%10.94+2.4%pass
Sequential read 64KB2.1334864.216.75 ms/op6.3211.2113.4324.5026214402.28-6.7%12.78+5.1%pass
Random write 64KB1.7528608.978.66 ms/op8.4510.4212.3925.6226214401.77-1.3%10.29+20.4%degraded
Random read 64KB1.0817627.8713.76 ms/op13.7017.1719.2753.0126214401.12-3.9%19.53-1.3%pass
Sequential write 256KB2.7011070.2721.54 ms/op23.2032.3736.4453.736553602.73-1.0%36.440.0%pass
Sequential read 256KB3.1212800.0017.88 ms/op19.2723.9925.5635.086553603.11+0.5%26.08-2.0%pass
Random write 256KB2.048372.0029.01 ms/op22.6838.0174.971356.556553602.51-18.6%30.28+147.6%fail
Random read 256KB2.369681.7825.65 ms/op25.5630.8034.3452.646553602.37-0.3%34.340.0%pass
Sequential write 1MB3.303374.6764.64 ms/op61.08143.65173.02337.221638403.45-4.5%191.89-9.8%pass
Sequential read 1MB2.722788.2982.36 ms/op91.75104.33109.58185.941638402.72+0.1%108.53+1.0%pass
Random write 1MB2.252307.28102.45 ms/op80.22164.63910.161047.291638403.09-27.1%229.64+296.3%fail
Random read 1MB2.182231.85110.10 ms/op111.67124.26131.60188.671638402.15+1.4%135.27-2.7%pass

Failures and attribution

Failure analysis

g-rustc-recursion-compile

  • Class: unknown_failure
  • Failure layer: build
  • Root-cause confidence: medium
  • Root cause: Rust compiler query recursion limit is exceeded while type-checking async test code in fallback_read_test and write_cache_test. Whether increased type complexity in curvine-tests or rustc 1.98 limits is primary remains unproven.
  • Recommendation: Bisect commits between last passing daily compilation SHA and current main; add recursion_limit attributes or refactor async test helpers if product-side, or adjust toolchain settings if environment-specific.

g-ltp-ftest-eio-worker-flush

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: high
  • Root cause: The worker Running-frame flush change in write_handler.rs (PR #1694) likely regressed read/readv durability on curvine-fuse for ftest workloads. Failure persists on de580be3; narrowed bisect against LTP-clean SHA 04d988e3 is still required.
  • Recommendation: Review flush-only Running frame handling in curvine-worker/src/worker/handler/write_handler.rs from PR #1694; add ftest read/readv regression coverage on curvine-fuse.

g-ltp-lftest-eio-worker-flush

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: high
  • Root cause: Likely the same worker Running-frame flush regression as ftest read EIO; write-path symptom on lftest01.
  • Recommendation: Address alongside PR #1694 write_handler flush regression in issue #1701.

g-ltp-stream-eio-worker-flush

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: medium
  • Root cause: Likely cascade from the worker flush regression affecting stream I/O on curvine-fuse.
  • Recommendation: Re-test stream cases after worker flush fix in issue #1701.

g-ltp-writetest-eio-worker-flush

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: medium
  • Root cause: Likely verify failure caused by incomplete writes from the worker flush regression.
  • Recommendation: Re-test writetest after worker flush fix in issue #1701.

g-ltp-inode-eio-worker-flush

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: high
  • Root cause: Likely the same worker flush regression manifesting on inode stress writes.
  • Recommendation: Re-test inode02 after worker flush fix in issue #1701.

g-ltp-fs-di-eio-worker-flush

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: medium
  • Root cause: Likely unable to create large test files because of the worker flush regression returning EIO on writes.
  • Recommendation: Re-test fs_di after worker flush fix in issue #1701.

g-ltp-fs-fill-eio-worker-flush

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: high
  • Root cause: Likely the same worker flush regression blocking loop-device file creation on curvine-fuse.
  • Recommendation: Re-test fs_fill after worker flush fix in issue #1701.

g-fuse-git-clone-eio-worker-flush

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: medium
  • Root cause: Likely the same worker flush regression returning EIO on sustained small writes during git metadata operations on curvine-fuse.
  • Recommendation: Re-test git clone on curvine-fuse after worker flush fix in issue #1701.

Failed case summary

CaseSuite / PackageStatusKey errorRoot group
integration profileintegration / integrationFAILEDprofile failedg-rustc-recursion-compile
daily profiledaily / dailyFAILEDprofile failedg-rustc-recursion-compile
fusefuse / fuseFAILED“status”: “failed”g-fuse-git-clone-eio-worker-flush
inode02fs-jfs / ltpFAILEDinode02 1 TFAIL : inode02.c:851: Test failedg-ltp-inode-eio-worker-flush
stream01fs-jfs / ltpFAILEDstream01 1 TFAIL : stream01.c:92: bad contents in stream011.34423 / stream01 2 TFAIL : stream01.c:106: bad contents in stream012.34423 / stream01 3 TFAIL : stream01.c:115: Test failed.g-ltp-stream-eio-worker-flush
stream03fs-jfs / ltpFAILEDstream03 1 TFAIL : stream03.c:164: strlen(junk)=26: file pointer descrepancy 7 (pos=0) / stream03 2 TFAIL : stream03.c:175: Test failed in block0. / stream03 3 TFAIL : stream03.c:278: strlen(junk)=26: file pointer descrepancy 7 (opos=0)g-ltp-stream-eio-worker-flush
stream04fs-jfs / ltpFAILEDstream04 1 TFAIL : stream04.c:104: fread failed: Input/output errorg-ltp-stream-eio-worker-flush
stream05fs-jfs / ltpFAILEDstream05 2 TFAIL : stream05.c:115: read did not read right number / stream05 3 TFAIL : stream05.c:119: read returned bad values / stream05 4 TFAIL : stream05.c:125: Test failed in block1.g-ltp-stream-eio-worker-flush
ftest01fs-jfs / ltpFAILEDftest01 1 TFAIL : ftest01.c:343: Test[0]: read fail at 73800, errno = 5. / ftest01 1 TFAIL : ftest01.c:343: Test[1]: read fail at ce000, errno = 5. / ftest01 1 TFAIL : ftest01.c:343: Test[2]: read fail at 29000, errno = 5.g-ltp-ftest-eio-worker-flush
ftest02fs-jfs / ltpFAILEDftest02 1 TBROK : ftest02.c:413: Test[0]: error 5 on read / ftest02 2 TBROK : ftest02.c:413: Remaining cases broken / ftest02 1 TBROK : ftest02.c:413: Test[3]: error 5 on readg-ltp-ftest-eio-worker-flush
ftest03fs-jfs / ltpFAILEDftest03 1 TFAIL : ftest03.c:404: Test[0]: readv fail at 39000, errno = 5. / ftest03 1 TFAIL : ftest03.c:404: Test[1]: readv fail at 2c000, errno = 5. / ftest03 1 TFAIL : ftest03.c:404: Test[2]: readv fail at 86000, errno = 5.g-ltp-ftest-eio-worker-flush
ftest04fs-jfs / ltpFAILEDftest04 1 TFAIL : ftest04.c:327: Test[2]: readv fail at e2000, errno = 5. / ftest04 1 TFAIL : ftest04.c:327: Test[1]: readv fail at a3000, errno = 5. / ftest04 1 TFAIL : ftest04.c:327: Test[0]: readv fail at f2800, errno = 5.g-ltp-ftest-eio-worker-flush
ftest05fs-jfs / ltpFAILEDftest05 1 TFAIL : ftest05.c:337: Test[0]: read fail at 21800: errno=EIO(5): Input/output error / ftest05 1 TFAIL : ftest05.c:337: Test[1]: read fail at d7800: errno=EIO(5): Input/output error / ftest05 1 TFAIL : ftest05.c:337: Test[2]: read fail at 49000: errno=EIO(5): Input/output errorg-ltp-ftest-eio-worker-flush
ftest06fs-jfs / ltpFAILEDftest06 1 TFAIL : ftest06.c:431: Test[4]: error 5 on read / ftest06 1 TFAIL : ftest06.c:431: Test[0]: error 5 on read / ftest06 1 TFAIL : ftest06.c:431: Test[1]: error 5 on readg-ltp-ftest-eio-worker-flush
ftest07fs-jfs / ltpFAILEDftest07 1 TFAIL : ftest07.c:399: Test[0]: readv fail at 0, errno = 5. / ftest07 1 TFAIL : ftest07.c:399: Test[1]: readv fail at 91000, errno = 5. / ftest07 1 TFAIL : ftest07.c:399: Test[2]: readv fail at 3b000, errno = 5.g-ltp-ftest-eio-worker-flush
ftest08fs-jfs / ltpFAILEDftest08 1 TFAIL : ftest08.c:340: Test[2]: readv fail at b2800x, errno = 5. / ftest08 1 TFAIL : ftest08.c:340: Test[1]: readv fail at 87800x, errno = 5. / ftest08 1 TFAIL : ftest08.c:340: Test[3]: readv fail at b8000x, errno = 5.g-ltp-ftest-eio-worker-flush
lftest01fs-jfs / ltpFAILEDlftest.c:55: TFAIL: write() failed: EIO (5)g-ltp-lftest-eio-worker-flush
writetest01fs-jfs / ltpFAILEDwritetest 2 TFAIL : writetest.c:253: Verify: Failureg-ltp-writetest-eio-worker-flush
fs_difs-jfs / ltpFAILEDfs_di 10 TFAIL : ltpapicmd.c:188: Test Failed: Could not create testfile of size 30Mbg-ltp-fs-di-eio-worker-flush
fs_fillfs-jfs / ltpFAILEDtst_device.c:335: TBROK: Failed to acquire deviceg-ltp-fs-fill-eio-worker-flush
writetestfs-jfs / ltpTFAILwritetest.c:253: Verify: Failureg-ltp-writetest-eio-worker-flush

Failed case reconciliation

ProfileReported failuresSource failuresDelta
integration110
daily101
fuse110
ltp18180

Common root-cause groups

g-rustc-recursion-compile

  • Model class: unknown_failure; confidence: medium
  • Recommendation: Bisect commits between last passing daily compilation SHA and current main; add recursion_limit attributes or refactor async test helpers if product-side, or adjust toolchain settings if environment-specific.

g-ltp-ftest-eio-worker-flush

  • Model class: pre_existing_failure; confidence: high
  • Recommendation: Review flush-only Running frame handling in curvine-worker/src/worker/handler/write_handler.rs from PR #1694; add ftest read/readv regression coverage on curvine-fuse.

g-ltp-lftest-eio-worker-flush

  • Model class: pre_existing_failure; confidence: high
  • Recommendation: Address alongside PR #1694 write_handler flush regression in issue #1701.

g-ltp-stream-eio-worker-flush

  • Model class: pre_existing_failure; confidence: medium
  • Recommendation: Re-test stream cases after worker flush fix in issue #1701.

g-ltp-writetest-eio-worker-flush

  • Model class: pre_existing_failure; confidence: medium
  • Recommendation: Re-test writetest after worker flush fix in issue #1701.

g-ltp-inode-eio-worker-flush

  • Model class: pre_existing_failure; confidence: high
  • Recommendation: Re-test inode02 after worker flush fix in issue #1701.

g-ltp-fs-di-eio-worker-flush

  • Model class: pre_existing_failure; confidence: medium
  • Recommendation: Re-test fs_di after worker flush fix in issue #1701.

g-ltp-fs-fill-eio-worker-flush

  • Model class: pre_existing_failure; confidence: high
  • Recommendation: Re-test fs_fill after worker flush fix in issue #1701.

g-fuse-git-clone-eio-worker-flush

  • Model class: pre_existing_failure; confidence: medium
  • Recommendation: Re-test git clone on curvine-fuse after worker flush fix in issue #1701.

Follow-up

Defects and fixes

  • GitHub Issue: 1701 (open, reused)

Risks

  • A subset of green profiles does not override a full-chain NO-GO.
  • Report-only performance regressions still require human follow-up.

Next actions

  • Bisect commits between last passing daily compilation SHA and current main; add recursion_limit attributes or refactor async test helpers if product-side, or adjust toolchain settings if environment-specific.
  • Review flush-only Running frame handling in curvine-worker/src/worker/handler/write_handler.rs from PR #1694; add ftest read/readv regression coverage on curvine-fuse.
  • Address alongside PR #1694 write_handler flush regression in issue #1701.
  • Re-test stream cases after worker flush fix in issue #1701.
  • Re-test writetest after worker flush fix in issue #1701.
  • Re-test inode02 after worker flush fix in issue #1701.
  • Re-test fs_di after worker flush fix in issue #1701.
  • Re-test fs_fill after worker flush fix in issue #1701.
  • Re-test git clone on curvine-fuse after worker flush fix in issue #1701.
  • Rerun the full chain after fixes and require all release gates to pass.

Publication note

This public report contains only sanitized test outcomes. Private execution evidence remains internal.