Skip to content
2026-09-15 full-chain

Curvine full-chain daily test report - 2026-09-15

Quality conclusion

Executive summary

Caution

Release decision: NO-GO. Pipeline FAIL; ran 7 profiles, 3 passed, 3 failed.

Quality gates

GateCriterionActualVerdict
Full-chain resultAll required profiles passed3/7 passedFAIL
Failure attributionFailures classified2 root-cause groupsPASS
Resource cleanupAll profile cleanups succeeded7/7PASS

Conclusion

This full-chain run did not pass. Failed profiles require resolution before release.

Test results

Profile summary

ProfilePreflightResultDurationClassCleanup
fastPASSPASS12m 37spassedPASSED
integrationPASSFAIL5m 08sharness_failurePASSED
dailyPASSFAIL9m 14sunknown_failurePASSED
fusePASSFAIL3m 44sunknown_failurePASSED
ltpPASSPASS50m 42spassedPASSED
csiPASSPASS0m 36spassedPASSED
perf-benchmarkNOT_RECORDEDFAILED5m 30sfailedPASSED

LTP

  • Status: COMPLETED
  • Suites completed: 7
  • Suites remaining: 0
  • Stats: 1129 passed / 0 real failed / 141 skipped / 0 report-consistency errors
SuiteStatusPassedReal failedSkippedReport errorsReturn code
fs_perms_simplepassed180000
fsxpassed10000
fs_bindpassed10000
smoketestpassed120100
iopassed20000
fs-jfspassed270200
syscalls-jfspassed1068013800

Performance

Note

Gate policy: report only, non-blocking for the full-chain result. Mark yellow/red vs baseline for human follow-up.

  • Status: failed
  • Gate mode: report_only

Metadata performance (this run)

ITEMVALUEAVG COSTP50 msP95 msP99 msMAX msSAMPLESERRORSBaseline VALUEVALUE deltaBaseline P99 msP99 deltaStatus
Create file23483.41 ops/s1.69 ms/op2.054.094.095.34200000021668.74 ops/s+8.4%4.09+0.1%pass
Stat file69346.69 ops/s0.57 ms/op1.021.021.022.69200000063487.48 ops/s+9.2%2.05-50.1%pass
Open file68221.21 ops/s0.58 ms/op1.021.021.023.01200000063113.20 ops/s+8.1%2.05-50.1%pass
Rename file32209.79 ops/s1.23 ms/op2.054.094.095.03200000030314.48 ops/s+6.3%4.09+0.1%pass
Delete file31833.34 ops/s1.24 ms/op2.054.094.094.37200000031231.33 ops/s+1.9%4.09+0.1%pass

FIO read/write (this run)

ITEMSPEED GiB/sIOPSAVG COSTP50 msP95 msP99 msMAX msSAMPLESERRORSBaseline SPEED GiB/sSPEED deltaBaseline P99 msP99 deltaStatus
Sequential write 64KB1.7428503.218.75 ms/op8.7210.0310.8116.5726214401.71+1.7%10.94-1.2%pass
Sequential read 64KB2.4740435.606.03 ms/op5.7310.0312.3947.1726214402.28+8.2%12.78-3.1%pass
Random write 64KB1.7428574.678.68 ms/op8.2210.2915.27322.3626214401.77-1.5%10.29+48.4%fail
Random read 64KB1.1218316.3813.23 ms/op13.0416.4519.0142.0826214401.12-0.2%19.53-2.7%pass
Sequential write 256KB3.0012268.0619.03 ms/op20.0531.0636.4463.156553602.73+9.7%36.440.0%pass
Sequential read 256KB2.7711358.0620.77 ms/op22.9427.3929.2344.786553603.11-10.8%26.08+12.1%pass
Random write 256KB2.8011485.4520.82 ms/op20.0525.8228.97473.826553602.51+11.7%30.28-4.3%pass
Random read 256KB2.4510036.1424.36 ms/op24.2529.7533.1649.996553602.37+3.4%34.34-3.4%pass
Sequential write 1MB3.453532.5665.76 ms/op60.03149.95210.76301.901638403.450.0%191.89+9.8%pass
Sequential read 1MB2.852919.9879.09 ms/op87.56102.24108.53170.481638402.72+4.8%108.530.0%pass
Random write 1MB3.163234.7570.78 ms/op65.27105.38291.50607.091638403.09+2.2%229.64+26.9%fail
Random read 1MB2.262318.05104.67 ms/op105.38117.96125.30172.691638402.15+5.3%135.27-7.4%pass

Failures and attribution

Failure analysis

g-rustc-recursion-compile

  • Class: unknown_failure
  • Failure layer: build
  • Root-cause confidence: medium
  • Root cause: Rust compiler query recursion limit is exceeded while type-checking async test code in fallback_read_test and write_cache_test crates. Whether increased type complexity in curvine-tests or rustc 1.98 toolchain limits is the primary cause remains unproven.
  • Recommendation: Bisect commits between last passing daily SHA and current main to identify async test changes that increased type depth. If product-side, add recursion_limit attributes or refactor test helpers; if toolchain-specific, adjust rustc settings in the build environment.

g-fuse-buffered-io-eio

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: high
  • Root cause: curvine-fuse buffered read/write on /curvine-fuse returns Input/output error during sustained or verified writes. This matches the pre-existing defect tracked in GitHub issue #1646 related to FuseConf mount CLI override generation.
  • Recommendation: Review FuseConf-to-mount flag mapping in curvine-fuse mount CLI code introduced around e01cf918; restore correct buffered I/O behavior per open issue #1646 repair plan.

Failed case summary

CaseSuite / PackageStatusKey errorRoot group
integration profileintegration / integrationFAILEDprofile failedg-rustc-recursion-compile
daily profiledaily / dailyFAILEDprofile failedg-rustc-recursion-compile
FIO Sequential Write Test (256KB blocks)fio / fuseFAILFIO Sequential Write test failedg-fuse-buffered-io-eio
FIO Sequential Read Test (256KB blocks)fio / fuseFAILFIO Sequential Read test failedg-fuse-buffered-io-eio
FIO Random Write Test (256KB blocks)fio / fuseFAILFIO Random Write test failedg-fuse-buffered-io-eio
FIO Random Read Test (256KB blocks)fio / fuseFAILFIO Random Read test failedg-fuse-buffered-io-eio
FIO Mixed Random Read/Write Test (256KB blocks, 70% read, 30% write)fio / fuseFAILFIO Mixed Random Read/Write test failedg-fuse-buffered-io-eio
fusefuse / fuseFAILED“status”: “failed”g-fuse-buffered-io-eio

Failed case reconciliation

ProfileReported failuresSource failuresDelta
integration110
daily101
fuse660

Common root-cause groups

g-rustc-recursion-compile

  • Model class: unknown_failure; confidence: medium
  • Recommendation: Bisect commits between last passing daily SHA and current main to identify async test changes that increased type depth. If product-side, add recursion_limit attributes or refactor test helpers; if toolchain-specific, adjust rustc settings in the build environment.

g-fuse-buffered-io-eio

  • Model class: pre_existing_failure; confidence: high
  • Recommendation: Review FuseConf-to-mount flag mapping in curvine-fuse mount CLI code introduced around e01cf918; restore correct buffered I/O behavior per open issue #1646 repair plan.

Follow-up

Defects and fixes

  • GitHub Issue: 1646 (open, reused)

Risks

  • A subset of green profiles does not override a full-chain NO-GO.
  • Report-only performance regressions still require human follow-up.

Next actions

  • Bisect commits between last passing daily SHA and current main to identify async test changes that increased type depth. If product-side, add recursion_limit attributes or refactor test helpers; if toolchain-specific, adjust rustc settings in the build environment.
  • Review FuseConf-to-mount flag mapping in curvine-fuse mount CLI code introduced around e01cf918; restore correct buffered I/O behavior per open issue #1646 repair plan.
  • Rerun the full chain after fixes and require all release gates to pass.

Publication note

This public report contains only sanitized test outcomes. Private execution evidence remains internal.