Skip to content
2026-09-11 full-chain

Curvine full-chain daily test report - 2026-09-11

Quality conclusion

Executive summary

Caution

Release decision: NO-GO. Pipeline FAIL; ran 7 profiles, 3 passed, 3 failed.

Quality gates

GateCriterionActualVerdict
Full-chain resultAll required profiles passed3/7 passedFAIL
Failure attributionFailures classified2 root-cause groupsPASS
Resource cleanupAll profile cleanups succeeded6/7FAIL

Conclusion

This full-chain run did not pass. Failed profiles require resolution before release.

Test results

Profile summary

ProfilePreflightResultDurationClassCleanup
fastPASSPASS12m 28spassedPASSED
integrationPASSFAIL5m 07sharness_failurePASSED
dailyPASSFAIL9m 14sunknown_failurePASSED
fusePASSFAIL3m 41srunner_infrastructure_failureFAILED
ltpPASSPASS49m 54spassedPASSED
csiPASSPASS0m 36spassedPASSED
perf-benchmarkNOT_RECORDEDFAILED7m 39sfailedPASSED

LTP

  • Status: COMPLETED
  • Suites completed: 7
  • Suites remaining: 0
  • Stats: 1129 passed / 0 real failed / 141 skipped / 0 report-consistency errors
SuiteStatusPassedReal failedSkippedReport errorsReturn code
fs_perms_simplepassed180000
fsxpassed10000
fs_bindpassed10000
smoketestpassed120100
iopassed20000
fs-jfspassed270200
syscalls-jfspassed1068013800

Performance

Note

Gate policy: report only, non-blocking for the full-chain result. Mark yellow/red vs baseline for human follow-up.

  • Status: failed
  • Gate mode: report_only

Metadata performance (this run)

ITEMVALUEAVG COSTP50 msP95 msP99 msMAX msSAMPLESERRORSBaseline VALUEVALUE deltaBaseline P99 msP99 deltaStatus
Create file23049.70 ops/s1.73 ms/op2.054.094.096.50200000021668.74 ops/s+6.4%4.09+0.1%pass
Stat file68805.77 ops/s0.58 ms/op1.021.021.022.91200000063487.48 ops/s+8.4%2.05-50.1%pass
Open file67769.47 ops/s0.59 ms/op1.021.021.801.80200000063113.20 ops/s+7.4%2.05-12.4%pass
Rename file32382.44 ops/s1.22 ms/op2.054.094.094.67200000030314.48 ops/s+6.8%4.09+0.1%pass
Delete file32735.81 ops/s1.21 ms/op2.054.094.094.78200000031231.33 ops/s+4.8%4.09+0.1%pass

FIO read/write (this run)

ITEMSPEED GiB/sIOPSAVG COSTP50 msP95 msP99 msMAX msSAMPLESERRORSBaseline SPEED GiB/sSPEED deltaBaseline P99 msP99 deltaStatus
Sequential write 64KB1.7128069.818.89 ms/op8.8510.2911.0818.9826214401.71+0.2%10.94+1.2%pass
Sequential read 64KB2.1935821.816.52 ms/op6.1310.5512.7832.4726214402.28-4.1%12.780.0%pass
Random write 64KB1.8029487.518.42 ms/op8.229.5010.29272.9926214401.77+1.7%10.290.0%pass
Random read 64KB1.0517258.8113.92 ms/op13.8317.4320.5878.7526214401.12-5.9%19.53+5.4%pass
Sequential write 256KB2.8911831.7420.26 ms/op22.1531.0635.9143.886553602.73+5.8%36.44-1.4%pass
Sequential read 256KB2.429901.1924.71 ms/op26.0828.9730.2849.636553603.11-22.3%26.08+16.1%degraded
Random write 256KB2.5910623.4422.44 ms/op21.6326.8730.28453.636553602.51+3.3%30.280.0%pass
Random read 256KB2.289323.6626.10 ms/op26.0831.5934.3456.066553602.37-4.0%34.340.0%pass
Sequential write 1MB3.473557.8763.28 ms/op67.63137.36164.63305.721638403.45+0.7%191.89-14.2%pass
Sequential read 1MB2.522580.1691.84 ms/op99.09113.77121.11175.181638402.72-7.4%108.53+11.6%pass
Random write 1MB3.003070.4677.41 ms/op73.92108.53295.70558.281638403.09-3.0%229.64+28.8%fail
Random read 1MB2.092136.11114.39 ms/op115.87128.45135.27178.411638402.15-3.0%135.270.0%pass

Failures and attribution

Failure analysis

g-rustc-recursion-compile

  • Class: unknown_failure
  • Failure layer: build
  • Root-cause confidence: medium
  • Root cause: Rust compiler query recursion limit is exceeded while type-checking async test code in fallback_read_test and write_cache_test crates. Whether this is a new product regression from increased type complexity on a858336e or a toolchain/environment sensitivity is not yet proven; prior daily runs at earlier SHAs passed compilation.
  • Recommendation: Bisect a858336e against the last passing daily SHA to identify commits that increased async type depth in curvine-tests. If confirmed product-side, add recursion_limit attributes or refactor async test helpers; if toolchain-specific, pin or adjust rustc settings in the build image.

g-fuse-buffered-io-eio-cleanup-blocked

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: high
  • Root cause: curvine-fuse buffered read/write on /curvine-fuse returns Input/output error during sustained or verified writes. This matches the pre-existing defect tracked in GitHub issue #1646. Cleanup failed after tests with ‘could not remove owned worktree’, blocking automated issue handoff for this run.
  • Recommendation: Reuse existing repair track for GitHub issue #1646 (FuseConf mount CLI buffered I/O regression). No new issue from this run because cleanup did not succeed.

Failed case summary

CaseSuite / PackageStatusKey errorRoot group
integration profileintegration / integrationFAILEDprofile failedg-rustc-recursion-compile
daily profiledaily / dailyFAILEDprofile failedg-rustc-recursion-compile
FIO Sequential Write Test (256KB blocks)fio / fuseFAILFIO Sequential Write test failedg-fuse-buffered-io-eio-cleanup-blocked
FIO Sequential Read Test (256KB blocks)fio / fuseFAILFIO Sequential Read test failedg-fuse-buffered-io-eio-cleanup-blocked
FIO Random Write Test (256KB blocks)fio / fuseFAILFIO Random Write test failedg-fuse-buffered-io-eio-cleanup-blocked
FIO Random Read Test (256KB blocks)fio / fuseFAILFIO Random Read test failedg-fuse-buffered-io-eio-cleanup-blocked
FIO Mixed Random Read/Write Test (256KB blocks, 70% read, 30% write)fio / fuseFAILFIO Mixed Random Read/Write test failedg-fuse-buffered-io-eio-cleanup-blocked
fusefuse / fuseFAILED“status”: “failed”g-fuse-buffered-io-eio-cleanup-blocked

Failed case reconciliation

ProfileReported failuresSource failuresDelta
integration110
daily101
fuse660

Common root-cause groups

g-rustc-recursion-compile

  • Model class: unknown_failure; confidence: medium
  • Recommendation: Bisect a858336e against the last passing daily SHA to identify commits that increased async type depth in curvine-tests. If confirmed product-side, add recursion_limit attributes or refactor async test helpers; if toolchain-specific, pin or adjust rustc settings in the build image.

g-fuse-buffered-io-eio-cleanup-blocked

  • Model class: pre_existing_failure; confidence: high
  • Recommendation: Reuse existing repair track for GitHub issue #1646 (FuseConf mount CLI buffered I/O regression). No new issue from this run because cleanup did not succeed.

Follow-up

Defects and fixes

  • GitHub Issue: none

Risks

  • A subset of green profiles does not override a full-chain NO-GO.
  • Report-only performance regressions still require human follow-up.

Next actions

  • Bisect a858336e against the last passing daily SHA to identify commits that increased async type depth in curvine-tests. If confirmed product-side, add recursion_limit attributes or refactor async test helpers; if toolchain-specific, pin or adjust rustc settings in the build image.
  • Reuse existing repair track for GitHub issue #1646 (FuseConf mount CLI buffered I/O regression). No new issue from this run because cleanup did not succeed.
  • Rerun the full chain after fixes and require all release gates to pass.

Publication note

This public report contains only sanitized test outcomes. Private execution evidence remains internal.