Skip to content
2026-09-07 full-chain

Curvine full-chain daily test report - 2026-09-07

Quality conclusion

Executive summary

Caution

Release decision: NO-GO. Pipeline FAIL; ran 7 profiles, 3 passed, 3 failed.

Quality gates

GateCriterionActualVerdict
Full-chain resultAll required profiles passed3/7 passedFAIL
Failure attributionFailures classified2 root-cause groupsPASS
Resource cleanupAll profile cleanups succeeded7/7PASS

Conclusion

This full-chain run did not pass. Failed profiles require resolution before release.

Test results

Profile summary

ProfilePreflightResultDurationClassCleanup
fastPASSPASS12m 15spassedPASSED
integrationPASSFAIL12m 40sunknown_failurePASSED
dailyPASSFAIL20m 35sunknown_failurePASSED
fusePASSFAIL3m 07sunknown_failurePASSED
ltpPASSPASS49m 32spassedPASSED
csiPASSPASS0m 35spassedPASSED
perf-benchmarkNOT_RECORDEDFAILED5m 45sfailedPASSED

LTP

  • Status: COMPLETED
  • Suites completed: 7
  • Suites remaining: 0
  • Stats: 1129 passed / 0 real failed / 141 skipped / 0 report-consistency errors
SuiteStatusPassedReal failedSkippedReport errorsReturn code
fs_perms_simplepassed180000
fsxpassed10000
fs_bindpassed10000
smoketestpassed120100
iopassed20000
fs-jfspassed270200
syscalls-jfspassed1068013800

Performance

Note

Gate policy: report only, non-blocking for the full-chain result. Mark yellow/red vs baseline for human follow-up.

  • Status: failed
  • Gate mode: report_only

Metadata performance (this run)

ITEMVALUEAVG COSTP50 msP95 msP99 msMAX msSAMPLESERRORSBaseline VALUEVALUE deltaBaseline P99 msP99 deltaStatus
Create file23370.78 ops/s1.70 ms/op2.054.094.095.25200000021668.74 ops/s+7.9%4.09+0.1%pass
Stat file71017.91 ops/s0.56 ms/op1.021.021.023.18200000063487.48 ops/s+11.9%2.05-50.1%pass
Open file69220.37 ops/s0.57 ms/op1.021.021.021.93200000063113.20 ops/s+9.7%2.05-50.1%pass
Rename file32239.33 ops/s1.23 ms/op2.053.963.963.96200000030314.48 ops/s+6.3%4.09-3.1%pass
Delete file31892.21 ops/s1.24 ms/op2.054.094.094.30200000031231.33 ops/s+2.1%4.09+0.1%pass

FIO read/write (this run)

ITEMSPEED GiB/sIOPSAVG COSTP50 msP95 msP99 msMAX msSAMPLESERRORSBaseline SPEED GiB/sSPEED deltaBaseline P99 msP99 deltaStatus
Sequential write 64KB1.6727406.599.08 ms/op9.1110.5511.4722.8326214401.71-2.2%10.94+4.8%pass
Sequential read 64KB2.5641896.125.83 ms/op5.549.6311.7323.2626214402.28+12.2%12.78-8.2%pass
Random write 64KB1.7228233.068.73 ms/op8.599.9011.08294.4326214401.77-2.6%10.29+7.6%pass
Random read 64KB1.1719185.0112.90 ms/op12.7816.0618.7442.4726214401.12+4.6%19.53-4.0%pass
Sequential write 256KB2.4710113.5823.34 ms/op24.2533.8238.0153.766553602.73-9.6%36.44+4.3%pass
Sequential read 256KB3.1012681.1117.94 ms/op19.5323.9925.8238.956553603.11-0.5%26.08-1.0%pass
Random write 256KB2.329500.7225.39 ms/op23.2029.4932.64862.726553602.51-7.6%30.28+7.8%pass
Random read 256KB2.5510443.9823.84 ms/op23.7228.9733.1653.086553602.37+7.6%34.34-3.4%pass
Sequential write 1MB3.123193.7673.65 ms/op64.75156.24238.03339.161638403.45-9.6%191.89+24.0%degraded
Sequential read 1MB3.743834.3159.35 ms/op54.7976.0284.41116.921638402.72+37.7%108.53-22.2%pass
Random write 1MB2.402456.7495.47 ms/op78.12119.01977.271589.171638403.09-22.4%229.64+325.6%fail
Random read 1MB2.272323.64105.57 ms/op106.43119.01126.35182.861638402.15+5.5%135.27-6.6%pass

Failures and attribution

Failure analysis

g-fuse-buffered-io-eio

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: high
  • Root cause: curvine-fuse mount option generation from FuseConf (introduced in e01cf918) appears to mis-apply or omit settings required for buffered read/write, causing user I/O on /curvine-fuse to fail with Input/output error. Mount and metadata operations succeed, but sustained or verified writes fail deterministically. This is a pre-existing product defect still open as GitHub issue #1646.
  • Recommendation: Review e01cf918 changes in curvine-fuse/src/cli/mount_args.rs and crates/common/curvine-config/src/fuse_conf.rs ClientConfCliOverrides generation; compare effective mount flags against eb76f94a and restore correct buffered I/O behavior.

g-resize-fs-dir-lock-regression

  • Class: product_regression
  • Failure layer: product
  • Root-cause confidence: high
  • Root cause: Commit dc637c67 (PR #1663, feat(master): add inode id to resize and assign worker RPCs) refactored MasterFilesystem::resize to acquire fs_dir.write() at function entry and retain it while calling worker_manager.write().remove_blocks() and building the FileBlocks response. This violates the lock-scope contract established by PR #670 and tested by resize_lock_scope_test.rs, which requires resize to release fs_dir before blocking on worker_manager.
  • Recommendation: Restore the narrow lock scope in MasterFilesystem::resize: release fs_dir before acquiring worker_manager for remove_blocks, then re-acquire only as needed to build the response (or build the response before releasing fs_dir but after remove_blocks completes without holding fs_dir during worker_manager acquisition). Ensure resize_lock_scope_test passes.

Failed case summary

CaseSuite / PackageStatusKey errorRoot group
resize_does_not_hold_fs_lock_while_waiting_worker_managerresize_lock_scope_test / curvine-serverFAILEDcalled Result::unwrap() on an Err value: PoisonError { .. }g-resize-fs-dir-lock-regression
resize_does_not_hold_fs_lock_while_waiting_worker_managerresize_lock_scope_test / curvine-serverFAILEDcalled Result::unwrap() on an Err value: PoisonError { .. }g-resize-fs-dir-lock-regression
FIO Sequential Write Test (256KB blocks)fio / fuseFAILFIO Sequential Write test failedg-fuse-buffered-io-eio
FIO Sequential Read Test (256KB blocks)fio / fuseFAILFIO Sequential Read test failedg-fuse-buffered-io-eio
FIO Random Write Test (256KB blocks)fio / fuseFAILFIO Random Write test failedg-fuse-buffered-io-eio
FIO Random Read Test (256KB blocks)fio / fuseFAILFIO Random Read test failedg-fuse-buffered-io-eio
FIO Mixed Random Read/Write Test (256KB blocks, 70% read, 30% write)fio / fuseFAILFIO Mixed Random Read/Write test failedg-fuse-buffered-io-eio
fusefuse / fuseFAILED“status”: “failed”g-fuse-buffered-io-eio

Failed case reconciliation

ProfileReported failuresSource failuresDelta
integration110
daily110
fuse660

Common root-cause groups

g-fuse-buffered-io-eio

  • Model class: pre_existing_failure; confidence: high
  • Recommendation: Review e01cf918 changes in curvine-fuse/src/cli/mount_args.rs and crates/common/curvine-config/src/fuse_conf.rs ClientConfCliOverrides generation; compare effective mount flags against eb76f94a and restore correct buffered I/O behavior.

g-resize-fs-dir-lock-regression

  • Model class: product_regression; confidence: high
  • Recommendation: Restore the narrow lock scope in MasterFilesystem::resize: release fs_dir before acquiring worker_manager for remove_blocks, then re-acquire only as needed to build the response (or build the response before releasing fs_dir but after remove_blocks completes without holding fs_dir during worker_manager acquisition). Ensure resize_lock_scope_test passes.

Follow-up

Defects and fixes

  • GitHub Issue: 1671 (open, reused)

Risks

  • A subset of green profiles does not override a full-chain NO-GO.
  • Report-only performance regressions still require human follow-up.

Next actions

  • Review e01cf918 changes in curvine-fuse/src/cli/mount_args.rs and crates/common/curvine-config/src/fuse_conf.rs ClientConfCliOverrides generation; compare effective mount flags against eb76f94a and restore correct buffered I/O behavior.
  • Restore the narrow lock scope in MasterFilesystem::resize: release fs_dir before acquiring worker_manager for remove_blocks, then re-acquire only as needed to build the response (or build the response before releasing fs_dir but after remove_blocks completes without holding fs_dir during worker_manager acquisition). Ensure resize_lock_scope_test passes.
  • Rerun the full chain after fixes and require all release gates to pass.

Publication note

This public report contains only sanitized test outcomes. Private execution evidence remains internal.