Skip to content
2026-09-09 full-chain

Curvine full-chain daily test report - 2026-09-09

Quality conclusion

Executive summary

Caution

Release decision: NO-GO. Pipeline FAIL; ran 7 profiles, 4 passed, 2 failed.

Quality gates

GateCriterionActualVerdict
Full-chain resultAll required profiles passed4/7 passedFAIL
Failure attributionFailures classified2 root-cause groupsPASS
Resource cleanupAll profile cleanups succeeded7/7PASS

Conclusion

This full-chain run did not pass. Failed profiles require resolution before release.

Test results

Profile summary

ProfilePreflightResultDurationClassCleanup
fastPASSPASS12m 07spassedPASSED
integrationPASSPASS12m 06spassedPASSED
dailyPASSFAIL20m 12sunknown_failurePASSED
fusePASSFAIL3m 02sunknown_failurePASSED
ltpPASSPASS49m 15spassedPASSED
csiPASSPASS0m 36spassedPASSED
perf-benchmarkNOT_RECORDEDFAILED5m 41sfailedPASSED

LTP

  • Status: COMPLETED
  • Suites completed: 7
  • Suites remaining: 0
  • Stats: 1129 passed / 0 real failed / 141 skipped / 0 report-consistency errors
SuiteStatusPassedReal failedSkippedReport errorsReturn code
fs_perms_simplepassed180000
fsxpassed10000
fs_bindpassed10000
smoketestpassed120100
iopassed20000
fs-jfspassed270200
syscalls-jfspassed1068013800

Performance

Note

Gate policy: report only, non-blocking for the full-chain result. Mark yellow/red vs baseline for human follow-up.

  • Status: failed
  • Gate mode: report_only

Metadata performance (this run)

ITEMVALUEAVG COSTP50 msP95 msP99 msMAX msSAMPLESERRORSBaseline VALUEVALUE deltaBaseline P99 msP99 deltaStatus
Create file23168.30 ops/s1.72 ms/op2.054.094.094.57200000021668.74 ops/s+6.9%4.09+0.1%pass
Stat file69786.89 ops/s0.57 ms/op1.021.021.022.93200000063487.48 ops/s+9.9%2.05-50.1%pass
Open file69180.89 ops/s0.58 ms/op1.021.021.021.25200000063113.20 ops/s+9.6%2.05-50.1%pass
Rename file33539.96 ops/s1.19 ms/op2.053.613.613.61200000030314.48 ops/s+10.6%4.09-11.6%pass
Delete file32920.34 ops/s1.21 ms/op2.054.094.094.68200000031231.33 ops/s+5.4%4.09+0.1%pass

FIO read/write (this run)

ITEMSPEED GiB/sIOPSAVG COSTP50 msP95 msP99 msMAX msSAMPLESERRORSBaseline SPEED GiB/sSPEED deltaBaseline P99 msP99 deltaStatus
Sequential write 64KB1.6727320.909.15 ms/op9.1110.5511.4724.5826214401.71-2.5%10.94+4.8%pass
Sequential read 64KB2.3237958.886.39 ms/op5.9311.2113.1728.6526214402.28+1.6%12.78+3.1%pass
Random write 64KB1.7328389.008.72 ms/op8.599.7610.68292.6526214401.77-2.1%10.29+3.8%pass
Random read 64KB1.0517225.9214.04 ms/op13.9617.4319.5356.8526214401.12-6.1%19.530.0%pass
Sequential write 256KB2.5310359.7923.00 ms/op24.5132.9037.4947.836553602.73-7.4%36.44+2.9%pass
Sequential read 256KB3.2213183.6717.37 ms/op18.2224.5127.1358.056553603.11+3.5%26.08+4.0%pass
Random write 256KB2.4510051.5324.21 ms/op23.4628.9731.85476.796553602.51-2.2%30.28+5.2%pass
Random read 256KB2.319471.8925.98 ms/op25.8231.3334.3446.246553602.37-2.4%34.340.0%pass
Sequential write 1MB3.313388.6264.64 ms/op58.46135.27179.31287.721638403.45-4.1%191.89-6.6%pass
Sequential read 1MB3.223294.5969.32 ms/op77.0791.7594.90140.921638402.72+18.3%108.53-12.6%pass
Random write 1MB2.612667.9785.52 ms/op73.92117.96784.331347.351638403.09-15.7%229.64+241.5%fail
Random read 1MB2.322378.28104.32 ms/op105.38117.96128.45183.401638402.15+8.0%135.27-5.0%pass

Failures and attribution

Failure analysis

g-fuse-buffered-io-eio

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: high
  • Root cause: curvine-fuse mount option generation from FuseConf (introduced in e01cf918) appears to mis-apply or omit settings required for buffered read/write, causing user I/O on /curvine-fuse to fail with Input/output error. Mount and metadata operations succeed, but sustained or verified writes fail deterministically. This is a pre-existing product defect still open as GitHub issue #1646.
  • Recommendation: Review e01cf918 changes in curvine-fuse/src/cli/mount_args.rs and crates/common/curvine-config/src/fuse_conf.rs ClientConfCliOverrides generation; compare effective mount flags against eb76f94a and restore correct buffered I/O behavior.

g-raft-propose-apply-timeout-flaky

  • Class: flaky
  • Failure layer: test
  • Root-cause confidence: medium
  • Root cause: The propose_response_waits_until_committed_entry_is_applied unit test uses a 3-second timeout for apply to start after propose. Under load or slow Raft election timing (log shows ~14 seconds before election began), the committed entry may not reach BlockingApplyAppStorage within the timeout window. Daily passed on the immediately prior run at 5ecf8ca2, suggesting intermittent timing sensitivity rather than a deterministic product defect on 39069ae2.
  • Recommendation: If failures recur, increase the wait_started timeout in raft_node_test.rs or investigate Raft election/apply latency under parallel test load. No product code change warranted from intermittent timeout without stable reproduction across commits.

Failed case summary

CaseSuite / PackageStatusKey errorRoot group
propose_response_waits_until_committed_entry_is_appliedraft_node_test / curvine-raftFAILEDthread ‘propose_response_waits_until_committed_entry_is_applied’ (29715) panicked at crates/metadata/curvine-raft/tests/raft_node_test.rs:615:14:g-raft-propose-apply-timeout-flaky
FIO Sequential Write Test (256KB blocks)fio / fuseFAILFIO Sequential Write test failedg-fuse-buffered-io-eio
FIO Sequential Read Test (256KB blocks)fio / fuseFAILFIO Sequential Read test failedg-fuse-buffered-io-eio
FIO Random Write Test (256KB blocks)fio / fuseFAILFIO Random Write test failedg-fuse-buffered-io-eio
FIO Random Read Test (256KB blocks)fio / fuseFAILFIO Random Read test failedg-fuse-buffered-io-eio
FIO Mixed Random Read/Write Test (256KB blocks, 70% read, 30% write)fio / fuseFAILFIO Mixed Random Read/Write test failedg-fuse-buffered-io-eio
fusefuse / fuseFAILED“status”: “failed”g-fuse-buffered-io-eio

Failed case reconciliation

ProfileReported failuresSource failuresDelta
daily110
fuse660

Common root-cause groups

g-fuse-buffered-io-eio

  • Model class: pre_existing_failure; confidence: high
  • Recommendation: Review e01cf918 changes in curvine-fuse/src/cli/mount_args.rs and crates/common/curvine-config/src/fuse_conf.rs ClientConfCliOverrides generation; compare effective mount flags against eb76f94a and restore correct buffered I/O behavior.

g-raft-propose-apply-timeout-flaky

  • Model class: flaky; confidence: medium
  • Recommendation: If failures recur, increase the wait_started timeout in raft_node_test.rs or investigate Raft election/apply latency under parallel test load. No product code change warranted from intermittent timeout without stable reproduction across commits.

Follow-up

Defects and fixes

  • GitHub Issue: 1646 (open, reused)

Risks

  • A subset of green profiles does not override a full-chain NO-GO.
  • Report-only performance regressions still require human follow-up.

Next actions

  • Review e01cf918 changes in curvine-fuse/src/cli/mount_args.rs and crates/common/curvine-config/src/fuse_conf.rs ClientConfCliOverrides generation; compare effective mount flags against eb76f94a and restore correct buffered I/O behavior.
  • If failures recur, increase the wait_started timeout in raft_node_test.rs or investigate Raft election/apply latency under parallel test load. No product code change warranted from intermittent timeout without stable reproduction across commits.
  • Rerun the full chain after fixes and require all release gates to pass.

Publication note

This public report contains only sanitized test outcomes. Private execution evidence remains internal.