Skip to content
2026-09-06 full-chain

Curvine full-chain daily test report - 2026-09-06

Quality conclusion

Executive summary

Caution

Release decision: NO-GO. Pipeline FAIL; ran 7 profiles, 4 passed, 2 failed, 1 degraded.

Quality gates

GateCriterionActualVerdict
Full-chain resultAll required profiles passed4/7 passedFAIL
Failure attributionFailures classified2 root-cause groupsPASS
Resource cleanupAll profile cleanups succeeded7/7PASS

Conclusion

This full-chain run did not pass. Failed profiles require resolution before release.

Test results

Profile summary

ProfilePreflightResultDurationClassCleanup
fastPASSPASS1m 52spassedPASSED
integrationPASSPASS10m 58spassedPASSED
dailyPASSFAIL16m 55sunknown_failurePASSED
fusePASSFAIL2m 38sunknown_failurePASSED
ltpPASSPASS48m 43spassedPASSED
csiPASSPASS0m 35spassedPASSED
perf-benchmarkNOT_RECORDEDDEGRADED2m 07sdegradedPASSED

LTP

  • Status: COMPLETED
  • Suites completed: 7
  • Suites remaining: 0
  • Stats: 1129 passed / 0 real failed / 141 skipped / 0 report-consistency errors
SuiteStatusPassedReal failedSkippedReport errorsReturn code
fs_perms_simplepassed180000
fsxpassed10000
fs_bindpassed10000
smoketestpassed120100
iopassed20000
fs-jfspassed270200
syscalls-jfspassed1068013800

Performance

Note

Gate policy: report only, non-blocking for the full-chain result. Mark yellow/red vs baseline for human follow-up.

  • Status: degraded
  • Gate mode: report_only

Metadata performance (this run)

ITEMVALUEAVG COSTP50 msP95 msP99 msMAX msSAMPLESERRORSBaseline VALUEVALUE deltaBaseline P99 msP99 deltaStatus
Create file23097.31 ops/s1.73 ms/op2.054.094.09163.60200000021668.74 ops/s+6.6%4.09+0.1%pass
Stat file71578.65 ops/s0.56 ms/op1.021.021.021.19200000063487.48 ops/s+12.7%2.05-50.1%pass
Open file69457.74 ops/s0.57 ms/op1.021.021.022.49200000063113.20 ops/s+10.1%2.05-50.1%pass
Rename file32699.97 ops/s1.21 ms/op2.054.094.094.22200000030314.48 ops/s+7.9%4.09+0.1%pass
Delete file32370.10 ops/s1.23 ms/op2.054.094.094.39200000031231.33 ops/s+3.6%4.09+0.1%pass

FIO read/write (this run)

ITEMSPEED GiB/sIOPSAVG COSTP50 msP95 msP99 msMAX msSAMPLESERRORSBaseline SPEED GiB/sSPEED deltaBaseline P99 msP99 deltaStatus
Sequential write 64KB1.6727349.409.12 ms/op9.1110.5511.4717.3426214401.71-2.4%10.94+4.8%pass
Sequential read 64KB2.2837353.096.55 ms/op6.1910.4212.6522.6126214402.280.0%12.78-1.0%pass
Random write 64KB1.7328413.618.72 ms/op8.599.7610.68326.9426214401.77-2.0%10.29+3.8%pass
Random read 64KB1.0316821.3614.64 ms/op14.4818.2220.3239.4426214401.12-8.3%19.53+4.0%pass
Sequential write 256KB2.5910604.5322.51 ms/op24.2532.6437.4953.606553602.73-5.2%36.44+2.9%pass
Sequential read 256KB2.5310343.4322.96 ms/op25.0328.7030.5455.316553603.11-18.8%26.08+17.1%degraded
Random write 256KB2.4710115.1423.66 ms/op23.2028.9732.11444.196553602.51-1.6%30.28+6.1%pass
Random read 256KB2.178899.5127.44 ms/op27.3933.1636.4456.876553602.37-8.3%34.34+6.1%pass
Sequential write 1MB3.033105.9775.62 ms/op70.78152.04200.28362.411638403.45-12.1%191.89+4.4%pass
Sequential read 1MB2.152197.42115.08 ms/op115.87122.16127.40212.111638402.72-21.1%108.53+17.4%degraded
Random write 1MB2.772839.0283.35 ms/op82.31116.92261.10512.361638403.09-10.3%229.64+13.7%pass
Random read 1MB2.132181.62113.86 ms/op114.82127.40137.36199.541638402.15-0.9%135.27+1.5%pass

Failures and attribution

Failure analysis

g-daily-coverage-timeout

  • Class: flaky
  • Failure layer: test
  • Root-cause confidence: medium
  • Root cause: The daily profile passed all 1449 nextest unit tests, but the ut-coverage orchestration step timed out (exit 124) while cargo-llvm-cov re-ran tests for coverage. This points to intermittent resource or timing pressure during the coverage subprocess rather than a demonstrated stable product defect at ad604155.
  • Recommendation: Increase coverage collection timeout or isolate llvm-cov rerun from runner load; retry coverage on transient timeout; investigate whether ordinary runner compile pressure during parallel crate builds causes coverage step to exceed its limit.

g-fuse-buffered-io-eio

  • Class: pre_existing_failure
  • Failure layer: product
  • Root-cause confidence: high
  • Root cause: curvine-fuse mount option generation from FuseConf (introduced in e01cf918) appears to mis-apply or omit settings required for buffered read/write, causing user I/O on /curvine-fuse to fail with Input/output error. Mount and metadata operations succeed, but sustained or verified writes fail deterministically. This is a pre-existing product defect still open as GitHub issue #1646.
  • Recommendation: Review e01cf918 changes in curvine-fuse/src/cli/mount_args.rs and crates/common/curvine-config/src/fuse_conf.rs ClientConfCliOverrides generation; compare effective mount flags against eb76f94a and restore correct buffered I/O behavior.

Failed case summary

CaseSuite / PackageStatusKey errorRoot group
daily profiledaily / dailyFAILEDprofile failedg-daily-coverage-timeout
FIO Sequential Write Test (256KB blocks)fio / fuseFAILFIO Sequential Write test failedg-fuse-buffered-io-eio
FIO Sequential Read Test (256KB blocks)fio / fuseFAILFIO Sequential Read test failedg-fuse-buffered-io-eio
FIO Random Write Test (256KB blocks)fio / fuseFAILFIO Random Write test failedg-fuse-buffered-io-eio
FIO Random Read Test (256KB blocks)fio / fuseFAILFIO Random Read test failedg-fuse-buffered-io-eio
FIO Mixed Random Read/Write Test (256KB blocks, 70% read, 30% write)fio / fuseFAILFIO Mixed Random Read/Write test failedg-fuse-buffered-io-eio
fusefuse / fuseFAILED“status”: “failed”g-fuse-buffered-io-eio

Failed case reconciliation

ProfileReported failuresSource failuresDelta
daily101
fuse660

Common root-cause groups

g-daily-coverage-timeout

  • Model class: flaky; confidence: medium
  • Recommendation: Increase coverage collection timeout or isolate llvm-cov rerun from runner load; retry coverage on transient timeout; investigate whether ordinary runner compile pressure during parallel crate builds causes coverage step to exceed its limit.

g-fuse-buffered-io-eio

  • Model class: pre_existing_failure; confidence: high
  • Recommendation: Review e01cf918 changes in curvine-fuse/src/cli/mount_args.rs and crates/common/curvine-config/src/fuse_conf.rs ClientConfCliOverrides generation; compare effective mount flags against eb76f94a and restore correct buffered I/O behavior.

Follow-up

Defects and fixes

  • GitHub Issue: 1646 (open, reused)

Risks

  • A subset of green profiles does not override a full-chain NO-GO.
  • Report-only performance regressions still require human follow-up.

Next actions

  • Increase coverage collection timeout or isolate llvm-cov rerun from runner load; retry coverage on transient timeout; investigate whether ordinary runner compile pressure during parallel crate builds causes coverage step to exceed its limit.
  • Review e01cf918 changes in curvine-fuse/src/cli/mount_args.rs and crates/common/curvine-config/src/fuse_conf.rs ClientConfCliOverrides generation; compare effective mount flags against eb76f94a and restore correct buffered I/O behavior.
  • Rerun the full chain after fixes and require all release gates to pass.

Publication note

This public report contains only sanitized test outcomes. Private execution evidence remains internal.