Best Practices

This guide consolidates production-proven recommendations for deploying, configuring, and operating Curvine across common workload scenarios. Each section is self-contained — apply the practices relevant to your environment.

1. Deployment Best PracticesCopy Anchor Link

1.1 Cluster Sizing GuidelinesCopy Anchor Link

1.2 High AvailabilityCopy Anchor Link

• Always deploy 3 Master nodes in production to form a Raft group. A single Master is a single point of failure.
• Set journal_addrs to list all Raft peers explicitly:

[journal]
journal_addrs = [
  { id = 1, hostname = "master1", port = 8996 },
  { id = 2, hostname = "master2", port = 8996 },
  { id = 3, hostname = "master3", port = 8996 },
]

• Place Master nodes on separate physical hosts or availability zones to survive rack/zone failures.
• Use snapshot_interval = "6h" (default) to keep Raft log size bounded.

1.3 Production Configuration ChecklistCopy Anchor Link

# Disable format flags in production
format_master = false
format_worker = false

[master]
meta_dir = "/data/curvine-meta"   # Use dedicated SSD for metadata

[worker]
data_dir = [
  "[SSD]/data/ssd1",
  "[SSD]/data/ssd2",
  "[HDD]/data/hdd1"
]

[client]
master_addrs = [
  { hostname = "master1", port = 8995 },
  { hostname = "master2", port = 8995 },
  { hostname = "master3", port = 8995 },
]

• Set CURVINE_MASTER_HOSTNAME, CURVINE_WORKER_HOSTNAME, and CURVINE_CLIENT_HOSTNAME to routable addresses in multi-node environments.
• Never use localhost for any hostname in distributed deployments.

1.4 Kubernetes DeploymentCopy Anchor Link

• Use the official Helm chart for K8s environments:

helm repo add curvine https://curvineio.github.io/helm-charts
helm upgrade --install curvine curvine/curvine \
  --version 0.3.2-alpha \
  -n curvine --create-namespace \
  --wait --timeout 10m

• Use ReadWriteMany PVCs with the CSI driver for shared data access across pods.
• For latency-sensitive workloads, prefer embedded mount mode; for FUSE isolation, use standalone mode.

1.5 Kernel and OS TuningCopy Anchor Link

# Increase file descriptor limit
ulimit -n 128000

# Optimize TCP buffers for high-throughput data transfer
sysctl -w net.ipv4.tcp_rmem="8192 128000 33554432"
sysctl -w net.ipv4.tcp_wmem="8192 128000 33554432"
sysctl -w net.core.rmem_max=67108864
sysctl -w net.core.wmem_max=67108864

• Use Linux kernel 5.0+ (e.g., Ubuntu 22.04, Rocky Linux 9) for optimal FUSE performance.
• Avoid kernels below 4.15 in production — FUSE concurrency is severely limited.

2. Mount Mode SelectionCopy Anchor Link

Choosing the right mount mode is the single most impactful decision for your workload.

2.1 Decision MatrixCopy Anchor Link

2.2 cache_mode Best PracticesCopy Anchor Link

bin/cv mount \
  s3://bucket/analytics /analytics \
  --write-type cache_mode \
  --ttl-ms 7d \
  --read-verify-ufs false \
  -s ssd \
  -c s3.endpoint_url=https://s3.example.com \
  -c s3.credentials.access=YOUR_KEY \
  -c s3.credentials.secret=YOUR_SECRET \
  -c s3.region_name=us-east-1

• Enable --read-verify-ufs true only when UFS data changes frequently and staleness is unacceptable. It adds one extra UFS HEAD request per read.
• Use bin/cv fs free -r /path to release cache space safely without deleting UFS data.
• Warning: delete in cache_mode deletes UFS data. Use free for cache-only cleanup.

2.3 fs_mode Best PracticesCopy Anchor Link

bin/cv mount \
  s3://bucket/workspace /workspace \
  --write-type fs_mode \
  --ttl-ms 30d \
  --replicas 2 \
  --block-size 128M \
  -s ssd \
  -c s3.endpoint_url=https://s3.example.com \
  -c s3.credentials.access=YOUR_KEY \
  -c s3.credentials.secret=YOUR_SECRET

• Run bin/cv mount resync /workspace after external UFS changes to synchronize metadata.
• Never rename large directories on S3/OSS backends — it triggers full data copy.
• Set --replicas 2 or higher for critical data to survive Worker failures.

3. Tiered Storage StrategyCopy Anchor Link

3.1 Configure Multi-Tier WorkersCopy Anchor Link

[worker]
data_dir = [
  "[MEM]/dev/shm/curvine",
  "[SSD]/data/nvme0/curvine",
  "[SSD]/data/nvme1/curvine",
  "[HDD]/data/hdd0/curvine"
]

3.2 Match Storage Tier to Data TemperatureCopy Anchor Link

3.3 Client-Side Tier PreferenceCopy Anchor Link

[client]
storage_type = "ssd"   # Preferred write tier; auto-fallback to next tier on space exhaustion

4. Performance TuningCopy Anchor Link

4.1 Server-Side TuningCopy Anchor Link

[master]
io_threads = 32          # Match CPU core count
worker_threads = 64      # 2x CPU cores for mixed workloads

[worker]
io_threads = 32
worker_threads = 64
enable_send_file = true  # Zero-copy data transfer

4.2 Client-Side Tuning by WorkloadCopy Anchor Link

Sequential read (large files > 100GB):

[client]
read_parallel = 8          # 4-16 recommended
read_chunk_size = "256KB"

High-concurrency small files (> 1000 concurrent):

[client]
read_chunk_num = 1
write_chunk_num = 1        # Reduces memory by ~90%

Random read:

[client]
read_chunk_num = 1
read_chunk_size = "256KB"  # Match average I/O size
# For very large files, use block_size >= 1GB at write time to reduce connections

4.3 FUSE OptimizationCopy Anchor Link

• Always use FUSE3 in production — FUSE2 delivers only ~50% of FUSE3 throughput.
• If stuck on FUSE2, use multiple mount points:

[fuse]
mnt_number = 4   # 4-8 mount points ≈ 80% of FUSE3 performance

• For memory-constrained FUSE nodes:

[fuse]
fuse_channel_size = 1024
stream_channel_size = 16

• Curvine FUSE is fully asynchronous — attr_timeout and entry_timeout are optional. Increase them for read-heavy workloads with stable metadata:

[fuse]
entry_timeout = 60
attr_timeout = 60

• Set entry_timeout = 0 and attr_timeout = 0 if you need real-time visibility of changes from other clients.

5. Cache ManagementCopy Anchor Link

5.1 TTL StrategyCopy Anchor Link

5.2 Quota-Based EvictionCopy Anchor Link

Enable LRU eviction to prevent storage exhaustion:

[master]
enable_quota_eviction = true
quota_eviction_high_rate = 0.8   # Start eviction at 80% usage
quota_eviction_low_rate = 0.6    # Stop at 60% usage
quota_eviction_policy = "lru"

• Files with ttl_action = "none" are exempt from LRU eviction — use this for critical data.

5.3 Data PreloadingCopy Anchor Link

Warm up the cache before peak workloads:

# Proactive load (async by default)
bin/cv load s3://bucket/datasets/imagenet

# Check load job status
bin/cv load-status <job-id>

• Automatic caching is enabled per-mount when TTL > 0. First reads trigger background load jobs.
• Combine proactive + automatic caching for best hit ratio.

6. AI/ML Workload OptimizationCopy Anchor Link

6.1 Training Data AccessCopy Anchor Link

# Mount immutable training dataset with memory-tier caching
bin/cv mount \
  s3://datasets/imagenet /datasets/imagenet \
  --write-type cache_mode \
  --storage-type mem \
  --block-size 1GB \
  --replicas 2 \
  --ttl-ms 30d

• Use FUSE mount point directly in PyTorch/TensorFlow data loaders — no code changes needed.
• Set num_workers=8 in DataLoader for parallel FUSE reads.
• Pre-load datasets before training starts: bin/cv load s3://datasets/imagenet -w

6.2 Model ServingCopy Anchor Link

# Low-latency model file access
bin/cv mount \
  s3://models/bert-large /models/bert-large \
  --write-type cache_mode \
  --storage-type mem \
  --ttl-ms 0 \
  --read-verify-ufs true

• Use --ttl-ms 0 (no expiration) for model files that must always be available.
• Enable --read-verify-ufs to detect model version updates pushed to UFS.

6.3 Checkpoint StorageCopy Anchor Link

# Training checkpoints with fs_mode for POSIX write semantics
bin/cv mount \
  s3://checkpoints/run-001 /checkpoints/run-001 \
  --write-type fs_mode \
  --storage-type ssd \
  --replicas 2 \
  --ttl-ms 7d

7. Big Data Ecosystem IntegrationCopy Anchor Link

7.1 Spark IntegrationCopy Anchor Link

spark-submit \
  --conf spark.hadoop.fs.cv.impl=io.curvine.CurvineFileSystem \
  --conf spark.hadoop.fs.cv.master_addrs=master1:8995,master2:8995,master3:8995 \
  --jars curvine-hadoop-client.jar \
  --class com.example.App app.jar

• For zero-code acceleration, use the S3A transparent proxy:

<property>
  <name>fs.s3a.impl</name>
  <value>io.curvine.S3AProxyFileSystem</value>
</property>

Mounted paths automatically use Curvine cache; unmounted paths fall through to native S3.

7.2 Shuffle AccelerationCopy Anchor Link

• Use fs_mode mounts for shuffle data to benefit from full write acceleration.
• Set --storage-type ssd for shuffle — memory is too expensive for ephemeral data.
• Use short TTL (1h-4h) since shuffle data is transient.

7.3 Block Size SelectionCopy Anchor Link

8. Monitoring and ObservabilityCopy Anchor Link

8.1 Metrics EndpointsCopy Anchor Link

8.2 Key Metrics to Alert OnCopy Anchor Link

8.3 Prometheus + Grafana SetupCopy Anchor Link

# prometheus.yml scrape config
scrape_configs:
  - job_name: 'curvine-master'
    static_configs:
      - targets: ['master1:9001', 'master2:9001', 'master3:9001']
  - job_name: 'curvine-worker'
    static_configs:
      - targets: ['worker1:9002', 'worker2:9002', 'worker3:9002']
  - job_name: 'curvine-fuse'
    static_configs:
      - targets: ['app1:9003', 'app2:9003']

8.4 Web UICopy Anchor Link

• Master UI: http://<master-host>:9001 — cluster overview, file browser, worker status, configuration.
• Worker UI: http://<worker-host>:9002 — node-level storage and block statistics.

9. Transfer Service Best PracticesCopy Anchor Link

The S3 gateway is not part of the current Curvine server. Use the optional Transfer service for independent Load and Export scheduling instead.

9.1 High AvailabilityCopy Anchor Link

• Use a shared MySQL store_url before running more than one Transfer replica.
• The default SQLite store is suitable for a single Transfer instance only.
• Keep transfer.replicas = 1 when SQLite is in use.

9.2 Capacity and RetentionCopy Anchor Link

• Tune max_running_transfers to match available UFS bandwidth.
• Set terminal_retention according to how long completed jobs must remain queryable.
• Monitor the Transfer /metrics endpoint and the cv transfer list backlog.

curl http://<transfer-host>:9011/metrics
cv transfer list --state running

10. Common Pitfalls and TroubleshootingCopy Anchor Link

11. Security RecommendationsCopy Anchor Link

• Rotate UFS credentials regularly; use IAM roles where possible.
• Use network policies (K8s) or firewall rules to restrict access to Master RPC (8995) and Worker RPC (8997) ports.
• Enable audit logging (audit_logging_enabled = true) for compliance-sensitive environments.
• In multi-tenant environments, isolate namespaces by using separate Curvine paths per team with quota controls.

12. Upgrade and MaintenanceCopy Anchor Link

• Always take a metadata snapshot before upgrading Master nodes.
• Perform rolling upgrades: upgrade Followers first, then trigger leader transfer and upgrade the last node.
• After upgrading Workers, verify heartbeat registration via the Web UI or bin/cv report.
• Test FUSE remount after upgrades: bin/curvine-fuse.sh restart.
• Keep retain_checkpoint_num = 3 (default) to allow rollback to previous metadata states.