k8s Deployment

This chapter introduces how to deploy Curvine distributed storage cluster on Kubernetes using production-grade Helm Chart.

Prerequisites

Kubernetes 1.20+
Helm 3.0+
PV provisioner (if using PVC storage)

Quick Start

1. Load Images into Kubernetes Cluster

Refer to docker compile to build images and import them into the Kubernetes cluster.

2. Add Helm Repository (Optional)

# If Chart is published to repository
helm repo add curvine https://curvineio.github.io/helm-charts
helm repo update

3. Install Chart

Option A: Install from Helm Repository (Recommended)

Note: The current Helm version provided is based on the main branch and is intended for pre-release use only. To install, you must specify the --devel flag

# Install with default configuration
helm install curvine curvine/curvine -n curvine --create-namespace --devel

# Install with custom replica count
helm install curvine curvine/curvine -n curvine --create-namespace --devel \
  --set master.replicas=5 \
  --set worker.replicas=10

# Install with custom values file
helm install curvine curvine/curvine -n curvine --create-namespace --devel \
  -f https://curvineio.github.io/helm/charts/examples/values-prod.yaml

Option B: Install from Local Chart

Note: Run these commands from the helm-charts directory (parent directory of curvine-runtime folder)

# Install with default configuration
helm install curvine ./curvine-runtime -n curvine --create-namespace

# Install with custom replica count
helm install curvine ./curvine-runtime -n curvine --create-namespace \
  --set master.replicas=5 \
  --set worker.replicas=10

# Install with custom values file
helm install curvine ./curvine-runtime -n curvine --create-namespace \
  -f ./curvine-runtime/examples/values-prod.yaml

4. Verify Deployment

# Check Pod status
kubectl get pods -n curvine

# View Services
kubectl get svc -n curvine

# View PersistentVolumeClaims
kubectl get pvc -n curvine

# Run Helm tests
helm test curvine -n curvine

5. Access Cluster

# Port forward to access Master Web UI
kubectl port-forward -n curvine svc/curvine-master 9000:9000

# Access http://localhost:9000

Configuration

View Current Configuration

# View all current values
helm get values curvine -n curvine

# View specific version values in YAML format
helm get values curvine -n curvine -o yaml

# View rendered manifests
helm get manifest curvine -n curvine

# View Chart's values.yaml
cat ./curvine-runtime/values.yaml

# View specific parameters
helm get values curvine -n curvine | grep master.replicas

Common Parameter Usage Examples

Adjust Resource Limits

# Increase Master resources for high-load scenarios
helm install curvine ./curvine-runtime -n curvine --create-namespace \
  --set master.resources.requests.cpu=2000m \
  --set master.resources.requests.memory=4Gi \
  --set master.resources.limits.cpu=4000m \
  --set master.resources.limits.memory=8Gi

Configure Node Affinity

# Run Master on specific nodes
helm install curvine ./curvine-runtime -n curvine --create-namespace \
  --set 'master.nodeSelector.node-type=master' \
  --set 'worker.nodeSelector.node-type=worker'

Enable Worker Privileged Mode

# Enabled by default, but can be disabled if needed
helm install curvine ./curvine-runtime -n curvine --create-namespace \
  --set worker.privileged=false

Configure Multiple Data Directories for Worker

# Create values-multi-data.yaml with the following content:
# worker:
#   storage:
#     dataDirs:
#       - name: "data1"
#         type: "SSD"
#         enabled: true
#         size: "100Gi"
#         storageClass: "fast-ssd"
#         mountPath: "/data/data1"
#       - name: "data2"
#         type: "HDD"
#         enabled: true
#         size: "500Gi"
#         storageClass: "slow-hdd"
#         mountPath: "/data/data2"

helm install curvine ./curvine-runtime -n curvine --create-namespace \
  -f values-multi-data.yaml

Adjust Log Level

# Set log level to DEBUG for troubleshooting
helm install curvine ./curvine-runtime -n curvine --create-namespace \
  --set config.log.level=DEBUG

Configure External Master Service

# Expose Master Service via LoadBalancer
helm install curvine ./curvine-runtime -n curvine --create-namespace \
  --set service.masterExternal.enabled=true \
  --set service.masterExternal.type=LoadBalancer

Configuration Parameters

Global Parameters

Parameter	Description	Default Value
global.clusterDomain	Kubernetes cluster domain	cluster.local

Cluster Parameters

Parameter	Description	Default Value
cluster.id	Cluster identifier	curvine
cluster.formatMaster	Format Master data on startup	false
cluster.formatWorker	Format Worker data on startup	false
cluster.formatJournal	Format journal data on startup	false

Image Configuration

Parameter	Description	Default Value
image.repository	Container image repository	docker.io/curvine
image.tag	Container image tag	latest
image.pullPolicy	Image pull policy	IfNotPresent
image.pullSecrets	Image pull secrets	[]

Master Configuration

Parameter	Description	Default Value
master.replicas	Master replica count (must be odd: 1, 3, 5, 7...)	3
master.rpcPort	RPC port	8995
master.journalPort	Journal/Raft port	8996
master.webPort	Web UI port	9000
master.web1Port	Additional Web port	9001
master.storage.meta.enabled	Enable metadata storage	true
master.storage.meta.storageClass	Metadata storage class	"" (default)
master.storage.meta.size	Metadata storage size	10Gi
master.storage.meta.hostPath	Metadata host path (used when no storageClass)	""
master.storage.meta.mountPath	Metadata mount path	/opt/curvine/data/meta
master.storage.journal.enabled	Enable journal storage	true
master.storage.journal.storageClass	Journal storage class	"" (default)
master.storage.journal.size	Journal storage size	50Gi
master.storage.journal.hostPath	Journal host path (used when no storageClass)	""
master.storage.journal.mountPath	Journal mount path	/opt/curvine/data/journal
master.resources.requests.cpu	CPU request	1000m
master.resources.requests.memory	Memory request	2Gi
master.resources.limits.cpu	CPU limit	2000m
master.resources.limits.memory	Memory limit	4Gi
master.nodeSelector	Node selector labels
master.tolerations	Pod tolerations	[]
master.affinity	Pod affinity rules
master.labels	Additional labels
master.annotations	Additional annotations
master.extraEnv	Additional environment variables	[]
master.extraVolumes	Additional volumes	[]
master.extraVolumeMounts	Additional volume mounts	[]

Worker Configuration

Parameter	Description	Default Value
worker.replicas	Worker replica count	3
worker.rpcPort	RPC port	8997
worker.webPort	Web UI port	9001
worker.hostNetwork	Use host network	false
worker.dnsPolicy	DNS policy	ClusterFirst
worker.privileged	Privileged mode (required for FUSE)	true
worker.storage.dataDirs[0].name	Data directory name	data1
worker.storage.dataDirs[0].type	Storage type (SSD/HDD)	SSD
worker.storage.dataDirs[0].enabled	Enable data directory	true
worker.storage.dataDirs[0].size	Data directory size	10Gi
worker.storage.dataDirs[0].storageClass	Storage class	"" (default)
worker.storage.dataDirs[0].hostPath	Host path (used when no storageClass)	""
worker.storage.dataDirs[0].mountPath	Mount path	/data/data1
worker.resources.requests.cpu	CPU request	2000m
worker.resources.requests.memory	Memory request	4Gi
worker.resources.limits.cpu	CPU limit	4000m
worker.resources.limits.memory	Memory limit	8Gi
worker.nodeSelector	Node selector labels
worker.tolerations	Pod tolerations	[]
worker.antiAffinity.enabled	Enable Pod anti-affinity	true
worker.antiAffinity.type	Anti-affinity type (required/preferred)	preferred
worker.labels	Additional labels
worker.annotations	Additional annotations
worker.extraEnv	Additional environment variables	[]
worker.extraVolumes	Additional volumes	[]
worker.extraVolumeMounts	Additional volume mounts	[]

Service Configuration

Parameter	Description	Default Value
service.master.type	Master Service type	ClusterIP
service.master.annotations	Master Service annotations
service.masterExternal.enabled	Enable external Master Service	false
service.masterExternal.type	External Service type	ClusterIP
service.masterExternal.annotations	External Service annotations
service.masterExternal.nodePort	NodePort configuration
service.masterExternal.loadBalancerIP	LoadBalancer IP	""
service.masterExternal.loadBalancerSourceRanges	LoadBalancer source ranges	[]

Service Account & RBAC

Parameter	Description	Default Value
serviceAccount.create	Create Service Account	true
serviceAccount.name	Service Account name	"" (auto-generated)
serviceAccount.annotations	Service Account annotations
rbac.create	Create RBAC resources	true

Curvine Configuration

Parameter	Description	Default Value
config.master.metaDir	Master metadata directory	/opt/curvine/data/meta
config.journal.enable	Enable journal	true
config.journal.journalDir	Journal directory	/opt/curvine/data/journal
config.client.blockSizeStr	Client block size	64MB
config.log.level	Log level (INFO/DEBUG/WARN/ERROR)	INFO
config.log.logDir	Log directory	/opt/curvine/logs
configOverrides.master	Master configuration override
configOverrides.journal	Journal configuration override
configOverrides.worker	Worker configuration override
configOverrides.client	Client configuration override
configOverrides.log	Log configuration override

For the complete parameter list, please refer to values.yaml.

Configuration Examples

Development Environment (Minimal)

# Install from Helm repository
helm install curvine curvine/curvine -n curvine --create-namespace --devel \
  --set master.replicas=1 \
  --set worker.replicas=1

# Install from local Chart (run in helm-charts directory)
helm install curvine ./curvine-runtime -n curvine --create-namespace \
  -f ./curvine-runtime/examples/values-dev.yaml

Production Environment (High Availability)

# Install from Helm repository
helm install curvine curvine/curvine -n curvine --create-namespace --devel \
  --set master.replicas=5 \
  --set worker.replicas=10 \
  --set master.storage.meta.storageClass=fast-ssd \
  --set master.storage.journal.storageClass=fast-ssd

# Install from local Chart (run in helm-charts directory)
helm install curvine ./curvine-runtime -n curvine --create-namespace \
  -f ./curvine-runtime/examples/values-prod.yaml

Bare Metal Environment (Using hostPath)

# Install from local Chart (run in helm-charts directory)
helm install curvine ./curvine-runtime -n curvine --create-namespace \
  -f ./curvine-runtime/examples/values-baremetal.yaml

Custom Configuration

# Install from Helm repository
helm install curvine curvine/curvine -n curvine --create-namespace --devel \
  --set master.replicas=5 \
  --set worker.replicas=10 \
  --set master.storage.meta.storageClass=fast-ssd \
  --set worker.storage.dataDirs[0].storageClass=fast-ssd \
  --set worker.storage.dataDirs[0].size=500Gi

# Install from local Chart (run in helm-charts directory)
helm install curvine ./curvine-runtime -n curvine --create-namespace \
  --set master.replicas=5 \
  --set worker.replicas=10 \
  --set master.storage.meta.storageClass=fast-ssd \
  --set worker.storage.dataDirs[0].storageClass=fast-ssd \
  --set worker.storage.dataDirs[0].size=500Gi

Storage Configuration

Using PVC (Recommended for Cloud Environments)

master:
  storage:
    meta:
      storageClass: "fast-ssd"
      size: "20Gi"
    journal:
      storageClass: "fast-ssd"
      size: "100Gi"

worker:
  storage:
    dataDirs:
      - name: "data1"
        type: "SSD"
        enabled: true
        size: "100Gi"
        storageClass: "fast-ssd"
        mountPath: "/data/data1"

Using hostPath (Recommended for Bare Metal)

master:
  storage:
    meta:
      storageClass: ""
      hostPath: "/mnt/curvine/master/meta"
    journal:
      storageClass: ""
      hostPath: "/mnt/curvine/master/journal"

worker:
  storage:
    dataDirs:
      - name: "data1"
        type: "SSD"
        enabled: true
        size: "100Gi"
        storageClass: ""
        hostPath: "/mnt/nvme0n1/curvine"
        mountPath: "/data/data1"

Using emptyDir (For Testing)

master:
  storage:
    meta:
      storageClass: ""
      hostPath: ""
    journal:
      storageClass: ""
      hostPath: ""

worker:
  storage:
    dataDirs:
      - name: "data1"
        storageClass: ""
        hostPath: ""

Storage Configuration Examples

Quick Start with Default PVC

# Use default storage class (fastest startup method)
helm install curvine ./curvine-runtime -n curvine --create-namespace

Cloud Environment Fast SSD Configuration

# AWS/GCP/Azure fast SSD storage
helm install curvine ./curvine-runtime -n curvine --create-namespace \
  --set master.storage.meta.storageClass=fast-ssd \
  --set master.storage.journal.storageClass=fast-ssd \
  --set 'worker.storage.dataDirs[0].storageClass=fast-ssd' \
  --set 'worker.storage.dataDirs[0].size=500Gi'

Bare Metal Multi-Storage Type Configuration

# Create values-baremetal-multi.yaml:
# master:
#   storage:
#     meta:
#       storageClass: ""
#       hostPath: "/mnt/nvme/master/meta"
#     journal:
#       storageClass: ""
#       hostPath: "/mnt/nvme/master/journal"
# 
# worker:
#   storage:
#     dataDirs:
#       - name: "nvme"
#         type: "SSD"
#         enabled: true
#         size: "200Gi"
#         storageClass: ""
#         hostPath: "/mnt/nvme/worker"
#         mountPath: "/data/nvme"
#       - name: "ssd"
#         type: "SSD"
#         enabled: true
#         size: "500Gi"
#         storageClass: ""
#         hostPath: "/mnt/ssd/worker"
#         mountPath: "/data/ssd"
#       - name: "hdd"
#         type: "HDD"
#         enabled: true
#         size: "2000Gi"
#         storageClass: ""
#         hostPath: "/mnt/hdd/worker"
#         mountPath: "/data/hdd"

helm install curvine ./curvine-runtime -n curvine --create-namespace \
  -f values-baremetal-multi.yaml

Hybrid Cloud and Local Storage

# Master uses cloud PVC, Worker uses local hostPath
helm install curvine ./curvine-runtime -n curvine --create-namespace \
  --set master.storage.meta.storageClass=cloud-ssd \
  --set master.storage.journal.storageClass=cloud-ssd \
  --set 'worker.storage.dataDirs[0].storageClass=""' \
  --set 'worker.storage.dataDirs[0].hostPath=/mnt/local/data'

Upgrade

Update Configuration

# Scale Worker replicas (from Helm repository)
helm upgrade curvine curvine/curvine -n curvine --devel \
  --set worker.replicas=15

# Upgrade image version (from Helm repository)
helm upgrade curvine curvine/curvine -n curvine --devel \
  --set image.tag=v1.1.0

# Upgrade with new values file (from local Chart, run in helm-charts directory)
helm upgrade curvine ./curvine-runtime -n curvine \
  -f ./curvine-runtime/values-new.yaml

Note:

Master replica count and journal storage class cannot be modified during upgrades. To modify, delete and redeploy the cluster.

Parameters not modified during upgrade will be reset to default configuration. If Master replica count and journal storage class were modified during installation, these two parameters need to be included during updates.

Common Upgrade Scenarios

Scale Worker Nodes

# Increase Worker replicas from 3 to 10
helm upgrade curvine ./curvine-runtime -n curvine \
  --set worker.replicas=10

Increase Resource Limits

# Increase Master resources for better performance
helm upgrade curvine ./curvine-runtime -n curvine \
  --set master.resources.limits.cpu=4000m \
  --set master.resources.limits.memory=8Gi

Update Image Version

# Upgrade to new Curvine version
helm upgrade curvine ./curvine-runtime -n curvine \
  --set image.tag=v1.2.0

Enable Debug Logging

# Temporarily enable debug logging for troubleshooting
helm upgrade curvine ./curvine-runtime -n curvine \
  --set config.log.level=DEBUG

Change Storage Configuration

# Migrate to faster storage class
helm upgrade curvine ./curvine-runtime -n curvine \
  --set master.storage.meta.storageClass=ultra-ssd \
  --set master.storage.journal.storageClass=ultra-ssd

View Release History

helm history curvine -n curvine

Rollback

# Rollback to previous version
helm rollback curvine -n curvine

# Rollback to specific version
helm rollback curvine 2 -n curvine

Uninstall

# Uninstall Chart (keep PVC)
helm uninstall curvine -n curvine

# Delete PersistentVolumeClaims
kubectl delete pvc -n curvine -l app.kubernetes.io/instance=curvine

# Delete namespace
kubectl delete namespace curvine

Troubleshooting

Check Pod Status

kubectl get pods -n curvine
kubectl describe pod <pod-name> -n curvine
kubectl logs <pod-name> -n curvine

View ConfigMap

kubectl get configmap -n curvine
kubectl describe configmap curvine-config -n curvine

View Events

kubectl get events -n curvine --sort-by='.lastTimestamp'

Common Issues

Master Replica Validation Failed
1. Error: master.replicas must be an odd number
2. Solution: Ensure Master replica count is odd (1, 3, 5, 7...)
PVC Cannot Bind
1. Check if StorageClass exists
2. Verify PV provisioner is working properly
Pod Startup Failed
1. Verify container image exists
2. Check if resource quotas are sufficient
3. View Pod logs for details

Prerequisites​

Quick Start​

1. Load Images into Kubernetes Cluster​

2. Add Helm Repository (Optional)​

3. Install Chart​

Option A: Install from Helm Repository (Recommended)​

Option B: Install from Local Chart​

4. Verify Deployment​

5. Access Cluster​

Configuration​

View Current Configuration​

Common Parameter Usage Examples​

Adjust Resource Limits​

Configure Node Affinity​

Enable Worker Privileged Mode​

Configure Multiple Data Directories for Worker​

Adjust Log Level​

Configure External Master Service​

Configuration Parameters​

Global Parameters​

Cluster Parameters​

Image Configuration​

Master Configuration​

Worker Configuration​

Service Configuration​

Service Account & RBAC​

Curvine Configuration​

Configuration Examples​

Development Environment (Minimal)​

Production Environment (High Availability)​

Bare Metal Environment (Using hostPath)​

Custom Configuration​

Storage Configuration​

Using PVC (Recommended for Cloud Environments)​

Using hostPath (Recommended for Bare Metal)​

Using emptyDir (For Testing)​

Storage Configuration Examples​

Quick Start with Default PVC​

Cloud Environment Fast SSD Configuration​

Bare Metal Multi-Storage Type Configuration​

Hybrid Cloud and Local Storage​

Upgrade​

Update Configuration​

Common Upgrade Scenarios​

Scale Worker Nodes​

Increase Resource Limits​

Update Image Version​

Enable Debug Logging​

Change Storage Configuration​

View Release History​

Rollback​

Uninstall​

Troubleshooting​

Check Pod Status​

View ConfigMap​

View Events​

Common Issues​