Curvine is an AI-Native and Cloud-Native distributed cache file system, written entirely in Rust. Born at OPPO and now a CNCF Landscape project, it layers full POSIX semantics over cloud object storage — delivering local-disk speed for AI training, inference, and big data workloads while keeping S3, OSS, GCS, Azure Blob, and HDFS as the durable backbone.

This post walks through Curvine's architecture at three levels: the overall system context, the container-level service breakdown, and the end-to-end data flow for read and write operations.

Introduction​

Over the past two years, multimodal data lakes have become a standard answer for AI infrastructure: images, videos, text, and embeddings all live in object storage; columnar formats such as Lance and Parquet provide unified data management; and engines such as LanceDB run vector, full-text, and hybrid search directly on the lake. Compute and storage are decoupled, capacity is virtually unlimited, and costs stay low. It sounds almost perfect.

But teams that move online retrieval workloads onto this architecture encounter the same problem: object storage has plenty of capacity and bandwidth, but not low latency. A single vector-retrieval request may trigger dozens or hundreds of small random reads. Each read pays the object's time-to-first-byte penalty, and the longer the path, the worse the tail latency becomes.

We ran a benchmark on Alibaba Cloud using the same LanceDB version, the same one-million-row dataset of 1,536-dimensional vectors, and the same benchmark scripts. The only change was replacing direct OSS access with Curvine, a high-performance distributed cache system written in Rust. Vector-search p50 latency fell from 11 ms to 2 ms, QPS rose from 69 to 333, and p99 latency dropped from 62 ms to 10 ms.

This article presents the complete results and analysis, including the scenario where the improvement was limited—and why.

1. Where Is the Bottleneck? The I/O Profile of Vector Search​

A vector query over a data lake roughly follows these steps:

1. Read index metadata, such as IVF centroids and partition offsets.
2. Read several index fragments from the partitions selected by nprobes.
3. After obtaining candidate row IDs, read the original columns—vectors, title, text, and so on—to retrieve actual values for reranking.
4. Full-text search also reads inverted-index structures. Hybrid search executes both paths and then applies Reciprocal Rank Fusion (RRF).

This access pattern consists of small, highly fanned-out, random reads. Object storage delivers excellent throughput, but every range read incurs a fixed cost for a network round trip and server-side lookup, typically several to tens of milliseconds. When one query chains together dozens of these reads, fixed overhead dominates latency. Adding machines or bandwidth does not help because the bottleneck is not bandwidth; it is the inherent latency of each I/O operation.

This is exactly where Curvine fits. Curvine is a distributed cache file system that places hot data in Worker memory and on local NVMe or ESSD devices. Clients access it through native RPC, FUSE, or a Hadoop-compatible interface. Object storage remains the system of record, but each random read on the hot path changes from "access object storage over the network" to "access a cache node in the same availability zone plus a local disk." This reduces per-I/O latency by an order of magnitude.

2. Test Environment and Methodology​

To avoid the appearance of writing a benchmark tailored to our own system, we used LanceDB's official benchmark suite.

The cluster was deployed on Alibaba Cloud:

• Curvine test cluster: one Master and one Worker, both ecs.r8a.8xlarge instances with 32 vCPUs and 247 GB of memory. The Worker had an additional 1.3 TB ESSD cloud disk.
• Query node: one ecs.r8a.4xlarge instance with 16 vCPUs and 123 GB of memory.

This was a minimal, single-Worker cluster without multi-replica or multi-node parallel-read optimization. In other words, the results below represent a lower bound for Curvine, not its ceiling.

We tested four query types that cover common multimodal-retrieval patterns:

• vector: search using random 1,536-dimensional vectors.
• fts: randomly select terms from a built-in vocabulary and run full-text search on title.
• hybrid: combine a random vector and random text, followed by RRF reranking.
• vector_with_filter: combine a random vector with text LIKE '%term%' and prefilter=True.

The core variable was the storage path:

• oss: data and indexes both reside directly in OSS.
• curvine-index: data remains at its OSS URI, while only the index directory (_indices) is placed in Curvine's cache.
• curvine: both data and indexes use curvine:// and are served through the cache.

The second mode is especially important: it changes only the index location and barely affects the organization of an existing data lake. For many teams, it is the easiest way to get started.

3. Benchmark Results​

All latency values are in milliseconds. Higher QPS is better; lower latency percentiles are better.

3.1 Vector: Pure Vector Search​

With the full Curvine path, p50 latency fell to one-fifth-and-a-half of the OSS result, while QPS improved by approximately 4.8x. More importantly, p90 fell from 26 ms to 3 ms—about 8.7x lower—and p99 dropped from 62 ms to 10 ms. A better average is useful; flattening tail latency is what makes a system ready for production.

3.2 FTS: Full-Text Search​

Full-text search produced the most interesting benefit profile. Moving only the index into Curvine increased QPS by 2.3x and more than halved p50 latency. The reason is straightforward: accesses to an inverted index are more fragmented than accesses to a vector index and are therefore more sensitive to per-I/O latency. Accelerating only the index captures much of the benefit. The full Curvine path reduced p50 to 5 ms and p99 from 88 ms to 15 ms.

3.3 Hybrid: Vector and Full-Text Search with RRF Reranking​

Hybrid search is closest to a real multimodal workload. It pays the I/O cost of both the vector and full-text paths, so its OSS baseline is the worst, with a p50 of 36 ms. Curvine reduced p50 to 13 ms—approximately 2.8x lower—and increased QPS by 2.9x.

The latency distribution is the standout result: 13 ms at p50, 14 ms at p90, 15 ms at p95, and 21 ms at p99. The curve is nearly flat. With OSS, the same percentiles rise continuously from 36 to 58, 66, and 90 ms. For a retrieval service embedded in a RAG pipeline, predictable latency is often more valuable than a lower average alone.

3.4 Vector with Filter​

We include this result as measured: curvine-index provided almost no improvement over OSS, while the full Curvine path improved performance by only about 1.5x.

The reason is that this query has a completely different cost profile. With prefilter=True, text LIKE '%term%' first performs a full scan and match over the text column, then feeds the result into vector search. Overall p50 latency is approximately 250–380 ms. The bottleneck lies in filtering and scan computation, not index I/O. Caching can optimize only a small part of the total runtime, so it cannot produce a 5x gain.

This result reinforces the credibility of the first three tests. Curvine accelerates I/O latency: I/O-bound scenarios benefit greatly, while CPU- or scan-bound scenarios see limited gains. The appropriate solution for this kind of filtered query is to build a scalar index on the filter field or replace LIKE with an inverted index—not to keep adding cache capacity.

4. Summary: Improvement over OSS​

Four conclusions stand out:

1. The full Curvine path performed best across all query types. Vector and full-text search benefited most: p50 fell to roughly one-fifth of the OSS result, while QPS improved by 4–5x.
2. Caching only the index still delivered meaningful gains. The effect was especially clear for full-text and hybrid search, at 1.5–2.2x, with a smaller improvement for vector search.
3. Vector search with a filter was the exception. Its bottleneck was filtering and scanning rather than index I/O, so it requires better index design instead of more caching.
4. Latency consistency improved even more than average latency. Both p90 and p99 were lower across the Curvine path, and hybrid-search latency became nearly flat across percentiles.

5. Deployment: Two Adoption Strategies​

The results suggest a practical two-stage adoption path.

Option 1: Cache Only the Index for a Low-Cost Trial​

Keep the existing OSS layout of the data lake and point only Lance's _indices directory to Curvine. This requires minimal change, no data migration, and no disruption to existing ingestion or archival workflows. It can still deliver 1.5–2.2x gains for full-text and hybrid search. Because the cache needs to cover only the index footprint, the cost is low.

This option is well suited to teams that want to validate the benefit first, or whose hot index data is much smaller than the full dataset.

Option 2: Use Curvine End to End for Maximum Performance​

Mount the complete dataset under curvine:// so that the entire read path uses the cache. This approach delivers improvements on the order of 5x and much flatter tail latency, but the cache must be large enough to hold the working set.

For latency-sensitive paths such as online retrieval services and RAG recall, this is the recommended architecture.

Curvine supports several access methods and can be adopted without changing application code. For example, mount it through FUSE and use it as a local directory:

bin/curvine-fuse.sh start
ls /curvine-fuse

Curvine also provides a native Rust API and a Hadoop-compatible interface using cv://:

Configuration conf = new Configuration();
conf.set("fs.cv.impl", "io.curvine.CurvineFileSystem");
FileSystem fs = FileSystem.get(URI.create("cv://master:8995"), conf);

6. Final Thoughts​

A multimodal data lake solves the question of where to store data, but not how quickly to read it. Object storage is economical because data is kept farther away; online retrieval feels fast when data is brought closer. A cache layer is needed to reconcile those two goals.

The main lesson from this benchmark is simple: in a disaggregated compute-and-storage architecture, reducing the latency of small random I/O often provides better value than switching to a more powerful retrieval engine or adding more query nodes. We used the same LanceDB, the same data, and the same query-node specification. Changing only the storage path delivered a 5x latency improvement, even with a minimal Curvine cluster containing just one Worker.

The test that did not improve is equally important. Caching is not a universal remedy; it targets I/O latency precisely. When the bottleneck moves to scanning and computation, it is time to use a different tool. Knowing where a technology does not help matters as much as knowing where it does.

Curvine is open source under the Apache License 2.0. Beyond accelerating vector retrieval, it is used for training-data acceleration, model distribution, hot-table acceleration, big-data shuffle acceleration, and multi-cloud caching. Try it, open an issue, or contribute:

• GitHub: github.com/CurvineIO/curvine
• Documentation: curvineio.github.io
• Quick Start: Deploy Curvine

Benchmark note: Results were measured on Alibaba Cloud with LanceDB 0.34.0 and the dbpedia-entities-openai-1M dataset. Each test group ran 10,000 queries in a single process. The Curvine deployment used a minimal one-Master, one-Worker cluster. Results will vary with hardware, dataset size, and concurrency; validate performance with your own production workload.

I. The Storage Challenge of Large-Scale AI Agent Deployment​

In 2026, AI infrastructure is undergoing a fundamental architectural shift: from a centralized model where one large model instance serves all requests, to a distributed model where thousands of Agent instances run independently.

This is not a conceptual change. Take frontend development with Vite as an example: during debugging, an Agent needs to run build tools like Vite. The Vite dev server depends on high-speed random reads of node_modules and source directories for on-demand compilation and Hot Module Replacement (HMR). Developers expect to see page changes within milliseconds after saving a file. This places high demands on small-file random read performance and low-latency file system event notifications (inotify/fswatch). If the storage layer cannot keep up, the entire development experience becomes sluggish.

Every Agent instance is not a stateless HTTP handler, but a stateful process with its own working directory and persistent storage needs. Platforms like OpenClaw build their entire memory and collaboration system on files: SOUL.md defines the agent's personality and behavioral boundaries, AGENTS.md describes behavior rules and session workflows, MEMORY.md stores long-term memory across sessions, plus USER.md, TOOLS.md, HEARTBEAT.md, and other Markdown files automatically loaded at startup — along with project source code, node_modules, and .git history. Each agent instance runs in an isolated sandbox where these files are constantly read, written, and updated. When a platform allocates dedicated agent instances to thousands of developers simultaneously, you get a classic "tens of thousands of independent file systems" requirement: each instance needs an isolated POSIX workspace with dense file counts and frequent I/O.

From Kubernetes' perspective, this means your cluster must simultaneously support thousands or even tens of thousands of independent PersistentVolumeClaims. Each PVC must provision quickly — Agent elastic scaling cannot wait minutes for storage. You need low-latency file I/O, and data must remain accessible after a Pod is rescheduled to another node.

This "massive small-scale stateful instances" workload pattern is fundamentally different from traditional stateful applications like databases and message queues, which typically use a few large PVCs. The former requires tens of thousands of small PVCs. Native AWS storage services each have different strengths and weaknesses when facing this new pattern.

1.1 Amazon EBS: Strong Isolation, but Attachment Limits Are Hard Constraints​

EBS excels at isolation. One independent EBS volume per Agent means performance does not interfere across tenants, and failures do not spread. For scenarios requiring strict tenant isolation, this is the cleanest approach.

The problem: the number of EBS volumes a single EC2 instance can attach has a hard upper limit. For most Nitro instances (including the r6g series used in this test), the maximum attachment count is 28, shared with network interfaces and NVMe instance store. After accounting for the primary network interface, the practical EBS volume limit is around 27. Some 7th-generation instance types (such as M7i and R7i) introduced independent EBS volume limits with higher quotas based on instance size, but for r6g instances still under shared limits, 28 is the ceiling. (Reference: Amazon EBS volume limits for Amazon EC2 instances)

What does this mean in practice? In our test scenario, each r6g.4xlarge node ran approximately 100 Agent Pods. If EBS allocated an independent volume to each Pod, a single node could mount at most 28 volumes — requiring nearly 4× the number of nodes to carry the same Pod density, dropping compute resource utilization from 88% to around 20%.

Additionally, EBS volumes are bound to a single Availability Zone and cannot be used across AZs. Once a Pod is scheduled to a node in a different AZ, the original EBS volume becomes unreachable, forcing Pod scheduling constraints to specific AZs. For Agent platforms requiring cross-AZ high availability and rapid elastic scaling, this is an architectural hard constraint.

1.2 Amazon EFS: Mature Isolation, but Mass Provisioning Is the Bottleneck​

As a managed NFS file system, EFS combined with Access Points can allocate an isolated directory view for each Agent. Each Access Point has independent POSIX permissions and a root directory, achieving file-system-level tenant isolation. EFS supports ReadWriteMany, so Pods can be rescheduled across nodes without detach/attach operations — ideal from a scheduling flexibility perspective. Since February 2025, a single EFS file system supports up to 10,000 Access Points, sufficient for tens-of-thousands Agent scenarios from a quota perspective.

However, in actual large-scale deployments, the bottleneck appears in provisioning speed. When dynamically provisioning thousands of PVCs simultaneously through the EFS CSI Driver, each PVC corresponds to an Access Point creation. The EFS API has rate limits on Access Point creation, and the CSI controller must call the API serially or in small batches.

1.3 Amazon S3: Unlimited Capacity, but Not a File System​

S3's scalability and cost efficiency are undeniable — it works well as the final archival layer for Agent data. But Agents at runtime need POSIX file semantics: open, read, write, seek, rename, and list directory. S3 is object storage — it does not support in-place modification, atomic rename, or consistent directory listing semantics.

Mountpoint for Amazon S3 provides a FUSE mount solution, but it explicitly supports only sequential writes to new files and reads of existing files — not random writes or modifications to existing files. For Agent workflows that repeatedly modify context files, append logs, and update checkpoints, this is not a viable runtime storage solution.

II. Curvine: A Distributed Cache File System Designed for Agent Scale​

Curvine is a high-performance distributed cache file system written from scratch in Rust. The name comes from "Curvature Engine" — the faster-than-light propulsion device from Liu Cixin's The Three-Body Problem — symbolizing extreme acceleration of data access.

Its core approach: build a distributed file system cache layer on top of cloud object storage (such as S3), providing complete POSIX semantics upward and using object storage as the persistence layer. For Kubernetes workloads, it integrates natively through a CSI driver, mounted directly as PVCs.

2.1 Core Architecture​

Curvine uses a Master-Worker architecture:

• Master nodes: Manage metadata, coordinate Workers, and handle load balancing. Raft consensus ensures metadata consistency and high availability.
• Worker nodes: Handle actual data caching and serving. Support multi-tier caching across memory, SSD, and HDD, with hot data automatically promoted to faster tiers.
• Client access: POSIX file system interface via FUSE mount; also compatible with S3 and HDFS protocols for integration with existing AI/big data ecosystems.

In Kubernetes environments, Curvine integrates as a CSI driver: the CSI Controller handles dynamic PVC provisioning, and the CSI Node DaemonSet runs on each node to handle FUSE mounts. This means storage provisioning does not call external cloud APIs — it simply creates a directory on the distributed file system, completing in milliseconds.

2.2 How Curvine Differs from JuiceFS​

JuiceFS is a pioneer in this space, implemented in Go with a similar "metadata engine + object storage" architecture. Key differences:

• Performance: Curvine implements core read/write paths in Rust with zero-copy techniques, targeting 100μs-level latency and 100K+ stable QPS. Rust's async runtime (tokio) and GC-free memory model theoretically offer advantages over Go's runtime in high-concurrency small-file scenarios.
• Metadata capacity: Curvine supports up to 5 billion small files per cluster — tens of thousands of Agents each producing small files creates significant metadata pressure.
• Metadata independence: Curvine's file metadata paths correspond one-to-one with underlying S3 object paths. Even if Curvine services fail, files on S3 retain their original structure and remain independently accessible. JuiceFS splits files into blocks, making original files unrecognizable from S3 object names — metadata availability strongly depends on JuiceFS itself.
• Cache architecture: Curvine natively supports automatic multi-tier grading from memory → SSD → HDD. JuiceFS also has local cache capabilities, but Curvine offers more fine-grained cache scheduling strategies.
• Positioning: JuiceFS focuses more on general scenarios and deep cloud vendor integration. Curvine explicitly targets AI training acceleration and Agent cloud-native storage as primary use cases.

Both are excellent open-source projects with active communities, each optimized for different design goals. Specific choices should be evaluated through actual testing based on workload characteristics, team technology stack, and operational preferences. This article does not constitute a recommendation.

2.3 CSI Integration​

StorageClass configuration for Curvine on EKS:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: curvine-sc
provisioner: curvine
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
parameters:
  master-addrs: "curvine-master-0.curvine-master.curvine.svc.cluster.local:8995"
  fs-path: "/k8s-volumes"
  path-type: "DirectoryOrCreate"
  io-threads: "4"
  worker-threads: "8"

volumeBindingMode: Immediate means PVCs bind immediately upon creation (without waiting for Pod scheduling). path-type: DirectoryOrCreate means each PVC corresponds to a directory on the Curvine file system — creation is far faster than solutions requiring cloud API calls.

III. 10,000-Pod Benchmark: Validating Scale Feasibility​

We conducted a scale validation test on Amazon EKS to answer one question: can Curvine reliably provide persistent storage for 10,000 independent stateful Pods on a production-grade EKS cluster?

3.1 Test Environment​

3.2 Workload Configuration​

3.3 Curvine Storage Cluster​

The key number: the storage cluster serving 10,000 PVCs consists of only 1 Master + 3 Worker = 4 core Pods. The CSI Node DaemonSet is a lightweight mount agent that does not consume significant compute resources.

3.4 Test Results​

Provisioning Success Rate

• 10,000 PVCs: all Bound, zero Pending, zero Failed
• 10,000 Pods: all Running, zero CrashLoopBackOff, zero storage-related errors

Storage Provisioning Specs

• Each PVC requested 1Gi, actually provisioned ~30Gi (Curvine's minimum allocation unit)
• Total provisioned storage capacity: approximately 300TB
• File system type: curvinefs (FUSE mount)

Data Durability Verification

• Wrote and verified data persistence on pod-0, pod-5000, and pod-9999
• Data persisted after Pod restarts
• Consistent file system view after cross-node rescheduling

Pod Distribution Uniformity

102 pods × 98 nodes = 9,996 pods
  4 pods ×  1 node  =     4 pods
Total                = 10,000 pods

Each r6g.4xlarge node stably carried ~100 Agent Pods, with the CSI DaemonSet coexisting without resource contention.

3.5 Storage View Inside Each Pod​

Filesystem    Size      Used     Available  Use%  Mounted on
curvinefs     29.8G     354.6M   29.5G      1%    /usr/share/nginx/html

Each Pod has an independent file system view, invisible to others — the same logical isolation as EBS, but bypassing EBS attachment count limits.

3.6 Key Conclusions​

1. Provisioning does not depend on cloud control plane APIs: With EBS, creating a PVC triggers the CSI driver to call EBS CreateVolume and AttachVolume APIs. With EFS, it calls CreateAccessPoint. These cloud APIs have rate limits that cause queuing and slowdowns during mass concurrent creation. Curvine creates a PVC by essentially running mkdir on its own distributed file system — no cloud vendor API calls, fast and unconstrained by cloud API rate limits.
2. Minimal storage cluster resource overhead: 4 core Pods serve tens of thousands of PVCs — no need to reserve large compute resources for the storage system itself.
3. Per-node Pod density no longer limited by storage: 100 Pods share the same FUSE mount point at different paths, without EBS's 28/128 attachment ceiling.
4. Clear horizontal scaling path: Add Worker nodes for greater scale or throughput without affecting existing PVCs.

IV. Getting Started​

If you are building an AI Agent platform on EKS and face any of these scenarios:

• Each Agent instance needs an independent POSIX file system workspace
• Total instances in the thousands to tens of thousands with rapid elastic scaling
• High-density stateful Pods per node (>28)
• Sensitive to small-file I/O latency (microseconds vs. milliseconds)

Curvine is worth including in your storage technology evaluation.

Quick Start

• GitHub: github.com/CurvineIO/curvine
• Documentation: curvineio.github.io
• EKS Integration: Deploy Curvine cluster + CSI driver via Helm Chart, configure StorageClass, and start using

Recommended validation path: start with 100–500 Pods to verify provisioning speed and I/O performance; if results meet expectations, gradually scale to thousands or tens of thousands. The storage cluster itself can start with 1 Master + 1 Worker, adding Worker nodes as needed to scale throughput and cache capacity.

Conclusion​

The "massive small-scale stateful instances" workload pattern brought by AI Agents poses new challenges for Kubernetes storage. EBS's per-instance attachment limits, EFS's provisioning bottlenecks, and S3's lack of POSIX semantics each fall short in different ways. Curvine, as a distributed cache file system designed for this scenario, validated on EKS that 10,000 independent PVCs can be provisioned and served reliably with minimal storage cluster overhead — providing a practical storage foundation for Agent platform scale-out.

Introduction​

As data volumes continue to grow explosively, storage performance has become a key factor in application responsiveness and user experience. Traditional storage I/O paths are often constrained by operating-system kernel overhead, making it difficult to fully utilize modern high-speed storage devices such as NVMe SSDs. To break through this bottleneck, many high-performance storage systems have started to adopt kernel-bypass technologies. This article takes a closer look at how the Curvine storage platform integrates SPDK (Storage Performance Development Kit) to deliver extreme NVMe storage performance, bringing users lower latency and higher throughput.

The Challenge with Traditional I/O: The Kernel VFS Bottleneck​

In a traditional storage architecture without SPDK, application I/O requests must pass through the operating system's Virtual File System (VFS) layer. VFS provides a unified file-operation interface, but it also introduces overhead such as context switches, data copies, and interrupt handling. For NVMe devices, which already provide extremely low latency at the hardware level, these kernel overheads can become the main barrier to further performance improvement. When Curvine Worker processes read and write data through system calls such as pread and pwrite, the system cannot fully unleash the raw performance of NVMe hardware.

Enter SPDK: Kernel Bypass and User-Space I/O​

The core idea behind SPDK is to move storage I/O processing from the kernel into user space. By bypassing the kernel VFS path, SPDK allows applications to communicate directly with NVMe devices, significantly reducing unnecessary overhead.

Curvine takes advantage of this capability by integrating SPDK into its architecture and achieving several key improvements:

1. Ultra-low latency: With kernel bypass, I/O requests no longer travel through a complex kernel path. They complete directly in user space, greatly reducing end-to-end latency.
2. Extremely high throughput: By reducing CPU overhead and context switches, a single CPU core can handle more I/O operations, resulting in higher IOPS. According to official SPDK data, SPDK can natively exceed 10 million IOPS on a single core [1].
3. Efficient resource utilization: SPDK uses hugepages for DMA buffer management, reducing TLB misses and improving memory-access efficiency.

Deep Integration Between Curvine and SPDK: Architecture Overview​

Curvine uses SPDK as its remote block-device layer. Local physical NVMe drives, dedicated SPDK target containers, and remote storage nodes can all communicate with Curvine Worker processes through NVMe-oF (NVMe over Fabrics).

This integration is not a thin wrapper around SPDK calls. It reaches deeply into Curvine's architecture.

SpdkEnv: The Foundation of the Global Environment​

SpdkEnv is the brain of Curvine's SPDK module. It initializes the entire SPDK application framework and acts as a singleton to ensure unified resource management.

Its main responsibilities include:

• Hugepage allocation: Reserves large memory pages for DMA buffers to optimize performance.
• CPU affinity configuration: Uses reactor_mask to bind SPDK I/O threads to specific CPU cores, avoiding scheduling overhead.
• NVMe-oF target discovery: Automatically discovers and connects to available NVMe-oF storage targets.
• SpdkPoller thread creation: Starts dedicated I/O polling threads, which are central to SPDK's high-efficiency execution model.

SpdkPoller: The Always-On I/O Engine​

SpdkPoller is a critical component in the SPDK architecture. Because SPDK requires NVMe command submission and completion handling to occur on the same thread, SpdkPoller takes on this responsibility. It is a dedicated I/O thread that bridges asynchronous processing threads with SPDK's synchronous polling loop.

SpdkPoller provides two important capabilities:

• Intelligent polling: When there are no I/O requests, SpdkPoller enters an idle state and uses eventfd to avoid wasting CPU cycles on busy spinning. When new requests arrive, it wakes up, enters an active state, efficiently submits NVMe commands, and polls completion queues. This spin-before-sleep strategy minimizes system-call overhead and keeps the I/O path smooth.
• Error handling: When a queue pair (qpair) encounters an error, SpdkPoller can mark it as orphaned and force-complete all pending I/O on that queue pair, preserving system stability.

SpdkBdev and DMA Buffers: The Basis for Efficient Data Transfer​

SpdkBdev is Curvine's abstract handle for SPDK block devices, similar in role to LocalFile in a traditional file system. It manages connections to NVMe namespaces and queue pairs, and it owns the key DMA buffers (DmaBuf).

The design focuses on efficient and predictable data transfer:

• Hugepage-backed DMA buffers: DmaBuf is a preallocated, fixed-size buffer backed by hugepages. Data can move directly between the NVMe device and memory without CPU involvement, avoiding extra copies and TLB misses.
• Buffer reuse: SpdkBdev allocates read and write buffers during initialization and reuses them across I/O operations. This avoids frequent memory allocation and release overhead, reaching a zero-cost-after-first-allocation model.
• Chunked processing for large I/O: For large I/O requests, SpdkBdev splits them into chunks of dma_buf_size (1 MB by default) and processes them sequentially through the same fixed buffer. This keeps the implementation efficient while avoiding the complexity of dynamic memory management.

BdevOffsetAllocator: User-Space Block Management​

With SPDK, Curvine bypasses the kernel file system and directly works with the flat byte-address space provided by NVMe devices. This means the traditional kernel block allocator is no longer applicable.

BdevOffsetAllocator fills that gap by implementing block management in user space:

• Unique byte-range allocation: Allocates a unique byte range for each Curvine block and reclaims that range when the block is deleted.
• Efficient space management: Uses a bump cursor for new allocations and a free list to track reclaimed ranges. Adjacent free ranges are coalesced to reduce fragmentation.
• Persistence and thread safety: Allocator state, including the cursor position and free list, is persisted to RocksDB through SpdkMetaStore, ensuring consistency after restart. The allocator itself is also thread-safe.

Optimized Read and Write Paths​

After integrating SPDK, Curvine's read and write paths become significantly more efficient.

• Read path: The handler submits an IoRequest directly to SpdkPoller. SpdkPoller communicates with the NVMe-oF target and notifies the handler after I/O completion. Data is copied directly from the DMA buffer to the application, keeping the path extremely short.
• Write path: SPDK requires block-aligned I/O. For unaligned writes, Curvine uses a read-modify-write strategy: it first reads the full aligned block, modifies the target range, and writes the complete aligned block back. For aligned writes, Curvine writes directly, minimizing unnecessary steps.

The Core Value SPDK Brings to Curvine​

Curvine's deep SPDK integration is more than the introduction of a technical component. It is a comprehensive upgrade to Curvine's high-performance storage architecture. Through kernel bypass, user-space I/O, dedicated polling threads, hugepage-backed DMA buffers, and user-space block management, SPDK brings Curvine several important benefits:

• Outstanding performance: Under I/O-intensive workloads, Curvine can significantly outperform traditional kernel-VFS-based storage paths, delivering lower latency and higher throughput.
• Higher resource utilization: Reduced CPU and memory overhead allows hardware resources to serve business logic more efficiently.
• Stronger scalability: Modular design and user-space control provide a solid foundation for future Curvine features and performance optimizations.

On the path toward extreme performance, the combination of Curvine and SPDK is a strong example of what modern software-defined storage can achieve. It demonstrates the potential of software-defined storage and points toward the future direction of high-performance storage systems.

References​

[1] SPDK official performance data: SPDK

In distributed file systems, metadata memory efficiency, concurrent request handling, and small-file throughput are core indicators of overall system capability. Curvine recently completed a high-intensity metadata benchmark, and the results were clear: Curvine reached a new high-water mark for open-source metadata efficiency while delivering core capabilities comparable to commercial distributed storage products.

🔥 Key Takeaways​

• Efficient memory usage: With 800,000 directories and 300 million files, and one block written per file, Curvine used just 38 GB of memory. That is roughly on par with the metadata-memory capability described for the commercial edition of JuiceFS in reference [1].
• Low latency under massive concurrency: With 100,000 clients looping operations, throughput held steady at 53,000 ops/s. Average command latency stayed below 2 ms, and P99 latency stayed below 9 ms.
• High small-file throughput: Under heavy concurrent small-file writes, Curvine sustained 12 million small files per hour, with an average write time of 0.3 ms per file.

📝 Test Setup​

• Curvine cluster: one Master and one Worker
• Benchmark machine: Alibaba Cloud ecs.i5.8xlarge, 32 cores, 256 GB RAM
• Clients: 100,000 FUSE clients
• Operations: repeated high-frequency commands such as mkdir, touch, file writes, and ls

📊 Core Benchmark Results​

🧠 Memory Efficiency: A New Open-Source High-Water Mark​

• Managed scale: 800,000 directories + 300 million files
• Per-file data written: 1 block
• Total memory usage: 38 GB
• Comparison point: comparable to the metadata-memory capability described for the commercial edition of JuiceFS

⏱️ High Concurrency, Low Latency at 100,000 Clients​

• Concurrent clients: 100,000 FUSE clients
• Stable throughput: 53,000 ops/s
• Average latency: up to 2 ms
• P99 latency: up to 9 ms

Connection overhead was also low: 100,000 live connections consumed only 1.1 GB, or about 11.5 KB per connection.

Once the benchmark stopped, Master memory dropped immediately from 39.1 GB back to 38 GB.

🚀 Small-File Throughput: Built for Scale​

• Files written per hour: 12 million small files
• Average write time per file: 0.3 ms
• Throughput remained saturated even under high concurrency

At 15:00, Curvine had written 287 million files:

At 16:00, the total had reached 299 million files:

🏗️ Metadata Architecture​

Curvine's metadata subsystem stands out not just in large-scale memory efficiency and high-concurrency performance, but also in comparison with other open-source systems. Those results come from a deliberately designed metadata architecture.

💡 Design Principles​

1. A single Master should support very large namespaces and massive numbers of small files.
2. The system should provide high concurrency and low latency for frequent operations such as create, delete, and update.
3. External dependencies should be minimized to reduce operational complexity while keeping the system stable.

Based on those goals, Curvine combines an in-memory directory tree, standalone RocksDB, and a Raft-based consistency mechanism. This three-layer design balances performance, scale, and stability.

🛡️ FsMode: Working with UFS for Safe Durability​

Curvine also supports FsMode, which synchronizes metadata and file data to the underlying file system (UFS). This creates a dual safety model of local storage plus disk-backed fallback, preventing data loss without sacrificing runtime performance.

🚀 Future Directions​

Curvine's metadata system will keep pushing forward in three areas:

1. 10 billion files on a single node: continue deepening single-node capability until a standard 512 GB memory machine can manage metadata for 10 billion files.
2. Federation: improve cluster-scale metadata expansion with an HDFS Federation-like model that partitions by directory and can scale beyond 100 billion files. Federation is especially strong for centralized metadata operations such as mv and ls, but it requires directory planning up front.
3. Pluggable metadata management: abstract the metadata interface and support pluggable metadata backends for better flexibility and adaptability.

📚 References​

1. https://mp.weixin.qq.com/s/zbBUQ4P53PPWQjOHQmw8uw
2. https://hadoop.apache.org/docs/r3.4.0/hadoop-project-dist/hadoop-hdfs-rbf/HDFS%20RouterFederation.html

👇 Follow Us​

We regularly share hands-on work on distributed storage, metadata optimization, and high-concurrency benchmarking.

GitHub: https://github.com/CurvineIO/curvine

In our practice with distributed cache, we have identified core pain points users face: insufficient POSIX semantics support, high resource consumption, and complex operations. To better address these challenges, we have restructured Curvine's architecture and development roadmap, creating a next-generation unified data access layer that combines strong POSIX semantics with high-performance caching, delivering a qualitative improvement in remote data access experience.

Two Core Mount Modes for Diverse Business Needs​

Curvine optimizes data read/write patterns into two concise mount point modes: CacheMode and FsMode, each targeting different business scenarios. This design balances cache acceleration with complete semantics support, allowing users to choose flexibly based on actual requirements.

CacheMode: Lightweight Read Cache Acceleration, Tightly Coupled with UFS​

CacheMode centers on UFS (Underlying File System), primarily serving as a read cache accelerator and unified proxy for UFS. All read/write operations are UFS-centric:

• Metadata caching effectively accelerates common operations like ls
• Write operations go directly to UFS
• Users maintain strong awareness of UFS without changing existing data operation habits

This is the preferred solution for lightweight UFS read performance improvement.

FsMode: Full Performance Acceleration with Strong POSIX Semantics​

FsMode places Curvine itself at the core, with metadata independently managed by Curvine. Its path structure maps one-to-one with UFS, while UFS serves only as Curvine's cold storage layer. This mode provides:

• Comprehensive read/write cache acceleration
• Better POSIX semantics support for read/write operations
• Ideal for large-scale file performance acceleration
• Best choice for scenarios requiring both semantic completeness and high performance

FsMode Deep Dive: Layered Design Balancing Performance and Consistency​

As Curvine's core mode, FsMode employs a layered filesystem mount/write design. Through clear semantic definitions and process planning, it achieves balance among performance, semantics, and data consistency. Let's examine its core design details.

Core Semantic Rules​

1. Unified IO Entry Point: All data read/write operations go through Curvine. Applications should use only Curvine paths; direct UFS access is not recommended as it cannot guarantee data consistency.

2. Asynchronous Cold Storage Writes: When writing data, it first lands in Curvine (metadata + blocks). The Master side periodically submits Load/Dump tasks based on policies to asynchronously flush data to UFS (e.g., S3), making frontend write operations more efficient.

3. Intelligent Read Backfill: When reading data, Curvine is prioritized. If data has been evicted or exists only in UFS, a Load operation backfills UFS data to Curvine. The current read passes through directly from UFS, balancing read speed with data availability.

4. Flexible Replica Forms: Allows states where only UFS contains data. In such cases, UFS data serves as the file's sole replica, maximizing storage resource utilization.

5. On-Demand Metadata Synchronization: The mount operation synchronizes all metadata under the directory. Full synchronization is not performed actively afterward. If needed, use the mount resync command to manually update mount point metadata (synchronizing only file metadata that exists solely in UFS).

6. Lazy Cache Loading: If a file being read has no metadata in Curvine but exists in UFS, the first read will fail. On retry, UFS files are actively fetched. Alternatively, manually trigger the mount resync command in advance to synchronize metadata and avoid read failures.

7. Fault Tolerance Design: When Master fails, users can access data normally through UFS interfaces. When Worker fails, other replicas can be accessed in multi-replica scenarios; in single-replica scenarios, direct UFS reading ensures business continuity.

Consistency Design: Usable Now, Better in Future​

The current FsMode consistency implementation:

• When a UFS path is first mounted to Curvine, all metadata under that directory is mapped to Curvine as a whole
• If files are written directly to UFS bypassing Curvine, Curvine cannot automatically perceive these changes
• Manual re-synchronization via sync commands is required in such cases

Future Planning: Curvine will implement automatic perception of UFS metadata change events, enabling near real-time UFS metadata synchronization. This will technically guarantee data consistency completely, allowing users to operate without concerning themselves with synchronization—truly seamless usage.

FsMode Core Design Objectives​

All FsMode designs revolve around clear objectives, ensuring every capability precisely addresses business pain points:

• Unified Entry Point: All operations go through Curvine paths. Applications don't need to adapt to UFS, reducing development and operations costs.
• POSIX Semantics: Supports complete POSIX filesystem semantics, including directory trees, random read/write, renaming, atomicity, strong consistency, etc., adapting to various traditional and new applications.
• Tiered Storage: Curvine layer stores hot data (metadata + optional local blocks), UFS serves as persistent/cold replica, achieving hot/cold data separation for improved storage efficiency and access performance.
• Background Flushing: Master periodically submits Load/Dump tasks based on business operations and preset policies to asynchronously flush Curvine data to UFS without affecting frontend business.
• UFS-Only Replica: Supports scenarios where data exists only in UFS (e.g., S3). Data is backfilled on-demand or read-through as needed, balancing storage flexibility with data accessibility.

CacheMode vs FsMode: Core Differences at a Glance​

To help you clearly distinguish between the two modes and precisely match business scenarios, we've organized the core comparison dimensions:

Extreme Resource Optimization, Lighter and More Friendly​

Beyond refining functionality and performance, Curvine has made deep optimizations in resource consumption, upgrading comprehensively from underlying technology stack to implementation methods:

Curvine is built on Rust, naturally possessing high performance and low resource consumption characteristics. It also employs cutting-edge optimization techniques such as asynchronous operations and zero-copy, further reducing resource footprint.

Online practice data shows: A single Curvine Worker process occupies less than 1GB of memory. In large-scale cluster deployments, this effectively reduces server resource investment and operational costs, making it easily adaptable even in resource-constrained scenarios.

Product Philosophy: Not a Replacement, Just a Better Data Access Method​

Curvine has had a clear product positioning from the start:

Not pursuing to become a general-purpose POSIX filesystem, nor attempting to replace any storage product.

We have always focused on one thing: making remote data access so fast that you can't feel the "remote" aspect, without changing users' original data operation habits. This is not only a technical challenge but also Curvine's core product philosophy—the best infrastructure is the kind that makes you unaware of its existence.

In the AI era, data volume is exploding. Remote data access performance and experience have become key factors affecting business efficiency. Curvine integrates the wisdom of distributed cache technology with POSIX completeness of distributed filesystems, while adhering to the core principle of "metadata transparency, unchanged file structure." Users can achieve a leap in remote data access performance without major modifications to existing business systems.

In the future, Curvine will continue to深耕 the unified data access layer field, continuously optimizing performance, stability, improving semantics support, and simplifying operations processes. We are committed to becoming a key part of data infrastructure in the AI era, providing efficient, stable, and lightweight data access support for digital upgrades of various businesses.

Finally, we hope more open-source enthusiasts from the storage and Rust fields will join us in building and sharing together!

Powered by OPPO Bigdata.

After six months of open-source journey and surveying multiple users, we have gained a clear understanding of the pros and cons of the distributed cache model. In light of the limitations of distributed cache application scenarios, here is a simple reflection.

In today's booming era of big data and artificial intelligence, "storage-compute separation" has become the mainstream paradigm of cloud-native data architecture. Computing resources can elastically scale, while data is uniformly settled in low-cost object storage (such as S3, OSS). However, this architecture brings a fatal pain point: the high latency and low throughput of object storage seriously drag down computing performance. Thus, the distributed cache layer emerged as needed—it was hoped to become a high-speed bridge connecting "flexible computing" and "cheap storage."

Distributed file cache systems can "transparently accelerate" access to remote storage and provide a unified namespace. However, when enterprises eagerly introduce them into production environments, they often fall into the predicament of underwhelming performance, operational complexity, and semantic mismatch. This article will deeply analyze the gap between the technical ideal of distributed cache and its landing reality, revealing the structural limitations of distributed cache in general scenarios.

I. The Ideal of Distributed Cache: Unified, Transparent, High-Performance​

The original design intention was highly attractive:

• Unified Namespace: Mount heterogeneous storage such as HDFS, S3, GCS into a single directory tree, applications only need to access xx://;
• Transparent Cache: After the first read of remote data, automatically cache to memory/SSD, subsequent responses in milliseconds;
• Ecosystem Compatibility: Seamlessly integrate with mainstream computing engines such as Spark, Presto, Pytorch, without code modifications;
• Tiered Storage: Support memory → SSD → HDD multi-level caching, balancing performance and cost.

In demonstration environments, distributed cache can indeed significantly improve the access performance of object storage, especially in scenarios where model training repeatedly reads input.

II. Reality's Pain: Three Structural Defects​

However, the ideal is full, but the reality is bony. Distributed cache exposes three unavoidable defects in real business scenarios.

Pain 1: Incomplete POSIX Semantics, Limited Versatility​

Distributed cache provides POSIX-like interfaces through FUSE, enabling traditional applications to read remote data like accessing local files. However, its support for POSIX semantics is highly incomplete:

• ❌ No random write support: Cannot modify bytes in the middle of a file, only allows creating new files or full overwrites;
• ❌ No truncate, hard links, or file locks;
• ⚠️ Strong consistency missing: Multiple clients may read expired cache, requiring manual metadata refresh.

This means distributed cache simply cannot run databases, log systems, or any programs requiring in-place updates. It is essentially designed for WORM (Write-Once-Read-Many), not a general-purpose file system. Many teams, after attempting to "seamlessly migrate" existing business to distributed cache, discover their applications crash due to write operation failures.

Truth: Distributed cache is not a "distributed POSIX file system," but a "data orchestration layer optimized for batch processing."

Pain 2: High Resource Consumption​

Currently, distributed cache systems using Java or Go languages generally have high resource consumption problems.

For example, Java processes often occupy tens of GB of memory, which is somewhat wasteful for systems that use memory as cache.

Pain 3: Operational Complexity, ROI Hard to Deliver​

The deployment of distributed cache involves multiple components such as Master, Worker, Journal, UFS connectors, etc., and resource tuning (memory allocation, cache policies, network configuration) is extremely complex. More fatally:

• Cache hit rate depends on data access patterns: If jobs are one-time scans (such as ETL), caching has no value;
• Resource competition: Memory/SSD occupied by Workers competes with Spark Executors for node resources;
• Troubleshooting difficulties: Issues like cache inconsistency, block loss, and UFS synchronization failures require deep source code analysis to locate.

Many teams invest months in building and tuning cache clusters, only to find limited performance improvement but doubled operational burden, forcing them to abandon it.

III. Reflection: Can Distributed Cache Replace File Systems?​

The dilemma of distributed cache reflects a deeper issue: attempting to use a general-purpose intermediate layer to solve all I/O problems is itself a form of technical dogmatism.

Distributed cache has significant effects in large-scale data I/O scenarios such as big data and AI training. However, as a company, purchasing or deploying a distributed cache cluster cannot achieve universal application across scenarios, nor can it fully utilize its capabilities.

Trend: "Specialized is better than generalized" — Rather than maintaining a heavyweight distributed cluster, it is better to build a more versatile tiered file system with a cache acceleration layer in the middle, providing more general support with file system semantics.

IV. Way Out: Rational Choice, Scenario-Driven​

Distributed cache is not without merit. It still has value in the following scenarios:

• ✅ Hybrid cloud/multi-cloud architecture: Unified access to object storage from different cloud providers;
• ✅ High-reuse read-only datasets: Such as benchmark datasets repeatedly used in AI training;
• ✅ Dedicated platform teams available: Can bear its operational and tuning costs.

However, for most enterprises, a more pragmatic path is:

1. First evaluate whether I/O is truly a bottleneck: Confirm through profiling;
2. Prioritize optimizing data formats and query logic: Use Iceberg/Lance instead of raw files;
3. Avoid "using cache for the sake of using cache": Caching is a means, not an end;
4. Build general-purpose file system capabilities: Build cache that rivals file system capabilities, fully exploring versatility.

Conclusion​

Distributed cache is a phased technical experiment that has promoted the development of data orchestration concepts. However, its "pain" also warns us: there is no silver bullet, only trade-offs. On the road to pursuing high performance, blindly introducing general-purpose middleware often backfires. True engineering wisdom lies in understanding the essence of business and choosing the most matching tool, even if it's not "cool" enough.

Distributed cache should not be a standard configuration of architecture, but rather a precise scalpel for specific scenarios. Only by building more general, lightweight, and efficient tiered file systems can we avoid falling into the "cache pain" and let data truly flow, rather than being trapped in layers of abstraction.

Powered by OPPO Bigdata.