Executive Summary

This report provides a comprehensive comparison between Retrieval-Augmented Generation (RAG) and
Cache-Augmented Generation (CAG) architectures for a RAG application serving 100 concurrent users
with a company dataset exceeding 100 GB. The analysis covers architectural paradigms, performance
characteristics, hardware requirements, evaluation metrics, and suitability with LLaMA 3.1 (8B and 70B)
models. Based on data size, concurrency, and model considerations, the report offers
recommendations for selecting an optimal architecture.

Introduction
Modern applications increasingly leverage large language models (LLMs) alongside external knowledge
sources. RAG has been widely adopted to integrate domain-specific information at inference time, while
CAG has emerged as an alternative that preloads and caches external knowledge, reducing retrieval
latency. This report examines both approaches in the context of a sizable dataset (>100 GB) and
moderate concurrency (100 users), using LLaMA 3.1 in 8B and 70B configurations.

Definitions and Concepts

Retrieval-Augmented Generation (RAG)

RAG enhances LLM responses by retrieving relevant documents at inference time and incorporating
them into the model input. Typical steps include: 1. Query encoding and retrieval from a vector store or
database. 2. Selecting top-k documents based on similarity scores. 3. Constructing a prompt combining
retrieved context and the user query. 4. Generating responses with the LLM using retrieved context.

This dynamic retrieval enables up-to-date and extensive knowledge integration without embedding all
data within the model parameters (en.wikipedia.org).

Cache-Augmented Generation (CAG)

CAG preloads relevant external knowledge into an extended context or precomputes internal model
caches (e.g., key-value caches) ahead of inference, enabling the model to generate responses without
on-demand retrieval. By loading and caching data in advance, CAG aims to: - Eliminate real-time
retrieval overhead. - Simplify the inference pipeline by removing the retrieval component. - Reduce
latency for frequently asked or similar queries by serving directly from cache (medium.com,
medium.com).

CAG is most effective when the knowledge base is stable, of manageable size relative to model context
capabilities, and when repeated queries access similar subsets of data.

Architectural Comparison

Workflow and Components

• RAG Workflow:

1
 • Retriever: Vector database (e.g., FAISS, Pinecone) indexing embeddings of >100 GB data.
• Ranking: Scoring and selecting top-k passages per query.
• Prompt Assembly: Combining retrieved passages with query.
• Generation: LLaMA 3.1 generates output.

• Cache Layer (optional): Caching recent retrieval results to optimize repeat queries.

• CAG Workflow:

• Preload Phase: Embedding and loading relevant documents into model context or precompute
KV caches; potentially segmented into partitions if full dataset exceeds context capacity.
• Inference Phase: LLM uses preloaded context/caches; no retrieval component invoked on each
query.

Data Handling

• RAG handles large datasets by indexing embeddings on disk or distributed storage. Scalable to
hundreds of GBs by sharding vector store and using approximate nearest neighbor search.
Retrieval is per query, incurring I/O and compute overhead but manageable with optimized
vector indices and caching layers.
• CAG requires the entire relevant subset to fit within the model’s extended context window or
within precomputed caches. With >100 GB data, preloading all data into LLaMA 3.1’s context
(e.g., even with 32K or larger token windows) is infeasible. Partitioning or partial caching may be
possible, but reduces benefit of full preload and complicates architecture (adasci.org, arxiv.org).

Concurrency and Scalability

• RAG: Designed for concurrency by scaling vector store and retrieval services horizontally. For 100
concurrent users, retrieval services must handle simultaneous embedding lookups and ranking.
Caching repeated queries (since similar queries retrieve similar data) can reduce load. Load
balancing and concurrency controls on vector database nodes ensure consistent latency.
• CAG: Concurrency largely depends on inference throughput. Preloaded caches benefit all users if
they query cached content. However, if different users need disparate data subsets exceeding
cache capacity, performance degrades. Managing cached partitions per user group adds
complexity. For 100 concurrent users with varied queries, ensuring cache hit rates is challenging
unless dataset subsets are heavily overlapping and small enough to preload.

Latency and Throughput

• RAG Latency: Includes vector search latency (disk or memory-based), prompt construction, and
generation. Typical vector search latency can be optimized to tens to low hundreds of
milliseconds per query with ANN and caching layers. Generation latency depends on model size
and hardware (e.g., 8B vs. 70B LLaMA 3.1) but is independent of retrieval.
• CAG Latency: Removes retrieval latency, so end-to-end inference may be faster per query if
context fits and caches are warm. However, initial preload or cache-building phase can be time-
and resource-intensive. For stable datasets and high repetition, amortized latency benefits
manifest. With >100 GB data and limited context window, full CAG benefit is limited; partial
caching yields inconsistent latency improvements.

2
Hardware Requirements

Model Serving (LLaMA 3.1)

• 8B Model:
• VRAM: Approximately 16–24 GB GPU memory for inference with typical sequence lengths.
• Compute: Single high-end GPU (e.g., NVIDIA A100 40GB) can serve multiple concurrent 8B
inference streams, depending on batching and latency requirements.

• CPU/RAM: Moderate CPU resources for preprocessing; 64–128 GB RAM to support data pipelines
and retrieval services.

• 70B Model:

• VRAM: Approximately 80+ GB GPU memory, often requiring model parallelism across multiple
GPUs (e.g., 2×A100 80GB or clusters with NVLink).
• Compute: Multi-GPU setup, higher complexity in serving infrastructure; lower throughput per
GPU due to larger size.
• CPU/RAM: Higher RAM (128–256 GB) for data handling; more CPU cores to orchestrate
distributed inference.

Retrieval Infrastructure (RAG)

• Vector Store:
• Storage to index embeddings of >100 GB data: likely several hundred GB to 1 TB depending on
embedding size and index overhead.
• Memory: For faster ANN (e.g., in-memory indices), require servers with large RAM (256+ GB) or
use hybrid disk+RAM approaches.
• CPU/GPU: CPU-based ANN often sufficient; GPU-accelerated search can improve latency but
increases cost.

• Horizontal scaling: Cluster of nodes for sharding data and handling 100 concurrent queries with
caching layers.

• Cache Layer:

• In-memory cache (e.g., Redis) to store embeddings or retrieval results for repeated queries to
reduce vector store load.

Caching Infrastructure (CAG)

• Preload Phase:
• Storage: Precompute and store model KV caches. For >100 GB data, precomputing KV caches for
all documents may require storage proportional to dataset size × model state size, potentially
hundreds of GB or more.
• GPU/Memory: Preloading context may require GPUs with large memory or specialized hardware.
If partial caching, multiple GPUs may be involved.
• Inference:
• Serving model with extended context: GPUs must hold preloaded context in memory; may
exceed capacity, requiring splitting or streaming context.
• Complexity: Managing large preloads for multiple user groups increases infrastructure
complexity.

3
Performance Metrics and Evaluation

Retrieval Quality and Relevance

• RAG: Evaluate retrieval accuracy via metrics such as recall@k, precision, and end-to-end answer
quality (e.g., BLEU, ROUGE, or human evaluation). Retrieval errors directly impact generation
accuracy (en.wikipedia.org).
• CAG: If cache includes comprehensive relevant content, retrieval step is implicit. Evaluate
coverage of cached knowledge vs. dataset: if cache misses relevant content, generation quality
suffers. Measuring cache hit ratio and effective coverage is critical (coforge.com).

Latency and Throughput Benchmarks

• RAG: Benchmark vector search latency under load (100 concurrent queries), generation latency
for 8B vs. 70B. Use load testing tools to simulate concurrency; measure p50, p95, p99 latencies.
• CAG: Benchmark after preload: measure generation latency without retrieval. Account for
partitioning strategies: measure average latency when data fits in cache vs. misses requiring
fallback (if hybrid approach used).

Scalability

• RAG: Scalability depends on vector store sharding, cache hit improvements, and autoscaling of
retrieval and inference services.
• CAG: Scalability constrained by cache capacity and model context window; scaling to new data
requires re-preload; dynamic data updates are challenging.

Cost Considerations

• RAG: Costs include vector store infrastructure, storage, retrieval compute, and inference GPUs.
Scaling costs linearly with data growth and query volume.
• CAG: High upfront cost for cache precomputation and storage; lower per-query retrieval cost but
may need frequent refresh for dynamic data. Infrastructure complexity can increase
maintenance costs.

Suitability for >100 GB Dataset and 100 Concurrent Users

• More predictable infrastructure footprint: separate retrieval and inference services can autoscale
independently.

• CAG:

• Preloading >100 GB data into model context or KV caches is impractical due to context window
and memory constraints. Partial caching strategies reduce benefits.
• High complexity to manage cache partitions, especially if query patterns vary. Suitable only if a
small subset (~few GB) of data accounts for most queries and can be preloaded.
• For dynamic data, frequent cache rebuilds hamper deployment agility.

4
Model Choice Implications (LLaMA 3.1 8B vs. 70B)
• Inference Latency:
• 8B: Lower latency, can serve more concurrent requests per GPU. Suitable when response
complexity is moderate.

• 70B: Higher latency, requires multi-GPU; use only if substantial quality improvements justify cost
and complexity.

• Context Window:

• Both models limited by maximum token length (e.g., 8K–32K tokens). Neither can directly hold
>100 GB content. CAG infeasible for full dataset regardless of model size.

• Generation Quality:

• 70B may produce more coherent and accurate responses, especially with complex prompts.
However, incremental benefit over 8B must be evaluated via benchmarks on domain-specific
tasks.

• Cost and Infrastructure:

• 8B: Lower GPU memory requirements (one A100 40GB or equivalent), lower inference cost;
easier to scale horizontally.
• 70B: Requires multi-GPU or high-memory GPUs, complex orchestration, higher cost. For many
enterprise applications, 8B with optimized prompts and retrieval yields sufficient performance.

Recommendations
1. Adopt RAG Architecture: Given dataset size (>100 GB) and concurrency requirements, RAG with
a scalable vector store and caching layer is the practical choice.

2. Model Selection: Start with LLaMA 3.1 8B for cost-effective inference. Conduct benchmarking to
compare response quality against 70B on representative queries. If 70B delivers significantly
better ROI, consider deploying 70B for critical use cases.

3. Retrieval Optimization:

4. Use efficient ANN indexes (e.g., HNSW) and sharding to handle >100 GB embeddings.
5. Implement caching of retrieval results for repeated queries to reduce load and latency.

6. Monitor retrieval quality; periodically evaluate embedding model and update if necessary.

7. Infrastructure Planning:

8. Provision GPU servers for inference: multi-instance 8B deployments, autoscaling based on load.
9. Deploy vector store nodes with sufficient RAM and disk for embeddings; use SSDs or NVMe for
low-latency storage.

10. Implement monitoring and logging for latency, throughput, error rates.

5
 11. Fallback and Hybrid Strategies:

12. For very frequent queries targeting a small subset of data, implement in-memory caching at
application layer to serve responses quickly without repeated retrieval.

13. Explore partial caching (CAG-like) for hot data segments if analysis identifies a stable subset
representing most queries.

14. Evaluation Framework:

15. Define benchmarks: query sets reflecting expected usage; measure retrieval accuracy,
generation quality (automated and human evaluation), latency under concurrency.

16. Compare LLaMA 3.1 8B vs. 70B using same retrieval contexts.

17. Data Update Process:

18. Implement pipelines to index new or updated documents promptly (incremental embedding and
index update).

19. For CAG-like caching of hot segments, schedule periodic refresh based on data change
frequency.

20. Security and Compliance:

21. Secure vector store and inference endpoints; ensure data encryption at rest and in transit.
22. Comply with licensing requirements of LLaMA 3.1 for commercial usage; verify usage terms.

Conclusion
For a RAG application with 100 concurrent users and over 100 GB of company data, traditional RAG
architecture is recommended due to its scalability, flexibility, and manageable infrastructure complexity.
CAG offers benefits in scenarios with small, stable datasets and highly repetitive queries but is
impractical for large datasets exceeding model context capacities. Starting with LLaMA 3.1 8B under a
RAG setup, combined with retrieval caching and performance benchmarks, provides a cost-effective and
scalable solution. Consider LLaMA 3.1 70B only after validating significant quality gains justifying
increased hardware and operational costs.