← Back
CacheBlendPlus
Semantic KV cache reuse for faster RAG inference
Reimplements CacheBlend's selective KV cache reuse for RAG inference in PyTorch to cut redundant recomputation across document chunks, adds an adaptive recomputation budget driven by divergence scoring, and uses embedding based semantic deduplication with a custom CUDA kernel to profile and speed up the inference pipeline.
Technologies
CUDA · Machine Learning · Python