gh-jamie-bitflight-claude-skills-development-skills/skills/python3-development/references/modern-modules/python-diskcache.md

title, library_name, pypi_package, category, python_compatibility, last_updated, official_docs, official_repository, maintenance_status

title	library_name	pypi_package	category	python_compatibility	last_updated	official_docs	official_repository	maintenance_status
python-diskcache - SQLite-Backed Persistent Cache for Python	python-diskcache	diskcache	caching	3.0+	2025-11-02	https://grantjenks.com/docs/diskcache	https://github.com/grantjenks/python-diskcache	active

python-diskcache - SQLite-Backed Persistent Cache for Python

Overview

python-diskcache is an Apache2-licensed disk and file-backed cache library written in pure Python. It provides persistent, thread-safe, and process-safe caching using SQLite as the backend, making it suitable for applications that need caching without running a separate cache server like Redis or Memcached.

Official Repository: https://github.com/grantjenks/python-diskcache @ grantjenks/python-diskcache Documentation: https://grantjenks.com/docs/diskcache/ @ grantjenks.com PyPI Package: diskcache @ pypi.org/project/diskcache License: Apache License 2.0 @ github.com/grantjenks/python-diskcache Current Version: 5.6.3 (August 31, 2023) @ pypi.org Maintenance Status: Actively maintained, 2,647+ GitHub stars @ github.com/grantjenks/python-diskcache

Core Purpose

Problem diskcache Solves

1. Persistent Caching Without External Services: Provides disk-backed caching without requiring Redis/Memcached servers @ grantjenks.com/docs/diskcache
2. Thread and Process Safety: SQLite-backed cache with atomic operations safe for multi-threaded and multi-process applications @ grantjenks.com/docs/diskcache/tutorial.html
3. Leveraging Unused Disk Space: Utilizes empty disk space instead of competing for scarce memory in cloud environments @ github.com/grantjenks/python-diskcache/README.rst
4. Django's Broken File Cache: Replaces Django's problematic file-based cache with linear scaling issues @ github.com/grantjenks/python-diskcache/README.rst

What Would Be "Reinventing the Wheel"

Without diskcache, you would need to:

• Implement SQLite-based caching with proper locking and atomicity manually @ grantjenks.com/docs/diskcache
• Build eviction policies (LRU, LFU) from scratch @ grantjenks.com/docs/diskcache/tutorial.html
• Manage thread-safe and process-safe file system operations @ grantjenks.com/docs/diskcache
• Handle serialization, compression, and expiration logic manually @ grantjenks.com/docs/diskcache/tutorial.html
• Implement cache stampede prevention for memoization @ grantjenks.com/docs/diskcache/case-study-landing-page-caching.html

When to Use diskcache

Use diskcache When

1. Single-Machine Persistent Cache: You need persistent caching on one server without distributed requirements @ grantjenks.com/docs/diskcache
2. No External Cache Server: You want to avoid running and managing Redis/Memcached @ github.com/grantjenks/python-diskcache/README.rst
3. Process-Safe Caching: Multiple processes need to share cache data safely (web workers, background tasks) @ grantjenks.com/docs/diskcache/tutorial.html
4. Large Cache Size: You need gigabytes of cache that would be expensive in memory @ github.com/grantjenks/python-diskcache/README.rst
5. Django File Cache Replacement: Django's file cache is too slow for your needs @ grantjenks.com/docs/diskcache/djangocache-benchmarks.html
6. Memoization with Persistence: Function results should persist across process restarts @ grantjenks.com/docs/diskcache/tutorial.html
7. Tag-Based Eviction: You need to invalidate related cache entries by tag @ grantjenks.com/docs/diskcache/tutorial.html
8. Offline/Local Development: No network cache available in development environment @ grantjenks.com/docs/diskcache

Use Redis When

1. Distributed Caching: Multiple servers need to share the same cache @ grantjenks.com/docs/diskcache
2. Sub-Millisecond Latency Critical: Network latency acceptable for extreme speed requirements @ grantjenks.com/docs/diskcache/cache-benchmarks.html
3. Advanced Data Structures: Need Redis-specific types (sets, sorted sets, pub/sub) @ redis.io
4. Cache Replication: Require high availability and replication across nodes @ redis.io
5. Horizontal Scaling: Cache must scale across multiple machines @ redis.io

Use functools.lru_cache When

1. In-Memory Only: Cache doesn't need to persist across process restarts @ python.org/docs
2. Single Process: No multi-process cache sharing needed @ python.org/docs
3. Small Cache Size: Cache fits comfortably in memory (megabytes, not gigabytes) @ python.org/docs
4. Simple Memoization: No expiration, tags, or complex eviction needed @ python.org/docs

Decision Matrix

┌──────────────────────────────┬───────────┬─────────┬────────────────┬──────────┐
│ Requirement                  │ diskcache │ Redis   │ lru_cache      │ shelve   │
├──────────────────────────────┼───────────┼─────────┼────────────────┼──────────┤
│ Persistent storage           │ ✓         │ ✓*      │ ✗              │ ✓        │
│ Thread-safe                  │ ✓         │ ✓       │ ✓              │ ✗        │
│ Process-safe                 │ ✓         │ ✓       │ ✗              │ ✗        │
│ No external server           │ ✓         │ ✗       │ ✓              │ ✓        │
│ Eviction policies            │ LRU/LFU   │ LRU/LFU │ LRU only       │ None     │
│ Tag-based invalidation       │ ✓         │ Manual  │ ✗              │ ✗        │
│ Expiration support           │ ✓         │ ✓       │ ✗              │ ✗        │
│ Distributed caching          │ ✗         │ ✓       │ ✗              │ ✗        │
│ Django integration           │ ✓         │ ✓       │ ✗              │ ✗        │
│ Transactions                 │ ✓         │ ✓       │ ✗              │ ✗        │
│ Atomic operations            │ Always    │ ✓       │ ✓              │ Maybe    │
│ Memoization decorators       │ ✓         │ Manual  │ ✓              │ ✗        │
│ Typical latency (get)        │ 25 µs     │ 190 µs  │ 0.1 µs         │ 36 µs    │
│ Pure Python                  │ ✓         │ ✗       │ ✓              │ ✓        │
└──────────────────────────────┴───────────┴─────────┴────────────────┴──────────┘

@ Compiled from grantjenks.com/docs/diskcache, github.com/grantjenks/python-diskcache

Note: Redis persistence is optional and primarily for durability, not primary storage model.

Python Version Compatibility

Minimum Python Version: 3.0 @ github.com/grantjenks/python-diskcache/setup.py Officially Tested Versions: 3.6, 3.7, 3.8, 3.9, 3.10 @ github.com/grantjenks/python-diskcache/README.rst Development Version: 3.10 @ github.com/grantjenks/python-diskcache/README.rst

Python 3.11-3.14 Status:

• 3.11: Expected to work (no known incompatibilities)
• 3.12: Expected to work (no known incompatibilities)
• 3.13: Expected to work (no known incompatibilities)
• 3.14: Expected to work (pure Python with no C dependencies)

Dependencies: None - pure Python with standard library only @ github.com/grantjenks/python-diskcache/setup.py

Real-World Usage Examples

Example Projects Using diskcache

1. morss (722+ stars) @ github.com/pictuga/morss

   • Full-text RSS feed generator
   • Pattern: Caching HTTP responses and parsed feed data
   • URL: https://github.com/pictuga/morss

2. git-pandas (192+ stars) @ github.com/wdm0006/git-pandas

   • Git repository analysis with pandas dataframes
   • Pattern: Caching expensive git repository queries
   • URL: https://github.com/wdm0006/git-pandas

3. High-Traffic Website Caching @ grantjenks.com/docs/diskcache

   • Testimonial: "Reduced Elasticsearch queries by over 25% for 1M+ users/day (100+ hits/second)" - Daren Hasenkamp
   • Pattern: Database query result caching in production web applications

4. Ansible Automation @ grantjenks.com/docs/diskcache

   • Testimonial: "Sped up Ansible runs by almost 3 times" - Mathias Petermann
   • Pattern: Caching lookup module results across playbook runs

Common Usage Patterns @ grantjenks.com/docs/diskcache, exa.ai

# Pattern 1: Basic Cache Operations
from diskcache import Cache

cache = Cache('/tmp/mycache')

# Dictionary-like interface
cache['key'] = 'value'
print(cache['key'])  # 'value'
print('key' in cache)  # True
del cache['key']

# Method-based interface with expiration
cache.set('key', 'value', expire=300)  # 5 minutes
value = cache.get('key')
cache.delete('key')

# Cleanup
cache.close()

# Pattern 2: Function Memoization with Cache Decorator
from diskcache import Cache

cache = Cache('/tmp/mycache')

@cache.memoize()
def expensive_function(x, y):
    # Expensive computation
    import time
    time.sleep(2)
    return x + y

# First call takes 2 seconds
result = expensive_function(1, 2)  # Slow

# Second call is instant (cached)
result = expensive_function(1, 2)  # Fast!

# Pattern 3: Cache Stampede Prevention
from diskcache import Cache, memoize_stampede
import time

cache = Cache('/tmp/mycache')

@memoize_stampede(cache, expire=60, beta=0.3)
def generate_landing_page():
    """Prevents thundering herd when cache expires"""
    time.sleep(0.2)  # Simulate expensive computation
    return "<html>Landing Page</html>"

# Multiple concurrent requests won't cause stampede
result = generate_landing_page()

# Pattern 4: FanoutCache for High Concurrency
from diskcache import FanoutCache

# Sharded cache for concurrent writes
cache = FanoutCache('/tmp/mycache', shards=8, timeout=1.0)

# Same API as Cache but with better write concurrency
cache.set('key', 'value')
value = cache.get('key')

# Pattern 5: Tag-Based Eviction
from diskcache import Cache
from io import BytesIO

cache = Cache('/tmp/mycache', tag_index=True)  # Enable tag index

# Set items with tags
cache.set('user:1:profile', data1, tag='user:1')
cache.set('user:1:posts', data2, tag='user:1')
cache.set('user:1:friends', data3, tag='user:1')

# Evict all items for a specific tag
cache.evict('user:1')

# Pattern 6: Web Crawler with Persistent Storage
from diskcache import Index

# Persistent dictionary for crawled URLs
results = Index('data/results')

# Store crawled data
results['https://example.com'] = {
    'html': '<html>...</html>',
    'timestamp': '2025-10-21',
    'status': 200
}

# Query persistent results
print(len(results))
if 'https://example.com' in results:
    data = results['https://example.com']

# Pattern 7: Django Cache Configuration
# settings.py
CACHES = {
    'default': {
        'BACKEND': 'diskcache.DjangoCache',
        'LOCATION': '/var/cache/django',
        'TIMEOUT': 300,
        'SHARDS': 8,
        'DATABASE_TIMEOUT': 0.010,  # 10 milliseconds
        'OPTIONS': {
            'size_limit': 2 ** 30  # 1 GB
        },
    },
}

# Pattern 8: Async Operation with asyncio
import asyncio
from diskcache import Cache

cache = Cache('/tmp/mycache')

async def set_async(key, value):
    loop = asyncio.get_running_loop()
    await loop.run_in_executor(None, cache.set, key, value)

async def get_async(key):
    loop = asyncio.get_running_loop()
    return await loop.run_in_executor(None, cache.get, key)

# Use in async functions
await set_async('test-key', 'test-value')
value = await get_async('test-key')

# Pattern 9: Custom Serialization with JSONDisk
import json
import zlib
from diskcache import Cache, Disk, UNKNOWN

class JSONDisk(Disk):
    def __init__(self, directory, compress_level=1, **kwargs):
        self.compress_level = compress_level
        super().__init__(directory, **kwargs)

    def put(self, key):
        json_bytes = json.dumps(key).encode('utf-8')
        data = zlib.compress(json_bytes, self.compress_level)
        return super().put(data)

    def get(self, key, raw):
        data = super().get(key, raw)
        return json.loads(zlib.decompress(data).decode('utf-8'))

    def store(self, value, read, key=UNKNOWN):
        if not read:
            json_bytes = json.dumps(value).encode('utf-8')
            value = zlib.compress(json_bytes, self.compress_level)
        return super().store(value, read, key=key)

    def fetch(self, mode, filename, value, read):
        data = super().fetch(mode, filename, value, read)
        if not read:
            data = json.loads(zlib.decompress(data).decode('utf-8'))
        return data

# Use custom disk implementation
cache = Cache('/tmp/mycache', disk=JSONDisk, disk_compress_level=6)

# Pattern 10: Cross-Process Locking
from diskcache import Lock
import time

lock = Lock(cache, 'resource-name')

with lock:
    # Critical section - only one process executes at a time
    print("Exclusive access to resource")
    time.sleep(1)

# Pattern 11: Rate Limiting / Throttling
from diskcache import throttle

@throttle(cache, count=10, seconds=60)
def api_call():
    """Allow only 10 calls per minute"""
    return make_expensive_api_request()

# Raises exception if rate limit exceeded
try:
    api_call()
except Exception:
    print("Rate limit exceeded")

@ Compiled from grantjenks.com/docs/diskcache, exa.ai/get_code_context

Integration Patterns

Django Integration @ grantjenks.com/docs/diskcache/tutorial.html

# settings.py
CACHES = {
    'default': {
        'BACKEND': 'diskcache.DjangoCache',
        'LOCATION': '/path/to/cache/directory',
        'TIMEOUT': 300,
        'SHARDS': 8,
        'DATABASE_TIMEOUT': 0.010,
        'OPTIONS': {
            'size_limit': 2 ** 30   # 1 gigabyte
        },
    },
}

# Usage in views
from django.core.cache import cache

def my_view(request):
    result = cache.get('my_key')
    if result is None:
        result = expensive_computation()
        cache.set('my_key', result, timeout=300)
    return result

FastAPI with Async Caching @ exa.ai, calmcode.io

from fastapi import FastAPI
import httpx
from diskcache import Cache
import asyncio

app = FastAPI()
cache = Cache('/tmp/api_cache')

async def cached_api_call(url: str):
    # Check cache
    if url in cache:
        print(f'Using cached content for {url}')
        return cache[url]

    print(f'Making new request for {url}')
    # Make async request
    async with httpx.AsyncClient(timeout=10) as client:
        response = await client.get(url)
        html = response.text
        cache[url] = html
        return html

@app.get("/fetch")
async def fetch_data(url: str):
    content = await cached_api_call(url)
    return {"content": content[:1000]}

Multi-Process Web Crawler @ grantjenks.com/docs/diskcache/case-study-web-crawler.html

from diskcache import Index, Deque
from multiprocessing import Process
import requests

# Shared queue and results across processes
todo = Deque('data/todo')
results = Index('data/results')

def crawl():
    while True:
        try:
            url = todo.popleft()
        except IndexError:
            break

        response = requests.get(url)
        results[url] = response.text

        # Add discovered URLs to queue
        for link in extract_links(response.text):
            todo.append(link)

# Start multiple crawler processes
processes = [Process(target=crawl) for _ in range(4)]
for process in processes:
    process.start()
for process in processes:
    process.join()

print(f"Crawled {len(results)} pages")

Installation

Basic Installation @ grantjenks.com/docs/diskcache

pip install diskcache

Using uv (Recommended) @ astral.sh

uv add diskcache

Development Installation @ grantjenks.com/docs/diskcache/development.rst

git clone https://github.com/grantjenks/python-diskcache.git
cd python-diskcache
pip install -r requirements.txt

Core API Components

Cache Class @ grantjenks.com/docs/diskcache/tutorial.html

The basic cache implementation backed by SQLite.

from diskcache import Cache

# Initialize cache
cache = Cache(directory='/tmp/mycache')

# Dictionary-like operations
cache['key'] = 'value'
value = cache['key']
'key' in cache  # True
del cache['key']

# Method-based operations
cache.set('key', 'value', expire=60, tag='category')
value = cache.get('key', default=None, read=False,
                  expire_time=False, tag=False)
cache.delete('key')
cache.clear()

# Statistics and management
cache.volume()  # Estimated disk usage
cache.stats(enable=True, reset=False)  # (hits, misses)
cache.evict('tag')  # Remove all entries with tag
cache.expire()  # Remove expired entries
cache.close()

FanoutCache Class @ grantjenks.com/docs/diskcache/tutorial.html

Sharded cache for high-concurrency write scenarios.

from diskcache import FanoutCache

# Sharded cache (default 8 shards)
cache = FanoutCache(
    directory='/tmp/mycache',
    shards=8,
    timeout=1.0,
    disk=Disk,
    disk_min_file_size=2 ** 15
)

# Same API as Cache
cache.set('key', 'value')
value = cache.get('key')

Eviction Policies @ grantjenks.com/docs/diskcache/tutorial.html

Four eviction policies control what happens when cache size limit is reached:

from diskcache import Cache

# least-recently-stored (default) - fastest
cache = Cache(eviction_policy='least-recently-stored')

# least-recently-used - updates on read
cache = Cache(eviction_policy='least-recently-used')

# least-frequently-used - tracks access count
cache = Cache(eviction_policy='least-frequently-used')

# none - no eviction, unbounded growth
cache = Cache(eviction_policy='none')

Performance Characteristics:

• least-recently-stored: Fastest (no read updates)
• least-recently-used: Slower (updates timestamp on read)
• least-frequently-used: Slowest (increments counter on read)
• none: Fastest (no eviction overhead)

Deque and Index Classes @ grantjenks.com/docs/diskcache/tutorial.html

Persistent, process-safe data structures.

from diskcache import Deque, Index

# Persistent deque (FIFO queue)
deque = Deque('data/queue')
deque.append('item')
deque.appendleft('item')
item = deque.pop()
item = deque.popleft()

# Persistent dictionary
index = Index('data/index')
index['key'] = 'value'
value = index['key']

Performance Benchmarks

Single Process Performance @ grantjenks.com/docs/diskcache/cache-benchmarks.html

diskcache.Cache:
  get:    19.073 µs (median)
  set:   114.918 µs (median)
  delete: 87.976 µs (median)

pylibmc.Client (Memcached):
  get:    42.915 µs (median)
  set:    44.107 µs (median)
  delete: 41.962 µs (median)

Comparison vs alternatives:
  dbm:      get 36µs, set 900µs, delete 740µs
  shelve:   get 41µs, set 928µs, delete 702µs
  sqlitedict: get 513µs, set 697µs, delete 1717µs
  pickleDB: get 92µs, set 1020µs, delete 1020µs

Multi-Process Performance (8 processes) @ grantjenks.com/docs/diskcache/cache-benchmarks.html

diskcache.Cache:
  get:    20.027 µs (median)
  set:   129.700 µs (median)
  delete: 97.036 µs (median)

redis.StrictRedis:
  get:   187.874 µs (median)
  set:   192.881 µs (median)
  delete: 185.966 µs (median)

pylibmc.Client:
  get:    95.844 µs (median)
  set:    97.036 µs (median)
  delete: 94.891 µs (median)

Key Insight: diskcache is faster than network-based caches (Redis, Memcached) for single-machine workloads, especially for reads. @ grantjenks.com/docs/diskcache

Django Cache Backend Performance @ grantjenks.com/docs/diskcache/djangocache-benchmarks.html

diskcache DjangoCache:
  get:    55.075 µs (median)
  set:   303.984 µs (median)
  delete: 228.882 µs (median)
  Total:  98.465s

redis DjangoCache:
  get:   214.100 µs (median)
  set:   230.789 µs (median)
  delete: 195.742 µs (median)
  Total: 174.069s

filebased DjangoCache:
  get:   114.918 µs (median)
  set:    11.289 ms (median)
  delete: 432.014 µs (median)
  Total: 907.537s

Key Insight: diskcache is 1.8x faster than Redis and 9.2x faster than Django's file-based cache. @ grantjenks.com/docs/diskcache/djangocache-benchmarks.html

When NOT to Use diskcache

Scenarios Where diskcache May Not Be Suitable

1. Distributed Systems @ grantjenks.com/docs/diskcache

   • diskcache is single-machine only
   • Use Redis, Memcached, or distributed caches for multi-server architectures
   • Cannot share cache across network nodes

2. Extremely Low Latency Required @ grantjenks.com/docs/diskcache/cache-benchmarks.html

   • In-memory caches (lru_cache, dict) are faster for frequently accessed data
   • diskcache adds disk I/O overhead (~20µs vs ~0.1µs)
   • Consider in-memory + diskcache two-tier strategy

3. Small Cache (< 100MB) @ github.com/grantjenks/python-diskcache

   • functools.lru_cache more appropriate for small in-memory caches
   • Overhead of SQLite not justified for tiny caches
   • Use lru_cache for simplicity

4. Read-Only Access Patterns @ grantjenks.com/docs/diskcache

   • If cache is never updated after initialization
   • Simple dict or frozen data structures may be simpler
   • No eviction or expiration needed

5. Cache Needs to Survive Disk Failures @ grantjenks.com/docs/diskcache

   • diskcache stores on local disk
   • Disk failure = cache loss
   • Redis with persistence and replication for critical caches

6. Need Atomic Multi-Key Operations @ grantjenks.com/docs/diskcache

   • diskcache operations are single-key atomic
   • No native support for transactions across multiple keys
   • Redis supports MULTI/EXEC for atomic multi-key operations

7. Advanced Data Structures Required @ redis.io

   • diskcache is key-value only
   • Redis provides sets, sorted sets, lists, streams, etc.
   • Use Redis if you need these structures

Key Features

Thread and Process Safety @ grantjenks.com/docs/diskcache/tutorial.html

All operations are atomic and safe for concurrent access:

from diskcache import Cache
from multiprocessing import Process

cache = Cache('/tmp/shared')

def worker(worker_id):
    for i in range(1000):
        cache[f'worker_{worker_id}_key_{i}'] = f'value_{i}'

# Safe concurrent writes from multiple processes
processes = [Process(target=worker, args=(i,)) for i in range(4)]
for p in processes:
    p.start()
for p in processes:
    p.join()

Expiration and TTL @ grantjenks.com/docs/diskcache/tutorial.html

from diskcache import Cache
import time

cache = Cache()

# Set with expiration
cache.set('key', 'value', expire=5)  # 5 seconds

time.sleep(6)
print(cache.get('key'))  # None (expired)

# Manual expiration cleanup
cache.expire()  # Remove all expired entries

Tag-Based Invalidation @ grantjenks.com/docs/diskcache/tutorial.html

from diskcache import Cache

cache = Cache(tag_index=True)  # Enable tag index for performance

# Tag cache entries
cache.set('user:1:profile', data1, tag='user:1')
cache.set('user:1:settings', data2, tag='user:1')
cache.set('user:2:profile', data3, tag='user:2')

# Evict all entries for a tag
count = cache.evict('user:1')
print(f"Evicted {count} entries")

Statistics and Monitoring @ grantjenks.com/docs/diskcache/tutorial.html

from diskcache import Cache

cache = Cache()

# Enable statistics tracking
cache.stats(enable=True)

# Perform operations
for i in range(100):
    cache.set(i, i)

for i in range(150):
    cache.get(i)

# Get statistics
hits, misses = cache.stats(enable=False, reset=True)
print(f"Hits: {hits}, Misses: {misses}")  # Hits: 100, Misses: 50

# Get cache size
volume = cache.volume()
print(f"Cache volume: {volume} bytes")

Custom Serialization @ grantjenks.com/docs/diskcache/tutorial.html

from diskcache import Cache, Disk, UNKNOWN
import pickle
import zlib

class CompressedDisk(Disk):
    def put(self, key):
        data = pickle.dumps(key)
        compressed = zlib.compress(data)
        return super().put(compressed)

    def get(self, key, raw):
        compressed = super().get(key, raw)
        data = zlib.decompress(compressed)
        return pickle.loads(data)

cache = Cache(disk=CompressedDisk)

Migration and Compatibility

From functools.lru_cache @ python.org/docs, grantjenks.com/docs/diskcache

# Before: In-memory only
from functools import lru_cache

@lru_cache(maxsize=128)
def expensive_function(x):
    return x * 2

# After: Persistent across restarts
from diskcache import Cache

cache = Cache('/tmp/mycache')

@cache.memoize()
def expensive_function(x):
    return x * 2

From Django File Cache @ grantjenks.com/docs/diskcache/tutorial.html

# Before: Django's slow file cache
CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.filebased.FileBasedCache',
        'LOCATION': '/var/tmp/django_cache',
    }
}

# After: Fast diskcache
CACHES = {
    'default': {
        'BACKEND': 'diskcache.DjangoCache',
        'LOCATION': '/var/tmp/django_cache',
        'TIMEOUT': 300,
        'SHARDS': 8,
        'OPTIONS': {
            'size_limit': 2 ** 30
        }
    }
}

From Redis (Single Machine) @ grantjenks.com/docs/diskcache

# Before: Redis client
import redis
r = redis.Redis(host='localhost', port=6379)
r.set('key', 'value')
value = r.get('key')

# After: diskcache (no server needed)
from diskcache import Cache
cache = Cache('/tmp/mycache')
cache.set('key', 'value')
value = cache.get('key')

Advanced Patterns

Cache Warming @ grantjenks.com/docs/diskcache

from diskcache import Cache

def warm_cache():
    cache = Cache('/tmp/mycache')

    # Pre-populate cache with common queries
    common_queries = load_common_queries()

    for query in common_queries:
        result = expensive_database_query(query)
        cache.set(f'query:{query}', result, expire=3600)

    print(f"Warmed cache with {len(common_queries)} entries")

Two-Tier Caching @ grantjenks.com/docs/diskcache

from functools import lru_cache
from diskcache import Cache

disk_cache = Cache('/tmp/mycache')

@lru_cache(maxsize=100)  # Fast in-memory tier
def get_from_memory(key):
    # Fall back to disk cache
    return disk_cache.get(key)

def get_value(key):
    # Try memory first (fast)
    value = get_from_memory(key)

    if value is None:
        # Fetch from source and cache both tiers
        value = expensive_operation(key)
        disk_cache.set(key, value, expire=3600)
        get_from_memory.cache_clear()  # Invalidate memory
        get_from_memory(key)  # Warm memory cache

    return value

Testing and Development

Temporary Cache for Tests @ grantjenks.com/docs/diskcache

import tempfile
import shutil
from diskcache import Cache

def test_cache_operations():
    # Create temporary cache directory
    tmpdir = tempfile.mkdtemp()

    try:
        cache = Cache(tmpdir)

        # Test operations
        cache.set('key', 'value')
        assert cache.get('key') == 'value'

        cache.close()
    finally:
        # Cleanup
        shutil.rmtree(tmpdir, ignore_errors=True)

Context Manager for Cleanup @ grantjenks.com/docs/diskcache/tutorial.html

from diskcache import Cache

# Automatic cleanup with context manager
with Cache('/tmp/mycache') as cache:
    cache.set('key', 'value')
    value = cache.get('key')
# cache.close() called automatically

Additional Resources

Official Documentation @ grantjenks.com/docs/diskcache

• Tutorial: https://grantjenks.com/docs/diskcache/tutorial.html
• Cache Benchmarks: https://grantjenks.com/docs/diskcache/cache-benchmarks.html
• Django Benchmarks: https://grantjenks.com/docs/diskcache/djangocache-benchmarks.html
• Case Study - Web Crawler: https://grantjenks.com/docs/diskcache/case-study-web-crawler.html
• Case Study - Landing Page: https://grantjenks.com/docs/diskcache/case-study-landing-page-caching.html
• API Reference: https://grantjenks.com/docs/diskcache/api.html

Community Resources

• GitHub Repository: https://github.com/grantjenks/python-diskcache @ github.com
• Issue Tracker: https://github.com/grantjenks/python-diskcache/issues @ github.com
• PyPI Package: https://pypi.org/project/diskcache/ @ pypi.org
• Author's Blog: https://grantjenks.com/ @ grantjenks.com

Related Projects by Author

• sortedcontainers: Fast pure-Python sorted collections @ github.com/grantjenks/python-sortedcontainers
• wordsegment: English word segmentation @ github.com/grantjenks/python-wordsegment
• runstats: Online statistics and regression @ github.com/grantjenks/python-runstats

Summary

diskcache is the ideal choice for single-machine persistent caching when you need:

• Process-safe caching without running a separate server
• Gigabytes of cache using disk space instead of memory
• Better performance than Django's file cache or network caches for local workloads
• Memoization that persists across process restarts
• Tag-based invalidation for related cache entries
• Multiple eviction policies (LRU, LFU)

It provides production-grade reliability with 100% test coverage, extensive benchmarking, and stress testing. For distributed systems or when network latency is acceptable, Redis remains the better choice. For small in-memory caches, use functools.lru_cache.

Performance Highlight: diskcache can be faster than Redis and Memcached for single-machine workloads because it eliminates network overhead (19µs get vs 187µs for Redis). @ grantjenks.com/docs/diskcache/cache-benchmarks.html

---

Research completed: 2025-10-21 @ Claude Code Agent Sources verified: GitHub, Context7, PyPI, Official Documentation, Exa Code Context @ Multiple verified sources Confidence level: High - All information cross-referenced from official sources and benchmarks