zhongwei/gh-dnvriend-gemini-file-search-tool-plugins-gemini-file-search-tool

Files

Zhongwei Li 962cc0c926 Initial commit

2025-11-29 18:23:15 +08:00

21 KiB

Raw Blame History

name, description

name	description
skill-gemini-file-search-tool	Manage Gemini RAG stores with Code-RAG support

When to use

Managing Gemini File Search stores and documents
Uploading documents for RAG queries (including codebases)
Querying stores with natural language
Building Code-RAG systems for semantic code search

Gemini File Search Tool Skill

Purpose

Comprehensive guide for managing Google's fully managed RAG (Retrieval-Augmented Generation) system using the gemini-file-search-tool CLI. This tool eliminates the complexity of vector databases, embeddings, and retrieval infrastructure by providing a production-ready interface to Gemini File Search.

When to Use This Skill

Use this skill when:

Creating and managing Gemini File Search stores
Uploading documents for semantic search and RAG queries
Building Code-RAG systems (semantic code search with natural language)
Querying document stores with natural language
Managing upload caches and operation status
Implementing document search in AI applications

Do NOT use this skill for:

General document processing (use document-skills instead)
File system operations (use Read/Write tools)
Cloud infrastructure management (use AWS/GCP CLIs)

CLI Tool: gemini-file-search-tool

Production-ready CLI and Python library for Google's Gemini File Search API, a fully managed RAG system that automatically handles document ingestion, chunking, embedding generation, and semantic retrieval with zero infrastructure overhead.

Installation

# Clone repository
git clone https://github.com/dnvriend/gemini-file-search-tool.git
cd gemini-file-search-tool

# Install globally with uv
uv tool install .

# Verify installation
gemini-file-search-tool --help

Prerequisites

Authentication (Choose one):

Option 1: Gemini Developer API (Recommended for development)

export GEMINI_API_KEY="your-api-key-here"
# Or
export GOOGLE_API_KEY="your-api-key-here"

Get API key from: https://aistudio.google.com/apikey

Option 2: Vertex AI (For production)

export GOOGLE_GENAI_USE_VERTEXAI=true
export GOOGLE_CLOUD_PROJECT="your-project-id"
export GOOGLE_CLOUD_LOCATION="us-central1"

Quick Start

# Create a store
gemini-file-search-tool create-store "my-docs"

# Upload documents
gemini-file-search-tool upload "*.pdf" --store "my-docs" -v

# Query with natural language
gemini-file-search-tool query "What is the main topic?" --store "my-docs" --show-cost

Progressive Disclosure

📖 Store Management Commands (Click to expand)

create-store - Create New Store

Create a new Gemini File Search store for document storage and RAG queries.

Usage:

gemini-file-search-tool create-store "STORE_NAME" [--display-name NAME] [-v]

Arguments:

STORE_NAME: Store identifier (required, positional)
--display-name NAME: Human-readable display name (optional)
-v: Verbose logging

Examples:

# Create store with auto-generated display name
gemini-file-search-tool create-store "research-papers"

# Create store with custom display name
gemini-file-search-tool create-store "docs" --display-name "Project Documentation"

# Capture output for processing
gemini-file-search-tool create-store "code" | jq '.name'

Output:

{
  "name": "fileSearchStores/abc123",
  "display_name": "research-papers",
  "create_time": "2025-11-20T10:30:00Z",
  "update_time": "2025-11-20T10:30:00Z"
}

list-stores - List All Stores

List all available Gemini File Search stores with their metadata.

Usage:

gemini-file-search-tool list-stores [-v]

Arguments:

-v: Verbose logging

Examples:

# List all stores
gemini-file-search-tool list-stores

# List with verbose logging
gemini-file-search-tool list-stores -v

# Filter stores with jq
gemini-file-search-tool list-stores | jq '.[] | select(.display_name | contains("docs"))'

# Count stores
gemini-file-search-tool list-stores | jq 'length'

Output:

[
  {
    "name": "fileSearchStores/abc123",
    "display_name": "research-papers",
    "create_time": "2025-11-20T10:30:00Z",
    "update_time": "2025-11-20T10:30:00Z"
  }
]

get-store - Get Store Details

Get detailed information about a specific store.

Usage:

gemini-file-search-tool get-store "STORE_NAME" [-v]

Arguments:

STORE_NAME: Store name/ID (accepts display names, IDs, or full resource names)
-v: Verbose logging

Examples:

# Get by display name
gemini-file-search-tool get-store "research-papers"

# Get by ID
gemini-file-search-tool get-store "abc123"

# Extract specific field
gemini-file-search-tool get-store "docs" | jq '.create_time'

update-store - Update Store

Update a store's display name or metadata.

Usage:

gemini-file-search-tool update-store "STORE_NAME" --display-name "NEW_NAME" [-v]

Examples:

# Rename store
gemini-file-search-tool update-store "docs" --display-name "Production Documentation"

delete-store - Delete Store

Delete a Gemini File Search store and all its documents.

Usage:

gemini-file-search-tool delete-store "STORE_NAME" [--force] [-v]

Arguments:

STORE_NAME: Store to delete (required)
--force: Skip confirmation prompt
-v: Verbose logging

Examples:

# Delete with confirmation
gemini-file-search-tool delete-store "old-docs"

# Delete without confirmation
gemini-file-search-tool delete-store "temp-store" --force

Note: Shows cache statistics before deletion and automatically removes cache file after successful deletion.

📁 Document Management Commands (Click to expand)

upload - Upload Documents

Upload files to a Gemini File Search store with intelligent caching, glob support, and parallel processing.

Usage:

gemini-file-search-tool upload FILES... --store "STORE_NAME" [OPTIONS]

Arguments:

FILES: File paths or glob patterns (required, positional)
--store NAME: Target store name (required)
--title TEXT: Custom metadata title (optional)
--url URL: Custom metadata URL (optional)
--max-tokens N: Max tokens per chunk (default: 200)
--max-overlap N: Max overlap tokens (default: 20)
--num-workers N: Concurrent workers (default: CPU cores)
--skip-validation: Skip file validation checks
--ignore-gitignore: Ignore .gitignore patterns
--dry-run: Preview files without uploading
--rebuild-cache: Force re-upload all files
--no-wait: Async upload without polling
-v/-vv/-vvv: Verbosity (INFO/DEBUG/TRACE)

Key Features:

Intelligent Caching: Automatically skips unchanged files using mtime-based optimization (O(1) check)
Glob Patterns: Supports *.pdf, docs/**/*.md, src/**/*.py
Gitignore Support: Respects .gitignore patterns by default
Parallel Processing: Concurrent uploads with configurable workers
System File Filtering: Auto-skips __pycache__, .pyc, .DS_Store
Async Mode: Fire-and-forget uploads with --no-wait

Examples:

# Upload single file
gemini-file-search-tool upload document.pdf --store "papers"

# Upload multiple files
gemini-file-search-tool upload doc1.pdf doc2.pdf --store "papers"

# Upload with glob pattern
gemini-file-search-tool upload "*.pdf" --store "papers" -v

# Upload recursive with markdown files
gemini-file-search-tool upload "docs/**/*.md" --store "documentation" -v

# Upload codebase for Code-RAG
gemini-file-search-tool upload "src/**/*.py" --store "my-codebase" -v

# Async upload with 8 workers
gemini-file-search-tool upload "*.pdf" --store "papers" --no-wait --num-workers 8

# Dry-run to preview files
gemini-file-search-tool upload "**/*.py" --store "code" --dry-run -v

# Rebuild cache (force re-upload)
gemini-file-search-tool upload "**/*.py" --store "code" --rebuild-cache

Output:

[
  {"file": "doc1.pdf", "status": "completed", "document": {"name": "documents/123"}},
  {"file": "doc2.pdf", "status": "skipped", "reason": "Already exists"},
  {"file": "doc3.pdf", "status": "pending", "operation": "operations/456"}
]

list-documents - List Documents in Store

List all documents in a Gemini File Search store.

Usage:

gemini-file-search-tool list-documents --store "STORE_NAME" [-v/-vv/-vvv]

Arguments:

--store NAME: Store name (required)
-v/-vv/-vvv: Verbosity levels

Examples:

# List all documents
gemini-file-search-tool list-documents --store "papers"

# With verbose logging
gemini-file-search-tool list-documents --store "papers" -vv

# Count documents
gemini-file-search-tool list-documents --store "papers" | jq 'length'

Note: Uses REST API workaround due to SDK bug #1661. Only works with Developer API (not Vertex AI).

🔍 Query Commands (Click to expand)

query - Natural Language RAG Queries

Query a Gemini File Search store with natural language using Retrieval-Augmented Generation.

Usage:

gemini-file-search-tool query "PROMPT" --store "STORE_NAME" [OPTIONS]

Arguments:

PROMPT: Natural language query (required, positional)
--store NAME: Store to query (required)
--query-model {flash|pro}: Model for RAG query (default: flash)
--query-grounding-metadata: Include source citations in response
--metadata-filter "key=value": Filter documents by metadata
--show-cost: Include token usage and cost estimation
-v/-vv/-vvv: Verbosity (INFO/DEBUG/TRACE)

Enhancement Options (Use Sparingly):

--enhance-mode {generic|code-rag|obsidian}: Query optimization mode (disabled by default)
--enhancement-model {flash|pro}: Model for enhancement (default: flash)
--show-enhancement: Display enhanced query
--dry-run-enhancement: Preview enhancement without executing query

Examples:

# Basic query
gemini-file-search-tool query "What is DORA?" --store "papers"

# Query with citations and cost tracking
gemini-file-search-tool query "How does authentication work?" \
  --store "codebase" --query-grounding-metadata --show-cost -v

# Query with Pro model
gemini-file-search-tool query "Explain the architecture" \
  --store "docs" --query-model pro

# Code-RAG query (semantic code search)
gemini-file-search-tool query "Where is error handling implemented?" \
  --store "my-codebase" --query-grounding-metadata

# With metadata filtering
gemini-file-search-tool query "Python best practices" \
  --store "docs" --metadata-filter "language=python"

Output:

{
  "response_text": "DORA stands for DevOps Research and Assessment...",
  "usage_metadata": {
    "prompt_token_count": 150,
    "candidates_token_count": 320,
    "total_token_count": 470
  },
  "grounding_metadata": {
    "grounding_chunks": [
      {
        "retrieved_context": {
          "title": "/path/to/doc.pdf",
          "text": "Relevant excerpt..."
        }
      }
    ]
  },
  "estimated_cost": {
    "total_cost_usd": 0.00010725,
    "model": "gemini-2.5-flash"
  }
}

Important Notes:

Enhancement is disabled by default: Benchmarks show simple queries work best for RAG
Flash model recommended: 12x more cost-efficient than Pro, adequate for RAG
Only use --enhance-mode for vague or poorly worded queries

💾 Cache Management Commands (Click to expand)

sync-cache - Sync Pending Operations

Synchronize pending upload operations and update cache with final status.

Usage:

gemini-file-search-tool sync-cache --store "STORE_NAME" [OPTIONS]

Arguments:

--store NAME: Store name (required)
--num-workers N: Parallel workers (default: 4)
--text: Human-readable output (default: JSON)
-v: Verbose logging

Examples:

# Sync with default workers
gemini-file-search-tool sync-cache --store "papers"

# Sync with 8 parallel workers
gemini-file-search-tool sync-cache --store "codebase" --num-workers 8 -v

# Human-readable output
gemini-file-search-tool sync-cache --store "docs" --text

When to Use:

After upload --no-wait operations
To check status of pending uploads
To update cache with final document IDs

flush-cache - Clear Cache File

Delete the cache file for a specific store to start fresh.

Usage:

gemini-file-search-tool flush-cache --store "STORE_NAME" [--force] [-v]

Arguments:

--store NAME: Store name (required)
--force: Skip confirmation prompt
-v: Verbose logging

Examples:

# Flush with confirmation
gemini-file-search-tool flush-cache --store "old-docs"

# Flush without confirmation
gemini-file-search-tool flush-cache --store "temp" --force

When to Use:

Before rebuilding cache
When cache is corrupted
For clean slate troubleshooting

cache-report - Generate Cache Status Report

Show cache statistics and operation status.

Usage:

gemini-file-search-tool cache-report --store "STORE_NAME" [FILTERS] [OPTIONS]

Arguments:

--store NAME: Store name (required)
--pending-only: Show only pending operations
--errors-only: Show only failed operations
--completed-only: Show only completed operations
--all: Show all cached files
--text: Human-readable output (default: JSON)
-v: Verbose logging

Examples:

# Show summary + pending operations
gemini-file-search-tool cache-report --store "papers"

# Show only failed operations
gemini-file-search-tool cache-report --store "docs" --errors-only

# Show all cached files
gemini-file-search-tool cache-report --store "code" --all --text

💻 Code-RAG: Semantic Code Search (Click to expand)

What is Code-RAG?

Code-RAG (Retrieval-Augmented Generation for Code) enables uploading entire codebases and querying them with natural language. This is powerful for:

Codebase Onboarding: New developers ask questions about architecture
Code Discovery: Find implementations without grepping
Architecture Analysis: Understand design patterns and structure
Documentation Generation: Generate contextual docs from code
AI Coding Assistants: Build agents that answer codebase questions

Code-RAG Workflow

Step 1: Upload Codebase

# Upload all Python files
gemini-file-search-tool upload "src/**/*.py" --store "my-project" -v

# Upload multiple languages
gemini-file-search-tool upload "src/**/*.{py,js,go}" --store "polyglot-project" -v

# Upload with custom chunking
gemini-file-search-tool upload "**/*.py" --store "large-project" \
  --max-tokens 500 --max-overlap 50 --num-workers 8

Step 2: Query with Natural Language

# Architectural questions
gemini-file-search-tool query "How does the authentication system work?" \
  --store "my-project" --query-grounding-metadata -v

# Implementation discovery
gemini-file-search-tool query "Where is error handling for API calls implemented?" \
  --store "my-project" --show-cost

# Design pattern analysis
gemini-file-search-tool query "What design patterns are used in this codebase?" \
  --store "my-project" --query-model pro

Step 3: Get Source References

# Query with full citations
gemini-file-search-tool query "Explain the database layer" \
  --store "my-project" --query-grounding-metadata | \
  jq '.grounding_metadata.grounding_chunks[].retrieved_context.title'

Code-RAG Best Practices

Comprehensive Docstrings: Well-documented code provides better semantic search results
Modular Structure: Clear file/function organization improves retrieval accuracy
Use Flash Model: Cost-effective and sufficient for code search
Enable Grounding: Always use --query-grounding-metadata to verify sources
Gitignore Support: Automatically skips __pycache__, build artifacts, etc.

Meta Note

This tool itself was built using Code-RAG! During development, we uploaded the codebase to a Gemini File Search store and queried it to understand implementation details, find bugs, and plan features. The tool enables the very functionality it provides.

⚙️ Advanced Features (Click to expand)

Intelligent Caching System

Location: ~/.config/gemini-file-search-tool/stores/

How it Works:

Tracks uploaded files per store in JSON cache files
Uses mtime-based optimization (O(1) check before O(n) hash)
For 1000 unchanged files: ~0.1s (mtime) vs ~5-10s (full hash)
Automatically skips unchanged files on re-upload
Per-store isolation prevents cache conflicts

Cache Structure:

{
  "/absolute/path/to/file.py": {
    "hash": "sha256-hash",
    "mtime": 1731969000.0,
    "remote_id": "documents/123",
    "last_uploaded": "2025-11-18T22:30:00Z"
  }
}

Cache States:

Completed: Has remote_id, file successfully uploaded
Pending: Has operation, async upload in progress
Missing: Not in cache, needs upload

Parallel Processing

Upload Concurrency:

# Default: Uses CPU core count
gemini-file-search-tool upload "*.pdf" --store "papers"

# Custom workers
gemini-file-search-tool upload "*.pdf" --store "papers" --num-workers 8

Sync Concurrency:

# Parallel operation status checking
gemini-file-search-tool sync-cache --store "papers" --num-workers 8

Cost Tracking

Query Cost Estimation:

gemini-file-search-tool query "What is this?" --store "docs" --show-cost

Output:

{
  "estimated_cost": {
    "input_cost_usd": 0.00001125,
    "output_cost_usd": 0.000096,
    "total_cost_usd": 0.00010725,
    "model": "gemini-2.5-flash"
  }
}

Current Pricing (2025-01):

Flash: $0.075 input / $0.30 output per 1M tokens
Pro: $1.25 input / $5.00 output per 1M tokens

Authentication Methods

Developer API (Recommended for Development):

export GEMINI_API_KEY="AIza..."
gemini-file-search-tool list-stores

Vertex AI (Recommended for Production):

export GOOGLE_GENAI_USE_VERTEXAI=true
export GOOGLE_CLOUD_PROJECT="my-project"
export GOOGLE_CLOUD_LOCATION="us-central1"
gemini-file-search-tool list-stores

🔧 Troubleshooting (Click to expand)

Common Issues

Issue: Authentication error - missing API key

Error: GEMINI_API_KEY or GOOGLE_API_KEY environment variable is required

Solution:

Get API key from https://aistudio.google.com/apikey
Set environment variable:
```
export GEMINI_API_KEY="your-api-key"
```

Issue: File not uploading

Solution:

Check if file is in cache and unchanged:

gemini-file-search-tool cache-report --store "your-store" | grep "your-file"

Force re-upload with --rebuild-cache:

gemini-file-search-tool upload "file.pdf" --store "your-store" --rebuild-cache

Issue: Pending operations not completing

Solution: Run sync-cache to check operation status:

gemini-file-search-tool sync-cache --store "your-store" -v

Issue: Store not found

Solution:

List all stores:
```
gemini-file-search-tool list-stores
```
Use exact display name or ID from list output

Issue: Query returning no results

Solution:

Verify documents were uploaded successfully:

gemini-file-search-tool list-documents --store "your-store"

Check if uploads are pending:

gemini-file-search-tool cache-report --store "your-store" --pending-only

Try simpler query (avoid over-specification)

Getting Help

# General help
gemini-file-search-tool --help

# Command-specific help
gemini-file-search-tool upload --help
gemini-file-search-tool query --help

# Verbose logging for debugging
gemini-file-search-tool upload "file.pdf" --store "test" -vvv

Known Issues

SDK Bug #1661 - Document Listing

list-documents uses REST API workaround
Only works with Developer API (not Vertex AI)
GitHub: https://github.com/googleapis/python-genai/issues/1661

Exit Codes

0: Success
1: Error (authentication, validation, API error)

Output Formats

JSON (Default): All commands output JSON to stdout, logs to stderr for easy piping:

gemini-file-search-tool list-stores | jq '.[] | .display_name'

Text (Select Commands): Some commands support --text flag for human-readable output:

gemini-file-search-tool cache-report --store "docs" --text

Best Practices

Use Caching: Let the tool manage duplicates automatically, don't manually track uploads
Flash Model for RAG: 12x more cost-efficient than Pro, perfectly adequate for semantic search
Enable Grounding: Always use --query-grounding-metadata to verify sources and validate answers
Simple Queries: Keep queries simple and direct, avoid over-specification (RAG relies on semantic similarity)
Code-RAG Documentation: Comprehensive docstrings maximize retrieval quality for code search
Parallel Processing: Use --num-workers to speed up large uploads
Async + Sync: Use --no-wait for bulk uploads, then sync-cache to check final status

Resources

GitHub: https://github.com/dnvriend/gemini-file-search-tool
Gemini File Search API: https://ai.google.dev/gemini-api/docs/file-search
Google AI Studio: https://aistudio.google.com/
API Keys: https://aistudio.google.com/apikey
Pricing: https://ai.google.dev/pricing

21 KiB Raw Blame History

When to use

Gemini File Search Tool Skill

Purpose

When to Use This Skill

CLI Tool: gemini-file-search-tool

Installation

Prerequisites

Quick Start

Progressive Disclosure

create-store - Create New Store

list-stores - List All Stores

get-store - Get Store Details

update-store - Update Store

delete-store - Delete Store

upload - Upload Documents

list-documents - List Documents in Store

query - Natural Language RAG Queries

sync-cache - Sync Pending Operations

flush-cache - Clear Cache File

cache-report - Generate Cache Status Report

What is Code-RAG?

Code-RAG Workflow

Code-RAG Best Practices

Meta Note

Intelligent Caching System

Parallel Processing

Cost Tracking

Authentication Methods

Common Issues

Getting Help

Known Issues

Exit Codes

Output Formats

Best Practices

Resources

21 KiB

Raw Blame History