zhongwei/gh-jamie-bitflight-claude-skills-development-skills

Files

Zhongwei Li 5007abf04b Initial commit

2025-11-29 18:49:58 +08:00

20 KiB

Raw Blame History

title, library_name, pypi_package, category, python_compatibility, last_updated, official_docs, official_repository, maintenance_status

title	library_name	pypi_package	category	python_compatibility	last_updated	official_docs	official_repository	maintenance_status
Fabric: High-Level SSH Command Execution and Deployment	fabric	fabric	ssh-automation	3.6+	2025-11-02	https://docs.fabfile.org	https://github.com/fabric/fabric	stable

Fabric: High-Level SSH Command Execution and Deployment

Core Purpose

Fabric is a high-level Python library designed to execute shell commands remotely over SSH, yielding useful Python objects in return. It solves the problem of programmatic remote server management and deployment automation by providing a Pythonic interface to SSH operations.

What Problem Does Fabric Solve?

Fabric eliminates the need to manually SSH into multiple servers and run commands repeatedly. It provides:

Programmatic SSH Execution: Execute commands on remote servers from Python code
Multi-Host Management: Run commands across multiple servers in parallel or serially
File Transfer: Upload and download files over SSH/SFTP
Deployment Automation: Orchestrate complex deployment workflows
Task Definition: Define reusable deployment tasks with the @task decorator
Connection Management: Handle SSH authentication, connection pooling, and error handling

When Should You Use Fabric?

Use Fabric when:

You need to execute commands on remote servers over SSH
You're automating deployment processes (copying files, restarting services, running migrations)
You need to manage multiple servers programmatically
You want to define reusable deployment tasks in Python
You're building continuous integration/deployment pipelines
You need more than just subprocess (which only works locally)

Use subprocess when:

You only need to run commands on your local machine
You don't need SSH connectivity to remote hosts
Your automation is purely local process execution

Use Ansible when:

You need declarative configuration management across many hosts
You require idempotency guarantees
You need a large ecosystem of pre-built modules
Your team prefers YAML over Python
You're managing infrastructure state, not just running scripts

Use Paramiko directly when:

You need low-level SSH protocol control
You're building custom SSH clients or servers
Fabric's higher-level abstractions are too restrictive

Architecture and Dependencies

Fabric is built on two core libraries:

Invoke (>=2.0): Subprocess command execution and command-line task features
Paramiko (>=2.4): SSH protocol implementation

Fabric extends their APIs to provide:

Remote execution via Connection.run()
File transfer via Connection.put() and Connection.get()
Sudo support via Connection.sudo()
Group operations via SerialGroup and ThreadingGroup

Python Version Compatibility

Python Version	Fabric 2.x	Fabric 3.x	Status
3.6	✓	✓	Minimum version
3.7	✓	✓	Supported
3.8	✓	✓	Supported
3.9	✓	✓	Supported
3.10	✓	✓	Supported
3.11	✓	✓	Supported
3.12	?	?	Likely supported
3.13	?	?	Likely supported
3.14	?	?	Unknown

Note: Fabric follows semantic versioning. Fabric 2.x and 3.x share similar APIs with minor breaking changes. Fabric 1.x (legacy) is incompatible with 2.x/3.x.

Fabric Version Differences

Fabric 1.x (legacy): Python 2.7 only, different API, no longer maintained
Fabric 2.x: Modern API, Python 3.6+, built on Invoke/Paramiko
Fabric 3.x: Current stable, incremental improvements over 2.x, Python 3.6+

Installation

# Standard installation
pip install fabric

# For migration from Fabric 1.x (side-by-side installation)
pip install fabric2

# Development installation
pip install -e git+https://github.com/fabric/fabric

# With pytest fixtures support
pip install fabric[pytest]

Core Usage Patterns

1. Basic Remote Command Execution

@https://docs.fabfile.org/en/latest/getting-started.html

from fabric import Connection

# Simple connection and command execution
result = Connection('web1.example.com').run('uname -s', hide=True)
print(f"Ran {result.command!r} on {result.connection.host}")
print(f"Exit code: {result.exited}")
print(f"Output: {result.stdout.strip()}")

Result object attributes:

result.stdout: Command output
result.stderr: Error output
result.exited: Exit code
result.ok: Boolean (True if exit code was 0)
result.command: The command that was run
result.connection: The Connection object used

2. Connection with Authentication

@https://docs.fabfile.org/en/latest/getting-started.html

from fabric import Connection

# User@host:port format
c = Connection('deploy@web1.example.com:2202')

# Or explicit parameters
c = Connection(
    host='web1.example.com',
    user='deploy',
    port=2202,
    connect_kwargs={
        "key_filename": "/path/to/private/key",
        # or
        "password": "mypassword"
    }
)

# Execute commands
c.run('whoami')
c.run('ls -la /var/www')

3. File Transfer Operations

@https://docs.fabfile.org/en/latest/getting-started.html

from fabric import Connection

c = Connection('web1')

# Upload file
result = c.put('myfiles.tgz', remote='/opt/mydata/')
print(f"Uploaded {result.local} to {result.remote}")

# Download file
c.get('/var/log/app.log', local='./logs/')

# Upload and extract
c.put('myfiles.tgz', '/opt/mydata')
c.run('tar -C /opt/mydata -xzvf /opt/mydata/myfiles.tgz')

4. Sudo Operations

@https://docs.fabfile.org/en/latest/getting-started.html

import getpass
from fabric import Connection, Config

# Configure sudo password
sudo_pass = getpass.getpass("What's your sudo password?")
config = Config(overrides={'sudo': {'password': sudo_pass}})

c = Connection('db1', config=config)

# Run with sudo using helper method
c.sudo('whoami', hide='stderr')  # Output: root
c.sudo('useradd mydbuser')
c.run('id -u mydbuser')  # Verify user created

# Alternative: Manual sudo with password responder
from invoke import Responder

sudopass = Responder(
    pattern=r'\[sudo\] password:',
    response=f'{sudo_pass}\n',
)
c.run('sudo whoami', pty=True, watchers=[sudopass])

5. Multi-Host Execution (Serial)

@https://docs.fabfile.org/en/latest/getting-started.html

from fabric import SerialGroup as Group

# Execute on multiple hosts serially
pool = Group('web1', 'web2', 'web3')

# Run command on all hosts
results = pool.run('uname -s')
for connection, result in results.items():
    print(f"{connection.host}: {result.stdout.strip()}")

# File operations on all hosts
pool.put('myfiles.tgz', '/opt/mydata')
pool.run('tar -C /opt/mydata -xzvf /opt/mydata/myfiles.tgz')

6. Multi-Host Execution (Parallel)

@https://docs.fabfile.org/en/latest/getting-started.html

from fabric import ThreadingGroup as Group

# Execute on multiple hosts in parallel
pool = Group('web1', 'web2', 'web3', 'web4', 'web5')

# Run command concurrently
results = pool.run('hostname')

# Process results
for connection, result in results.items():
    print(f"{connection.host}: {result.stdout.strip()}")

7. Defining Reusable Tasks

@https://docs.fabfile.org/en/latest/getting-started.html

from fabric import task

@task
def deploy(c):
    """Deploy application to remote server"""
    code_dir = "/srv/django/myproject"

    # Check if directory exists
    if not c.run(f"test -d {code_dir}", warn=True):
        # Clone repository
        c.run(f"git clone user@vcshost:/path/to/repo/.git {code_dir}")

    # Update code
    c.run(f"cd {code_dir} && git pull")

    # Restart application
    c.run(f"cd {code_dir} && touch app.wsgi")

@task
def update_servers(c):
    """Run system updates"""
    c.sudo('apt update')
    c.sudo('apt upgrade -y')
    c.sudo('systemctl restart nginx')

# Use with fab command:
# fab -H web1,web2,web3 deploy

8. Task Composition and Workflow

@https://docs.fabfile.org/en/latest/getting-started.html

from fabric import task
from invoke import Exit
from invocations.console import confirm

@task
def test(c):
    """Run local tests"""
    result = c.local("./manage.py test my_app", warn=True)
    if not result and not confirm("Tests failed. Continue anyway?"):
        raise Exit("Aborting at user request.")

@task
def commit(c):
    """Commit changes"""
    c.local("git add -p && git commit")

@task
def push(c):
    """Push to remote"""
    c.local("git push")

@task
def prepare_deploy(c):
    """Prepare for deployment"""
    test(c)
    commit(c)
    push(c)

@task(hosts=['web1.example.com', 'web2.example.com'])
def deploy(c):
    """Deploy to remote servers"""
    code_dir = "/srv/django/myproject"
    c.run(f"cd {code_dir} && git pull")
    c.run(f"cd {code_dir} && touch app.wsgi")

# Usage:
# fab prepare_deploy deploy

9. Connection with Gateway/Bastion Host

@https://docs.fabfile.org/en/latest/concepts/networking.html

from fabric import Connection

# Connect to internal host through gateway
gateway = Connection('bastion.example.com')
c = Connection('internal-db.local', gateway=gateway)

# Now all operations go through the gateway
c.run('hostname')
c.run('df -h')

10. Error Handling and Conditional Logic

@https://docs.fabfile.org/en/latest/getting-started.html

from fabric import SerialGroup as Group

def upload_and_unpack(c):
    """Upload file only if it doesn't exist"""
    # Check if file exists (don't fail on non-zero exit)
    if c.run('test -f /opt/mydata/myfile', warn=True).failed:
        c.put('myfiles.tgz', '/opt/mydata')
        c.run('tar -C /opt/mydata -xzvf /opt/mydata/myfiles.tgz')
    else:
        print(f"File already exists on {c.host}, skipping upload")

# Apply to group
for connection in Group('web1', 'web2', 'web3'):
    upload_and_unpack(connection)

Real-World Integration Patterns

Pattern 1: Django/Web Application Deployment

@https://www.oreilly.com/library/view/test-driven-development-with/9781449365141/ch09.html

from fabric import task, Connection

@task
def deploy_django(c):
    """Deploy Django application"""
    # Pull latest code
    c.run('cd /var/www/myapp && git pull origin main')

    # Install dependencies
    c.run('cd /var/www/myapp && pip install -r requirements.txt')

    # Run migrations
    c.run('cd /var/www/myapp && python manage.py migrate')

    # Collect static files
    c.run('cd /var/www/myapp && python manage.py collectstatic --noinput')

    # Restart services
    c.sudo('systemctl restart gunicorn')
    c.sudo('systemctl restart nginx')

Pattern 2: Database Backup and Restore

@Exa:fabric deployment examples

from fabric import task
from datetime import datetime

@task
def backup_database(c):
    """Create database backup"""
    timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
    backup_file = f"backup_{timestamp}.sql"

    # Create backup
    c.run(f"mysqldump -u dbuser -p database_name > /backups/{backup_file}")

    # Compress backup
    c.run(f"gzip /backups/{backup_file}")

    # Download backup
    c.get(f"/backups/{backup_file}.gz", local=f"./backups/{backup_file}.gz")

    print(f"Backup completed: {backup_file}.gz")

Pattern 3: Log Collection and Analysis

from fabric import SerialGroup as Group

def collect_logs(c):
    """Collect application logs from remote server"""
    hostname = c.run('hostname', hide=True).stdout.strip()
    c.get('/var/log/app/error.log', local=f'logs/{hostname}_error.log')
    c.get('/var/log/app/access.log', local=f'logs/{hostname}_access.log')

# Collect from all servers
pool = Group('web1', 'web2', 'web3', 'web4')
for conn in pool:
    collect_logs(conn)

Pattern 4: Service Health Check

from fabric import task, SerialGroup as Group

@task
def health_check(c):
    """Check service health across servers"""
    servers = Group('web1', 'web2', 'db1', 'cache1')

    for conn in servers:
        print(f"\nChecking {conn.host}...")

        # Check disk space
        result = conn.run("df -h / | tail -n1 | awk '{print $5}'", hide=True)
        disk_usage = result.stdout.strip()
        print(f"  Disk usage: {disk_usage}")

        # Check memory
        result = conn.run("free -m | grep Mem | awk '{print $3/$2 * 100.0}'", hide=True)
        mem_usage = float(result.stdout.strip())
        print(f"  Memory usage: {mem_usage:.1f}%")

        # Check service status
        result = conn.run("systemctl is-active nginx", warn=True, hide=True)
        service_status = result.stdout.strip()
        print(f"  Nginx status: {service_status}")

When NOT to Use Fabric

1. Simple Local Automation

# DON'T use Fabric for local operations
from fabric import Connection
c = Connection('localhost')
c.run('ls -la')

# DO use subprocess instead
import subprocess
subprocess.run(['ls', '-la'])

2. Large-Scale Infrastructure Management

If you need to manage hundreds of servers with complex configuration requirements, Ansible or SaltStack provide better:

Declarative configuration syntax
Idempotency guarantees
Large module ecosystem
Built-in inventory management
Role-based organization

3. Container Orchestration

For Docker/Kubernetes deployments, use native orchestration tools:

Docker Compose
Kubernetes manifests
Helm charts
ArgoCD

4. Configuration Drift Detection

Fabric executes commands but doesn't track state. For configuration management with drift detection, use:

Ansible
Chef
Puppet
Terraform (for infrastructure)

5. Windows Remote Management

For Windows automation, use:

PowerShell Remoting
WinRM libraries
Ansible (with WinRM)

Decision Matrix

Scenario	Fabric	Ansible	Subprocess	Paramiko
Deploy to 1-10 Linux servers	✓✓	✓	✗	✓
Deploy to 100+ servers	✓	✓✓	✗	✗
Run local commands	✗	✗	✓✓	✗
Configuration management	✗	✓✓	✗	✗
Ad-hoc SSH automation	✓✓	✓	✗	✓
Custom SSH protocol work	✗	✗	✗	✓✓
Python-first workflow	✓✓	✗	✓✓	✓✓
YAML-first workflow	✗	✓✓	✗	✗
File transfer over SSH	✓✓	✓	✗	✓
Parallel execution	✓✓	✓✓	✓	✗
Windows targets	✗	✓✓	✓	✗

Legend: ✓✓ = Excellent fit, ✓ = Suitable, ✗ = Not appropriate

Common Gotchas and Solutions

1. Separate Shell Sessions

@https://www.fabfile.org/faq.html

# WRONG: cd doesn't persist across run() calls
@task
def deploy(c):
    c.run("cd /path/to/application")
    c.run("./update.sh")  # This runs in home directory!

# CORRECT: Use shell && operator
@task
def deploy(c):
    c.run("cd /path/to/application && ./update.sh")

# ALTERNATIVE: Use absolute paths
@task
def deploy(c):
    c.run("/path/to/application/update.sh")

2. Sudo Password Prompts

# WRONG: Sudo hangs waiting for password
c.run('sudo systemctl restart nginx')

# CORRECT: Use pty=True and watchers
from invoke import Responder

sudopass = Responder(
    pattern=r'\[sudo\] password:',
    response='mypassword\n',
)
c.run('sudo systemctl restart nginx', pty=True, watchers=[sudopass])

# BETTER: Use Connection.sudo() helper
c.sudo('systemctl restart nginx')  # Uses configured password

3. Connection Reuse

# INEFFICIENT: Creates new connection each time
for i in range(10):
    Connection('web1').run(f'echo {i}')

# EFFICIENT: Reuse connection
c = Connection('web1')
for i in range(10):
    c.run(f'echo {i}')

Testing with Fabric

@https://docs.fabfile.org/en/latest/testing.html

from fabric.testing import MockRemote

def test_deployment():
    """Test deployment logic without real SSH"""
    with MockRemote(commands={
        'test -d /srv/app': (1, '', ''),  # Exit 1 = doesn't exist
        'git clone ...': (0, 'Cloning...', ''),
        'cd /srv/app && git pull': (0, 'Already up to date', ''),
    }) as remote:
        c = remote.connection
        deploy(c)

        # Verify commands were called
        assert 'git clone' in remote.calls

Migration from Fabric 1.x to 2.x/3.x

@https://docs.fabfile.org/en/latest/upgrading.html

Key changes:

No more env global dictionary
Tasks must accept Connection or Context as first argument
No more @hosts decorator (use @task(hosts=[...]))
run() is now c.run() on Connection object
Import from fabric not fabric.api

# Fabric 1.x (OLD)
from fabric.api import env, run, task

env.hosts = ['web1', 'web2']

@task
def deploy():
    run('git pull')

# Fabric 2.x/3.x (NEW)
from fabric import task

@task(hosts=['web1', 'web2'])
def deploy(c):
    c.run('git pull')

Performance Considerations

Parallel vs Serial Execution:
- Use ThreadingGroup for I/O-bound tasks (network operations)
- Consider SerialGroup for order-dependent operations
- Default thread pool size is 10 connections
Connection Pooling:
- Reuse Connection objects when possible
- Close connections explicitly with c.close() or use context managers
Output Buffering:
- Use hide=True to suppress output and improve performance
- Large output can slow down execution

Resources and Examples

Summary

Use Fabric when you need to:

Execute commands on remote Linux servers via SSH
Automate deployment of web applications
Manage small to medium server fleets (1-50 servers)
Transfer files between local and remote systems
Define reusable deployment tasks in Python
Integrate deployment into CI/CD pipelines

Don't use Fabric when:

You only need local command execution (use subprocess)
You're managing large infrastructure (>100 servers, use Ansible)
You need configuration drift detection (use Ansible/Chef/Puppet)
You're working with Windows servers primarily
You need declarative infrastructure as code (use Terraform/Ansible)

Fabric excels at programmatic SSH automation for deployment workflows where you want the full power of Python combined with remote execution capabilities. It's the sweet spot between low-level Paramiko and heavyweight configuration management tools.

20 KiB Raw Blame History

Fabric: High-Level SSH Command Execution and Deployment

Core Purpose

What Problem Does Fabric Solve?

When Should You Use Fabric?

Architecture and Dependencies

Python Version Compatibility

Fabric Version Differences

Installation

Core Usage Patterns

1. Basic Remote Command Execution

2. Connection with Authentication

3. File Transfer Operations

4. Sudo Operations

5. Multi-Host Execution (Serial)

6. Multi-Host Execution (Parallel)

7. Defining Reusable Tasks

8. Task Composition and Workflow

9. Connection with Gateway/Bastion Host

10. Error Handling and Conditional Logic

Real-World Integration Patterns

Pattern 1: Django/Web Application Deployment

Pattern 2: Database Backup and Restore

Pattern 3: Log Collection and Analysis

Pattern 4: Service Health Check

When NOT to Use Fabric

1. Simple Local Automation

2. Large-Scale Infrastructure Management

3. Container Orchestration

4. Configuration Drift Detection

5. Windows Remote Management

Decision Matrix

Common Gotchas and Solutions

1. Separate Shell Sessions

2. Sudo Password Prompts

3. Connection Reuse

Testing with Fabric

Migration from Fabric 1.x to 2.x/3.x

Performance Considerations

Resources and Examples

Official Documentation

GitHub Examples

Community Resources

Summary

20 KiB

Raw Blame History