gh-openshift-eng-ai-helpers-plugins-must-gather/commands/ovn-dbs.md at master

zhongwei/gh-openshift-eng-ai-helpers-plugins-must-gather

Fork 0

Files

Zhongwei Li cf9da06850 Initial commit

2025-11-30 08:46:06 +08:00

10 KiB

Raw Permalink Blame History

description, argument-hint

description

argument-hint

Analyze OVN databases from a must-gather using ovsdb-tool

must-gather-path

Name

must-gather:ovn-dbs

Synopsis

/must-gather:ovn-dbs [must-gather-path] [--node <node-name>] [--query <json>]

Description

The ovn-dbs command analyzes OVN Northbound and Southbound databases collected from clusters. It uses ovsdb-tool to query the binary database files (.db) collected per-node, providing detailed information about the logical network topology, pods, ACLs, and routers on each node.

The command automatically maps ovnkube pods to their corresponding nodes by reading pod specifications from the must-gather data.

Two modes of operation:

Standard Analysis (default): Runs pre-built analysis showing switches, ports, ACLs, and routers
Query Mode (--query): Run custom OVSDB JSON queries for specific data extraction

What it analyzes:

Per-zone logical network topology
Logical Switches and their ports
Pod Logical Switch Ports with namespace, pod name, and IP addresses
Access Control Lists (ACLs) with priorities, directions, and match rules
Logical Routers and their ports

Important: This command only works with must-gathers from clusters, where each node/zone has its own database files.

Prerequisites

The must-gather should contain:

network_logs/
└── ovnk_database_store.tar.gz

Required Tools:

ovsdb-tool must be installed (from openvswitch package)
- Check with: which ovsdb-tool
- Install: sudo dnf install openvswitch or sudo apt install openvswitch-common

Analysis Script:

The script is bundled with this plugin:

<plugin-root>/skills/must-gather-analyzer/scripts/analyze_ovn_dbs.py

Where <plugin-root> is the directory where this plugin is installed (typically ~/.cursor/commands/ai-helpers/plugins/must-gather/ or similar).

Claude will automatically locate it by searching for the script in the plugin installation directory, regardless of your current working directory.

Implementation

The command performs the following steps:

Locate Analysis Script:

SCRIPT_PATH=$(find ~ -name "analyze_ovn_dbs.py" -path "*/must-gather/skills/must-gather-analyzer/scripts/*" 2>/dev/null | head -1)

if [ -z "$SCRIPT_PATH" ]; then
    echo "ERROR: analyze_ovn_dbs.py script not found."
    echo "Please ensure the must-gather plugin from ai-helpers is properly installed."
    exit 1
fi

SCRIPTS_DIR=$(dirname "$SCRIPT_PATH")

Extract Database Tarball:
- Locate network_logs/ovnk_database_store.tar.gz
- Extract if not already extracted
- Find all *_nbdb and *_sbdb files

Query Each Zone's Database: For each zone (node), query the Northbound database using ovsdb-tool query:

ovsdb-tool query <zone>_nbdb '["OVN_Northbound", {"op":"select", "table":"<table>", "where":[], "columns":[...]}]'

Analyze and Display:
- Logical Switches: Names and port counts
- Logical Switch Ports: Filter for pods (external_ids.pod=true), show namespace, pod name, and IP
- ACLs: Priority, direction, match rules, and actions
- Logical Routers: Names and port counts
Present Zone Summary:
- Total counts per zone
- Detailed breakdowns
- Sorted and formatted output

Return Value

The command outputs structured analysis for each node:

Found 6 node(s)

================================================================================
Node: ip-10-0-26-145.us-east-2.compute.internal
Pod:  ovnkube-node-79cbh
================================================================================
  Logical Switches:      4
  Logical Switch Ports:  55
  ACLs:                  7
  Logical Routers:       2

  LOGICAL SWITCHES (4):
  NAME                                                         PORTS
  --------------------------------------------------------------------------------
  transit_switch                                               6
  ip-10-0-1-10.us-east-2.compute.internal                      7
  ext_ip-10-0-1-10.us-east-2.compute.internal                  2
  join                                                         2

  POD LOGICAL SWITCH PORTS (5):
  NAMESPACE                                POD                                           IP
  ------------------------------------------------------------------------------------------------------------------------
  openshift-dns                            dns-default-abc123                            10.128.0.5
  openshift-monitoring                     prometheus-k8s-0                              10.128.0.10
  openshift-etcd                           etcd-master-0                                 10.128.0.3
  ...

  ACCESS CONTROL LISTS (7):
  PRIORITY   DIRECTION       ACTION          MATCH
  ------------------------------------------------------------------------------------------------------------------------
  1012       from-lport      allow           inport == @a4743249366342378346 && (ip4.mcast ...
  1011       to-lport        drop            (ip4.mcast || mldv1 || mldv2 || ...
  1001       to-lport        allow-related   ip4.src==10.128.0.2
  ...

  LOGICAL ROUTERS (2):
  NAME                                                         PORTS
  --------------------------------------------------------------------------------
  ovn_cluster_router                                           3
  GR_ip-10-0-1-10.us-east-2.compute.internal                   2

Examples

Analyze all nodes in a must-gather:

/must-gather:ovn-dbs ./must-gather/registry-ci-openshift-org-origin-4-20-...-sha256-abc123/

Shows logical network topology for all nodes.

Analyze specific node:
```
/must-gather:ovn-dbs ./must-gather/.../ --node ip-10-0-26-145
```
Shows OVN database information only for the specified node (supports partial name matching).
Analyze master node:
```
/must-gather:ovn-dbs ./must-gather/.../ --node master-0
```
Filter to a specific master node using partial name matching.
Interactive usage without path:
```
/must-gather:ovn-dbs
```
The command will ask for the must-gather path.
Check if pod exists in OVN:
```
/must-gather:ovn-dbs ./must-gather/.../
```
Then search the output for the pod name to see which node it's on and its IP allocation.
Investigate ACL rules on a specific node:
```
/must-gather:ovn-dbs ./must-gather/.../ --node worker-1
```
Review the ACL section for a specific node to understand traffic filtering rules.

Run custom OVSDB query (Query Mode):

/must-gather:ovn-dbs ./must-gather/.../ --query '["OVN_Northbound", {"op":"select", "table":"ACL", "where":[["priority", ">", 1000]], "columns":["priority","match","action"]}]'

Query ACLs with priority > 1000 across all nodes. Claude can construct the JSON query for any OVSDB table.

Query specific node with custom query:

/must-gather:ovn-dbs ./must-gather/.../ --node master-0 --query '["OVN_Northbound", {"op":"select", "table":"Logical_Switch", "where":[], "columns":["name","ports"]}]'

List all logical switches with their ports on master-0.

Query specific table (Claude constructs JSON): Just ask Claude to query a specific OVSDB table and it will construct the appropriate JSON query. For example:
- "Show all Logical_Router_Static_Route entries"
- "Find ACLs with action 'drop'"
- "List Logical_Switch_Port entries where external_ids contains 'openshift-etcd'"

Error Handling

Missing ovsdb-tool:

Error: ovsdb-tool not found. Please install openvswitch package.

Solution: Install openvswitch: sudo dnf install openvswitch

Missing database tarball:

Error: Database tarball not found: network_logs/ovnk_database_store.tar.gz

Solution: Ensure this is a must-gather from an OVN cluster.

Node not found:

Error: No databases found for node matching 'master-5'

Available nodes:
  - ip-10-0-77-117.us-east-2.compute.internal
  - ip-10-0-26-145.us-east-2.compute.internal
  - ip-10-0-1-194.us-east-2.compute.internal

Solution: Use one of the listed node names or a partial match.

Notes

Binary Database Format: Uses ovsdb-tool to read OVSDB binary files directly
Per-Node Analysis: Each node in IC mode has its own database (one NB and one SB per zone)
Node Mapping: Automatically correlates ovnkube pods to nodes by reading pod specs from must-gather
Pod Discovery: Pods are identified by external_ids with pod=true
IP Extraction: Pod IPs are parsed from the addresses field (format: "MAC IP")
ACL Priorities: Higher priority ACLs are processed first (shown at top)
Node Filtering: Supports partial name matching for convenience (e.g., "--node master" matches all masters)
Query Mode: Accepts raw OVSDB JSON queries in the format ["OVN_Northbound", {"op":"select", "table":"...", ...}]
Claude Query Construction: Claude can automatically construct OVSDB JSON queries based on natural language requests
Performance: Querying large databases may take a few seconds per node

Use Cases

Verify Pod Network Configuration:
- Check if pods are registered in OVN
- Verify IP address assignments
- Confirm logical switch port creation
Troubleshoot Connectivity Issues:
- Review ACL rules blocking traffic
- Check if pods are in correct logical switches
- Verify router configurations
Understand Topology:
- See how zones are interconnected via transit_switch
- Review gateway router configurations
- Understand logical network structure
Audit Network Policies:
- See ACL rules generated from NetworkPolicies
- Identify overly permissive or restrictive rules
- Check rule priorities and match conditions

Arguments

$1 (must-gather-path): Optional. Path to the must-gather directory containing network_logs/. If not provided, user will be prompted.
--node, -n (node-name): Optional. Filter analysis to a specific node. Supports partial name matching (e.g., "master-0", "ip-10-0-26-145"). If no match is found, displays list of available nodes.
--query, -q (json-query): Optional. Run a raw OVSDB JSON query instead of standard analysis. Claude can construct the JSON query based on OVSDB transaction format. When provided, outputs raw JSON results instead of formatted analysis.

10 KiB Raw Permalink Blame History