zhongwei/gh-basher83-lunar-claude-plugins-infrastructure-proxmox-infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
master
gh-basher83-lunar-claude-pl…/skills/proxmox-infrastructure/workflows/cluster-formation.md
Zhongwei Li 0c6988a884 Initial commit
2025-11-29 18:00:27 +08:00

647 lines
17 KiB
Markdown
Raw Permalink Blame History

# Proxmox Cluster Formation Workflow
Complete guide to forming a Proxmox VE cluster using Ansible automation with idempotent patterns.
## Overview
This workflow automates the creation of a Proxmox VE cluster with:
- Hostname resolution configuration
- SSH key distribution for cluster operations
- Idempotent cluster initialization
- Corosync network configuration
- Quorum and health verification
## Prerequisites
Before forming a cluster:
1. **All nodes must have:**
- Proxmox VE 9.x installed
- Network connectivity on management network
- Dedicated corosync network configured (VLAN 9 for Matrix)
- Unique hostnames
- Synchronized time (NTP configured)
2. **Minimum requirements:**
- At least 3 nodes for quorum (production)
- 1 node for development/testing (non-recommended)
3. **Network requirements:**
- All nodes must be able to resolve each other's hostnames
- Corosync network must be isolated (no VM traffic)
- Low latency between nodes (<2ms recommended)
- MTU 1500 on management network
## Phase 1: Prepare Cluster Nodes
### Step 1: Verify Prerequisites
```yaml
# roles/proxmox_cluster/tasks/prerequisites.yml
---
- name: Check Proxmox VE is installed
ansible.builtin.stat:
path: /usr/bin/pvecm
register: pvecm_binary
failed_when: not pvecm_binary.stat.exists
- name: Get Proxmox VE version
ansible.builtin.command:
cmd: pveversion
register: pve_version
changed_when: false
- name: Verify minimum Proxmox VE version
ansible.builtin.assert:
that:
- "'pve-manager/9' in pve_version.stdout or 'pve-manager/8' in pve_version.stdout"
fail_msg: "Proxmox VE 8.x or 9.x required"
- name: Verify minimum node count for production
ansible.builtin.assert:
that:
- groups[cluster_group] | length >= 3
fail_msg: "Production cluster requires at least 3 nodes for quorum"
when: cluster_environment == 'production'
- name: Check no existing cluster membership
ansible.builtin.command:
cmd: pvecm status
register: existing_cluster
failed_when: false
changed_when: false
- name: Display cluster warning if already member
ansible.builtin.debug:
msg: |
WARNING: Node {{ inventory_hostname }} is already a cluster member.
Current cluster: {{ existing_cluster.stdout }}
This playbook will attempt to join the target cluster.
when:
- existing_cluster.rc == 0
- cluster_name not in existing_cluster.stdout
```
### Step 2: Configure Hostname Resolution
```yaml
# roles/proxmox_cluster/tasks/hosts_config.yml
---
- name: Ensure cluster nodes in /etc/hosts (management IP)
ansible.builtin.lineinfile:
path: /etc/hosts
regexp: "^{{ item.management_ip }}\\s+"
line: "{{ item.management_ip }} {{ item.fqdn }} {{ item.short_name }}"
state: present
loop: "{{ cluster_nodes }}"
loop_control:
label: "{{ item.short_name }}"
- name: Ensure corosync IPs in /etc/hosts
ansible.builtin.lineinfile:
path: /etc/hosts
regexp: "^{{ item.corosync_ip }}\\s+"
line: "{{ item.corosync_ip }} {{ item.short_name }}-corosync"
state: present
loop: "{{ cluster_nodes }}"
loop_control:
label: "{{ item.short_name }}"
- name: Verify hostname resolution (forward)
ansible.builtin.command:
cmd: "getent hosts {{ item.fqdn }}"
register: host_lookup
failed_when: host_lookup.rc != 0
changed_when: false
loop: "{{ cluster_nodes }}"
loop_control:
label: "{{ item.fqdn }}"
- name: Verify hostname resolution (reverse)
ansible.builtin.command:
cmd: "getent hosts {{ item.management_ip }}"
register: reverse_lookup
failed_when:
- reverse_lookup.rc != 0
changed_when: false
loop: "{{ cluster_nodes }}"
loop_control:
label: "{{ item.management_ip }}"
- name: Test corosync network connectivity
ansible.builtin.command:
cmd: "ping -c 3 -W 2 {{ item.corosync_ip }}"
register: corosync_ping
changed_when: false
when: item.short_name != inventory_hostname_short
loop: "{{ cluster_nodes }}"
loop_control:
label: "{{ item.short_name }}"
```
### Step 3: Distribute SSH Keys
```yaml
# roles/proxmox_cluster/tasks/ssh_keys.yml
---
- name: Generate SSH key for root (if not exists)
ansible.builtin.user:
name: root
generate_ssh_key: true
ssh_key_type: ed25519
ssh_key_comment: "root@{{ inventory_hostname }}"
register: root_ssh_key
- name: Fetch public keys from all nodes
ansible.builtin.slurp:
src: /root/.ssh/id_ed25519.pub
register: node_public_keys
- name: Distribute SSH keys to all nodes
ansible.posix.authorized_key:
user: root
state: present
key: "{{ hostvars[item].node_public_keys.content | b64decode }}"
comment: "cluster-{{ item }}"
loop: "{{ groups[cluster_group] }}"
when: item != inventory_hostname
- name: Populate known_hosts with node SSH keys
ansible.builtin.shell:
cmd: "ssh-keyscan -H {{ item }} >> /root/.ssh/known_hosts"
when: item != inventory_hostname
loop: "{{ groups[cluster_group] }}"
loop_control:
label: "{{ item }}"
changed_when: true
- name: Test SSH connectivity to all nodes
ansible.builtin.command:
cmd: "ssh -o ConnectTimeout=5 {{ item }} hostname"
register: ssh_test
changed_when: false
when: item != inventory_hostname
loop: "{{ groups[cluster_group] }}"
loop_control:
label: "{{ item }}"
```
## Phase 2: Initialize Cluster
### Step 4: Create Cluster (First Node Only)
```yaml
# roles/proxmox_cluster/tasks/cluster_init.yml
---
- name: Check existing cluster status
ansible.builtin.command:
cmd: pvecm status
register: cluster_status
failed_when: false
changed_when: false
- name: Get cluster nodes list
ansible.builtin.command:
cmd: pvecm nodes
register: cluster_nodes_check
failed_when: false
changed_when: false
- name: Set cluster facts
ansible.builtin.set_fact:
in_target_cluster: "{{ cluster_status.rc == 0 and cluster_name in cluster_status.stdout }}"
- name: Create new cluster on first node
ansible.builtin.command:
cmd: "pvecm create {{ cluster_name }} --link0 {{ corosync_link0_address }}"
when: not in_target_cluster
register: cluster_create
changed_when: cluster_create.rc == 0
- name: Wait for cluster to initialize
ansible.builtin.pause:
seconds: 10
when: cluster_create.changed
- name: Verify cluster creation
ansible.builtin.command:
cmd: pvecm status
register: cluster_verify
changed_when: false
failed_when: cluster_name not in cluster_verify.stdout
- name: Display cluster status
ansible.builtin.debug:
var: cluster_verify.stdout_lines
when: cluster_create.changed or ansible_verbosity > 0
```
### Step 5: Join Nodes to Cluster
```yaml
# roles/proxmox_cluster/tasks/cluster_join.yml
---
- name: Check if already in cluster
ansible.builtin.command:
cmd: pvecm status
register: cluster_status
failed_when: false
changed_when: false
- name: Set membership facts
ansible.builtin.set_fact:
is_cluster_member: "{{ cluster_status.rc == 0 }}"
in_target_cluster: "{{ cluster_status.rc == 0 and cluster_name in cluster_status.stdout }}"
- name: Get first node hostname
ansible.builtin.set_fact:
first_node_hostname: "{{ hostvars[groups[cluster_group][0]].inventory_hostname }}"
- name: Join cluster
ansible.builtin.command:
cmd: >
pvecm add {{ first_node_hostname }}
--link0 {{ corosync_link0_address }}
when:
- not is_cluster_member or not in_target_cluster
register: cluster_join
changed_when: cluster_join.rc == 0
failed_when:
- cluster_join.rc != 0
- "'already in a cluster' not in cluster_join.stderr"
- name: Wait for node to join cluster
ansible.builtin.pause:
seconds: 10
when: cluster_join.changed
- name: Verify cluster membership
ansible.builtin.command:
cmd: pvecm status
register: join_verify
changed_when: false
failed_when:
- "'Quorate: Yes' not in join_verify.stdout"
```
## Phase 3: Configure Corosync
### Step 6: Corosync Network Configuration
```yaml
# roles/proxmox_cluster/tasks/corosync.yml
---
- name: Get current corosync configuration
ansible.builtin.slurp:
src: /etc/pve/corosync.conf
register: corosync_conf_current
- name: Parse current corosync config
ansible.builtin.set_fact:
current_corosync: "{{ corosync_conf_current.content | b64decode }}"
- name: Check if corosync config needs update
ansible.builtin.set_fact:
corosync_needs_update: "{{ corosync_network not in current_corosync }}"
- name: Backup corosync.conf
ansible.builtin.copy:
src: /etc/pve/corosync.conf
dest: "/etc/pve/corosync.conf.{{ ansible_date_time.epoch }}.bak"
remote_src: true
mode: '0640'
when: corosync_needs_update
delegate_to: "{{ groups[cluster_group][0] }}"
run_once: true
- name: Update corosync configuration
ansible.builtin.template:
src: corosync.conf.j2
dest: /etc/pve/corosync.conf.new
validate: corosync-cfgtool -c %s
mode: '0640'
when: corosync_needs_update
delegate_to: "{{ groups[cluster_group][0] }}"
run_once: true
- name: Apply new corosync configuration
ansible.builtin.copy:
src: /etc/pve/corosync.conf.new
dest: /etc/pve/corosync.conf
remote_src: true
mode: '0640'
when: corosync_needs_update
notify:
- reload corosync
delegate_to: "{{ groups[cluster_group][0] }}"
run_once: true
```
**Corosync Template Example:**
```jinja2
# templates/corosync.conf.j2
totem {
version: 2
cluster_name: {{ cluster_name }}
transport: knet
crypto_cipher: aes256
crypto_hash: sha256
interface {
linknumber: 0
knet_link_priority: 255
}
}
nodelist {
{% for node in cluster_nodes %}
node {
name: {{ node.short_name }}
nodeid: {{ node.node_id }}
quorum_votes: 1
ring0_addr: {{ node.corosync_ip }}
}
{% endfor %}
}
quorum {
provider: corosync_votequorum
{% if cluster_nodes | length == 2 %}
two_node: 1
{% endif %}
}
logging {
to_logfile: yes
logfile: /var/log/corosync/corosync.log
to_syslog: yes
timestamp: on
}
```
## Phase 4: Verify Cluster Health
### Step 7: Health Checks
```yaml
# roles/proxmox_cluster/tasks/verify.yml
---
- name: Wait for cluster to stabilize
ansible.builtin.pause:
seconds: 15
- name: Check cluster quorum
ansible.builtin.command:
cmd: pvecm status
register: cluster_health
changed_when: false
failed_when: "'Quorate: Yes' not in cluster_health.stdout"
- name: Get cluster node count
ansible.builtin.command:
cmd: pvecm nodes
register: cluster_nodes_final
changed_when: false
- name: Verify expected node count
ansible.builtin.assert:
that:
- cluster_nodes_final.stdout_lines | length >= groups[cluster_group] | length
fail_msg: "Expected {{ groups[cluster_group] | length }} nodes but found {{ cluster_nodes_final.stdout_lines | length }}"
- name: Check corosync ring status
ansible.builtin.command:
cmd: corosync-cfgtool -s
register: corosync_status
changed_when: false
- name: Verify all nodes in corosync
ansible.builtin.assert:
that:
- "'online' in corosync_status.stdout"
fail_msg: "Corosync ring issues detected"
- name: Get cluster configuration version
ansible.builtin.command:
cmd: corosync-cmapctl -b totem.config_version
register: config_version
changed_when: false
- name: Display cluster health summary
ansible.builtin.debug:
msg: |
Cluster: {{ cluster_name }}
Quorum: {{ 'Yes' if 'Quorate: Yes' in cluster_health.stdout else 'No' }}
Nodes: {{ cluster_nodes_final.stdout_lines | length }}
Config Version: {{ config_version.stdout }}
```
## Matrix Cluster Example Configuration
```yaml
# group_vars/matrix_cluster.yml
---
cluster_name: "Matrix"
cluster_group: "matrix_cluster"
cluster_environment: "production"
# Corosync configuration
corosync_network: "192.168.8.0/24" # VLAN 9
# Node configuration
cluster_nodes:
- short_name: foxtrot
fqdn: foxtrot.matrix.spaceships.work
management_ip: 192.168.3.5
corosync_ip: 192.168.8.5
node_id: 1
- short_name: golf
fqdn: golf.matrix.spaceships.work
management_ip: 192.168.3.6
corosync_ip: 192.168.8.6
node_id: 2
- short_name: hotel
fqdn: hotel.matrix.spaceships.work
management_ip: 192.168.3.7
corosync_ip: 192.168.8.7
node_id: 3
# Set per-node corosync address
corosync_link0_address: "{{ cluster_nodes | selectattr('short_name', 'equalto', inventory_hostname_short) | map(attribute='corosync_ip') | first }}"
```
## Complete Playbook Example
```yaml
# playbooks/cluster-init.yml
---
- name: Initialize Proxmox Cluster
hosts: "{{ cluster_group | default('matrix_cluster') }}"
become: true
serial: 1 # One node at a time for safety
pre_tasks:
- name: Validate cluster group is defined
ansible.builtin.assert:
that:
- cluster_group is defined
- cluster_name is defined
- cluster_nodes is defined
fail_msg: "Required variables not defined in group_vars"
- name: Display cluster configuration
ansible.builtin.debug:
msg: |
Forming cluster: {{ cluster_name }}
Nodes: {{ cluster_nodes | map(attribute='short_name') | join(', ') }}
Corosync network: {{ corosync_network }}
run_once: true
tasks:
- name: Verify prerequisites
ansible.builtin.include_tasks: "{{ role_path }}/tasks/prerequisites.yml"
- name: Configure /etc/hosts
ansible.builtin.include_tasks: "{{ role_path }}/tasks/hosts_config.yml"
- name: Distribute SSH keys
ansible.builtin.include_tasks: "{{ role_path }}/tasks/ssh_keys.yml"
# First node creates cluster
- name: Initialize cluster on first node
ansible.builtin.include_tasks: "{{ role_path }}/tasks/cluster_init.yml"
when: inventory_hostname == groups[cluster_group][0]
# Wait for first node
- name: Wait for first node to complete
ansible.builtin.pause:
seconds: 20
when: inventory_hostname != groups[cluster_group][0]
# Other nodes join
- name: Join cluster on other nodes
ansible.builtin.include_tasks: "{{ role_path }}/tasks/cluster_join.yml"
when: inventory_hostname != groups[cluster_group][0]
- name: Configure corosync
ansible.builtin.include_tasks: "{{ role_path }}/tasks/corosync.yml"
- name: Verify cluster health
ansible.builtin.include_tasks: "{{ role_path }}/tasks/verify.yml"
post_tasks:
- name: Display final cluster status
ansible.builtin.command:
cmd: pvecm status
register: final_status
changed_when: false
delegate_to: "{{ groups[cluster_group][0] }}"
run_once: true
- name: Show cluster status
ansible.builtin.debug:
var: final_status.stdout_lines
run_once: true
handlers:
- name: reload corosync
ansible.builtin.systemd:
name: corosync
state: reloaded
throttle: 1
```
## Usage
### Initialize Matrix Cluster
```bash
# Check syntax
ansible-playbook playbooks/cluster-init.yml --syntax-check
# Dry run (limited functionality)
ansible-playbook playbooks/cluster-init.yml --check --diff
# Initialize cluster
ansible-playbook playbooks/cluster-init.yml --limit matrix_cluster
# Verify cluster status
ansible -i inventory/proxmox.yml foxtrot -m shell -a "pvecm status"
ansible -i inventory/proxmox.yml foxtrot -m shell -a "pvecm nodes"
```
### Add mise Task
```toml
# .mise.toml
[tasks."cluster:init"]
description = "Initialize Proxmox cluster"
run = """
cd ansible
uv run ansible-playbook playbooks/cluster-init.yml
"""
[tasks."cluster:status"]
description = "Show cluster status"
run = """
ansible -i ansible/inventory/proxmox.yml foxtrot -m shell -a "pvecm status"
"""
```
## Troubleshooting
### Node Won't Join Cluster
**Symptoms:**
- `pvecm add` fails with timeout or connection error
**Solutions:**
1. Verify SSH connectivity: `ssh root@first-node hostname`
2. Check /etc/hosts: `getent hosts first-node`
3. Verify corosync network: `ping -c 3 192.168.8.5`
4. Check firewall: `iptables -L | grep 5404`
### Cluster Shows No Quorum
**Symptoms:**
- `pvecm status` shows `Quorate: No`
**Solutions:**
1. Check node count: Must have majority (2 of 3, 3 of 5, etc.)
2. Verify corosync: `systemctl status corosync`
3. Check corosync ring: `corosync-cfgtool -s`
4. Review logs: `journalctl -u corosync -n 50`
### Configuration Sync Issues
**Symptoms:**
- Changes on one node don't appear on others
**Solutions:**
1. Verify pmxcfs: `systemctl status pve-cluster`
2. Check filesystem: `pvecm status | grep -i cluster`
3. Restart cluster filesystem: `systemctl restart pve-cluster`
## Related Workflows
- [CEPH Deployment](ceph-deployment.md) - Deploy CEPH after cluster formation
- [Network Configuration](../reference/networking.md) - Configure cluster networking
- [Cluster Maintenance](cluster-maintenance.md) - Add/remove nodes, upgrades
## References
- ProxSpray analysis: `docs/proxspray-analysis.md` (lines 1318-1428)
- Proxmox VE Cluster Manager documentation
- Corosync configuration guide
- [Ansible cluster automation pattern](../../ansible-best-practices/patterns/cluster-automation.md)
Reference in New Issue View Git Blame Copy Permalink