gh-basher83-lunar-claude-plugins-infrastructure-proxmox-infrastructure/skills/proxmox-infrastructure/workflows/cluster-formation.md

Proxmox Cluster Formation Workflow

Complete guide to forming a Proxmox VE cluster using Ansible automation with idempotent patterns.

Overview

This workflow automates the creation of a Proxmox VE cluster with:

• Hostname resolution configuration
• SSH key distribution for cluster operations
• Idempotent cluster initialization
• Corosync network configuration
• Quorum and health verification

Prerequisites

Before forming a cluster:

1. All nodes must have:

   • Proxmox VE 9.x installed
   • Network connectivity on management network
   • Dedicated corosync network configured (VLAN 9 for Matrix)
   • Unique hostnames
   • Synchronized time (NTP configured)

2. Minimum requirements:

   • At least 3 nodes for quorum (production)
   • 1 node for development/testing (non-recommended)

3. Network requirements:

   • All nodes must be able to resolve each other's hostnames
   • Corosync network must be isolated (no VM traffic)
   • Low latency between nodes (<2ms recommended)
   • MTU 1500 on management network

Phase 1: Prepare Cluster Nodes

Step 1: Verify Prerequisites

# roles/proxmox_cluster/tasks/prerequisites.yml
---
- name: Check Proxmox VE is installed
  ansible.builtin.stat:
    path: /usr/bin/pvecm
  register: pvecm_binary
  failed_when: not pvecm_binary.stat.exists

- name: Get Proxmox VE version
  ansible.builtin.command:
    cmd: pveversion
  register: pve_version
  changed_when: false

- name: Verify minimum Proxmox VE version
  ansible.builtin.assert:
    that:
      - "'pve-manager/9' in pve_version.stdout or 'pve-manager/8' in pve_version.stdout"
    fail_msg: "Proxmox VE 8.x or 9.x required"

- name: Verify minimum node count for production
  ansible.builtin.assert:
    that:
      - groups[cluster_group] | length >= 3
    fail_msg: "Production cluster requires at least 3 nodes for quorum"
  when: cluster_environment == 'production'

- name: Check no existing cluster membership
  ansible.builtin.command:
    cmd: pvecm status
  register: existing_cluster
  failed_when: false
  changed_when: false

- name: Display cluster warning if already member
  ansible.builtin.debug:
    msg: |
      WARNING: Node {{ inventory_hostname }} is already a cluster member.
      Current cluster: {{ existing_cluster.stdout }}
      This playbook will attempt to join the target cluster.
  when:
    - existing_cluster.rc == 0
    - cluster_name not in existing_cluster.stdout

Step 2: Configure Hostname Resolution

# roles/proxmox_cluster/tasks/hosts_config.yml
---
- name: Ensure cluster nodes in /etc/hosts (management IP)
  ansible.builtin.lineinfile:
    path: /etc/hosts
    regexp: "^{{ item.management_ip }}\\s+"
    line: "{{ item.management_ip }} {{ item.fqdn }} {{ item.short_name }}"
    state: present
  loop: "{{ cluster_nodes }}"
  loop_control:
    label: "{{ item.short_name }}"

- name: Ensure corosync IPs in /etc/hosts
  ansible.builtin.lineinfile:
    path: /etc/hosts
    regexp: "^{{ item.corosync_ip }}\\s+"
    line: "{{ item.corosync_ip }} {{ item.short_name }}-corosync"
    state: present
  loop: "{{ cluster_nodes }}"
  loop_control:
    label: "{{ item.short_name }}"

- name: Verify hostname resolution (forward)
  ansible.builtin.command:
    cmd: "getent hosts {{ item.fqdn }}"
  register: host_lookup
  failed_when: host_lookup.rc != 0
  changed_when: false
  loop: "{{ cluster_nodes }}"
  loop_control:
    label: "{{ item.fqdn }}"

- name: Verify hostname resolution (reverse)
  ansible.builtin.command:
    cmd: "getent hosts {{ item.management_ip }}"
  register: reverse_lookup
  failed_when:
    - reverse_lookup.rc != 0
  changed_when: false
  loop: "{{ cluster_nodes }}"
  loop_control:
    label: "{{ item.management_ip }}"

- name: Test corosync network connectivity
  ansible.builtin.command:
    cmd: "ping -c 3 -W 2 {{ item.corosync_ip }}"
  register: corosync_ping
  changed_when: false
  when: item.short_name != inventory_hostname_short
  loop: "{{ cluster_nodes }}"
  loop_control:
    label: "{{ item.short_name }}"

Step 3: Distribute SSH Keys

# roles/proxmox_cluster/tasks/ssh_keys.yml
---
- name: Generate SSH key for root (if not exists)
  ansible.builtin.user:
    name: root
    generate_ssh_key: true
    ssh_key_type: ed25519
    ssh_key_comment: "root@{{ inventory_hostname }}"
  register: root_ssh_key

- name: Fetch public keys from all nodes
  ansible.builtin.slurp:
    src: /root/.ssh/id_ed25519.pub
  register: node_public_keys

- name: Distribute SSH keys to all nodes
  ansible.posix.authorized_key:
    user: root
    state: present
    key: "{{ hostvars[item].node_public_keys.content | b64decode }}"
    comment: "cluster-{{ item }}"
  loop: "{{ groups[cluster_group] }}"
  when: item != inventory_hostname

- name: Populate known_hosts with node SSH keys
  ansible.builtin.shell:
    cmd: "ssh-keyscan -H {{ item }} >> /root/.ssh/known_hosts"
  when: item != inventory_hostname
  loop: "{{ groups[cluster_group] }}"
  loop_control:
    label: "{{ item }}"
  changed_when: true

- name: Test SSH connectivity to all nodes
  ansible.builtin.command:
    cmd: "ssh -o ConnectTimeout=5 {{ item }} hostname"
  register: ssh_test
  changed_when: false
  when: item != inventory_hostname
  loop: "{{ groups[cluster_group] }}"
  loop_control:
    label: "{{ item }}"

Phase 2: Initialize Cluster

Step 4: Create Cluster (First Node Only)

# roles/proxmox_cluster/tasks/cluster_init.yml
---
- name: Check existing cluster status
  ansible.builtin.command:
    cmd: pvecm status
  register: cluster_status
  failed_when: false
  changed_when: false

- name: Get cluster nodes list
  ansible.builtin.command:
    cmd: pvecm nodes
  register: cluster_nodes_check
  failed_when: false
  changed_when: false

- name: Set cluster facts
  ansible.builtin.set_fact:
    in_target_cluster: "{{ cluster_status.rc == 0 and cluster_name in cluster_status.stdout }}"

- name: Create new cluster on first node
  ansible.builtin.command:
    cmd: "pvecm create {{ cluster_name }} --link0 {{ corosync_link0_address }}"
  when: not in_target_cluster
  register: cluster_create
  changed_when: cluster_create.rc == 0

- name: Wait for cluster to initialize
  ansible.builtin.pause:
    seconds: 10
  when: cluster_create.changed

- name: Verify cluster creation
  ansible.builtin.command:
    cmd: pvecm status
  register: cluster_verify
  changed_when: false
  failed_when: cluster_name not in cluster_verify.stdout

- name: Display cluster status
  ansible.builtin.debug:
    var: cluster_verify.stdout_lines
  when: cluster_create.changed or ansible_verbosity > 0

Step 5: Join Nodes to Cluster

# roles/proxmox_cluster/tasks/cluster_join.yml
---
- name: Check if already in cluster
  ansible.builtin.command:
    cmd: pvecm status
  register: cluster_status
  failed_when: false
  changed_when: false

- name: Set membership facts
  ansible.builtin.set_fact:
    is_cluster_member: "{{ cluster_status.rc == 0 }}"
    in_target_cluster: "{{ cluster_status.rc == 0 and cluster_name in cluster_status.stdout }}"

- name: Get first node hostname
  ansible.builtin.set_fact:
    first_node_hostname: "{{ hostvars[groups[cluster_group][0]].inventory_hostname }}"

- name: Join cluster
  ansible.builtin.command:
    cmd: >
      pvecm add {{ first_node_hostname }}
      --link0 {{ corosync_link0_address }}
  when:
    - not is_cluster_member or not in_target_cluster
  register: cluster_join
  changed_when: cluster_join.rc == 0
  failed_when:
    - cluster_join.rc != 0
    - "'already in a cluster' not in cluster_join.stderr"

- name: Wait for node to join cluster
  ansible.builtin.pause:
    seconds: 10
  when: cluster_join.changed

- name: Verify cluster membership
  ansible.builtin.command:
    cmd: pvecm status
  register: join_verify
  changed_when: false
  failed_when:
    - "'Quorate: Yes' not in join_verify.stdout"

Phase 3: Configure Corosync

Step 6: Corosync Network Configuration

# roles/proxmox_cluster/tasks/corosync.yml
---
- name: Get current corosync configuration
  ansible.builtin.slurp:
    src: /etc/pve/corosync.conf
  register: corosync_conf_current

- name: Parse current corosync config
  ansible.builtin.set_fact:
    current_corosync: "{{ corosync_conf_current.content | b64decode }}"

- name: Check if corosync config needs update
  ansible.builtin.set_fact:
    corosync_needs_update: "{{ corosync_network not in current_corosync }}"

- name: Backup corosync.conf
  ansible.builtin.copy:
    src: /etc/pve/corosync.conf
    dest: "/etc/pve/corosync.conf.{{ ansible_date_time.epoch }}.bak"
    remote_src: true
    mode: '0640'
  when: corosync_needs_update
  delegate_to: "{{ groups[cluster_group][0] }}"
  run_once: true

- name: Update corosync configuration
  ansible.builtin.template:
    src: corosync.conf.j2
    dest: /etc/pve/corosync.conf.new
    validate: corosync-cfgtool -c %s
    mode: '0640'
  when: corosync_needs_update
  delegate_to: "{{ groups[cluster_group][0] }}"
  run_once: true

- name: Apply new corosync configuration
  ansible.builtin.copy:
    src: /etc/pve/corosync.conf.new
    dest: /etc/pve/corosync.conf
    remote_src: true
    mode: '0640'
  when: corosync_needs_update
  notify:
    - reload corosync
  delegate_to: "{{ groups[cluster_group][0] }}"
  run_once: true

Corosync Template Example:

# templates/corosync.conf.j2
totem {
  version: 2
  cluster_name: {{ cluster_name }}
  transport: knet
  crypto_cipher: aes256
  crypto_hash: sha256

  interface {
    linknumber: 0
    knet_link_priority: 255
  }
}

nodelist {
{% for node in cluster_nodes %}
  node {
    name: {{ node.short_name }}
    nodeid: {{ node.node_id }}
    quorum_votes: 1
    ring0_addr: {{ node.corosync_ip }}
  }
{% endfor %}
}

quorum {
  provider: corosync_votequorum
{% if cluster_nodes | length == 2 %}
  two_node: 1
{% endif %}
}

logging {
  to_logfile: yes
  logfile: /var/log/corosync/corosync.log
  to_syslog: yes
  timestamp: on
}

Phase 4: Verify Cluster Health

Step 7: Health Checks

# roles/proxmox_cluster/tasks/verify.yml
---
- name: Wait for cluster to stabilize
  ansible.builtin.pause:
    seconds: 15

- name: Check cluster quorum
  ansible.builtin.command:
    cmd: pvecm status
  register: cluster_health
  changed_when: false
  failed_when: "'Quorate: Yes' not in cluster_health.stdout"

- name: Get cluster node count
  ansible.builtin.command:
    cmd: pvecm nodes
  register: cluster_nodes_final
  changed_when: false

- name: Verify expected node count
  ansible.builtin.assert:
    that:
      - cluster_nodes_final.stdout_lines | length >= groups[cluster_group] | length
    fail_msg: "Expected {{ groups[cluster_group] | length }} nodes but found {{ cluster_nodes_final.stdout_lines | length }}"

- name: Check corosync ring status
  ansible.builtin.command:
    cmd: corosync-cfgtool -s
  register: corosync_status
  changed_when: false

- name: Verify all nodes in corosync
  ansible.builtin.assert:
    that:
      - "'online' in corosync_status.stdout"
    fail_msg: "Corosync ring issues detected"

- name: Get cluster configuration version
  ansible.builtin.command:
    cmd: corosync-cmapctl -b totem.config_version
  register: config_version
  changed_when: false

- name: Display cluster health summary
  ansible.builtin.debug:
    msg: |
      Cluster: {{ cluster_name }}
      Quorum: {{ 'Yes' if 'Quorate: Yes' in cluster_health.stdout else 'No' }}
      Nodes: {{ cluster_nodes_final.stdout_lines | length }}
      Config Version: {{ config_version.stdout }}

Matrix Cluster Example Configuration

# group_vars/matrix_cluster.yml
---
cluster_name: "Matrix"
cluster_group: "matrix_cluster"
cluster_environment: "production"

# Corosync configuration
corosync_network: "192.168.8.0/24"  # VLAN 9

# Node configuration
cluster_nodes:
  - short_name: foxtrot
    fqdn: foxtrot.matrix.spaceships.work
    management_ip: 192.168.3.5
    corosync_ip: 192.168.8.5
    node_id: 1

  - short_name: golf
    fqdn: golf.matrix.spaceships.work
    management_ip: 192.168.3.6
    corosync_ip: 192.168.8.6
    node_id: 2

  - short_name: hotel
    fqdn: hotel.matrix.spaceships.work
    management_ip: 192.168.3.7
    corosync_ip: 192.168.8.7
    node_id: 3

# Set per-node corosync address
corosync_link0_address: "{{ cluster_nodes | selectattr('short_name', 'equalto', inventory_hostname_short) | map(attribute='corosync_ip') | first }}"

Complete Playbook Example

# playbooks/cluster-init.yml
---
- name: Initialize Proxmox Cluster
  hosts: "{{ cluster_group | default('matrix_cluster') }}"
  become: true
  serial: 1  # One node at a time for safety

  pre_tasks:
    - name: Validate cluster group is defined
      ansible.builtin.assert:
        that:
          - cluster_group is defined
          - cluster_name is defined
          - cluster_nodes is defined
        fail_msg: "Required variables not defined in group_vars"

    - name: Display cluster configuration
      ansible.builtin.debug:
        msg: |
          Forming cluster: {{ cluster_name }}
          Nodes: {{ cluster_nodes | map(attribute='short_name') | join(', ') }}
          Corosync network: {{ corosync_network }}
      run_once: true

  tasks:
    - name: Verify prerequisites
      ansible.builtin.include_tasks: "{{ role_path }}/tasks/prerequisites.yml"

    - name: Configure /etc/hosts
      ansible.builtin.include_tasks: "{{ role_path }}/tasks/hosts_config.yml"

    - name: Distribute SSH keys
      ansible.builtin.include_tasks: "{{ role_path }}/tasks/ssh_keys.yml"

    # First node creates cluster
    - name: Initialize cluster on first node
      ansible.builtin.include_tasks: "{{ role_path }}/tasks/cluster_init.yml"
      when: inventory_hostname == groups[cluster_group][0]

    # Wait for first node
    - name: Wait for first node to complete
      ansible.builtin.pause:
        seconds: 20
      when: inventory_hostname != groups[cluster_group][0]

    # Other nodes join
    - name: Join cluster on other nodes
      ansible.builtin.include_tasks: "{{ role_path }}/tasks/cluster_join.yml"
      when: inventory_hostname != groups[cluster_group][0]

    - name: Configure corosync
      ansible.builtin.include_tasks: "{{ role_path }}/tasks/corosync.yml"

    - name: Verify cluster health
      ansible.builtin.include_tasks: "{{ role_path }}/tasks/verify.yml"

  post_tasks:
    - name: Display final cluster status
      ansible.builtin.command:
        cmd: pvecm status
      register: final_status
      changed_when: false
      delegate_to: "{{ groups[cluster_group][0] }}"
      run_once: true

    - name: Show cluster status
      ansible.builtin.debug:
        var: final_status.stdout_lines
      run_once: true

  handlers:
    - name: reload corosync
      ansible.builtin.systemd:
        name: corosync
        state: reloaded
      throttle: 1

Usage

Initialize Matrix Cluster

# Check syntax
ansible-playbook playbooks/cluster-init.yml --syntax-check

# Dry run (limited functionality)
ansible-playbook playbooks/cluster-init.yml --check --diff

# Initialize cluster
ansible-playbook playbooks/cluster-init.yml --limit matrix_cluster

# Verify cluster status
ansible -i inventory/proxmox.yml foxtrot -m shell -a "pvecm status"
ansible -i inventory/proxmox.yml foxtrot -m shell -a "pvecm nodes"

Add mise Task

# .mise.toml
[tasks."cluster:init"]
description = "Initialize Proxmox cluster"
run = """
cd ansible
uv run ansible-playbook playbooks/cluster-init.yml
"""

[tasks."cluster:status"]
description = "Show cluster status"
run = """
ansible -i ansible/inventory/proxmox.yml foxtrot -m shell -a "pvecm status"
"""

Troubleshooting

Node Won't Join Cluster

Symptoms:

• pvecm add fails with timeout or connection error

Solutions:

1. Verify SSH connectivity: ssh root@first-node hostname
2. Check /etc/hosts: getent hosts first-node
3. Verify corosync network: ping -c 3 192.168.8.5
4. Check firewall: iptables -L | grep 5404

Cluster Shows No Quorum

Symptoms:

• pvecm status shows Quorate: No

Solutions:

1. Check node count: Must have majority (2 of 3, 3 of 5, etc.)
2. Verify corosync: systemctl status corosync
3. Check corosync ring: corosync-cfgtool -s
4. Review logs: journalctl -u corosync -n 50

Configuration Sync Issues

Symptoms:

• Changes on one node don't appear on others

Solutions:

1. Verify pmxcfs: systemctl status pve-cluster
2. Check filesystem: pvecm status | grep -i cluster
3. Restart cluster filesystem: systemctl restart pve-cluster

Related Workflows

• CEPH Deployment - Deploy CEPH after cluster formation
• Network Configuration - Configure cluster networking
• Cluster Maintenance - Add/remove nodes, upgrades

References

• ProxSpray analysis: docs/proxspray-analysis.md (lines 1318-1428)
• Proxmox VE Cluster Manager documentation
• Corosync configuration guide
• Ansible cluster automation pattern