Dapper Cluster Documentation

This documentation covers the architecture, configuration, and operations of the Dapper Kubernetes cluster, a high-performance home lab infrastructure with GPU capabilities.

Cluster Overview

graph TD
    subgraph Control Plane
        CP1[Control Plane 1<br>4 CPU, 16GB]
        CP2[Control Plane 2<br>4 CPU, 16GB]
        CP3[Control Plane 3<br>4 CPU, 16GB]
    end

    subgraph Worker Nodes
        W1[Worker 1<br>16 CPU, 128GB]
        W2[Worker 2<br>16 CPU, 128GB]
        GPU[GPU Node<br>16 CPU, 128GB<br>2x Tesla P100]
    end

    CP1 --- CP2
    CP2 --- CP3
    CP3 --- CP1

    Control Plane --> Worker Nodes

Hardware Specifications

Control Plane

• 3 nodes for high availability
• 4 CPU cores per node
• 16GB RAM per node
• Dedicated to cluster control plane operations

Worker Nodes

• 8 worker nodes total: 1 GPU node, 3 large (16 CPU / 128GB), 4 small (6 CPU / 16GB)
• Handles general workloads, databases, media, and applications

GPU Node

• Specialized GPU worker node
• 16 CPU cores
• 128GB RAM
• 2x NVIDIA Tesla P100 GPUs
• Handles ML/AI and GPU-accelerated workloads

Key Features

• High-availability Kubernetes cluster
• GPU acceleration support
• Automated deployment using Flux CD
• Secure secrets management with SOPS + External Secrets (Infisical)
• External Ceph storage (RBD + CephFS) via Rook
• Comprehensive monitoring and observability
• Media services automation

Infrastructure Components

graph TD
    subgraph Core Services
        Flux[Flux CD]
        Storage[Storage Layer]
        Network[Network Layer]
    end

    subgraph Applications
        Media[Media Stack]
        Monitor[Monitoring]
        GPU[GPU Workloads]
    end

    Core Services --> Applications

    Storage --> |Ceph RBD/CephFS| Applications
    Network --> |Envoy Gateway/DNS| Applications

Documentation Structure

• Architecture: Detailed technical documentation about cluster design and components

  • High-availability control plane design
  • Storage architecture and configuration
  • Network topology and policies
  • GPU integration and management

• Applications: Information about deployed applications and their configurations

  • Media services stack
  • Monitoring and observability
  • GPU-accelerated applications

• Operations: Guides for installation, maintenance, and troubleshooting

  • Cluster setup procedures
  • Node management
  • GPU configuration
  • Maintenance tasks

Getting Started

For new users, we recommend starting with:

1. Architecture Overview - Understanding the cluster design
2. Installation Guide - Setting up the cluster
3. Application Stack - Deploying applications

Quick Links

• Storage Configuration
• Network Setup
• Maintenance Guide
• Troubleshooting

Architecture Overview

Cluster Architecture

graph TD
    subgraph Control Plane
        CP1[Control Plane 1<br>4 CPU, 16GB]
        CP2[Control Plane 2<br>4 CPU, 16GB]
        CP3[Control Plane 3<br>4 CPU, 16GB]

        CP1 --- CP2
        CP2 --- CP3
        CP3 --- CP1
    end

    subgraph Worker Nodes
        L[3x Large<br>16 CPU, 128GB]
        S[4x Small<br>6 CPU, 16GB]
    end

    subgraph GPU Node
        GPU[GPU Worker<br>16 CPU, 128GB<br>2x Tesla P100]
    end

    Control Plane --> Worker Nodes
    Control Plane --> GPU

Core Components

Control Plane

• High Availability: 3-node control plane configuration
• Resource Allocation: 4 CPU, 16GB RAM per node
• Components:
  • etcd cluster
  • API Server
  • Controller Manager
  • Scheduler

Worker Nodes

• General Purpose Workers: 7 nodes (3 large + 4 small)
• Resources:
  • Large: 16 CPU cores, 128GB RAM (x3)
  • Small: 6 CPU cores, 16GB RAM (x4)
• Workload Types:
  • Application deployments
  • Database workloads
  • Media services
  • Monitoring systems

GPU Node

• Specialized Worker: 1 node
• Hardware:
  • 16 CPU cores
  • 128GB RAM
  • 2x NVIDIA Tesla P100 GPUs
• Workload Types:
  • ML/AI workloads
  • Video transcoding
  • GPU-accelerated applications

Network Architecture

graph TD
    subgraph External
        Internet((Internet))
        DNS((DNS))
    end

    subgraph Network Edge
        FW[Firewall]
        LB[Load Balancer]
    end

    subgraph Kubernetes Network
        CP[Control Plane]
        Workers[Worker Nodes]
        GPUNode[GPU Node]

        subgraph Services
            Gateway[Envoy Gateway]
            CoreDNS[CoreDNS]
            CNI[Cilium CNI]
        end
    end

    Internet --> FW
    DNS --> FW
    FW --> LB
    LB --> CP
    CP --> Workers
    CP --> GPUNode
    Services --> Workers
    Services --> GPUNode

Storage Architecture

All persistent storage is backed by an external Ceph cluster on the Proxmox hosts, accessed through Rook's CSI drivers.

graph TD
    subgraph Storage Classes
        RBD[ceph-rbd<br/>RWO block]
        CEPHFS[cephfs-shared / cephfs-static<br/>RWX filesystem]
    end

    subgraph Persistent Volumes
        RBD --> RBD_PV[RBD PVs]
        CEPHFS --> FS_PV[CephFS PVs]
    end

    subgraph Workload Types
        RBD_PV --> DB[Databases]
        RBD_PV --> App[Single-pod apps]
        FS_PV --> Media[Media Storage]
        FS_PV --> Shared[Shared Config]
    end

Security Considerations

• Network segmentation using Kubernetes network policies
• Encrypted secrets management with SOPS
• TLS encryption for all external services
• Regular security updates via automated pipelines
• GPU access controls and resource quotas

Scalability

The cluster architecture is designed to be scalable:

• High-availability control plane (3 nodes)
• Expandable worker node pool
• Specialized GPU node for compute-intensive tasks
• Dynamic storage provisioning
• Load balancing for external services
• Resource quotas and limits management

Monitoring and Observability

graph LR
    subgraph Monitoring Stack
        Prom[Prometheus]
        Graf[Grafana]
        Alert[Alertmanager]
    end

    subgraph Node Types
        CP[Control Plane Metrics]
        Work[Worker Metrics]
        GPU[GPU Metrics]
    end

    CP --> Prom
    Work --> Prom
    GPU --> Prom
    Prom --> Graf
    Prom --> Alert

Resource Management

Control Plane

• Reserved for kubernetes control plane components
• Optimized for control plane operations
• High availability configuration

Worker Nodes

• General purpose workloads
• Balanced resource allocation
• Flexible scheduling options

GPU Node

• Dedicated for GPU workloads
• NVIDIA GPU operator integration
• Specialized resource scheduling

Network Architecture

This document covers the Kubernetes application-level networking. For physical network topology and VLAN configuration, see Network Topology.

Container Networking (CNI)

Cilium CNI

The cluster uses Cilium as the primary Container Network Interface (CNI):

• Pod CIDR: 10.69.0.0/16 (native routing mode)
• Service CIDR: 10.96.0.0/16
• Mode: Non-exclusive (paired with Multus for multi-network support)
• Kube-Proxy Replacement: Enabled (eBPF-based service load balancing)
• Load Balancing Algorithm: Maglev with DSR (Direct Server Return)
• Network Policy: Endpoint routes enabled
• BPF Masquerading: Enabled for outbound traffic

Key Features:

• High-performance eBPF data plane
• Native Kubernetes network policy support
• L2 announcements for external load balancer IPs
• Advanced observability and monitoring

Multus CNI (Multiple Networks)

Multus provides additional network interfaces to pods beyond the primary Cilium network:

• Primary Use: IoT network attachment (VLAN-based isolation)
• Network Attachment: macvlan on ens19 interface
• Mode: Bridge mode with DHCP IPAM
• Purpose: Enable pods to connect to additional networks (e.g., IoT devices, legacy systems)

Pods can request additional networks via annotations:

metadata:
  annotations:
    k8s.v1.cni.cncf.io/networks: macvlan-conf

Gateway API (Envoy Gateway)

The cluster routes all HTTP(S) traffic through Envoy Gateway (Gateway API), not Ingress objects. A single GatewayClass named envoy backs three Gateway resources, each with its own Cilium LB-IPAM load-balancer IP:

Each Gateway terminates TLS using wildcard certificates (cert-manager) and exposes per-domain HTTPS listeners for the cluster's domains. Applications attach to a Gateway by creating an HTTPRoute whose hostname matches a listener — the listener is auto-selected by hostname, so routes generally don't pin a sectionName. Plain HTTP (port 80) listeners redirect to HTTPS.

Authentication: Login protection is applied declaratively at the Gateway via Authentik. Apps annotated for forward-auth get a Kyverno-generated SecurityPolicy (ext-auth to the Authentik outpost); some apps use native Authentik OIDC instead. See the authentik-auth repo skill.

Load Balancer IP Management

Cilium L2 Announcements

Cilium's L2 announcement feature provides load balancer IPs for services:

• How it works: Cilium announces load balancer IPs via L2 (ARP/NDP)
• Policy-based: L2AnnouncementPolicy defines which services get announced
• Benefits:
  • No external load balancer required
  • Native Kubernetes LoadBalancer service type support
  • High availability through leader election
  • Automatic failover

Configuration: See kubernetes/apps/kube-system/cilium/config/l2.yaml

This is how each Envoy Gateway (and any other LoadBalancer service) receives an external IP accessible from the broader network.

Network Policies

graph LR
    subgraph Policies
        Default[Default Deny]
        Allow[Allowed Routes]
    end

    subgraph Apps
        Media[Media Stack]
        Monitor[Monitoring]
        DB[Databases]
    end

    Allow --> Media
    Allow --> Monitor
    Default --> DB

DNS Configuration

Internal DNS (k8s-gateway)

• Purpose: Authoritative DNS for the cluster's private domains
• Answers: Hostnames resolve to the internal Envoy Gateway LB IP (10.100.0.20)
• Delegation: The OPNsense resolver forwards the private domains to k8s-gateway

External DNS (Cloudflare)

• Provider: Cloudflare (single ExternalDNS instance)
• Source: Watches HTTPRoute/Gateway resources on the public domains
• Purpose: Publishes public DNS records; external traffic enters via the Cloudflare Tunnel (cloudflared)

How DNS Works

1. Create an HTTPRoute attached to the internal, external, or media Gateway
2. Internal hostnames are answered by k8s-gateway; public hostnames are synced to Cloudflare by ExternalDNS
3. Services become reachable at their configured hostnames

Security

Network Policies

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

TLS Configuration

• Automatic certificate management via cert-manager
• Let's Encrypt integration (wildcard certs per domain)
• TLS terminated at the Envoy Gateways

Traffic Flow

graph LR
    subgraph Edge
        CF[Cloudflare Tunnel]
        LAN[LAN Clients]
    end

    subgraph Gateways
        EXT[external Gateway]
        INT[internal Gateway]
    end

    subgraph Services
        App1[Service 1]
        App2[Service 2]
        DB[Database]
    end

    CF --> EXT
    LAN --> INT
    EXT --> App1
    INT --> App2
    App1 --> DB
    App2 --> DB

External clients reach public services through the Cloudflare Tunnel into the external Gateway; LAN clients resolve private hostnames via k8s-gateway and hit the internal Gateway directly.

Best Practices

1. Security

   • Implement default deny policies
   • Use TLS everywhere
   • Regular security audits
   • Network segmentation

2. Performance

   • Load balancer optimization
   • Connection pooling
   • Proper resource allocation
   • Traffic monitoring

3. Reliability

   • High availability configuration
   • Failover planning
   • Backup routes
   • Health checks

4. Monitoring

   • Network metrics collection
   • Traffic analysis
   • Latency monitoring
   • Bandwidth usage tracking

Troubleshooting

Common network issues and resolution steps:

1. Connectivity Issues

   • Check network policies
   • Verify DNS resolution
   • Inspect service endpoints
   • Review ingress configuration

2. Performance Problems

   • Monitor network metrics
   • Check for bottlenecks
   • Analyze traffic patterns
   • Review resource allocation

Network Topology

Current physical / L2 / L3 topology of the homelab switching fabric. Verified live from every switch on 2026-06-30 (Brocade, Arista, Aruba, both Mikrotik NRay radios). The Mikrotik CSS326 is SwOS web-only; its role here is confirmed from neighbor discovery on adjacent devices.

The network spans two buildings — house and garage/shop — joined by a single 60GHz wireless bridge (~1 Gbps). The garage holds the core (Brocade) and the high-speed Ceph distribution switch (Arista); the house holds the access switch (Aruba) and the OPNsense router.

Bandwidth & Link Diagram

At-a-glance view of devices, links, and aggregated speeds. Rendered from the Mermaid source below (kept in-doc as the source of truth for the topology infographic).

flowchart LR
internet([Internet — AT&T Fiber — 2.5 Gbps]):::wan

  subgraph HOUSE
    direction TB
    opn[OPNsense<br/>Router / Firewall — .1]:::fw
    aruba[Aruba S2500-48P<br/>Access / PoE — .26]:::sw
    ws[10G workstation]:::client
    clients[Clients / PoE]:::client
    nray7[MikroTik NRay — .7]:::radio
  end

  subgraph GARAGE_SHOP[GARAGE / SHOP]
    direction TB
    nray8[MikroTik NRay — .8]:::radio
    css[MikroTik CSS326 — .27]:::sw
    brocade[Brocade ICX6610<br/>Core / L3 — .20]:::core
    arista[Arista 7050<br/>Ceph Distribution — .21]:::core
    pmox[Proxmox cluster x4<br/>Talos K8s + Ceph]:::server
    netboot[netboot.xyz<br/>PXE — .10 · Proxmox LXC]:::server
  end

  internet -->|2.5G| opn
  opn ==>|10G| aruba
  aruba ==>|10G| ws
  aruba -->|1G| clients
  aruba -->|1G| nray7
  nray7 -. "60GHz Wireless Bridge<br/>~1 Gbps · bottleneck / SPOF" .-> nray8
  nray8 -->|1G| css
  css ==>|10G SFP+| brocade
  brocade ====|"80G LACP (2×40G)"| arista
  arista ====|"2×40G Ceph → Arista (80G bond)"| pmox
  brocade ==>|"2×10G VM LAG (20G)"| pmox
  brocade -->|"2×1G mgmt LAG (2G)"| pmox
  brocade -->|1G IPMI| pmox
  netboot -.-> pmox

  classDef wan     fill:#0b2a3a,stroke:#38bdf8,color:#e6f6ff,stroke-width:2px;
  classDef fw      fill:#12324a,stroke:#38bdf8,color:#e6f6ff,stroke-width:2px;
  classDef sw      fill:#12324a,stroke:#5eead4,color:#e6fffb,stroke-width:2px;
  classDef core    fill:#1a2e12,stroke:#a3e635,color:#f7ffe6,stroke-width:2px;
  classDef server  fill:#2a1436,stroke:#e879f9,color:#fdebff,stroke-width:2px;
  classDef radio   fill:#3a2a0b,stroke:#f59e0b,color:#fff7e6,stroke-width:2px;
  classDef client  fill:#1b2430,stroke:#94a3b8,color:#e2e8f0,stroke-width:2px;

Device Inventory

All switch management interfaces sit on VLAN 1 (192.168.1.0/24).

VLANs

• VLAN 1 is the untagged/native VLAN on every switch trunk port (Brocade dual-mode 1, Arista/Aruba native 1).
• VLAN 20 (LAB) is defined and trunked on the Brocade only. Arista carries it in its trunk allowed-lists but it is not instantiated in the Arista VLAN database; Aruba does not carry it.
• VLAN 150 has no L3 interface on any switch — purely L2, jumbo frames, gateway-less.
• VLAN 200 has a single SVI: Brocade ve200 10.200.0.254/24. It is otherwise internal/east-west only.

Kubernetes internal networks (Cilium, not switched)

Layer 3 / Routing (verified)

• OPNsense (house) — default gateway for VLAN 1 (192.168.1.1) and VLAN 100 (10.100.0.1). Upstream internet: AT&T Fiber, 2.5 Gbps. Connects to the house fabric over 10G fiber (mce0/mce1, 10Gbase-SR, both active) — this is the house backbone.
• Brocade ICX6610 — SVIs: ve1 192.168.1.20/24, ve100 10.100.0.10/24, ve200 10.200.0.254/24. Jumbo frames enabled globally. Two default routes point at OPNsense: 0.0.0.0/0 → 192.168.1.1 and 0.0.0.0/0 → 10.100.0.1.
• Arista 7050 — no ip routing; pure L2. Management only: Management1 192.168.1.21/24, default route 0.0.0.0/0 → 192.168.1.1.
• Aruba S2500 — L2 access. vlan 1 IP via DHCP (currently 192.168.1.26), vlan 100 10.100.0.26/24, dedicated OOB mgmt port 172.16.0.254/24. 10G SFP+ to OPNsense (house backbone); can serve 10G to select house clients (e.g. a workstation), 1G to the rest.

DNS Resolution (verified 2026-07-01)

Name resolution is layered, with a garage-local failover so the cluster keeps resolving through a 60GHz bridge outage:

• OPNsense Unbound — 192.168.1.1 / 10.100.0.1 (house). Primary resolver for all VLANs: recursive + DNSBL filtering, and forwards the internal split-horizon domains to the in-cluster k8s-gateway.
• Garage Unbound — 10.100.0.2 (Proxmox LXC unbound-garage, CT 110 on proxmox-04, VLAN 100). Failover resolver, garage-local. Mirrors the house: forwards the same internal domains to k8s-gateway, and forwards everything else upstream to OPNsense (10.100.0.1) so external DNS still passes the firewall's DNSBL/logging.
• k8s-gateway — LB VIP 10.100.0.21. Authoritative for the internal split-horizon domains (chelonianlabs.com, turtleassmedia.com, derekmackley.com, dapperdivers.com), returning in-cluster gateway VIPs. Both resolvers above delegate these domains to it.

Talos node resolvers (machine.network.nameservers, all 11 nodes): 10.100.0.1 primary, 10.100.0.2 fallback. During a bridge cut the garage resolver keeps internal + cached names resolving; external names need WAN and are unavailable regardless.

Network Boot (PXE)

netboot.xyz runs as Proxmox LXC netboot-xyz (CT 111, 192.168.1.10, mgmt VLAN). OPNsense DHCP hands PXE clients on both VLAN 1 and VLAN 100 next-server 192.168.1.10 plus the netboot.xyz bootloader over TFTP, which chainloads the netboot.xyz menu over HTTP. Talos VMs boot disk-first and only fall through to PXE if the disk is unbootable; netboot.xyz's built-in Talos entry can then boot the stock Talos image into maintenance mode for talosctl.

Inter-Site Path (house ↔ garage)

The only data path between buildings is the 60GHz wireless bridge. The verified end-to-end chain:

House clients / OPNsense (192.168.1.1)
  └─ Aruba S2500  ── GE0/0/21 ──▶ NRay-house ether1
        NRay-house  ))) 60GHz, 58320 MHz, ~1 Gbps  (((  NRay-shop
                                   NRay-shop ether1 ──▶ CSS326
                                        CSS326 ──(SFP+ 10G)──▶ Brocade 1/3/1 "Uplink-10Gb-Primary"
                                                                  └─ garage fabric

• Aruba GE0/0/21 → NRay-house ether1 — LLDP-confirmed from the Aruba.
• NRay-shop ether1 → CSS326 (192.168.1.27) — MNDP/neighbor-confirmed on the NRay.
• CSS326 → Brocade 1/3/1 ("Uplink-10Gb-Primary", up at 10G) — the Brocade's only active uplink port and the CSS326's SFP+ uplink; this is the garage-side landing of the bridge path.
• Brocade 1/1/1 ("Uplink-1G-Backup") is administratively disabled.
• The bridge is transparent L2 — both NRays bridge ether1+wlan60 with no VLAN filtering, so all VLANs (1/100/…) cross it. Effective throughput is gated to ~1 Gbps by the 60GHz link and the radios' gigabit ether1 ports.

60GHz link health (verified 2026-06-30)

Brocade ICX6610 — Core (verified port map)

Module layout: 1/1 = 48× 1G PoE, 1/2 = QSFP/10G module, 1/3 = 8× 10G dual-mode. Every port is dual-mode 1 (untagged VLAN 1) and tagged for VLANs 1, 20, 100, 150, 200.

Server / host LAGs (dynamic = LACP)

Each hypervisor presents three connections to the core: a single 1G IPMI port, a 2×1G management LAG, and a 2×10G VM-traffic LAG.

• Athena — all of its links (IPMI, mgmt LAG, VM LAG) are currently down (host offline at verification time).
• Tower has the same 1G/10G attachment to the core as a hypervisor but no 40G link to the Arista (it is not a Ceph OSD node).

Interconnect & infrastructure ports

SVIs

interface ve 1    ip address 192.168.1.20  255.255.255.0
interface ve 100  ip address 10.100.0.10   255.255.255.0
interface ve 200  ip address 10.200.0.254  255.255.255.0

Arista 7050 — Ceph Distribution (verified port map)

L2-only (no ip routing), MSTP, all switched ports MTU 9214 (jumbo), trunking VLANs 1, 20, 100, 150, 200.

• Et9/1 and Et10/1 are spare 40G ports (no channel-group).
• VLAN database: 1 (default), 100 (kubernetes), 150 (ceph-public), 200 (ceph-storage).

The four Proxmox/Ceph hypervisors each carry their VLAN 200 (and 150) Ceph traffic over a dedicated 2×40G LACP bond to the Arista. Ceph OSD replication stays east-west on this switch and does not traverse the Brocade.

Aruba S2500-48P — House Access (verified)

ArubaOS MAS, MSTP + LACP, carries VLAN 1 (untagged/native) and VLAN 100 only — no Ceph VLANs.

• Default ports: trunk (VLAN 1 native + VLAN 100 tagged).
• VLAN-100 access ports: GE0/0/38 and GE0/0/40 (switching-profile "Servers", native VLAN 100).
• Garage uplink: GE0/0/21 → NRay-house ether1 (LLDP-confirmed).
• Management: VLAN 1 via DHCP (192.168.1.26), VLAN 100 = 10.100.0.26/24, OOB mgmt = 172.16.0.254/24.

LLDP neighbors at verification time: GE0/0/21 MikroTik NRay-house, GE0/0/25 ACMesh AP, GE0/0/29 UniFi U6-Plus AP, GE0/0/20 a wired client, GE0/1/0 a server-class host.

Mikrotik 60GHz Bridge (verified)

Both radios run RouterOS 7.18.2 on nRAYG-60ad hardware, each with a single bridge named "Wireless Bridge" containing ether1 + the 60GHz interface, vlan-filtering=no (transparent), MTU 1500.

• NRay-house (192.168.1.7) — wlan60-1 mode=bridge (AP). ether1 → Aruba GE0/0/21.
• NRay-shop (192.168.1.8) — wlan60-1 mode=station-bridge. ether1 → CSS326.
• Both addresses are DHCP leases from OPNsense on VLAN 1.

Mikrotik CSS326 (verified from config 2026-06-30)

CSS326-24G-2S+ running SwOS 2.18 at 192.168.1.27 — the garage-side aggregation point for the wireless bridge. The NRay-shop's ether1 lands on a 1G port here (neighbor-confirmed), and one of the SFP+ ports (SFP1/SFP2) provides the 10G uplink to the Brocade core (1/3/1).

• 24× 1G (Port1–24) + 2× SFP+ (SFP1/SFP2); all ports use default names (the config carries no custom labels, so the NRay-shop vs Brocade ports are identified only by neighbor discovery).
• Transparent L2 passthrough: per-port VLAN mode = optional with default VID 1, full-mesh forwarding (no port isolation), RSTP on all ports. It forwards tagged VLANs by their tag and treats untagged frames as VLAN 1 — so VLAN 1 (mgmt) and 100 (servers) cross it unchanged.
• Management: DHCP (currently 192.168.1.27), static fallback 192.168.88.1. Identity is still the default "MikroTik"; SNMP enabled with community public.
• The switch also has a set of defined-but-unenforced VLAN names left in its table — 2 (MGMT), 3 (PTP), 20 (LAB), 30 (DEMO), 100 (KUBERNETES). Because VLAN mode is optional (not strict), these are not enforced and do not match the rest of the fabric; they appear to be stale/experimental entries. No LACP, no ACLs configured.

Compute Attachment Summary

Five hosts attach to the core; four of them are also Ceph OSD nodes on the Arista.

Per-host management IPs are 192.168.1.62–.66 (IPMI 192.168.1.162–.165); see hardware.md for chassis detail and the Talos/k8s VM addressing in the cluster configuration.

Traffic Segregation

Verification Commands

# Reachability
ping 192.168.1.20   # Brocade   192.168.1.21  # Arista   192.168.1.26  # Aruba
ping 192.168.1.7    # NRay-house 192.168.1.8   # NRay-shop 192.168.1.27 # CSS326

# Brocade (FastIron)
show vlan brief
show interface brief
show running-config

# Arista (EOS)
show port-channel summary
show interfaces description
show lldp neighbors

# Aruba (ArubaOS MAS)
show vlan
show port status
show lldp neighbor

# Mikrotik NRay (RouterOS)
/interface w60g monitor wlan60-1 once
/ip neighbor print detail

Storage Architecture

Storage Overview

The Dapper Cluster uses Rook Ceph as its only storage solution, providing unified block and shared-filesystem storage for all Kubernetes workloads. The external Ceph cluster (Ceph 18.2.x Reef) runs on the Proxmox hosts and is connected to Kubernetes via Rook's external cluster mode. CephFS uses three data pools: cephfs_data, cephfs_media, and cephfs_backups.

graph TD
    subgraph External Ceph Cluster
        MON[Ceph Monitors]
        OSD[Ceph OSDs]
        MDS[Ceph MDS - CephFS]
    end

    subgraph Kubernetes Cluster
        ROOK[Rook Operator]
        CSI[Ceph CSI Drivers]
        SC[Storage Classes]
    end

    subgraph Applications
        APPS[Application Pods]
        PVC[Persistent Volume Claims]
    end

    MON --> ROOK
    ROOK --> CSI
    CSI --> SC
    SC --> PVC
    PVC --> APPS
    CSI --> MDS
    CSI --> OSD

Storage Architecture Decision

Why Rook Ceph?

The cluster uses Rook Ceph (external mode) for several key reasons:

1. Unified Storage Platform: Single storage solution for all workload types
2. External Cluster Design: Leverages the Proxmox Ceph cluster infrastructure
3. High Performance: Direct Ceph integration over the dedicated 40Gb storage network
4. Scalability: Native Ceph scalability for growing storage needs
5. Feature Rich: Snapshots, cloning, expansion, and advanced storage features
6. ReadWriteMany Support: CephFS provides shared filesystem access
7. Production Proven: Mature, widely-adopted storage solution

Current Storage Classes

The live storage classes are: ceph-rbd (default), cephfs-shared, cephfs-static, and cephfs-backups.

RBD Block Storage (Default)

Storage Class: ceph-rbd

The default storage class. Any PVC without an explicit storageClassName is provisioned as an RBD block volume.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ceph-rbd
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
  clusterID: rook-ceph
  pool: rook-pvc-pool
  imageFeatures: layering
  csi.storage.k8s.io/fstype: ext4
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: Immediate

Characteristics:

• Access Mode: ReadWriteOnce (RWO) - single-pod exclusive access
• Use Cases: Databases, etcd, and any single-pod app needing high IOPS
• Performance: Superior to CephFS for block workloads
• Default: Yes

CephFS Shared Storage

Storage Class: cephfs-shared

Dynamically-provisioned shared filesystem for workloads needing RWX.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: cephfs-shared
provisioner: rook-ceph.cephfs.csi.ceph.com
parameters:
  clusterID: rook-ceph
  fsName: cephfs
  pool: cephfs_data
  subvolumeGroup: csi
  mounter: kernel
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: Immediate

Characteristics:

• Access Mode: ReadWriteMany (RWX) - Multiple pods can read/write simultaneously
• Use Cases:
  • Applications requiring shared storage
  • Configuration storage
  • General multi-pod application storage
• Performance: Good performance for most workloads, shared filesystem overhead

CephFS Static Storage

Storage Class: cephfs-static

Used for pre-existing CephFS paths that need to be mounted into Kubernetes.

Characteristics:

• Access Mode: ReadWriteMany (RWX)
• Use Cases:
  • Mounting existing CephFS directories (e.g., /media)
  • Large media libraries
  • Shared configuration repositories
• Provisioning: Manual - requires creating both PV and PVC
• Pattern: See "Static PV Pattern" section below

Example: Media storage at /media on cephfs_media pool

apiVersion: v1
kind: PersistentVolume
metadata:
  name: media-cephfs-pv
spec:
  capacity:
    storage: 100Ti
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: cephfs-static
  csi:
    driver: rook-ceph.cephfs.csi.ceph.com
    nodeStageSecretRef:
      name: rook-csi-cephfs-static
      namespace: rook-ceph
    volumeAttributes:
      clusterID: rook-ceph
      fsName: cephfs
      staticVolume: "true"
      rootPath: /media
      pool: cephfs_media
      mounter: kernel

CephFS Backups

Storage Class: cephfs-backups

Dedicated CephFS class on the cephfs_backups pool with a Retain reclaim policy, used for backup repositories so data survives PVC deletion.

Characteristics:

• Access Mode: ReadWriteMany (RWX)
• Reclaim Policy: Retain
• Use Cases: VolSync Restic repositories and other backup data

Storage Provisioning Patterns

Dynamic Provisioning (Default)

For most applications, simply create a PVC and Kubernetes will automatically provision storage. With no storageClassName, the default ceph-rbd (RWO block) class is used; set storageClassName: cephfs-shared when you need RWX:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-app-data
  namespace: my-namespace
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  # No storageClassName specified = uses default (ceph-rbd, RWO)

Static PV Pattern

For mounting pre-existing CephFS paths:

Step 1: Create PersistentVolume

apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-static-pv
spec:
  capacity:
    storage: 5Ti
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: cephfs-static
  csi:
    driver: rook-ceph.cephfs.csi.ceph.com
    nodeStageSecretRef:
      name: rook-csi-cephfs-static
      namespace: rook-ceph
    volumeAttributes:
      clusterID: rook-ceph
      fsName: cephfs
      staticVolume: "true"
      rootPath: /my-data # Path in CephFS
      pool: cephfs_data # Target data pool
      mounter: kernel # Kernel mounter for better performance

Step 2: Create matching PersistentVolumeClaim

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-static-pvc
  namespace: my-namespace
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Ti
  storageClassName: cephfs-static
  volumeName: my-static-pv

Current Static PVs in Use:

• media-cephfs-pv → /media on cephfs_media pool (100Ti)
• minio-cephfs-pv → /minio on cephfs_data pool (10Ti)
• paperless-cephfs-pv → /paperless on cephfs_data pool (5Ti)

Storage Decision Matrix

Backup Strategy

VolSync

All persistent data is backed up using VolSync:

• Backup Frequency: Hourly snapshots via ReplicationSource
• Repository Storage: CephFS (cephfs-backups pool), Restic repositories
• Restore/cache PVCs: Default to ceph-rbd
• Retention: Configurable per-application
• Recovery: Supports restore to same or different PVC

VolSync Repository Location: /repository/{APP} on CephFS

Network Configuration

Ceph Networks

The external Ceph cluster uses two networks:

• Public Network: 10.150.0.0/24
  • Client connections from Kubernetes
  • Ceph monitor communication
  • Used by CSI drivers
• Cluster Network: 10.200.0.0/24
  • OSD-to-OSD replication
  • Not directly accessed by Kubernetes

Connection Method

Kubernetes connects to Ceph via:

1. Rook Operator: Manages connection to external cluster
2. CSI Drivers: cephfs.csi.ceph.com for CephFS volumes
3. Mon Endpoints: ConfigMap with Ceph monitor addresses
4. Authentication: Ceph client.kubernetes credentials

Performance Characteristics

CephFS Performance

• Sequential Read: Excellent (limited by network, ~10 Gbps)
• Sequential Write: Very Good (COW overhead, CRUSH rebalancing)
• Random I/O: Good (shared filesystem overhead)
• Concurrent Access: Excellent (native RWX support)
• Metadata Operations: Good (dedicated MDS servers)

Optimization Tips

1. Use RWO when possible: Even on CephFS, specify RWO if no sharing needed
2. Size appropriately: CephFS handles small and large files well
3. Monitor MDS health: CephFS performance depends on MDS responsiveness
4. Enable client caching: Default CSI settings enable attribute caching

Storage Operations

Common Operations

Expand a PVC:

kubectl patch pvc my-pvc -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'

Check storage usage:

kubectl get pvc -A
kubectl exec -it <pod> -- df -h

Monitor Ceph cluster health:

kubectl -n rook-ceph get cephcluster
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph status

List CephFS mounts:

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph fs status

Troubleshooting

PVC stuck in Pending:

kubectl describe pvc <pvc-name>
kubectl -n rook-ceph logs -l app=rook-ceph-operator

Slow performance:

# Check Ceph cluster health
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph health detail

# Check MDS status
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph fs status

# Check OSD performance
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd perf

Mount issues:

# Check CSI driver logs
kubectl -n rook-ceph logs -l app=csi-cephfsplugin

# Verify connection to monitors
kubectl -n rook-ceph get configmap rook-ceph-mon-endpoints -o yaml

Best Practices

Storage Selection

1. Databases and single-pod apps: Use ceph-rbd for best performance
2. Shared storage needs: Use cephfs-shared for RWX access
3. Use static PVs for existing data: Don't duplicate large datasets
4. Specify requests accurately: Helps with capacity planning
5. Choose appropriate access modes: RWO for RBD, RWX for CephFS

Capacity Planning

1. Monitor Ceph cluster capacity:

   kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph df
   

2. Set appropriate PVC sizes: CephFS supports expansion
3. Plan for growth: Ceph cluster can scale by adding OSDs
4. Regular capacity reviews: Check usage trends

Data Protection

1. Enable VolSync: For all stateful applications
2. Test restores regularly: Ensure backup viability
3. Monitor backup success: Check ReplicationSource status
4. Retain snapshots appropriately: Balance storage cost vs recovery needs

Security

1. Use namespace isolation: PVCs are namespace-scoped
2. Limit access with RBAC: Control who can create PVCs
3. Monitor access patterns: Unusual I/O may indicate issues
4. Rotate Ceph credentials: Periodically update client keys

Monitoring and Observability

Key Metrics

Monitor these metrics via Prometheus/Grafana:

• Ceph cluster health status
• OSD utilization and performance
• MDS cache hit rates
• PVC capacity usage
• CSI operation latencies
• VolSync backup success rates

Alerts

Critical alerts configured:

• Ceph cluster health warnings
• High OSD utilization (>80%)
• MDS performance degradation
• PVC approaching capacity
• VolSync backup failures

References

• Rook Documentation: rook.io/docs
• Ceph Documentation: docs.ceph.com
• Local Setup: kubernetes/apps/rook-ceph/README.md
• Storage Classes: kubernetes/apps/rook-ceph/rook-ceph-cluster/app/storageclasses.yaml

Media Applications

Media Stack Overview

The media stack provides automated media management and streaming services using the *arr suite of applications and Plex Media Server.

graph TD
    subgraph Content Acquisition
        SONARR[Sonarr - TV Shows]
        SONARR_UHD[Sonarr UHD - 4K TV]
        RADARR[Radarr - Movies]
        RADARR_UHD[Radarr UHD - 4K Movies]
        BAZARR[Bazarr - Subtitles]
        BAZARR_UHD[Bazarr UHD - 4K Subtitles]
    end

    subgraph Download Clients
        SABNZBD[SABnzbd - Usenet]
        NZBGET[NZBget - Usenet Alt]
    end

    subgraph Media Storage
        CEPHFS[CephFS Static PV<br>/media]
    end

    subgraph Media Server
        PLEX[Plex Media Server]
        TAUTULLI[Tautulli - Analytics]
        OVERSEERR[Overseerr - Requests]
    end

    subgraph Post Processing
        TDARR[Tdarr - Transcoding]
        KOMETA[Kometa - Metadata]
    end

    SONARR --> SABNZBD
    RADARR --> SABNZBD
    SABNZBD --> CEPHFS
    CEPHFS --> PLEX
    PLEX --> TAUTULLI
    OVERSEERR --> SONARR
    OVERSEERR --> RADARR
    TDARR --> CEPHFS
    KOMETA --> PLEX

Storage Configuration

Media Library - CephFS

The media library is stored on CephFS using a static PV that mounts the /media path on the dedicated cephfs_media pool. The same path can also be mounted directly from the Proxmox hosts via a standard CephFS mount.

Configuration: kubernetes/apps/media/storage/app/media-cephfs-pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: media-cephfs-pv
spec:
  capacity:
    storage: 100Ti
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: cephfs-static
  csi:
    driver: rook-ceph.cephfs.csi.ceph.com
    volumeHandle: media-static
    nodeStageSecretRef:
      name: rook-csi-cephfs-static
      namespace: rook-ceph
    volumeAttributes:
      clusterID: rook-ceph
      fsName: cephfs
      staticVolume: "true"
      rootPath: /media
      pool: cephfs_media
      mounter: kernel

Mount Pattern: Applications mount this PVC at /media or specific subdirectories:

• /media/downloads - Download staging area
• /media/tv - TV show library
• /media/movies - Movie library
• /media/music - Music library
• /media/books - Book library

Benefits:

• ReadWriteMany: Multiple pods can access simultaneously
• High Performance: Direct CephFS access, no NFS overhead
• Shared Access: All media apps see the same filesystem
• Snapshots: VolSync backups protect the data

All media apps mount the single media-cephfs-pvc — there is one unified library on CephFS.

Core Components

Media Server

Plex Media Server

Namespace: media Purpose: Media streaming and library management

Plex is the primary media server, providing:

• Streaming to multiple devices
• Hardware transcoding (Intel Quick Sync)
• Library organization and metadata
• User management and sharing
• Remote access

Configuration: kubernetes/apps/media/plex/app/helmrelease.yaml

Storage Mounts:

• Media library: CephFS static PV (media-cephfs-pvc) at /media
• Configuration: ceph-rbd PVC
• Transcoding cache: EmptyDir (temporary)

Library Configuration:

• Plex libraries scan the unified /media CephFS library

Resource Allocation:

resources:
  requests:
    cpu: 2000m
    memory: 4Gi
    gpu.intel.com/i915: 1
  limits:
    cpu: 8000m
    memory: 16Gi
    gpu.intel.com/i915: 1

Hardware Acceleration: Intel Quick Sync enabled for transcoding

Tautulli

Namespace: media Purpose: Plex analytics and monitoring

Provides:

• Watch history and statistics
• User activity monitoring
• Notification triggers
• API for automation

Storage: CephFS dynamic PVC (5Gi) for database and logs

Content Acquisition (*arr Suite)

Sonarr / Sonarr UHD

Purpose: TV show automation

• Sonarr: Standard quality TV shows
• Sonarr UHD: 4K/UHD TV shows

Features:

• TV series tracking and monitoring
• Episode search and download
• Quality profiles and upgrades
• Calendar and schedule tracking

Storage:

• Configuration: CephFS dynamic PVC (10Gi)
• Media access: CephFS static PV (shared /media)

Radarr / Radarr UHD

Purpose: Movie automation

• Radarr: Standard quality movies
• Radarr UHD: 4K/UHD movies

Features:

• Movie library management
• Automated downloads
• Quality management
• List integration (IMDb, Trakt)

Storage:

• Configuration: CephFS dynamic PVC (10Gi)
• Media access: CephFS static PV (shared /media)

Bazarr / Bazarr UHD

Purpose: Subtitle management

Automated subtitle downloading for:

• TV shows (via Sonarr integration)
• Movies (via Radarr integration)
• Multiple languages
• Subtitle providers

Storage: CephFS dynamic PVC (5Gi)

Prowlarr

Purpose: Unified indexer management

Centrally manages indexers/trackers and syncs them to Sonarr, Radarr, and their UHD instances, so indexer configuration lives in one place. Recyclarr, Autobrr, Omegabrr, Cross-seed, and Huntarr complement the *arr suite for quality profiles, release automation, and cross-seeding.

Download Clients

SABnzbd

Namespace: media Purpose: Primary Usenet download client

Features:

• NZB file processing
• Automated post-processing
• Category-based handling
• Integration with *arr apps

Storage Mounts:

• Configuration: ceph-rbd PVC
• Downloads: CephFS static PV at /media/downloads/usenet
• Incomplete: dynamic PVC (temporary downloads)

Post-Processing: Automatically moves completed downloads to the appropriate /media library folders (then imported by the *arr apps)

NZBget

Namespace: media Purpose: Alternative Usenet client

Lightweight alternative to SABnzbd for specific use cases.

Storage: Similar pattern to SABnzbd

Post-Processing

Tdarr

Purpose: Media transcoding and file optimization

Components:

1. Tdarr Server: Manages transcoding queue
2. Tdarr Node: CPU-based transcoding workers
3. Tdarr Node GPU: GPU-accelerated transcoding

Use Cases:

• Convert media to h265/HEVC
• Reduce file sizes
• Standardize formats
• Remove unwanted audio/subtitle tracks

Storage:

• Configuration: CephFS dynamic PVC (25Gi)
• Media access: CephFS static PV (shared /media)
• Transcode cache: CephFS dynamic PVC (100Gi)

Resource Intensive: Uses significant CPU/GPU resources during transcoding

Kometa (formerly Plex Meta Manager)

Purpose: Enhanced Plex metadata and collections

Features:

• Automated collections (e.g., "Top Rated 2023")
• Poster and artwork management
• Rating and tag synchronization
• Scheduled metadata updates

Storage: CephFS dynamic PVC (5Gi) for configuration

User Management

Overseerr

Namespace: media Purpose: Media request management

User-facing application for:

• Media requests (movies/TV shows)
• Request approval workflow
• User quotas and limits
• Integration with Sonarr/Radarr

Authentication: Integrated with Plex accounts

Storage: CephFS dynamic PVC (5Gi)

Network Configuration

Internal Access

Most media apps are exposed via an HTTPRoute attached to the internal Envoy Gateway, resolved by k8s-gateway. The app-template route block looks like:

route:
  app:
    hostnames: ["sonarr.example.com"]
    parentRefs:
      - name: internal
        namespace: network
        sectionName: https

External / Media Access

Plex and other streaming services attach to the dedicated media Gateway (LB IP 10.100.0.31), which is reachable WAN-direct and bypasses the Cloudflare Tunnel for high-bandwidth streaming. Other externally-shared apps route through the external Gateway and the Cloudflare Tunnel. See Network Applications.

Service Discovery

Applications discover each other via Kubernetes services:

• sonarr.media.svc.cluster.local:8989
• radarr.media.svc.cluster.local:7878
• sabnzbd.media.svc.cluster.local:8080
• plex.media.svc.cluster.local:32400

Backup Strategy

Application Configuration

All *arr application configurations are backed up via VolSync:

Backup Schedule: Hourly Retention:

• Hourly: 24 snapshots
• Daily: 7 snapshots
• Weekly: 4 snapshots

Backup Pattern:

apiVersion: volsync.backube/v1alpha1
kind: ReplicationSource
metadata:
  name: sonarr
  namespace: media
spec:
  sourcePVC: sonarr-config
  trigger:
    schedule: "0 * * * *"
  restic:
    repository: sonarr-restic-secret
    retain:
      hourly: 24
      daily: 7
      weekly: 4

Media Library

Media files are NOT backed up via VolSync due to size (100Ti+)

Protection Strategy:

• Ceph replication across OSDs
• Replaceable content (can be re-downloaded)
• Critical media manually backed up externally

Configuration Backup: All *arr databases and settings are backed up

Resource Management

Resource Allocation Strategy

Media applications have varying resource needs:

High Resource:

• Plex: 2-8 CPU, 4-16Gi RAM, GPU for transcoding
• Tdarr: 4-16 CPU, 8-32Gi RAM, GPU optional

Medium Resource:

• Sonarr/Radarr: 500m-2 CPU, 512Mi-2Gi RAM
• SABnzbd: 1-4 CPU, 1-4Gi RAM

Low Resource:

• Bazarr: 100m-500m CPU, 128Mi-512Mi RAM
• Overseerr: 100m-500m CPU, 256Mi-1Gi RAM

Storage Quotas

Dynamic PVCs sized appropriately:

• Configuration: 5-10Gi (databases, logs)
• Download buffers: 100Gi (temporary downloads)
• Transcode cache: 100Gi (Tdarr working space)

Maintenance

Regular Tasks

Weekly:

• Review failed downloads
• Check disk space usage
• Verify backup completion
• Update metadata (Kometa)

Monthly:

• Library maintenance (Plex)
• Database optimization (*arr apps)
• Review and cleanup old downloads
• Check for application updates (Renovate handles this)

Health Monitoring

Key Metrics:

• Plex stream count and transcoding sessions
• SABnzbd download queue and speed
• *arr indexer health and search failures
• Storage capacity and growth rate

Alerts:

• Download failures
• Indexer connectivity issues
• Storage capacity warnings
• Failed backup jobs

Troubleshooting

Common Issues

Plex can't see media files:

# Check PVC mount
kubectl exec -n media deployment/plex -- ls -la /media

# Verify permissions
kubectl exec -n media deployment/plex -- ls -ld /media/movies /media/tv

# Check Ceph health
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph status

Downloads not moving to library:

# Check SABnzbd logs
kubectl logs -n media deployment/sabnzbd --tail=100

# Verify shared storage access
kubectl exec -n media deployment/sabnzbd -- ls -la /media/downloads/usenet

# Check Sonarr/Radarr import
kubectl logs -n media deployment/sonarr --tail=100 | grep -i import

Slow transcoding:

# Verify GPU allocation
kubectl describe pod -n media -l app.kubernetes.io/name=plex | grep -A5 "Limits\|Requests"

# Check GPU utilization (on node)
intel_gpu_top

# Review transcode logs
kubectl logs -n media deployment/plex | grep -i transcode

Storage full:

# Check PVC usage
kubectl get pvc -n media

# Check storage usage in pod
kubectl exec -n media deployment/plex -- df -h | grep media

# Check Ceph cluster capacity
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph df

Best Practices

Storage Organization

Directory Structure:

/media/
├── downloads/
│   ├── usenet/          # SABnzbd downloads
│   └── complete/        # Completed downloads
├── movies/              # Radarr managed
│   ├── 4k/             # UHD content
│   └── 1080p/          # HD content
├── tv/                  # Sonarr managed
│   ├── 4k/
│   └── 1080p/
├── music/
└── books/

Quality Profiles

• Use separate instances for 4K content (Sonarr UHD, Radarr UHD)
• Configure appropriate quality cutoffs
• Enable upgrades for better releases
• Set size limits to prevent excessive downloads

Download Management

• Configure category-based post-processing in SABnzbd
• Use download client categories in *arr apps
• Enable completed download handling
• Set appropriate retention for download history

Performance Optimization

• Use hardware transcoding (Intel Quick Sync)
• Pre-optimize media with Tdarr (h265/HEVC)
• Adjust Plex transcoder quality settings
• Enable Plex optimize versions for common devices

Security Considerations

Access Control

• Internal Network Only: Media apps exposed only via internal ingress
• Authentication Required: All apps require login
• Plex Managed Auth: User access controlled via Plex sharing
• Overseerr Integration: Request permissions via Plex accounts

API Keys

• All API keys stored in Kubernetes secrets
• External Secrets integration with Infisical
• Regular key rotation via automation
• Least privilege access between services

Future Improvements

Under Consideration

• Jellyfin: Alternative media server for comparison
• Readarr: Book management automation
• Lidarr: Music management automation

References

• Media Storage: kubernetes/apps/media/storage/
• Plex: kubernetes/apps/media/plex/
• Sonarr: kubernetes/apps/media/sonarr/
• Radarr: kubernetes/apps/media/radarr/
• SABnzbd: kubernetes/apps/media/sabnzbd/
• Storage Architecture: docs/src/architecture/storage.md

Network Applications

This document covers the applications running in the network namespace: the cluster's ingress, DNS, notification, and CNI plumbing.

All HTTP(S) ingress is handled by Envoy Gateway (Gateway API) — there is no Ingress controller. A single GatewayClass named envoy backs three Gateway resources, split by where the traffic comes from and where it is allowed to go:

Each Gateway gets its VIP from Cilium LB-IPAM via an lbipam.cilium.io/ips annotation on the Gateway's managed Envoy LoadBalancer Service. TLS is terminated with per-domain wildcard certificates issued by cert-manager; the HTTPS listener is auto-selected by request hostname (apps attach with an HTTPRoute and omit sectionName), and the :80 listener carries only a global HTTP→HTTPS 301 redirect.

For the underlying DNS design (private domains, OPNsense forwarding, public records), see Network Architecture.

Traffic Flow

graph LR
    subgraph Internet
        WAN[WAN clients]
        CF[Cloudflare]
    end
    subgraph LAN
        LANC[LAN clients]
        OPN[OPNsense resolver]
    end
    subgraph "network namespace"
        CFD[cloudflared]
        KGW[k8s-gateway DNS]
        GEXT[external Gateway<br/>10.100.0.22]
        GINT[internal Gateway<br/>10.100.0.20]
        GMED[media Gateway<br/>10.100.0.31]
    end
    APPS[Application HTTPRoutes]

    WAN -->|public hostname| CF -->|Tunnel| CFD --> GEXT --> APPS
    WAN -->|media hostname, DDNS| GMED --> APPS
    LANC -->|private hostname| OPN --> KGW
    LANC --> GINT --> APPS
    GINT --> APPS

Ingress and Gateways

Envoy Gateway

Namespace: network Type: Helm Release + Gateway API resources Purpose: Gateway API controller and the cluster's three Gateways

Configuration: kubernetes/apps/network/envoy-gateway/

• gateway/gatewayclass.yaml — the envoy GatewayClass, parameterized by a class-wide EnvoyProxy.
• gateway/gateway-internal.yaml — internal Gateway (VIP 10.100.0.20). Per-domain HTTPS listeners; :80 restricted to from: Same so only the redirect route attaches.
• gateway/gateway-external.yaml — external Gateway (VIP 10.100.0.22), labelled external-dns: "true". external-dns derives the public record target from the Gateway and points it at the Cloudflare Tunnel CNAME. Carries both wildcard and apex listeners.
• gateway/gateway-media.yaml — media Gateway (VIP 10.100.0.31). Uses a dedicated EnvoyProxy running Envoy as a DaemonSet with externalTrafficPolicy: Local (plus a control-plane toleration) so the real client IP survives to Plex. Reached WAN-direct via an unproxied DDNS record, not the tunnel.
• gateway/httproutes-redirect.yaml — global HTTP→HTTPS 301 redirect routes on each Gateway's :80 listener.

# Gateways and their assigned VIPs
kubectl -n network get gateways

# Routes attached to a Gateway
kubectl get httproutes -A

# Envoy Gateway controller
kubectl -n network get pods -l app.kubernetes.io/name=gateway-helm

cloudflared

Namespace: network Type: Helm Release (app-template) Purpose: Cloudflare Tunnel — secure public ingress to the external Gateway

Configuration: kubernetes/apps/network/external/cloudflared/

cloudflared establishes an outbound tunnel to Cloudflare, so no inbound ports are opened on the firewall. Public requests arrive at Cloudflare and are forwarded down the tunnel to the in-cluster Envoy external Service. Each ingress rule sets originServerName to a hostname matching the correct per-domain HTTPS listener (Envoy selects its listener by TLS SNI; the Host header still selects the HTTPRoute).

kubectl -n network logs -l app.kubernetes.io/name=cloudflared -f

external-dns

Namespace: network Type: Helm Release Purpose: Syncs public DNS records to Cloudflare from Gateway/HTTPRoute resources

Configuration: kubernetes/apps/network/external/external-dns/

A single external-dns instance watches Gateways labelled external-dns: "true" (the external and media Gateways) and their attached HTTPRoutes, publishing the corresponding records to Cloudflare. The record target is taken from the Gateway annotation — the Cloudflare Tunnel CNAME for external, and an unproxied DDNS record for media.

k8s-gateway

Namespace: network Type: Helm Release Purpose: Authoritative internal DNS for the cluster's private domains

Configuration: kubernetes/apps/network/internal/k8s-gateway/

k8s-gateway is a CoreDNS-based server that answers for the private SECRET_DOMAIN* domains, returning the internal Gateway VIP (10.100.0.20) for any hostname backed by an HTTPRoute or Service. It runs as a LoadBalancer Service (Cilium LB-IPAM VIP) on port 53. The OPNsense resolver forwards those domains to it, so LAN clients resolve cluster hostnames to the internal Gateway. See Network Architecture for the resolver chain.

home-reverse-proxy

Namespace: network Type: HTTPRoutes + Envoy Backend resources Purpose: Give non-cluster LAN hosts cluster hostnames and TLS

Configuration: kubernetes/apps/network/internal/home-reverse-proxy/

A set of HTTPRoutes (with ExternalName Services and Envoy Backend resources) that proxy to physical LAN devices such as Proxmox (circe.manor) and the OPNsense firewall. This lets those off-cluster services be reached through the internal Gateway with a friendly cluster hostname and a valid wildcard certificate, including for self-signed HTTPS upstreams (insecureSkipVerify).

Notifications and Mail

ntfy

Namespace: network Type: Helm Release (app-template) Purpose: Self-hosted push-notification server

Configuration: kubernetes/apps/network/ntfy/

Self-hosted ntfy (binwiederhier/ntfy) used as the cluster's notification sink. Both Gatus (uptime) and Alertmanager (via alertmanager-ntfy) publish to it. See Observability Applications for the alert-routing detail.

smtp-relay

Namespace: network Type: Helm Release (app-template) Purpose: Outbound SMTP relay for cluster apps

Configuration: kubernetes/apps/network/smtp-relay/

A maddy-based SMTP relay providing a single in-cluster mail submission endpoint, so applications that send email use one shared, authenticated upstream rather than each holding provider credentials.

CNI

Multus

Namespace: network Type: Helm Release + NetworkAttachmentDefinition Purpose: Attach additional pod network interfaces beyond the primary Cilium network

Configuration: kubernetes/apps/network/multus/

Multus is a meta-CNI that lets selected pods receive a second interface in addition to the primary Cilium network. The multus-iot NetworkAttachmentDefinition uses macvlan over a host interface with DHCP IPAM, placing those pods directly on an IoT/legacy L2 segment — used by workloads that must talk to devices outside the cluster's pod network.

kubectl -n network get net-attach-def
kubectl -n network get pods -l app.kubernetes.io/name=multus

References

• Envoy Gateway: kubernetes/apps/network/envoy-gateway/
• cloudflared / external-dns: kubernetes/apps/network/external/
• k8s-gateway / home-reverse-proxy: kubernetes/apps/network/internal/
• Network Architecture: docs/src/architecture/network.md

Storage Applications

This document covers the storage-related applications and services running in the cluster. All persistent storage is backed by the external Ceph cluster on the Proxmox hosts; CephFS uses the cephfs_data, cephfs_media, and cephfs_backups pools.

Storage Stack Overview

graph TD
    subgraph External Infrastructure
        CEPH[Proxmox Ceph Cluster]
    end

    subgraph Kubernetes Storage Layer
        ROOK[Rook Ceph Operator]
        CSI[Ceph CSI Drivers]
        VOLSYNC[VolSync]
    end

    subgraph Storage Consumers
        APPS[Applications]
        BACKUPS[Backup Repositories]
    end

    CEPH --> ROOK
    ROOK --> CSI
    CSI --> APPS
    VOLSYNC --> BACKUPS
    CSI --> VOLSYNC

Core Components

Rook Ceph Operator

Namespace: rook-ceph Type: Helm Release Purpose: Manages connection to external Ceph cluster and provides CSI drivers

The Rook operator is the bridge between Kubernetes and the external Ceph cluster. It:

• Manages CSI driver deployments
• Maintains connection to Ceph monitors
• Handles authentication and secrets
• Provides CephFS filesystem access

Configuration: kubernetes/apps/rook-ceph/rook-ceph-operator/app/helmrelease.yaml

Current Setup:

• CephFS Driver: Enabled ✅
• RBD Driver: Enabled ✅
• Connection Mode: External cluster
• Network: Public network 10.150.0.0/24

Key Resources:

# Check operator status
kubectl -n rook-ceph get pods -l app=rook-ceph-operator

# View operator logs
kubectl -n rook-ceph logs -l app=rook-ceph-operator -f

# Check CephCluster resource
kubectl -n rook-ceph get cephcluster

Rook Ceph Cluster Configuration

Namespace: rook-ceph Type: CephCluster Custom Resource Purpose: Defines external Ceph cluster connection

Configuration: kubernetes/apps/rook-ceph/rook-ceph-cluster/app/cluster-external.yaml

This resource tells Rook how to connect to the external Ceph cluster:

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  external:
    enable: true
  dataDirHostPath: /var/lib/rook
  cephVersion:
    image: quay.io/ceph/ceph:v18

Monitor Configuration: Defined in ConfigMap rook-ceph-mon-endpoints

• Contains Ceph monitor IP addresses
• Critical for cluster connectivity
• Automatically referenced by CSI drivers

Authentication: Stored in Secret rook-ceph-mon

• Contains client.kubernetes Ceph credentials
• Encrypted with SOPS
• Referenced by all CSI operations

Ceph CSI Drivers

Namespace: rook-ceph Type: DaemonSet (nodes) + Deployment (provisioner) Purpose: Enable Kubernetes to mount RBD and CephFS volumes

Both the RBD and CephFS CSI drivers are deployed:

RBD (block, RWO):

1. csi-rbdplugin (DaemonSet) — maps/unmaps RBD images on each node
2. csi-rbdplugin-provisioner (Deployment) — creates/deletes RBD images, snapshots, expansion

CephFS (shared filesystem, RWX):

1. csi-cephfsplugin (DaemonSet) — mounts CephFS volumes on each node
2. csi-cephfsplugin-provisioner (Deployment) — creates/deletes CephFS subvolumes, expansion

Monitoring:

# Check CSI pods
kubectl -n rook-ceph get pods -l app=csi-cephfsplugin

# View CSI driver logs
kubectl -n rook-ceph logs -l app=csi-cephfsplugin -c csi-cephfsplugin

# Check provisioner
kubectl -n rook-ceph get pods -l app=csi-cephfsplugin-provisioner

Storage Classes

Configuration: kubernetes/apps/rook-ceph/rook-ceph-cluster/app/storageclasses.yaml

Live classes: ceph-rbd (default), cephfs-shared, cephfs-static, cephfs-backups.

ceph-rbd (Default)

RBD block storage; the default class for any PVC without an explicit storageClassName.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ceph-rbd
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
  clusterID: rook-ceph
  pool: rook-pvc-pool
  imageFeatures: layering
  csi.storage.k8s.io/fstype: ext4
allowVolumeExpansion: true
reclaimPolicy: Delete

Usage: Default for all PVCs without an explicit storageClassName (RWO block)

cephfs-shared

Dynamically-provisioned shared filesystem for RWX workloads (cephfs_data pool).

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: cephfs-shared
provisioner: rook-ceph.cephfs.csi.ceph.com
parameters:
  clusterID: rook-ceph
  fsName: cephfs
  pool: cephfs_data
  subvolumeGroup: csi
  mounter: kernel
allowVolumeExpansion: true
reclaimPolicy: Delete

Usage: Set storageClassName: cephfs-shared when an app needs ReadWriteMany

cephfs-static

For mounting pre-existing CephFS directories:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: cephfs-static
provisioner: rook-ceph.cephfs.csi.ceph.com
# Used with manually created PVs pointing to existing paths

Usage: Requires manual PV creation, see examples below

cephfs-backups

CephFS class on the cephfs_backups pool with Retain reclaim policy, used for VolSync Restic repositories.

VolSync

Namespace: storage Type: Helm Release Purpose: Backup and recovery for Persistent Volume Claims

VolSync provides automated backup of all stateful applications using Restic.

Configuration: kubernetes/apps/storage/volsync/app/helmrelease.yaml

Backup Repository: CephFS-backed PVC (cephfs_backups pool)

• Location: volsync-cephfs-pvc
• Path: /repository/{APP}/ for each application
• Restore/cache PVCs: default to ceph-rbd

How It Works:

1. Applications create ReplicationSource resources
2. VolSync creates backup pods with mover containers
3. Mover mounts both application PVC and repository PVC
4. Restic backs up data to repository
5. Retention policies keep configured snapshot count

Backup Pattern:

apiVersion: volsync.backube/v1alpha1
kind: ReplicationSource
metadata:
  name: my-app
  namespace: my-namespace
spec:
  sourcePVC: my-app-data
  trigger:
    schedule: "0 * * * *" # Hourly
  restic:
    repository: my-app-restic-secret
    retain:
      hourly: 24
      daily: 7
      weekly: 4

Common Operations:

# Manual backup trigger
task volsync:snapshot NS=<namespace> APP=<app>

# List snapshots
task volsync:run NS=<namespace> REPO=<app> -- snapshots

# Unlock repository (if locked)
task volsync:unlock-local NS=<namespace> APP=<app>

# Restore to new PVC
task volsync:restore NS=<namespace> APP=<app>

Repository PVC Configuration: kubernetes/apps/storage/volsync/app/volsync-cephfs-pv.yaml

Static PV Examples

Media Storage

Large media library using dedicated cephfs_media pool:

Location: kubernetes/apps/media/storage/app/media-cephfs-pv.yaml

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: media-cephfs-pv
spec:
  capacity:
    storage: 100Ti
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: cephfs-static
  csi:
    driver: rook-ceph.cephfs.csi.ceph.com
    nodeStageSecretRef:
      name: rook-csi-cephfs-static
      namespace: rook-ceph
    volumeAttributes:
      clusterID: rook-ceph
      fsName: cephfs
      staticVolume: "true"
      rootPath: /media
      pool: cephfs_media
      mounter: kernel
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: media-cephfs-pvc
  namespace: media
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 100Ti
  storageClassName: cephfs-static
  volumeName: media-cephfs-pv

Usage: Mounted by Plex, Sonarr, Radarr, etc. for media library access

Minio Object Storage

Minio data stored on CephFS using cephfs_data pool:

Location: kubernetes/apps/storage/minio/app/minio-cephfs-pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: minio-cephfs-pv
spec:
  capacity:
    storage: 10Ti
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: cephfs-static
  csi:
    driver: rook-ceph.cephfs.csi.ceph.com
    nodeStageSecretRef:
      name: rook-csi-cephfs-static
      namespace: rook-ceph
    volumeAttributes:
      clusterID: rook-ceph
      fsName: cephfs
      staticVolume: "true"
      rootPath: /minio
      pool: cephfs_data
      mounter: kernel

Paperless-ngx Document Storage

Document management system storage using cephfs_data pool:

Location: kubernetes/apps/selfhosted/paperless-ngx/app/paperless-cephfs-pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: paperless-cephfs-pv
spec:
  capacity:
    storage: 5Ti
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: cephfs-static
  csi:
    driver: rook-ceph.cephfs.csi.ceph.com
    nodeStageSecretRef:
      name: rook-csi-cephfs-static
      namespace: rook-ceph
    volumeAttributes:
      clusterID: rook-ceph
      fsName: cephfs
      staticVolume: "true"
      rootPath: /paperless
      pool: cephfs_data
      mounter: kernel

Storage Operations

Creating a New Static PV

Step 1: Create directory in CephFS (on Proxmox Ceph node)

# SSH to a Proxmox node and mount CephFS
ceph-fuse /mnt/cephfs
mkdir -p /mnt/cephfs/my-app
chmod 777 /mnt/cephfs/my-app  # Or appropriate permissions
fusermount -u /mnt/cephfs

Step 2: Create PV manifest

apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-app-cephfs-pv
spec:
  capacity:
    storage: 1Ti
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: cephfs-static
  csi:
    driver: rook-ceph.cephfs.csi.ceph.com
    nodeStageSecretRef:
      name: rook-csi-cephfs-static
      namespace: rook-ceph
    volumeAttributes:
      clusterID: rook-ceph
      fsName: cephfs
      staticVolume: "true"
      rootPath: /my-app
      pool: cephfs_data # Or cephfs_media, cephfs_backups
      mounter: kernel

Step 3: Create PVC manifest

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-app-cephfs-pvc
  namespace: my-namespace
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Ti
  storageClassName: cephfs-static
  volumeName: my-app-cephfs-pv

Step 4: Apply and verify

kubectl apply -f pv.yaml
kubectl apply -f pvc.yaml
kubectl get pv my-app-cephfs-pv
kubectl get pvc -n my-namespace my-app-cephfs-pvc

Expanding a PVC

CephFS supports online volume expansion:

# Edit PVC to increase size
kubectl patch pvc my-pvc -n my-namespace -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'

# Verify expansion
kubectl get pvc -n my-namespace my-pvc -w

Note: Size can only increase, not decrease

Troubleshooting Mount Issues

PVC stuck in Pending:

# Check PVC events
kubectl describe pvc -n <namespace> <pvc-name>

# Check CSI driver logs
kubectl -n rook-ceph logs -l app=csi-cephfsplugin -c csi-cephfsplugin --tail=100

# Verify storage class exists
kubectl get sc cephfs-shared

Pod can't mount volume:

# Check pod events
kubectl describe pod -n <namespace> <pod-name>

# Verify Ceph cluster connectivity
kubectl -n rook-ceph get cephcluster

# Check Ceph health
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph status

# Verify CephFS is available
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph fs status

Slow I/O performance:

# Check MDS performance
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph fs status

# Check OSD performance
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd perf

# Identify slow operations
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph health detail

Monitoring and Alerts

Key Metrics

Monitor these via Prometheus/Grafana:

1. Storage Capacity

   • Ceph cluster utilization
   • Individual PVC usage
   • Growth trends

2. Performance

   • CSI operation latency
   • MDS cache hit ratio
   • OSD I/O rates

3. Reliability

   • VolSync backup success rate
   • Ceph health status
   • CSI driver availability

Useful Queries

Check all PVCs by size:

kubectl get pvc -A -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,SIZE:.spec.resources.requests.storage,STORAGECLASS:.spec.storageClassName --sort-by=.spec.resources.requests.storage

Group PVCs by storage class:

kubectl get pvc -A -o json | jq -r '.items[] | "\(.spec.storageClassName)\t\(.metadata.namespace)/\(.metadata.name)"' | sort

Check Ceph cluster capacity:

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph df

Monitor VolSync backups:

# Check all ReplicationSources
kubectl get replicationsource -A

# Check specific backup status
kubectl get replicationsource -n <namespace> <app> -o jsonpath='{.status.lastSyncTime}'

Backup and Recovery

VolSync Backup Workflow

1. Application creates ReplicationSource
2. VolSync creates backup pod (every hour by default)
3. Restic backs up PVC to repository
4. Snapshots retained per retention policy
5. Status updated in ReplicationSource

Restore Procedures

Restore to original PVC:

# Scale down application
kubectl scale deployment -n <namespace> <app> --replicas=0

# Run restore
task volsync:restore NS=<namespace> APP=<app>

# Scale up application
kubectl scale deployment -n <namespace> <app> --replicas=1

Restore to new PVC:

1. Create ReplicationDestination pointing to new PVC
2. VolSync will restore data from repository
3. Update application to use new PVC
4. Verify data integrity

Disaster Recovery

Complete cluster rebuild:

1. Deploy new Kubernetes cluster
2. Install Rook with same external Ceph connection
3. Recreate storage classes
4. Deploy VolSync
5. Restore all applications from backups

CephFS corruption:

1. Check Ceph health and repair if possible
2. If unrecoverable, restore from VolSync backups
3. VolSync repository is on CephFS, so ensure repository is intact
4. Consider external backup of VolSync repository

Security Considerations

Ceph Authentication

• Client Key: client.kubernetes Ceph user
• Permissions: Limited to CephFS pools only
• Storage: SOPS-encrypted in rook-ceph-mon secret
• Rotation: Should be rotated periodically

PVC Access Control

• Namespace Isolation: PVCs are namespace-scoped
• RBAC: Control who can create/delete PVCs
• Pod Security: Pods must have appropriate security context
• Network Policies: Limit which pods can access storage

Backup Security

• VolSync Repository: Protected by Kubernetes RBAC
• Restic Encryption: Repository encryption with per-app keys
• Snapshot Access: Controlled via ReplicationSource ownership

Pools

• rook-pvc-pool: RBD block pool backing ceph-rbd (databases and single-pod apps)
• cephfs_data: General application data (minio, paperless, etc.)
• cephfs_media: Large media files (plex, *arr, etc.)
• cephfs_backups: VolSync backup repositories

References

• Rook Operator: kubernetes/apps/rook-ceph/rook-ceph-operator/
• Cluster Config: kubernetes/apps/rook-ceph/rook-ceph-cluster/
• Storage Classes: kubernetes/apps/rook-ceph/rook-ceph-cluster/app/storageclasses.yaml
• VolSync: kubernetes/apps/storage/volsync/
• Architecture: docs/src/architecture/storage.md

Observability Applications

This document covers the applications running in the observability namespace: metrics, logging, uptime monitoring, alert routing, and resource right-sizing.

Metrics are collected by Prometheus and surfaced in Grafana; container logs are shipped to VictoriaLogs; uptime is tracked by Gatus. Alerts fan out to the self-hosted ntfy server, with critical alerts additionally paged to Pushover.

Observability Flow

graph TD
    subgraph Metrics
        PROM[Prometheus]
        AM[Alertmanager]
        EXP[Exporters / blackbox]
    end
    subgraph Logs
        FB[fluent-bit]
        VL[VictoriaLogs]
    end
    subgraph Uptime
        GAT[Gatus]
    end
    subgraph Visualize
        GRAF[Grafana]
    end
    subgraph Notify
        NTFY[ntfy]
        PO[Pushover]
    end

    EXP --> PROM
    PROM --> AM
    PROM --> GRAF
    AM -->|all alerts| NTFY
    AM -->|severity=critical| PO
    FB --> VL --> GRAF
    GAT --> NTFY

Metrics and Visualization

kube-prometheus-stack

Namespace: observability Type: Helm Release Purpose: Core metrics platform — Prometheus, Alertmanager, node-exporter, kube-state-metrics

Configuration: kubernetes/apps/observability/kube-prometheus-stack/

The bundle deploys Prometheus (scraping and recording/alerting rules), Alertmanager (alert routing), node-exporter (per-node host metrics), and kube-state-metrics (Kubernetes object state). Additional scrape targets and Alertmanager routing live alongside the HelmRelease in scrapeconfig.yaml and alertmanagerconfig.yaml.

Alertmanager routing: the default receiver is ntfy; severity=critical alerts route to Pushover (a loud phone push) and continue through to ntfy as well, so criticals are still delivered if Pushover is rate-limited.

kubectl -n observability get prometheus,alertmanager
kubectl -n observability get pods -l app.kubernetes.io/name=prometheus

Grafana

Namespace: observability Type: Helm Release Purpose: Visualization layer for metrics and logs

Configuration: kubernetes/apps/observability/grafana/

Grafana is the dashboard front-end. Its provisioned datasources are Prometheus (prometheus-operated), VictoriaLogs (victoria-logs-server), and Alertmanager. Dashboards are provisioned declaratively, including via ConfigMaps discovered by the Grafana sidecar.

kubectl -n observability get pods -l app.kubernetes.io/name=grafana

Logging

VictoriaLogs

Namespace: observability Type: Helm Release Purpose: Log storage and query backend

Configuration: kubernetes/apps/observability/victoria-logs/

VictoriaLogs is the cluster's log store, exposed at victoria-logs-server.observability.svc.cluster.local:9428 and queried from Grafana via the VictoriaLogs datasource.

fluent-bit

Namespace: observability Type: Helm Release (DaemonSet) Purpose: Ships container logs into VictoriaLogs

Configuration: kubernetes/apps/observability/fluent-bit/

fluent-bit runs on every node, tails container logs, enriches them with Kubernetes metadata, and forwards them over HTTP to the VictoriaLogs server.

kubectl -n observability get pods -l app.kubernetes.io/name=fluent-bit

Uptime and Probing

Gatus

Namespace: observability Type: Helm Release (gatus-sidecar) Purpose: Uptime / status page

Configuration: kubernetes/apps/observability/gatus/

Gatus uses the gatus-sidecar model: endpoints are discovered automatically from HTTPRoute resources, with base probe configuration inherited from annotations on the parent Gateway (internal routes become DNS-only "guarded" checks; external/media routes become real HTTP 200 probes resolved via public DNS). Gatus alerts are delivered to ntfy. See the gatus-monitoring concept for how to add or tune a check.

kubectl -n observability get pods -l app.kubernetes.io/name=gatus

blackbox-exporter

Namespace: observability Type: Helm Release Purpose: Black-box probing of endpoints

Configuration: kubernetes/apps/observability/blackbox-exporter/

blackbox-exporter performs synthetic probes (HTTP, TCP, ICMP) against targets defined in probes.yaml, exposing the results to Prometheus.

Exporters

Namespace: observability Type: Helm Releases / DaemonSets Purpose: Assorted Prometheus exporters

Configuration: kubernetes/apps/observability/exporters/

Additional exporters feeding Prometheus:

• dcgm-exporter — NVIDIA GPU metrics.
• nut-exporter — UPS metrics from NUT.
• snmp-exporter — SNMP metrics from network devices.

Alerting

alertmanager-ntfy

Namespace: observability Type: Helm Release Purpose: Bridges Alertmanager notifications to the self-hosted ntfy server

Configuration: kubernetes/apps/observability/alertmanager-ntfy/

alertmanager-ntfy receives Alertmanager webhook notifications and republishes them as ntfy messages, so Prometheus alerts land on the self-hosted ntfy server alongside Gatus uptime notifications.

silence-operator

Namespace: observability Type: Helm Release + Silence CRs Purpose: Declarative, GitOps-managed Alertmanager silences

Configuration: kubernetes/apps/observability/silence-operator/

The silence-operator reconciles Silence custom resources (silences/silences.yaml) into Alertmanager, so silences are version-controlled in Git rather than clicked into the UI.

Resource Right-Sizing

VPA and Goldilocks

Namespace: observability Type: Helm Releases Purpose: Vertical Pod Autoscaler with the Goldilocks recommendation dashboard

Configuration: kubernetes/apps/observability/vpa/, kubernetes/apps/observability/goldilocks/

The Vertical Pod Autoscaler computes recommended CPU/memory requests for workloads; Goldilocks presents those recommendations in a dashboard to guide right-sizing of resource requests.

kubectl -n observability get vpa -A

Status Reporting

kromgo

Namespace: observability Type: Helm Release Purpose: Exposes selected cluster metrics as safe public endpoints

Configuration: kubernetes/apps/observability/kromgo/

kromgo serves a curated allow-list of Prometheus queries as public, shareable endpoints — used to power the status badges in the repository README without exposing Prometheus itself.

peanut

Namespace: observability Type: Helm Release Purpose: NUT (UPS) monitoring dashboard

Configuration: kubernetes/apps/observability/peanut/

peanut provides a web dashboard for the UPS hardware monitored via NUT (battery charge, load, runtime), complementing the Prometheus nut-exporter metrics.

References

• kube-prometheus-stack: kubernetes/apps/observability/kube-prometheus-stack/
• Grafana: kubernetes/apps/observability/grafana/
• VictoriaLogs / fluent-bit: kubernetes/apps/observability/victoria-logs/, kubernetes/apps/observability/fluent-bit/
• Gatus: kubernetes/apps/observability/gatus/
• ntfy (notification sink): kubernetes/apps/network/ntfy/

Installation Guide

Prerequisites

graph TD
    subgraph Hardware
        CP[Control Plane Nodes]
        GPU[GPU Worker Node]
        Worker[Worker Nodes]
    end

    subgraph Software
        OS[Operating System]
        Tools[Required Tools]
        Network[Network Setup]
    end

    subgraph Configuration
        Git[Git Repository]
        Secrets[SOPS Setup]
        Certs[Certificates]
    end

Hardware Requirements

Control Plane Nodes (3x)

• CPU: 4 cores per node
• RAM: 16GB per node
• Role: Cluster control plane

GPU Worker Node (1x)

• CPU: 16 cores
• RAM: 128GB
• GPU: 2x NVIDIA Tesla P100
• Role: GPU-accelerated workloads

Worker Nodes (2x)

• CPU: 16 cores per node
• RAM: 128GB per node
• Role: General workloads

Software Prerequisites

1. Operating System

   • Linux distribution
   • Updated system packages
   • Required kernel modules
   • NVIDIA drivers (for GPU node)

2. Required Tools

   • kubectl
   • flux
   • SOPS
   • age/gpg
   • task

Initial Setup

1. Repository Setup

# Clone the repository
git clone https://github.com/username/dapper-cluster.git
cd dapper-cluster

# Create configuration
cp config.sample.yaml config.yaml

2. Configuration

graph LR
    Config[Configuration] --> Secrets[Secrets Management]
    Config --> Network[Network Settings]
    Config --> Storage[Storage Setup]
    Secrets --> SOPS[SOPS Encryption]
    Network --> DNS[DNS Setup]
    Storage --> CSI[CSI Drivers]

Edit Configuration

cluster:
  name: dapper-cluster
  domain: example.com

network:
  cidr: 10.0.0.0/16

storage:
  ceph:
    # External Ceph cluster connection
    # Configured via Rook operator after bootstrap
    monitors: [] # Set during Rook deployment

3. Secrets Management

• Generate age key
• Configure SOPS
• Encrypt sensitive files

4. Bootstrap Process

graph TD
    Start[Start Installation] --> CP[Bootstrap Control Plane]
    CP --> Workers[Join Worker Nodes]
    Workers --> GPU[Configure GPU Node]
    GPU --> Flux[Install Flux]
    Flux --> Apps[Deploy Apps]

Bootstrap Commands

# Bootstrap Talos, then deploy apps (installs Flux)
task bootstrap:talos
task bootstrap:apps

# Force Flux to reconcile from Git
task reconcile

# Verify GPU support
kubectl get nodes -o wide
nvidia-smi # on GPU node

Post-Installation

1. Verify Components

• Check control plane health
• Verify worker node status
• Test GPU functionality
• Check Rook Ceph connection
• Verify storage classes
• Verify network connectivity

2. Deploy Applications

• Deploy core services
• Configure monitoring
• Setup backup systems
• Deploy GPU-enabled workloads

3. Security Setup

• Configure network policies
• Setup certificate management
• Enable monitoring and alerts
• Secure GPU access

Troubleshooting

Common installation issues and solutions:

1. Control Plane Issues

   • Verify etcd cluster health
   • Check control plane components
   • Review system logs

2. Worker Node Issues

   • Verify node join process
   • Check kubelet status
   • Review node logs

3. GPU Node Issues

   • Verify NVIDIA driver installation
   • Check NVIDIA container runtime
   • Validate GPU visibility in cluster

4. Storage Issues

   • Verify Ceph cluster connectivity
   • Check Rook operator status
   • Verify storage class configuration
   • Review CephCluster resource health
   • Check PV/PVC status

5. Network Problems

   • Check DNS resolution
   • Verify network policies
   • Review ingress configuration

Maintenance

Regular Tasks

1. System updates
2. Certificate renewal
3. Backup verification
4. Security audits
5. GPU driver updates

Health Checks

• Component status
• Resource usage
• Storage capacity
• Network connectivity
• GPU health

Next Steps

After successful installation:

1. Review Architecture Overview
2. Configure Storage
3. Setup Network
4. Deploy Applications

Maintenance Guide

Maintenance Overview

graph TD
    subgraph Regular Tasks
        Updates[System Updates]
        Backups[Backup Tasks]
        Monitoring[Health Checks]
    end

    subgraph Periodic Tasks
        Audit[Security Audits]
        Cleanup[Resource Cleanup]
        Review[Config Review]
    end

    Updates --> Verify[Verification]
    Backups --> Test[Backup Testing]
    Monitoring --> Alert[Alert Response]

Regular Maintenance

Daily Tasks

1. Monitor system health
   • Check cluster status
   • Review resource usage
   • Verify backup completion
   • Check alert status

Weekly Tasks

1. Review system logs
2. Check storage usage
3. Verify backup integrity
4. Update documentation

Monthly Tasks

1. Security updates
2. Certificate rotation
3. Resource optimization
4. Performance review

Update Procedures

Flux Updates

graph LR
    PR[Pull Request] --> Review[Review Changes]
    Review --> Test[Test Environment]
    Test --> Deploy[Deploy to Prod]
    Deploy --> Monitor[Monitor Status]

Application Updates

1. Review release notes
2. Test in staging if available
3. Update flux manifests
4. Monitor deployment
5. Verify functionality

Backup Management

Backup Strategy

graph TD
    Apps[Applications] --> Data[Data Backup]
    Config[Configurations] --> Git[Git Repository]
    Secrets[Secrets] --> Vault[Secret Storage]

    Data --> Verify[Verification]
    Git --> Verify
    Vault --> Verify

Backup Verification

• Regular restore testing
• Data integrity checks
• Recovery time objectives
• Backup retention policy

Resource Management

Cleanup Procedures

1. Remove unused resources

   • Orphaned PVCs
   • Completed jobs
   • Old backups
   • Unused configs

2. Storage optimization

   • Compress old logs
   • Archive unused data
   • Clean container cache

Monitoring and Alerts

Key Metrics

• Node health
• Pod status
• Resource usage
• Storage capacity
• Network performance

Alert Response

1. Acknowledge alert
2. Assess impact
3. Investigate root cause
4. Apply fix
5. Document resolution

Security Maintenance

Regular Tasks

graph TD
    Audit[Security Audit] --> Review[Review Findings]
    Review --> Update[Update Policies]
    Update --> Test[Test Changes]
    Test --> Document[Document Changes]

Security Checklist

• Review network policies
• Check certificate expiration
• Audit access controls
• Review secret rotation
• Scan for vulnerabilities

Troubleshooting Guide

Common Issues

1. Node Problems

   • Check node status
   • Review system logs
   • Verify resource usage
   • Check connectivity

2. Storage Issues

   • Check Ceph cluster health
   • Verify CephFS status
   • Monitor storage capacity
   • Review OSD performance
   • Check MDS responsiveness
   • Verify PVC mount status

3. Network Problems

   • Check DNS resolution
   • Verify network policies
   • Review ingress status
   • Test connectivity

Recovery Procedures

1. Node Recovery

# Check node status
kubectl get nodes

# Drain node for maintenance
kubectl drain node-name

# Perform maintenance
# ...

# Uncordon node
kubectl uncordon node-name

2. Storage Recovery

# Check Ceph cluster health
kubectl -n rook-ceph get cephcluster
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph status

# Check PV status
kubectl get pv

# Check PVC status
kubectl get pvc -A

# Verify storage class
kubectl get sc

# Check CephFS status
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph fs status

# Check OSD status
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd tree

Documentation

Maintenance Logs

• Keep detailed records
• Document changes
• Track issues
• Update procedures

Review Process

1. Regular documentation review
2. Update procedures
3. Verify accuracy
4. Add new sections

Best Practices

1. Change Management

   • Use git workflow
   • Test changes
   • Document updates
   • Monitor results

2. Resource Management

   • Regular cleanup
   • Optimize usage
   • Monitor trends
   • Plan capacity

3. Security

   • Regular audits
   • Update policies
   • Monitor access
   • Review logs

Troubleshooting Guide

Diagnostic Workflow

graph TD
    Issue[Issue Detected] --> Triage[Triage]
    Triage --> Diagnose[Diagnose]
    Diagnose --> Fix[Apply Fix]
    Fix --> Verify[Verify]
    Verify --> Document[Document]

Common Issues

1. Cluster Health Issues

Node Problems

graph TD
    Node[Node Issue] --> Check[Check Status]
    Check --> |Healthy| Resources[Resource Issue]
    Check --> |Unhealthy| System[System Issue]
    Resources --> Memory[Memory]
    Resources --> CPU[CPU]
    Resources --> Disk[Disk]
    System --> Logs[Check Logs]
    System --> Network[Network]

Diagnosis Steps:

# Check node status
kubectl get nodes
kubectl describe node <node-name>

# Check system resources
kubectl top nodes
kubectl top pods --all-namespaces

# Check system logs
kubectl logs -n kube-system <pod-name>

2. Storage Issues

Volume Problems

graph LR
    PV[PV Issue] --> Status[Check Status]
    Status --> |Bound| Access[Access Issue]
    Status --> |Pending| Provision[Provisioning Issue]
    Status --> |Failed| Storage[Storage System]

Resolution Steps:

# Check PV/PVC status
kubectl get pv,pvc --all-namespaces

# Check storage class
kubectl get sc

# Check provisioner pods
kubectl get pods -n storage

3. Network Issues

Connectivity Problems

graph TD
    Net[Network Issue] --> DNS[DNS Check]
    Net --> Ingress[Ingress Check]
    Net --> Policy[Network Policy]

    DNS --> CoreDNS[CoreDNS Pods]
    Ingress --> Traefik[Traefik Logs]
    Policy --> Rules[Policy Rules]

Diagnostic Commands:

# Check DNS
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns

# Check ingress
kubectl get ingress --all-namespaces
kubectl describe ingress <ingress-name> -n <namespace>

4. Application Issues

Pod Problems

graph TD
    Pod[Pod Issue] --> Status[Check Status]
    Status --> |Pending| Schedule[Scheduling]
    Status --> |CrashLoop| Crash[Container Crash]
    Status --> |Error| Logs[Check Logs]

Troubleshooting Steps:

# Check pod status
kubectl get pods -n <namespace>
kubectl describe pod <pod-name> -n <namespace>

# Check logs
kubectl logs <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace> --previous

Flux Issues

GitOps Troubleshooting

graph TD
    Flux[Flux Issue] --> Source[Source Controller]
    Flux --> Kust[Kustomize Controller]
    Flux --> Helm[Helm Controller]

    Source --> Git[Git Repository]
    Kust --> Sync[Sync Status]
    Helm --> Release[Release Status]

Resolution Steps:

# Check Flux components
flux check

# Check sources
flux get sources git
flux get sources helm

# Check reconciliation
flux get kustomizations
flux get helmreleases

Performance Issues

Resource Constraints

graph LR
    Perf[Performance] --> CPU[CPU Usage]
    Perf --> Memory[Memory Usage]
    Perf --> IO[I/O Usage]

    CPU --> Limit[Resource Limits]
    Memory --> Constraint[Memory Constraints]
    IO --> Bottleneck[I/O Bottleneck]

Analysis Commands:

# Check resource usage
kubectl top pods -n <namespace>
kubectl top nodes

# Check resource quotas
kubectl get resourcequota -n <namespace>

Recovery Procedures

1. Node Recovery

1. Drain node
2. Perform maintenance
3. Uncordon node
4. Verify workloads

2. Storage Recovery

1. Backup data
2. Fix storage issues
3. Restore data
4. Verify access

3. Network Recovery

1. Check connectivity
2. Verify DNS
3. Test ingress
4. Update policies

Best Practices

1. Logging

• Maintain detailed logs
• Set appropriate retention
• Use structured logging
• Enable audit logging

2. Monitoring

• Set up alerts
• Monitor resources
• Track metrics
• Use dashboards

3. Documentation

• Document issues
• Record solutions
• Update procedures
• Share knowledge

Emergency Procedures

Critical Issues

1. Assess impact
2. Implement temporary fix
3. Plan permanent solution
4. Update documentation

Contact Information

• Maintain escalation paths
• Keep contact list updated
• Document response times
• Track incidents

SSO Cluster Access (kubectl via Authentik OIDC)

Daily kubectl access authenticates against Authentik (OIDC). Group membership in Authentik maps to cluster RBAC. This is layered on top of the existing x509 admin kubeconfig, which remains the untouchable break-glass path.

How it fits together

The kube-apiserver runs multiple authenticators at once — OIDC is additive. The apiserver validates the x509 admin cert entirely locally against the cluster CA, so it works even if Authentik is completely down. OIDC uses the structured Authentication Configuration (apiserver.config.k8s.io/v1): one jwt authenticator per Authentik issuer — Authentik gives every application its own issuer URL, so the kubectl CLI and Headlamp each get their own entry. JWKS is fetched lazily, so an unreachable Authentik degrades only OIDC token validation, never apiserver startup.

• Providers: Authentik Kubernetes OIDC (public/PKCE, client_id: kubectl) and Headlamp OIDC (confidential, client_id: headlamp).
• Issuers: https://sso.chelonianlabs.com/application/o/kubectl/ and …/o/headlamp/ (trailing slash required).
• apiserver config: kubernetes/bootstrap/talos/patches/controller/apiserver-oidc.yaml — the auth config is written via machine.files to /var/lib/kubernetes/authentication/ (must be a writable /var path — a machine.files entry under /etc/kubernetes makes Talos overlay that dir read-only and breaks the kubelet's PKI writes) and mounted into the apiserver pod. Adding another OIDC app = one more jwt: block. Applying this patch requires a per-node reboot (machine.files changes are not immediate-mode).
• RBAC: kubernetes/apps/kube-system/oidc-rbac/ binds oidc:kubernetes-admins → cluster-admin and oidc:kubernetes-viewers → view; both issuers map claims with the same oidc: prefix, so bindings apply to both.
• Headlamp: https://headlamp.chelonianlabs.com — same login, per-user RBAC via token pass-through.

Access tiers

Grant access by adding a user to the group in Authentik; revoke by removing them.

Client setup (one-time)

Install the exec credential plugin:

kubectl krew install oidc-login   # provides `kubectl oidc-login` / kubelogin

Add an OIDC user and context to ~/.kube/config (leave the existing admin context untouched):

users:
  - name: oidc
    user:
      exec:
        apiVersion: client.authentication.k8s.io/v1beta1
        command: kubectl
        args:
          - oidc-login
          - get-token
          - --oidc-issuer-url=https://sso.chelonianlabs.com/application/o/kubectl/
          - --oidc-client-id=kubectl
          - --oidc-pkce-method=S256
          - --oidc-extra-scope=profile
          - --oidc-extra-scope=email
          - --oidc-extra-scope=offline_access

Then add a context that pairs this user with the existing cluster, e.g.:

kubectl config set-context dapper-oidc --cluster=<existing-cluster> --user=oidc

--oidc-pkce-method=S256 means no client secret is needed; offline_access gives a refresh token so you are not re-prompted every hour.

Verify:

kubectl --context dapper-oidc auth whoami          # -> oidc:<you> + oidc:kubernetes-admins
kubectl --context dapper-oidc auth can-i '*' '*'   # -> yes (admins)

Break-glass (when Authentik is down or misbehaving)

Neither path below touches Authentik:

1. x509 admin kubeconfig — switch back to the original admin context:

   kubectl config use-context <admin-context>
   

2. Regenerate from Talos — the Talos mTLS PKI is fully independent of Authentik:

   talosctl kubeconfig ./break-glass-kubeconfig
   export KUBECONFIG=./break-glass-kubeconfig
   

Keep a copy of talosconfig / the admin kubeconfig backed up off-cluster. That backup is what makes the circular-dependency concern (Authentik runs on the cluster you need it to reach) a non-issue.

Network Operations Runbook

Overview

This runbook provides step-by-step procedures for common network operations, troubleshooting, and emergency recovery scenarios for the Dapper Cluster network.

Quick Reference:

• Network Topology Documentation
• Brocade Core: 192.168.1.20
• Arista Distribution: 192.168.1.21
• Aruba Access: 192.168.1.26

Table of Contents

1. Common Operations
2. Troubleshooting Procedures
3. Emergency Procedures
4. Switch Configuration
5. Performance Monitoring
6. Maintenance Windows

Common Operations

Accessing Network Equipment

SSH to Switches

Brocade ICX6610:

ssh admin@192.168.1.20
# [TODO: Document default credentials location]

Arista 7050:

ssh admin@192.168.1.21
# [TODO: Document default credentials location]

Aruba S2500-48p:

ssh admin@192.168.1.26
# [TODO: Document default credentials location]

Console Access

When SSH is unavailable:

# [TODO: Document console server or direct serial access]
# Brocade: Serial settings [TODO: baud rate, etc]
# Arista: Serial settings [TODO: baud rate, etc]

Checking Switch Health

Brocade ICX6610

# Basic health check
show version
show chassis
show cpu
show memory
show log tail 50

# Temperature and power
show inline power
show environment

# Check for errors
show logging | include error
show logging | include warn

Arista 7050

# Basic health check
show version
show environment all
show processes top

# Check for errors
show logging last 100
show logging | grep -i error

Verifying VLAN Configuration

Check VLAN Assignments

Brocade:

show vlan

# Check specific VLAN
show vlan 100
show vlan 150
show vlan 200

# Check which ports are in which VLANs
show vlan ethernet 1/1/1

Arista:

show vlan

# Check VLAN details
show vlan id 200

# Show interfaces by VLAN
show interfaces status

Verify Trunk Ports

Brocade:

# Show trunk configuration
show interface brief | include Trunk

# Show specific trunk
show interface ethernet 1/1/41
show interface ethernet 1/1/42

Arista:

# Show trunk ports
show interface trunk

# Show specific interface
show interface ethernet 49
show interface ethernet 50

Checking Link Aggregation (LAG) Status

Brocade LAG Status

# Show all LAG groups
show lag brief

# Show specific LAG details
show lag [lag-id]

# Show which ports are in LAG
show lag | include active

# Check individual LAG port status
show interface ethernet 1/1/41
show interface ethernet 1/1/42

Expected Output When Working:

LAG "brocade-to-arista" (lag-id [X]) has 2 active ports:
  ethernet 1/1/41 (40Gb) - Active
  ethernet 1/1/42 (40Gb) - Active

Arista Port-Channel Status

# Show port-channel summary
show port-channel summary

# Show specific port-channel
show interface port-channel 1

# Check member interfaces
show interface ethernet 49 port-channel
show interface ethernet 50 port-channel

Expected Output When Working:

Port-Channel1:
  Active Ports: 2
  Et49: Active
  Et50: Active
  Protocol: LACP

Monitoring Traffic and Bandwidth

Real-Time Interface Statistics

Brocade:

# Show interface rates
show interface ethernet 1/1/41 | include rate
show interface ethernet 1/1/42 | include rate

# Show all interface statistics
show interface ethernet 1/1/41

# Monitor in real-time (if supported)
monitor interface ethernet 1/1/41

Arista:

# Show interface counters
show interface ethernet 49 counters

# Show interface rates
show interface ethernet 49 | grep rate

# Real-time monitoring
watch 1 show interface ethernet 49 counters rate

Identify Top Talkers

Brocade:

# [TODO: Document method to identify top talkers]
# May require SNMP monitoring or sFlow

Arista:

# Check interface utilization
show interface counters utilization

# If sFlow configured:
# [TODO: Document sFlow commands]

Testing Connectivity

From Your Workstation

Test Management Plane:

# Ping all management interfaces
ping -c 4 192.168.1.20  # Brocade
ping -c 4 192.168.1.21  # Arista
ping -c 4 192.168.1.26  # Aruba
ping -c 4 192.168.1.7   # Mikrotik House
ping -c 4 192.168.1.8   # Mikrotik Shop

# Test wireless bridge latency
ping -c 100 192.168.1.8 | tail -3

Test Server Network (VLAN 100):

# Test Kubernetes nodes
ping -c 4 10.100.0.40   # K8s VIP
ping -c 4 10.100.0.50   # talos-control-1
ping -c 4 10.100.0.51   # talos-control-2
ping -c 4 10.100.0.52   # talos-control-3

Test from Kubernetes Nodes:

# SSH to a Talos node (if enabled) or use kubectl exec
kubectl exec -it -n default <pod-name> -- sh

# Test connectivity
ping 10.150.0.10   # Storage network
ping 10.100.0.1    # Gateway
ping 8.8.8.8       # Internet

MTU Testing (Jumbo Frames)

Test VLAN 150/200 MTU 9000:

# From a host on VLAN 150
ping -M do -s 8972 10.150.0.10

# -M do: Don't fragment
# -s 8972: 8972 + 28 (IP+ICMP headers) = 9000

# If this fails but smaller packets work, MTU is misconfigured

Path Testing

Trace route across networks:

# From your workstation
traceroute 10.100.0.50

# Expected path (if everything is working):
# 1. Local gateway
# 2. Wireless bridge
# 3. Brocade/OPNsense
# 4. Destination

Troubleshooting Procedures

Issue: No Connectivity to Garage Switches

Symptoms:

• Cannot ping/SSH to Brocade (192.168.1.20) or Arista (192.168.1.21)
• Can ping Aruba switch (192.168.1.26)

Diagnosis:

1. Test wireless bridge:

   ping 192.168.1.7   # Mikrotik House
   ping 192.168.1.8   # Mikrotik Shop
   

   • If 192.168.1.7 responds but 192.168.1.8 doesn't: Wireless link down
   • If neither respond: Mikrotik issue or config problem

2. Check Aruba-to-Mikrotik connection:

   # SSH to Aruba
   ssh admin@192.168.1.26
   
   # Check port status for Mikrotik connection
   show interface [TODO: port ID]
   

Resolution:

If wireless bridge is down:

1. Check Mikrotik radios web interface (192.168.1.7, 192.168.1.8)
2. Check alignment and signal strength
3. Verify power to both radios
4. Check for interference (weather, obstacles)
5. Emergency: Use physical console access to switches in garage

If Mikrotik is up but switches unreachable:

1. Check VLAN 1 configuration on trunk ports
2. Verify Mikrotik is not blocking traffic
3. Check Brocade port connected to Mikrotik is up

Issue: Kubernetes Pods Can't Access Storage

Symptoms:

• Pods stuck in ContainerCreating
• PVC stuck in Pending
• Errors about unable to mount CephFS

Diagnosis:

1. Check Rook/Ceph health:

   kubectl -n rook-ceph get cephcluster
   kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph status
   

2. Check network connectivity from Kubernetes nodes to Ceph monitors:

   # From a Talos node or debug pod
   ping 10.150.0.10   # Test VLAN 150 connectivity
   
   # Test Ceph monitor port
   nc -zv <monitor-ip> 6789
   

3. Verify VLAN 150 MTU:

   # Test jumbo frames
   ping -M do -s 8972 10.150.0.10
   

4. Check CSI driver logs:

   kubectl -n rook-ceph logs -l app=csi-cephfsplugin --tail=100
   

Resolution:

If MTU mismatch:

1. Verify MTU 9000 on all VLAN 150 interfaces
2. Check Proxmox bridge MTU settings
3. Check switch port MTU configuration

If connectivity issue:

1. Check VLAN 150 is properly tagged on trunk ports
2. Verify Proxmox host network configuration
3. Check Brocade routing for VLAN 150

Issue: Slow Ceph Performance

Symptoms:

• Slow pod startup times
• High I/O latency in applications
• Ceph health warnings about slow ops

Diagnosis:

1. Check Ceph cluster health:

   kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph health detail
   kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd perf
   

2. Check network bandwidth utilization:

   On Brocade (VLAN 150 - Ceph Public):

   # Check 10Gb bonds to Proxmox hosts
   show interface ethernet 1/1/[TODO: ports] | include rate
   

   On Arista (VLAN 200 - Ceph Cluster):

   # Check 40Gb links to Proxmox hosts
   show interface ethernet [TODO: ports] counters rate
   

3. Identify bottlenecks:

   • Are 10Gb links saturated? (VLAN 150)
   • Are 40Gb links saturated? (VLAN 200)
   • Is the Brocade-Arista link saturated?

Resolution:

If Brocade-Arista link is bottleneck:

• Primary Issue: Only one 40Gb link active (see below to enable second link)
• Enabling second 40Gb link will double bandwidth to 80Gbps

If MTU not configured:

• Verify MTU 9000 on VLAN 150 and 200
• Check each hop in the path

If switch CPU is high:

• Check for broadcast storms
• Verify STP is working correctly
• Look for loops in topology

Issue: Network Loop / Broadcast Storm

Symptoms:

• Network performance severely degraded
• High CPU usage on switches
• Connectivity flapping
• Massive packet rates on interfaces

Diagnosis:

1. Check for duplicate MAC addresses:

   # Brocade
   show mac-address
   
   # Look for same MAC on multiple ports
   

2. Check STP status:

   # Brocade
   show spanning-tree
   
   # Arista
   show spanning-tree
   

3. Look for physical loops:

   • Review physical topology diagram
   • Check for accidental double connections
   • Known issue: Brocade-Arista 2x 40Gb links not in LAG

Resolution:

Immediate (Emergency):

1. Disable one link causing loop:

   # On Arista (already done in current config)
   configure
   interface ethernet 50
   shutdown
   

2. Verify spanning-tree is enabled:

   # Brocade
   show spanning-tree
   
   # If not enabled:
   configure terminal
   spanning-tree
   

Permanent Fix:

• Configure proper LAG/port-channel (see section below)

Issue: Proxmox Host Loses Network Connectivity

Symptoms:

• Cannot ping Proxmox host management IP
• VMs on host also offline
• IPMI still accessible

Diagnosis:

1. Access via IPMI console:

   # [TODO: Document IPMI access method]
   

2. Check bond status on Proxmox:

   # From Proxmox console
   ip link show
   
   # Check bond interfaces
   cat /proc/net/bonding/bond0
   cat /proc/net/bonding/bond1
   

3. Check switch ports:

   # On Brocade
   show interface ethernet 1/1/[TODO: ports for this host]
   show lag [TODO: lag-id for this host]
   

Resolution:

If bond is down on Proxmox:

1. Check physical cables
2. Restart networking on Proxmox (WARNING: will disrupt VMs)
3. Check switch port status

If ports down on switch:

1. Check for error counters
2. Re-enable port if administratively down
3. Check for physical issues (SFP, cable)

Issue: High Latency Across Wireless Bridge

Symptoms:

• Ping times to garage > 10ms (normally 1-2ms)
• Slow access to services in garage
• Packet loss

Diagnosis:

1. Test latency:

   ping -c 100 192.168.1.8
   
   # Look at:
   # - Average latency
   # - Packet loss %
   # - Jitter (variation)
   

2. Check Mikrotik radio status:

   • Access web interface: 192.168.1.7 and 192.168.1.8
   • Check signal strength
   • Check throughput/bandwidth utilization
   • Look for interference

3. Test with iperf:

   # On server side (garage)
   iperf3 -s
   
   # On client side (house)
   iperf3 -c 192.168.1.8 -t 30
   
   # Should see ~1 Gbps
   

Resolution:

If signal degraded:

1. Check for obstructions (trees, weather)
2. Check alignment
3. Check for interference sources
4. Consider backup link or failover

If bandwidth saturated:

1. Identify high-bandwidth users/applications
2. Implement QoS if available
3. Consider upgrade to higher bandwidth link

Emergency Procedures

Complete Network Outage (Wireless Bridge Down)

Impact:

• No remote access to garage infrastructure
• Kubernetes cluster still functions internally
• No internet access from garage
• Management access requires physical presence

Emergency Access Methods:

1. Physical console access:

   # [TODO: Document where console cables are stored]
   # Connect laptop directly to switch console port
   

2. IPMI access (if VPN or alternative route exists):

   # [TODO: Document IPMI network topology]
   

Restoration Steps:

1. Check Mikrotik radios:

   • Physical inspection of both radios
   • Power cycle if needed
   • Check alignment

2. Temporary workaround:

   • [TODO: Document backup connectivity method]
   • VPN tunnel over alternative route?
   • Temporary cable run?

3. Verify restoration:

   ping 192.168.1.8
   ping 192.168.1.20
   ssh admin@192.168.1.20
   

Core Switch (Brocade) Failure

Impact:

• Loss of VLAN 150/200 routing
• Kubernetes cluster degraded (storage issues)
• Loss of 10Gb connectivity to Proxmox hosts

Emergency Actions:

1. Do NOT reboot all Proxmox hosts simultaneously

   • Cluster may be operational on running workloads
   • Storage connections via VLAN 200 through Arista may still work

2. Check Brocade status:

   • Physical inspection (power, fans, LEDs)
   • Console access
   • Review logs

3. If Brocade must be replaced:

   • [TODO: Document backup configuration location]
   • [TODO: Document restoration procedure]
   • [TODO: Document spare hardware location]

Spanning Tree Failure / Network Loop

Impact:

• Network completely unusable
• High CPU on all switches
• Broadcast storm

Emergency Actions:

1. Disconnect Brocade-Arista links:

   # On Arista (fastest access if SSH still works)
   configure
   interface ethernet 49
   shutdown
   interface ethernet 50
   shutdown
   

2. Or physically disconnect:

   • Unplug both 40Gb QSFP+ cables between Brocade and Arista

3. Wait for network to stabilize (30-60 seconds)

4. Reconnect ONE link only:

   # On Arista
   configure
   interface ethernet 49
   no shutdown
   

5. Verify stability before enabling second link

Accidental Configuration Change

Symptoms:

• Network suddenly degraded after change
• New errors appearing
• Connectivity loss

Emergency Actions:

1. Rollback configuration:

   Brocade:

   # Show configuration history
   show configuration
   
   # Revert to previous config
   # [TODO: Document Brocade config rollback method]
   

   Arista:

   # Show rollback options
   show configuration sessions
   
   # Rollback to previous
   configure session rollback <session-name>
   

2. If rollback not available:

   • Reboot switch (loads startup-config)
   • WARNING: Brief outage during reboot

Switch Configuration

Configure Brocade-Arista LAG (Fix Loop Issue)

Prerequisites:

• Maintenance window scheduled
• Both 40Gb QSFP+ cables connected and working
• Console access to both switches available
• Configuration backed up

Step 1: Pre-Change Verification

# Verify current state
# On Brocade:
show interface ethernet 1/1/41
show interface ethernet 1/1/42

# On Arista:
show interface ethernet 49
show interface ethernet 50  # Currently disabled

# Document current traffic levels
show interface ethernet 1/1/41 | include rate

Step 2: Configure Brocade LAG

# SSH to Brocade
ssh admin@192.168.1.20

# Enter configuration mode
enable
configure terminal

# Create LAG
lag brocade-to-arista dynamic id [TODO: Choose available LAG ID, e.g., 10]
  ports ethernet 1/1/41 to 1/1/42
  primary-port 1/1/41
  lacp-timeout short
  deploy
exit

# Configure VLAN on LAG
vlan 1
  tagged lag [LAG-ID]
exit

vlan 100
  tagged lag [LAG-ID]
exit

vlan 150
  tagged lag [LAG-ID]
exit

vlan 200
  tagged lag [LAG-ID]
exit

# Apply to interfaces
interface ethernet 1/1/41
  link-aggregate active
exit

interface ethernet 1/1/42
  link-aggregate active
exit

# Save configuration
write memory

# Verify
show lag brief
show lag [LAG-ID]

Step 3: Configure Arista Port-Channel

# SSH to Arista
ssh admin@192.168.1.21

# Enter configuration mode
enable
configure

# Create port-channel
interface Port-Channel1
  description Link to Brocade ICX6610
  switchport mode trunk
  switchport trunk allowed vlan 1,100,150,200
exit

# Add member interfaces
interface Ethernet49
  description Brocade 40G Link 1
  channel-group 1 mode active
  lacp rate fast
exit

interface Ethernet50
  description Brocade 40G Link 2
  channel-group 1 mode active
  lacp rate fast
exit

# Save configuration
write memory

# Verify
show port-channel summary
show interface Port-Channel1
show lacp neighbor

Step 4: Verify Configuration

# On Brocade:
show lag [LAG-ID]
# Should show: 2 ports active

show lacp
# Should show: Negotiated with neighbor

# On Arista:
show port-channel summary
# Should show: Po1(U) with Et49(P), Et50(P)

show lacp neighbor
# Should show: Brocade as partner

# Test traffic balancing
show interface Port-Channel1 counters
show interface ethernet 49 counters
show interface ethernet 50 counters
# Both Et49 and Et50 should show traffic

Step 5: Monitor for Issues

# Watch for 15 minutes
# On Arista:
watch 10 show port-channel summary

# Check for errors
show logging | grep -i Port-Channel1

# Monitor CPU (should be normal)
show processes top

Rollback Plan (if issues occur):

# On Arista (fastest to disable)
configure
interface ethernet 50
shutdown

# On Brocade (if needed)
configure terminal
no lag brocade-to-arista
interface ethernet 1/1/41
  no link-aggregate
interface ethernet 1/1/42
  no link-aggregate

Adding a New VLAN

Example: Adding VLAN 300 for IoT devices

Step 1: Plan VLAN

• VLAN ID: 300
• Network: [TODO: e.g., 10.30.0.0/24]
• Gateway: [TODO: Which device?]
• Required on: [TODO: Which switches/trunks?]

Step 2: Create VLAN on Brocade

ssh admin@192.168.1.20
enable
configure terminal

# Create VLAN
vlan 300
  name IoT-Network
  tagged ethernet 1/1/41 to 1/1/42  # Trunk to Arista
  tagged ethernet 1/1/[TODO]         # Trunk to Mikrotik
  untagged ethernet 1/1/[TODO]       # Access ports if needed
exit

# If Brocade is gateway:
interface ve 300
  ip address [TODO: IP]/24
exit

# Save
write memory

Step 3: Add to other switches as needed

# On Arista:
configure
vlan 300
  name IoT-Network
exit

interface Port-Channel1
  switchport trunk allowed vlan add 300
exit

write memory

Configuring Jumbo Frames (MTU 9000)

For VLAN 150 and 200 (Ceph networks)

On Brocade:

# Configure MTU on VLAN interfaces
interface ve 150
  mtu 9000
exit

interface ve 200
  mtu 9000
exit

# Configure MTU on physical/LAG interfaces
interface ethernet 1/1/[TODO: storage network ports]
  mtu 9000
exit

write memory

On Arista:

# Configure MTU on interfaces carrying VLAN 150/200
interface ethernet [TODO: ports]
  mtu 9216  # 9000 + overhead
exit

write memory

Verify MTU:

# From Talos node
ping -M do -s 8972 10.150.0.10
# Should succeed without fragmentation

Performance Monitoring

Key Metrics to Monitor

Switch Health:

• CPU utilization (should be <30% normally)
• Memory utilization (should be <70%)
• Temperature (within operating range)
• Power supply status

Interface Health:

• Error counters (input/output errors)
• CRC errors
• Interface resets
• Utilization percentage

Traffic Patterns:

• Bandwidth utilization per interface
• Top talkers per VLAN
• Broadcast/multicast rates

Setting Up Monitoring

[TODO: Document monitoring setup]

Options:

1. SNMP monitoring to Prometheus
2. sFlow for traffic analysis
3. Switch logging to VictoriaLogs
4. Grafana dashboards

Example Prometheus Targets:

# [TODO: Example prometheus config for SNMP exporter]

Baseline Performance Metrics

Normal Operating Conditions:

[TODO: Add baseline measurements]

Maintenance Windows

Pre-Maintenance Checklist

Before any network maintenance:

• Schedule maintenance window

• Notify all users

• Back up switch configurations

  # Brocade
  show running-config > backup-$(date +%Y%m%d).cfg
  
  # Arista
  show running-config > backup-$(date +%Y%m%d).cfg
  

• Document current state

• Have rollback plan ready

• Ensure console access available

• Test backup connectivity method

Post-Maintenance Checklist

After any network maintenance:

• Verify all links are up

  show interface brief
  show lag brief  # Brocade
  show port-channel summary  # Arista
  

• Check for errors

  show logging | include error
  

• Test connectivity to all VLANs
• Monitor for 30 minutes for issues
• Update documentation with any changes
• Save configurations

  write memory
  

Regular Maintenance Tasks

Weekly:

• Review switch logs for errors/warnings
• Check interface error counters
• Verify wireless bridge performance

Monthly:

• Review bandwidth utilization trends
• Check for firmware updates
• Verify backup configurations are current

Quarterly:

• Review and update network documentation
• Test emergency procedures
• Review and optimize switch configurations

Configuration Backup

Backing Up Switch Configurations

Brocade ICX6610:

# Method 1: Copy to TFTP server
copy running-config tftp [TODO: TFTP server IP] brocade-backup-$(date +%Y%m%d).cfg

# Method 2: Display and save manually
show running-config > /tmp/brocade-config.txt

# [TODO: Document automated backup method]

Arista 7050:

# Show running config
show running-config

# Copy to USB (if available)
copy running-config usb:/brocade-backup-$(date +%Y%m%d).cfg

# [TODO: Document automated backup method]

Storage Location:

• [TODO: Document where configurations are backed up]
• Consider: Git repository for version control
• Consider: Automated daily backups via Ansible

Restoring Configurations

Brocade:

# Load config from file
copy tftp running-config [TFTP-IP] [filename]

# Or manually paste config
configure terminal
# Paste configuration

Arista:

# Copy config from file
copy usb:/backup.cfg running-config

# Or configure manually
configure
# Paste configuration

Security Considerations

Access Control

[TODO: Document security policies]

• Who has access to switch management?
• How are credentials managed?
• Is 2FA available/configured?
• Are management VLANs isolated?

Security Best Practices

1. Change default passwords
2. Disable unused ports
3. Enable port security where appropriate
4. Configure DHCP snooping
5. Enable storm control
6. Regular firmware updates
7. Monitor for unauthorized devices

Useful Commands Reference

Brocade ICX6610 Quick Reference

# Basic show commands
show version
show running-config
show interface brief
show vlan
show lag
show mac-address
show spanning-tree
show log

# Interface management
interface ethernet 1/1/1
  enable
  disable
  description [text]

# Save configuration
write memory

Arista 7050 Quick Reference

# Basic show commands
show version
show running-config
show interfaces status
show vlan
show port-channel summary
show mac address-table
show spanning-tree
show logging

# Interface management
configure
interface ethernet 1
  shutdown
  no shutdown
  description [text]

# Save configuration
write memory

Contacts and Escalation

[TODO: Fill in contact information]

References

• Network Topology Documentation
• Storage Architecture - For Ceph network details
• Brocade ICX6610 Documentation: [TODO: Link]
• Arista 7050 Documentation: [TODO: Link]
• [TODO: Add other relevant documentation links]

VolSync Resource Flow and Cleanup Guide

Reference for the VolSync resource hierarchy and the cleanup procedures used when making storage/infrastructure changes.

Overview

VolSync creates a complex chain of Kubernetes resources for backup and replication. Understanding this flow is critical when making infrastructure changes to avoid orphaned resources or provisioning conflicts.

Resource Hierarchy

Flux Kustomization (volsync)
  └── App Kustomization (e.g., archon)
      ├── ReplicationSource (archon)
      │   ├── Snapshot PVC (volsync-archon-src)
      │   ├── Cache PVC (volsync-src-archon-cache) [RBD]
      │   ├── Repository PVC (volsync-archon-repo)
      │   │   └── Repository PV (volsync-archon-repo-pv) [Static CephFS]
      │   └── Mover Pod (volsync-src-archon-xxxxx)
      │
      ├── ReplicationSource (archon-r2) [Remote/R2]
      │   ├── Snapshot PVC (volsync-archon-r2-src)
      │   ├── Cache PVC (volsync-src-archon-r2-cache) [RBD]
      │   └── Mover Pod (volsync-src-archon-r2-xxxxx)
      │
      └── ReplicationDestination (archon-dst)
          ├── Destination PVC (volsync-archon-dst-dest)
          ├── Cache PVC (volsync-dst-archon-dst-cache) [RBD]
          ├── Repository PVC (volsync-archon-repo)
          │   └── Repository PV (volsync-archon-repo-pv) [Static CephFS]
          └── Mover Pod (volsync-dst-archon-dst-xxxxx)

Resource Types Explained

Static Resources (Pre-created via Kustomize)

1. Repository PV (volsync-${APP}-repo-pv)

   • Type: Static CephFS PersistentVolume
   • Points to: /k8s-backups/volsync/${APP} on external Ceph cluster
   • Purpose: Houses restic repository data
   • Created by: flux/components/volsync/repository-pv.yaml template
   • Key attributes:

     volumeAttributes:
       clusterID: rook-ceph
       fsName: cephfs
       staticVolume: "true"
       pool: k8s-backups
       rootPath: "/k8s-backups/volsync/${APP}"
     

2. Repository PVC (volsync-${APP}-repo)

   • Type: PersistentVolumeClaim
   • Binds to: Static PV above via volumeName field
   • Created by: flux/components/volsync/repository-pvc.yaml template

Dynamic Resources (Created by VolSync Operator)

3. ReplicationSource/Destination

   • Type: VolSync CRDs
   • Created by: App kustomizations (when volsync component is included)
   • Manages: Backup/restore jobs

4. Cache PVCs (e.g., volsync-src-${APP}-cache, volsync-dst-${APP}-dst-cache)

   • Type: Dynamic RBD PersistentVolumeClaims
   • Created by: VolSync operator (from ReplicationSource/Destination specs)
   • Purpose: Temporary staging for restic operations
   • Storage Class: ceph-rbd (block storage)

5. Snapshot PVCs (e.g., volsync-${APP}-src)

   • Type: Dynamic CephFS PersistentVolumeClaims
   • Created by: VolSync operator from source data snapshots
   • Storage Class: cephfs-shared

6. Mover Pods

   • Type: Kubernetes Pods
   • Created by: VolSync operator
   • Purpose: Execute restic backup/restore operations
   • Mounts: Source/destination PVCs, cache PVCs, repository PVC

Common Cleanup Scenarios

Scenario 1: Changing Repository PV Configuration

When: Modifying repository-pv.yaml (path, pool, monitors, etc.)

Why cleanup needed: Static PVs are immutable once created. Changes require delete/recreate.

Cleanup Order:

# 1. Suspend the volsync kustomization to prevent recreation
flux suspend kustomization volsync -n storage

# 2. Scale down volsync operator
kubectl scale deployment volsync -n storage --replicas=0

# 3. Delete ReplicationSources
kubectl get replicationsource -A -o json | \
  jq -r '.items[] | "\(.metadata.namespace) \(.metadata.name)"' | \
  xargs -n 2 sh -c 'kubectl delete replicationsource -n $0 $1'

# 4. Delete ReplicationDestinations
kubectl get replicationdestination -A -o json | \
  jq -r '.items[] | "\(.metadata.namespace) \(.metadata.name)"' | \
  xargs -n 2 sh -c 'kubectl delete replicationdestination -n $0 $1'

# 5. Delete Repository PVCs (unbinds from PVs)
kubectl get pvc -A -o json | \
  jq -r '.items[] | select(.metadata.name | test("volsync-.*-repo$")) | "\(.metadata.namespace) \(.metadata.name)"' | \
  xargs -n 2 sh -c 'kubectl delete pvc -n $0 $1'

# 6. Delete Repository PVs
kubectl get pv -o name | \
  grep "persistentvolume/volsync-.*-repo-pv$" | \
  xargs kubectl delete

# 7. Wait for cleanup to complete
sleep 10

# 8. Commit and push your repository-pv.yaml changes
git add kubernetes/flux/components/volsync/repository-pv.yaml
git commit -m "fix(volsync): update repository PV configuration"
git push

# 9. Resume volsync kustomization (recreates PVs/PVCs with new config)
flux resume kustomization volsync -n storage

# 10. Scale volsync operator back up
kubectl scale deployment volsync -n storage --replicas=1

# 11. Watch for recreation
watch kubectl get pv,pvc -A | grep volsync

Scenario 2: Stuck Cache PVC Provisioning

When: Cache PVCs stuck in Pending with "Volume ID already exists" errors

Why this happens: Orphaned provisioning operations in Ceph RBD from previous deletion attempts

Cleanup Order:

# 1. Delete stuck cache PVCs
kubectl get pvc -A -o json | \
  jq -r '.items[] | select(.metadata.name | test("cache")) | select(.status.phase == "Pending") | "\(.metadata.namespace) \(.metadata.name)"' | \
  xargs -n 2 sh -c 'kubectl delete pvc -n $0 $1'

# 2. Wait for VolSync operator to recreate with fresh UIDs
# Cache PVCs are owned by ReplicationSource/Destination resources
# and will auto-recreate with new UIDs that don't conflict

Scenario 3: Full VolSync Reset

When: Major infrastructure changes, migrating storage backends, etc.

Cleanup Order:

# Complete reset - destroys all VolSync resources but preserves data in /k8s-backups/volsync/

# 1. Suspend volsync
flux suspend kustomization volsync -n storage

# 2. Scale down operator
kubectl scale deployment volsync -n storage --replicas=0

# 3. Delete all ReplicationSources
kubectl get replicationsource -A -o json | \
  jq -r '.items[] | "\(.metadata.namespace) \(.metadata.name)"' | \
  xargs -n 2 sh -c 'kubectl delete replicationsource -n $0 $1'

# 4. Delete all ReplicationDestinations
kubectl get replicationdestination -A -o json | \
  jq -r '.items[] | "\(.metadata.namespace) \(.metadata.name)"' | \
  xargs -n 2 sh -c 'kubectl delete replicationdestination -n $0 $1'

# 5. Delete all cache PVCs
kubectl get pvc -A -o json | \
  jq -r '.items[] | select(.metadata.name | test("cache")) | "\(.metadata.namespace) \(.metadata.name)"' | \
  xargs -n 2 sh -c 'kubectl delete pvc -n $0 $1'

# 6. Delete all snapshot PVCs
kubectl get pvc -A -o json | \
  jq -r '.items[] | select(.metadata.name | test("volsync-.*-src$|volsync-.*-dst-dest$")) | "\(.metadata.namespace) \(.metadata.name)"' | \
  xargs -n 2 sh -c 'kubectl delete pvc -n $0 $1'

# 7. Delete repository PVCs
kubectl get pvc -A -o json | \
  jq -r '.items[] | select(.metadata.name | test("volsync-.*-repo$")) | "\(.metadata.namespace) \(.metadata.name)"' | \
  xargs -n 2 sh -c 'kubectl delete pvc -n $0 $1'

# 8. Delete repository PVs
kubectl get pv -o name | \
  grep "persistentvolume/volsync-.*-repo-pv$" | \
  xargs kubectl delete

# 9. Resume after changes
flux resume kustomization volsync -n storage
kubectl scale deployment volsync -n storage --replicas=1

Critical Configuration Details

Static CephFS PV Requirements

For static PVs pointing to existing CephFS directories:

volumeAttributes:
  clusterID: rook-ceph # References mon-endpoints ConfigMap
  fsName: cephfs # CephFS filesystem name
  staticVolume: "true" # REQUIRED for static PVs
  pool: k8s-backups # Data pool (doesn't affect mount path)
  rootPath: "/k8s-backups/volsync/${APP}" # FULL path in filesystem

Important Notes:

• staticVolume: "true" is mandatory - without it, CSI driver won't properly discover monitors
• pool parameter specifies which data pool stores the data, NOT the mount path
• rootPath must be the full filesystem path where data exists
• monitors field is NOT required - auto-discovered from rook-ceph-mon-endpoints ConfigMap when staticVolume: "true"

Why Pool Doesn't Change Mount Path

The pool parameter in CephFS tells Ceph which data pool to use for new data storage, but:

• Mount paths are at the filesystem level, not pool level
• All pools share the same namespace/directory structure
• Path /k8s-backups/volsync/archon exists in the filesystem regardless of pool
• The pool only matters for data placement on OSDs

Example:

Filesystem: cephfs
  ├── /volumes/csi/...          (data in cephfs_data pool)
  ├── /k8s-backups/volsync/...  (data in k8s-backups pool)
  └── /media/...                (data in cephfs_media pool)

Troubleshooting

Mount Error: "No such file or directory"

mount error 2 = No such file or directory

Cause: rootPath doesn't exist in CephFS filesystem Solution: Verify path exists: ls /mnt/cephfs/k8s-backups/volsync/${APP} Fix: Update rootPath in repository-pv.yaml to match actual filesystem path

DNS SRV Lookup Warnings

unable to get monitor info from DNS SRV with service name: ceph-mon

Impact: Warning only - mount proceeds with explicit monitors from ConfigMap Cause: Ceph client tries DNS discovery before using explicit monitors Action: None required if mount succeeds

Cache PVC: "Volume ID already exists"

rpc error: code = Aborted desc = an operation with the given Volume ID already exists

Cause: Orphaned provisioning operation in RBD from previous PVC with same UID Solution: Delete the PVC - VolSync will recreate with fresh UID

Data Safety

What Gets Deleted

When following cleanup procedures:

• ✅ Kubernetes PVs/PVCs (metadata only)
• ✅ ReplicationSource/Destination CRDs
• ✅ Temporary cache volumes
• ✅ Snapshot PVCs

What Gets Preserved

The cleanup procedures DO NOT delete:

• ✅ Actual restic repository data in /k8s-backups/volsync/${APP}/
• ✅ Application data in source PVCs
• ✅ R2/remote backup data

Recovery: Once resources are recreated, VolSync reconnects to existing restic repositories automatically.

Monitoring Recovery

After cleanup and resume:

# Watch PVs being created
watch kubectl get pv | grep volsync

# Watch PVCs binding
watch kubectl get pvc -A | grep volsync

# Watch ReplicationSources
watch kubectl get replicationsource -A

# Check mover pod logs
kubectl logs -n <namespace> <volsync-src-pod> -f

# Verify mounts succeeded
kubectl get events -A --sort-by='.lastTimestamp' | grep -i mount | grep volsync

Quick Reference Commands

# Count resources
echo "PVs: $(kubectl get pv | grep 'volsync-.*-repo-pv' | wc -l)"
echo "Repo PVCs: $(kubectl get pvc -A | grep 'volsync.*repo' | wc -l)"
echo "Cache PVCs: $(kubectl get pvc -A | grep 'cache' | wc -l)"
echo "Sources: $(kubectl get replicationsource -A --no-headers | wc -l)"
echo "Destinations: $(kubectl get replicationdestination -A --no-headers | wc -l)"

# Suspend/Resume
flux suspend kustomization volsync -n storage
flux resume kustomization volsync -n storage

# Scale operator
kubectl scale deployment volsync -n storage --replicas=0
kubectl scale deployment volsync -n storage --replicas=1

# Check volsync kustomization status
kubectl get ks volsync -n storage

Known Failure Modes

Mount failure: "missing required field monitors"

Cause: Missing staticVolume: "true" attribute in the PV spec — the CSI driver can't discover monitors from the ConfigMap without it. Fix: Add staticVolume: "true" to volumeAttributes.

Mount failure: "No such file or directory"

Cause: rootPath doesn't match the actual filesystem path (e.g. /volsync/${APP} vs /k8s-backups/volsync/${APP}). Fix: Set rootPath to the full path where the data exists.

Cache PVCs stuck provisioning

Cause: Orphaned RBD volume IDs from rapid delete/create cycles ("Volume ID already exists"). Fix: Delete the stuck PVCs and let VolSync recreate them with fresh UIDs.

Persistent "Volume ID already exists" after mass PVC deletion

Cause: When many PVCs are deleted at once, their PVs linger in Released state and keep holding the Ceph RBD volume IDs. New PVCs reuse those IDs and fail to provision.

Resolution steps:

# 1. Identify the deadlock
kubectl get pvc -A | grep Pending  # PVCs stuck
kubectl get pods -A | grep volsync-dst | grep Pending  # Mover pods can't start
kubectl logs -n rook-ceph deploy/csi-rbdplugin-provisioner -c csi-rbdplugin --tail=50 | grep "already exists"

# 2. Break the deadlock - delete ReplicationDestinations
while read namespace app; do
  kubectl delete replicationdestination -n "$namespace" "${app}-dst" --wait=false
done < apps_list.txt

# 3. Wait for mover pods and PVCs to clean up
sleep 10

# 4. Delete all Released PVs (THE KEY STEP!)
kubectl get pv -o json | jq -r '.items[] | select(.spec.csi.driver == "rook-ceph.rbd.csi.ceph.com") | select(.status.phase == "Released") | .metadata.name' | xargs -r kubectl delete pv

# 5. Optionally restart RBD provisioner for fresh state
kubectl rollout restart deployment -n rook-ceph csi-rbdplugin-provisioner

# 6. Reconcile apps to recreate everything
flux reconcile kustomization -n <namespace> <app>

Why this worked:

• PVCs were deleted → PVs became "Released" but not deleted (reclaimPolicy: Delete doesn't apply immediately)
• Released PVs held references to Ceph RBD volume IDs
• New PVCs got same UIDs, tried to create volumes with existing IDs
• Deleting Released PVs freed the volume IDs in Ceph RBD
• Fresh provisioning succeeded with clean slate

Key Insight: When doing mass PVC migrations, always check for and clean up Released PVs immediately!

Related Documentation

• VolSync Official Docs
• Ceph CSI Static PV Guide
• Rook CephFS Documentation