Data & Backups

All persistent data in HALO resides on the Nexus host—either in Docker volumes, bind mounts, or PostgreSQL databases. This document describes data locations, retention policies, backup strategies, and recovery procedures for ensuring data durability across the ecosystem.

Data Storage Architecture

Storage Types

Docker Named Volumes:

Managed by Docker, stored in /var/lib/docker/volumes/
Isolated from host filesystem, portable across host systems
Used for: PostgreSQL data, Grafana dashboards, Frigate configuration

Bind Mounts:

Direct filesystem paths mounted into containers
Easier to backup with standard filesystem tools
Used for: Frigate recordings, Home Assistant config, n8n workflows export

PostgreSQL Databases:

Centralized database server storing structured data
Schema-isolated databases for n8n, Home Assistant, Omnia API, Grafana
Single backup target for all relational data

MQTT State:

Mosquitto broker stores persistent messages and subscriptions
Typically not backed up—services re-publish state on restart

Data Locations by Service

Core Infrastructure

PostgreSQL (Database Server)

Volume: nexus_postgres_data
Path: /var/lib/docker/volumes/nexus_postgres_data/_data/
Schemas:
- n8n - Workflow definitions, execution logs, credentials
- home_assistant - Entity history, events, recorder database
- omnia_api - User preferences, widget configurations
- grafana - Dashboard definitions, data source configs
Size: 5-50GB depending on history retention
Backup Method: pg_dump for logical backups, volume snapshot for binary backups

Redis (Message Broker)

Volume: nexus_redis_data
Path: /var/lib/docker/volumes/nexus_redis_data/_data/
Contents: n8n job queues, session data
Size: 100MB-1GB (transient data, periodic snapshots)
Backup Priority: Low—queues are ephemeral, can replay from n8n workflows

Traefik (Ingress Controller)

Volume: nexus_traefik_letsencrypt
Path: /var/lib/docker/volumes/nexus_traefik_letsencrypt/_data/acme.json
Contents: TLS certificates from Let’s Encrypt
Size: <10MB
Backup Priority: Medium—certificates auto-renew, but backup avoids rate limits

Workflows & Automation

n8n (Workflow Engine)

Database: PostgreSQL schema n8n
Workflow Definitions: Stored in PostgreSQL
Credentials: Encrypted in PostgreSQL (requires N8N_ENCRYPTION_KEY)
Execution History: Last 30 days in PostgreSQL (configurable retention)
Backup Method: pg_dump for PostgreSQL, environment variable backup for encryption key

Node-RED (Reactive Automation)

Bind Mount: /opt/node-red/data → ./data/node-red/ on host
Contents: flows.json (flow definitions), settings.js, credentials
Size: <100MB
Backup Method: Copy bind mount directory, backup flows via Node-RED UI export

HALO Applications

Home Assistant

Bind Mount: /config → configs/home-assistant/ on host
Contents:
- configuration.yaml - System configuration
- automations.yaml - Automation rules
- scripts.yaml - Reusable scripts
- home-assistant.log - Application logs
- .storage/ - Entity registry, device registry, lovelace dashboards
Database: PostgreSQL schema home_assistant (history, state)
Size: 1-5GB (config), 5-50GB (database history)
Backup Method: Copy bind mount directory, pg_dump for history database

Omnia (API)

Database: PostgreSQL schema omnia_api
Contents: User profiles, widget configurations, preferences
Size: <500MB
Backup Method: pg_dump for database schema

Frigate (Security Cameras)

Bind Mount: /media/frigate → /mnt/recordings/frigate/ on host
Contents:
- recordings/ - Continuous recordings (24x7)
- clips/ - Event-triggered video clips (motion, person detection)
- frigate.db - SQLite database of events and metadata
Retention:
- Recordings: 7 days (auto-delete)
- Clips: 30 days (auto-delete)
Size: 500GB-2TB depending on camera count and retention
Backup Priority: Low—recordings are time-series data, not critical for recovery

Zigbee2MQTT

Bind Mount: /app/data → ./data/zigbee2mqtt/ on host
Contents:
- configuration.yaml - MQTT broker connection, device settings
- database.db - SQLite database of paired devices
Size: <50MB
Backup Method: Copy bind mount directory

Monitoring & Visualization

Grafana

Database: PostgreSQL schema grafana (dashboards, data sources, users)
Volume: nexus_grafana_data (plugins, logs)
Size: <500MB
Backup Method: pg_dump for database, Grafana dashboard export API

Retention Policies

Hot Data (Active Operational Data)

n8n Execution Logs: 30 days (configurable via EXECUTIONS_DATA_PRUNE_MAX_AGE)

Home Assistant History: 14 days in PostgreSQL (configurable via purge_keep_days)

Frigate Recordings: 7 days continuous, 30 days clips (configurable per camera in config.yml)

PostgreSQL Logs: 7 days (rotate via log_rotation_age)

Traefik Access Logs: 30 days (rotate via logrotate)

Warm Data (Historical Backups)

Full System Backups: 7 daily, 4 weekly, 12 monthly (tiered retention)

Database Dumps: 14 daily, 8 weekly (incremental cleanup)

Configuration Snapshots: 30 days (on-change backups)

Cold Data (Archival)

Major Release Backups: Indefinite retention, stored off-site

Incident Response Snapshots: Retained for 1 year for forensic analysis

Backup Strategies

PostgreSQL Backups

Daily Logical Dumps:

#!/bin/bash
# /opt/nexus/scripts/backup-postgres.sh

BACKUP_DIR="/mnt/backups/postgres"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)

# Dump all databases
docker exec postgres-db pg_dumpall -U postgres | gzip > \
  "$BACKUP_DIR/postgres_full_$TIMESTAMP.sql.gz"

# Per-schema dumps (faster restoration)
for schema in n8n home_assistant omnia_api grafana; do
  docker exec postgres-db pg_dump -U postgres -n $schema postgres | gzip > \
    "$BACKUP_DIR/postgres_${schema}_$TIMESTAMP.sql.gz"
done

# Cleanup old backups (keep 14 days)
find "$BACKUP_DIR" -name "postgres_*.sql.gz" -mtime +14 -delete

Run Daily:

# /etc/cron.d/postgres-backup
0 2 * * * root /opt/nexus/scripts/backup-postgres.sh

Binary Backups (Point-in-Time Recovery):

# Enable WAL archiving in postgresql.conf
archive_mode = on
archive_command = 'cp %p /mnt/backups/postgres/wal_archive/%f'

# Base backup
docker exec postgres-db pg_basebackup -U postgres -D /mnt/backups/postgres/base -Ft -z -P

Docker Volume Backups

Snapshot All Named Volumes:

#!/bin/bash
# /opt/nexus/scripts/backup-volumes.sh

BACKUP_DIR="/mnt/backups/volumes"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)

# List all nexus volumes
VOLUMES=$(docker volume ls -q | grep ^nexus_)

for volume in $VOLUMES; do
  echo "Backing up $volume..."

  # Use temporary container to tar volume contents
  docker run --rm \
    -v "$volume:/volume" \
    -v "$BACKUP_DIR:/backup" \
    alpine \
    tar czf "/backup/${volume}_${TIMESTAMP}.tar.gz" -C /volume .
done

# Cleanup old backups (keep 7 days)
find "$BACKUP_DIR" -name "nexus_*.tar.gz" -mtime +7 -delete

Run Weekly:

# /etc/cron.d/volume-backup
0 3 * * 0 root /opt/nexus/scripts/backup-volumes.sh

Configuration Backups

Home Assistant, Node-RED, Frigate Configs:

#!/bin/bash
# /opt/nexus/scripts/backup-configs.sh

BACKUP_DIR="/mnt/backups/configs"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="$BACKUP_DIR/configs_$TIMESTAMP.tar.gz"

# Backup all config directories
tar czf "$BACKUP_FILE" \
  -C /opt/halo \
  configs/home-assistant \
  configs/frigate \
  configs/node-red \
  configs/zigbee2mqtt \
  nexus/compose \
  nexus/env

# Cleanup old backups (keep 30 days)
find "$BACKUP_DIR" -name "configs_*.tar.gz" -mtime +30 -delete

Run Daily:

# /etc/cron.d/config-backup
0 1 * * * root /opt/nexus/scripts/backup-configs.sh

Git Version Control (Best Practice):

# Track config changes in git
cd /opt/halo/configs/home-assistant
git add -A
git commit -m "Auto-commit: $(date +%Y-%m-%d)"
git push origin main

Provides granular history and easy rollback of configuration changes.

Automated Backup System

Systemd Timers

Replace cron with systemd for better logging and dependency management:

Service Unit:

# /etc/systemd/system/nexus-backup-postgres.service
[Unit]
Description=Backup PostgreSQL databases for Nexus
After=docker.service

[Service]
Type=oneshot
ExecStart=/opt/nexus/scripts/backup-postgres.sh
User=root
StandardOutput=journal
StandardError=journal

Timer Unit:

# /etc/systemd/system/nexus-backup-postgres.timer
[Unit]
Description=Daily PostgreSQL backup timer

[Timer]
OnCalendar=daily
OnCalendar=02:00
Persistent=true

[Install]
WantedBy=timers.target

Enable and Start:

systemctl enable --now nexus-backup-postgres.timer
systemctl list-timers  # Verify scheduled

Backup Monitoring

Check Backup Success:

#!/bin/bash
# /opt/nexus/scripts/check-backups.sh

BACKUP_DIR="/mnt/backups/postgres"
THRESHOLD_HOURS=26  # Expect daily backups

LATEST=$(find "$BACKUP_DIR" -name "postgres_full_*.sql.gz" -mtime -1 | wc -l)

if [ "$LATEST" -eq 0 ]; then
  echo "ERROR: No PostgreSQL backup found in last $THRESHOLD_HOURS hours"
  # Send alert via n8n webhook or email
  exit 1
fi

echo "OK: Recent PostgreSQL backup found"

Alert Integration:

Use n8n workflow triggered by webhook to send notifications on backup failures.

Restoration Procedures

PostgreSQL Restore

Full Restore:

# Stop services using database
docker compose -f nexus/compose/n8n.yml down

# Restore dump
gunzip -c /mnt/backups/postgres/postgres_full_20250115_020000.sql.gz | \
  docker exec -i postgres-db psql -U postgres

# Restart services
docker compose -f nexus/compose/n8n.yml up -d

Single Schema Restore:

# Restore only n8n schema
gunzip -c /mnt/backups/postgres/postgres_n8n_20250115_020000.sql.gz | \
  docker exec -i postgres-db psql -U postgres -d postgres

Docker Volume Restore

Restore Specific Volume:

# Stop container using volume
docker stop grafana

# Delete existing volume (or rename for safety)
docker volume rm nexus_grafana_data

# Create new volume
docker volume create nexus_grafana_data

# Extract backup into volume
docker run --rm \
  -v nexus_grafana_data:/volume \
  -v /mnt/backups/volumes:/backup \
  alpine \
  tar xzf /backup/nexus_grafana_data_20250115_030000.tar.gz -C /volume

# Restart container
docker start grafana

Configuration Restore

Restore All Configs:

# Extract backup
tar xzf /mnt/backups/configs/configs_20250115_010000.tar.gz -C /opt/halo

# Restart affected services
docker compose -f nexus/compose/n8n.yml restart

Git Rollback:

cd /opt/halo/configs/home-assistant
git log --oneline  # Find commit to restore
git checkout abc1234 -- automations.yaml
docker restart homeassistant

Off-Site Backups

Backup Strategies

Cloud Storage (Recommended):

# Sync to cloud storage (S3, Backblaze B2, etc.)
rclone sync /mnt/backups/postgres remote:halo-backups/postgres \
  --transfers 4 --checkers 8 --exclude "*.tmp"

Remote NAS:

# Rsync to remote NAS over SSH
rsync -avz --delete /mnt/backups/ \
  backup-user@nas.local:/volume1/halo-backups/

External USB Drive:

# Weekly manual copy to USB drive
rsync -avz /mnt/backups/ /media/usb-backup/halo/

Encryption

Encrypt Backups Before Upload:

# Encrypt PostgreSQL dump with GPG
gpg --symmetric --cipher-algo AES256 \
  -o postgres_full_encrypted.sql.gz.gpg \
  postgres_full_20250115_020000.sql.gz

# Upload encrypted file
rclone copy postgres_full_encrypted.sql.gz.gpg remote:halo-backups/

Decrypt During Restore:

gpg --decrypt postgres_full_encrypted.sql.gz.gpg | \
  gunzip | docker exec -i postgres-db psql -U postgres

Disaster Recovery Testing

Quarterly DR Drills

Procedure:

Provision Clean System: Fresh Ubuntu install or VM
Restore Docker Compose Files: From config backup
Restore PostgreSQL: From latest database dump
Restore Volumes: From latest volume backups
Start Services: docker compose up -d
Validate Functionality: Test critical workflows (n8n executions, Home Assistant automation, Frigate detection)
Measure Recovery Time: Document RTO (Recovery Time Objective)

Success Criteria:

All services start without errors
n8n workflows executable
Home Assistant automation triggers
Frigate cameras recording
Omnia dashboard loads correctly

Recovery Time Objectives (RTO)

Target RTOs:

Component	RTO
PostgreSQL	30 minutes
Home Assistant	15 minutes
n8n Workflows	20 minutes
Frigate Cameras	10 minutes
Full System	2 hours

Recovery Point Objectives (RPO):

Data Type	RPO
PostgreSQL	24 hours (daily backup)
Configurations	24 hours (daily backup)
Docker Volumes	7 days (weekly backup)
Frigate Recordings	N/A (rolling retention)

Storage Requirements

Backup Storage Sizing

Minimum Capacity:

PostgreSQL Dumps: 50GB (14 days × ~3GB/day)
Volume Snapshots: 100GB (7 days × ~15GB/week)
Configuration Backups: 10GB (30 days × ~300MB/day)
Total: 200GB minimum

Recommended Capacity:

500GB-1TB for local backup storage
Separate physical drive from primary storage (SATA SSD)
Off-site storage equal to local capacity

Primary Storage Requirements

Service	Data Type	Size
PostgreSQL	Database	20-100GB
Frigate Recordings	Video	500GB-2TB
Home Assistant Config	Files	1-5GB
Docker Volumes	Mixed	10-50GB
Total		600GB-2.2TB

Nexus Host Configuration:

NVMe SSD: 512GB (OS, containers, PostgreSQL)
SATA SSD: 2TB (Frigate recordings, backups)

Best Practices

Automate Everything: Use systemd timers or cron for hands-free backups

Test Restores: Regularly verify backups are restorable (quarterly DR drills)

Monitor Backup Health: Alert on backup failures, check file sizes for corruption

Version Control Configs: Use Git for Home Assistant/Node-RED/n8n configs to track changes

Off-Site Copies: Always maintain at least one off-site backup copy

Encryption: Encrypt backups containing credentials or sensitive data before off-site storage

Document Procedures: Keep restoration procedures updated and accessible (not only on Nexus!)

Incremental Backups: Use tools like pg_basebackup or restic for efficient incremental backups

Retention Alignment: Match retention policies to recovery requirements and storage capacity

Nexus Architecture - Overall system design
Container Reference - Services and data volumes
Hardware Specifications - Storage hardware details
Portability & Migrations - Volume migration procedures

Data & Backups

Data Storage Architecture

Storage Types

Data Locations by Service

Core Infrastructure

Workflows & Automation

HALO Applications

Monitoring & Visualization

Retention Policies

Hot Data (Active Operational Data)

Warm Data (Historical Backups)

Cold Data (Archival)

Backup Strategies

PostgreSQL Backups

Docker Volume Backups

Configuration Backups

Automated Backup System

Systemd Timers

Backup Monitoring

Restoration Procedures

PostgreSQL Restore

Docker Volume Restore

Configuration Restore

Off-Site Backups

Backup Strategies

Encryption

Disaster Recovery Testing

Quarterly DR Drills

Recovery Time Objectives (RTO)

Storage Requirements

Backup Storage Sizing

Primary Storage Requirements

Best Practices

Related Documentation