The Fuckup

I wanted to migrate from Hugo to WordPress. Seemed simple. It wasn’t.

I spent hours trying to install WordPress on my T620 thin client. Each time I “fixed” it, something else broke. By the end, both WordPress AND my Hugo site were completely fucked. Nginx wouldn’t start. The main blog was down. I had wasted hours on something that should have taken 30 minutes.

Then I remembered: I had full backups.

Without them, I would have lost everything.

What Went Wrong

The problem was trying to run nginx with conflicting server blocks. The main /etc/nginx/nginx.conf had a default server listening on port 80. My WordPress config tried to add another. Then I tried disabling the default. Then I accidentally commented out the include statements that load /etc/nginx/conf.d/.

Each fix made it worse. Nginx wouldn’t even syntax-check anymore.

The real lesson: never try to fix infrastructure without backups in place first.

The Backup Setup I Should Have Used From The Start

Here’s what saved me.

Cron Job: Automated Weekly Backups (T620 → G4)

Every Sunday at 3am, a cron job runs on T620 and backs up:

  • /etc/nginx/ - all web configs
  • /etc/systemd/system/ - all systemd services
  • ~/hugo/kingtolga/ - the entire Hugo site
  • /var/www/kingtolga/ - the compiled HTML

Old backups older than 28 days auto-delete.

Set up the cron job on T620:

echo '0 3 * * 0 root cp /etc/fstab /root/backup-staging/fstab && tar -czf /root/backup-$(date +\%Y\%m\%d).tar.gz /root/backup-staging /etc/ssh /etc/samba /etc/nginx /etc/systemd/system /home/tolga/hugo /var/www && chmod 600 /root/backup-$(date +\%Y\%m\%d).tar.gz && find /root -maxdepth 1 -name "backup-*.tar.gz" -mtime +28 -delete' | sudo tee /etc/cron.d/config-backup

Verify it’s set:

cat /etc/cron.d/config-backup
sudo systemctl status crond

Manual Backup Command (Run Anytime)

If you want to backup right now without waiting for Sunday:

sudo cp /etc/fstab /root/backup-staging/fstab
sudo tar -czf /root/backup-$(date +%Y%m%d).tar.gz /root/backup-staging /etc/ssh /etc/samba /etc/nginx /etc/systemd/system /home/tolga/hugo /var/www
sudo chmod 600 /root/backup-$(date +%Y%m%d).tar.gz

Verify:

sudo ls -lah /root/backup-*.tar.gz
sudo tar -tzf /root/backup-$(date +%Y%m%d).tar.gz | head -30

Copy Backups to G4 (From T620)

Every backup sits on T620. But if T620 dies, you’ve got nothing. Copy to G4:

sudo mkdir -p /root/backup-staging
sudo scp /root/backup-*.tar.gz tolga@192.168.0.xxx:/home/tolga/HUGO-BACKUP/

Verify on G4:

ls -lah ~/HUGO-BACKUP/

Systemd Services for Automatic Backups

If you want automated backups via systemd instead of cron:

Create /etc/systemd/system/backup.timer:

sudo bash << 'EOF'
cat > /etc/systemd/system/backup.timer << 'TIMER'
[Unit]
Description=My weekly backup timer
Requires=backup.service

[Timer]
OnBootSec=5min
OnUnitActiveSec=1w
Persistent=true

[Install]
WantedBy=timers.target
TIMER

cat > /etc/systemd/system/backup.service << 'SERVICE'
[Unit]
Description=T620 weekly backup to G4
After=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup-to-g4.sh

[Install]
WantedBy=multi-user.target
SERVICE

systemctl daemon-reload
systemctl enable --now backup.timer
systemctl status backup.timer
EOF

Create the backup script /usr/local/bin/backup-to-g4.sh:

sudo bash << 'EOF'
cat > /usr/local/bin/backup-to-g4.sh << 'SCRIPT'
#!/bin/bash
set -e

backup_file="/root/backup-$(date +%Y%m%d).tar.gz"

# Copy fstab to staging
cp /etc/fstab /root/backup-staging/fstab

# Create archive
tar -czf "$backup_file" \
  /root/backup-staging \
  /etc/ssh \
  /etc/samba \
  /etc/nginx \
  /etc/systemd/system \
  /home/tolga/hugo \
  /var/www

chmod 600 "$backup_file"

# Copy to G4
scp "$backup_file" tolga@192.168.0.xxx:/home/tolga/HUGO-BACKUP/

# Clean old backups (older than 28 days)
find /root -maxdepth 1 -name "backup-*.tar.gz" -mtime +28 -delete

echo "Backup complete: $backup_file"
SCRIPT

chmod 755 /usr/local/bin/backup-to-g4.sh
EOF

Enable and test:

sudo systemctl start backup.timer
sudo systemctl status backup.timer

How to Restore From Backups

When things go catastrophically wrong (like they did for me):

Restore Everything on T620 from G4 Backups

# On T620, pull backups from G4
cd ~
scp tolga@192.168.0.xxx:/home/tolga/HUGO-BACKUP/backup-*.tar.gz ~/

# Restore each piece
sudo tar -xzf ~/backup-*.tar.gz -C /

# Restart services
sudo systemctl daemon-reload
sudo systemctl restart nginx
sudo systemctl restart tailscale-funnel

# Verify Hugo site
curl http://localhost

# Verify Tailscale Funnel
curl https://jacksparrow2.tail9e758e.ts.net/

Or Restore Just Specific Parts

# Just restore Hugo
tar -xzf ~/backup-*.tar.gz -C ~/ home/tolga/hugo/

# Just restore nginx
sudo tar -xzf ~/backup-*.tar.gz -C / etc/nginx/

# Just restore systemd
sudo tar -xzf ~/backup-*.tar.gz -C / etc/systemd/system/

Scripts to Keep Handy

T620 → G4 Backup Script (t620-to-g4-backup.sh):

#!/bin/bash
# Run this anytime to backup T620 to G4

echo "Creating backup..."
sudo cp /etc/fstab /root/backup-staging/fstab
sudo tar -czf /root/backup-$(date +%Y%m%d).tar.gz \
  /root/backup-staging \
  /etc/ssh \
  /etc/samba \
  /etc/nginx \
  /etc/systemd/system \
  /home/tolga/hugo \
  /var/www

sudo chmod 600 /root/backup-*.tar.gz

echo "Sending to G4..."
sudo scp /root/backup-*.tar.gz tolga@192.168.0.27:/home/tolga/HUGO-BACKUP/

echo "Done. Backups on G4:"
ls -lah ~/HUGO-BACKUP/

G4 → T620 Restore Script (g4-to-t620-restore.sh):

#!/bin/bash
# Run this on T620 to restore from G4

echo "Pulling backups from G4..."
cd ~
scp tolga@192.168.0.xxx:/home/tolga/HUGO-BACKUP/backup-*.tar.gz ./

echo "Restoring nginx..."
sudo tar -xzf backup-*.tar.gz -C / etc/nginx

echo "Restoring systemd services..."
sudo tar -xzf backup-*.tar.gz -C / etc/systemd/system

echo "Restoring Hugo site..."
tar -xzf backup-*.tar.gz -C ~/ home/tolga/hugo

echo "Restoring www..."
sudo tar -xzf backup-*.tar.gz -C / var/www

echo "Reloading systemd..."
sudo systemctl daemon-reload
sudo systemctl restart nginx
sudo systemctl restart tailscale-funnel

echo "Done. Testing site..."
sleep 2
curl http://localhost

Save these scripts to /usr/local/bin/ on T620 and G4, make them executable:

sudo chmod 755 /usr/local/bin/t620-to-g4-backup.sh
sudo chmod 755 /usr/local/bin/g4-to-t620-restore.sh

The Lesson

I broke my Hugo blog because I didn’t have a tested restore procedure in place before things went wrong.

The rule: If you don’t have a backup and a tested way to restore from it, you don’t have a backup.

Now I do. Every Sunday at 3am, T620 backs itself up. Once a week manually, I push those backups to G4. If T620 explodes tomorrow, I restore in 15 minutes.

Never skip backups. Never.

-Tolga