Chapter 17 — Maintenance & Operations

Launching is just the beginning. Day-2 operations — keeping the system running, updated, and healthy — is where most of the work lives.

OS & Dependency Updates

# Automatic security updates (Ubuntu)
sudo apt install unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades

# /etc/apt/apt.conf.d/50unattended-upgrades
Unattended-Upgrade::Allowed-Origins {
    "${distro_id}:${distro_codename}-security";
};
Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-Time "03:00";

# Application dependency updates — use Dependabot or Renovate
# .github/dependabot.yml
version: 2
updates:
  - package-ecosystem: "npm"
    directory: "/"
    schedule:
      interval: "weekly"
    open-pull-requests-limit: 5

Zero-Downtime Deployments

# Method 1: Rolling restart with multiple instances behind LB
# Deploy to instance 1, health check passes, deploy to instance 2...

# Method 2: Docker with Nginx upstream reload
# deploy.sh
docker pull myregistry/myapp:$NEW_VERSION
docker run -d --name myapp-new -p 3001:3000 myregistry/myapp:$NEW_VERSION

# Wait for health check
until curl -sf http://localhost:3001/health; do sleep 1; done

# Switch Nginx upstream
sed -i 's/127.0.0.1:3000/127.0.0.1:3001/' /etc/nginx/conf.d/upstream.conf
nginx -s reload

# Stop old container
docker stop myapp-old && docker rm myapp-old
docker rename myapp-new myapp-old

Rollback Procedures

# Docker rollback — instant (previous image still cached)
docker stop myapp
docker run -d --name myapp -p 3000:3000 myregistry/myapp:PREVIOUS_VERSION

# Kubernetes rollback
kubectl rollout undo deployment/myapp
kubectl rollout status deployment/myapp    # Watch progress

# Database rollback — this is the hard part
# Always make migrations reversible:
# - migration_001_add_column.up.sql
# - migration_001_add_column.down.sql
# Run: migrate down 1
Database migrations are the #1 cause of failed rollbacks. Rules:
• Never drop columns in the same deploy that removes the code using them
• Use expand-contract pattern: add new column → deploy code using both → remove old column
• Always write down migrations
• Test migrations against a production-size dataset before deploying

Disaster Recovery Plan

MetricDefinitionYour Target
RTO (Recovery Time Objective)Max acceptable downtimee.g., 1 hour
RPO (Recovery Point Objective)Max acceptable data losse.g., 15 minutes
# Disaster Recovery Runbook Template:

## Scenario: Complete server failure
1. Spin up new server from Terraform (5 min)
2. Restore latest database backup (10 min)
3. Deploy latest Docker image (2 min)
4. Update DNS to new server IP (5 min + propagation)
5. Verify application health
6. Notify stakeholders

## Scenario: Database corruption
1. Stop application (prevent further writes)
2. Identify last good backup
3. Restore to point-in-time (RDS: use PITR)
4. Verify data integrity
5. Restart application
6. Post-mortem: identify root cause

## Scenario: Security breach
1. Isolate affected systems (revoke access, block IPs)
2. Preserve evidence (don't destroy logs)
3. Rotate ALL credentials and secrets
4. Assess scope of breach
5. Patch vulnerability
6. Notify affected users (legal requirement in many jurisdictions)
7. Post-mortem and remediation plan

Runbooks & On-Call