rb/od-backup-schedule.md

# Runbook: Open Dental — Schedule & monitor backups (Windows)

**Applies to:** The Open Dental database server, once [`od-db-backup.md`](od-db-backup.md) is proven to run by hand.
**Goal:** Make the backup run **automatically every day**, replicate it **off-site**, and **monitor** it so a silent failure gets noticed within a day — not the day you need a restore.

Open Dental's floor is **at least one backup per day**, with a combination of methods and at least one **automated**. ([Open Dental — Backups](https://opendental.com/manual/backups.html)) This runbook automates and watches the cold backup.

> [!IMPORTANT]
> **Monitoring is the half everyone skips.** A backup job that silently stopped, a full disk, or a powered-off server sends **no error email** — the absence of failure looks exactly like success. The only reliable signal is a **positive heartbeat that goes missing** (a dead-man's-switch). Build that, or you don't have monitoring.

**Placeholders:**

| Placeholder | Meaning |
|---|---|
| `<DEST>` | Backup destination root (holds the `od-backup-<timestamp>` folders) |
| `<BACKUP_SCRIPT>` | Filled-in local copy of `od-db-backup.ps1` (paths baked in) — **private tier** |
| `<CHECK_SCRIPT>` | Filled-in local copy of `od-backup-check.ps1` (`<DEST>` + heartbeat baked in) — **private tier** |
| `<HEARTBEAT_URL>` | Dead-man's-switch ping URL (healthchecks.io, Uptime Kuma push, RMM, etc.) — **private tier** |

---

## The jobs and their cadence

| Job | What | When | Runbook |
|---|---|---|---|
| **1. Cold backup** | Stop DB, copy data + images, restart | Nightly, off-hours | [`od-db-backup.md`](od-db-backup.md) |
| **2. Off-site upload** | Replicate `<DEST>` to cloud/immutable | After job 1 finishes | [`od-db-backup.md` §6](od-db-backup.md) |
| **3. Health check** | Verify newest backup + heartbeat ping | After job 1 finishes | this runbook + [`od-backup-check.ps1`](scripts/od-backup-check.ps1) |
| **4. Test-restore** | Full restore into isolated VM | Monthly | [`od-backup-verify.md`](od-backup-verify.md) |

Stagger them so nothing reads a half-written folder, e.g. **backup 23:30 → check 00:45 → off-site upload 01:00**.

## Why a scheduled *local copy*, not `irm | iex`

`irm | iex` prompts interactively — it can't run unattended. For scheduling, keep a **filled-in copy** of each script (`<BACKUP_SCRIPT>`, `<CHECK_SCRIPT>`) with `<DEST>`, service name, and paths baked in, stored in the **private tier** (never committed here). The public scripts stay the interactive/spot-check version.

---

## 1. Schedule the nightly cold backup

Register `<BACKUP_SCRIPT>` to run as **SYSTEM**, highest privileges, off-hours. From an elevated prompt:

```
schtasks /Create /TN "OD Nightly Cold Backup" ^
  /TR "powershell -NoProfile -ExecutionPolicy Bypass -File <BACKUP_SCRIPT>" ^
  /SC DAILY /ST 23:30 /RU SYSTEM /RL HIGHEST
```

- `<BACKUP_SCRIPT>` is the filled-in copy — it must **not** prompt (no `Read-Host` for the scheduled path) and must still do the stop → verify-stopped → copy → **always-restart** sequence.
- Ensure the task is set to **run whether or not a user is logged on** and, if the server sleeps, **wake the computer to run** (Task Scheduler → task → *Conditions*).
- Confirm the server actually **stays on** overnight (disable sleep/hibernate on the server).

## 2. Schedule the off-site upload (after the backup)

Point your off-site tool (Duplicati → Backblaze B2, `rclone`, Veeam, etc.) at **`<DEST>`** — it uploads *this backup*, it does **not** re-run the stop/copy against the live database. Schedule it **after** job 1 completes and stagger the start. Enable that tool's **own** email/report and **object-lock/immutability** for the ransomware-resistant off-site copy. Details: [`od-db-backup.md` §6](od-db-backup.md).

## 3. Schedule the health check (and heartbeat)

Register `<CHECK_SCRIPT>` to run shortly **after** the backup window. It verifies the newest `od-backup-<timestamp>` is fresh, complete, and sensibly sized, appends to `<DEST>\backup-check.log`, and — when healthy — pings `<HEARTBEAT_URL>`.

```
schtasks /Create /TN "OD Backup Health Check" ^
  /TR "powershell -NoProfile -ExecutionPolicy Bypass -File <CHECK_SCRIPT>" ^
  /SC DAILY /ST 00:45 /RU SYSTEM /RL HIGHEST
```

The check is **read-only** — no DB, no service, no file changes — so it's safe to run any time, including a manual spot-check:

```
irm https://gitea.ivangodwin.com/ops/rb/raw/branch/main/scripts/od-backup-check.ps1 | iex
```

---

## Monitoring — the three layers of "is it actually working?"

Use all three; each catches what the others miss.

### Layer 1 — Did the job run? (Task Scheduler)

`Last Run Result = 0x0` and a recent `Last Run Time` on both tasks:

```
Get-ScheduledTaskInfo -TaskName "OD Nightly Cold Backup"
Get-ScheduledTaskInfo -TaskName "OD Backup Health Check"
```

Enable **All Tasks History** in Task Scheduler so you can see misfires. Catches: task disabled, wrong credentials, server was off.

### Layer 2 — Is the output good? (the health check)

`od-backup-check.ps1` / `<CHECK_SCRIPT>` confirms the newest backup is:

- **Fresh** — written within 24 h (else the job silently stopped),
- **Complete** — has `data\`, `OpenDentImages\`, `MANIFEST.txt`, and a `backup.log` that ends in success with no `ERROR`/`CRITICAL`,
- **Sane size** — `data\` isn't near-empty and isn't a fraction of the prior run (catches truncation / a filling disk).

Read `<DEST>\backup-check.log` for the running PASS/FAIL trail. Catches: partial copies, missing images, service that didn't restart, dying disk.

### Layer 3 — Dead-man's-switch (the one that catches everything)

Register a check with an **external** monitor — [healthchecks.io](https://healthchecks.io), Uptime Kuma (push), or your RMM — that expects a daily ping. `<CHECK_SCRIPT>` pings `<HEARTBEAT_URL>` **only when the backup is healthy**. If the backup breaks, the check fails, the script crashes, or **the whole server is offline**, the ping never arrives and the monitor alerts you.

- Set the monitor's **period to ~1 day** with a grace window past your backup+check schedule.
- healthchecks.io users: `<CHECK_SCRIPT>` can hit `<HEARTBEAT_URL>/fail` on failure for an **immediate** alert instead of waiting out the grace period.
- This is the layer that turns "no news" into an actual alarm. Without it, a dead backup is invisible until a restore fails.

### Layer 4 — Off-site tool's own report

Your cloud tool (Duplicati/Veeam/rclone wrapper) should send its **own** success/failure summary and expose versions/immutability in the provider console. Confirms the copy actually left the building.

---

## On failure — triage

When Layer 2/3 flags a problem, in rough order:

1. **Is the database up?** `Get-Service <DB_SERVICE>` — if the backup died mid-run, confirm the service **restarted** (the script's `finally` should have; verify). The practice being able to work comes first.
2. **Destination full / offline?** Free space on `<DEST>`; is the disk/UNC reachable? Prune old `od-backup-<timestamp>` generations if space-bound.
3. **Partial/most-recent folder incomplete?** Check that run's `backup.log` for the `ERROR`/`CRITICAL` line; re-run the backup by hand ([`od-db-backup.md`](od-db-backup.md)).
4. **Version drift?** A recent Open Dental/MySQL update can change paths — reconcile against `MANIFEST.txt`.
5. **Off-site not uploading?** Check the cloud tool's log and that it sources `<DEST>` (not the live datadir).

Then re-run the health check and confirm the heartbeat goes green.

## Records / evidence

Keep the `<DEST>\backup-check.log`, the monitor's uptime history, and the monthly **test-restore** results ([`od-backup-verify.md`](od-backup-verify.md)) together. That trail is your DR evidence for E&O / cyber insurance and HIPAA contingency-plan testing.

---

## Security note

`<HEARTBEAT_URL>` is a capability — anyone with it can spoof "backup healthy." Treat it as a secret: keep it in the **private tier**, out of this repo, out of screenshots. Don't let monitor check names or heartbeat URLs encode a client's identity. The `<DEST>` folders and `<BACKUP_SCRIPT>`/`<CHECK_SCRIPT>` may reference real paths — keep the filled-in copies private and the destination encrypted/access-controlled (it holds PHI).

## References

- Open Dental manual — Backups (daily minimum; automated + combined methods): <https://opendental.com/manual/backups.html>
- Microsoft — `schtasks` / Scheduled Tasks: <https://learn.microsoft.com/windows-server/administration/windows-commands/schtasks>
- healthchecks.io — dead-man's-switch cron monitoring: <https://healthchecks.io/>
- Companion runbooks — [`od-db-backup.md`](od-db-backup.md) (produce the backup), [`od-backup-verify.md`](od-backup-verify.md) (test-restore)