# Runbook: Open Dental — Schedule & monitor backups (Windows) **Applies to:** The Open Dental database server, once [`od-db-backup.md`](od-db-backup.md) is proven to run by hand. **Goal:** Make the backup run **automatically every day**, replicate it **off-site**, and **monitor** it so a silent failure gets noticed within a day — not the day you need a restore. Open Dental's floor is **at least one backup per day**, with a combination of methods and at least one **automated**. ([Open Dental — Backups](https://opendental.com/manual/backups.html)) This runbook automates and watches the cold backup. > [!IMPORTANT] > **Monitoring is the half everyone skips.** A backup job that silently stopped, a full disk, or a powered-off server sends **no error email** — the absence of failure looks exactly like success. The only reliable signal is a **positive heartbeat that goes missing** (a dead-man's-switch). Build that, or you don't have monitoring. **Placeholders:** | Placeholder | Meaning | |---|---| | `` | Backup destination root (holds the `od-backup-` folders) | | `` | Filled-in local copy of `od-db-backup.ps1` (paths baked in) — **private tier** | | `` | Filled-in local copy of `od-backup-check.ps1` (`` + heartbeat baked in) — **private tier** | | `` | Dead-man's-switch ping URL (healthchecks.io, Uptime Kuma push, RMM, etc.) — **private tier** | --- ## The jobs and their cadence | Job | What | When | Runbook | |---|---|---|---| | **1. Cold backup** | Stop DB, copy data + images, restart | Nightly, off-hours | [`od-db-backup.md`](od-db-backup.md) | | **2. Off-site upload** | Replicate `` to cloud/immutable | After job 1 finishes | [`od-db-backup.md` §6](od-db-backup.md) | | **3. Health check** | Verify newest backup + heartbeat ping | After job 1 finishes | this runbook + [`od-backup-check.ps1`](scripts/od-backup-check.ps1) | | **4. Test-restore** | Full restore into isolated VM | Monthly | [`od-backup-verify.md`](od-backup-verify.md) | Stagger them so nothing reads a half-written folder, e.g. **backup 23:30 → check 00:45 → off-site upload 01:00**. ## Why a scheduled *local copy*, not `irm | iex` `irm | iex` prompts interactively — it can't run unattended. For scheduling, keep a **filled-in copy** of each script (``, ``) with ``, service name, and paths baked in, stored in the **private tier** (never committed here). The public scripts stay the interactive/spot-check version. --- ## 1. Schedule the nightly cold backup Register `` to run as **SYSTEM**, highest privileges, off-hours. From an elevated prompt: ``` schtasks /Create /TN "OD Nightly Cold Backup" ^ /TR "powershell -NoProfile -ExecutionPolicy Bypass -File " ^ /SC DAILY /ST 23:30 /RU SYSTEM /RL HIGHEST ``` - `` is the filled-in copy — it must **not** prompt (no `Read-Host` for the scheduled path) and must still do the stop → verify-stopped → copy → **always-restart** sequence. - Ensure the task is set to **run whether or not a user is logged on** and, if the server sleeps, **wake the computer to run** (Task Scheduler → task → *Conditions*). - Confirm the server actually **stays on** overnight (disable sleep/hibernate on the server). ## 2. Schedule the off-site upload (after the backup) Point your off-site tool (Duplicati → Backblaze B2, `rclone`, Veeam, etc.) at **``** — it uploads *this backup*, it does **not** re-run the stop/copy against the live database. Schedule it **after** job 1 completes and stagger the start. Enable that tool's **own** email/report and **object-lock/immutability** for the ransomware-resistant off-site copy. Details: [`od-db-backup.md` §6](od-db-backup.md). ## 3. Schedule the health check (and heartbeat) Register `` to run shortly **after** the backup window. It verifies the newest `od-backup-` is fresh, complete, and sensibly sized, appends to `\backup-check.log`, and — when healthy — pings ``. ``` schtasks /Create /TN "OD Backup Health Check" ^ /TR "powershell -NoProfile -ExecutionPolicy Bypass -File " ^ /SC DAILY /ST 00:45 /RU SYSTEM /RL HIGHEST ``` The check is **read-only** — no DB, no service, no file changes — so it's safe to run any time, including a manual spot-check: ``` irm https://gitea.ivangodwin.com/ops/rb/raw/branch/main/scripts/od-backup-check.ps1 | iex ``` --- ## Monitoring — the three layers of "is it actually working?" Use all three; each catches what the others miss. ### Layer 1 — Did the job run? (Task Scheduler) `Last Run Result = 0x0` and a recent `Last Run Time` on both tasks: ``` Get-ScheduledTaskInfo -TaskName "OD Nightly Cold Backup" Get-ScheduledTaskInfo -TaskName "OD Backup Health Check" ``` Enable **All Tasks History** in Task Scheduler so you can see misfires. Catches: task disabled, wrong credentials, server was off. ### Layer 2 — Is the output good? (the health check) `od-backup-check.ps1` / `` confirms the newest backup is: - **Fresh** — written within 24 h (else the job silently stopped), - **Complete** — has `data\`, `OpenDentImages\`, `MANIFEST.txt`, and a `backup.log` that ends in success with no `ERROR`/`CRITICAL`, - **Sane size** — `data\` isn't near-empty and isn't a fraction of the prior run (catches truncation / a filling disk). Read `\backup-check.log` for the running PASS/FAIL trail. Catches: partial copies, missing images, service that didn't restart, dying disk. ### Layer 3 — Dead-man's-switch (the one that catches everything) Register a check with an **external** monitor — [healthchecks.io](https://healthchecks.io), Uptime Kuma (push), or your RMM — that expects a daily ping. `` pings `` **only when the backup is healthy**. If the backup breaks, the check fails, the script crashes, or **the whole server is offline**, the ping never arrives and the monitor alerts you. - Set the monitor's **period to ~1 day** with a grace window past your backup+check schedule. - healthchecks.io users: `` can hit `/fail` on failure for an **immediate** alert instead of waiting out the grace period. - This is the layer that turns "no news" into an actual alarm. Without it, a dead backup is invisible until a restore fails. ### Layer 4 — Off-site tool's own report Your cloud tool (Duplicati/Veeam/rclone wrapper) should send its **own** success/failure summary and expose versions/immutability in the provider console. Confirms the copy actually left the building. --- ## On failure — triage When Layer 2/3 flags a problem, in rough order: 1. **Is the database up?** `Get-Service ` — if the backup died mid-run, confirm the service **restarted** (the script's `finally` should have; verify). The practice being able to work comes first. 2. **Destination full / offline?** Free space on ``; is the disk/UNC reachable? Prune old `od-backup-` generations if space-bound. 3. **Partial/most-recent folder incomplete?** Check that run's `backup.log` for the `ERROR`/`CRITICAL` line; re-run the backup by hand ([`od-db-backup.md`](od-db-backup.md)). 4. **Version drift?** A recent Open Dental/MySQL update can change paths — reconcile against `MANIFEST.txt`. 5. **Off-site not uploading?** Check the cloud tool's log and that it sources `` (not the live datadir). Then re-run the health check and confirm the heartbeat goes green. ## Records / evidence Keep the `\backup-check.log`, the monitor's uptime history, and the monthly **test-restore** results ([`od-backup-verify.md`](od-backup-verify.md)) together. That trail is your DR evidence for E&O / cyber insurance and HIPAA contingency-plan testing. --- ## Security note `` is a capability — anyone with it can spoof "backup healthy." Treat it as a secret: keep it in the **private tier**, out of this repo, out of screenshots. Don't let monitor check names or heartbeat URLs encode a client's identity. The `` folders and ``/`` may reference real paths — keep the filled-in copies private and the destination encrypted/access-controlled (it holds PHI). ## References - Open Dental manual — Backups (daily minimum; automated + combined methods): - Microsoft — `schtasks` / Scheduled Tasks: - healthchecks.io — dead-man's-switch cron monitoring: - Companion runbooks — [`od-db-backup.md`](od-db-backup.md) (produce the backup), [`od-backup-verify.md`](od-backup-verify.md) (test-restore)