rb/od-backup-schedule.md

Runbook: Open Dental — Schedule & monitor backups (Windows)

Applies to: The Open Dental database server, once od-db-backup.md is proven to run by hand. Goal: Make the backup run automatically every day, replicate it off-site, and monitor it so a silent failure gets noticed within a day — not the day you need a restore.

Open Dental's floor is at least one backup per day, with a combination of methods and at least one automated. (Open Dental — Backups) This runbook automates and watches the cold backup.

Important

Monitoring is the half everyone skips. A backup job that silently stopped, a full disk, or a powered-off server sends no error email — the absence of failure looks exactly like success. The only reliable signal is a positive heartbeat that goes missing (a dead-man's-switch). Build that, or you don't have monitoring.

Placeholders:

Placeholder	Meaning
<DEST>	Backup destination root (holds the od-backup-<timestamp> folders)
<BACKUP_SCRIPT>	Filled-in local copy of od-db-backup.ps1 (paths baked in) — private tier
<CHECK_SCRIPT>	Filled-in local copy of od-backup-check.ps1 (<DEST> + heartbeat baked in) — private tier
<HEARTBEAT_URL>	Dead-man's-switch ping URL (healthchecks.io, Uptime Kuma push, RMM, etc.) — private tier

---

The jobs and their cadence

Job	What	When	Runbook
1. Cold backup	Stop DB, copy data + images, restart	Nightly, off-hours	od-db-backup.md
2. Off-site upload	Replicate <DEST> to cloud/immutable	After job 1 finishes	od-db-backup.md §6
3. Health check	Verify newest backup + heartbeat ping	After job 1 finishes	this runbook + od-backup-check.ps1
4. Test-restore	Full restore into isolated VM	Monthly	od-backup-verify.md

Stagger them so nothing reads a half-written folder, e.g. backup 23:30 → check 00:45 → off-site upload 01:00.

Why a scheduled local copy, not irm | iex

irm | iex prompts interactively — it can't run unattended. For scheduling, keep a filled-in copy of each script (<BACKUP_SCRIPT>, <CHECK_SCRIPT>) with <DEST>, service name, and paths baked in, stored in the private tier (never committed here). The public scripts stay the interactive/spot-check version.

---

1. Schedule the nightly cold backup

Register <BACKUP_SCRIPT> to run as SYSTEM, highest privileges, off-hours. From an elevated prompt:

schtasks /Create /TN "OD Nightly Cold Backup" ^
  /TR "powershell -NoProfile -ExecutionPolicy Bypass -File <BACKUP_SCRIPT>" ^
  /SC DAILY /ST 23:30 /RU SYSTEM /RL HIGHEST

• <BACKUP_SCRIPT> is the filled-in copy — it must not prompt (no Read-Host for the scheduled path) and must still do the stop → verify-stopped → copy → always-restart sequence.
• Ensure the task is set to run whether or not a user is logged on and, if the server sleeps, wake the computer to run (Task Scheduler → task → Conditions).
• Confirm the server actually stays on overnight (disable sleep/hibernate on the server).

2. Schedule the off-site upload (after the backup)

Point your off-site tool (Duplicati → Backblaze B2, rclone, Veeam, etc.) at <DEST> — it uploads this backup, it does not re-run the stop/copy against the live database. Schedule it after job 1 completes and stagger the start. Enable that tool's own email/report and object-lock/immutability for the ransomware-resistant off-site copy. Details: od-db-backup.md §6.

3. Schedule the health check (and heartbeat)

Register <CHECK_SCRIPT> to run shortly after the backup window. It verifies the newest od-backup-<timestamp> is fresh, complete, and sensibly sized, appends to <DEST>\backup-check.log, and — when healthy — pings <HEARTBEAT_URL>.

schtasks /Create /TN "OD Backup Health Check" ^
  /TR "powershell -NoProfile -ExecutionPolicy Bypass -File <CHECK_SCRIPT>" ^
  /SC DAILY /ST 00:45 /RU SYSTEM /RL HIGHEST

The check is read-only — no DB, no service, no file changes — so it's safe to run any time, including a manual spot-check:

irm https://gitea.ivangodwin.com/ops/rb/raw/branch/main/scripts/od-backup-check.ps1 | iex

---

Monitoring — the three layers of "is it actually working?"

Use all three; each catches what the others miss.

Layer 1 — Did the job run? (Task Scheduler)

Last Run Result = 0x0 and a recent Last Run Time on both tasks:

Get-ScheduledTaskInfo -TaskName "OD Nightly Cold Backup"
Get-ScheduledTaskInfo -TaskName "OD Backup Health Check"

Enable All Tasks History in Task Scheduler so you can see misfires. Catches: task disabled, wrong credentials, server was off.

Layer 2 — Is the output good? (the health check)

od-backup-check.ps1 / <CHECK_SCRIPT> confirms the newest backup is:

• Fresh — written within 24 h (else the job silently stopped),
• Complete — has data\, OpenDentImages\, MANIFEST.txt, and a backup.log that ends in success with no ERROR/CRITICAL,
• Sane size — data\ isn't near-empty and isn't a fraction of the prior run (catches truncation / a filling disk).

Read <DEST>\backup-check.log for the running PASS/FAIL trail. Catches: partial copies, missing images, service that didn't restart, dying disk.

Layer 3 — Dead-man's-switch (the one that catches everything)

Register a check with an external monitor — healthchecks.io, Uptime Kuma (push), or your RMM — that expects a daily ping. <CHECK_SCRIPT> pings <HEARTBEAT_URL> only when the backup is healthy. If the backup breaks, the check fails, the script crashes, or the whole server is offline, the ping never arrives and the monitor alerts you.

• Set the monitor's period to ~1 day with a grace window past your backup+check schedule.
• healthchecks.io users: <CHECK_SCRIPT> can hit <HEARTBEAT_URL>/fail on failure for an immediate alert instead of waiting out the grace period.
• This is the layer that turns "no news" into an actual alarm. Without it, a dead backup is invisible until a restore fails.

Layer 4 — Off-site tool's own report

Your cloud tool (Duplicati/Veeam/rclone wrapper) should send its own success/failure summary and expose versions/immutability in the provider console. Confirms the copy actually left the building.

---

On failure — triage

When Layer 2/3 flags a problem, in rough order:

1. Is the database up? Get-Service <DB_SERVICE> — if the backup died mid-run, confirm the service restarted (the script's finally should have; verify). The practice being able to work comes first.
2. Destination full / offline? Free space on <DEST>; is the disk/UNC reachable? Prune old od-backup-<timestamp> generations if space-bound.
3. Partial/most-recent folder incomplete? Check that run's backup.log for the ERROR/CRITICAL line; re-run the backup by hand (od-db-backup.md).
4. Version drift? A recent Open Dental/MySQL update can change paths — reconcile against MANIFEST.txt.
5. Off-site not uploading? Check the cloud tool's log and that it sources <DEST> (not the live datadir).

Then re-run the health check and confirm the heartbeat goes green.

Records / evidence

Keep the <DEST>\backup-check.log, the monitor's uptime history, and the monthly test-restore results (od-backup-verify.md) together. That trail is your DR evidence for E&O / cyber insurance and HIPAA contingency-plan testing.

---

Security note

<HEARTBEAT_URL> is a capability — anyone with it can spoof "backup healthy." Treat it as a secret: keep it in the private tier, out of this repo, out of screenshots. Don't let monitor check names or heartbeat URLs encode a client's identity. The <DEST> folders and <BACKUP_SCRIPT>/<CHECK_SCRIPT> may reference real paths — keep the filled-in copies private and the destination encrypted/access-controlled (it holds PHI).

References

• Open Dental manual — Backups (daily minimum; automated + combined methods): https://opendental.com/manual/backups.html
• Microsoft — schtasks / Scheduled Tasks: https://learn.microsoft.com/windows-server/administration/windows-commands/schtasks
• healthchecks.io — dead-man's-switch cron monitoring: https://healthchecks.io/
• Companion runbooks — od-db-backup.md (produce the backup), od-backup-verify.md (test-restore)