Add od-backup-schedule runbook + od-backup-check script: schedule & monitor backups

Automate the nightly cold backup (Task Scheduler, filled-in local copy in
private tier), stagger the off-site upload, and MONITOR with three layers:
Task Scheduler last-run, a read-only health check (freshness/completeness/
size), and a dead-man's-switch heartbeat that pings an external monitor only
on success so silent failures and offline servers get caught. od-backup-check.ps1
is read-only (no DB/service), iex-safe, and pings <HEARTBEAT_URL> on PASS.
Cross-linked with od-db-backup.md and od-backup-verify.md; README updated.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

README.md

@@ -70,9 +70,11 @@ run time — never commit filled-in values.
 | [`od-scan-duplex.md`](od-scan-duplex.md) | Open Dental — duplex ADF scanner captures only one side (TWAIN, Show TWAIN UI branches) |
 | [`od-db-backup.md`](od-db-backup.md) | Open Dental — rock-solid cold backup of the database + images (stop/copy/start MySQL/MariaDB) |
 | [`od-backup-verify.md`](od-backup-verify.md) | Open Dental — verify a backup by test-restoring into an isolated Hyper-V VM (health checklist) |
+| [`od-backup-schedule.md`](od-backup-schedule.md) | Open Dental — schedule the backup + off-site upload and monitor it (dead-man's-switch heartbeat) |
 | [`scripts/cg-disable.ps1`](scripts/cg-disable.ps1) | Disable Credential Guard, then reboot (prompts to confirm) |
 | [`scripts/od-cfg-acl.ps1`](scripts/od-cfg-acl.ps1) | Grant Users Modify on FreeDentalConfig.xml (Option B of od-cfg-persist) |
 | [`scripts/od-db-backup.ps1`](scripts/od-db-backup.ps1) | Cold backup: stop MySQL/MariaDB, copy whole data dir + OpenDentImages, always restart (od-db-backup) |
+| [`scripts/od-backup-check.ps1`](scripts/od-backup-check.ps1) | Read-only backup health check: freshness/completeness/size + heartbeat ping (od-backup-schedule) |
 
 ## Tiers
 

od-backup-schedule.md

@@ -0,0 +1,141 @@
+# Runbook: Open Dental — Schedule & monitor backups (Windows)
+
+**Applies to:** The Open Dental database server, once [`od-db-backup.md`](od-db-backup.md) is proven to run by hand.
+**Goal:** Make the backup run **automatically every day**, replicate it **off-site**, and **monitor** it so a silent failure gets noticed within a day — not the day you need a restore.
+
+Open Dental's floor is **at least one backup per day**, with a combination of methods and at least one **automated**. ([Open Dental — Backups](https://opendental.com/manual/backups.html)) This runbook automates and watches the cold backup.
+
+> [!IMPORTANT]
+> **Monitoring is the half everyone skips.** A backup job that silently stopped, a full disk, or a powered-off server sends **no error email** — the absence of failure looks exactly like success. The only reliable signal is a **positive heartbeat that goes missing** (a dead-man's-switch). Build that, or you don't have monitoring.
+
+**Placeholders:**
+
+| Placeholder | Meaning |
+|---|---|
+| `<DEST>` | Backup destination root (holds the `od-backup-<timestamp>` folders) |
+| `<BACKUP_SCRIPT>` | Filled-in local copy of `od-db-backup.ps1` (paths baked in) — **private tier** |
+| `<CHECK_SCRIPT>` | Filled-in local copy of `od-backup-check.ps1` (`<DEST>` + heartbeat baked in) — **private tier** |
+| `<HEARTBEAT_URL>` | Dead-man's-switch ping URL (healthchecks.io, Uptime Kuma push, RMM, etc.) — **private tier** |
+
+---
+
+## The jobs and their cadence
+
+| Job | What | When | Runbook |
+|---|---|---|---|
+| **1. Cold backup** | Stop DB, copy data + images, restart | Nightly, off-hours | [`od-db-backup.md`](od-db-backup.md) |
+| **2. Off-site upload** | Replicate `<DEST>` to cloud/immutable | After job 1 finishes | [`od-db-backup.md` §6](od-db-backup.md) |
+| **3. Health check** | Verify newest backup + heartbeat ping | After job 1 finishes | this runbook + [`od-backup-check.ps1`](scripts/od-backup-check.ps1) |
+| **4. Test-restore** | Full restore into isolated VM | Monthly | [`od-backup-verify.md`](od-backup-verify.md) |
+
+Stagger them so nothing reads a half-written folder, e.g. **backup 23:30 → check 00:45 → off-site upload 01:00**.
+
+## Why a scheduled *local copy*, not `irm | iex`
+
+`irm | iex` prompts interactively — it can't run unattended. For scheduling, keep a **filled-in copy** of each script (`<BACKUP_SCRIPT>`, `<CHECK_SCRIPT>`) with `<DEST>`, service name, and paths baked in, stored in the **private tier** (never committed here). The public scripts stay the interactive/spot-check version.
+
+---
+
+## 1. Schedule the nightly cold backup
+
+Register `<BACKUP_SCRIPT>` to run as **SYSTEM**, highest privileges, off-hours. From an elevated prompt:
+
+```
+schtasks /Create /TN "OD Nightly Cold Backup" ^
+  /TR "powershell -NoProfile -ExecutionPolicy Bypass -File <BACKUP_SCRIPT>" ^
+  /SC DAILY /ST 23:30 /RU SYSTEM /RL HIGHEST
+```
+
+- `<BACKUP_SCRIPT>` is the filled-in copy — it must **not** prompt (no `Read-Host` for the scheduled path) and must still do the stop → verify-stopped → copy → **always-restart** sequence.
+- Ensure the task is set to **run whether or not a user is logged on** and, if the server sleeps, **wake the computer to run** (Task Scheduler → task → *Conditions*).
+- Confirm the server actually **stays on** overnight (disable sleep/hibernate on the server).
+
+## 2. Schedule the off-site upload (after the backup)
+
+Point your off-site tool (Duplicati → Backblaze B2, `rclone`, Veeam, etc.) at **`<DEST>`** — it uploads *this backup*, it does **not** re-run the stop/copy against the live database. Schedule it **after** job 1 completes and stagger the start. Enable that tool's **own** email/report and **object-lock/immutability** for the ransomware-resistant off-site copy. Details: [`od-db-backup.md` §6](od-db-backup.md).
+
+## 3. Schedule the health check (and heartbeat)
+
+Register `<CHECK_SCRIPT>` to run shortly **after** the backup window. It verifies the newest `od-backup-<timestamp>` is fresh, complete, and sensibly sized, appends to `<DEST>\backup-check.log`, and — when healthy — pings `<HEARTBEAT_URL>`.
+
+```
+schtasks /Create /TN "OD Backup Health Check" ^
+  /TR "powershell -NoProfile -ExecutionPolicy Bypass -File <CHECK_SCRIPT>" ^
+  /SC DAILY /ST 00:45 /RU SYSTEM /RL HIGHEST
+```
+
+The check is **read-only** — no DB, no service, no file changes — so it's safe to run any time, including a manual spot-check:
+
+```
+irm https://gitea.ivangodwin.com/ops/rb/raw/branch/main/scripts/od-backup-check.ps1 | iex
+```
+
+---
+
+## Monitoring — the three layers of "is it actually working?"
+
+Use all three; each catches what the others miss.
+
+### Layer 1 — Did the job run? (Task Scheduler)
+
+`Last Run Result = 0x0` and a recent `Last Run Time` on both tasks:
+
+```
+Get-ScheduledTaskInfo -TaskName "OD Nightly Cold Backup"
+Get-ScheduledTaskInfo -TaskName "OD Backup Health Check"
+```
+
+Enable **All Tasks History** in Task Scheduler so you can see misfires. Catches: task disabled, wrong credentials, server was off.
+
+### Layer 2 — Is the output good? (the health check)
+
+`od-backup-check.ps1` / `<CHECK_SCRIPT>` confirms the newest backup is:
+
+- **Fresh** — written within 24 h (else the job silently stopped),
+- **Complete** — has `data\`, `OpenDentImages\`, `MANIFEST.txt`, and a `backup.log` that ends in success with no `ERROR`/`CRITICAL`,
+- **Sane size** — `data\` isn't near-empty and isn't a fraction of the prior run (catches truncation / a filling disk).
+
+Read `<DEST>\backup-check.log` for the running PASS/FAIL trail. Catches: partial copies, missing images, service that didn't restart, dying disk.
+
+### Layer 3 — Dead-man's-switch (the one that catches everything)
+
+Register a check with an **external** monitor — [healthchecks.io](https://healthchecks.io), Uptime Kuma (push), or your RMM — that expects a daily ping. `<CHECK_SCRIPT>` pings `<HEARTBEAT_URL>` **only when the backup is healthy**. If the backup breaks, the check fails, the script crashes, or **the whole server is offline**, the ping never arrives and the monitor alerts you.
+
+- Set the monitor's **period to ~1 day** with a grace window past your backup+check schedule.
+- healthchecks.io users: `<CHECK_SCRIPT>` can hit `<HEARTBEAT_URL>/fail` on failure for an **immediate** alert instead of waiting out the grace period.
+- This is the layer that turns "no news" into an actual alarm. Without it, a dead backup is invisible until a restore fails.
+
+### Layer 4 — Off-site tool's own report
+
+Your cloud tool (Duplicati/Veeam/rclone wrapper) should send its **own** success/failure summary and expose versions/immutability in the provider console. Confirms the copy actually left the building.
+
+---
+
+## On failure — triage
+
+When Layer 2/3 flags a problem, in rough order:
+
+1. **Is the database up?** `Get-Service <DB_SERVICE>` — if the backup died mid-run, confirm the service **restarted** (the script's `finally` should have; verify). The practice being able to work comes first.
+2. **Destination full / offline?** Free space on `<DEST>`; is the disk/UNC reachable? Prune old `od-backup-<timestamp>` generations if space-bound.
+3. **Partial/most-recent folder incomplete?** Check that run's `backup.log` for the `ERROR`/`CRITICAL` line; re-run the backup by hand ([`od-db-backup.md`](od-db-backup.md)).
+4. **Version drift?** A recent Open Dental/MySQL update can change paths — reconcile against `MANIFEST.txt`.
+5. **Off-site not uploading?** Check the cloud tool's log and that it sources `<DEST>` (not the live datadir).
+
+Then re-run the health check and confirm the heartbeat goes green.
+
+## Records / evidence
+
+Keep the `<DEST>\backup-check.log`, the monitor's uptime history, and the monthly **test-restore** results ([`od-backup-verify.md`](od-backup-verify.md)) together. That trail is your DR evidence for E&O / cyber insurance and HIPAA contingency-plan testing.
+
+---
+
+## Security note
+
+`<HEARTBEAT_URL>` is a capability — anyone with it can spoof "backup healthy." Treat it as a secret: keep it in the **private tier**, out of this repo, out of screenshots. Don't let monitor check names or heartbeat URLs encode a client's identity. The `<DEST>` folders and `<BACKUP_SCRIPT>`/`<CHECK_SCRIPT>` may reference real paths — keep the filled-in copies private and the destination encrypted/access-controlled (it holds PHI).
+
+## References
+
+- Open Dental manual — Backups (daily minimum; automated + combined methods): <https://opendental.com/manual/backups.html>
+- Microsoft — `schtasks` / Scheduled Tasks: <https://learn.microsoft.com/windows-server/administration/windows-commands/schtasks>
+- healthchecks.io — dead-man's-switch cron monitoring: <https://healthchecks.io/>
+- Companion runbooks — [`od-db-backup.md`](od-db-backup.md) (produce the backup), [`od-backup-verify.md`](od-backup-verify.md) (test-restore)

od-db-backup.md

@@ -106,6 +106,8 @@ Register the cold backup as a nightly **Task Scheduler** job (runs off-hours, as
 schtasks /Create /TN "OD Nightly Cold Backup" /TR "powershell -NoProfile -ExecutionPolicy Bypass -File C:\ops\od-db-backup-local.ps1" /SC DAILY /ST 23:30 /RU SYSTEM /RL HIGHEST
 ```
 
+Full scheduling **and monitoring** (staggering the off-site upload, a daily health check, and a dead-man's-switch heartbeat so a silent failure gets caught): **[`od-backup-schedule.md`](od-backup-schedule.md)**.
+
 ## 5. Verify — a backup you haven't restored is a guess
 
 - **Test-restore** to an **isolated** machine periodically (matching Open Dental + MySQL/MariaDB versions): `net stop`, rename the existing `opendental` folder aside, drop in the backup's `data\` contents, `net start`, launch Open Dental, spot-check patients/images. Full step-by-step with a health checklist: **[`od-backup-verify.md`](od-backup-verify.md)** (isolated Hyper-V test-restore).

scripts/od-backup-check.ps1

@@ -0,0 +1,147 @@
+<#
+.SYNOPSIS
+  Read-only health check of the latest Open Dental cold backup: freshness,
+  completeness, and size sanity. Optionally pings a dead-man's-switch heartbeat
+  URL on success so an external monitor alerts when a backup silently stops.
+
+.DESCRIPTION
+  Points at the backup destination root (<DEST>) produced by od-db-backup.ps1,
+  finds the newest od-backup-<timestamp> folder, and verifies it is:
+    - FRESH      : written within the max-age window (default 24h)
+    - COMPLETE   : has data\, (optionally) OpenDentImages\, MANIFEST.txt, and a
+                   backup.log that ends in success with no ERROR/CRITICAL lines
+    - SANE SIZE  : data\ isn't near-empty, and isn't a fraction of the prior run
+  Prints a clear PASS/FAIL summary, appends a line to <DEST>\backup-check.log,
+  and — if healthy and a heartbeat URL was given — pings it. Pinging ONLY on
+  success is deliberate: if this check fails, crashes, or the whole server is
+  off, no ping arrives and the external monitor raises the alarm. "No news" is
+  never mistaken for good news.
+
+  Fully READ-ONLY: it does not touch the database, the service, or the backup
+  files. Safe to schedule and safe to run ad hoc.
+
+.NOTES
+  Manual spot-check:
+      irm https://gitea.ivangodwin.com/ops/rb/raw/branch/main/scripts/od-backup-check.ps1 | iex
+
+  For unattended monitoring, schedule a filled-in LOCAL copy (with <DEST> and the
+  heartbeat URL baked in) — keep that copy in the private tier, not here. See
+  od-backup-schedule.md.
+
+  iex-safe: uses `return`, never `exit` (which would close an interactive
+  session). PUBLIC REPO: no client specifics hard-coded; prompt at run time.
+#>
+
+$ErrorActionPreference = 'Stop'
+
+Write-Host '== Open Dental: backup health check (read-only) ==' -ForegroundColor Cyan
+
+# --- Prompts (no param block; iex-safe) ---
+$destRoot = Read-Host 'Backup destination root (<DEST>, e.g. E:\OD-Backups)'
+if (-not (Test-Path $destRoot)) {
+    Write-Warning "Destination not found: $destRoot"
+    return
+}
+$maxAgeInput = Read-Host 'Max age (hours) for a healthy backup [24]'
+$maxAgeHrs   = if ($maxAgeInput -match '^\d+$') { [int]$maxAgeInput } else { 24 }
+$expectImages = (Read-Host 'Should each backup include an OpenDentImages folder? (Y/n)') -ne 'n'
+$heartbeat    = Read-Host 'Heartbeat/ping URL to hit on SUCCESS (<HEARTBEAT_URL>) [blank to skip]'
+
+$problems = New-Object System.Collections.Generic.List[string]
+function Fail($msg) { $problems.Add($msg) }
+
+function Get-DirSizeBytes($path) {
+    if (-not (Test-Path $path)) { return $null }
+    (Get-ChildItem -LiteralPath $path -Recurse -Force -File -ErrorAction SilentlyContinue |
+        Measure-Object -Property Length -Sum).Sum
+}
+
+# --- Find the newest backup folder ---
+$folders = @(Get-ChildItem -LiteralPath $destRoot -Directory -Filter 'od-backup-*' -ErrorAction SilentlyContinue |
+             Sort-Object LastWriteTime -Descending)
+if ($folders.Count -eq 0) {
+    Fail "No od-backup-* folders found under $destRoot."
+}
+else {
+    $latest = $folders[0]
+    Write-Host "Latest backup: $($latest.Name)  ($($latest.LastWriteTime))" -ForegroundColor Green
+
+    # 1) Freshness
+    $ageHrs = [math]::Round(((Get-Date) - $latest.LastWriteTime).TotalHours, 1)
+    if ($ageHrs -gt $maxAgeHrs) {
+        Fail "Stale: newest backup is $ageHrs h old (limit $maxAgeHrs h). Backup may have stopped running."
+    }
+    if (((Get-Date) - $latest.LastWriteTime).TotalMinutes -lt 5) {
+        Write-Warning 'Newest folder was written <5 min ago — a backup may be in progress. Re-check after it finishes.'
+    }
+
+    # 2) Completeness
+    $dataDir  = Join-Path $latest.FullName 'data'
+    $imgDir   = Join-Path $latest.FullName 'OpenDentImages'
+    $manifest = Join-Path $latest.FullName 'MANIFEST.txt'
+    $logFile  = Join-Path $latest.FullName 'backup.log'
+
+    if (-not (Test-Path $dataDir))  { Fail "Missing data\ directory in $($latest.Name)." }
+    if (-not (Test-Path $manifest)) { Fail "Missing MANIFEST.txt (backup may not have completed) in $($latest.Name)." }
+    if ($expectImages -and -not (Test-Path $imgDir)) {
+        Fail "Missing OpenDentImages\ in $($latest.Name) — a DB-only backup loses all documents."
+    }
+    if (Test-Path $logFile) {
+        $tail = Get-Content -LiteralPath $logFile -Tail 40 -ErrorAction SilentlyContinue
+        if ($tail -match 'ERROR|CRITICAL') { Fail "backup.log contains ERROR/CRITICAL lines in $($latest.Name)." }
+        if (-not ($tail -match 'completed successfully')) {
+            Fail "backup.log does not show a successful completion in $($latest.Name)."
+        }
+    } else {
+        Fail "Missing backup.log in $($latest.Name)."
+    }
+
+    # 3) Size sanity
+    $dataBytes = Get-DirSizeBytes $dataDir
+    if ($null -ne $dataBytes) {
+        $dataMB = [math]::Round($dataBytes / 1MB, 1)
+        Write-Host "  data\ size: $dataMB MB"
+        if ($dataBytes -lt 1MB) { Fail "data\ is only $dataMB MB — implausibly small; likely a broken/partial backup." }
+        if ($folders.Count -gt 1) {
+            $prevData = Get-DirSizeBytes (Join-Path $folders[1].FullName 'data')
+            if ($prevData -and $dataBytes -lt ($prevData * 0.5)) {
+                Fail ("data\ shrank to {0}% of the previous run ({1} MB -> {2} MB) — possible truncation." -f `
+                    [math]::Round($dataBytes / $prevData * 100), [math]::Round($prevData/1MB,1), $dataMB)
+            }
+        }
+    }
+}
+
+# --- Report ---
+$healthy = ($problems.Count -eq 0)
+$stamp   = Get-Date -Format 'yyyy-MM-dd HH:mm:ss'
+Write-Host ''
+if ($healthy) {
+    Write-Host "PASS  Backup is healthy as of $stamp." -ForegroundColor Green
+} else {
+    Write-Host "FAIL  Backup health problems:" -ForegroundColor Red
+    $problems | ForEach-Object { Write-Host "  - $_" -ForegroundColor Red }
+}
+
+# On-disk trail at the destination
+try {
+    $line = "$stamp  $(if($healthy){'PASS'}else{'FAIL'})  $(if($healthy){'healthy'}else{$problems -join ' | '})"
+    Add-Content -LiteralPath (Join-Path $destRoot 'backup-check.log') -Value $line
+} catch { Write-Warning "Could not write backup-check.log: $($_.Exception.Message)" }
+
+# --- Dead-man's-switch heartbeat: ping ONLY on success ---
+if ($heartbeat) {
+    if ($healthy) {
+        try {
+            Invoke-WebRequest -Uri $heartbeat -UseBasicParsing -TimeoutSec 20 -Method Get | Out-Null
+            Write-Host 'Heartbeat ping sent.' -ForegroundColor Green
+        } catch {
+            Write-Warning "Heartbeat ping failed: $($_.Exception.Message) (monitor will alarm on the missing check-in)."
+        }
+    } else {
+        Write-Host 'Unhealthy — NOT pinging heartbeat. The external monitor should alert on the missing ping.' -ForegroundColor Yellow
+        Write-Host '(healthchecks.io users: append /fail to the URL for an immediate alert.)'
+    }
+}
+
+if (-not $healthy) { Write-Warning 'Investigate now — see od-backup-schedule.md triage.' }