From dfe37d3e4a2e9ee3bc0fee3f739055e1022c68a7 Mon Sep 17 00:00:00 2001 From: Ivan Godwin Date: Mon, 13 Jul 2026 01:07:47 -0700 Subject: [PATCH] Add od-backup-schedule runbook + od-backup-check script: schedule & monitor backups Automate the nightly cold backup (Task Scheduler, filled-in local copy in private tier), stagger the off-site upload, and MONITOR with three layers: Task Scheduler last-run, a read-only health check (freshness/completeness/ size), and a dead-man's-switch heartbeat that pings an external monitor only on success so silent failures and offline servers get caught. od-backup-check.ps1 is read-only (no DB/service), iex-safe, and pings on PASS. Cross-linked with od-db-backup.md and od-backup-verify.md; README updated. Co-Authored-By: Claude Opus 4.8 --- README.md | 2 + od-backup-schedule.md | 141 ++++++++++++++++++++++++++++++++++ od-db-backup.md | 2 + scripts/od-backup-check.ps1 | 147 ++++++++++++++++++++++++++++++++++++ 4 files changed, 292 insertions(+) create mode 100644 od-backup-schedule.md create mode 100644 scripts/od-backup-check.ps1 diff --git a/README.md b/README.md index 9be58ec..4ebb99b 100644 --- a/README.md +++ b/README.md @@ -70,9 +70,11 @@ run time — never commit filled-in values. | [`od-scan-duplex.md`](od-scan-duplex.md) | Open Dental — duplex ADF scanner captures only one side (TWAIN, Show TWAIN UI branches) | | [`od-db-backup.md`](od-db-backup.md) | Open Dental — rock-solid cold backup of the database + images (stop/copy/start MySQL/MariaDB) | | [`od-backup-verify.md`](od-backup-verify.md) | Open Dental — verify a backup by test-restoring into an isolated Hyper-V VM (health checklist) | +| [`od-backup-schedule.md`](od-backup-schedule.md) | Open Dental — schedule the backup + off-site upload and monitor it (dead-man's-switch heartbeat) | | [`scripts/cg-disable.ps1`](scripts/cg-disable.ps1) | Disable Credential Guard, then reboot (prompts to confirm) | | [`scripts/od-cfg-acl.ps1`](scripts/od-cfg-acl.ps1) | Grant Users Modify on FreeDentalConfig.xml (Option B of od-cfg-persist) | | [`scripts/od-db-backup.ps1`](scripts/od-db-backup.ps1) | Cold backup: stop MySQL/MariaDB, copy whole data dir + OpenDentImages, always restart (od-db-backup) | +| [`scripts/od-backup-check.ps1`](scripts/od-backup-check.ps1) | Read-only backup health check: freshness/completeness/size + heartbeat ping (od-backup-schedule) | ## Tiers diff --git a/od-backup-schedule.md b/od-backup-schedule.md new file mode 100644 index 0000000..0ccdc58 --- /dev/null +++ b/od-backup-schedule.md @@ -0,0 +1,141 @@ +# Runbook: Open Dental — Schedule & monitor backups (Windows) + +**Applies to:** The Open Dental database server, once [`od-db-backup.md`](od-db-backup.md) is proven to run by hand. +**Goal:** Make the backup run **automatically every day**, replicate it **off-site**, and **monitor** it so a silent failure gets noticed within a day — not the day you need a restore. + +Open Dental's floor is **at least one backup per day**, with a combination of methods and at least one **automated**. ([Open Dental — Backups](https://opendental.com/manual/backups.html)) This runbook automates and watches the cold backup. + +> [!IMPORTANT] +> **Monitoring is the half everyone skips.** A backup job that silently stopped, a full disk, or a powered-off server sends **no error email** — the absence of failure looks exactly like success. The only reliable signal is a **positive heartbeat that goes missing** (a dead-man's-switch). Build that, or you don't have monitoring. + +**Placeholders:** + +| Placeholder | Meaning | +|---|---| +| `` | Backup destination root (holds the `od-backup-` folders) | +| `` | Filled-in local copy of `od-db-backup.ps1` (paths baked in) — **private tier** | +| `` | Filled-in local copy of `od-backup-check.ps1` (`` + heartbeat baked in) — **private tier** | +| `` | Dead-man's-switch ping URL (healthchecks.io, Uptime Kuma push, RMM, etc.) — **private tier** | + +--- + +## The jobs and their cadence + +| Job | What | When | Runbook | +|---|---|---|---| +| **1. Cold backup** | Stop DB, copy data + images, restart | Nightly, off-hours | [`od-db-backup.md`](od-db-backup.md) | +| **2. Off-site upload** | Replicate `` to cloud/immutable | After job 1 finishes | [`od-db-backup.md` §6](od-db-backup.md) | +| **3. Health check** | Verify newest backup + heartbeat ping | After job 1 finishes | this runbook + [`od-backup-check.ps1`](scripts/od-backup-check.ps1) | +| **4. Test-restore** | Full restore into isolated VM | Monthly | [`od-backup-verify.md`](od-backup-verify.md) | + +Stagger them so nothing reads a half-written folder, e.g. **backup 23:30 → check 00:45 → off-site upload 01:00**. + +## Why a scheduled *local copy*, not `irm | iex` + +`irm | iex` prompts interactively — it can't run unattended. For scheduling, keep a **filled-in copy** of each script (``, ``) with ``, service name, and paths baked in, stored in the **private tier** (never committed here). The public scripts stay the interactive/spot-check version. + +--- + +## 1. Schedule the nightly cold backup + +Register `` to run as **SYSTEM**, highest privileges, off-hours. From an elevated prompt: + +``` +schtasks /Create /TN "OD Nightly Cold Backup" ^ + /TR "powershell -NoProfile -ExecutionPolicy Bypass -File " ^ + /SC DAILY /ST 23:30 /RU SYSTEM /RL HIGHEST +``` + +- `` is the filled-in copy — it must **not** prompt (no `Read-Host` for the scheduled path) and must still do the stop → verify-stopped → copy → **always-restart** sequence. +- Ensure the task is set to **run whether or not a user is logged on** and, if the server sleeps, **wake the computer to run** (Task Scheduler → task → *Conditions*). +- Confirm the server actually **stays on** overnight (disable sleep/hibernate on the server). + +## 2. Schedule the off-site upload (after the backup) + +Point your off-site tool (Duplicati → Backblaze B2, `rclone`, Veeam, etc.) at **``** — it uploads *this backup*, it does **not** re-run the stop/copy against the live database. Schedule it **after** job 1 completes and stagger the start. Enable that tool's **own** email/report and **object-lock/immutability** for the ransomware-resistant off-site copy. Details: [`od-db-backup.md` §6](od-db-backup.md). + +## 3. Schedule the health check (and heartbeat) + +Register `` to run shortly **after** the backup window. It verifies the newest `od-backup-` is fresh, complete, and sensibly sized, appends to `\backup-check.log`, and — when healthy — pings ``. + +``` +schtasks /Create /TN "OD Backup Health Check" ^ + /TR "powershell -NoProfile -ExecutionPolicy Bypass -File " ^ + /SC DAILY /ST 00:45 /RU SYSTEM /RL HIGHEST +``` + +The check is **read-only** — no DB, no service, no file changes — so it's safe to run any time, including a manual spot-check: + +``` +irm https://gitea.ivangodwin.com/ops/rb/raw/branch/main/scripts/od-backup-check.ps1 | iex +``` + +--- + +## Monitoring — the three layers of "is it actually working?" + +Use all three; each catches what the others miss. + +### Layer 1 — Did the job run? (Task Scheduler) + +`Last Run Result = 0x0` and a recent `Last Run Time` on both tasks: + +``` +Get-ScheduledTaskInfo -TaskName "OD Nightly Cold Backup" +Get-ScheduledTaskInfo -TaskName "OD Backup Health Check" +``` + +Enable **All Tasks History** in Task Scheduler so you can see misfires. Catches: task disabled, wrong credentials, server was off. + +### Layer 2 — Is the output good? (the health check) + +`od-backup-check.ps1` / `` confirms the newest backup is: + +- **Fresh** — written within 24 h (else the job silently stopped), +- **Complete** — has `data\`, `OpenDentImages\`, `MANIFEST.txt`, and a `backup.log` that ends in success with no `ERROR`/`CRITICAL`, +- **Sane size** — `data\` isn't near-empty and isn't a fraction of the prior run (catches truncation / a filling disk). + +Read `\backup-check.log` for the running PASS/FAIL trail. Catches: partial copies, missing images, service that didn't restart, dying disk. + +### Layer 3 — Dead-man's-switch (the one that catches everything) + +Register a check with an **external** monitor — [healthchecks.io](https://healthchecks.io), Uptime Kuma (push), or your RMM — that expects a daily ping. `` pings `` **only when the backup is healthy**. If the backup breaks, the check fails, the script crashes, or **the whole server is offline**, the ping never arrives and the monitor alerts you. + +- Set the monitor's **period to ~1 day** with a grace window past your backup+check schedule. +- healthchecks.io users: `` can hit `/fail` on failure for an **immediate** alert instead of waiting out the grace period. +- This is the layer that turns "no news" into an actual alarm. Without it, a dead backup is invisible until a restore fails. + +### Layer 4 — Off-site tool's own report + +Your cloud tool (Duplicati/Veeam/rclone wrapper) should send its **own** success/failure summary and expose versions/immutability in the provider console. Confirms the copy actually left the building. + +--- + +## On failure — triage + +When Layer 2/3 flags a problem, in rough order: + +1. **Is the database up?** `Get-Service ` — if the backup died mid-run, confirm the service **restarted** (the script's `finally` should have; verify). The practice being able to work comes first. +2. **Destination full / offline?** Free space on ``; is the disk/UNC reachable? Prune old `od-backup-` generations if space-bound. +3. **Partial/most-recent folder incomplete?** Check that run's `backup.log` for the `ERROR`/`CRITICAL` line; re-run the backup by hand ([`od-db-backup.md`](od-db-backup.md)). +4. **Version drift?** A recent Open Dental/MySQL update can change paths — reconcile against `MANIFEST.txt`. +5. **Off-site not uploading?** Check the cloud tool's log and that it sources `` (not the live datadir). + +Then re-run the health check and confirm the heartbeat goes green. + +## Records / evidence + +Keep the `\backup-check.log`, the monitor's uptime history, and the monthly **test-restore** results ([`od-backup-verify.md`](od-backup-verify.md)) together. That trail is your DR evidence for E&O / cyber insurance and HIPAA contingency-plan testing. + +--- + +## Security note + +`` is a capability — anyone with it can spoof "backup healthy." Treat it as a secret: keep it in the **private tier**, out of this repo, out of screenshots. Don't let monitor check names or heartbeat URLs encode a client's identity. The `` folders and ``/`` may reference real paths — keep the filled-in copies private and the destination encrypted/access-controlled (it holds PHI). + +## References + +- Open Dental manual — Backups (daily minimum; automated + combined methods): +- Microsoft — `schtasks` / Scheduled Tasks: +- healthchecks.io — dead-man's-switch cron monitoring: +- Companion runbooks — [`od-db-backup.md`](od-db-backup.md) (produce the backup), [`od-backup-verify.md`](od-backup-verify.md) (test-restore) diff --git a/od-db-backup.md b/od-db-backup.md index 69aaea8..9c75c48 100644 --- a/od-db-backup.md +++ b/od-db-backup.md @@ -106,6 +106,8 @@ Register the cold backup as a nightly **Task Scheduler** job (runs off-hours, as schtasks /Create /TN "OD Nightly Cold Backup" /TR "powershell -NoProfile -ExecutionPolicy Bypass -File C:\ops\od-db-backup-local.ps1" /SC DAILY /ST 23:30 /RU SYSTEM /RL HIGHEST ``` +Full scheduling **and monitoring** (staggering the off-site upload, a daily health check, and a dead-man's-switch heartbeat so a silent failure gets caught): **[`od-backup-schedule.md`](od-backup-schedule.md)**. + ## 5. Verify — a backup you haven't restored is a guess - **Test-restore** to an **isolated** machine periodically (matching Open Dental + MySQL/MariaDB versions): `net stop`, rename the existing `opendental` folder aside, drop in the backup's `data\` contents, `net start`, launch Open Dental, spot-check patients/images. Full step-by-step with a health checklist: **[`od-backup-verify.md`](od-backup-verify.md)** (isolated Hyper-V test-restore). diff --git a/scripts/od-backup-check.ps1 b/scripts/od-backup-check.ps1 new file mode 100644 index 0000000..e955ce5 --- /dev/null +++ b/scripts/od-backup-check.ps1 @@ -0,0 +1,147 @@ +<# +.SYNOPSIS + Read-only health check of the latest Open Dental cold backup: freshness, + completeness, and size sanity. Optionally pings a dead-man's-switch heartbeat + URL on success so an external monitor alerts when a backup silently stops. + +.DESCRIPTION + Points at the backup destination root () produced by od-db-backup.ps1, + finds the newest od-backup- folder, and verifies it is: + - FRESH : written within the max-age window (default 24h) + - COMPLETE : has data\, (optionally) OpenDentImages\, MANIFEST.txt, and a + backup.log that ends in success with no ERROR/CRITICAL lines + - SANE SIZE : data\ isn't near-empty, and isn't a fraction of the prior run + Prints a clear PASS/FAIL summary, appends a line to \backup-check.log, + and — if healthy and a heartbeat URL was given — pings it. Pinging ONLY on + success is deliberate: if this check fails, crashes, or the whole server is + off, no ping arrives and the external monitor raises the alarm. "No news" is + never mistaken for good news. + + Fully READ-ONLY: it does not touch the database, the service, or the backup + files. Safe to schedule and safe to run ad hoc. + +.NOTES + Manual spot-check: + irm https://gitea.ivangodwin.com/ops/rb/raw/branch/main/scripts/od-backup-check.ps1 | iex + + For unattended monitoring, schedule a filled-in LOCAL copy (with and the + heartbeat URL baked in) — keep that copy in the private tier, not here. See + od-backup-schedule.md. + + iex-safe: uses `return`, never `exit` (which would close an interactive + session). PUBLIC REPO: no client specifics hard-coded; prompt at run time. +#> + +$ErrorActionPreference = 'Stop' + +Write-Host '== Open Dental: backup health check (read-only) ==' -ForegroundColor Cyan + +# --- Prompts (no param block; iex-safe) --- +$destRoot = Read-Host 'Backup destination root (, e.g. E:\OD-Backups)' +if (-not (Test-Path $destRoot)) { + Write-Warning "Destination not found: $destRoot" + return +} +$maxAgeInput = Read-Host 'Max age (hours) for a healthy backup [24]' +$maxAgeHrs = if ($maxAgeInput -match '^\d+$') { [int]$maxAgeInput } else { 24 } +$expectImages = (Read-Host 'Should each backup include an OpenDentImages folder? (Y/n)') -ne 'n' +$heartbeat = Read-Host 'Heartbeat/ping URL to hit on SUCCESS () [blank to skip]' + +$problems = New-Object System.Collections.Generic.List[string] +function Fail($msg) { $problems.Add($msg) } + +function Get-DirSizeBytes($path) { + if (-not (Test-Path $path)) { return $null } + (Get-ChildItem -LiteralPath $path -Recurse -Force -File -ErrorAction SilentlyContinue | + Measure-Object -Property Length -Sum).Sum +} + +# --- Find the newest backup folder --- +$folders = @(Get-ChildItem -LiteralPath $destRoot -Directory -Filter 'od-backup-*' -ErrorAction SilentlyContinue | + Sort-Object LastWriteTime -Descending) +if ($folders.Count -eq 0) { + Fail "No od-backup-* folders found under $destRoot." +} +else { + $latest = $folders[0] + Write-Host "Latest backup: $($latest.Name) ($($latest.LastWriteTime))" -ForegroundColor Green + + # 1) Freshness + $ageHrs = [math]::Round(((Get-Date) - $latest.LastWriteTime).TotalHours, 1) + if ($ageHrs -gt $maxAgeHrs) { + Fail "Stale: newest backup is $ageHrs h old (limit $maxAgeHrs h). Backup may have stopped running." + } + if (((Get-Date) - $latest.LastWriteTime).TotalMinutes -lt 5) { + Write-Warning 'Newest folder was written <5 min ago — a backup may be in progress. Re-check after it finishes.' + } + + # 2) Completeness + $dataDir = Join-Path $latest.FullName 'data' + $imgDir = Join-Path $latest.FullName 'OpenDentImages' + $manifest = Join-Path $latest.FullName 'MANIFEST.txt' + $logFile = Join-Path $latest.FullName 'backup.log' + + if (-not (Test-Path $dataDir)) { Fail "Missing data\ directory in $($latest.Name)." } + if (-not (Test-Path $manifest)) { Fail "Missing MANIFEST.txt (backup may not have completed) in $($latest.Name)." } + if ($expectImages -and -not (Test-Path $imgDir)) { + Fail "Missing OpenDentImages\ in $($latest.Name) — a DB-only backup loses all documents." + } + if (Test-Path $logFile) { + $tail = Get-Content -LiteralPath $logFile -Tail 40 -ErrorAction SilentlyContinue + if ($tail -match 'ERROR|CRITICAL') { Fail "backup.log contains ERROR/CRITICAL lines in $($latest.Name)." } + if (-not ($tail -match 'completed successfully')) { + Fail "backup.log does not show a successful completion in $($latest.Name)." + } + } else { + Fail "Missing backup.log in $($latest.Name)." + } + + # 3) Size sanity + $dataBytes = Get-DirSizeBytes $dataDir + if ($null -ne $dataBytes) { + $dataMB = [math]::Round($dataBytes / 1MB, 1) + Write-Host " data\ size: $dataMB MB" + if ($dataBytes -lt 1MB) { Fail "data\ is only $dataMB MB — implausibly small; likely a broken/partial backup." } + if ($folders.Count -gt 1) { + $prevData = Get-DirSizeBytes (Join-Path $folders[1].FullName 'data') + if ($prevData -and $dataBytes -lt ($prevData * 0.5)) { + Fail ("data\ shrank to {0}% of the previous run ({1} MB -> {2} MB) — possible truncation." -f ` + [math]::Round($dataBytes / $prevData * 100), [math]::Round($prevData/1MB,1), $dataMB) + } + } + } +} + +# --- Report --- +$healthy = ($problems.Count -eq 0) +$stamp = Get-Date -Format 'yyyy-MM-dd HH:mm:ss' +Write-Host '' +if ($healthy) { + Write-Host "PASS Backup is healthy as of $stamp." -ForegroundColor Green +} else { + Write-Host "FAIL Backup health problems:" -ForegroundColor Red + $problems | ForEach-Object { Write-Host " - $_" -ForegroundColor Red } +} + +# On-disk trail at the destination +try { + $line = "$stamp $(if($healthy){'PASS'}else{'FAIL'}) $(if($healthy){'healthy'}else{$problems -join ' | '})" + Add-Content -LiteralPath (Join-Path $destRoot 'backup-check.log') -Value $line +} catch { Write-Warning "Could not write backup-check.log: $($_.Exception.Message)" } + +# --- Dead-man's-switch heartbeat: ping ONLY on success --- +if ($heartbeat) { + if ($healthy) { + try { + Invoke-WebRequest -Uri $heartbeat -UseBasicParsing -TimeoutSec 20 -Method Get | Out-Null + Write-Host 'Heartbeat ping sent.' -ForegroundColor Green + } catch { + Write-Warning "Heartbeat ping failed: $($_.Exception.Message) (monitor will alarm on the missing check-in)." + } + } else { + Write-Host 'Unhealthy — NOT pinging heartbeat. The external monitor should alert on the missing ping.' -ForegroundColor Yellow + Write-Host '(healthchecks.io users: append /fail to the URL for an immediate alert.)' + } +} + +if (-not $healthy) { Write-Warning 'Investigate now — see od-backup-schedule.md triage.' }