Add od-backup-schedule runbook + od-backup-check script: schedule & monitor backups
Automate the nightly cold backup (Task Scheduler, filled-in local copy in private tier), stagger the off-site upload, and MONITOR with three layers: Task Scheduler last-run, a read-only health check (freshness/completeness/ size), and a dead-man's-switch heartbeat that pings an external monitor only on success so silent failures and offline servers get caught. od-backup-check.ps1 is read-only (no DB/service), iex-safe, and pings <HEARTBEAT_URL> on PASS. Cross-linked with od-db-backup.md and od-backup-verify.md; README updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -70,9 +70,11 @@ run time — never commit filled-in values.
|
||||
| [`od-scan-duplex.md`](od-scan-duplex.md) | Open Dental — duplex ADF scanner captures only one side (TWAIN, Show TWAIN UI branches) |
|
||||
| [`od-db-backup.md`](od-db-backup.md) | Open Dental — rock-solid cold backup of the database + images (stop/copy/start MySQL/MariaDB) |
|
||||
| [`od-backup-verify.md`](od-backup-verify.md) | Open Dental — verify a backup by test-restoring into an isolated Hyper-V VM (health checklist) |
|
||||
| [`od-backup-schedule.md`](od-backup-schedule.md) | Open Dental — schedule the backup + off-site upload and monitor it (dead-man's-switch heartbeat) |
|
||||
| [`scripts/cg-disable.ps1`](scripts/cg-disable.ps1) | Disable Credential Guard, then reboot (prompts to confirm) |
|
||||
| [`scripts/od-cfg-acl.ps1`](scripts/od-cfg-acl.ps1) | Grant Users Modify on FreeDentalConfig.xml (Option B of od-cfg-persist) |
|
||||
| [`scripts/od-db-backup.ps1`](scripts/od-db-backup.ps1) | Cold backup: stop MySQL/MariaDB, copy whole data dir + OpenDentImages, always restart (od-db-backup) |
|
||||
| [`scripts/od-backup-check.ps1`](scripts/od-backup-check.ps1) | Read-only backup health check: freshness/completeness/size + heartbeat ping (od-backup-schedule) |
|
||||
|
||||
## Tiers
|
||||
|
||||
|
||||
@@ -0,0 +1,141 @@
|
||||
# Runbook: Open Dental — Schedule & monitor backups (Windows)
|
||||
|
||||
**Applies to:** The Open Dental database server, once [`od-db-backup.md`](od-db-backup.md) is proven to run by hand.
|
||||
**Goal:** Make the backup run **automatically every day**, replicate it **off-site**, and **monitor** it so a silent failure gets noticed within a day — not the day you need a restore.
|
||||
|
||||
Open Dental's floor is **at least one backup per day**, with a combination of methods and at least one **automated**. ([Open Dental — Backups](https://opendental.com/manual/backups.html)) This runbook automates and watches the cold backup.
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **Monitoring is the half everyone skips.** A backup job that silently stopped, a full disk, or a powered-off server sends **no error email** — the absence of failure looks exactly like success. The only reliable signal is a **positive heartbeat that goes missing** (a dead-man's-switch). Build that, or you don't have monitoring.
|
||||
|
||||
**Placeholders:**
|
||||
|
||||
| Placeholder | Meaning |
|
||||
|---|---|
|
||||
| `<DEST>` | Backup destination root (holds the `od-backup-<timestamp>` folders) |
|
||||
| `<BACKUP_SCRIPT>` | Filled-in local copy of `od-db-backup.ps1` (paths baked in) — **private tier** |
|
||||
| `<CHECK_SCRIPT>` | Filled-in local copy of `od-backup-check.ps1` (`<DEST>` + heartbeat baked in) — **private tier** |
|
||||
| `<HEARTBEAT_URL>` | Dead-man's-switch ping URL (healthchecks.io, Uptime Kuma push, RMM, etc.) — **private tier** |
|
||||
|
||||
---
|
||||
|
||||
## The jobs and their cadence
|
||||
|
||||
| Job | What | When | Runbook |
|
||||
|---|---|---|---|
|
||||
| **1. Cold backup** | Stop DB, copy data + images, restart | Nightly, off-hours | [`od-db-backup.md`](od-db-backup.md) |
|
||||
| **2. Off-site upload** | Replicate `<DEST>` to cloud/immutable | After job 1 finishes | [`od-db-backup.md` §6](od-db-backup.md) |
|
||||
| **3. Health check** | Verify newest backup + heartbeat ping | After job 1 finishes | this runbook + [`od-backup-check.ps1`](scripts/od-backup-check.ps1) |
|
||||
| **4. Test-restore** | Full restore into isolated VM | Monthly | [`od-backup-verify.md`](od-backup-verify.md) |
|
||||
|
||||
Stagger them so nothing reads a half-written folder, e.g. **backup 23:30 → check 00:45 → off-site upload 01:00**.
|
||||
|
||||
## Why a scheduled *local copy*, not `irm | iex`
|
||||
|
||||
`irm | iex` prompts interactively — it can't run unattended. For scheduling, keep a **filled-in copy** of each script (`<BACKUP_SCRIPT>`, `<CHECK_SCRIPT>`) with `<DEST>`, service name, and paths baked in, stored in the **private tier** (never committed here). The public scripts stay the interactive/spot-check version.
|
||||
|
||||
---
|
||||
|
||||
## 1. Schedule the nightly cold backup
|
||||
|
||||
Register `<BACKUP_SCRIPT>` to run as **SYSTEM**, highest privileges, off-hours. From an elevated prompt:
|
||||
|
||||
```
|
||||
schtasks /Create /TN "OD Nightly Cold Backup" ^
|
||||
/TR "powershell -NoProfile -ExecutionPolicy Bypass -File <BACKUP_SCRIPT>" ^
|
||||
/SC DAILY /ST 23:30 /RU SYSTEM /RL HIGHEST
|
||||
```
|
||||
|
||||
- `<BACKUP_SCRIPT>` is the filled-in copy — it must **not** prompt (no `Read-Host` for the scheduled path) and must still do the stop → verify-stopped → copy → **always-restart** sequence.
|
||||
- Ensure the task is set to **run whether or not a user is logged on** and, if the server sleeps, **wake the computer to run** (Task Scheduler → task → *Conditions*).
|
||||
- Confirm the server actually **stays on** overnight (disable sleep/hibernate on the server).
|
||||
|
||||
## 2. Schedule the off-site upload (after the backup)
|
||||
|
||||
Point your off-site tool (Duplicati → Backblaze B2, `rclone`, Veeam, etc.) at **`<DEST>`** — it uploads *this backup*, it does **not** re-run the stop/copy against the live database. Schedule it **after** job 1 completes and stagger the start. Enable that tool's **own** email/report and **object-lock/immutability** for the ransomware-resistant off-site copy. Details: [`od-db-backup.md` §6](od-db-backup.md).
|
||||
|
||||
## 3. Schedule the health check (and heartbeat)
|
||||
|
||||
Register `<CHECK_SCRIPT>` to run shortly **after** the backup window. It verifies the newest `od-backup-<timestamp>` is fresh, complete, and sensibly sized, appends to `<DEST>\backup-check.log`, and — when healthy — pings `<HEARTBEAT_URL>`.
|
||||
|
||||
```
|
||||
schtasks /Create /TN "OD Backup Health Check" ^
|
||||
/TR "powershell -NoProfile -ExecutionPolicy Bypass -File <CHECK_SCRIPT>" ^
|
||||
/SC DAILY /ST 00:45 /RU SYSTEM /RL HIGHEST
|
||||
```
|
||||
|
||||
The check is **read-only** — no DB, no service, no file changes — so it's safe to run any time, including a manual spot-check:
|
||||
|
||||
```
|
||||
irm https://gitea.ivangodwin.com/ops/rb/raw/branch/main/scripts/od-backup-check.ps1 | iex
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Monitoring — the three layers of "is it actually working?"
|
||||
|
||||
Use all three; each catches what the others miss.
|
||||
|
||||
### Layer 1 — Did the job run? (Task Scheduler)
|
||||
|
||||
`Last Run Result = 0x0` and a recent `Last Run Time` on both tasks:
|
||||
|
||||
```
|
||||
Get-ScheduledTaskInfo -TaskName "OD Nightly Cold Backup"
|
||||
Get-ScheduledTaskInfo -TaskName "OD Backup Health Check"
|
||||
```
|
||||
|
||||
Enable **All Tasks History** in Task Scheduler so you can see misfires. Catches: task disabled, wrong credentials, server was off.
|
||||
|
||||
### Layer 2 — Is the output good? (the health check)
|
||||
|
||||
`od-backup-check.ps1` / `<CHECK_SCRIPT>` confirms the newest backup is:
|
||||
|
||||
- **Fresh** — written within 24 h (else the job silently stopped),
|
||||
- **Complete** — has `data\`, `OpenDentImages\`, `MANIFEST.txt`, and a `backup.log` that ends in success with no `ERROR`/`CRITICAL`,
|
||||
- **Sane size** — `data\` isn't near-empty and isn't a fraction of the prior run (catches truncation / a filling disk).
|
||||
|
||||
Read `<DEST>\backup-check.log` for the running PASS/FAIL trail. Catches: partial copies, missing images, service that didn't restart, dying disk.
|
||||
|
||||
### Layer 3 — Dead-man's-switch (the one that catches everything)
|
||||
|
||||
Register a check with an **external** monitor — [healthchecks.io](https://healthchecks.io), Uptime Kuma (push), or your RMM — that expects a daily ping. `<CHECK_SCRIPT>` pings `<HEARTBEAT_URL>` **only when the backup is healthy**. If the backup breaks, the check fails, the script crashes, or **the whole server is offline**, the ping never arrives and the monitor alerts you.
|
||||
|
||||
- Set the monitor's **period to ~1 day** with a grace window past your backup+check schedule.
|
||||
- healthchecks.io users: `<CHECK_SCRIPT>` can hit `<HEARTBEAT_URL>/fail` on failure for an **immediate** alert instead of waiting out the grace period.
|
||||
- This is the layer that turns "no news" into an actual alarm. Without it, a dead backup is invisible until a restore fails.
|
||||
|
||||
### Layer 4 — Off-site tool's own report
|
||||
|
||||
Your cloud tool (Duplicati/Veeam/rclone wrapper) should send its **own** success/failure summary and expose versions/immutability in the provider console. Confirms the copy actually left the building.
|
||||
|
||||
---
|
||||
|
||||
## On failure — triage
|
||||
|
||||
When Layer 2/3 flags a problem, in rough order:
|
||||
|
||||
1. **Is the database up?** `Get-Service <DB_SERVICE>` — if the backup died mid-run, confirm the service **restarted** (the script's `finally` should have; verify). The practice being able to work comes first.
|
||||
2. **Destination full / offline?** Free space on `<DEST>`; is the disk/UNC reachable? Prune old `od-backup-<timestamp>` generations if space-bound.
|
||||
3. **Partial/most-recent folder incomplete?** Check that run's `backup.log` for the `ERROR`/`CRITICAL` line; re-run the backup by hand ([`od-db-backup.md`](od-db-backup.md)).
|
||||
4. **Version drift?** A recent Open Dental/MySQL update can change paths — reconcile against `MANIFEST.txt`.
|
||||
5. **Off-site not uploading?** Check the cloud tool's log and that it sources `<DEST>` (not the live datadir).
|
||||
|
||||
Then re-run the health check and confirm the heartbeat goes green.
|
||||
|
||||
## Records / evidence
|
||||
|
||||
Keep the `<DEST>\backup-check.log`, the monitor's uptime history, and the monthly **test-restore** results ([`od-backup-verify.md`](od-backup-verify.md)) together. That trail is your DR evidence for E&O / cyber insurance and HIPAA contingency-plan testing.
|
||||
|
||||
---
|
||||
|
||||
## Security note
|
||||
|
||||
`<HEARTBEAT_URL>` is a capability — anyone with it can spoof "backup healthy." Treat it as a secret: keep it in the **private tier**, out of this repo, out of screenshots. Don't let monitor check names or heartbeat URLs encode a client's identity. The `<DEST>` folders and `<BACKUP_SCRIPT>`/`<CHECK_SCRIPT>` may reference real paths — keep the filled-in copies private and the destination encrypted/access-controlled (it holds PHI).
|
||||
|
||||
## References
|
||||
|
||||
- Open Dental manual — Backups (daily minimum; automated + combined methods): <https://opendental.com/manual/backups.html>
|
||||
- Microsoft — `schtasks` / Scheduled Tasks: <https://learn.microsoft.com/windows-server/administration/windows-commands/schtasks>
|
||||
- healthchecks.io — dead-man's-switch cron monitoring: <https://healthchecks.io/>
|
||||
- Companion runbooks — [`od-db-backup.md`](od-db-backup.md) (produce the backup), [`od-backup-verify.md`](od-backup-verify.md) (test-restore)
|
||||
@@ -106,6 +106,8 @@ Register the cold backup as a nightly **Task Scheduler** job (runs off-hours, as
|
||||
schtasks /Create /TN "OD Nightly Cold Backup" /TR "powershell -NoProfile -ExecutionPolicy Bypass -File C:\ops\od-db-backup-local.ps1" /SC DAILY /ST 23:30 /RU SYSTEM /RL HIGHEST
|
||||
```
|
||||
|
||||
Full scheduling **and monitoring** (staggering the off-site upload, a daily health check, and a dead-man's-switch heartbeat so a silent failure gets caught): **[`od-backup-schedule.md`](od-backup-schedule.md)**.
|
||||
|
||||
## 5. Verify — a backup you haven't restored is a guess
|
||||
|
||||
- **Test-restore** to an **isolated** machine periodically (matching Open Dental + MySQL/MariaDB versions): `net stop`, rename the existing `opendental` folder aside, drop in the backup's `data\` contents, `net start`, launch Open Dental, spot-check patients/images. Full step-by-step with a health checklist: **[`od-backup-verify.md`](od-backup-verify.md)** (isolated Hyper-V test-restore).
|
||||
|
||||
@@ -0,0 +1,147 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Read-only health check of the latest Open Dental cold backup: freshness,
|
||||
completeness, and size sanity. Optionally pings a dead-man's-switch heartbeat
|
||||
URL on success so an external monitor alerts when a backup silently stops.
|
||||
|
||||
.DESCRIPTION
|
||||
Points at the backup destination root (<DEST>) produced by od-db-backup.ps1,
|
||||
finds the newest od-backup-<timestamp> folder, and verifies it is:
|
||||
- FRESH : written within the max-age window (default 24h)
|
||||
- COMPLETE : has data\, (optionally) OpenDentImages\, MANIFEST.txt, and a
|
||||
backup.log that ends in success with no ERROR/CRITICAL lines
|
||||
- SANE SIZE : data\ isn't near-empty, and isn't a fraction of the prior run
|
||||
Prints a clear PASS/FAIL summary, appends a line to <DEST>\backup-check.log,
|
||||
and — if healthy and a heartbeat URL was given — pings it. Pinging ONLY on
|
||||
success is deliberate: if this check fails, crashes, or the whole server is
|
||||
off, no ping arrives and the external monitor raises the alarm. "No news" is
|
||||
never mistaken for good news.
|
||||
|
||||
Fully READ-ONLY: it does not touch the database, the service, or the backup
|
||||
files. Safe to schedule and safe to run ad hoc.
|
||||
|
||||
.NOTES
|
||||
Manual spot-check:
|
||||
irm https://gitea.ivangodwin.com/ops/rb/raw/branch/main/scripts/od-backup-check.ps1 | iex
|
||||
|
||||
For unattended monitoring, schedule a filled-in LOCAL copy (with <DEST> and the
|
||||
heartbeat URL baked in) — keep that copy in the private tier, not here. See
|
||||
od-backup-schedule.md.
|
||||
|
||||
iex-safe: uses `return`, never `exit` (which would close an interactive
|
||||
session). PUBLIC REPO: no client specifics hard-coded; prompt at run time.
|
||||
#>
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
Write-Host '== Open Dental: backup health check (read-only) ==' -ForegroundColor Cyan
|
||||
|
||||
# --- Prompts (no param block; iex-safe) ---
|
||||
$destRoot = Read-Host 'Backup destination root (<DEST>, e.g. E:\OD-Backups)'
|
||||
if (-not (Test-Path $destRoot)) {
|
||||
Write-Warning "Destination not found: $destRoot"
|
||||
return
|
||||
}
|
||||
$maxAgeInput = Read-Host 'Max age (hours) for a healthy backup [24]'
|
||||
$maxAgeHrs = if ($maxAgeInput -match '^\d+$') { [int]$maxAgeInput } else { 24 }
|
||||
$expectImages = (Read-Host 'Should each backup include an OpenDentImages folder? (Y/n)') -ne 'n'
|
||||
$heartbeat = Read-Host 'Heartbeat/ping URL to hit on SUCCESS (<HEARTBEAT_URL>) [blank to skip]'
|
||||
|
||||
$problems = New-Object System.Collections.Generic.List[string]
|
||||
function Fail($msg) { $problems.Add($msg) }
|
||||
|
||||
function Get-DirSizeBytes($path) {
|
||||
if (-not (Test-Path $path)) { return $null }
|
||||
(Get-ChildItem -LiteralPath $path -Recurse -Force -File -ErrorAction SilentlyContinue |
|
||||
Measure-Object -Property Length -Sum).Sum
|
||||
}
|
||||
|
||||
# --- Find the newest backup folder ---
|
||||
$folders = @(Get-ChildItem -LiteralPath $destRoot -Directory -Filter 'od-backup-*' -ErrorAction SilentlyContinue |
|
||||
Sort-Object LastWriteTime -Descending)
|
||||
if ($folders.Count -eq 0) {
|
||||
Fail "No od-backup-* folders found under $destRoot."
|
||||
}
|
||||
else {
|
||||
$latest = $folders[0]
|
||||
Write-Host "Latest backup: $($latest.Name) ($($latest.LastWriteTime))" -ForegroundColor Green
|
||||
|
||||
# 1) Freshness
|
||||
$ageHrs = [math]::Round(((Get-Date) - $latest.LastWriteTime).TotalHours, 1)
|
||||
if ($ageHrs -gt $maxAgeHrs) {
|
||||
Fail "Stale: newest backup is $ageHrs h old (limit $maxAgeHrs h). Backup may have stopped running."
|
||||
}
|
||||
if (((Get-Date) - $latest.LastWriteTime).TotalMinutes -lt 5) {
|
||||
Write-Warning 'Newest folder was written <5 min ago — a backup may be in progress. Re-check after it finishes.'
|
||||
}
|
||||
|
||||
# 2) Completeness
|
||||
$dataDir = Join-Path $latest.FullName 'data'
|
||||
$imgDir = Join-Path $latest.FullName 'OpenDentImages'
|
||||
$manifest = Join-Path $latest.FullName 'MANIFEST.txt'
|
||||
$logFile = Join-Path $latest.FullName 'backup.log'
|
||||
|
||||
if (-not (Test-Path $dataDir)) { Fail "Missing data\ directory in $($latest.Name)." }
|
||||
if (-not (Test-Path $manifest)) { Fail "Missing MANIFEST.txt (backup may not have completed) in $($latest.Name)." }
|
||||
if ($expectImages -and -not (Test-Path $imgDir)) {
|
||||
Fail "Missing OpenDentImages\ in $($latest.Name) — a DB-only backup loses all documents."
|
||||
}
|
||||
if (Test-Path $logFile) {
|
||||
$tail = Get-Content -LiteralPath $logFile -Tail 40 -ErrorAction SilentlyContinue
|
||||
if ($tail -match 'ERROR|CRITICAL') { Fail "backup.log contains ERROR/CRITICAL lines in $($latest.Name)." }
|
||||
if (-not ($tail -match 'completed successfully')) {
|
||||
Fail "backup.log does not show a successful completion in $($latest.Name)."
|
||||
}
|
||||
} else {
|
||||
Fail "Missing backup.log in $($latest.Name)."
|
||||
}
|
||||
|
||||
# 3) Size sanity
|
||||
$dataBytes = Get-DirSizeBytes $dataDir
|
||||
if ($null -ne $dataBytes) {
|
||||
$dataMB = [math]::Round($dataBytes / 1MB, 1)
|
||||
Write-Host " data\ size: $dataMB MB"
|
||||
if ($dataBytes -lt 1MB) { Fail "data\ is only $dataMB MB — implausibly small; likely a broken/partial backup." }
|
||||
if ($folders.Count -gt 1) {
|
||||
$prevData = Get-DirSizeBytes (Join-Path $folders[1].FullName 'data')
|
||||
if ($prevData -and $dataBytes -lt ($prevData * 0.5)) {
|
||||
Fail ("data\ shrank to {0}% of the previous run ({1} MB -> {2} MB) — possible truncation." -f `
|
||||
[math]::Round($dataBytes / $prevData * 100), [math]::Round($prevData/1MB,1), $dataMB)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# --- Report ---
|
||||
$healthy = ($problems.Count -eq 0)
|
||||
$stamp = Get-Date -Format 'yyyy-MM-dd HH:mm:ss'
|
||||
Write-Host ''
|
||||
if ($healthy) {
|
||||
Write-Host "PASS Backup is healthy as of $stamp." -ForegroundColor Green
|
||||
} else {
|
||||
Write-Host "FAIL Backup health problems:" -ForegroundColor Red
|
||||
$problems | ForEach-Object { Write-Host " - $_" -ForegroundColor Red }
|
||||
}
|
||||
|
||||
# On-disk trail at the destination
|
||||
try {
|
||||
$line = "$stamp $(if($healthy){'PASS'}else{'FAIL'}) $(if($healthy){'healthy'}else{$problems -join ' | '})"
|
||||
Add-Content -LiteralPath (Join-Path $destRoot 'backup-check.log') -Value $line
|
||||
} catch { Write-Warning "Could not write backup-check.log: $($_.Exception.Message)" }
|
||||
|
||||
# --- Dead-man's-switch heartbeat: ping ONLY on success ---
|
||||
if ($heartbeat) {
|
||||
if ($healthy) {
|
||||
try {
|
||||
Invoke-WebRequest -Uri $heartbeat -UseBasicParsing -TimeoutSec 20 -Method Get | Out-Null
|
||||
Write-Host 'Heartbeat ping sent.' -ForegroundColor Green
|
||||
} catch {
|
||||
Write-Warning "Heartbeat ping failed: $($_.Exception.Message) (monitor will alarm on the missing check-in)."
|
||||
}
|
||||
} else {
|
||||
Write-Host 'Unhealthy — NOT pinging heartbeat. The external monitor should alert on the missing ping.' -ForegroundColor Yellow
|
||||
Write-Host '(healthchecks.io users: append /fail to the URL for an immediate alert.)'
|
||||
}
|
||||
}
|
||||
|
||||
if (-not $healthy) { Write-Warning 'Investigate now — see od-backup-schedule.md triage.' }
|
||||
Reference in New Issue
Block a user