igodwin
f2a979f047
The repo is public and files are fetched by raw URL, so a reader who lands on one runbook never sees the README -- the repo's context does not travel with the file. Each .md now carries two lines under the title, each .ps1 the equivalent at the end of its .NOTES block. Deliberately two lines, not a paragraph. These files are read through `| more` on a client console mid-incident, and the top of the file is where the procedure-specific warnings live -- never a live chart, stop the service before copying, confirm authorization before acting. A legal preamble above those competes with them and trains people to skip past. Wording aims at a stranger who found the repo, not at the quality of the procedure: these double as documented-procedure evidence for E&O, and language implying the content is unreliable works against that. MIT rather than no license: the warranty and liability disclaimer is the part that does the work, and leaving it unlicensed makes reuse ambiguous rather than disclaimed. Also fixes 5 stale ops/rb URLs in scripts/*.ps1 that the previous commit missed -- it only swept the .md files. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HwcG1jLs1T425QRMxtjxP7
145 lines
8.8 KiB
Markdown
145 lines
8.8 KiB
Markdown
# Runbook: Open Dental — Schedule & monitor backups (Windows)
|
|
|
|
> For qualified IT professionals, on systems they are authorized to administer.
|
|
> Provided as-is, without warranty — verify it fits your environment. See LICENSE.
|
|
|
|
**Applies to:** The Open Dental database server, once [`od-db-backup.md`](od-db-backup.md) is proven to run by hand.
|
|
**Goal:** Make the backup run **automatically every day**, replicate it **off-site**, and **monitor** it so a silent failure gets noticed within a day — not the day you need a restore.
|
|
|
|
Open Dental's floor is **at least one backup per day**, with a combination of methods and at least one **automated**. ([Open Dental — Backups](https://opendental.com/manual/backups.html)) This runbook automates and watches the cold backup.
|
|
|
|
> [!IMPORTANT]
|
|
> **Monitoring is the half everyone skips.** A backup job that silently stopped, a full disk, or a powered-off server sends **no error email** — the absence of failure looks exactly like success. The only reliable signal is a **positive heartbeat that goes missing** (a dead-man's-switch). Build that, or you don't have monitoring.
|
|
|
|
**Placeholders:**
|
|
|
|
| Placeholder | Meaning |
|
|
|---|---|
|
|
| `<DEST>` | Backup destination root (holds the `od-backup-<timestamp>` folders) |
|
|
| `<BACKUP_SCRIPT>` | Filled-in local copy of `od-db-backup.ps1` (paths baked in) — **private tier** |
|
|
| `<CHECK_SCRIPT>` | Filled-in local copy of `od-backup-check.ps1` (`<DEST>` + heartbeat baked in) — **private tier** |
|
|
| `<HEARTBEAT_URL>` | Dead-man's-switch ping URL (healthchecks.io, Uptime Kuma push, RMM, etc.) — **private tier** |
|
|
|
|
---
|
|
|
|
## The jobs and their cadence
|
|
|
|
| Job | What | When | Runbook |
|
|
|---|---|---|---|
|
|
| **1. Cold backup** | Stop DB, copy data + images, restart | Nightly, off-hours | [`od-db-backup.md`](od-db-backup.md) |
|
|
| **2. Off-site upload** | Replicate `<DEST>` to cloud/immutable | After job 1 finishes | [`od-db-backup.md` §6](od-db-backup.md) |
|
|
| **3. Health check** | Verify newest backup + heartbeat ping | After job 1 finishes | this runbook + [`od-backup-check.ps1`](scripts/od-backup-check.ps1) |
|
|
| **4. Test-restore** | Full restore into isolated VM | Monthly | [`od-backup-verify.md`](od-backup-verify.md) |
|
|
|
|
Stagger them so nothing reads a half-written folder, e.g. **backup 23:30 → check 00:45 → off-site upload 01:00**.
|
|
|
|
## Why a scheduled *local copy*, not `irm | iex`
|
|
|
|
`irm | iex` prompts interactively — it can't run unattended. For scheduling, keep a **filled-in copy** of each script (`<BACKUP_SCRIPT>`, `<CHECK_SCRIPT>`) with `<DEST>`, service name, and paths baked in, stored in the **private tier** (never committed here). The public scripts stay the interactive/spot-check version.
|
|
|
|
---
|
|
|
|
## 1. Schedule the nightly cold backup
|
|
|
|
Register `<BACKUP_SCRIPT>` to run as **SYSTEM**, highest privileges, off-hours. From an elevated prompt:
|
|
|
|
```
|
|
schtasks /Create /TN "OD Nightly Cold Backup" ^
|
|
/TR "powershell -NoProfile -ExecutionPolicy Bypass -File <BACKUP_SCRIPT>" ^
|
|
/SC DAILY /ST 23:30 /RU SYSTEM /RL HIGHEST
|
|
```
|
|
|
|
- `<BACKUP_SCRIPT>` is the filled-in copy — it must **not** prompt (no `Read-Host` for the scheduled path) and must still do the stop → verify-stopped → copy → **always-restart** sequence.
|
|
- Ensure the task is set to **run whether or not a user is logged on** and, if the server sleeps, **wake the computer to run** (Task Scheduler → task → *Conditions*).
|
|
- Confirm the server actually **stays on** overnight (disable sleep/hibernate on the server).
|
|
|
|
## 2. Schedule the off-site upload (after the backup)
|
|
|
|
Point your off-site tool (Duplicati → Backblaze B2, `rclone`, Veeam, etc.) at **`<DEST>`** — it uploads *this backup*, it does **not** re-run the stop/copy against the live database. Schedule it **after** job 1 completes and stagger the start. Enable that tool's **own** email/report and **object-lock/immutability** for the ransomware-resistant off-site copy. Details: [`od-db-backup.md` §6](od-db-backup.md).
|
|
|
|
## 3. Schedule the health check (and heartbeat)
|
|
|
|
Register `<CHECK_SCRIPT>` to run shortly **after** the backup window. It verifies the newest `od-backup-<timestamp>` is fresh, complete, and sensibly sized, appends to `<DEST>\backup-check.log`, and — when healthy — pings `<HEARTBEAT_URL>`.
|
|
|
|
```
|
|
schtasks /Create /TN "OD Backup Health Check" ^
|
|
/TR "powershell -NoProfile -ExecutionPolicy Bypass -File <CHECK_SCRIPT>" ^
|
|
/SC DAILY /ST 00:45 /RU SYSTEM /RL HIGHEST
|
|
```
|
|
|
|
The check is **read-only** — no DB, no service, no file changes — so it's safe to run any time, including a manual spot-check:
|
|
|
|
```
|
|
irm rb.godwinsystems.com/scripts/od-backup-check.ps1 | iex
|
|
```
|
|
|
|
---
|
|
|
|
## Monitoring — the three layers of "is it actually working?"
|
|
|
|
Use all three; each catches what the others miss.
|
|
|
|
### Layer 1 — Did the job run? (Task Scheduler)
|
|
|
|
`Last Run Result = 0x0` and a recent `Last Run Time` on both tasks:
|
|
|
|
```
|
|
Get-ScheduledTaskInfo -TaskName "OD Nightly Cold Backup"
|
|
Get-ScheduledTaskInfo -TaskName "OD Backup Health Check"
|
|
```
|
|
|
|
Enable **All Tasks History** in Task Scheduler so you can see misfires. Catches: task disabled, wrong credentials, server was off.
|
|
|
|
### Layer 2 — Is the output good? (the health check)
|
|
|
|
`od-backup-check.ps1` / `<CHECK_SCRIPT>` confirms the newest backup is:
|
|
|
|
- **Fresh** — written within 24 h (else the job silently stopped),
|
|
- **Complete** — has `data\`, `OpenDentImages\`, `MANIFEST.txt`, and a `backup.log` that ends in success with no `ERROR`/`CRITICAL`,
|
|
- **Sane size** — `data\` isn't near-empty and isn't a fraction of the prior run (catches truncation / a filling disk).
|
|
|
|
Read `<DEST>\backup-check.log` for the running PASS/FAIL trail. Catches: partial copies, missing images, service that didn't restart, dying disk.
|
|
|
|
### Layer 3 — Dead-man's-switch (the one that catches everything)
|
|
|
|
Register a check with an **external** monitor — [healthchecks.io](https://healthchecks.io), Uptime Kuma (push), or your RMM — that expects a daily ping. `<CHECK_SCRIPT>` pings `<HEARTBEAT_URL>` **only when the backup is healthy**. If the backup breaks, the check fails, the script crashes, or **the whole server is offline**, the ping never arrives and the monitor alerts you.
|
|
|
|
- Set the monitor's **period to ~1 day** with a grace window past your backup+check schedule.
|
|
- healthchecks.io users: `<CHECK_SCRIPT>` can hit `<HEARTBEAT_URL>/fail` on failure for an **immediate** alert instead of waiting out the grace period.
|
|
- This is the layer that turns "no news" into an actual alarm. Without it, a dead backup is invisible until a restore fails.
|
|
|
|
### Layer 4 — Off-site tool's own report
|
|
|
|
Your cloud tool (Duplicati/Veeam/rclone wrapper) should send its **own** success/failure summary and expose versions/immutability in the provider console. Confirms the copy actually left the building.
|
|
|
|
---
|
|
|
|
## On failure — triage
|
|
|
|
When Layer 2/3 flags a problem, in rough order:
|
|
|
|
1. **Is the database up?** `Get-Service <DB_SERVICE>` — if the backup died mid-run, confirm the service **restarted** (the script's `finally` should have; verify). The practice being able to work comes first.
|
|
2. **Destination full / offline?** Free space on `<DEST>`; is the disk/UNC reachable? Prune old `od-backup-<timestamp>` generations if space-bound.
|
|
3. **Partial/most-recent folder incomplete?** Check that run's `backup.log` for the `ERROR`/`CRITICAL` line; re-run the backup by hand ([`od-db-backup.md`](od-db-backup.md)).
|
|
4. **Version drift?** A recent Open Dental/MySQL update can change paths — reconcile against `MANIFEST.txt`.
|
|
5. **Off-site not uploading?** Check the cloud tool's log and that it sources `<DEST>` (not the live datadir).
|
|
|
|
Then re-run the health check and confirm the heartbeat goes green.
|
|
|
|
## Records / evidence
|
|
|
|
Keep the `<DEST>\backup-check.log`, the monitor's uptime history, and the monthly **test-restore** results ([`od-backup-verify.md`](od-backup-verify.md)) together. That trail is your DR evidence for E&O / cyber insurance and HIPAA contingency-plan testing.
|
|
|
|
---
|
|
|
|
## Security note
|
|
|
|
`<HEARTBEAT_URL>` is a capability — anyone with it can spoof "backup healthy." Treat it as a secret: keep it in the **private tier**, out of this repo, out of screenshots. Don't let monitor check names or heartbeat URLs encode a client's identity. The `<DEST>` folders and `<BACKUP_SCRIPT>`/`<CHECK_SCRIPT>` may reference real paths — keep the filled-in copies private and the destination encrypted/access-controlled (it holds PHI).
|
|
|
|
## References
|
|
|
|
- Open Dental manual — Backups (daily minimum; automated + combined methods): <https://opendental.com/manual/backups.html>
|
|
- Microsoft — `schtasks` / Scheduled Tasks: <https://learn.microsoft.com/windows-server/administration/windows-commands/schtasks>
|
|
- healthchecks.io — dead-man's-switch cron monitoring: <https://healthchecks.io/>
|
|
- Companion runbooks — [`od-db-backup.md`](od-db-backup.md) (produce the backup), [`od-backup-verify.md`](od-backup-verify.md) (test-restore)
|