Add od-backup-verify runbook: isolated Hyper-V test-restore + health checklist

Verify an od-db-backup cold backup by restoring it into a throwaway,
network-isolated Hyper-V VM (revert to checkpoint after), never over live
prod. Covers matching versions from MANIFEST, whole-datadir restore for
InnoDB, re-pointing the image path, and a pass/fail health checklist
(connects, Help>About version, Database Maintenance Check, recent data,
images open, no missing tables, optional mysqlcheck). Cites Open Dental
Backups / Manual Backups / Database Maintenance. Cross-linked with
od-db-backup.md; README updated.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

README.md

@@ -69,6 +69,7 @@ run time — never commit filled-in values.
 | [`od-cfg-persist.md`](od-cfg-persist.md) | Open Dental — persist "Do not show this window on startup" (writable FreeDentalConfig.xml) |
 | [`od-scan-duplex.md`](od-scan-duplex.md) | Open Dental — duplex ADF scanner captures only one side (TWAIN, Show TWAIN UI branches) |
 | [`od-db-backup.md`](od-db-backup.md) | Open Dental — rock-solid cold backup of the database + images (stop/copy/start MySQL/MariaDB) |
+| [`od-backup-verify.md`](od-backup-verify.md) | Open Dental — verify a backup by test-restoring into an isolated Hyper-V VM (health checklist) |
 | [`scripts/cg-disable.ps1`](scripts/cg-disable.ps1) | Disable Credential Guard, then reboot (prompts to confirm) |
 | [`scripts/od-cfg-acl.ps1`](scripts/od-cfg-acl.ps1) | Grant Users Modify on FreeDentalConfig.xml (Option B of od-cfg-persist) |
 | [`scripts/od-db-backup.ps1`](scripts/od-db-backup.ps1) | Cold backup: stop MySQL/MariaDB, copy whole data dir + OpenDentImages, always restart (od-db-backup) |

od-backup-verify.md

@@ -0,0 +1,116 @@
+# Runbook: Open Dental — Verify a backup by test-restoring into an isolated Hyper-V VM
+
+**Applies to:** Any Open Dental cold backup produced by [`od-db-backup.md`](od-db-backup.md) (an `od-backup-<timestamp>` folder holding `data\`, `OpenDentImages\`, `FreeDentalConfig.xml`, `MANIFEST.txt`).
+**Goal:** Prove a backup is actually restorable and complete by restoring it into a **throwaway, network-isolated Hyper-V virtual machine** and running a short health checklist. A backup you have never restored is a guess.
+
+> [!IMPORTANT]
+> **Never restore over a live production database** — Open Dental warns this can cause irreversible data loss. The whole point of this runbook is to restore somewhere that *cannot* touch production: an isolated VM, off the office network. ([Open Dental — Backups](https://opendental.com/manual/backups.html))
+
+**Cadence:** Test-restore **at least monthly**, and any time you change backup jobs, versions, or hardware. Open Dental recommends restoring to a machine **not connected to the office network** to verify quality. ([Open Dental — Manual Backups](https://www.opendental.com/manual/backupsmanual.html))
+
+**Why Hyper-V:** it gives you that isolated machine on demand, with a **checkpoint** you revert after every test — so each verification starts from a known-clean baseline and the PHI-laden copy is destroyed when you're done. Any spare offline PC or a doctor's disconnected laptop works too; the steps are identical inside.
+
+**Placeholders:**
+
+| Placeholder | Meaning |
+|---|---|
+| `<BACKUP_FOLDER>` | The `od-backup-<timestamp>` folder being verified |
+| `<OD_VERSION>` | Open Dental version from `MANIFEST.txt` — must match on the VM |
+| `<DB_VERSION>` | MySQL/MariaDB version from `MANIFEST.txt` — must match on the VM |
+| `<DB_SERVICE>` | The MySQL/MariaDB service name **inside the VM** |
+| `<VM_DATA_DIR>` | The VM's MySQL data directory (e.g. `C:\mysql\data`) |
+| `<VM_IMAGES_DIR>` | The VM's A-to-Z images folder (e.g. `C:\OpenDentImages`) |
+
+---
+
+## Phase 0 — Read the manifest first
+
+Open `<BACKUP_FOLDER>\MANIFEST.txt`. Note **`Open Dental ver`** (`<OD_VERSION>`) and **`DB engine ver`** (`<DB_VERSION>`). You must install **matching** versions in the VM — a version mismatch either refuses to open or silently converts the database. ([Open Dental — Manual Backups](https://www.opendental.com/manual/backupsmanual.html))
+
+## Phase 1 — Build the isolated verification VM (once, then reuse)
+
+Do this one time; you'll revert to a checkpoint for every future test.
+
+1. **Enable Hyper-V** on a Windows Pro/Enterprise/Server host (elevated PowerShell; reboots):
+   ```
+   Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V -All
+   ```
+2. **Create an isolated network** so the VM can never reach the office network or the live database. In Hyper-V Manager → *Virtual Switch Manager*, create a **Private** (or **Internal**) switch — **not** External. Or leave the VM with **no network adapter** at all.
+   > This isolation is the safety guarantee. An "External" switch would put the test VM on the practice LAN, where a mistake could reach production shares or the live DB. Don't.
+3. **Create the VM** (Gen 2, enough RAM/disk for the database + images), install a supported **Windows** guest.
+4. Inside the VM, install **MySQL/MariaDB `<DB_VERSION>`** and **Open Dental `<OD_VERSION>`** (matching the manifest). Do a normal Open Dental install so `<DB_SERVICE>`, `<VM_DATA_DIR>`, and a local `opendental` database exist.
+5. Shut the VM down cleanly and take a **checkpoint** named `clean-baseline`. This is your reset point.
+
+## Phase 2 — Get the backup into the VM (keep it offline)
+
+Copy `<BACKUP_FOLDER>` into the VM **without** using the office network — e.g. Hyper-V Enhanced Session copy/paste, or attach the backup as a VHD/passthrough disk, or a scratch VHDX. The VM stays isolated the entire time.
+
+## Phase 3 — Restore into the VM
+
+Run these **inside the VM**, elevated. This mirrors Open Dental's documented restore, adapted for our **whole-datadir** cold copy.
+
+1. **Stop the database:**
+   ```
+   net stop <DB_SERVICE>
+   Get-Service <DB_SERVICE>          # confirm Status = Stopped before touching files
+   ```
+2. **Move the VM's existing data aside** (don't delete — lets you retry):
+   ```
+   Rename-Item "<VM_DATA_DIR>" "<VM_DATA_DIR>-old"
+   New-Item -ItemType Directory "<VM_DATA_DIR>" | Out-Null
+   ```
+3. **Restore the whole data directory** from the backup — copy the **entire contents of the backup's `data\` folder**, not just `data\opendental\`:
+   ```
+   robocopy "<BACKUP_FOLDER>\data" "<VM_DATA_DIR>" /E /COPY:DAT /R:2 /W:5
+   ```
+   > For **InnoDB** you must restore the shared tablespace and logs (`ibdata1`, `ib_logfile*`) that live at the datadir root alongside the `opendental` folder — that's why we restore the whole `data\`, matching how [`od-db-backup.md`](od-db-backup.md) captured it. Restoring only the `opendental` subfolder yields a database that won't start.
+4. **Restore the images:**
+   ```
+   robocopy "<BACKUP_FOLDER>\OpenDentImages" "<VM_IMAGES_DIR>" /E /COPY:DAT /R:2 /W:5
+   ```
+5. **Start the database:**
+   ```
+   net start <DB_SERVICE>
+   Get-Service <DB_SERVICE>          # confirm Running
+   ```
+6. **Launch Open Dental in the VM**, connecting to the **local** `opendental` database (localhost). Configure a fresh local connection — do **not** reuse the production `FreeDentalConfig.xml` connection string (it points at the real server). The captured `FreeDentalConfig.xml` is only a **reference** of production settings (e.g. the A-to-Z path).
+7. If Open Dental reports it **can't find the images**, point the image path at `<VM_IMAGES_DIR>`: *Setup → Imaging / Image path* (the A-to-Z path stored in the database is production's; update it to the VM's local folder so restored images resolve).
+
+## Phase 4 — Health checklist: is this backup good?
+
+Work top to bottom. **All must pass** for the backup to count as verified.
+
+- [ ] **Connects & opens** — Open Dental launches against the restored DB with **no connection or startup errors**.
+- [ ] **Version matches** — *Help → About* shows `<OD_VERSION>` (no unexpected DB-conversion prompt).
+- [ ] **Database Maintenance is clean** — *Main Menu → Tools → Database Maintenance*, run **Check** (not Fix). It "checks the database for improper settings, inconsistencies, or corruption." Review the results for **unexpected** corruption/integrity errors. ([Open Dental — Database Maintenance](https://www.opendental.com/manual/databasemaintenance.html))
+- [ ] **Recent data is present and current** — open the **Appointments** module on a recent production date; open 2–3 known patients; check a recent **payment/ledger** entry. Confirm the newest data lines up with production **as of the backup time** (proves the backup is current, not stale).
+- [ ] **Images actually open** — pick a patient known to have scanned docs/X-rays in the Imaging module and **open the files**. This proves the A-to-Z folder restored *and* the image path resolves — a DB-only restore silently loses every document.
+- [ ] **No missing modules/tables** — click through Chart, Family, Account, Imaging; watch for "table doesn't exist" / read-only errors.
+- [ ] *(Optional, deeper)* **Engine-level integrity** — from the VM's MySQL shell, `mysqlcheck -u root -p --check opendental` (or `CHECK TABLE`) to confirm no table-level corruption.
+
+## Phase 5 — Record the result (keep as evidence)
+
+Log, outside the VM: **date verified**, `<BACKUP_FOLDER>` timestamp, `<OD_VERSION>`/`<DB_VERSION>`, each checklist item **pass/fail**, who ran it, and any Database Maintenance findings. This is your DR evidence for E&O / cyber insurance and HIPAA contingency-plan testing.
+
+> A **fail** on any item means the backup is not trustworthy — fix the backup job (not just this restore) and re-verify. Common causes: images folder never included; only `data\opendental\` copied (InnoDB won't start); backup taken hot without stopping the service; version drift.
+
+## Phase 6 — Tear down / reset
+
+1. In Open Dental, close out; **stop the MySQL service** in the VM before shutting down.
+   > Open Dental warns: if a machine used to verify backups **sleeps while MySQL is running, it can corrupt the database.** Stop the service first. ([Open Dental — Manual Backups](https://www.opendental.com/manual/backupsmanual.html))
+2. **Revert the VM to the `clean-baseline` checkpoint.** This resets to a known-clean state for next time **and destroys the restored PHI copy**.
+3. Never reconnect this VM to the office network with a restored database attached.
+
+---
+
+## Security note
+
+The moment you restore, the VM's virtual disk holds the **entire practice's PHI** — treat it exactly like production. Keep the VM **isolated** (Private/Internal switch or no NIC), keep the host and VHDX **encrypted** (BitLocker / encrypted VHDX), and **revert/destroy** the restored state when done so PHI doesn't linger on a test box. HIPAA applies to the copy just as much as to production. See the security note in [`od-db-backup.md`](od-db-backup.md) — on direct-connection setups the captured `FreeDentalConfig.xml` also holds the (reversibly obfuscated) DB password.
+
+## References
+
+- Open Dental manual — Backups (overview; test-restore off-network): <https://opendental.com/manual/backups.html>
+- Open Dental manual — Manual Backups (restore steps; verify on disconnected machine; sleep/corruption warning): <https://www.opendental.com/manual/backupsmanual.html>
+- Open Dental manual — Database Maintenance (Check/Fix health tool): <https://www.opendental.com/manual/databasemaintenance.html>
+- Open Dental manual — Backup Tool: <https://www.opendental.com/manual/backuptool.html>
+- Companion runbook — producing the backup: [`od-db-backup.md`](od-db-backup.md)

od-db-backup.md

@@ -108,7 +108,7 @@ schtasks /Create /TN "OD Nightly Cold Backup" /TR "powershell -NoProfile -Execut
 
 ## 5. Verify — a backup you haven't restored is a guess
 
-- **Test-restore** to an **isolated** machine periodically (matching Open Dental + MySQL/MariaDB versions): `net stop`, rename the existing `opendental` folder aside, drop in the backup's `data\` contents, `net start`, launch Open Dental, spot-check patients/images.
+- **Test-restore** to an **isolated** machine periodically (matching Open Dental + MySQL/MariaDB versions): `net stop`, rename the existing `opendental` folder aside, drop in the backup's `data\` contents, `net start`, launch Open Dental, spot-check patients/images. Full step-by-step with a health checklist: **[`od-backup-verify.md`](od-backup-verify.md)** (isolated Hyper-V test-restore).
 - **Never restore over a live production database** — data loss is irreversible.
 - Confirm the nightly job is actually producing dated folders and that they leave the building (off-site / immutable copy) — ransomware that reaches the server will reach on-line backups too.