diff --git a/README.md b/README.md index 4b1f5f6..8fba2de 100644 --- a/README.md +++ b/README.md @@ -1,67 +1,90 @@ -# Kobo Highlight Exfiltrator +# kb_exfiltrator -A simple, statically-linked binary to extract highlights from your Kobo e-reader. +A statically-linked ARM binary that pulls data off a Kobo e-reader. Zero runtime +dependencies — the Kobo has no `curl`, no `sqlite3` CLI, and a busybox `wget` that cannot +POST, so a compiled tool is required rather than convenient. -## Features +## Modes -- **Zero dependencies**: Fully statically linked with musl libc and SQLite -- **Cross-platform**: Builds for both x86_64 (development) and ARM (Kobo) -- **Small binary**: ~780KB for ARM release build -- **Simple**: Just point it at your KoboReader.sqlite database +``` +kb_exfiltrator snapshot [endpoint] [timeout-secs] # primary +kb_exfiltrator highlights [endpoint] # legacy +kb_exfiltrator [endpoint] # legacy alias +``` + +**`snapshot`** uploads a consistent copy of the *whole database*, letting the server parse +whatever it wants — so adding a new statistic never means touching the device again. It: + +1. `VACUUM INTO`s a snapshot to `/tmp` (tmpfs, so nothing touches flash) +2. **closes the database before any network I/O** +3. POSTs it as `application/octet-stream` with `x-kobo-sha256` and `x-kobo-device` headers +4. deletes the temp file + +Omit the endpoint to leave the snapshot on disk for inspection. + +**`highlights`** is the original mode: extract `Bookmark` rows and POST them as JSON. Retained +for debugging and backwards compatibility; the snapshot pipeline supersedes it. + +## Why `VACUUM INTO` rather than copying the file + +Nickel keeps `KoboReader.sqlite` in WAL mode, and recent commits live in the `-wal` file until +a checkpoint. Copying the bare `.sqlite` silently loses reading progress — **measured at 19 +hours of drift on a live device**. Copying all three files instead is not atomic: a checkpoint +landing mid-copy yields a pre-checkpoint main file plus a post-reset WAL, losing data with no +error. `VACUUM INTO` runs inside a read transaction, so it sees WAL-resident commits and emits +a single compacted `journal_mode=delete` file. + +## Timeouts + +A watchdog thread hard-exits the process after `timeout-secs` (default 120), armed for **every** +mode before dispatch. This is not theoretical: a v1 run hung for **16 days** because the server +accepted the connection and never responded, and `std.http.Client` has no timeout. It held the +SQLite handle open the whole time — which is also why snapshot mode releases the database +before it touches the network. ## Building -### For local testing (x86_64) +**Zig 0.15.2 specifically.** Not "or later" — 0.16 (what pacman ships) breaks this code. + ```bash -zig build -./zig-out/bin/kb_exfiltrator local-dev.sqlite +zig build -Dtarget=arm-linux-musleabihf -Doptimize=ReleaseSmall # the Kobo +zig build test -Dtarget=x86_64-linux-musl # tests ``` -### For Kobo (ARM) -```bash -./build-kobo.sh -# or manually: -zig build -Dtarget=arm-linux-musleabihf -Doptimize=ReleaseSmall -``` +The **host** build fails on CachyOS with `unhandled relocation type R_X86_64_PC64` in glibc's +`crt1.o` `.sframe` sections. That is an environment quirk, not a code problem — build and test +against musl as above. -## Usage on Kobo +## Kobo portability constraints -1. Copy `zig-out/bin/kb_exfiltrator` to your Kobo (via USB or SSH) -2. Run: - ```bash - ./kb_exfiltrator /mnt/onboard/.kobo/KoboReader.sqlite - ``` +Both were found only by running on real hardware: -3. Or save to a file: - ```bash - ./kb_exfiltrator /mnt/onboard/.kobo/KoboReader.sqlite > highlights.txt - ``` +- **No `statx`.** The Kobo Elipsa runs Linux 4.9.77; `statx` landed in 4.11. Zig's + `File.stat()` *and* `File.getEndPos()` both issue it, returning ENOSYS as a bare + `error.Unexpected` — and it surfaces *after* the snapshot is written, so it presents as a + post-write failure. Use `readToEndAlloc` with no size hint. **Avoid `File.stat()` anywhere + in this codebase.** +- **`usize` is 32-bit on armv7**, so a `u64` stat size will not coerce. Only the ARM build + catches this. -## How It Works +Zig also collects `test` blocks only from a test unit's root file, hence +`test { _ = @import("snapshot.zig"); }` in `main.zig` — without it the snapshot tests silently +do not run (symptom: `1/1 tests passed` instead of `4/4`). -The program: -1. Opens the SQLite database at the path you provide -2. Queries the `Bookmark` table for all highlights (entries with non-NULL `Text`) -3. Displays each highlight with: - - Date created - - Book identifier (ContentID) - - Highlight text - - Any annotations/notes you added +## Deployment and operations -## Dependencies +This repo is only the binary. The pipeline it feeds — ingest service, database schema, the +automatic dhcpcd trigger, router-based monitoring, and the recovery runbook for when a Kobo +firmware update wipes the rootfs — lives in **`c0smere_devops/kobo-sync/`**: -All dependencies are bundled into the binary: -- SQLite 3.48.0 (amalgamation build) -- musl libc (for ARM target) +| doc | what | +|---|---| +| `kobo-sync/README.md` | how the whole pipeline fits together | +| `kobo-sync/NOTES.md` | device findings and design rationale — **read before changing device behaviour** | +| `kobo-sync/RECOVERY.md` | restoring the trigger after a firmware update | +| `kobo-sync/STORYGRAPH-PLAN.md` | plan for the not-yet-built StoryGraph connector | -## Requirements +## Bundled -- Zig 0.15.2 or later - -## Notes - -- The binary is statically linked, so you can copy just the single file to your Kobo -- Uses musl libc for smaller binary size and better portability -- SQLite is compiled with: - - `SQLITE_THREADSAFE=0`: No threading support (smaller, faster) - - `SQLITE_OMIT_LOAD_EXTENSION`: No dynamic extensions (more secure) +- SQLite 3.48.0 (amalgamation), `SQLITE_THREADSAFE=0` + `SQLITE_OMIT_LOAD_EXTENSION` +- musl libc for the ARM target