kb_exfiltrator/README.md

kb_exfiltrator

A statically-linked ARM binary that pulls data off a Kobo e-reader. Zero runtime dependencies — the Kobo has no curl, no sqlite3 CLI, and a busybox wget that cannot POST, so a compiled tool is required rather than convenient.

Modes

kb_exfiltrator snapshot   <db> [endpoint] [timeout-secs]   # primary
kb_exfiltrator highlights <db> [endpoint]                  # legacy
kb_exfiltrator            <db> [endpoint]                  # legacy alias

snapshot uploads a consistent copy of the whole database, letting the server parse whatever it wants — so adding a new statistic never means touching the device again. It:

1. VACUUM INTOs a snapshot to /tmp (tmpfs, so nothing touches flash)
2. closes the database before any network I/O
3. POSTs it as application/octet-stream with x-kobo-sha256 and x-kobo-device headers
4. deletes the temp file

Omit the endpoint to leave the snapshot on disk for inspection.

highlights is the original mode: extract Bookmark rows and POST them as JSON. Retained for debugging and backwards compatibility; the snapshot pipeline supersedes it.

Why VACUUM INTO rather than copying the file

Nickel keeps KoboReader.sqlite in WAL mode, and recent commits live in the -wal file until a checkpoint. Copying the bare .sqlite silently loses reading progress — measured at 19 hours of drift on a live device. Copying all three files instead is not atomic: a checkpoint landing mid-copy yields a pre-checkpoint main file plus a post-reset WAL, losing data with no error. VACUUM INTO runs inside a read transaction, so it sees WAL-resident commits and emits a single compacted journal_mode=delete file.

Timeouts

A watchdog thread hard-exits the process after timeout-secs (default 120), armed for every mode before dispatch. This is not theoretical: a v1 run hung for 16 days because the server accepted the connection and never responded, and std.http.Client has no timeout. It held the SQLite handle open the whole time — which is also why snapshot mode releases the database before it touches the network.

Building

Zig 0.15.2 specifically. Not "or later" — 0.16 (what pacman ships) breaks this code.

zig build -Dtarget=arm-linux-musleabihf -Doptimize=ReleaseSmall   # the Kobo
zig build test -Dtarget=x86_64-linux-musl                         # tests

The host build fails on CachyOS with unhandled relocation type R_X86_64_PC64 in glibc's crt1.o .sframe sections. That is an environment quirk, not a code problem — build and test against musl as above.

Kobo portability constraints

Both were found only by running on real hardware:

• No statx. The Kobo Elipsa runs Linux 4.9.77; statx landed in 4.11. Zig's File.stat() and File.getEndPos() both issue it, returning ENOSYS as a bare error.Unexpected — and it surfaces after the snapshot is written, so it presents as a post-write failure. Use readToEndAlloc with no size hint. Avoid File.stat() anywhere in this codebase.
• usize is 32-bit on armv7, so a u64 stat size will not coerce. Only the ARM build catches this.

Zig also collects test blocks only from a test unit's root file, hence test { _ = @import("snapshot.zig"); } in main.zig — without it the snapshot tests silently do not run (symptom: 1/1 tests passed instead of 4/4).

Deployment and operations

This repo is only the binary. The pipeline it feeds — ingest service, database schema, the automatic dhcpcd trigger, router-based monitoring, and the recovery runbook for when a Kobo firmware update wipes the rootfs — lives in c0smere_devops/kobo-sync/:

doc	what
kobo-sync/README.md	how the whole pipeline fits together
kobo-sync/NOTES.md	device findings and design rationale — read before changing device behaviour
kobo-sync/RECOVERY.md	restoring the trigger after a firmware update
kobo-sync/STORYGRAPH-PLAN.md	plan for the not-yet-built StoryGraph connector

Bundled

• SQLite 3.48.0 (amalgamation), SQLITE_THREADSAFE=0 + SQLITE_OMIT_LOAD_EXTENSION
• musl libc for the ARM target