wesandClaude Opus 5
fc93fc3a45
The README still described the binary as a highlights extractor, which has not been the primary mode since snapshot was added. Records the two constraints that only real hardware surfaced, because both cost real time and neither is discoverable from a host build: the Elipsa's 4.9 kernel predates statx, so Zig's File.stat and getEndPos fail as a bare error.Unexpected after the snapshot is already written; and usize is 32-bit on armv7, so a u64 stat size will not coerce. Also pins Zig to 0.15.2 rather than "or later", since the 0.16 in pacman breaks the build, and notes that the host build fails on CachyOS for reasons unrelated to this code. Points at c0smere_devops/kobo-sync for deployment and operations rather than duplicating them here — this repo is just the binary; the ingest service, schema, trigger, monitoring and recovery runbook live with the rest of the infrastructure. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
91 lines
4.1 KiB
Markdown
91 lines
4.1 KiB
Markdown
# kb_exfiltrator
|
|
|
|
A statically-linked ARM binary that pulls data off a Kobo e-reader. Zero runtime
|
|
dependencies — the Kobo has no `curl`, no `sqlite3` CLI, and a busybox `wget` that cannot
|
|
POST, so a compiled tool is required rather than convenient.
|
|
|
|
## Modes
|
|
|
|
```
|
|
kb_exfiltrator snapshot <db> [endpoint] [timeout-secs] # primary
|
|
kb_exfiltrator highlights <db> [endpoint] # legacy
|
|
kb_exfiltrator <db> [endpoint] # legacy alias
|
|
```
|
|
|
|
**`snapshot`** uploads a consistent copy of the *whole database*, letting the server parse
|
|
whatever it wants — so adding a new statistic never means touching the device again. It:
|
|
|
|
1. `VACUUM INTO`s a snapshot to `/tmp` (tmpfs, so nothing touches flash)
|
|
2. **closes the database before any network I/O**
|
|
3. POSTs it as `application/octet-stream` with `x-kobo-sha256` and `x-kobo-device` headers
|
|
4. deletes the temp file
|
|
|
|
Omit the endpoint to leave the snapshot on disk for inspection.
|
|
|
|
**`highlights`** is the original mode: extract `Bookmark` rows and POST them as JSON. Retained
|
|
for debugging and backwards compatibility; the snapshot pipeline supersedes it.
|
|
|
|
## Why `VACUUM INTO` rather than copying the file
|
|
|
|
Nickel keeps `KoboReader.sqlite` in WAL mode, and recent commits live in the `-wal` file until
|
|
a checkpoint. Copying the bare `.sqlite` silently loses reading progress — **measured at 19
|
|
hours of drift on a live device**. Copying all three files instead is not atomic: a checkpoint
|
|
landing mid-copy yields a pre-checkpoint main file plus a post-reset WAL, losing data with no
|
|
error. `VACUUM INTO` runs inside a read transaction, so it sees WAL-resident commits and emits
|
|
a single compacted `journal_mode=delete` file.
|
|
|
|
## Timeouts
|
|
|
|
A watchdog thread hard-exits the process after `timeout-secs` (default 120), armed for **every**
|
|
mode before dispatch. This is not theoretical: a v1 run hung for **16 days** because the server
|
|
accepted the connection and never responded, and `std.http.Client` has no timeout. It held the
|
|
SQLite handle open the whole time — which is also why snapshot mode releases the database
|
|
before it touches the network.
|
|
|
|
## Building
|
|
|
|
**Zig 0.15.2 specifically.** Not "or later" — 0.16 (what pacman ships) breaks this code.
|
|
|
|
```bash
|
|
zig build -Dtarget=arm-linux-musleabihf -Doptimize=ReleaseSmall # the Kobo
|
|
zig build test -Dtarget=x86_64-linux-musl # tests
|
|
```
|
|
|
|
The **host** build fails on CachyOS with `unhandled relocation type R_X86_64_PC64` in glibc's
|
|
`crt1.o` `.sframe` sections. That is an environment quirk, not a code problem — build and test
|
|
against musl as above.
|
|
|
|
## Kobo portability constraints
|
|
|
|
Both were found only by running on real hardware:
|
|
|
|
- **No `statx`.** The Kobo Elipsa runs Linux 4.9.77; `statx` landed in 4.11. Zig's
|
|
`File.stat()` *and* `File.getEndPos()` both issue it, returning ENOSYS as a bare
|
|
`error.Unexpected` — and it surfaces *after* the snapshot is written, so it presents as a
|
|
post-write failure. Use `readToEndAlloc` with no size hint. **Avoid `File.stat()` anywhere
|
|
in this codebase.**
|
|
- **`usize` is 32-bit on armv7**, so a `u64` stat size will not coerce. Only the ARM build
|
|
catches this.
|
|
|
|
Zig also collects `test` blocks only from a test unit's root file, hence
|
|
`test { _ = @import("snapshot.zig"); }` in `main.zig` — without it the snapshot tests silently
|
|
do not run (symptom: `1/1 tests passed` instead of `4/4`).
|
|
|
|
## Deployment and operations
|
|
|
|
This repo is only the binary. The pipeline it feeds — ingest service, database schema, the
|
|
automatic dhcpcd trigger, router-based monitoring, and the recovery runbook for when a Kobo
|
|
firmware update wipes the rootfs — lives in **`c0smere_devops/kobo-sync/`**:
|
|
|
|
| doc | what |
|
|
|---|---|
|
|
| `kobo-sync/README.md` | how the whole pipeline fits together |
|
|
| `kobo-sync/NOTES.md` | device findings and design rationale — **read before changing device behaviour** |
|
|
| `kobo-sync/RECOVERY.md` | restoring the trigger after a firmware update |
|
|
| `kobo-sync/STORYGRAPH-PLAN.md` | plan for the not-yet-built StoryGraph connector |
|
|
|
|
## Bundled
|
|
|
|
- SQLite 3.48.0 (amalgamation), `SQLITE_THREADSAFE=0` + `SQLITE_OMIT_LOAD_EXTENSION`
|
|
- musl libc for the ARM target
|