Document snapshot mode and the Kobo portability constraints
The README still described the binary as a highlights extractor, which has not been the primary mode since snapshot was added. Records the two constraints that only real hardware surfaced, because both cost real time and neither is discoverable from a host build: the Elipsa's 4.9 kernel predates statx, so Zig's File.stat and getEndPos fail as a bare error.Unexpected after the snapshot is already written; and usize is 32-bit on armv7, so a u64 stat size will not coerce. Also pins Zig to 0.15.2 rather than "or later", since the 0.16 in pacman breaks the build, and notes that the host build fails on CachyOS for reasons unrelated to this code. Points at c0smere_devops/kobo-sync for deployment and operations rather than duplicating them here — this repo is just the binary; the ingest service, schema, trigger, monitoring and recovery runbook live with the rest of the infrastructure. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
@@ -1,67 +1,90 @@
|
||||
# Kobo Highlight Exfiltrator
|
||||
# kb_exfiltrator
|
||||
|
||||
A simple, statically-linked binary to extract highlights from your Kobo e-reader.
|
||||
A statically-linked ARM binary that pulls data off a Kobo e-reader. Zero runtime
|
||||
dependencies — the Kobo has no `curl`, no `sqlite3` CLI, and a busybox `wget` that cannot
|
||||
POST, so a compiled tool is required rather than convenient.
|
||||
|
||||
## Features
|
||||
## Modes
|
||||
|
||||
- **Zero dependencies**: Fully statically linked with musl libc and SQLite
|
||||
- **Cross-platform**: Builds for both x86_64 (development) and ARM (Kobo)
|
||||
- **Small binary**: ~780KB for ARM release build
|
||||
- **Simple**: Just point it at your KoboReader.sqlite database
|
||||
```
|
||||
kb_exfiltrator snapshot <db> [endpoint] [timeout-secs] # primary
|
||||
kb_exfiltrator highlights <db> [endpoint] # legacy
|
||||
kb_exfiltrator <db> [endpoint] # legacy alias
|
||||
```
|
||||
|
||||
**`snapshot`** uploads a consistent copy of the *whole database*, letting the server parse
|
||||
whatever it wants — so adding a new statistic never means touching the device again. It:
|
||||
|
||||
1. `VACUUM INTO`s a snapshot to `/tmp` (tmpfs, so nothing touches flash)
|
||||
2. **closes the database before any network I/O**
|
||||
3. POSTs it as `application/octet-stream` with `x-kobo-sha256` and `x-kobo-device` headers
|
||||
4. deletes the temp file
|
||||
|
||||
Omit the endpoint to leave the snapshot on disk for inspection.
|
||||
|
||||
**`highlights`** is the original mode: extract `Bookmark` rows and POST them as JSON. Retained
|
||||
for debugging and backwards compatibility; the snapshot pipeline supersedes it.
|
||||
|
||||
## Why `VACUUM INTO` rather than copying the file
|
||||
|
||||
Nickel keeps `KoboReader.sqlite` in WAL mode, and recent commits live in the `-wal` file until
|
||||
a checkpoint. Copying the bare `.sqlite` silently loses reading progress — **measured at 19
|
||||
hours of drift on a live device**. Copying all three files instead is not atomic: a checkpoint
|
||||
landing mid-copy yields a pre-checkpoint main file plus a post-reset WAL, losing data with no
|
||||
error. `VACUUM INTO` runs inside a read transaction, so it sees WAL-resident commits and emits
|
||||
a single compacted `journal_mode=delete` file.
|
||||
|
||||
## Timeouts
|
||||
|
||||
A watchdog thread hard-exits the process after `timeout-secs` (default 120), armed for **every**
|
||||
mode before dispatch. This is not theoretical: a v1 run hung for **16 days** because the server
|
||||
accepted the connection and never responded, and `std.http.Client` has no timeout. It held the
|
||||
SQLite handle open the whole time — which is also why snapshot mode releases the database
|
||||
before it touches the network.
|
||||
|
||||
## Building
|
||||
|
||||
### For local testing (x86_64)
|
||||
**Zig 0.15.2 specifically.** Not "or later" — 0.16 (what pacman ships) breaks this code.
|
||||
|
||||
```bash
|
||||
zig build
|
||||
./zig-out/bin/kb_exfiltrator local-dev.sqlite
|
||||
zig build -Dtarget=arm-linux-musleabihf -Doptimize=ReleaseSmall # the Kobo
|
||||
zig build test -Dtarget=x86_64-linux-musl # tests
|
||||
```
|
||||
|
||||
### For Kobo (ARM)
|
||||
```bash
|
||||
./build-kobo.sh
|
||||
# or manually:
|
||||
zig build -Dtarget=arm-linux-musleabihf -Doptimize=ReleaseSmall
|
||||
```
|
||||
The **host** build fails on CachyOS with `unhandled relocation type R_X86_64_PC64` in glibc's
|
||||
`crt1.o` `.sframe` sections. That is an environment quirk, not a code problem — build and test
|
||||
against musl as above.
|
||||
|
||||
## Usage on Kobo
|
||||
## Kobo portability constraints
|
||||
|
||||
1. Copy `zig-out/bin/kb_exfiltrator` to your Kobo (via USB or SSH)
|
||||
2. Run:
|
||||
```bash
|
||||
./kb_exfiltrator /mnt/onboard/.kobo/KoboReader.sqlite
|
||||
```
|
||||
Both were found only by running on real hardware:
|
||||
|
||||
3. Or save to a file:
|
||||
```bash
|
||||
./kb_exfiltrator /mnt/onboard/.kobo/KoboReader.sqlite > highlights.txt
|
||||
```
|
||||
- **No `statx`.** The Kobo Elipsa runs Linux 4.9.77; `statx` landed in 4.11. Zig's
|
||||
`File.stat()` *and* `File.getEndPos()` both issue it, returning ENOSYS as a bare
|
||||
`error.Unexpected` — and it surfaces *after* the snapshot is written, so it presents as a
|
||||
post-write failure. Use `readToEndAlloc` with no size hint. **Avoid `File.stat()` anywhere
|
||||
in this codebase.**
|
||||
- **`usize` is 32-bit on armv7**, so a `u64` stat size will not coerce. Only the ARM build
|
||||
catches this.
|
||||
|
||||
## How It Works
|
||||
Zig also collects `test` blocks only from a test unit's root file, hence
|
||||
`test { _ = @import("snapshot.zig"); }` in `main.zig` — without it the snapshot tests silently
|
||||
do not run (symptom: `1/1 tests passed` instead of `4/4`).
|
||||
|
||||
The program:
|
||||
1. Opens the SQLite database at the path you provide
|
||||
2. Queries the `Bookmark` table for all highlights (entries with non-NULL `Text`)
|
||||
3. Displays each highlight with:
|
||||
- Date created
|
||||
- Book identifier (ContentID)
|
||||
- Highlight text
|
||||
- Any annotations/notes you added
|
||||
## Deployment and operations
|
||||
|
||||
## Dependencies
|
||||
This repo is only the binary. The pipeline it feeds — ingest service, database schema, the
|
||||
automatic dhcpcd trigger, router-based monitoring, and the recovery runbook for when a Kobo
|
||||
firmware update wipes the rootfs — lives in **`c0smere_devops/kobo-sync/`**:
|
||||
|
||||
All dependencies are bundled into the binary:
|
||||
- SQLite 3.48.0 (amalgamation build)
|
||||
- musl libc (for ARM target)
|
||||
| doc | what |
|
||||
|---|---|
|
||||
| `kobo-sync/README.md` | how the whole pipeline fits together |
|
||||
| `kobo-sync/NOTES.md` | device findings and design rationale — **read before changing device behaviour** |
|
||||
| `kobo-sync/RECOVERY.md` | restoring the trigger after a firmware update |
|
||||
| `kobo-sync/STORYGRAPH-PLAN.md` | plan for the not-yet-built StoryGraph connector |
|
||||
|
||||
## Requirements
|
||||
## Bundled
|
||||
|
||||
- Zig 0.15.2 or later
|
||||
|
||||
## Notes
|
||||
|
||||
- The binary is statically linked, so you can copy just the single file to your Kobo
|
||||
- Uses musl libc for smaller binary size and better portability
|
||||
- SQLite is compiled with:
|
||||
- `SQLITE_THREADSAFE=0`: No threading support (smaller, faster)
|
||||
- `SQLITE_OMIT_LOAD_EXTENSION`: No dynamic extensions (more secure)
|
||||
- SQLite 3.48.0 (amalgamation), `SQLITE_THREADSAFE=0` + `SQLITE_OMIT_LOAD_EXTENSION`
|
||||
- musl libc for the ARM target
|
||||
|
||||
Reference in New Issue
Block a user