Add complete Kobo highlight sync system with NickelMenu integration

- Add text cleaning to remove page reference numbers (e.g., "10After" -> "After")
- Implement cleanHighlightText() function with boundary detection
- Add NickelMenu configuration for UI integration
- Create sync wrapper scripts (with and without FBInk notifications)
- Add installation guides for Kobo deployment
- Include pointer/slice explanation examples for learning

Features:
- Extracts highlights from Kobo SQLite database
- Cleans page references from academic texts
- Computes SHA-256 hashes for deduplication
- POSTs JSON to Flask API endpoint
- Supports ARM static linking for Kobo hardware
- Optional FBInk notifications for user feedback

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

FBINK_GUIDE.md

@@ -0,0 +1,98 @@
+# FBInk Integration Guide
+
+## What is FBInk?
+
+FBInk is a tool for drawing text and images on e-ink displays. It's commonly bundled with KOReader and other Kobo mods.
+
+## Check if FBInk is installed
+
+```bash
+ssh root@192.168.88.110
+which fbink
+# or
+fbink --version
+```
+
+If not installed, it usually comes with KOReader at `/usr/local/koreader/fbink`
+
+## Basic Usage in Scripts
+
+### Show a message
+```bash
+fbink -pm "Hello, World!"
+```
+
+### Show message at specific position
+```bash
+fbink -pm -y 10 "Message at row 10"
+```
+
+### Clear the screen
+```bash
+fbink -s
+```
+
+### Show without refresh (faster)
+```bash
+fbink -q -pm "Quick message"
+```
+
+## Common Options
+
+- `-pm` - Print centered (middle)
+- `-q` - Quiet mode (no verbose output)
+- `-y N` - Print at row N
+- `-s` - Clear screen
+- `-h` - Refresh (full screen)
+- `-f` - Flash (for animations)
+
+## Testing FBInk
+
+SSH into your Kobo and try:
+```bash
+fbink -pm "Testing FBInk!"
+sleep 2
+fbink -s  # Clear it
+```
+
+## Enhanced sync_highlights.sh
+
+The `sync_highlights_with_fbink.sh` script shows:
+- "Syncing highlights..." while running
+- "✓ Synced N highlights!" on success
+- "✗ Sync failed! Check log" on error
+
+Auto-clears after 2-3 seconds so it doesn't clutter the screen.
+
+## Installation
+
+Replace your existing sync script:
+```bash
+scp sync_highlights_with_fbink.sh root@192.168.88.110:/usr/local/bin/sync_highlights.sh
+ssh root@192.168.88.110 chmod +x /usr/local/bin/sync_highlights.sh
+```
+
+## Custom Messages
+
+You can customize the messages in the script:
+
+```bash
+# Change this line:
+$FBINK -q -pm -y 10 "✓ Synced $COUNT highlights!"
+
+# To something like:
+$FBINK -q -pm -y 10 "📚 $COUNT new highlights saved!"
+```
+
+## Advanced: Progress Bar
+
+For future enhancements, FBInk can show progress bars:
+```bash
+fbink -pm -P 50  # Show 50% progress
+```
+
+Could be useful if syncing takes a while!
+
+## Fallback Behavior
+
+The script automatically detects if FBInk is available. If not found, it runs silently (just logs to file). So it works either way!

INSTALL_KOBO.md

@@ -0,0 +1,85 @@
+# Installing kb_exfiltrator on Kobo
+
+## Prerequisites
+
+1. **NickelMenu** installed on your Kobo (usually comes with KOReader)
+2. SSH access to your Kobo
+3. Kobo connected to WiFi
+
+## Installation
+
+### 1. Copy files to Kobo
+
+```bash
+# Copy the binary
+scp kb_exfiltrator-kobo-arm root@192.168.88.110:/usr/local/bin/kb_exfiltrator
+
+# Copy the wrapper script
+scp sync_highlights.sh root@192.168.88.110:/usr/local/bin/sync_highlights.sh
+
+# Copy NickelMenu config
+scp nickel_menu_config root@192.168.88.110:/mnt/onboard/.adds/nm/kb_exfiltrator
+```
+
+### 2. Set permissions via SSH
+
+```bash
+ssh root@192.168.88.110
+
+chmod +x /usr/local/bin/kb_exfiltrator
+chmod +x /usr/local/bin/sync_highlights.sh
+```
+
+### 3. Update the endpoint URL
+
+Edit the script on the Kobo to use your server's IP/hostname:
+
+```bash
+vi /usr/local/bin/sync_highlights.sh
+# Change: ENDPOINT="http://192.168.88.69:42070/api/highlights"
+# To your actual server URL
+```
+
+### 4. Restart Kobo
+
+The NickelMenu config is read on boot, so restart your Kobo:
+```bash
+reboot
+```
+
+## Usage
+
+After rebooting, you'll see "Sync Highlights" in:
+- Main menu (press the top-left Kobo icon)
+- Reader menu (while reading a book)
+
+Tap it to sync your highlights to your server!
+
+## Logs
+
+Check sync logs at: `/mnt/onboard/.adds/kb_exfiltrator.log`
+
+## Manual Usage (via SSH)
+
+```bash
+ssh root@192.168.88.110
+/usr/local/bin/kb_exfiltrator /mnt/onboard/.kobo/KoboReader.sqlite http://yourserver:42070/api/highlights
+```
+
+## Troubleshooting
+
+### Menu item doesn't appear
+- Check `/mnt/onboard/.adds/nm/kb_exfiltrator` exists
+- Restart Kobo
+- Check NickelMenu is installed
+
+### Sync fails
+- Check log: `cat /mnt/onboard/.adds/kb_exfiltrator.log`
+- Verify WiFi is connected
+- Test manually via SSH
+- Verify server is running: `curl http://192.168.88.69:42070/`
+
+### "No route to host"
+- Kobo WiFi disconnected
+- Server IP changed
+- Edit `/usr/local/bin/sync_highlights.sh` with correct IP

nickel_menu_config

@@ -0,0 +1,9 @@
+# NickelMenu configuration for kb_exfiltrator
+# Copy this to: /mnt/onboard/.adds/nm/kb_exfiltrator
+# Or append to existing NickelMenu config
+
+menu_item :main    :Sync Highlights    :cmd_spawn         :quiet :exec /usr/local/bin/sync_highlights.sh
+menu_item :reader  :Sync Highlights    :cmd_spawn         :quiet :exec /usr/local/bin/sync_highlights.sh
+
+# Alternative: Show output in a dialog
+# menu_item :main    :Sync Highlights    :cmd_output        :500:/usr/local/bin/sync_highlights.sh

pointer_explanation.zig

@@ -0,0 +1,28 @@
+const std = @import("std");
+
+pub fn main() !void {
+    // Example showing the conversion steps
+
+    // Step 1: SQLite gives us this type (simplified)
+    const sqlite_ptr: ?*const u8 = "hello";
+
+    // Step 2: Check if it's not null
+    if (sqlite_ptr != null) {
+        // Step 3: Cast to sentinel-terminated pointer
+        // @as(TargetType, value) - explicit type conversion
+        // [*:0]const u8 - pointer to many u8s, terminated by 0
+        const sentinel_ptr = @as([*:0]const u8, @ptrCast(sqlite_ptr));
+
+        // Step 4: Convert sentinel pointer to slice
+        // std.mem.span finds the 0 terminator and creates a slice
+        const slice: []const u8 = std.mem.span(sentinel_ptr);
+
+        std.debug.print("Slice: {s}, Length: {}\n", .{slice, slice.len});
+    }
+
+    // The one-liner version from our code:
+    const text = if (sqlite_ptr != null)
+        std.mem.span(@as([*:0]const u8, @ptrCast(sqlite_ptr)))
+    else
+        "";
+}

src/main.zig

@@ -3,6 +3,61 @@ const c = @cImport({
     @cInclude("sqlite3.h");
 });
 
+// Helper function to clean highlight text by removing page/margin reference numbers
+// Removes patterns like "10After" -> "After" and " 5good" -> " good"
+fn cleanHighlightText(allocator: std.mem.Allocator, text: []const u8) ![]const u8 {
+    var result = std.ArrayList(u8).initCapacity(allocator, 0) catch unreachable;
+    defer result.deinit(allocator);
+    const writer = result.writer(allocator);
+
+    var i: usize = 0;
+    while (i < text.len) {
+        // Check if we're at start of text or after whitespace/newline
+        const at_boundary = (i == 0) or (text[i - 1] == ' ') or (text[i - 1] == '\n') or (text[i - 1] == '\t');
+
+        if (at_boundary and std.ascii.isDigit(text[i])) {
+            // Count consecutive digits
+            var digit_end = i;
+            while (digit_end < text.len and std.ascii.isDigit(text[digit_end])) {
+                digit_end += 1;
+            }
+
+            // If digits are followed by a letter (like "10After"), skip the digits
+            if (digit_end < text.len and std.ascii.isAlphabetic(text[digit_end])) {
+                i = digit_end; // Skip the digits
+                continue;
+            }
+        }
+
+        try writer.writeByte(text[i]);
+        i += 1;
+    }
+
+    return try allocator.dupe(u8, result.items);
+}
+
+// Helper function to escape JSON strings
+fn escapeJson(allocator: std.mem.Allocator, s: []const u8) ![]const u8 {
+    var result = std.ArrayList(u8).initCapacity(allocator, 0) catch unreachable;
+    defer result.deinit(allocator);
+    const writer = result.writer(allocator);
+
+    try writer.writeAll("\"");
+    for (s) |ch| {
+        switch (ch) {
+            '"' => try writer.writeAll("\\\""),
+            '\\' => try writer.writeAll("\\\\"),
+            '\n' => try writer.writeAll("\\n"),
+            '\r' => try writer.writeAll("\\r"),
+            '\t' => try writer.writeAll("\\t"),
+            else => try writer.writeByte(ch),
+        }
+    }
+    try writer.writeAll("\"");
+
+    return try allocator.dupe(u8, result.items);
+}
+
 const Highlight = struct {
     text: []const u8, // Owned copy
     date: []const u8, // Owned copy
@@ -11,9 +66,32 @@ const Highlight = struct {
     book_hash: [64]u8, // Stack allocated - no cleanup needed
     allocator: std.mem.Allocator,
 
+    // JSON-serializable version (no allocator field)
+    const JsonHighlight = struct {
+        text: []const u8,
+        date: []const u8,
+        book: []const u8,
+        text_hash: []const u8, // Slice view of the hash array
+        book_hash: []const u8, // Slice view of the hash array
+    };
+
+    fn toJson(self: *const Highlight) JsonHighlight {
+        return JsonHighlight{
+            .text = self.text,
+            .date = self.date,
+            .book = self.book,
+            .text_hash = &self.text_hash,
+            .book_hash = &self.book_hash,
+        };
+    }
+
     fn init(allocator: std.mem.Allocator, text: []const u8, date: []const u8, book: []const u8, content_id_raw: []const u8) !Highlight {
+        // Clean the text to remove page reference numbers
+        const cleaned_text = try cleanHighlightText(allocator, text);
+        defer allocator.free(cleaned_text);
+
         var highlight = Highlight{
-            .text = try allocator.dupe(u8, text), // Copy from SQLite buffer
+            .text = try allocator.dupe(u8, cleaned_text), // Copy cleaned text
             .date = try allocator.dupe(u8, date), // Copy from SQLite buffer
             .book = try allocator.dupe(u8, book), // Copy from SQLite buffer
             .text_hash = undefined,
@@ -21,8 +99,8 @@ const Highlight = struct {
             .allocator = allocator,
         };
 
-        // Now compute hashes into the struct fields
+        // Now compute hashes into the struct fields (use cleaned text for hash)
-        computeHash(text, &highlight.text_hash);
+        computeHash(cleaned_text, &highlight.text_hash);
         computeHash(content_id_raw, &highlight.book_hash);
 
         return highlight;
@@ -39,7 +117,11 @@ const Highlight = struct {
         std.crypto.hash.sha2.Sha256.hash(data, &hash, .{});
 
         // Convert 32 bytes to 64 hex characters
-        _ = std.fmt.bufPrint(out_hex, "{s}", .{std.fmt.fmtSliceHexLower(&hash)}) catch unreachable;
+        const hex_chars = "0123456789abcdef";
+        for (hash, 0..) |byte, i| {
+            out_hex[i * 2] = hex_chars[byte >> 4];
+            out_hex[i * 2 + 1] = hex_chars[byte & 0x0F];
+        }
     }
 };
 
@@ -53,12 +135,14 @@ pub fn main() !void {
     defer std.process.argsFree(allocator, args);
 
     if (args.len < 2) {
-        std.debug.print("Usage: {s} <path-to-kobo-database.sqlite>\n", .{args[0]});
+        std.debug.print("Usage: {s} <path-to-kobo-database.sqlite> [flask-endpoint-url]\n", .{args[0]});
         std.debug.print("Example: {s} /mnt/kobo/.kobo/KoboReader.sqlite\n", .{args[0]});
+        std.debug.print("Example: {s} /mnt/kobo/.kobo/KoboReader.sqlite https://myserver.com/api/highlights\n", .{args[0]});
         return error.MissingArgument;
     }
 
     const db_path = args[1];
+    const endpoint_url = if (args.len >= 3) args[2] else null;
 
     // Open database
     var db: ?*c.sqlite3 = null;
@@ -77,7 +161,8 @@ pub fn main() !void {
         \\  SUBSTR(
         \\    SUBSTR(ContentID, 40, INSTR(ContentID, '.epub') - 40 + 5),
         \\    INSTR(SUBSTR(ContentID, 40, INSTR(ContentID, '.epub') - 40 + 5), '/') + 1
-        \\  ) AS Book
+        \\  ) AS Book,
+        \\  ContentID
         \\FROM Bookmark
         \\WHERE Text IS NOT NULL
         \\ORDER BY DateCreated DESC;
@@ -90,46 +175,105 @@ pub fn main() !void {
     }
     defer _ = c.sqlite3_finalize(stmt);
 
-    // Set up buffered stdout writer
+    // Create ArrayList to collect highlights
-    const stdout_file = std.fs.File{ .handle = std.posix.STDOUT_FILENO };
+    var highlights = std.ArrayList(Highlight).initCapacity(allocator, 0) catch unreachable;
-    var stdout_buffer: [4096]u8 = undefined;
+    defer {
-    var stdout_writer = stdout_file.writer(&stdout_buffer);
+        for (highlights.items) |*highlight| {
-    const stdout = &stdout_writer.interface;
+            highlight.deinit();
-    var count: usize = 0;
+        }
+        highlights.deinit(allocator);
+    }
 
-    // Execute query and print results
+    // Execute query and collect results
     while (c.sqlite3_step(stmt) == c.SQLITE_ROW) {
-        count += 1;
 
-        // Get the text (highlight)
+        // Get the text (highlight) - convert C pointer to Zig slice
         const text_ptr = c.sqlite3_column_text(stmt, 0);
         const text = if (text_ptr != null)
             std.mem.span(@as([*:0]const u8, @ptrCast(text_ptr)))
         else
             "";
 
-        // Get date
+        // Get date - convert C pointer to Zig slice
         const date_ptr = c.sqlite3_column_text(stmt, 1);
         const date = if (date_ptr != null)
             std.mem.span(@as([*:0]const u8, @ptrCast(date_ptr)))
         else
             "unknown";
 
-        // Get book title
+        // Get book title - convert C pointer to Zig slice
         const book_ptr = c.sqlite3_column_text(stmt, 2);
         const book = if (book_ptr != null)
             std.mem.span(@as([*:0]const u8, @ptrCast(book_ptr)))
         else
             "unknown";
 
-        // Print formatted output
+        // Get raw ContentID for hash - convert C pointer to Zig slice
-        try stdout.print("\n========== Highlight #{d} ==========\n", .{count});
+        const content_id_ptr = c.sqlite3_column_text(stmt, 3);
-        try stdout.print("Date: {s}\n", .{date});
+        const content_id = if (content_id_ptr != null)
-        try stdout.print("Book: {s}\n", .{book});
+            std.mem.span(@as([*:0]const u8, @ptrCast(content_id_ptr)))
-        try stdout.print("\n{s}\n", .{text});
+        else
-        try stdout.print("===================================\n", .{});
+            "";
+
+        // Create Highlight struct (makes owned copies and computes hashes)
+        const highlight = try Highlight.init(allocator, text, date, book, content_id);
+
+        // Add to ArrayList
+        try highlights.append(allocator, highlight);
     }
 
-    try stdout.print("\nTotal highlights extracted: {d}\n", .{count});
+    std.debug.print("Extracted {d} highlights from database\n", .{highlights.items.len});
-    try stdout_writer.interface.flush();
+
+    // Build JSON manually
+    var json_buffer = std.ArrayList(u8).initCapacity(allocator, 0) catch unreachable;
+    defer json_buffer.deinit(allocator);
+    const writer = json_buffer.writer(allocator);
+
+    try writer.writeAll("[");
+    for (highlights.items, 0..) |*highlight, i| {
+        if (i > 0) try writer.writeAll(",");
+        try writer.writeAll("{");
+
+        const escaped_text = try escapeJson(allocator, highlight.text);
+        defer allocator.free(escaped_text);
+        const escaped_book = try escapeJson(allocator, highlight.book);
+        defer allocator.free(escaped_book);
+
+        try writer.print("\"text\":{s},", .{escaped_text});
+        try writer.print("\"date\":\"{s}\",", .{highlight.date});
+        try writer.print("\"book\":{s},", .{escaped_book});
+        try writer.print("\"text_hash\":\"{s}\",", .{highlight.text_hash});
+        try writer.print("\"book_hash\":\"{s}\"", .{highlight.book_hash});
+        try writer.writeAll("}");
+    }
+    try writer.writeAll("]");
+
+    // If endpoint URL provided, POST the JSON; otherwise just print it
+    if (endpoint_url) |url| {
+        std.debug.print("Sending {d} highlights ({d} bytes) to {s}...\n", .{highlights.items.len, json_buffer.items.len, url});
+
+        var client: std.http.Client = .{ .allocator = allocator };
+        defer client.deinit();
+
+        const uri = try std.Uri.parse(url);
+
+        const result = try client.fetch(.{
+            .location = .{ .uri = uri },
+            .method = .POST,
+            .payload = json_buffer.items,
+            .headers = .{
+                .content_type = .{ .override = "application/json" },
+            },
+        });
+
+        const status_code = @intFromEnum(result.status);
+        if (status_code >= 200 and status_code < 300) {
+            std.debug.print("✓ Successfully sent highlights to server (HTTP {})\n", .{status_code});
+        } else {
+            std.debug.print("✗ Server returned error: HTTP {}\n", .{status_code});
+            return error.HttpRequestFailed;
+        }
+    } else {
+        std.debug.print("No endpoint provided. JSON payload ({d} bytes):\n{s}\n", .{json_buffer.items.len, json_buffer.items});
+    }
 }

sync_highlights.sh

@@ -0,0 +1,26 @@
+#!/bin/sh
+# Sync Kobo highlights to home server
+# Place at: /usr/local/bin/sync_highlights.sh
+
+DB="/mnt/onboard/.kobo/KoboReader.sqlite"
+ENDPOINT="http://192.168.88.69:42070/api/highlights"
+LOGFILE="/mnt/onboard/.adds/kb_exfiltrator.log"
+
+# Create log directory if needed
+mkdir -p /mnt/onboard/.adds
+
+# Log with timestamp
+echo "=== $(date) ===" >> "$LOGFILE"
+
+# Run exfiltrator and capture output
+if /usr/local/bin/kb_exfiltrator "$DB" "$ENDPOINT" >> "$LOGFILE" 2>&1; then
+    # Success
+    echo "Success: Highlights synced!" >> "$LOGFILE"
+    # Optional: Show a notification (requires FBInk or similar)
+    # fbink -g file=success.png
+    exit 0
+else
+    # Failure
+    echo "Error: Sync failed. Check log at .adds/kb_exfiltrator.log" >> "$LOGFILE"
+    exit 1
+fi

sync_highlights_with_fbink.sh

@@ -0,0 +1,74 @@
+#!/bin/sh
+# Sync Kobo highlights to home server with FBInk notifications
+# Place at: /usr/local/bin/sync_highlights.sh
+
+DB="/mnt/onboard/.kobo/KoboReader.sqlite"
+ENDPOINT="http://192.168.88.69:42070/api/highlights"
+LOGFILE="/mnt/onboard/.adds/kb_exfiltrator.log"
+
+# Create log directory if needed
+mkdir -p /mnt/onboard/.adds
+
+# Check if FBInk is available
+FBINK=""
+if command -v fbink >/dev/null 2>&1; then
+    FBINK="fbink"
+elif [ -x /usr/local/bin/fbink ]; then
+    FBINK="/usr/local/bin/fbink"
+elif [ -x /usr/bin/fbink ]; then
+    FBINK="/usr/bin/fbink"
+fi
+
+# Show "Syncing..." message
+if [ -n "$FBINK" ]; then
+    $FBINK -q -pm -y 10 "Syncing highlights..."
+fi
+
+# Log with timestamp
+echo "=== $(date) ===" >> "$LOGFILE"
+
+# Run exfiltrator and capture output
+OUTPUT=$(/usr/local/bin/kb_exfiltrator "$DB" "$ENDPOINT" 2>&1)
+EXIT_CODE=$?
+
+# Log the output
+echo "$OUTPUT" >> "$LOGFILE"
+
+# Parse the output for stats (if successful)
+if [ $EXIT_CODE -eq 0 ]; then
+    # Try to extract highlight count from output
+    COUNT=$(echo "$OUTPUT" | grep -o 'Extracted [0-9]* highlights' | grep -o '[0-9]*')
+
+    if [ -n "$FBINK" ]; then
+        if [ -n "$COUNT" ]; then
+            $FBINK -q -pm -y 10 "✓ Synced $COUNT highlights!"
+        else
+            $FBINK -q -pm -y 10 "✓ Highlights synced!"
+        fi
+    fi
+
+    echo "Success: Highlights synced!" >> "$LOGFILE"
+
+    # Clear notification after 2 seconds
+    sleep 2
+    if [ -n "$FBINK" ]; then
+        $FBINK -q -s
+    fi
+
+    exit 0
+else
+    # Failure
+    if [ -n "$FBINK" ]; then
+        $FBINK -q -pm -y 10 "✗ Sync failed! Check log"
+    fi
+
+    echo "Error: Sync failed" >> "$LOGFILE"
+
+    # Clear notification after 3 seconds
+    sleep 3
+    if [ -n "$FBINK" ]; then
+        $FBINK -q -s
+    fi
+
+    exit 1
+fi

why_conversion.zig

@@ -0,0 +1,25 @@
+const std = @import("std");
+
+pub fn main() !void {
+    // Why we need the conversion:
+
+    // SQLite stores strings as C-style null-terminated strings
+    const c_string: [*:0]const u8 = "Hello, World!";
+
+    // But Zig functions expect slices (pointer + length)
+    // For example, this function wants a slice:
+    printText("Direct string");  // This works - string literals are slices
+
+    // Can't do this - type mismatch:
+    // printText(c_string);  // ERROR: expected []const u8, found [*:0]const u8
+
+    // Must convert pointer to slice:
+    const slice = std.mem.span(c_string);
+    printText(slice);  // Now it works!
+
+    // Our Highlight.init() is the same - it expects slices, not pointers
+}
+
+fn printText(text: []const u8) void {
+    std.debug.print("Text: {s} (length: {})\n", .{text, text.len});
+}