Publish opted-in marimo notebooks at /notebooks
Notebooks from the marimo server can now appear in the garden. A notebook
opts in with an HTML comment in one of its markdown cells — invisible when
rendered, greppable in the .py source:
<!-- garden:publish
title: Dream of Spotification
order: 30
-->
Default-deny on purpose: garden.c0smere.net is public and the export bakes
each notebook's executed output into the page, not just its code.
export-notebooks.py also carries a NEVER_PUBLISH list (genome_*, coursework)
so a marker pasted into one of those refuses loudly instead of publishing.
Pipeline: export-notebooks.py runs each marked notebook in a one-shot
marimo container (same image/env/GPU as the live server, so it hits the real
databases) into notebooks-export/ + index.json; copy-notebooks.mjs stages
those to public/nb/; Astro reads index.json to build the pages and the
sidebar section.
Isolation choices worth keeping:
- Own timer and own lock, separate from the 5-min garden build — executing a
notebook takes minutes and must never hold up a build tick.
- Cached on notebook content; both index.json and .cache.json go through
write_if_changed, since auto-build.sh hashes mtimes under notebooks-export/
and an unconditional rewrite would force a full rebuild every 30 minutes.
- Per-notebook timeout; a failure keeps the previous export and continues.
- Runs against a throwaway copy of the notebook dir, so notebooks that write
scratch files don't dirty the notebooks repo.
- Notebooks are NOT injected into the garden collection — they aren't vault
notes, and doing so would move noteCount and the sitemap.
build.format:'file' makes the listing a file (notebooks.html) beside a
directory of detail pages. Verified against the running nginx: /notebooks and
/notebooks/<slug> resolve through the generic try_files but /notebooks/ does
not, so nginx.conf gets an explicit location for the trailing-slash form.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -2,6 +2,12 @@ node_modules/
|
||||
dist/
|
||||
.astro/
|
||||
public/assets/
|
||||
# notebook exports are build artifacts: tracking them would make every
|
||||
# export dirty the tree, which flips the auto-build signature and breaks
|
||||
# the git pull at the top of every build tick
|
||||
notebooks-export/
|
||||
public/nb/
|
||||
.build_state
|
||||
.build.lock
|
||||
.notebooks.lock
|
||||
.rebuild.log
|
||||
|
||||
@@ -37,6 +37,68 @@ Defaults to the kotov vault copy. On cyrion use `deploy/build.sh`
|
||||
`docker compose -f deploy/docker-compose.yml up -d` serves `dist/` on
|
||||
port 18100, proxied as `garden.c0smere.net` (NPM public + Caddy internal).
|
||||
|
||||
## marimo notebooks (`/notebooks`)
|
||||
|
||||
Notebooks from the marimo server (`/home/nox/docker/marimo/notebooks/`) can
|
||||
be published into the garden. **A notebook publishes only if it opts in**, by
|
||||
carrying this HTML comment in one of its markdown cells — invisible in the
|
||||
rendered page, greppable in the `.py` source:
|
||||
|
||||
```
|
||||
<!-- garden:publish
|
||||
title: Dream of Spotification
|
||||
description: A decade of Spotify history, worked two ways.
|
||||
order: 30
|
||||
-->
|
||||
```
|
||||
|
||||
Only the `garden:publish` line is required. Optional keys: `title`,
|
||||
`description`, `order` (sort weight, default 100), `slug` (URL override,
|
||||
default derived from the filename), `timeout` (export seconds, default 600).
|
||||
Delete the marker to unpublish — the next export run removes the page.
|
||||
|
||||
This is deliberately default-deny: garden.c0smere.net is public and the
|
||||
export bakes each notebook's **executed output** into the page, not just its
|
||||
code. `deploy/export-notebooks.py` also carries a `NEVER_PUBLISH` list
|
||||
(`genome_*`, coursework) as a second latch, so a marker pasted into one of
|
||||
those refuses loudly instead of publishing.
|
||||
|
||||
Pipeline:
|
||||
|
||||
1. `deploy/export-notebooks.py` (own timer, `garden-notebooks-export.timer`,
|
||||
every 30 min; own lock) scans for markers and runs each opted-in notebook
|
||||
in a one-shot `marimo:latest` container — same image and `.env` as the
|
||||
live server, so it hits the real databases. Output lands in
|
||||
`notebooks-export/<slug>.html` plus an `index.json`.
|
||||
- Cached on notebook content: a run after no edits does no work.
|
||||
- Per-notebook timeout, and a failure keeps the previous export and
|
||||
continues. A broken notebook can't take the garden down.
|
||||
- It runs against a throwaway *copy* of the notebook dir, so notebooks
|
||||
that write scratch files don't dirty the notebooks git repo.
|
||||
2. `scripts/copy-notebooks.mjs` stages those into `public/nb/<slug>.html`.
|
||||
3. Astro reads `index.json` (`src/lib/notebooks.mjs`) to build `/notebooks`
|
||||
and `/notebooks/<slug>`, which wrap the raw export in an iframe with the
|
||||
garden chrome, plus the sidebar section.
|
||||
|
||||
The raw export is at `/nb/<slug>.html`, deliberately *not* under
|
||||
`/notebooks` — with `build.format: 'file'` the page and the raw file would
|
||||
collide on the same URL.
|
||||
|
||||
URL shapes here are worth knowing, since `build.format: 'file'` makes the
|
||||
listing a *file* (`dist/notebooks.html`) that sits beside a *directory* of
|
||||
detail pages (`dist/notebooks/<slug>.html`). Verified against the running
|
||||
nginx: `/notebooks` and `/notebooks/<slug>` resolve through the generic
|
||||
`try_files`, but `/notebooks/` does not — a bare directory has no
|
||||
`index.html`. `deploy/nginx.conf` carries an explicit `location = /notebooks/`
|
||||
so the trailing-slash form works like every other folder URL on the site.
|
||||
|
||||
Both `notebooks-export/` and `public/nb/` are gitignored: tracking them would
|
||||
make every export dirty the tree, which flips `auto-build.sh`'s change
|
||||
signature and breaks the `git pull` at the top of each build tick. The
|
||||
signature hashes `notebooks-export/` separately instead.
|
||||
|
||||
Force a full re-export with `deploy/export-notebooks.py --force`.
|
||||
|
||||
## Layout extras
|
||||
|
||||
Every page carries a left sidebar tree of the whole garden (folders
|
||||
|
||||
@@ -21,12 +21,15 @@ flock 9
|
||||
# a dirty tree can make pull fail; still rebuild whatever is checked out
|
||||
git pull -q || echo "git pull failed (dirty tree?) — building local state"
|
||||
|
||||
# signature covers committed HEAD, any uncommitted repo edits, and vault mtimes
|
||||
# signature covers committed HEAD, any uncommitted repo edits, vault mtimes,
|
||||
# and the notebook exports (gitignored, so git status/diff can't see them —
|
||||
# they land here on their own timer via deploy/export-notebooks.py)
|
||||
sig=$({
|
||||
git rev-parse HEAD
|
||||
git status --porcelain=v1
|
||||
git diff
|
||||
find "$VAULT" -name .obsidian -prune -o -type f -printf '%T@ %p\n' | sort
|
||||
find "$REPO/notebooks-export" -type f -printf '%T@ %p\n' 2>/dev/null | sort
|
||||
} | sha256sum | cut -d' ' -f1)
|
||||
|
||||
if [ "$FORCE" -eq 0 ] && [ -f "$STATE" ] && [ "$(cat "$STATE")" = "$sig" ]; then
|
||||
|
||||
Executable
+320
@@ -0,0 +1,320 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Export the marimo notebooks that opt in to publication into the garden.
|
||||
|
||||
A notebook opts in by putting an HTML comment block in one of its markdown
|
||||
cells (invisible in the rendered page, greppable in the .py source):
|
||||
|
||||
<!-- garden:publish
|
||||
title: Dream of Spotification
|
||||
description: A decade of my own Spotify history, worked two ways.
|
||||
order: 20
|
||||
-->
|
||||
|
||||
Only `garden:publish` is required; every other key is optional. Recognized
|
||||
keys: title, description, order (sort weight, default 100), slug (URL
|
||||
override, default derived from the filename), timeout (seconds, default 600).
|
||||
|
||||
Nothing publishes without the marker. garden.c0smere.net is public and a
|
||||
static export bakes the notebook's *executed output* into the page, not just
|
||||
its code — so this is deliberately default-deny, with NEVER_PUBLISH below as
|
||||
a second latch under the files where an accidental marker would be worst.
|
||||
|
||||
Each opted-in notebook is executed against the real databases (same image and
|
||||
env as the live marimo server) and snapshotted to notebooks-export/<slug>.html
|
||||
next to an index.json the Astro build reads. Exports are cached on notebook
|
||||
content, so a run that follows no notebook edits does no work at all.
|
||||
|
||||
Failure is per-notebook: a broken or hung notebook is logged, its previous
|
||||
export is kept, and the run continues — it can never take the garden down.
|
||||
"""
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from fnmatch import fnmatch
|
||||
from pathlib import Path
|
||||
|
||||
REPO = Path(__file__).resolve().parent.parent
|
||||
NOTEBOOK_DIR = Path(os.environ.get("MARIMO_NOTEBOOKS", "/home/nox/docker/marimo/notebooks"))
|
||||
MARIMO_ENV = Path(os.environ.get("MARIMO_ENV", "/home/nox/docker/marimo/.env"))
|
||||
# The live server's HOME, holding .config/marimo/marimo.toml — the export
|
||||
# copies that config in so pages come out in the same theme Wes edits in.
|
||||
MARIMO_HOME = Path(os.environ.get("MARIMO_HOME", "/home/nox/docker/marimo/config"))
|
||||
MARIMO_IMAGE = os.environ.get("MARIMO_IMAGE", "marimo:latest")
|
||||
DOCKER_NETWORK = os.environ.get("MARIMO_NETWORK", "services_net")
|
||||
OUT_DIR = REPO / "notebooks-export"
|
||||
CACHE_FILE = OUT_DIR / ".cache.json"
|
||||
|
||||
# Bumped when the export command or output shape changes, to invalidate the
|
||||
# cache without anyone having to touch the notebooks.
|
||||
EXPORTER_VERSION = "2"
|
||||
|
||||
# Generous by default: these notebooks fit models against real data, and an
|
||||
# export only runs when the notebook actually changed, so a slow one costs
|
||||
# nothing on a steady-state tick.
|
||||
DEFAULT_TIMEOUT = 1800
|
||||
|
||||
# Second latch under the marker. These never publish even if one of them
|
||||
# picks up a marker by copy-paste: genome_* renders Wes's real genotypes
|
||||
# (claude_reader is revoked on that DB precisely so this data stays put), and
|
||||
# the coursework file is graded UoPeople work. db.py is a helper, not a
|
||||
# notebook. To genuinely publish one of these, remove it from this list —
|
||||
# the point is that it takes a deliberate edit here, not just a marker.
|
||||
NEVER_PUBLISH = ("genome_*.py", "MTH1211*.py", "db.py")
|
||||
|
||||
MARKER_RE = re.compile(r"<!--\s*garden:publish\b(?P<body>.*?)-->", re.DOTALL | re.IGNORECASE)
|
||||
|
||||
|
||||
def log(msg):
|
||||
print(f"export-notebooks: {msg}", flush=True)
|
||||
|
||||
|
||||
def slugify(text):
|
||||
s = re.sub(r"[^a-z0-9]+", "-", text.lower()).strip("-")
|
||||
return s or "notebook"
|
||||
|
||||
|
||||
def parse_marker(source):
|
||||
"""Return the marker's key/value dict, or None if the notebook opted out."""
|
||||
m = MARKER_RE.search(source)
|
||||
if not m:
|
||||
return None
|
||||
meta = {}
|
||||
for line in m.group("body").splitlines():
|
||||
line = line.strip()
|
||||
if not line or line.startswith("#"):
|
||||
continue
|
||||
key, sep, value = line.partition(":")
|
||||
if not sep:
|
||||
continue
|
||||
meta[key.strip().lower()] = value.strip()
|
||||
return meta
|
||||
|
||||
|
||||
def discover():
|
||||
"""All notebooks in the directory, partitioned into published and skipped."""
|
||||
published, skipped = [], []
|
||||
for path in sorted(NOTEBOOK_DIR.glob("*.py")):
|
||||
blocked = any(fnmatch(path.name, pat) for pat in NEVER_PUBLISH)
|
||||
try:
|
||||
source = path.read_text(encoding="utf-8", errors="replace")
|
||||
except OSError as exc:
|
||||
log(f"WARNING: cannot read {path.name}: {exc}")
|
||||
continue
|
||||
meta = parse_marker(source)
|
||||
if meta is None:
|
||||
skipped.append((path.name, "no marker"))
|
||||
continue
|
||||
if blocked:
|
||||
# Loud: a marker on a denylisted file means someone either
|
||||
# copy-pasted it or genuinely meant to change policy.
|
||||
log(f"REFUSING {path.name}: matches NEVER_PUBLISH despite carrying a publish marker")
|
||||
skipped.append((path.name, "denylisted"))
|
||||
continue
|
||||
slug = meta.get("slug") or slugify(path.stem)
|
||||
try:
|
||||
order = int(meta.get("order", 100))
|
||||
except ValueError:
|
||||
order = 100
|
||||
try:
|
||||
timeout = int(meta.get("timeout", DEFAULT_TIMEOUT))
|
||||
except ValueError:
|
||||
timeout = DEFAULT_TIMEOUT
|
||||
published.append(
|
||||
{
|
||||
"slug": slug,
|
||||
"source": path.name,
|
||||
"title": meta.get("title") or path.stem.replace("_", " ").title(),
|
||||
"description": meta.get("description", ""),
|
||||
"order": order,
|
||||
"timeout": timeout,
|
||||
"digest": hashlib.sha256(
|
||||
source.encode("utf-8") + f"|{EXPORTER_VERSION}|{slug}".encode()
|
||||
).hexdigest(),
|
||||
}
|
||||
)
|
||||
return published, skipped
|
||||
|
||||
|
||||
def staged_workdir(tmp):
|
||||
"""A writable copy of the notebook dir.
|
||||
|
||||
Notebooks import sibling modules (db.py) and some write scratch files
|
||||
while running, so the export needs a real working directory — but the
|
||||
live notebooks dir is a git repo that gets committed nightly, and export
|
||||
runs must not leave anything in it.
|
||||
"""
|
||||
work = Path(tmp) / "notebooks"
|
||||
shutil.copytree(
|
||||
NOTEBOOK_DIR,
|
||||
work,
|
||||
ignore=shutil.ignore_patterns(".git", "__pycache__", "__marimo__", ".gitignore"),
|
||||
)
|
||||
# Throwaway HOME carrying a copy of the live server's marimo.toml, so the
|
||||
# export inherits Wes's theme. A copy rather than a mount of the real
|
||||
# config dir: marimo writes cache into HOME, and that must not land in
|
||||
# the running server's config.
|
||||
home = Path(tmp) / "home"
|
||||
(home / ".config" / "marimo").mkdir(parents=True)
|
||||
toml = MARIMO_HOME / ".config" / "marimo" / "marimo.toml"
|
||||
if toml.is_file():
|
||||
shutil.copyfile(toml, home / ".config" / "marimo" / "marimo.toml")
|
||||
else:
|
||||
log(f"WARNING: no marimo.toml at {toml} — exports will use default theme")
|
||||
return work, home
|
||||
|
||||
|
||||
def export_one(nb, work, home, out_dir):
|
||||
"""Run one notebook to HTML. Returns True on success."""
|
||||
container = f"garden-nb-{nb['slug']}-{os.getpid()}"
|
||||
tmp_out = f"{nb['slug']}.html.part"
|
||||
cmd = [
|
||||
"docker", "run", "--rm", "--name", container,
|
||||
"--user", "1000:1000",
|
||||
"--network", DOCKER_NETWORK,
|
||||
"--env-file", str(MARIMO_ENV),
|
||||
# same GPU wiring as the live marimo service — without it xgboost
|
||||
# silently falls back to CPU and a fit that takes seconds on the
|
||||
# P2000 takes many minutes
|
||||
"--runtime", "nvidia",
|
||||
"-e", "NVIDIA_VISIBLE_DEVICES=all",
|
||||
"-e", "NVIDIA_DRIVER_CAPABILITIES=compute,utility",
|
||||
"-e", "HOME=/nbhome",
|
||||
"-e", "TZ=America/New_York",
|
||||
# notebooks that draw without an explicit backend must not try to
|
||||
# open a display inside the container
|
||||
"-e", "MPLBACKEND=Agg",
|
||||
"-v", f"{work}:/work",
|
||||
"-v", f"{home}:/nbhome",
|
||||
"-v", f"{out_dir}:/out",
|
||||
"-w", "/work",
|
||||
MARIMO_IMAGE,
|
||||
"marimo", "export", "html", nb["source"], "-o", f"/out/{tmp_out}", "-f",
|
||||
]
|
||||
started = time.monotonic()
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
cmd, capture_output=True, text=True, timeout=nb["timeout"]
|
||||
)
|
||||
except subprocess.TimeoutExpired:
|
||||
log(f"FAILED {nb['source']}: timed out after {nb['timeout']}s")
|
||||
subprocess.run(["docker", "rm", "-f", container], capture_output=True)
|
||||
(out_dir / tmp_out).unlink(missing_ok=True)
|
||||
return False
|
||||
if proc.returncode != 0:
|
||||
tail = (proc.stderr or proc.stdout or "").strip().splitlines()[-15:]
|
||||
log(f"FAILED {nb['source']}: exit {proc.returncode}")
|
||||
for line in tail:
|
||||
log(f" | {line}")
|
||||
(out_dir / tmp_out).unlink(missing_ok=True)
|
||||
return False
|
||||
part = out_dir / tmp_out
|
||||
if not part.exists() or part.stat().st_size == 0:
|
||||
log(f"FAILED {nb['source']}: export produced no output")
|
||||
part.unlink(missing_ok=True)
|
||||
return False
|
||||
# only swap in the new export once it is known-good, so a failure always
|
||||
# leaves the previously published page intact
|
||||
part.replace(out_dir / f"{nb['slug']}.html")
|
||||
log(f"exported {nb['source']} -> {nb['slug']}.html ({time.monotonic() - started:.0f}s)")
|
||||
return True
|
||||
|
||||
|
||||
def write_if_changed(path, text):
|
||||
"""Avoid rewriting identical files — auto-build.sh hashes mtimes."""
|
||||
if path.exists() and path.read_text(encoding="utf-8") == text:
|
||||
return False
|
||||
path.write_text(text, encoding="utf-8")
|
||||
return True
|
||||
|
||||
|
||||
def main():
|
||||
force = "--force" in sys.argv
|
||||
OUT_DIR.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
published, skipped = discover()
|
||||
if not NOTEBOOK_DIR.is_dir():
|
||||
log(f"notebook dir {NOTEBOOK_DIR} missing — nothing to do")
|
||||
return 0
|
||||
|
||||
try:
|
||||
cache = json.loads(CACHE_FILE.read_text())
|
||||
except (OSError, ValueError):
|
||||
cache = {}
|
||||
|
||||
live_slugs = {nb["slug"] for nb in published}
|
||||
# drop exports whose notebook lost its marker, was renamed, or was deleted
|
||||
for stale in OUT_DIR.glob("*.html"):
|
||||
if stale.stem not in live_slugs:
|
||||
log(f"unpublishing {stale.name} (no longer marked)")
|
||||
stale.unlink()
|
||||
cache.pop(stale.stem, None)
|
||||
|
||||
entries, failures = [], 0
|
||||
todo = [
|
||||
nb for nb in published
|
||||
if force
|
||||
or cache.get(nb["slug"], {}).get("digest") != nb["digest"]
|
||||
or not (OUT_DIR / f"{nb['slug']}.html").exists()
|
||||
]
|
||||
|
||||
if todo:
|
||||
with tempfile.TemporaryDirectory(prefix="garden-nb-") as tmp:
|
||||
work, home = staged_workdir(tmp)
|
||||
for nb in todo:
|
||||
if export_one(nb, work, home, OUT_DIR):
|
||||
cache[nb["slug"]] = {
|
||||
"digest": nb["digest"],
|
||||
"exported_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
|
||||
}
|
||||
else:
|
||||
failures += 1
|
||||
|
||||
for nb in sorted(published, key=lambda n: (n["order"], n["title"].lower())):
|
||||
if not (OUT_DIR / f"{nb['slug']}.html").exists():
|
||||
# never exported successfully — leave it out of the index rather
|
||||
# than publish a link to a 404
|
||||
continue
|
||||
cached = cache.get(nb["slug"], {})
|
||||
entries.append(
|
||||
{
|
||||
"slug": nb["slug"],
|
||||
"title": nb["title"],
|
||||
"description": nb["description"],
|
||||
"order": nb["order"],
|
||||
"source": nb["source"],
|
||||
"exportedAt": cached.get("exported_at"),
|
||||
# the notebook changed but its re-export failed: the page is
|
||||
# real, just behind the notebook
|
||||
"stale": cached.get("digest") != nb["digest"],
|
||||
}
|
||||
)
|
||||
|
||||
# Both writes go through write_if_changed: auto-build.sh's signature
|
||||
# hashes the mtime of every file under notebooks-export/, dotfiles
|
||||
# included, so rewriting an identical cache would force a full garden
|
||||
# rebuild on every single export tick.
|
||||
write_if_changed(CACHE_FILE, json.dumps(cache, indent=2, sort_keys=True) + "\n")
|
||||
changed = write_if_changed(
|
||||
OUT_DIR / "index.json", json.dumps(entries, indent=2) + "\n"
|
||||
)
|
||||
|
||||
for name, why in skipped:
|
||||
log(f"skipped {name} ({why})")
|
||||
log(
|
||||
f"{len(entries)} published, {len(skipped)} skipped, {failures} failed"
|
||||
f"{', index updated' if changed else ''}"
|
||||
)
|
||||
# a failed notebook must not fail the run — the garden still builds with
|
||||
# whatever exported cleanly. Exit 1 only so systemd surfaces it.
|
||||
return 1 if failures else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,9 @@
|
||||
[Unit]
|
||||
Description=Export marimo notebooks marked for publication into the garden
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
# Own lock, separate from the garden build's .build.lock: executing a
|
||||
# notebook can take minutes, and it must never hold up the 5-min build tick.
|
||||
# -n means a tick that lands while an export is still running just skips.
|
||||
ExecStart=/usr/bin/flock -n /home/nox/docker/garden-astro/.notebooks.lock /home/nox/docker/garden-astro/deploy/export-notebooks.py
|
||||
@@ -0,0 +1,13 @@
|
||||
[Unit]
|
||||
Description=Periodic marimo notebook export check (every 30 min)
|
||||
|
||||
[Timer]
|
||||
# Notebooks change far less often than the vault and each export executes
|
||||
# real code against the databases, so this runs much lazier than the garden
|
||||
# build. The offset keeps it off the :00/:30 build ticks.
|
||||
OnCalendar=*:07/30
|
||||
RandomizedDelaySec=60
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -10,6 +10,15 @@ server {
|
||||
try_files $uri $uri.html $uri/index.html =404;
|
||||
}
|
||||
|
||||
# The notebooks listing builds to notebooks.html *and* has a sibling
|
||||
# notebooks/ directory of detail pages. try_files above resolves
|
||||
# /notebooks and /notebooks/<slug>, but not the trailing-slash form —
|
||||
# a bare directory has no index.html. Map it explicitly so every folder
|
||||
# URL on the site keeps working with or without the slash.
|
||||
location = /notebooks/ {
|
||||
try_files /notebooks.html =404;
|
||||
}
|
||||
|
||||
location /assets/ {
|
||||
expires 7d;
|
||||
add_header Cache-Control "public";
|
||||
|
||||
+1
-1
@@ -5,7 +5,7 @@
|
||||
"private": true,
|
||||
"scripts": {
|
||||
"dev": "astro dev",
|
||||
"build": "node scripts/copy-assets.mjs && astro build",
|
||||
"build": "node scripts/copy-assets.mjs && node scripts/copy-notebooks.mjs && astro build",
|
||||
"preview": "astro preview"
|
||||
},
|
||||
"dependencies": {
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
// Stage the marimo notebook exports produced by deploy/export-notebooks.py
|
||||
// into public/, where Astro will pick them up as static files.
|
||||
//
|
||||
// The raw marimo HTML lands at public/nb/<slug>.html (served /nb/<slug>.html)
|
||||
// and is what the notebook pages iframe. It deliberately does NOT live under
|
||||
// /notebooks/ — those URLs belong to the Astro pages that wrap it in the
|
||||
// garden chrome, and with build.format:'file' the two would collide on a
|
||||
// trailing slash.
|
||||
//
|
||||
// No exports (export script never ran, or nothing is marked) is a normal
|
||||
// state, not an error: the notebooks section simply doesn't render.
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
const REPO = path.join(path.dirname(fileURLToPath(import.meta.url)), '..');
|
||||
const SRC_DIR = path.join(REPO, 'notebooks-export');
|
||||
const OUT_DIR = path.join(REPO, 'public', 'nb');
|
||||
|
||||
fs.rmSync(OUT_DIR, { recursive: true, force: true });
|
||||
|
||||
let index = [];
|
||||
try {
|
||||
index = JSON.parse(fs.readFileSync(path.join(SRC_DIR, 'index.json'), 'utf8'));
|
||||
} catch {
|
||||
console.log('copy-notebooks: no notebooks-export/index.json — nothing to stage');
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
fs.mkdirSync(OUT_DIR, { recursive: true });
|
||||
|
||||
let copied = 0;
|
||||
for (const nb of index) {
|
||||
const src = path.join(SRC_DIR, `${nb.slug}.html`);
|
||||
if (!fs.existsSync(src)) {
|
||||
// export-notebooks.py only indexes notebooks it exported, so this means
|
||||
// the two got out of sync — warn rather than ship a link to a 404
|
||||
console.warn(`copy-notebooks: MISSING export for indexed notebook ${nb.slug}`);
|
||||
continue;
|
||||
}
|
||||
fs.copyFileSync(src, path.join(OUT_DIR, `${nb.slug}.html`));
|
||||
copied++;
|
||||
}
|
||||
|
||||
console.log(`copy-notebooks: ${copied}/${index.length} staged -> ${OUT_DIR}`);
|
||||
+63
-2
@@ -5,6 +5,7 @@ import { buildNavTree } from '../lib/nav.mjs';
|
||||
import { COMMIT, BUILT_AT } from '../lib/build-info.mjs';
|
||||
import NavTree from '../components/NavTree.astro';
|
||||
import Toc from '../components/Toc.astro';
|
||||
import { loadNotebooks, notebookUrl } from '../lib/notebooks.mjs';
|
||||
|
||||
interface Props {
|
||||
title: string;
|
||||
@@ -13,12 +14,27 @@ interface Props {
|
||||
// which /og/<slug>.png card this page advertises; synthetic listing
|
||||
// pages fall back to the site card
|
||||
ogSlug?: string;
|
||||
// widen the content column — for embedded notebooks, which are far too
|
||||
// wide for the 44rem prose measure
|
||||
wide?: boolean;
|
||||
}
|
||||
const { title, description, headings = [], ogSlug = 'index' } = Astro.props;
|
||||
const {
|
||||
title,
|
||||
description,
|
||||
headings = [],
|
||||
ogSlug = 'index',
|
||||
wide = false,
|
||||
} = Astro.props;
|
||||
const isHome = Astro.url.pathname === '/';
|
||||
const entries = await getCollection('garden');
|
||||
const navTree = buildNavTree(entries);
|
||||
const noteCount = entries.length;
|
||||
// Notebooks are a separate section, NOT injected into the garden collection:
|
||||
// they aren't vault notes, so they must not move noteCount or the sitemap.
|
||||
const notebooks = loadNotebooks();
|
||||
const navPath = decodeURIComponent(Astro.url.pathname)
|
||||
.replace(/index\.html$/, '')
|
||||
.replace(/\.html$/, '');
|
||||
const showToc = headings.filter((h) => h.depth <= 3).length >= 2;
|
||||
const ogImage = new URL(`/og/${ogSlug}.png`, Astro.site);
|
||||
const pageUrl = new URL(Astro.url.pathname, Astro.site);
|
||||
@@ -60,7 +76,7 @@ const pageUrl = new URL(Astro.url.pathname, Astro.site);
|
||||
src="https://plausible.io/js/script.js"></script>
|
||||
</head>
|
||||
<body class={isHome ? 'home' : undefined}>
|
||||
<div class="layout">
|
||||
<div class:list={['layout', wide && 'layout-wide']}>
|
||||
<div class="content-col">
|
||||
<header>
|
||||
<div class="site-head">
|
||||
@@ -150,6 +166,29 @@ const pageUrl = new URL(Astro.url.pathname, Astro.site);
|
||||
<aside class="sidebar">
|
||||
<p class="sidebar-title"><a href="/">the garden</a></p>
|
||||
<NavTree nodes={navTree} currentPath={Astro.url.pathname} />
|
||||
{
|
||||
notebooks.length > 0 && (
|
||||
<div class="sidebar-section">
|
||||
<p class="sidebar-title">
|
||||
<a href="/notebooks">notebooks</a>
|
||||
</p>
|
||||
<ul class="nav-tree">
|
||||
{notebooks.map((nb) => (
|
||||
<li>
|
||||
<a
|
||||
href={notebookUrl(nb.slug)}
|
||||
aria-current={
|
||||
navPath === notebookUrl(nb.slug) ? 'page' : undefined
|
||||
}
|
||||
>
|
||||
{nb.title}
|
||||
</a>
|
||||
</li>
|
||||
))}
|
||||
</ul>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
</aside>
|
||||
{
|
||||
showToc && (
|
||||
@@ -692,11 +731,24 @@ const pageUrl = new URL(Astro.url.pathname, Astro.site);
|
||||
padding: 1.5rem 0.5rem 1.5rem 0;
|
||||
}
|
||||
}
|
||||
/* notebook pages: the embedded marimo frame needs far more than the
|
||||
44rem prose measure, so the content column takes whatever is left */
|
||||
@media (min-width: 72rem) {
|
||||
.layout-wide {
|
||||
grid-template-columns: 17rem minmax(0, 1fr);
|
||||
max-width: 96rem;
|
||||
}
|
||||
}
|
||||
@media (min-width: 88rem) {
|
||||
.layout {
|
||||
grid-template-columns: 17rem minmax(0, 44rem) 13rem;
|
||||
max-width: 80rem;
|
||||
}
|
||||
/* wide pages keep the two-column shape — no TOC rail */
|
||||
.layout-wide {
|
||||
grid-template-columns: 17rem minmax(0, 1fr);
|
||||
max-width: 96rem;
|
||||
}
|
||||
.toc-col {
|
||||
display: block;
|
||||
position: sticky;
|
||||
@@ -931,6 +983,15 @@ const pageUrl = new URL(Astro.url.pathname, Astro.site);
|
||||
font-weight: 700;
|
||||
color: var(--fg);
|
||||
}
|
||||
/* second sidebar block (notebooks) — hairline off the garden tree */
|
||||
.sidebar-section {
|
||||
margin-top: 1.75rem;
|
||||
padding-top: 1rem;
|
||||
border-top: 1px solid var(--hair);
|
||||
}
|
||||
.sidebar-section > ul {
|
||||
padding-left: 0;
|
||||
}
|
||||
|
||||
/* ---- table of contents ---- */
|
||||
.toc-title {
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
// Published marimo notebooks, as recorded by deploy/export-notebooks.py.
|
||||
//
|
||||
// The export script is the only thing that reads the publish markers in the
|
||||
// notebooks themselves; it hands the build this index.json. That keeps the
|
||||
// Astro build free of any dependency on the notebook directory — it isn't
|
||||
// even mounted into the build container — and means a malformed marker can
|
||||
// only ever break an export, never the garden.
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
const INDEX = path.join(
|
||||
path.dirname(fileURLToPath(import.meta.url)),
|
||||
'..',
|
||||
'..',
|
||||
'notebooks-export',
|
||||
'index.json',
|
||||
);
|
||||
|
||||
export function loadNotebooks() {
|
||||
try {
|
||||
const parsed = JSON.parse(fs.readFileSync(INDEX, 'utf8'));
|
||||
return Array.isArray(parsed) ? parsed : [];
|
||||
} catch {
|
||||
return []; // never exported: the section just doesn't render
|
||||
}
|
||||
}
|
||||
|
||||
export const notebookUrl = (slug) => `/notebooks/${slug}`;
|
||||
export const notebookRawUrl = (slug) => `/nb/${slug}.html`;
|
||||
@@ -0,0 +1,74 @@
|
||||
---
|
||||
import Base from '../../layouts/Base.astro';
|
||||
import { loadNotebooks, notebookRawUrl } from '../../lib/notebooks.mjs';
|
||||
|
||||
export function getStaticPaths() {
|
||||
return loadNotebooks().map((nb) => ({ params: { slug: nb.slug }, props: { nb } }));
|
||||
}
|
||||
|
||||
const { nb } = Astro.props;
|
||||
const raw = notebookRawUrl(nb.slug);
|
||||
const ranAt = nb.exportedAt
|
||||
? new Date(nb.exportedAt).toLocaleString('en-US', {
|
||||
year: 'numeric',
|
||||
month: 'short',
|
||||
day: 'numeric',
|
||||
hour: 'numeric',
|
||||
minute: '2-digit',
|
||||
timeZone: 'America/New_York',
|
||||
})
|
||||
: null;
|
||||
---
|
||||
|
||||
<Base title={nb.title} description={nb.description} wide>
|
||||
<h1>{nb.title}</h1>
|
||||
{nb.description && <p>{nb.description}</p>}
|
||||
|
||||
<p class="nb-bar">
|
||||
<a href={raw} target="_blank" rel="noopener">open standalone ↗</a>
|
||||
{ranAt && <span class="nb-when">· last run {ranAt} ET</span>}
|
||||
{nb.stale && (
|
||||
<span class="nb-stale">
|
||||
· the notebook has changed since this run
|
||||
</span>
|
||||
)}
|
||||
</p>
|
||||
|
||||
{/* marimo pages are tall and set their own scrolling, so the frame gets a
|
||||
fixed viewport-relative height rather than trying to autosize */}
|
||||
<iframe
|
||||
class="nb-frame"
|
||||
src={raw}
|
||||
title={`${nb.title} — marimo notebook`}
|
||||
loading="lazy"></iframe>
|
||||
|
||||
<style>
|
||||
.nb-bar {
|
||||
margin: 0.75rem 0 1rem;
|
||||
font-size: 0.8rem;
|
||||
color: var(--muted);
|
||||
}
|
||||
.nb-bar a {
|
||||
font-size: 0.8rem;
|
||||
}
|
||||
.nb-when,
|
||||
.nb-stale {
|
||||
margin-left: 0.25rem;
|
||||
}
|
||||
.nb-stale {
|
||||
color: #d97706;
|
||||
}
|
||||
.nb-frame {
|
||||
display: block;
|
||||
width: 100%;
|
||||
height: min(88vh, 1200px);
|
||||
min-height: 32rem;
|
||||
border: 1px solid var(--hair);
|
||||
border-radius: 4px;
|
||||
/* exports inherit the marimo server's dark theme, so the frame sits
|
||||
flush with the page instead of flashing a white slab while it loads */
|
||||
background: var(--bg);
|
||||
color-scheme: dark;
|
||||
}
|
||||
</style>
|
||||
</Base>
|
||||
@@ -0,0 +1,71 @@
|
||||
---
|
||||
import Base from '../../layouts/Base.astro';
|
||||
import { loadNotebooks, notebookUrl } from '../../lib/notebooks.mjs';
|
||||
|
||||
const notebooks = loadNotebooks();
|
||||
const fmt = (iso) =>
|
||||
iso
|
||||
? new Date(iso).toLocaleDateString('en-US', {
|
||||
year: 'numeric',
|
||||
month: 'short',
|
||||
day: 'numeric',
|
||||
timeZone: 'America/New_York',
|
||||
})
|
||||
: null;
|
||||
---
|
||||
|
||||
<Base
|
||||
title="notebooks"
|
||||
description="Interactive marimo notebooks — data science work from the homelab, executed against live data and snapshotted."
|
||||
>
|
||||
<h1>notebooks</h1>
|
||||
<p>
|
||||
marimo notebooks from the homelab, re-executed against their real data
|
||||
sources on a schedule and snapshotted here. Code is shown alongside its
|
||||
output — these are the working documents, not writeups.
|
||||
</p>
|
||||
|
||||
{
|
||||
notebooks.length === 0 ? (
|
||||
<p class="nb-empty">No notebooks are published right now.</p>
|
||||
) : (
|
||||
<ul class="nb-list">
|
||||
{notebooks.map((nb) => (
|
||||
<li>
|
||||
<a href={notebookUrl(nb.slug)}>{nb.title}</a>
|
||||
{nb.description && <p class="nb-desc">{nb.description}</p>}
|
||||
{fmt(nb.exportedAt) && (
|
||||
<p class="nb-meta">last run {fmt(nb.exportedAt)}</p>
|
||||
)}
|
||||
</li>
|
||||
))}
|
||||
</ul>
|
||||
)
|
||||
}
|
||||
|
||||
<style>
|
||||
.nb-list {
|
||||
list-style: none;
|
||||
margin: 2rem 0 0;
|
||||
padding: 0;
|
||||
}
|
||||
.nb-list li {
|
||||
margin: 0 0 1.6rem;
|
||||
padding-left: 0.9rem;
|
||||
border-left: 2px solid var(--hair);
|
||||
}
|
||||
.nb-list > li > a {
|
||||
font-size: 1.05rem;
|
||||
}
|
||||
.nb-desc {
|
||||
margin: 0.3rem 0 0;
|
||||
color: var(--muted);
|
||||
}
|
||||
.nb-meta,
|
||||
.nb-empty {
|
||||
margin: 0.3rem 0 0;
|
||||
font-size: 0.75rem;
|
||||
color: var(--muted);
|
||||
}
|
||||
</style>
|
||||
</Base>
|
||||
Reference in New Issue
Block a user