Publish opted-in marimo notebooks at /notebooks
Notebooks from the marimo server can now appear in the garden. A notebook
opts in with an HTML comment in one of its markdown cells — invisible when
rendered, greppable in the .py source:
<!-- garden:publish
title: Dream of Spotification
order: 30
-->
Default-deny on purpose: garden.c0smere.net is public and the export bakes
each notebook's executed output into the page, not just its code.
export-notebooks.py also carries a NEVER_PUBLISH list (genome_*, coursework)
so a marker pasted into one of those refuses loudly instead of publishing.
Pipeline: export-notebooks.py runs each marked notebook in a one-shot
marimo container (same image/env/GPU as the live server, so it hits the real
databases) into notebooks-export/ + index.json; copy-notebooks.mjs stages
those to public/nb/; Astro reads index.json to build the pages and the
sidebar section.
Isolation choices worth keeping:
- Own timer and own lock, separate from the 5-min garden build — executing a
notebook takes minutes and must never hold up a build tick.
- Cached on notebook content; both index.json and .cache.json go through
write_if_changed, since auto-build.sh hashes mtimes under notebooks-export/
and an unconditional rewrite would force a full rebuild every 30 minutes.
- Per-notebook timeout; a failure keeps the previous export and continues.
- Runs against a throwaway copy of the notebook dir, so notebooks that write
scratch files don't dirty the notebooks repo.
- Notebooks are NOT injected into the garden collection — they aren't vault
notes, and doing so would move noteCount and the sitemap.
build.format:'file' makes the listing a file (notebooks.html) beside a
directory of detail pages. Verified against the running nginx: /notebooks and
/notebooks/<slug> resolve through the generic try_files but /notebooks/ does
not, so nginx.conf gets an explicit location for the trailing-slash form.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -2,6 +2,12 @@ node_modules/
|
|||||||
dist/
|
dist/
|
||||||
.astro/
|
.astro/
|
||||||
public/assets/
|
public/assets/
|
||||||
|
# notebook exports are build artifacts: tracking them would make every
|
||||||
|
# export dirty the tree, which flips the auto-build signature and breaks
|
||||||
|
# the git pull at the top of every build tick
|
||||||
|
notebooks-export/
|
||||||
|
public/nb/
|
||||||
.build_state
|
.build_state
|
||||||
.build.lock
|
.build.lock
|
||||||
|
.notebooks.lock
|
||||||
.rebuild.log
|
.rebuild.log
|
||||||
|
|||||||
@@ -37,6 +37,68 @@ Defaults to the kotov vault copy. On cyrion use `deploy/build.sh`
|
|||||||
`docker compose -f deploy/docker-compose.yml up -d` serves `dist/` on
|
`docker compose -f deploy/docker-compose.yml up -d` serves `dist/` on
|
||||||
port 18100, proxied as `garden.c0smere.net` (NPM public + Caddy internal).
|
port 18100, proxied as `garden.c0smere.net` (NPM public + Caddy internal).
|
||||||
|
|
||||||
|
## marimo notebooks (`/notebooks`)
|
||||||
|
|
||||||
|
Notebooks from the marimo server (`/home/nox/docker/marimo/notebooks/`) can
|
||||||
|
be published into the garden. **A notebook publishes only if it opts in**, by
|
||||||
|
carrying this HTML comment in one of its markdown cells — invisible in the
|
||||||
|
rendered page, greppable in the `.py` source:
|
||||||
|
|
||||||
|
```
|
||||||
|
<!-- garden:publish
|
||||||
|
title: Dream of Spotification
|
||||||
|
description: A decade of Spotify history, worked two ways.
|
||||||
|
order: 30
|
||||||
|
-->
|
||||||
|
```
|
||||||
|
|
||||||
|
Only the `garden:publish` line is required. Optional keys: `title`,
|
||||||
|
`description`, `order` (sort weight, default 100), `slug` (URL override,
|
||||||
|
default derived from the filename), `timeout` (export seconds, default 600).
|
||||||
|
Delete the marker to unpublish — the next export run removes the page.
|
||||||
|
|
||||||
|
This is deliberately default-deny: garden.c0smere.net is public and the
|
||||||
|
export bakes each notebook's **executed output** into the page, not just its
|
||||||
|
code. `deploy/export-notebooks.py` also carries a `NEVER_PUBLISH` list
|
||||||
|
(`genome_*`, coursework) as a second latch, so a marker pasted into one of
|
||||||
|
those refuses loudly instead of publishing.
|
||||||
|
|
||||||
|
Pipeline:
|
||||||
|
|
||||||
|
1. `deploy/export-notebooks.py` (own timer, `garden-notebooks-export.timer`,
|
||||||
|
every 30 min; own lock) scans for markers and runs each opted-in notebook
|
||||||
|
in a one-shot `marimo:latest` container — same image and `.env` as the
|
||||||
|
live server, so it hits the real databases. Output lands in
|
||||||
|
`notebooks-export/<slug>.html` plus an `index.json`.
|
||||||
|
- Cached on notebook content: a run after no edits does no work.
|
||||||
|
- Per-notebook timeout, and a failure keeps the previous export and
|
||||||
|
continues. A broken notebook can't take the garden down.
|
||||||
|
- It runs against a throwaway *copy* of the notebook dir, so notebooks
|
||||||
|
that write scratch files don't dirty the notebooks git repo.
|
||||||
|
2. `scripts/copy-notebooks.mjs` stages those into `public/nb/<slug>.html`.
|
||||||
|
3. Astro reads `index.json` (`src/lib/notebooks.mjs`) to build `/notebooks`
|
||||||
|
and `/notebooks/<slug>`, which wrap the raw export in an iframe with the
|
||||||
|
garden chrome, plus the sidebar section.
|
||||||
|
|
||||||
|
The raw export is at `/nb/<slug>.html`, deliberately *not* under
|
||||||
|
`/notebooks` — with `build.format: 'file'` the page and the raw file would
|
||||||
|
collide on the same URL.
|
||||||
|
|
||||||
|
URL shapes here are worth knowing, since `build.format: 'file'` makes the
|
||||||
|
listing a *file* (`dist/notebooks.html`) that sits beside a *directory* of
|
||||||
|
detail pages (`dist/notebooks/<slug>.html`). Verified against the running
|
||||||
|
nginx: `/notebooks` and `/notebooks/<slug>` resolve through the generic
|
||||||
|
`try_files`, but `/notebooks/` does not — a bare directory has no
|
||||||
|
`index.html`. `deploy/nginx.conf` carries an explicit `location = /notebooks/`
|
||||||
|
so the trailing-slash form works like every other folder URL on the site.
|
||||||
|
|
||||||
|
Both `notebooks-export/` and `public/nb/` are gitignored: tracking them would
|
||||||
|
make every export dirty the tree, which flips `auto-build.sh`'s change
|
||||||
|
signature and breaks the `git pull` at the top of each build tick. The
|
||||||
|
signature hashes `notebooks-export/` separately instead.
|
||||||
|
|
||||||
|
Force a full re-export with `deploy/export-notebooks.py --force`.
|
||||||
|
|
||||||
## Layout extras
|
## Layout extras
|
||||||
|
|
||||||
Every page carries a left sidebar tree of the whole garden (folders
|
Every page carries a left sidebar tree of the whole garden (folders
|
||||||
|
|||||||
@@ -21,12 +21,15 @@ flock 9
|
|||||||
# a dirty tree can make pull fail; still rebuild whatever is checked out
|
# a dirty tree can make pull fail; still rebuild whatever is checked out
|
||||||
git pull -q || echo "git pull failed (dirty tree?) — building local state"
|
git pull -q || echo "git pull failed (dirty tree?) — building local state"
|
||||||
|
|
||||||
# signature covers committed HEAD, any uncommitted repo edits, and vault mtimes
|
# signature covers committed HEAD, any uncommitted repo edits, vault mtimes,
|
||||||
|
# and the notebook exports (gitignored, so git status/diff can't see them —
|
||||||
|
# they land here on their own timer via deploy/export-notebooks.py)
|
||||||
sig=$({
|
sig=$({
|
||||||
git rev-parse HEAD
|
git rev-parse HEAD
|
||||||
git status --porcelain=v1
|
git status --porcelain=v1
|
||||||
git diff
|
git diff
|
||||||
find "$VAULT" -name .obsidian -prune -o -type f -printf '%T@ %p\n' | sort
|
find "$VAULT" -name .obsidian -prune -o -type f -printf '%T@ %p\n' | sort
|
||||||
|
find "$REPO/notebooks-export" -type f -printf '%T@ %p\n' 2>/dev/null | sort
|
||||||
} | sha256sum | cut -d' ' -f1)
|
} | sha256sum | cut -d' ' -f1)
|
||||||
|
|
||||||
if [ "$FORCE" -eq 0 ] && [ -f "$STATE" ] && [ "$(cat "$STATE")" = "$sig" ]; then
|
if [ "$FORCE" -eq 0 ] && [ -f "$STATE" ] && [ "$(cat "$STATE")" = "$sig" ]; then
|
||||||
|
|||||||
Executable
+320
@@ -0,0 +1,320 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Export the marimo notebooks that opt in to publication into the garden.
|
||||||
|
|
||||||
|
A notebook opts in by putting an HTML comment block in one of its markdown
|
||||||
|
cells (invisible in the rendered page, greppable in the .py source):
|
||||||
|
|
||||||
|
<!-- garden:publish
|
||||||
|
title: Dream of Spotification
|
||||||
|
description: A decade of my own Spotify history, worked two ways.
|
||||||
|
order: 20
|
||||||
|
-->
|
||||||
|
|
||||||
|
Only `garden:publish` is required; every other key is optional. Recognized
|
||||||
|
keys: title, description, order (sort weight, default 100), slug (URL
|
||||||
|
override, default derived from the filename), timeout (seconds, default 600).
|
||||||
|
|
||||||
|
Nothing publishes without the marker. garden.c0smere.net is public and a
|
||||||
|
static export bakes the notebook's *executed output* into the page, not just
|
||||||
|
its code — so this is deliberately default-deny, with NEVER_PUBLISH below as
|
||||||
|
a second latch under the files where an accidental marker would be worst.
|
||||||
|
|
||||||
|
Each opted-in notebook is executed against the real databases (same image and
|
||||||
|
env as the live marimo server) and snapshotted to notebooks-export/<slug>.html
|
||||||
|
next to an index.json the Astro build reads. Exports are cached on notebook
|
||||||
|
content, so a run that follows no notebook edits does no work at all.
|
||||||
|
|
||||||
|
Failure is per-notebook: a broken or hung notebook is logged, its previous
|
||||||
|
export is kept, and the run continues — it can never take the garden down.
|
||||||
|
"""
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import shutil
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import time
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from fnmatch import fnmatch
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
REPO = Path(__file__).resolve().parent.parent
|
||||||
|
NOTEBOOK_DIR = Path(os.environ.get("MARIMO_NOTEBOOKS", "/home/nox/docker/marimo/notebooks"))
|
||||||
|
MARIMO_ENV = Path(os.environ.get("MARIMO_ENV", "/home/nox/docker/marimo/.env"))
|
||||||
|
# The live server's HOME, holding .config/marimo/marimo.toml — the export
|
||||||
|
# copies that config in so pages come out in the same theme Wes edits in.
|
||||||
|
MARIMO_HOME = Path(os.environ.get("MARIMO_HOME", "/home/nox/docker/marimo/config"))
|
||||||
|
MARIMO_IMAGE = os.environ.get("MARIMO_IMAGE", "marimo:latest")
|
||||||
|
DOCKER_NETWORK = os.environ.get("MARIMO_NETWORK", "services_net")
|
||||||
|
OUT_DIR = REPO / "notebooks-export"
|
||||||
|
CACHE_FILE = OUT_DIR / ".cache.json"
|
||||||
|
|
||||||
|
# Bumped when the export command or output shape changes, to invalidate the
|
||||||
|
# cache without anyone having to touch the notebooks.
|
||||||
|
EXPORTER_VERSION = "2"
|
||||||
|
|
||||||
|
# Generous by default: these notebooks fit models against real data, and an
|
||||||
|
# export only runs when the notebook actually changed, so a slow one costs
|
||||||
|
# nothing on a steady-state tick.
|
||||||
|
DEFAULT_TIMEOUT = 1800
|
||||||
|
|
||||||
|
# Second latch under the marker. These never publish even if one of them
|
||||||
|
# picks up a marker by copy-paste: genome_* renders Wes's real genotypes
|
||||||
|
# (claude_reader is revoked on that DB precisely so this data stays put), and
|
||||||
|
# the coursework file is graded UoPeople work. db.py is a helper, not a
|
||||||
|
# notebook. To genuinely publish one of these, remove it from this list —
|
||||||
|
# the point is that it takes a deliberate edit here, not just a marker.
|
||||||
|
NEVER_PUBLISH = ("genome_*.py", "MTH1211*.py", "db.py")
|
||||||
|
|
||||||
|
MARKER_RE = re.compile(r"<!--\s*garden:publish\b(?P<body>.*?)-->", re.DOTALL | re.IGNORECASE)
|
||||||
|
|
||||||
|
|
||||||
|
def log(msg):
|
||||||
|
print(f"export-notebooks: {msg}", flush=True)
|
||||||
|
|
||||||
|
|
||||||
|
def slugify(text):
|
||||||
|
s = re.sub(r"[^a-z0-9]+", "-", text.lower()).strip("-")
|
||||||
|
return s or "notebook"
|
||||||
|
|
||||||
|
|
||||||
|
def parse_marker(source):
|
||||||
|
"""Return the marker's key/value dict, or None if the notebook opted out."""
|
||||||
|
m = MARKER_RE.search(source)
|
||||||
|
if not m:
|
||||||
|
return None
|
||||||
|
meta = {}
|
||||||
|
for line in m.group("body").splitlines():
|
||||||
|
line = line.strip()
|
||||||
|
if not line or line.startswith("#"):
|
||||||
|
continue
|
||||||
|
key, sep, value = line.partition(":")
|
||||||
|
if not sep:
|
||||||
|
continue
|
||||||
|
meta[key.strip().lower()] = value.strip()
|
||||||
|
return meta
|
||||||
|
|
||||||
|
|
||||||
|
def discover():
|
||||||
|
"""All notebooks in the directory, partitioned into published and skipped."""
|
||||||
|
published, skipped = [], []
|
||||||
|
for path in sorted(NOTEBOOK_DIR.glob("*.py")):
|
||||||
|
blocked = any(fnmatch(path.name, pat) for pat in NEVER_PUBLISH)
|
||||||
|
try:
|
||||||
|
source = path.read_text(encoding="utf-8", errors="replace")
|
||||||
|
except OSError as exc:
|
||||||
|
log(f"WARNING: cannot read {path.name}: {exc}")
|
||||||
|
continue
|
||||||
|
meta = parse_marker(source)
|
||||||
|
if meta is None:
|
||||||
|
skipped.append((path.name, "no marker"))
|
||||||
|
continue
|
||||||
|
if blocked:
|
||||||
|
# Loud: a marker on a denylisted file means someone either
|
||||||
|
# copy-pasted it or genuinely meant to change policy.
|
||||||
|
log(f"REFUSING {path.name}: matches NEVER_PUBLISH despite carrying a publish marker")
|
||||||
|
skipped.append((path.name, "denylisted"))
|
||||||
|
continue
|
||||||
|
slug = meta.get("slug") or slugify(path.stem)
|
||||||
|
try:
|
||||||
|
order = int(meta.get("order", 100))
|
||||||
|
except ValueError:
|
||||||
|
order = 100
|
||||||
|
try:
|
||||||
|
timeout = int(meta.get("timeout", DEFAULT_TIMEOUT))
|
||||||
|
except ValueError:
|
||||||
|
timeout = DEFAULT_TIMEOUT
|
||||||
|
published.append(
|
||||||
|
{
|
||||||
|
"slug": slug,
|
||||||
|
"source": path.name,
|
||||||
|
"title": meta.get("title") or path.stem.replace("_", " ").title(),
|
||||||
|
"description": meta.get("description", ""),
|
||||||
|
"order": order,
|
||||||
|
"timeout": timeout,
|
||||||
|
"digest": hashlib.sha256(
|
||||||
|
source.encode("utf-8") + f"|{EXPORTER_VERSION}|{slug}".encode()
|
||||||
|
).hexdigest(),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return published, skipped
|
||||||
|
|
||||||
|
|
||||||
|
def staged_workdir(tmp):
|
||||||
|
"""A writable copy of the notebook dir.
|
||||||
|
|
||||||
|
Notebooks import sibling modules (db.py) and some write scratch files
|
||||||
|
while running, so the export needs a real working directory — but the
|
||||||
|
live notebooks dir is a git repo that gets committed nightly, and export
|
||||||
|
runs must not leave anything in it.
|
||||||
|
"""
|
||||||
|
work = Path(tmp) / "notebooks"
|
||||||
|
shutil.copytree(
|
||||||
|
NOTEBOOK_DIR,
|
||||||
|
work,
|
||||||
|
ignore=shutil.ignore_patterns(".git", "__pycache__", "__marimo__", ".gitignore"),
|
||||||
|
)
|
||||||
|
# Throwaway HOME carrying a copy of the live server's marimo.toml, so the
|
||||||
|
# export inherits Wes's theme. A copy rather than a mount of the real
|
||||||
|
# config dir: marimo writes cache into HOME, and that must not land in
|
||||||
|
# the running server's config.
|
||||||
|
home = Path(tmp) / "home"
|
||||||
|
(home / ".config" / "marimo").mkdir(parents=True)
|
||||||
|
toml = MARIMO_HOME / ".config" / "marimo" / "marimo.toml"
|
||||||
|
if toml.is_file():
|
||||||
|
shutil.copyfile(toml, home / ".config" / "marimo" / "marimo.toml")
|
||||||
|
else:
|
||||||
|
log(f"WARNING: no marimo.toml at {toml} — exports will use default theme")
|
||||||
|
return work, home
|
||||||
|
|
||||||
|
|
||||||
|
def export_one(nb, work, home, out_dir):
|
||||||
|
"""Run one notebook to HTML. Returns True on success."""
|
||||||
|
container = f"garden-nb-{nb['slug']}-{os.getpid()}"
|
||||||
|
tmp_out = f"{nb['slug']}.html.part"
|
||||||
|
cmd = [
|
||||||
|
"docker", "run", "--rm", "--name", container,
|
||||||
|
"--user", "1000:1000",
|
||||||
|
"--network", DOCKER_NETWORK,
|
||||||
|
"--env-file", str(MARIMO_ENV),
|
||||||
|
# same GPU wiring as the live marimo service — without it xgboost
|
||||||
|
# silently falls back to CPU and a fit that takes seconds on the
|
||||||
|
# P2000 takes many minutes
|
||||||
|
"--runtime", "nvidia",
|
||||||
|
"-e", "NVIDIA_VISIBLE_DEVICES=all",
|
||||||
|
"-e", "NVIDIA_DRIVER_CAPABILITIES=compute,utility",
|
||||||
|
"-e", "HOME=/nbhome",
|
||||||
|
"-e", "TZ=America/New_York",
|
||||||
|
# notebooks that draw without an explicit backend must not try to
|
||||||
|
# open a display inside the container
|
||||||
|
"-e", "MPLBACKEND=Agg",
|
||||||
|
"-v", f"{work}:/work",
|
||||||
|
"-v", f"{home}:/nbhome",
|
||||||
|
"-v", f"{out_dir}:/out",
|
||||||
|
"-w", "/work",
|
||||||
|
MARIMO_IMAGE,
|
||||||
|
"marimo", "export", "html", nb["source"], "-o", f"/out/{tmp_out}", "-f",
|
||||||
|
]
|
||||||
|
started = time.monotonic()
|
||||||
|
try:
|
||||||
|
proc = subprocess.run(
|
||||||
|
cmd, capture_output=True, text=True, timeout=nb["timeout"]
|
||||||
|
)
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
log(f"FAILED {nb['source']}: timed out after {nb['timeout']}s")
|
||||||
|
subprocess.run(["docker", "rm", "-f", container], capture_output=True)
|
||||||
|
(out_dir / tmp_out).unlink(missing_ok=True)
|
||||||
|
return False
|
||||||
|
if proc.returncode != 0:
|
||||||
|
tail = (proc.stderr or proc.stdout or "").strip().splitlines()[-15:]
|
||||||
|
log(f"FAILED {nb['source']}: exit {proc.returncode}")
|
||||||
|
for line in tail:
|
||||||
|
log(f" | {line}")
|
||||||
|
(out_dir / tmp_out).unlink(missing_ok=True)
|
||||||
|
return False
|
||||||
|
part = out_dir / tmp_out
|
||||||
|
if not part.exists() or part.stat().st_size == 0:
|
||||||
|
log(f"FAILED {nb['source']}: export produced no output")
|
||||||
|
part.unlink(missing_ok=True)
|
||||||
|
return False
|
||||||
|
# only swap in the new export once it is known-good, so a failure always
|
||||||
|
# leaves the previously published page intact
|
||||||
|
part.replace(out_dir / f"{nb['slug']}.html")
|
||||||
|
log(f"exported {nb['source']} -> {nb['slug']}.html ({time.monotonic() - started:.0f}s)")
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def write_if_changed(path, text):
|
||||||
|
"""Avoid rewriting identical files — auto-build.sh hashes mtimes."""
|
||||||
|
if path.exists() and path.read_text(encoding="utf-8") == text:
|
||||||
|
return False
|
||||||
|
path.write_text(text, encoding="utf-8")
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
force = "--force" in sys.argv
|
||||||
|
OUT_DIR.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
published, skipped = discover()
|
||||||
|
if not NOTEBOOK_DIR.is_dir():
|
||||||
|
log(f"notebook dir {NOTEBOOK_DIR} missing — nothing to do")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
try:
|
||||||
|
cache = json.loads(CACHE_FILE.read_text())
|
||||||
|
except (OSError, ValueError):
|
||||||
|
cache = {}
|
||||||
|
|
||||||
|
live_slugs = {nb["slug"] for nb in published}
|
||||||
|
# drop exports whose notebook lost its marker, was renamed, or was deleted
|
||||||
|
for stale in OUT_DIR.glob("*.html"):
|
||||||
|
if stale.stem not in live_slugs:
|
||||||
|
log(f"unpublishing {stale.name} (no longer marked)")
|
||||||
|
stale.unlink()
|
||||||
|
cache.pop(stale.stem, None)
|
||||||
|
|
||||||
|
entries, failures = [], 0
|
||||||
|
todo = [
|
||||||
|
nb for nb in published
|
||||||
|
if force
|
||||||
|
or cache.get(nb["slug"], {}).get("digest") != nb["digest"]
|
||||||
|
or not (OUT_DIR / f"{nb['slug']}.html").exists()
|
||||||
|
]
|
||||||
|
|
||||||
|
if todo:
|
||||||
|
with tempfile.TemporaryDirectory(prefix="garden-nb-") as tmp:
|
||||||
|
work, home = staged_workdir(tmp)
|
||||||
|
for nb in todo:
|
||||||
|
if export_one(nb, work, home, OUT_DIR):
|
||||||
|
cache[nb["slug"]] = {
|
||||||
|
"digest": nb["digest"],
|
||||||
|
"exported_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
|
||||||
|
}
|
||||||
|
else:
|
||||||
|
failures += 1
|
||||||
|
|
||||||
|
for nb in sorted(published, key=lambda n: (n["order"], n["title"].lower())):
|
||||||
|
if not (OUT_DIR / f"{nb['slug']}.html").exists():
|
||||||
|
# never exported successfully — leave it out of the index rather
|
||||||
|
# than publish a link to a 404
|
||||||
|
continue
|
||||||
|
cached = cache.get(nb["slug"], {})
|
||||||
|
entries.append(
|
||||||
|
{
|
||||||
|
"slug": nb["slug"],
|
||||||
|
"title": nb["title"],
|
||||||
|
"description": nb["description"],
|
||||||
|
"order": nb["order"],
|
||||||
|
"source": nb["source"],
|
||||||
|
"exportedAt": cached.get("exported_at"),
|
||||||
|
# the notebook changed but its re-export failed: the page is
|
||||||
|
# real, just behind the notebook
|
||||||
|
"stale": cached.get("digest") != nb["digest"],
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
# Both writes go through write_if_changed: auto-build.sh's signature
|
||||||
|
# hashes the mtime of every file under notebooks-export/, dotfiles
|
||||||
|
# included, so rewriting an identical cache would force a full garden
|
||||||
|
# rebuild on every single export tick.
|
||||||
|
write_if_changed(CACHE_FILE, json.dumps(cache, indent=2, sort_keys=True) + "\n")
|
||||||
|
changed = write_if_changed(
|
||||||
|
OUT_DIR / "index.json", json.dumps(entries, indent=2) + "\n"
|
||||||
|
)
|
||||||
|
|
||||||
|
for name, why in skipped:
|
||||||
|
log(f"skipped {name} ({why})")
|
||||||
|
log(
|
||||||
|
f"{len(entries)} published, {len(skipped)} skipped, {failures} failed"
|
||||||
|
f"{', index updated' if changed else ''}"
|
||||||
|
)
|
||||||
|
# a failed notebook must not fail the run — the garden still builds with
|
||||||
|
# whatever exported cleanly. Exit 1 only so systemd surfaces it.
|
||||||
|
return 1 if failures else 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Export marimo notebooks marked for publication into the garden
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=oneshot
|
||||||
|
# Own lock, separate from the garden build's .build.lock: executing a
|
||||||
|
# notebook can take minutes, and it must never hold up the 5-min build tick.
|
||||||
|
# -n means a tick that lands while an export is still running just skips.
|
||||||
|
ExecStart=/usr/bin/flock -n /home/nox/docker/garden-astro/.notebooks.lock /home/nox/docker/garden-astro/deploy/export-notebooks.py
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Periodic marimo notebook export check (every 30 min)
|
||||||
|
|
||||||
|
[Timer]
|
||||||
|
# Notebooks change far less often than the vault and each export executes
|
||||||
|
# real code against the databases, so this runs much lazier than the garden
|
||||||
|
# build. The offset keeps it off the :00/:30 build ticks.
|
||||||
|
OnCalendar=*:07/30
|
||||||
|
RandomizedDelaySec=60
|
||||||
|
Persistent=true
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=timers.target
|
||||||
@@ -10,6 +10,15 @@ server {
|
|||||||
try_files $uri $uri.html $uri/index.html =404;
|
try_files $uri $uri.html $uri/index.html =404;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# The notebooks listing builds to notebooks.html *and* has a sibling
|
||||||
|
# notebooks/ directory of detail pages. try_files above resolves
|
||||||
|
# /notebooks and /notebooks/<slug>, but not the trailing-slash form —
|
||||||
|
# a bare directory has no index.html. Map it explicitly so every folder
|
||||||
|
# URL on the site keeps working with or without the slash.
|
||||||
|
location = /notebooks/ {
|
||||||
|
try_files /notebooks.html =404;
|
||||||
|
}
|
||||||
|
|
||||||
location /assets/ {
|
location /assets/ {
|
||||||
expires 7d;
|
expires 7d;
|
||||||
add_header Cache-Control "public";
|
add_header Cache-Control "public";
|
||||||
|
|||||||
+1
-1
@@ -5,7 +5,7 @@
|
|||||||
"private": true,
|
"private": true,
|
||||||
"scripts": {
|
"scripts": {
|
||||||
"dev": "astro dev",
|
"dev": "astro dev",
|
||||||
"build": "node scripts/copy-assets.mjs && astro build",
|
"build": "node scripts/copy-assets.mjs && node scripts/copy-notebooks.mjs && astro build",
|
||||||
"preview": "astro preview"
|
"preview": "astro preview"
|
||||||
},
|
},
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
|
|||||||
@@ -0,0 +1,45 @@
|
|||||||
|
// Stage the marimo notebook exports produced by deploy/export-notebooks.py
|
||||||
|
// into public/, where Astro will pick them up as static files.
|
||||||
|
//
|
||||||
|
// The raw marimo HTML lands at public/nb/<slug>.html (served /nb/<slug>.html)
|
||||||
|
// and is what the notebook pages iframe. It deliberately does NOT live under
|
||||||
|
// /notebooks/ — those URLs belong to the Astro pages that wrap it in the
|
||||||
|
// garden chrome, and with build.format:'file' the two would collide on a
|
||||||
|
// trailing slash.
|
||||||
|
//
|
||||||
|
// No exports (export script never ran, or nothing is marked) is a normal
|
||||||
|
// state, not an error: the notebooks section simply doesn't render.
|
||||||
|
import fs from 'node:fs';
|
||||||
|
import path from 'node:path';
|
||||||
|
import { fileURLToPath } from 'node:url';
|
||||||
|
|
||||||
|
const REPO = path.join(path.dirname(fileURLToPath(import.meta.url)), '..');
|
||||||
|
const SRC_DIR = path.join(REPO, 'notebooks-export');
|
||||||
|
const OUT_DIR = path.join(REPO, 'public', 'nb');
|
||||||
|
|
||||||
|
fs.rmSync(OUT_DIR, { recursive: true, force: true });
|
||||||
|
|
||||||
|
let index = [];
|
||||||
|
try {
|
||||||
|
index = JSON.parse(fs.readFileSync(path.join(SRC_DIR, 'index.json'), 'utf8'));
|
||||||
|
} catch {
|
||||||
|
console.log('copy-notebooks: no notebooks-export/index.json — nothing to stage');
|
||||||
|
process.exit(0);
|
||||||
|
}
|
||||||
|
|
||||||
|
fs.mkdirSync(OUT_DIR, { recursive: true });
|
||||||
|
|
||||||
|
let copied = 0;
|
||||||
|
for (const nb of index) {
|
||||||
|
const src = path.join(SRC_DIR, `${nb.slug}.html`);
|
||||||
|
if (!fs.existsSync(src)) {
|
||||||
|
// export-notebooks.py only indexes notebooks it exported, so this means
|
||||||
|
// the two got out of sync — warn rather than ship a link to a 404
|
||||||
|
console.warn(`copy-notebooks: MISSING export for indexed notebook ${nb.slug}`);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
fs.copyFileSync(src, path.join(OUT_DIR, `${nb.slug}.html`));
|
||||||
|
copied++;
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log(`copy-notebooks: ${copied}/${index.length} staged -> ${OUT_DIR}`);
|
||||||
+63
-2
@@ -5,6 +5,7 @@ import { buildNavTree } from '../lib/nav.mjs';
|
|||||||
import { COMMIT, BUILT_AT } from '../lib/build-info.mjs';
|
import { COMMIT, BUILT_AT } from '../lib/build-info.mjs';
|
||||||
import NavTree from '../components/NavTree.astro';
|
import NavTree from '../components/NavTree.astro';
|
||||||
import Toc from '../components/Toc.astro';
|
import Toc from '../components/Toc.astro';
|
||||||
|
import { loadNotebooks, notebookUrl } from '../lib/notebooks.mjs';
|
||||||
|
|
||||||
interface Props {
|
interface Props {
|
||||||
title: string;
|
title: string;
|
||||||
@@ -13,12 +14,27 @@ interface Props {
|
|||||||
// which /og/<slug>.png card this page advertises; synthetic listing
|
// which /og/<slug>.png card this page advertises; synthetic listing
|
||||||
// pages fall back to the site card
|
// pages fall back to the site card
|
||||||
ogSlug?: string;
|
ogSlug?: string;
|
||||||
|
// widen the content column — for embedded notebooks, which are far too
|
||||||
|
// wide for the 44rem prose measure
|
||||||
|
wide?: boolean;
|
||||||
}
|
}
|
||||||
const { title, description, headings = [], ogSlug = 'index' } = Astro.props;
|
const {
|
||||||
|
title,
|
||||||
|
description,
|
||||||
|
headings = [],
|
||||||
|
ogSlug = 'index',
|
||||||
|
wide = false,
|
||||||
|
} = Astro.props;
|
||||||
const isHome = Astro.url.pathname === '/';
|
const isHome = Astro.url.pathname === '/';
|
||||||
const entries = await getCollection('garden');
|
const entries = await getCollection('garden');
|
||||||
const navTree = buildNavTree(entries);
|
const navTree = buildNavTree(entries);
|
||||||
const noteCount = entries.length;
|
const noteCount = entries.length;
|
||||||
|
// Notebooks are a separate section, NOT injected into the garden collection:
|
||||||
|
// they aren't vault notes, so they must not move noteCount or the sitemap.
|
||||||
|
const notebooks = loadNotebooks();
|
||||||
|
const navPath = decodeURIComponent(Astro.url.pathname)
|
||||||
|
.replace(/index\.html$/, '')
|
||||||
|
.replace(/\.html$/, '');
|
||||||
const showToc = headings.filter((h) => h.depth <= 3).length >= 2;
|
const showToc = headings.filter((h) => h.depth <= 3).length >= 2;
|
||||||
const ogImage = new URL(`/og/${ogSlug}.png`, Astro.site);
|
const ogImage = new URL(`/og/${ogSlug}.png`, Astro.site);
|
||||||
const pageUrl = new URL(Astro.url.pathname, Astro.site);
|
const pageUrl = new URL(Astro.url.pathname, Astro.site);
|
||||||
@@ -60,7 +76,7 @@ const pageUrl = new URL(Astro.url.pathname, Astro.site);
|
|||||||
src="https://plausible.io/js/script.js"></script>
|
src="https://plausible.io/js/script.js"></script>
|
||||||
</head>
|
</head>
|
||||||
<body class={isHome ? 'home' : undefined}>
|
<body class={isHome ? 'home' : undefined}>
|
||||||
<div class="layout">
|
<div class:list={['layout', wide && 'layout-wide']}>
|
||||||
<div class="content-col">
|
<div class="content-col">
|
||||||
<header>
|
<header>
|
||||||
<div class="site-head">
|
<div class="site-head">
|
||||||
@@ -150,6 +166,29 @@ const pageUrl = new URL(Astro.url.pathname, Astro.site);
|
|||||||
<aside class="sidebar">
|
<aside class="sidebar">
|
||||||
<p class="sidebar-title"><a href="/">the garden</a></p>
|
<p class="sidebar-title"><a href="/">the garden</a></p>
|
||||||
<NavTree nodes={navTree} currentPath={Astro.url.pathname} />
|
<NavTree nodes={navTree} currentPath={Astro.url.pathname} />
|
||||||
|
{
|
||||||
|
notebooks.length > 0 && (
|
||||||
|
<div class="sidebar-section">
|
||||||
|
<p class="sidebar-title">
|
||||||
|
<a href="/notebooks">notebooks</a>
|
||||||
|
</p>
|
||||||
|
<ul class="nav-tree">
|
||||||
|
{notebooks.map((nb) => (
|
||||||
|
<li>
|
||||||
|
<a
|
||||||
|
href={notebookUrl(nb.slug)}
|
||||||
|
aria-current={
|
||||||
|
navPath === notebookUrl(nb.slug) ? 'page' : undefined
|
||||||
|
}
|
||||||
|
>
|
||||||
|
{nb.title}
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
))}
|
||||||
|
</ul>
|
||||||
|
</div>
|
||||||
|
)
|
||||||
|
}
|
||||||
</aside>
|
</aside>
|
||||||
{
|
{
|
||||||
showToc && (
|
showToc && (
|
||||||
@@ -692,11 +731,24 @@ const pageUrl = new URL(Astro.url.pathname, Astro.site);
|
|||||||
padding: 1.5rem 0.5rem 1.5rem 0;
|
padding: 1.5rem 0.5rem 1.5rem 0;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
/* notebook pages: the embedded marimo frame needs far more than the
|
||||||
|
44rem prose measure, so the content column takes whatever is left */
|
||||||
|
@media (min-width: 72rem) {
|
||||||
|
.layout-wide {
|
||||||
|
grid-template-columns: 17rem minmax(0, 1fr);
|
||||||
|
max-width: 96rem;
|
||||||
|
}
|
||||||
|
}
|
||||||
@media (min-width: 88rem) {
|
@media (min-width: 88rem) {
|
||||||
.layout {
|
.layout {
|
||||||
grid-template-columns: 17rem minmax(0, 44rem) 13rem;
|
grid-template-columns: 17rem minmax(0, 44rem) 13rem;
|
||||||
max-width: 80rem;
|
max-width: 80rem;
|
||||||
}
|
}
|
||||||
|
/* wide pages keep the two-column shape — no TOC rail */
|
||||||
|
.layout-wide {
|
||||||
|
grid-template-columns: 17rem minmax(0, 1fr);
|
||||||
|
max-width: 96rem;
|
||||||
|
}
|
||||||
.toc-col {
|
.toc-col {
|
||||||
display: block;
|
display: block;
|
||||||
position: sticky;
|
position: sticky;
|
||||||
@@ -931,6 +983,15 @@ const pageUrl = new URL(Astro.url.pathname, Astro.site);
|
|||||||
font-weight: 700;
|
font-weight: 700;
|
||||||
color: var(--fg);
|
color: var(--fg);
|
||||||
}
|
}
|
||||||
|
/* second sidebar block (notebooks) — hairline off the garden tree */
|
||||||
|
.sidebar-section {
|
||||||
|
margin-top: 1.75rem;
|
||||||
|
padding-top: 1rem;
|
||||||
|
border-top: 1px solid var(--hair);
|
||||||
|
}
|
||||||
|
.sidebar-section > ul {
|
||||||
|
padding-left: 0;
|
||||||
|
}
|
||||||
|
|
||||||
/* ---- table of contents ---- */
|
/* ---- table of contents ---- */
|
||||||
.toc-title {
|
.toc-title {
|
||||||
|
|||||||
@@ -0,0 +1,30 @@
|
|||||||
|
// Published marimo notebooks, as recorded by deploy/export-notebooks.py.
|
||||||
|
//
|
||||||
|
// The export script is the only thing that reads the publish markers in the
|
||||||
|
// notebooks themselves; it hands the build this index.json. That keeps the
|
||||||
|
// Astro build free of any dependency on the notebook directory — it isn't
|
||||||
|
// even mounted into the build container — and means a malformed marker can
|
||||||
|
// only ever break an export, never the garden.
|
||||||
|
import fs from 'node:fs';
|
||||||
|
import path from 'node:path';
|
||||||
|
import { fileURLToPath } from 'node:url';
|
||||||
|
|
||||||
|
const INDEX = path.join(
|
||||||
|
path.dirname(fileURLToPath(import.meta.url)),
|
||||||
|
'..',
|
||||||
|
'..',
|
||||||
|
'notebooks-export',
|
||||||
|
'index.json',
|
||||||
|
);
|
||||||
|
|
||||||
|
export function loadNotebooks() {
|
||||||
|
try {
|
||||||
|
const parsed = JSON.parse(fs.readFileSync(INDEX, 'utf8'));
|
||||||
|
return Array.isArray(parsed) ? parsed : [];
|
||||||
|
} catch {
|
||||||
|
return []; // never exported: the section just doesn't render
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export const notebookUrl = (slug) => `/notebooks/${slug}`;
|
||||||
|
export const notebookRawUrl = (slug) => `/nb/${slug}.html`;
|
||||||
@@ -0,0 +1,74 @@
|
|||||||
|
---
|
||||||
|
import Base from '../../layouts/Base.astro';
|
||||||
|
import { loadNotebooks, notebookRawUrl } from '../../lib/notebooks.mjs';
|
||||||
|
|
||||||
|
export function getStaticPaths() {
|
||||||
|
return loadNotebooks().map((nb) => ({ params: { slug: nb.slug }, props: { nb } }));
|
||||||
|
}
|
||||||
|
|
||||||
|
const { nb } = Astro.props;
|
||||||
|
const raw = notebookRawUrl(nb.slug);
|
||||||
|
const ranAt = nb.exportedAt
|
||||||
|
? new Date(nb.exportedAt).toLocaleString('en-US', {
|
||||||
|
year: 'numeric',
|
||||||
|
month: 'short',
|
||||||
|
day: 'numeric',
|
||||||
|
hour: 'numeric',
|
||||||
|
minute: '2-digit',
|
||||||
|
timeZone: 'America/New_York',
|
||||||
|
})
|
||||||
|
: null;
|
||||||
|
---
|
||||||
|
|
||||||
|
<Base title={nb.title} description={nb.description} wide>
|
||||||
|
<h1>{nb.title}</h1>
|
||||||
|
{nb.description && <p>{nb.description}</p>}
|
||||||
|
|
||||||
|
<p class="nb-bar">
|
||||||
|
<a href={raw} target="_blank" rel="noopener">open standalone ↗</a>
|
||||||
|
{ranAt && <span class="nb-when">· last run {ranAt} ET</span>}
|
||||||
|
{nb.stale && (
|
||||||
|
<span class="nb-stale">
|
||||||
|
· the notebook has changed since this run
|
||||||
|
</span>
|
||||||
|
)}
|
||||||
|
</p>
|
||||||
|
|
||||||
|
{/* marimo pages are tall and set their own scrolling, so the frame gets a
|
||||||
|
fixed viewport-relative height rather than trying to autosize */}
|
||||||
|
<iframe
|
||||||
|
class="nb-frame"
|
||||||
|
src={raw}
|
||||||
|
title={`${nb.title} — marimo notebook`}
|
||||||
|
loading="lazy"></iframe>
|
||||||
|
|
||||||
|
<style>
|
||||||
|
.nb-bar {
|
||||||
|
margin: 0.75rem 0 1rem;
|
||||||
|
font-size: 0.8rem;
|
||||||
|
color: var(--muted);
|
||||||
|
}
|
||||||
|
.nb-bar a {
|
||||||
|
font-size: 0.8rem;
|
||||||
|
}
|
||||||
|
.nb-when,
|
||||||
|
.nb-stale {
|
||||||
|
margin-left: 0.25rem;
|
||||||
|
}
|
||||||
|
.nb-stale {
|
||||||
|
color: #d97706;
|
||||||
|
}
|
||||||
|
.nb-frame {
|
||||||
|
display: block;
|
||||||
|
width: 100%;
|
||||||
|
height: min(88vh, 1200px);
|
||||||
|
min-height: 32rem;
|
||||||
|
border: 1px solid var(--hair);
|
||||||
|
border-radius: 4px;
|
||||||
|
/* exports inherit the marimo server's dark theme, so the frame sits
|
||||||
|
flush with the page instead of flashing a white slab while it loads */
|
||||||
|
background: var(--bg);
|
||||||
|
color-scheme: dark;
|
||||||
|
}
|
||||||
|
</style>
|
||||||
|
</Base>
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
---
|
||||||
|
import Base from '../../layouts/Base.astro';
|
||||||
|
import { loadNotebooks, notebookUrl } from '../../lib/notebooks.mjs';
|
||||||
|
|
||||||
|
const notebooks = loadNotebooks();
|
||||||
|
const fmt = (iso) =>
|
||||||
|
iso
|
||||||
|
? new Date(iso).toLocaleDateString('en-US', {
|
||||||
|
year: 'numeric',
|
||||||
|
month: 'short',
|
||||||
|
day: 'numeric',
|
||||||
|
timeZone: 'America/New_York',
|
||||||
|
})
|
||||||
|
: null;
|
||||||
|
---
|
||||||
|
|
||||||
|
<Base
|
||||||
|
title="notebooks"
|
||||||
|
description="Interactive marimo notebooks — data science work from the homelab, executed against live data and snapshotted."
|
||||||
|
>
|
||||||
|
<h1>notebooks</h1>
|
||||||
|
<p>
|
||||||
|
marimo notebooks from the homelab, re-executed against their real data
|
||||||
|
sources on a schedule and snapshotted here. Code is shown alongside its
|
||||||
|
output — these are the working documents, not writeups.
|
||||||
|
</p>
|
||||||
|
|
||||||
|
{
|
||||||
|
notebooks.length === 0 ? (
|
||||||
|
<p class="nb-empty">No notebooks are published right now.</p>
|
||||||
|
) : (
|
||||||
|
<ul class="nb-list">
|
||||||
|
{notebooks.map((nb) => (
|
||||||
|
<li>
|
||||||
|
<a href={notebookUrl(nb.slug)}>{nb.title}</a>
|
||||||
|
{nb.description && <p class="nb-desc">{nb.description}</p>}
|
||||||
|
{fmt(nb.exportedAt) && (
|
||||||
|
<p class="nb-meta">last run {fmt(nb.exportedAt)}</p>
|
||||||
|
)}
|
||||||
|
</li>
|
||||||
|
))}
|
||||||
|
</ul>
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
<style>
|
||||||
|
.nb-list {
|
||||||
|
list-style: none;
|
||||||
|
margin: 2rem 0 0;
|
||||||
|
padding: 0;
|
||||||
|
}
|
||||||
|
.nb-list li {
|
||||||
|
margin: 0 0 1.6rem;
|
||||||
|
padding-left: 0.9rem;
|
||||||
|
border-left: 2px solid var(--hair);
|
||||||
|
}
|
||||||
|
.nb-list > li > a {
|
||||||
|
font-size: 1.05rem;
|
||||||
|
}
|
||||||
|
.nb-desc {
|
||||||
|
margin: 0.3rem 0 0;
|
||||||
|
color: var(--muted);
|
||||||
|
}
|
||||||
|
.nb-meta,
|
||||||
|
.nb-empty {
|
||||||
|
margin: 0.3rem 0 0;
|
||||||
|
font-size: 0.75rem;
|
||||||
|
color: var(--muted);
|
||||||
|
}
|
||||||
|
</style>
|
||||||
|
</Base>
|
||||||
Reference in New Issue
Block a user