Books
Ideas
- idea Move Kobo sync snapshot building out of the request path
-
Future implementation spec for annotation sync over the Kobo
reading_services_hostendpoints will need an auth model. Documenting the proposed approach here so the spec can reference it.Observed from the annotations research spike
The Kobo device sends annotation traffic with a Kobo-cloud-issued Bearer JWT in the
Authorizationheader. Example claims (captured vianc :5001):{ "iss": "https://auth.kobobooks.com", "aud": ["https://auth.kobobooks.com/resources", "profileauth", "public_api"], "client_id": "nickel", "kobo_user_id": "7081525b-05f9-4cbf-91e8-c93f936d17f6", "scope": ["openid", "profile", "kobo_profile", "public_api_authenticated", ...], "nbf": 1776709922, "exp": 1776713522 }The JWT itself rotates (~1 hour expiry). The
kobo_user_idclaim inside it is stable for the account.Proposed auth model
Trust-on-first-use on the
kobo_user_idclaim, mirroring the existing first-login-owner pattern (#19):- On first request to a reading-services endpoint, base64-decode the JWT payload (no signature verification — we can't, Kobo holds the signing key).
- Extract
kobo_user_id. - Persist as the server's bound Kobo user.
- On subsequent requests, decode again and reject any JWT whose
kobo_user_iddiffers from the bound value.
Surface:
/api/UserStorage/*,/api/v3/content/*, plus whatever else the annotations spec ends up owning.Why not validate the JWT signature
The JWT is signed with a Kobo-cloud key. We're a self-hosted replacement for Kobo cloud; we intentionally don't have Kobo's key and can't check it. TOFU on the user ID is the closest available analogue.
Related
- Annotations research spike findings:
docs/superpowers/specs/2026-04-20-kobo-annotations-spike-design.md(pending writeup atdocs/kobo-annotations-spike-findings.md) - First-login owner pattern: #19
-
`internal/web/handler.go` `EditPost` does the metadata UPDATE before the file rename. If the rename fails (typically because `book.FilePath` in the DB has drifted from disk), the user sees an error but the metadata write has already committed. Subsequent edits hit the same stale path and re-error.
Concrete repro from prod: book #110 had `file_path = "Scott, James C.;/Seeing Like a State.epub"` in DB while the actual file lived at `James C. Scott/Seeing Like a State.epub` (presumably from an earlier rename that updated DB but failed mid-way, or a manual move). Editing the author to "James C. Scott" then triggered:
``` file rename failed: open /opt/books/library/Scott, James C.;/Seeing Like a State.epub: no such file or directory ```
Two things worth doing:
-
Defensive rename. When `lib.Rename` finds the source missing AND the computed destination already exists, treat it as a successful no-op and let the handler update `file_path`/`cover_path` to match. (Risky if the destination collision is unrelated — collision-free naming uses " (2)" suffixing, so this case strongly suggests prior rename to the same path.)
-
Atomic metadata + rename. Don't commit the metadata UPDATE until the file rename succeeds, OR rollback the metadata change when rename fails. Today the partial-failure window leaves DB ahead of disk and every subsequent edit re-errors.
Workaround for stuck rows: `UPDATE books SET file_path='', cover_path='' WHERE id=` directly against SQLite.
-
-
Symptoms
Books are shelved on the server and marked
delivered=1in the latest sync session, but absent from the Kobo's library. Reproduced today: 2/12 shelved books missing on device.Root cause
internal/kobo/sync.go(line ~173) marks books delivered without emitting any envelope when theirmeta_revandfile_hashmatch the previous session — the "already there, nothing to do" fast path.If the Kobo's local DB lost a book (manual delete on device, factory reset, sync interruption, swap to new device), the server has no signal of the drift and never re-sends. The book is stuck in a "server thinks delivered, device doesn't have it" state with no recovery path.
Workaround (confirmed working)
On the shelf page: unshelve the missing book → sync from Kobo → reshelve → sync again. Forces removed→added envelopes.
Possible fixes
- Reset-sync action per device — single button on /shelf next to "Regenerate". Deletes all sync sessions for the device; next sync rebuilds from scratch (every shelved book goes through
added→NewEntitlement). Low risk, fixes drift in one shot. - Per-book "Re-send to Kobo" action — more granular, but more UI weight.
- Inferred-library debug view — log what the Kobo touches (covers, downloads, StateGet/StatePut) and surface "last seen by Kobo" per book. Half-picture (unopened books after initial sync are invisible), but useful telemetry. Would fit naturally on the errata page as another diagnostic.
Out of scope (Kobo protocol limit)
The storefront API has no way to query what books the Kobo currently holds. Only the device's local KoboReader.sqlite (USB-only) is authoritative.
Why not now
Workaround is acceptable. Errata page is the natural home if/when we build a diagnostic view.
- Reset-sync action per device — single button on /shelf next to "Regenerate". Deletes all sync sessions for the device; next sync rebuilds from scratch (every shelved book goes through
Design
- design 2026-04-10-reading-room.md
History
-
Surface per-book reading stats that Nickel already pushes via
PUT /v1/library/{id}/state, rendered as additional rows in the book page's.book-marginalia. Also captures three currently-droppedStatusInfofields and starts an append-only reading-state event log for future historical views.Summary
- Data model: migration 007 adds
times_started_reading,last_time_started_reading,last_time_finishedcolumns toreading_state; createsreading_state_eventstable with one row per meaningful PUT. - Capture:
PUT /v1/library/{id}/statenow persists the threeStatusInfofields;GETechoes them back. - UI:
/book/{uuid}shows status, time spent reading, time to finish, times opened, started reading, finished, and last read — each only when the underlying field has a value. Books without areading_staterow are unchanged. - Event log: populated but not queried; enables "minutes per day" / "books finished over time" style graphs without a future schema change.
Scope boundaries
- No aggregate views (total time, streaks, charts) — deferred.
- No invented metrics beyond what the Kobo sync API exposes.
- No multi-device aggregation — single device assumed;
GetLatestReadingStatepicks the most recent row across devices. /v1/analytics/eventbodies still accept-and-drop — reading-session telemetry channel left for a potential future investigation.
Spec + plan
- Spec:
docs/superpowers/specs/2026-04-20-stats-dashboard-design.md - Plan:
docs/superpowers/plans/2026-04-20-stats-dashboard.md
Test plan
- [ ]
go test ./...green. - [ ] Visit
/book/{uuid}for a book with areading_staterow — new rows render (status, time spent, time to finish, times opened, last read). - [ ] Visit
/book/{uuid}for a book without one — marginalia unchanged from before this PR. - [ ] Deploy the binary to abase and trigger a sync from the Kobo; confirm
reading_statenow hastimes_started_readingpopulated andreading_state_eventsstarts accumulating rows.
- Data model: migration 007 adds
-
Research spike into whether Kobo annotation sync is reachable via the
/api/kobo/*sync API. Yes, feasible — wire shapes captured for the three observed reading-services endpoints. Follow-up implementation spec is a separate sub-project; auth model tracked in #23.Summary
- Spec:
docs/superpowers/specs/2026-04-20-kobo-annotations-spike-design.md - Plan:
docs/superpowers/plans/2026-04-20-kobo-annotations-spike.md - Findings:
docs/kobo-annotations-spike-findings.md— methodology + results + wire-shape tables + recommendation.
Key discoveries
- Annotation sync runs in Nickel when the device is signed in to a Kobo cloud account. Traffic goes to
reading_services_host(defaultreadingservices.kobo.com) with a Kobo-cloud-issued Bearer JWT. - Three endpoints observed + characterised:
POST /api/v3/content/checkforchanges,PATCH /api/v3/content/{bookID}/annotations,GET /api/UserStorage/Metadata. - Nickel treats
reading_services_hostas host-only (path component discarded), so requests arrive at server root — not underapi_endpoint's prefix. nc :5001was the tool that unstuck the spike after passive catch-all + probe stubs both yielded null results.
Docs corrections (spin-off)
Two claims in
docs/kobo-api.mdthat the spike proved wrong, now corrected in the same PR:- Automatic host rewriting is storeapi-specific, not universal.
- Init write-back is selective, not universal — only a subset of Resources keys get persisted.
Plus a new Chapter 3 subsection documenting
reading_services_host's host-only quirk.Infrastructure aftermath
- Permanent: low-noise
slog.Info("unknown kobo endpoint", ...)drift detector in the catch-all. - Removed at teardown: transcript recorder, probe handlers, spike env vars, reading-services + notebooks init overrides, broadened catch-alls.
Test plan
- [ ] Read the findings doc; confirm wire shapes match your mental model.
- [ ] Read the docs corrections in
docs/kobo-api.md(diff against the mergeddocs/kobo-apifrom #22). - [ ] Verify
go test ./...green on this branch. - [ ] Verify abase server runs the teardown build cleanly (no transcript infrastructure, no spike env vars). Already deployed.
- [ ] Confirm the device's normal library sync still works end-to-end — spike cleanup should be invisible to Nickel.
Follow-ups
- Issue #23 — auth model (TOFU on
kobo_user_id) for reading-services endpoints. - Implementation spec for annotation sync — separate sub-project, separate branch.
- Spec:
-
Summary
- Replace
docs/kobo-api-findings.md(chronological field notes) withdocs/kobo-api.md— a reference-shaped document for external self-hosters documenting the Kobo sync protocol. - Single file, six chapters: Overview, Debugging, Conventions, Endpoint reference (one subsection per route, uniform template), Known failure modes, References. Appendix on device-side
KoboReader.sqlitestate. Contributing footer with three lightweight maintenance conventions. - Every route in
internal/kobo/handler.gonow has a subsection with Implementation, Called by, Purpose, Request, Response, Quirks, and Reference implementations fields. README.mdlinks updated to point at the new file.- Spec at
docs/superpowers/specs/2026-04-20-kobo-api-docs-refactor-design.md; implementation plan atdocs/superpowers/plans/2026-04-20-kobo-api-docs-refactor.md.
Test plan
- [ ] Render
docs/kobo-api.mdin a markdown viewer — ToC links resolve, code fences close, Chapter 5 anchor links jump to the correct Chapter 4 subsections. - [ ] Verify every route in
internal/kobo/handler.gohas a matching subsection (expected: 19 H4 endpoint headings). - [ ] Spot-check that the documented request/response shapes match the handler code for at least
/v1/initialization,/v1/library/sync,/v1/library/{id}/metadata,/v1/analytics/gettests, and the download endpoint. - [ ] Verify
docs/kobo-api-findings.mdis gone andREADME.mdno longer links to it.
- Replace
-
Summary
- Replace required
BOOKS_OIDC_OWNER_SUBwith first-login-wins ownership claim - New
settingskey/value table (migration 006);ClaimOwnerSubusesINSERT ... ON CONFLICT DO NOTHING *Authcaches the owner in anatomic.Pointer[string], loaded inNewand refreshed by the OIDC callback- Recovery is manual SQLite edit + restart (no admin API)
Test plan
- [x]
go test ./...passes - [ ] On a.bas.es: install books, log in via Pocket ID first, confirm shelf/admin views work
- [ ] Verify a second OIDC user logging in is treated as user (not owner)
- Replace required
-
The install task was downloading the binary directly to /usr/local/bin/books, so sha256sum -c (which looks up filenames listed in the manifest) couldn't find a match and exited 1. Download to the versioned name first, verify, then install.
-
OpenRC sources conf.d but doesn't export — prefix each line with
exportso the daemon actually receives BOOKS_OIDC_ISSUER, BOOKS_PUBLIC_URL, etc. Without this the service crashes on start with 'BOOKS_OIDC_ISSUER is required'. -
Summary
- New
Basefiledescribing books as an Alpine+OpenRC service for bases - Install task curls a prebuilt release asset from code.bas.es
- OIDC env vars renamed via
setup.auth.envto match what books reads - New
scripts/release.shcross-compileslinux/amd64, tags, pushes, and uploads the binary viatea
Test plan
- [ ] Run
scripts/release.sh v0.1.0and verify the asset lands on code.bas.es - [ ]
bases installon a.bas.es withBOOKS_OIDC_OWNER_SUBfilled in - [ ] Confirm the service starts via OpenRC and responds on books.a.bas.es
- New