Bookswarm

Bookswarm extracts deterministic text segments from EPUB or UTF-8 text files, renders each segment through a YAML-declared provider, stores audio in Kubo, and publishes signed content claims for verified peer reuse.

Requirements

• Python 3.9+
• Kubo RPC endpoint for audio storage
• Optional: a claim registry service and a trusted producer public key for peer reuse
• Docker-compatible runtime (such as Colima) for integration tests

Install the project and test dependencies:

python -m pip install -e '.[test]'

On macOS, the Docker-backed tests can be run with Colima:

brew install colima docker docker-compose
colima start

Generate signing keys

Each producer signs claims that associate request IDs with audio CIDs. A release publisher signs the complete audiobook manifest, and a curator signs endorsements of exact announcements. Generate separate raw 32-byte Ed25519 keys; each command prints its base64url public key:

bookswarm generate-producer-key producer.key
bookswarm generate-publisher-key publisher.key

Keep all key files private. The producer key is used for audio claims; the publisher key is used for releases; the curator key is used only for explicit endorsements.

bookswarm generate-curator-key curator.key

Create and inspect a release

bookswarm create-release book.txt \
  --producer-key producer.key \
  --publisher-key publisher.key \
  --tts-provider PROVIDER_ID \
  --ipfs-api-url http://127.0.0.1:5001/api/v0

Creation stores and pins signed audio and release bytes in Kubo but announces nothing. Publication is explicit and uses a normalized ISBN; multiple releases may share one ISBN.

Publish a release announcement to a relay (ingestion is open by default; operators may configure an optional publisher allowlist). The relay stores signed metadata only and never fetches audio:

bookswarm publish RELEASE_CID --publisher-key publisher.key \
  --relay-url http://127.0.0.1:8000 \
  [--isbn ISBN]

Search one or more relays with bounded, trust-aware cursor-paginated pages and exact signed renderer facets:

bookswarm search ISBN --relay-url http://127.0.0.1:8000 \
  --limit 50 --provider PROVIDER_ID --model MODEL_ID \
  --voice VOICE --language en-us --output-profile wav-pcm16-24k-mono

Use the returned next_cursor with --cursor to continue. Search normalizes ISBN-10 and ISBN-13 forms. Signature validity proves publisher authorship of the announcement, not that the ISBN assertion is truthful; trust and curation are separate. No command automatically retrieves release audio.

Phase 6 adds bounded sequence-based catalog synchronization, supervised foreground watch mode, immutable Kubo snapshots, and an explicit Nostr object bridge. Nostr uses optional nostr-sdk==0.45.1, experimental regular kind 30078, and a separate secp256k1 key; it is not automatic peer or catalog discovery. See docs/audit-phase-6.md.

Relays also expose an append-only signed-object catalog at GET /v1/catalog?limit=50&after=0. The response includes a persistent catalog_id, an inclusive through watermark, sequence-numbered announcement/endorsement objects, and next_after. Pin catalog_id and through for a stable traversal; new arrivals are excluded until a new capture. Existing databases backfill announcements and endorsements once in deterministic order without rewriting signed bytes. The inventory is only that relay's assertion: it does not prove global completeness, truth, or trust.

bookswarm catalog-sync --from-relay http://127.0.0.1:8000 --to-relay http://127.0.0.1:8001 --state sync.json --max-items 50
bookswarm snapshot-create --relay-url http://127.0.0.1:8000 --ipfs-api-url http://127.0.0.1:5001/api/v0
bookswarm generate-nostr-key nostr.key

bookswarm inspect-release RELEASE_CID \\
  --ipfs-api-url http://127.0.0.1:5001/api/v0

inspect-release verifies release signatures and structure, but does not express trust or fetch audio. Explicit publication arrives in Phase 4.

Keep producer.key private. Pass the producer public key to consumers with --trusted-key when using peer reuse.

Run the pipeline

bookswarm run book.txt \
  --tts-provider PROVIDER_ID \
  --asr-provider ASR_PROVIDER_ID \\
  --producer-key producer.key \
  --ipfs-api-url http://127.0.0.1:5001/api/v0 \
  --registry-url http://127.0.0.1:8000 \
  --trusted-key PRODUCER_PUBLIC_KEY \
  --verification-mode signature-plus-asr

# Equivalent: python -m bookswarm.run run book.txt ...

The command extracts the source into deterministic sections and segments. For every segment it derives a request ID from the normalized text and synthesis configuration, then:

1. Looks for a valid local or registry claim.
2. Retrieves the claimed CID from Kubo and verifies the signed claim and media metadata.
3. Pins a valid peer artifact instead of synthesizing it.
4. Otherwise synthesizes locally, stores the WAV in Kubo, signs a claim, and publishes it locally and to the configured registry.

Use --offline to avoid registry lookup and publication.

Verification modes

• signature-plus-asr (default): verifies the producer signature, strict media metadata, and ASR similarity through the selected declarative ASR provider. ASR provenance is listener-side verification metadata, not a signed claim field.
• signature-only: verifies the producer signature and strict WAV metadata without running ASR. This is suitable when a trusted signature is sufficient or model assets for ASR are unavailable.
• local-only: does not verify or reuse peer artifacts and does not contact the registry.

The committed provider catalog contains no secrets and has no active-provider selection; Kokoro and faster-whisper are not required. When --model is omitted, the provider catalog's model.id becomes the effective model in requests and signed renderer metadata. Provider/model provenance is recorded in signed claims and release announcements and displayed to listeners. Relay ingestion is open by default and may be restricted by operator policy; valid signatures remain distinct from trust or endorsement.

Registry

Run the registry service with a persistent SQLite database and its allowed producer keys:

BOOKSWARM_REGISTRY_DB=claims.sqlite3 \
BOOKSWARM_ALLOWED_KEYS=PRODUCER_PUBLIC_KEY \
python -m uvicorn bookswarm_registry.main:app --host 127.0.0.1 --port 8000

The registry stores signed claim envelopes. It is untrusted for artifact validity: consumers still validate signatures, retrieve the CID through their own Kubo node, inspect WAV metadata, and pin accepted artifacts.

Tests

Run unit tests:

pytest -m "not integration"

Run the real single-node Kubo test:

pytest tests/test_kubo_integration.py -m integration

Run the two-node peer-reuse proof:

pytest tests/test_two_node_reuse_integration.py -m integration

The two-node test creates an isolated Docker network. It starts two Kubo nodes and the registry, has Node A extract and synthesize a multi-segment text fixture, then has Node B independently extract the same source and reuse, verify, retrieve, and pin each signed artifact over direct Kubo peer connectivity. It then stops Node A and confirms Node B can still retrieve its pinned artifacts. The test does not use a DHT application mapping or a shared artifact filesystem.

Security notes

• Kubo RPC is administrative. Bind it to loopback or a private/isolated container network; never expose it publicly.
• A CID provides integrity, not secrecy. Publishing to a public IPFS node may make the audio retrievable.
• An isolated Docker test network is not an IPFS cryptographically private network. That would require a shared swarm.key and LIBP2P_FORCE_PNET configuration.
• Trust only producer public keys you intend to accept.