> ## Documentation Index > Fetch the complete documentation index at: https://docs.useharmonia.xyz/llms.txt > Use this file to discover all available pages before exploring further. # System architecture > How Harmonia wires Gemini, ADK, Firestore, Pub/Sub, Cloud Run, and the web UI together. ## System diagram The resident-autonomy control plane adds tenant-scoped Firestore cycles, leases, Micro-reflections, durable agendas, bounded experiments, and rollback. Scheduler only wakes private endpoints; deterministic policy owns eligibility, authority, cost, and transitions. See [Governed resident autonomy](/resident-autonomy). Progressively explore agents, APIs, state ownership, approval boundaries, effects, verification, and operational controls. ```mermaid theme={"system"} flowchart TB subgraph Clients UI[2:3 editorial studio
conversation + living canvas] TG[Telegram bot] IDP[Google sign-in
Identity Platform] end subgraph WebApp["Web application (Next.js / Cloud Run)"] API[REST + chat API routes] STREAM[Durable NDJSON chat runs
validated host events] A2UI[Harmonia A2UI catalog
official React renderer] UPLOAD[Resumable upload broker
magic-byte verification + quarantine] CHAT[Chat intent parser
Gemini structured output] DEC[Decision engine
policy + approval gates] CALSYNC[Operator-only Calendar sync
deterministic IDs + read-back] FS[(Firestore
workspaces/{workspaceId}/...)] LEDGER[(Budget reservations
+ immutable usage records)] PS[Pub/Sub topics] OUTBOX[(Stage outbox
pending + claimed triggers)] STOR[Asset store
.data/artifacts or GCS] end subgraph Worker["ADK worker (FastAPI / Cloud Run)"] PULL[Stage dispatcher
pull loop / push receiver] RUNTIME[Agent Engine client
mandatory, no fallback] CONTENT[Direct media operations
transcription + image generation] PROACT[Proactive agent
trend scan + engagement watch] SCHED[Scheduler dispatcher
due content items] TICK[Durable tick
isolated recovery arms] X[X API v2 client] FFMPEG[ffmpeg skills
clips, reels, images] end subgraph AgentPlatform["Vertex AI Agent Engine"] COORD[ADK coordinator
Gemini 3.5 Flash-Lite] SPECIALISTS[Single-turn specialists
Nimi analysis · Ryan strategy · Temi plan] DRAFTFLOW[Deterministic writing/review loop
selected item: Noni ↔ Dara] MEMORY[(Memory Bank
workspace + brand exact scope)] end subgraph External YT[YouTube
oEmbed + yt-dlp] HN[Hacker News front page
Algolia API] TRACE[Cloud Trace / Telemetry API
metadata-only OpenTelemetry] VEO[Veo 3.1 Fast
4-second b-roll] LYRIA[Lyria 3 Clip
30-second music] GCAL[Google Calendar API
app-created calendar only] end UI --> IDP --> API UI --> UPLOAD --> STOR API --> STREAM --> A2UI TG --> API API --> CHAT --> FS API --> DEC --> FS UI --> CALSYNC --> GCAL CALSYNC --> FS API --> LEDGER API -- transaction: state + trigger --> OUTBOX OUTBOX -- claimed publication --> PS PS --> PULL PULL --> CONTENT --> YT PULL --> RUNTIME --> COORD COORD --> SPECIALISTS COORD --> DRAFTFLOW COORD <--> MEMORY PULL --> FFMPEG PULL --> X DEC -- approved paid-media action --> PULL PULL --> VEO PULL --> LYRIA VEO --> STOR LYRIA --> STOR PROACT --> HN PROACT --> COORD SCHED --> X TICK --> OUTBOX TICK --> SCHED TICK --> PROACT CONTENT --> FS YT --> SPECIALISTS X --> FS FFMPEG --> STOR PULL --> FS API -. W3C trace context .-> PS PULL -. stage + agent + model spans .-> TRACE COORD -. delegation + validation spans .-> TRACE LEDGER -. traceId correlation .-> TRACE ``` ## Design principles ### Generative operator UI The full Console uses a versioned Harmonia catalog on Google's A2UI v0.9 wire protocol, rendered by pinned `@a2ui/react` 0.10.2 and `@a2ui/web_core` 0.10.6 packages. The catalog provides trusted renderers for messages, attachment cards, approval confirmations, safe reasoning summaries, activity traces, plans, queues, tool activity, tasks, citations, and measured context usage. A2UI messages are declarative data: the host accepts no generated JavaScript, HTML, CSS, arbitrary React component, or arbitrary callback. Unknown components, invalid HTTP(S) citations, and attachment preview URLs outside the authenticated chat route fail validation. The dashboard Console is a long-conversation editorial studio, not a stacked card feed. Desktop reserves two fifths for chaptered operator conversation and three fifths for a living cross-media canvas; narrower screens use an explicit Conversation/Studio canvas switcher. Pure selectors derive the active working set, chapters, artifact groups, and source links from persisted history and job records. Written posts, images, clips/reels, audio, transcript evidence, crop/caption-safe metadata, receipts, policy, cost, and verification appear only when corresponding records exist. Validated A2UI components are partitioned by product role: safe summaries and agent activity stay in conversation, artifacts/plans/citations stay on the canvas, and confirmation metadata stays inside the unchanged trusted approval boundary. `POST /api/chat/stream` creates a tenant-scoped Firestore run and emits schema-validated NDJSON records with monotonic sequence numbers. Each record is persisted before delivery and can be replayed from `GET /api/chat/runs/{id}/events?after={sequence}`. Activity and tool state arrive while the request is executing; the compatibility chat handler currently returns its Gemini answer as one result, which the stream then delivers in bounded deltas. This is transport streaming, not a claim that provider tokens are streamed directly. The composer creates an authenticated upload session before any bytes move. In cloud deployments the browser sends bytes directly to a tenant-scoped Cloud Storage resumable-upload URL. The server independently verifies size, type, and magic bytes, then requires a complete clean verdict from the configured private malware scanner before marking the attachment ready. Mismatches and infected objects are deleted; unavailable or malformed scan results remain quarantined. Local development uses the same fail-closed boundary. Uploaded video and audio can seed the existing ingest/transcribe/analyze pipeline. Uploading alone creates no job; the accompanying prompt still has to select the existing `create_job` intent. Image and document uploads are currently retained and rendered as conversation context, not silently treated as implemented analysis capabilities. The UI exposes only concise activity and reasoning summaries derived from validated application state. It never requests, stores, or renders hidden model chain-of-thought. Existing job actions continue through the same action-specific approval receipts and decision engine. The generic confirmation record is a typed, expiring, replay-resistant host contract for future registered tools; recording approval is not itself tool execution. ### Typed multi-agent judgment Harmonia, the ADK root coordinator, delegates evidence analysis to Nimi (`nimi_analyst`), strategy/content briefs to Ryan (`ryan_strategist`), and editorial operationalization to Temi (`temi_editorial_planner`). After independent human strategy approval, Temi proposes the complete four-week plan. Deterministic code validates it, computes its digest and lineage, persists it in Firestore, and selects exactly one eligible item. The worker then invokes Noni (`noni_copywriter`) and Dara (`dara_editor`) as separate typed ADK specialists in a deterministic loop: one original, one review, and at most one issue-bound revision plus final review. Deterministic stage code derives effect proposals from the exact accepted draft and evaluates each action independently; Firestore persists the complete trace and remains durable truth between Pub/Sub stages. All cognitive roles use policy `gear-2026-08-24`, pricing catalog `2026-08-23`, the `harmonia-standard` safety profile, and a proposed promotion quality floor of `0.95`. | Role | Default target | Temp / max output | Evaluation focus | State and tools | Prohibited authority | | ---------------------- | --------------------------- | ----------------: | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | | Harmonia coordinator | Gemini 3.5 Flash-Lite | 0.1 / 1,024 | exact one-specialist route | delegation only; invocation state | answer task, approve, publish | | Ryan strategist | Gemini 3.5 Flash | 0.4 / 4,096 | grounded four-week strategy and content briefs | typed Nimi/company/campaign/performance/memory context; one allow-listed filesystem strategy skill; optional isolated request-bound native search agent | cite skill guidance as evidence, open-ended research, invent evidence, schedule, approve, publish | | Nimi analyst | Gemini 3.5 Flash multimodal | 0.2 / 2,048 | strict source analysis with quote/time/frame and evidence-kind provenance | typed source package, verified performance, eligible memory, allow-listed filesystem skill; optional isolated public/private search agent | use skill as evidence, invent evidence/trends, define strategy, write copy, approve, schedule, publish | | Temi editorial planner | Gemini 3.5 Flash-Lite | 0.1 / 1,024 | coherent sequencing, cadence, supported channel/format, windows, deadlines, dependencies, priority | exact approved Ryan strategy, Nimi evidence, planning-method skill, six reads over one immutable Firestore planning snapshot | invent evidence/strategy, search, read outside the snapshot, write final copy, approve, schedule externally, publish | | Noni copywriter | Gemini 3.5 Flash | 0.8 / 2,048 | skill/research trace, provenance, grounding, and X length | selected item + exact brief + Nimi evidence + writing skill + verified prior posts + brief-scoped public sources | redefine strategy, access Memory Bank, inspect the rest of the plan, create facts, approve, publish | | Dara editor | Gemini 3.5 Flash | 0.2 / 2,048 | complete grounding, brief, brand, platform, CTA, safety, and clarity checks | exact Noni draft + immutable production input; one allow-listed filesystem editing skill | use skill prose as evidence, author review metadata, rewrite copy, create facts or authority, approve, publish | | Maya presenter | Gemini 3.5 Flash | 0.2 / 2,048 | exact-context component/entity graph | typed `UiContext`; no tools; host hydrates Firestore truth | invent state/IDs, emit lifecycle states, approve, publish, verify, execute actions | | Nova liaison | Gemini 3.5 Flash | 0.2 / 2,048 | grounded read-only answers | insight skills + read-only tools | mutate, approve, publish | All cognitive roles record input/output tokens under the immutable pricing catalog. The public ADK evalset and deterministic authority/grounding evaluators are committed; real-model quality, latency, and cost evidence remains pending until an authenticated private run is captured. The role catalog is strict: an unavailable configured target fails the stage. Model promotion is evaluation-gated. Candidates below the role quality floor or without known pricing are rejected before ranking; eligible candidates rank by pass rate, then cost, then p95 latency. Public fixtures contain no real transcripts or media. Live inputs and outputs remain in the private evidence workspace and mock mode is explicitly refused by the live runner. ### Actual multimodal analysis For a newly ingested YouTube job, Harmonia persists the source media digest and passes the authorized public video URI to Nimi as a Gemini `file_data` video part. The timed transcript is supplied as individually identified segments. Nimi loads its analysis skill and relevant method references before returning one strict `SourceAnalysis`. Optional current public or private context is fetched only by an isolated request-bound search agent. Runtime code validates the real trace and native grounding metadata, then rejects mismatched digests, invented references, non-verbatim quotes, invalid time/frame bounds, unsupported visual claims, and authority language before storing the full analysis, research lineage, metadata, and canonical digest. Brief-only jobs use one identified zero-time segment. Media bytes and source URIs are excluded from trace content. No agent has publishing, approval, credential, external-calendar, or destructive tools. Temi proposes editorial plans; Noni proposes grounded copy; Dara proposes structured review decisions. Deterministic code owns validation, canonical digests, persistence, item selection, the one-revision limit, and every external scheduling/effect mutation. It derives action proposals only from the exact accepted Noni draft. X and paid-media actions wait for human approval; safe internal packs, images, and ffmpeg renders remain explicitly approval-free and separate from publication. ### Managed runtime and persistent memory Agent Engine is the only cognitive execution runtime. For each cognitive operation the worker derives a stable hashed session ID, retrieves or creates that session, seeds the typed job snapshot only on creation, streams state deltas, and validates the required output keys. This permits process-restart continuation without making session state workflow authority. `AGENT_ENGINE_RESOURCE` is mandatory at worker startup. There is no in-process ADK execution branch and no runtime fallback. Memory Bank is mandatory in the managed deployment. Every retrieval and generation uses the exact `{workspace_id, brand_id}` values persisted on the job. Up to five facts are retrieved before analysis, strategy, or drafting. Writes are restricted to typed operator decisions and independently verified outcomes with durable Firestore evidence references; a learn-stage aggregate is not eligible until its own durable record exists. Raw source material, transcripts, prompts, drafts, errors, and unverified claims are ineligible. See the [state ownership matrix](/state-ownership) for every store's scope and authority. This separation is what makes the workflow architecture capable of spanning weeks; no elapsed-weeks run is claimed without evidence. Firestore—not an agent's prompt window—retains the current stage, delivery history, approvals, operation identifiers, receipts, and verification results. Pub/Sub can redeliver work after a process restart without duplicating an external effect. A restarted cognitive operation retrieves its deterministic managed session, while a new operation seeds a distinct session from Firestore and may request only eligible facts from the same workspace-and-brand Memory Bank scope. Authenticated cross-job Memory Bank retrieval remains an explicit cloud-evidence gate. ### Production-data and sovereignty boundary Harmonia does not claim a general-purpose enterprise compliance certification. It implements concrete controls that a deployment can demonstrate: * Google sign-in and server-side membership determine the tenant; callers cannot choose a workspace in a request body. * Customer collections and object names are workspace- and brand-scoped. Pub/Sub carries the same scope twice, and the worker rejects mismatched payload and attribute scopes. * Firestore is a shared SaaS database: IAM separates services, while workspace isolation inside that database is enforced by verified application context, nested paths, ownership checks, and cross-tenant tests. It is not represented as per-tenant IAM isolation. * Service accounts separate the public control plane from the private worker. Publishing still requires an action-specific human approval receipt. * OpenTelemetry spans contain identifiers, stages, role/model labels, counts, outcomes, and trace correlation only. Prompts, responses, transcripts, drafts, media, and private reasoning are excluded. * The deployment selects one `REGION` and rejects mismatched Agent Engine, Memory Bank, Veo, Firestore, or storage locations. Pub/Sub persistence and in-transit processing are restricted to that region. Because Lyria uses a global endpoint, generative media fails closed unless an operator explicitly acknowledges an approved global-processing policy exception. * Memory Bank accepts only typed eligible facts. Raw sources, creative text, failures, and unverified claims cannot be written by the normal learn path. Terminal job retention, per-job holds, administrator-confirmed job erasure, owner-confirmed workspace erasure, binary artifact cleanup, and metadata-only deletion tombstones are implemented. Tenant-wide export, customer-managed encryption, organization-policy integration, and formal regulatory certification remain deployment-specific work and must not be implied by these controls. ### SaaS identity and tenant isolation Google Identity Platform authenticates users through Google sign-in. A first sign-in creates one owner workspace and default brand. The server derives `userId`, `workspaceId`, and `brandId` from the verified session and membership record; request bodies cannot select ownership. All durable customer data lives below `workspaces/{workspaceId}`: jobs, event logs, content items, assets, connections, goals, chat history, proposals, notifications, proactive cadence, and Telegram configuration. Artifact object names include the workspace and brand. Pub/Sub messages carry the workspace and brand in both data and attributes; the worker rejects mismatches, restores invocation-local tenant context, and re-reads the job from that workspace. Agent Engine user IDs are namespaced as `{workspace}:{user}:{job}`. Each workspace has a cumulative budget guard in addition to per-job reservations. Social publishing and Telegram use that workspace's stored connection only. Background scans enumerate active workspaces and execute each scan inside a separate tenant context. ### One decision engine Every mutating action (approve, publish, schedule) flows through the same decision layer regardless of surface — dashboard button, chat message, or Telegram inline button. Telegram approvals additionally require an explicit callback tap; parsed text alone can never execute. Google Calendar is deliberately adjacent to, rather than inside, the worker action engine. A browser-session-only route accepts the operator’s explicit Add, Update, or Remove action; internal service authentication cannot invoke it. Firestore claims first-calendar provisioning, content revisions invalidate old projections, deterministic event IDs converge inserts, ETags guard updates, and Google read-back determines the stored sync state. The narrow `calendar.app.created` scope prevents Harmonia from reading unrelated personal calendar data. ### Async by Pub/Sub, state in Firestore Each stage transition and its deterministic pending outbox record are committed in one Firestore transaction. Publication then claims that record before sending the Pub/Sub trigger; the durable tick retries pending or expired claims. A crash can therefore cause a harmless duplicate trigger, but cannot leave a persisted stage without a recoverable trigger. The worker consumes stages through a push subscription on Cloud Run or an identical pull loop locally against the emulators. Jobs carry their full state (`transcriptSegments`, `moments`, `angles`, `drafts`, `actions`), so any stage can resume after a crash. The coordinator is the cognitive dispatcher, not the durable workflow engine. Pub/Sub owns asynchronous delivery and retry; Firestore owns resumable stage state, approvals, reservations, usage, provider-operation IDs, receipts, and verification evidence. Agent Engine hosts cognitive turns; it does not replace the durable stage machine. ### Correlated observability and cost controls The web process creates or continues W3C context and places `traceparent`/`tracestate` in Pub/Sub attributes. The worker continues that context through `harmonia.stage.execute`, `harmonia.agent.invoke`, `harmonia.model.generate`, and `harmonia.output.validate`. Delegation is an explicit `harmonia.agent.delegate` event. Internal HTTP callbacks carry the same context. Before dispatch, every real ADK role, transcription call, and image generation reserves a deterministic operation cost in a Firestore transaction. Successful calls finalize immutable usage records grouped by job, stage, role, and model. Retried operation IDs reuse their reservation and usage document, preventing double charging. `/api/metrics` exposes calls, units, and estimated cost by role/model plus current reserved cost. Veo and Lyria reserve their current catalog amount before provider execution. Veo's long-running operation name is persisted in a Firestore subcollection before the first poll, so Pub/Sub redelivery resumes the same operation. Completed media is stored and independently digest-verified through the existing asset boundary. Both paid-media action types always wait for explicit operator approval. Trace attributes are metadata-only. Harmonia disables both current and legacy ADK message-content capture and never attaches prompts, responses, transcripts, drafts, or media bytes. The trace shows the execution graph and explicit structured decisions; it does not expose private chain-of-thought. ### Idempotent external effects Every executable action derives a deterministic idempotency key from `(jobId, actionId, payload)` hashed with SHA-256. After durable approval, the publish stage atomically claims that key in Firestore before calling any provider or writing an artifact. Exactly one live claim may return `execute`; concurrent delivery returns `in_progress`, and a finalized claim returns `already_applied` with the original receipt identity without repeating the effect. An expired unresolved claim returns `uncertain` and requires operator reconciliation instead of an unsafe automatic retry. Receipt finalization atomically links the claim, immutable receipt, and action state. Only the explicit operator replay-proof route persists a replay observation. The deployment script converges a Cloud Scheduler job that invokes the private worker's `/durable/tick` endpoint once per minute with an OIDC service account that has only Cloud Run invocation authority. A Firestore transaction grants one 55-second workspace lease per minute. The tick runs stage-outbox recovery, scheduled-effect dispatch, proactive checks, and retention as isolated arms: one failure is recorded in the response and cannot suppress the others. Scheduled effects still enter the immutable command/effect-claim boundary. Resident Python threads remain development-only and are disabled by the full-deployment configuration. The web-only preview observed on 2026-08-26 has no deployed worker or Scheduler job and is not runtime proof of this topology. ### Independent verification Verification never trusts the executor. Published posts are re-fetched from the X API by id; exported packs are compared by digest; rendered assets are re-read from the asset store and compared byte-digests. Results land as immutable verification records on the job. ### Learnings feed forward The learn stage measures published posts via public metrics and stores evidence-linked takeaways per job. Eligible learnings re-enter Ryan's single strategy contract on a future job or revision—closing the governed learning loop without granting Memory Bank authority. Distinct role routing, Noni's skill-first writing trace, direct video attachment, managed-runtime adapter, exact-scope memory contracts, paid-media adapters, budget enforcement, usage normalization, and metadata-only trace behavior are verified offline. Authenticated Gemini team execution, Agent Engine, Memory Bank, Veo, Lyria, and exported Cloud Trace correlation remain unverified until separate live evidence is captured.