Async: what runs in parallel¶

ZeroAI is single-process, single-thread, asynchronous. One event loop (uvicorn + asyncio) serves every request. Nothing blocks it: network, database and model calls are all awaited, so while one request waits for a feed or for the model, the loop serves another.

Where time goes in one briefing¶

gantt
    title One briefing (illustrative seconds)
    dateFormat HH:mm:ss
    axisFormat %S s
    section Feeds (in parallel)
    Yahoo Finance        :a1, 00:00:00, 2s
    Google News          :a2, 00:00:00, 3s
    Nasdaq (slow)        :a3, 00:00:00, 3s
    section Model (one at a time)
    Wait for the gate    :crit, b1, 00:00:03, 1s
    Reasoning + answer   :b2, 00:00:04, 5s
    section After
    Store usage row      :c1, 00:00:09, 1s

The three feeds overlap, so fetching costs as long as the slowest feed, not the sum. Then the request takes its turn at the model.

Two briefings at the same time¶

gantt
    title Two requests overlap, but the model runs one at a time
    dateFormat HH:mm:ss
    axisFormat %S s
    section Request A
    Fetch feeds (parallel)  :a1, 00:00:00, 3s
    Model                   :a2, 00:00:03, 5s
    section Request B
    Fetch feeds (parallel)  :b1, 00:00:01, 3s
    Waiting for the model   :crit, b2, 00:00:04, 4s
    Model                   :b3, 00:00:08, 5s

B fetches its news while A is still working, so by the time the model is free B is ready. Only the model call queues. The UI shows "Waiting for the model" while it does.

What is parallel, serial, or limited¶

One request, step by step¶

flowchart TD
    R([GET /summary/stream]) --> D["Dependencies, awaited in turn:<br/>DB session, model config, enabled sources"]
    D --> F["asyncio.gather<br/>fetch every source concurrently"]
    F --> N[/"event news"/]
    N --> G{"LLMGate<br/>free slot?"}
    G -->|"no"| W[/"event status: Waiting for the model"/] --> G
    G -->|yes| M["PydanticAI run_stream_events<br/>(runs the agent in a background task)"]
    M -->|"reasoning tokens"| T[/"event thinking"/]
    M -->|"answer fragments"| S[/"event summary"/]
    M -->|"result + token usage"| U["Release the gate,<br/>write the usage row"]
    U --> DN[/"event done"/]

The route is an async generator: each yield hands one SSE event to Starlette, which sends it to the browser immediately. The generator only advances when the client is ready, which is natural backpressure.

The model call, in more detail¶

run_stream_events starts the agent loop in a background task and passes events to our generator through an in-memory channel. Our code consumes them one at a time:

• a reasoning delta becomes a thinking event (dropped on the server if Thinking mode is off);
• a fragment of the structured answer is appended to a buffer, parsed as partial JSON, and sent as a summary event when it changed;
• the final result carries the validated answer and the token usage.

If the model returns nothing usable, PydanticAI asks again (up to 3 retries) and our partial-JSON buffer is reset for the new attempt.

Cancellation¶

sequenceDiagram
    participant B as Browser
    participant S as Starlette
    participant G as stream_summary
    participant L as LLMGate
    participant M as Model request

    B->>S: stream open
    S->>G: iterate events
    G->>L: acquire (held)
    G->>M: streaming request
    B--xS: tab closed / reload / Stop
    S->>G: cancel (CancelledError at the next await)
    G->>M: request dropped
    G->>L: release (finally)
    Note over G: logs client_disconnected<br/>no usage row is written

Cancellation is cooperative: it lands at the next await, which for a model that streams tokens is milliseconds away. A request cancelled while still queued never reaches the model and gives its place back. This is covered by tests that cancel mid-call and while queued.

Shared resources, created once¶

What could go wrong, and why it does not¶