Go through Langfuse traces and catch quality drops

The agent reads the Langfuse public API: traces and the generations nested inside them, scores, sessions, prompt versions and the dataset items of the project. On top of that it builds the analysis — how many traces there were over the period and what share of them ended in an error, which steps lost time, how scores moved and which conversations dropped in quality the most. The same analysis arrives as a digest on schedule.

A trace here is not a flat record but a tree of observations, so slow at the trace level does not yet mean a slow generation: the time may be going into a neighbouring step, and the agent names the specific link. A score likewise lives as its own entity attached to a trace, while a session glues many traces into one conversation — the agent looks for a quality drop across that connection, not at a single rating detached from the dialogue.

The agent only reads observability data: it sends no traces of its own into your project and does not change prompt versions without an explicit request — editing a prompt stays your action, the agent merely ties it to the change in scores. This connection has no webhooks from Langfuse, so it reports a drop when you ask or on the digest schedule you agreed, not at the moment the drop happened.

The keys are stored encrypted and sent as basic auth with every request; revoking the pair in the key list kills the connection at once, with no waiting for an expiry. The pair covers a single project: neighbouring projects of the same organisation stay invisible, and each one needs its own pair. Long periods are fetched page by page at a deliberately modest request rate, so one report does not eat the project quota. When there is no data for the period, the agent says so directly instead of drawing an empty chart.