Abstract
Recurring personal work is stateful and accretes across days: a user should be able to return tomorrow to visible, editable task state an assistant created today. This calls for an agent-native app — an interactive application whose interface, state, and behavior are generated and maintained by an agent — yet each dominant interaction form fails it differently: chat is one-shot, a static mockup routes nowhere, hidden memory exposes no object to trust, and a GUI agent cannot create the missing workspace. We make the concept both buildable and falsifiable over one shared spine of five executable contracts: visible task state, action routing, real-API grounding, grounded visible update, and reopen persistence. GUIOS makes it buildable: a whiteboard runtime that turns one natural-language sentence into a validated spec package rendered as a live interactive app whose controls dispatch to a unified agent backend making real API calls. LiveGUI-Bench makes it falsifiable: 500 cases over nine AppWorld single-app domains plus the multi_tool and shared_world families, with 2,386 episodes and 10,480 deterministic assertions checked against execution traces, API-call records, and AppWorld database diffs. The primary metric, Full Workflow Resolution (FWR), credits a case only when every contract holds in every episode. Results from the 500-case OpenRouter rerun are pending; the prior 319-case pilot found that routing and persistence are largely solved while real-API grounding is the dominant bottleneck.
The Task: Agent-Native Apps, Made Executable
On Monday, a user asks an assistant to plan outfits for a rainy week. On Tuesday, the forecast changes and the user revises the plan. On Wednesday, they reopen the plan before leaving home. A chat answer is one-shot. A generated mockup looks like a planner, but its buttons dispatch to nothing. Hidden agent memory retains the facts but gives the user no visible object to inspect, edit, or trust. A GUI agent can operate an existing app, but cannot create the missing workspace itself.
Agent-native app — an interactive application whose interface, state, and behavior are generated and maintained by an agent. Its evaluable form is the Agent-as-GUI workflow surface: a generated interface the agent maintains as the user's visible, operable, recoverable task state.
Given an intention x and context c, a system produces generate(x, c) → surface | failure. The evaluator never grades the serialized package directly — it renders the surface and executes a reserved episode sequence against it, through user-level actions only (fill, click, observe):
Episodes mirror how a real user returns to a workspace over days; the reopen episode runs after full teardown.
Five executable contracts
Visible task state
Opening the surface renders task-relevant state and controls. Typical failure: blank shell, transcript answer.
Action routing
Events dispatch into the app-scoped agent workflow the surface declares. Typical failure: static controls, dead handlers.
Real-API grounding
The named AppWorld API is invoked and the application database changes — naming an API is not calling it. Typical failure: API named but never called.
Grounded visible update
The concrete returned entity (a playlist title, an amount, a file path) becomes user-visible state. Typical failure: hidden backend memory, generic echo.
Reopen persistence
After closing and reopening, the entity is still visible and editable. Typical failure: state reset, route loss.
Full Workflow Resolution (FWR)
A case is resolved only if every contract holds in every episode — no partial credit at the case level. Assertion Pass Rate (APR) is reported only as a diagnostic.
Dataset
LiveGUI-Bench targets recurring personal workflows where a persistent visible surface is more appropriate than a one-off answer: a weekly commute-playlist builder, a shared-expense settle-up tracker, a download-folder sorter. The 500 cases span nine single-app AppWorld domains (spotify, amazon, gmail, phone, venmo, splitwise, todoist, simple_note, file_system) plus two compositional families: multi_tool (one app spanning ≥3 services) and shared_world (two apps sharing one world, the second discovering state the first wrote). Splits are stratified within every domain and family: 147 dev / 172 public-test / 181 sealed-test; the sealed split is withheld for hosted evaluation.
Each case carries on average 11 deterministic assertions whose expected values are concrete case entities — 376 assert a named API invocation, 268 assert an application-database change, 382 assert a concrete entity in visible state. Intentions and assertions were authored by multi-agent pipelines and adversarially reviewed so every grounding value is a concrete entity, never a generic word. Two live-world soundness audits then repaired 187 and 61 cases via a tracked override layer — the second asking whether a correct app could still fail (underivable expected values, widget-prescriptive wording) — growing the fixture library to 118 files of seedable world state and quarantining 7 unsatisfiable cases; the audited suite ships as v0.2.0 (219 cases with multi_tool and shared_world compositional extensions).
{
"id": "spotify-weekly-commute-playlist__submit",
"kind": "submit",
"userMessage": "Vibe is upbeat indie, seed artists Phoenix and
Two Door Cinema Club, about 20 songs for Monday Commute.",
"assertions": [
{ "type": "route_enters_orchestrator", "value": true },
{ "type": "appworld_api_called", "value": "create_playlist" },
{ "type": "appworld_db_changed", "value": "spotify" },
{ "type": "visible_result_grounded", "value": "Monday Commute" }
]
}Illustrative episode schema (not a sealed-test case). Because AppWorld apps are real executable services with real databases, “the workflow created the Monday Commute playlist” is a checkable fact, not a rubric judgment.
Evaluation Harness
All systems are evaluated through a single adapter boundary: the harness supplies intention and context and receives a package or an explicit failure. The harness never repairs missing routes, missing state, or unrenderable surfaces — a broken package fails exactly as it would for a user. Routing is read from the runtime's agent trace; API invocation from per-action tool-call records; database change from AppWorld snapshots diffed before and after each call; persistence from a fresh state query after teardown.
/evaluator/gui) performs deterministic admission-quality checks on the package and its rendered A2UI model; the behavior evaluator (/evaluator/run) drives long-horizon exploratory sessions one user-level action at a time, ending in an LLM judgment over the full trajectory. All reported scores come from the deterministic runner; on a stratified 15-case probe the two evaluators agreed on 14 of 15 cases — the single disagreement was the behavior evaluator catching a spurious deterministic flake.
report.json / run-summary.json artifacts by checked-in scripts.Results
The 500-case suite is new, and every headline number will be compiled from archived report.json artifacts of the OpenRouter rerun — never transcribed by hand. Until those artifacts are archived, this section reports the measurement plan, not numbers.
What will be measured
Six scripted controls. Direct answer, static shell, route-only, hidden state, schema-first, and a multi-agent control — each constructed to pass some contracts and fail a targeted one, calibrating how far a single assertion-level score (APR) overstates workflow competence relative to FWR.
The GUIOS baseline on frontier models. GUIOS — the reference agent-native runtime and the benchmark's own baseline generator — runs end-to-end on multiple backing models behind one OpenRouter-compatible endpoint, over the 172-case public-test split; the sealed split is withheld for hosted evaluation.
A no-grounding ablation. The identical generator with the API-invoking executor disabled passes zero API and database assertions by construction, bounding exactly how much of the remaining score is surface behavior.
Contract localization. Per-contract pass rates (routing, API invocation, database change, grounded visibility, persistence) localize failures. The 319-case pilot found routing and persistence largely solved while only a small fraction of required API invocations actually happened — real-API grounding is expected to remain the dominant bottleneck, now stress-tested by the shared_world family where a second app must discover state the first really wrote.
Pilot observations (319-case suite, archived artifacts) are described qualitatively in the paper's case-study appendix; they are not comparable to the 500-case suite and are not shown here as current results.
Behavior Trajectories
The online behavior evaluator archives per-step snapshots of the rendered surface, which the harness renders into trajectory frames — the user's-side view of the same contracts the deterministic assertions check.
create_product_review API, and writes the returned entity (review id 15968) into the visible result — a grounded, passing step. Right: a Venmo recurring-rent surface with equally structured slots, whose action degenerates to echoing the app title — no grounded entity ever reaches the surface. Same generated-app machinery; real-API grounding is the discriminator.BibTeX
@article{liveguibench2026,
title = {LiveGUI-Bench: A Behavioral Benchmark for
Agent-as-GUI Workflow Surfaces},
author = {Anonymous Authors},
note = {Paper under double-blind review},
year = {2026},
url = {https://liveguibench.pages.dev/}
}