Abstract
Recurring personal work is stateful: a user should be able to return tomorrow to visible, editable task state an assistant created today, yet chat answers and static generated interfaces cannot support this, and no existing benchmark evaluates whether an agent can generate the surface through which that future user–agent work will occur. We introduce LiveGUI-Bench, a behavioral benchmark for agent-native apps — interactive applications whose interface, state, and behavior are generated and maintained by an agent — evaluated as Agent-as-GUI workflow surfaces that must actually drive the real services they name, grounded in the executable application APIs of AppWorld. Given an end-user intention and context, a system must generate a surface that satisfies five executable contracts across an open–submit–revise–reopen episode sequence: visible task state, action routing into an app-scoped agent workflow, real backend-API invocation, grounded write-back of returned entities into user-visible state, and persistence across reopening. LiveGUI-Bench contains 191 cases over eleven AppWorld app domains, with 764 episodes and 2,172 deterministic assertions checked against execution traces, API-call records, and application-database diffs — every score is read from executed behavior. The primary metric, Full Workflow Resolution (FWR), credits a case only when every contract holds in every episode, and a scripted control suite quantifies how assertion-level scores overstate workflow competence (up to an Assertion Pass Rate of 0.82 with FWR = 0). Frontier agentic systems resolve 5–15% of public workflows (gpt-5.4 0.05, claude-opus-4.7 0.15): the failure localizes sharply — routing (0.92–0.99) and persistence (0.72–0.75) are largely solved, but only 0.16–0.23 of required API invocations actually happen: real-API grounding is the dominant bottleneck.
The Task: Agent-Native Apps, Made Executable
On Monday, a user asks an assistant to plan outfits for a rainy week. On Tuesday, the forecast changes and the user revises the plan. On Wednesday, they reopen the plan before leaving home. A chat answer is one-shot. A generated mockup looks like a planner, but its buttons dispatch to nothing. Hidden agent memory retains the facts but gives the user no visible object to inspect, edit, or trust. A GUI agent can operate an existing app, but cannot create the missing workspace itself.
Agent-native app — an interactive application whose interface, state, and behavior are generated and maintained by an agent. Its evaluable form is the Agent-as-GUI workflow surface: a generated interface the agent maintains as the user's visible, operable, recoverable task state.
Given an intention x and context c, a system produces generate(x, c) → surface | failure. The evaluator never grades the serialized package directly — it renders the surface and executes a reserved episode sequence against it, through user-level actions only (fill, click, observe):
Episodes mirror how a real user returns to a workspace over days; the reopen episode runs after full teardown.
Five executable contracts
Visible task state
Opening the surface renders task-relevant state and controls. Typical failure: blank shell, transcript answer.
Action routing
Events dispatch into the app-scoped agent workflow the surface declares. Typical failure: static controls, dead handlers.
Real-API grounding
The named AppWorld API is invoked and the application database changes — naming an API is not calling it. Typical failure: API named but never called.
Grounded visible update
The concrete returned entity (a playlist title, an amount, a file path) becomes user-visible state. Typical failure: hidden backend memory, generic echo.
Reopen persistence
After closing and reopening, the entity is still visible and editable. Typical failure: state reset, route loss.
Full Workflow Resolution (FWR)
A case is resolved only if every contract holds in every episode — no partial credit at the case level. Assertion Pass Rate (APR) is reported only as a diagnostic.
Dataset
LiveGUI-Bench targets recurring personal workflows where a persistent visible surface is more appropriate than a one-off answer: a weekly commute-playlist builder, a shared-expense settle-up tracker, a download-folder sorter. The 191 cases span nine single-app domains over AppWorld services (spotify, amazon, gmail, phone, venmo, splitwise, todoist, simple_note, file_system) and two cross-app domains (amazon+venmo, gmail+todoist). Splits are stratified within every domain: 37 dev / 75 public-test / 79 sealed-test; the sealed split is withheld for hosted evaluation.
Each case carries on average 11 deterministic assertions whose expected values are concrete case entities — 376 assert a named API invocation, 268 assert an application-database change, 382 assert a concrete entity in visible state. Intentions and assertions were authored by multi-agent pipelines and adversarially reviewed so every grounding value is a concrete entity, never a generic word.
{
"id": "spotify-weekly-commute-playlist__submit",
"kind": "submit",
"userMessage": "Vibe is upbeat indie, seed artists Phoenix and
Two Door Cinema Club, about 20 songs for Monday Commute.",
"assertions": [
{ "type": "route_enters_orchestrator", "value": true },
{ "type": "appworld_api_called", "value": "create_playlist" },
{ "type": "appworld_db_changed", "value": "spotify" },
{ "type": "visible_result_grounded", "value": "Monday Commute" }
]
}Illustrative episode schema (not a sealed-test case). Because AppWorld apps are real executable services with real databases, “the workflow created the Monday Commute playlist” is a checkable fact, not a rubric judgment.
Evaluation Harness
All systems are evaluated through a single adapter boundary: the harness supplies intention and context and receives a package or an explicit failure. The harness never repairs missing routes, missing state, or unrenderable surfaces — a broken package fails exactly as it would for a user. Routing is read from the runtime's agent trace; API invocation from per-action tool-call records; database change from AppWorld snapshots diffed before and after each call; persistence from a fresh state query after teardown.
/evaluator/gui) performs deterministic admission-quality checks on the package and its rendered A2UI model; the behavior evaluator (/evaluator/run) drives long-horizon exploratory sessions one user-level action at a time, ending in an LLM judgment over the full trajectory. All reported scores come from the deterministic runner; on a stratified 15-case probe the two evaluators agreed on 14 of 15 cases — the single disagreement was the behavior evaluator catching a spurious deterministic flake.
report.json / run-summary.json artifacts by checked-in scripts.Results
Public-split results (75 cases). The upper block is a scripted control suite — each control is constructed to pass some contracts and fail a targeted one, calibrating what a single assertion-level score would overcount. All columns are pass rates in [0,1]; higher is better.
| System | Cases | FWR | APR | Route | API | DB | Grounded | Persist |
|---|---|---|---|---|---|---|---|---|
| Direct answer | 75 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| Static shell | 75 | 0.00 | 0.18 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| Route-only | 75 | 0.00 | 0.44 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| Hidden state | 75 | 0.00 | 0.82 | 1.00 | 1.00 | 1.00 | 0.00 | 1.00 |
| Schema-first | 75 | 0.00 | 0.35 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 |
| Multi-agent control | 75 | 0.00 | 0.79 | 1.00 | 1.00 | 0.00 | 1.00 | 0.00 |
livegui-agent (gpt-5.4) | 75 | 0.05 | 0.67 | 0.92 | 0.16 | 0.18 | 0.73 | 0.72 |
livegui-agent (claude-opus-4.7) | 75 | 0.15 | 0.72 | 0.99 | 0.23 | 0.27 | 0.80 | 0.75 |
no-grounding ablation (gpt-5.4) | 25 | 0.00 | 0.67 | 1.00 | 0.00 | 0.00 | 0.84 | 0.76 |
FWR = fraction of cases whose assertions all pass in every episode (primary metric); APR = fraction of individual assertions passing (diagnostic); Route = GUI events entering the app-scoped agent workflow; API = named AppWorld API actually invoked; DB = application database mutated; Grounded = concrete returned entity visible; Persist = entity survives reopen. Compiled from archived report.json artifacts.
Key findings
APR–FWR gap. The hidden-state control reaches APR 0.82 with FWR = 0 — it performs real API work, changes real databases, and persists state, yet never shows the user a result. The gap persists for real systems (APR 0.67–0.72 vs. FWR 0.05–0.15): a single assertion-level score overstates workflow competence by construction.
Contract localization. Both frontier systems route almost perfectly (0.92–0.99), persist state well (0.72–0.75), and echo requested entities into visible results (0.73–0.80) — but only 0.16–0.23 of required API invocations actually happen. Grounding GUI actions in authenticated, correctly-parameterized API execution is the dominant unsolved capability.
Model gap. claude-opus-4.7 resolves nearly three times more workflows than gpt-5.4 (0.15 vs. 0.05), with the advantage concentrated in the API-invocation contract — the benchmark discriminates tool-grounding ability, not surface fluency.
Per-domain difficulty. Single-user CRUD domains are tractable (spotify 5/8 resolved by claude-opus-4.7), while multi-party ledger domains (venmo, splitwise, todoist collaboration) resolve 0 for both models — workflows whose APIs must coordinate other principals' state remain fully open.
Behavior Trajectories
The online behavior evaluator archives per-step snapshots of the rendered surface, which the harness renders into trajectory frames — the user's-side view of the same contracts the deterministic assertions check.
create_product_review API, and writes the returned entity (review id 15968) into the visible result — a grounded, passing step. Right: a Venmo recurring-rent surface with equally structured slots, whose action degenerates to echoing the app title — no grounded entity ever reaches the surface. Same generated-app machinery; real-API grounding is the discriminator.BibTeX
@article{liveguibench2026,
title = {LiveGUI-Bench: A Behavioral Benchmark for
Agent-as-GUI Workflow Surfaces},
author = {Anonymous Authors},
note = {Paper under double-blind review},
year = {2026},
url = {https://liveguibench.pages.dev/}
}