The Agent-Native App: Building It (GUIOS) and Measuring It (LiveUIBench)

Abstract

The Task: Agent-Native Apps, Made Executable

On Monday, a user asks an assistant to plan outfits for a rainy week. On Tuesday, the forecast changes and the user revises the plan. On Wednesday, they reopen the plan before leaving home. A chat answer is one-shot. A generated mockup looks like a planner, but its buttons dispatch to nothing. Hidden agent memory retains the facts but gives the user no visible object to inspect, edit, or trust. A GUI agent can operate an existing app, but cannot create the missing workspace itself.

Given an intention x and context c, a system produces generate(x, c) → surface | failure. The evaluator never grades the serialized package directly — it renders the surface and executes a reserved episode sequence against it, through user-level actions only (fill, click, observe):

Episodes mirror how a real user returns to a workspace over days; the reopen episode runs after full teardown.

Five executable contracts

Dataset

LiveUIBench targets recurring personal workflows where a persistent visible surface is more appropriate than a one-off answer: a weekly commute-playlist builder, a shared-expense settle-up tracker, a download-folder sorter. The 500 cases span nine single-app AppWorld domains (spotify, amazon, gmail, phone, venmo, splitwise, todoist, simple_note, file_system) plus two compositional families: multi_tool (one app spanning ≥3 services) and shared_world (two apps sharing one world, the second discovering state the first wrote). Splits are stratified within every domain and family: 147 dev / 172 public-test / 181 sealed-test; the sealed split is withheld for hosted evaluation.

Each case carries on average 21 deterministic assertions whose expected values are concrete case entities — 2,115 assert a named API invocation, 1,418 assert an application-database change, 2,805 assert a concrete entity in visible state. Intentions and assertions were authored by multi-agent pipelines and adversarially reviewed so every grounding value is a concrete entity, never a generic word. Two live-world soundness audits then repaired 187 and 61 cases via a tracked override layer — the second asking whether a correct app could still fail (underivable expected values, widget-prescriptive wording) — growing the fixture library of seedable world state and quarantining unsatisfiable cases; the audited suite then grew to the 500-case v0.2.0 release with the multi_tool and shared_world compositional families.

{
  "id": "spotify-weekly-commute-playlist__submit",
  "kind": "submit",
  "userMessage": "Vibe is upbeat indie, seed artists Phoenix and
      Two Door Cinema Club, about 20 songs for Monday Commute.",
  "assertions": [
    { "type": "route_enters_orchestrator", "value": true },
    { "type": "appworld_api_called",      "value": "create_playlist" },
    { "type": "appworld_db_changed",      "value": "spotify" },
    { "type": "visible_result_grounded",  "value": "Monday Commute" }
  ]
}

Illustrative episode schema (not a sealed-test case). Because AppWorld apps are real executable services with real databases, “the workflow created the Monday Commute playlist” is a checkable fact, not a rubric judgment.

Evaluation Harness

All systems are evaluated through a single adapter boundary: the harness supplies intention and context and receives a package or an explicit failure. The harness never repairs missing routes, missing state, or unrenderable surfaces — a broken package fails exactly as it would for a user. Routing is read from the runtime's agent trace; API invocation from per-action tool-call records; database change from AppWorld snapshots diffed before and after each call; persistence from a fresh state query after teardown.

The GUIOS Baseline

To show the agent-native app is buildable — and to give the benchmark its strongest available baseline — we build GUIOS, a whiteboard runtime engineered contract-by-contract to satisfy the same five-contract spine the benchmark checks. GUIOS enters the benchmark through the same adapter boundary as any other generator: the system we build is the system we then try to refute, so any contract that still fails under it marks a genuine capability frontier rather than a weak baseline.

Results

The 500-case suite is new, and every headline number will be compiled from archived report.json artifacts of the OpenRouter rerun — never transcribed by hand. Until those artifacts are archived, this section reports the measurement plan, not numbers.

What will be measured

Pilot observations (319-case suite, archived artifacts) are described qualitatively in the paper's case-study appendix; they are not comparable to the 500-case suite and are not shown here as current results.

Behavior Trajectories

The online behavior evaluator archives per-step snapshots of the rendered surface, which the harness renders into trajectory frames — the user's-side view of the same contracts the deterministic assertions check. The frames below come from a five-session probe collected with GPT-5.5 (openai/gpt-5.5 via OpenRouter) driving the full generate–act–reopen loop; the deterministic runner and the trajectory judge agree on 4 of 5 verdicts.

BibTeX

@article{liveuibench2026,
  title   = {LiveUIBench: A Behavioral Benchmark for
             Agent-as-GUI Workflow Surfaces},
  author  = {Anonymous Authors},
  note    = {Paper under double-blind review},
  year    = {2026},
  url     = {https://liveuibench.pages.dev/}
}