Did Your OpenClaw Agent Actually Finish? A ThinkingBox-Inspired Testing Guide

An agent says “done.” The API returned success. But did the right record change—and did everything else stay untouched?

That is the operational question behind ThinkingBox, Microsoft's stateful agent-evaluation framework. It is also a useful question for anyone running OpenClaw workflows that edit tickets, manage calendars, or publish content through MCP tools.

The practical takeaway: test the resulting state, not just the agent's final message, and repeat the test from a clean starting point.

Why this matters now

OpenClaw's published 2026.10.10-beta.1 release notes highlight restart-safe replies and recovery of queued work and child results. Those are pre-release claims, not a guarantee about every installed version. But they make an important distinction worth testing: recovering an agent run is not the same thing as proving its external side effects were correct.

Separately, Microsoft's ThinkingBox explanation and the ThinkingBox research paper describe 507 business-workflow tasks evaluated with 20 attempts per task. Both report that the strongest evaluated model achieved 65.36% pass@1, but only 25.25% all-20 success, written as pass^20.

These are results for the authors' benchmark and configuration—not measured OpenClaw performance, a ranking of today's models, or a prediction about your automation. The two sources describe the same research, rather than independent replications. Their useful lesson is methodological: one successful demonstration does not establish repeatable completion.

1. Turn a request into a completion contract

Start with one narrow workflow in a disposable test system. For example:

Add the label needs-review to ticket TEST-104, leave its owner and status unchanged, and report the ticket ID.

Write the expected outcome before running the agent:

• The target ticket contains the requested label exactly once.
• Its owner and status match their initial values.
• No other ticket changed.
• No notification was sent unless the test explicitly authorized it.
• The final response identifies the same ticket and accurately describes the result.

The negative checks matter. An agent that updates the correct ticket and accidentally modifies a second one has not completed the task correctly.

Keep authorization separate from outcome. A correct final record does not excuse a write performed without required approval. Capture approval and access-control evidence alongside state checks.

2. Give the evaluator a separate view

ThinkingBox separates the agent's permitted tools from the harness that initializes the environment and inspects effects. Apply that pattern to your own tests: the agent gets only the tools needed for the task; a separate evaluator reads the authoritative records.

A read-only API check can verify a particular record. Detecting changes elsewhere usually requires a broader fixture snapshot or audit log that the agent cannot alter. Do not grant the agent production administrator access just to make evaluation easier.

Here is illustrative Python—not an OpenClaw configuration file or a ThinkingBox API example:

def check_ticket_update(before, after, events, target="TEST-104"):
    expected_labels = set(before[target]["labels"]) | {"needs-review"}
    assert set(after[target]["labels"]) == expected_labels
    assert after[target]["labels"].count("needs-review") == 1
    assert after[target]["owner"] == before[target]["owner"]
    assert after[target]["status"] == before[target]["status"]
    assert set(after) == set(before)
    for ticket_id in before:
        if ticket_id != target:
            assert after[ticket_id] == before[ticket_id]
    assert not any(e["kind"] == "notification_sent" for e in events)

Adapt the assertions to your actual schema and requirements. This small example is not a complete audit: it does not check every field on the target ticket or enforce approval policy. Add those checks where relevant.

For eventual-consistency APIs, allow a bounded observation window before declaring failure. Preserve an “unknown” outcome when the backend cannot be read; uncertainty is not proof that the write failed.

3. Repeat from a clean baseline

Run each case multiple times with a fresh conversation and reset fixtures. Keep the model, instructions, tool versions, permissions, and starting data fixed while measuring a baseline.

Record three simple numbers:

• Successful attempts divided by total attempts.
• Cases that succeeded at least once.
• Cases that succeeded on every attempt.

Do not label a single successful run “reliable.” Conversely, a small all-pass sample is not a statistical guarantee. Use it as a regression signal, then expand the cases and repetitions according to the consequences of failure.

Start with a handful of representative workflows rather than an expensive, unfocused benchmark. Keep evaluation results outside the agent's writable workspace so a later run cannot accidentally contaminate its own evidence.

4. Test uncertainty, not just the happy path

After the baseline, introduce controlled failures in the test environment:

• Write accepted, response lost: does recovery inspect state before repeating a mutation?
• Expired credentials: does the agent report the blocker without claiming completion?
• Missing user decision: does it pause rather than guess a consequential value?
• Interrupted run: after an authorized staging restart, does it reconcile existing effects?
• Ambiguous record match: does it avoid editing a similarly named item?

Use a stable operation identifier or idempotency key when the destination API supports one. Do not assume MCP itself provides exactly-once business operations. If an API offers neither idempotency nor a reliable way to reconcile an uncertain write, automatic replay may be inappropriate.

Never run destructive failure tests against live customer data. Use synthetic records and a test notification sink, with limits on execution time and spend.

5. Make the results a rollout gate

Keep a compact evidence record for each attempt: test case, configuration versions, final-state assertions, unauthorized effects, duration, cost, and the agent's reported outcome. Redact credentials and personal data from traces.

Repeat the suite when you change a model, an MCP server, a workflow instruction, or the runtime. Treat any unauthorized write as a release blocker; investigate functional regressions before expanding unattended operation.

For OpenClaw users, this is a testing pattern you can apply around existing workflows—not a claim that Clawly bundles ThinkingBox or has enabled a particular beta release. Managed hosting can handle infrastructure, but your workflow still needs its own definition of correct completion.

A trustworthy “done” is backed by the right state, the absence of forbidden effects, and repeated evidence—not a convincing sentence.

Sources

• OpenClaw release notes, including the 2026.10.10-beta.1 pre-release notes; checked October 16, 2026.
• Microsoft: How we built ThinkingBox to measure whether agents finish the job.
• ThinkingBox paper: One Success Isn't Reliability.