DREAMBENCH—SWE

The final task needs something only the first session saw.

Can a fact survive
the distance?

DreamBench-SWE turns a fragile memory into an executable software test.

Follow one fact

The eight-part evidence relay

  1. THE NEED

    One fact. Three sessions. No second chance.

    The final task needs something only the first session saw.

  2. SESSION 2 · THE DISTANCE

    The first session closes. New work begins.

    Session 2 moves on without the original view. By Session 3, only a saved record can carry the fact forward.

  3. THE RISK

    Memory gets crowded.

    New notes arrive. Some are useful. One looks right—but carries the wrong fact.

  4. THE TRAIL

    Keep the trail.

    A remembered note is safer when it can still point back to what really happened.

  5. THE CHOICE

    Choose for this task.

    The right fact passes through. The convincing wrong one stays behind.

  6. THE TEST

    Remembering is not solving.

    The fact reaches the final session. The agent still has to make the code work.

  7. THE MEASUREMENT

    Now stop imagining.

    DreamBench-SWE tests 60 different memory challenges, each rerun three controlled times.

  8. THE RESULT

    No winner. The test survives.

    Six more passes did not establish superiority—and did not establish equivalence.

THE LITERAL RESULT

A difference
is not a verdict.

An organized memory system that keeps both structured notes and verbatim evidence solved 95 of 180 tasks. A simple verbatim memory solved 89 of 180.

The six-task gap could easily arise by chance in this experiment. The exact paired test returned p = 0.518419; after correcting for multiple questions, p = 1.0.

Not proof of superiority.
Not proof of equivalence.

What happened in all 180 head-to-head checks
Outcome for the same challenge and rerunChecks
Both memory methods solved the task74
Neither memory method solved the task70
Only organized + verbatim memory solved it21
Only simple verbatim memory solved it15
What exactly was compared? +

Every challenge unfolds across three work sessions. The last session asks the agent to complete a real software task, but one necessary fact appeared only earlier and cannot be guessed from the final prompt or repository. We give the same challenge and rerun to two memory methods, then use code—not a human impression—to check whether each final task actually works.

Why isn’t six more wins a winner? +

Small gaps can appear just from which challenges and reruns happened to be included. Here, the exact paired p-value was 0.518419, and it became 1.0 after correcting for the several planned questions in the study. Those numbers do not give us enough evidence to call organized + verbatim memory better. They also do not prove that the methods are equal.

Why were two challenge types set aside? +

Two groups were meant to test whether memory avoids a false lesson and knows when to stay silent. But an agent with no memory already solved every case in those groups—12 of 12 in one and 6 of 6 in the other. With no room left for memory to improve the outcome, this experiment cannot support claims about those two abilities.

WHAT SURVIVES

The benchmark
refused to bluff.

DreamBench-SWE contributes a controlled, executable way to expose multi-session software-memory failures.

Its transparent null result and explicit limits are part of that contribution—not an apology for it.

Plain-language glossary

Memory challenge paper term: trap
A three-session software scenario built so the final task needs one specific fact from an earlier session.
Controlled rerun paper term: seed
A fresh repeat of the same challenge using controlled randomness, so a lucky or unlucky single run does not decide the result.
Head-to-head check paper term: paired outcome
The same challenge and rerun attempted once with each memory method. Pairing keeps the comparison fair.
Organized + verbatim memory paper term: Hybrid
The reference method: structured, source-linked notes plus the original verbatim evidence they came from.
Simple verbatim memory paper term: B5
A strong comparison method that stores deterministic event notes close to what originally happened, without the reference method’s full organization pipeline.
Executable check
A programmatic test of whether the final code works. It scores behavior rather than how convincing an answer sounds.
p-value
A measure of how surprising a gap this large would be if neither memory method had a real advantage. Larger values mean the observed gap is easier to explain by chance.
Exact paired test
A statistical test that compares the two methods on the very same challenges and reruns instead of treating their results as unrelated.
Multiple-question correction paper term: Holm adjustment
A safeguard that raises the bar when a study tests several planned claims, reducing the risk of announcing a chance result as a discovery.
No-memory check paper term: headroom
A sanity check that asks whether the task is already easy without memory. If no-memory performance is perfect, that task cannot show a memory benefit.
Inconclusive does not mean equal
“No winner” means this experiment did not separate the methods. It does not prove that their true performance is identical.

READ · VERIFY · BUILD ON IT

The work is open.

PUBLIC CODE + BENCHMARKExplore the GitHub repository FULL PAPER · PRE-ARXIVRead the full paper on Google Drive