Skip to content

docs: link AMB submission to provider PR #24 #213

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jack-arturo merged 1 commit into from
Jul 4, 2026
Merged

docs: link AMB submission to provider PR #24 #213

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ AutoMem **0.16.0** was run through the neutral [Agent Memory Benchmark](https://
- **Efficiency is architectural:** AutoMem feeds the answerer **~2.6–4.8k context tokens** at every scale (mean), versus 17–27k for the board leader on BEAM.
- **The honest other half:** on conversational Core-3, AutoMem **trails** the AMB leader Hindsight — locomo 85.1% vs 92%, longmemeval 74.4% vs 94.6%, personamem 76.1% vs 86.6%. Pick AutoMem for large-context scaling and efficiency, not for top-of-board verbatim recall.

Outputs are committed and public, and [`AUTOMEM_REPRODUCE.md`](https://automem.ai/benchmarks) gives one command per split so you can **run it yourself**. AutoMem is **submitted to the neutral board (provider PR under review)** — not yet live on the public leaderboard. Full head-to-head numbers live at [automem.ai/benchmarks](https://automem.ai/benchmarks).
Outputs are committed and public, and [`AUTOMEM_REPRODUCE.md`](https://automem.ai/benchmarks) gives one command per split so you can **run it yourself**. AutoMem is **submitted to the neutral board ([provider PR #24](https://github.com/vectorize-io/agent-memory-benchmark/pull/24), under review)** — not yet live on the public leaderboard. Full head-to-head numbers live at [automem.ai/benchmarks](https://automem.ai/benchmarks).

## Should you use AutoMem?

Expand Down
6 changes: 3 additions & 3 deletions benchmarks/EXPERIMENT_LOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,9 +57,9 @@ own harness (directional, not head-to-head) — only BEAM is the clean compariso
Hindsight is the apples-to-apples Core-3 yardstick (same AMB harness).

**Status — submitted, not official.** AMB results are submitted to the neutral
board; the vectorize-io provider PR is **under review**. Frame as "submitted, PR
under review" and "run it yourself" — never "live/official on the leaderboard"
until the PR merges. Outputs are committed and public; `AUTOMEM_REPRODUCE.md`
board; the vectorize-io provider PR ([#24](https://github.com/vectorize-io/agent-memory-benchmark/pull/24))
is **under review**. Frame as "submitted, PR under review" and "run it yourself"
— never "live/official on the leaderboard" until the PR merges. Outputs are committed and public; `AUTOMEM_REPRODUCE.md`
gives one command per split, and the public GHCR image
(`ghcr.io/verygoodplugins/automem:amb-v1`) self-spins the full stack with no API
keys. **No cross-system latency/speed claims** — AMB timings are AutoMem's own
Expand Down
Loading