Jerry Liu

Introducing ExtractBenchPro: the most challenging benchmark for real-world document extraction. We see a massive long tail of document complexity in terms of use cases brought on from our customers, that are completely unsolved from a price-performance standpoint. These include trial balances, well logs, canvass binders, site plan reviews, duty reconciliations, loss runs, and more. Most existing extraction benchmarks are saturated by the frontier models (our cheapest modes now score 93%+ on ExtractBench). So there is enormous economic value in building a complex extraction benchmark that reflects this long-tail of knowledge work. We spent a long time curating the most complex elements that we saw from real world extraction use cases: ✅ Document conventions: containing legends or rules where the rest of the document should be interpreted ✅ Density of internal jargon: specific definitions that must be cross-referenced within a doc ✅ Aggregation and reasoning: beyond returning exact values on a page, infer values derived from aggregating multiple sources of information ✅ Drawn relations: complex diagrams like procedures and circuits ✅ Charts inside documents: read precise points within a chart ✅ Long chart sheets: some data series like well logs cram all this data into an extremely long page Within the industries where document extraction is relevant, even small mistakes can have enormous negative impacts. Therefore our headline evaluation metric is based on perfect documents: the proportion of docs with perfect (100%) value f1 score. (Besides this we also measure value f1 and grounded f1) It’s an extremely hard task for even the latest frontier models to solve, topping out at ~58% or so in Value F1 score for Codex with Astra - which is quite good but extremely expensive (almost $1 per page!) Our LlamaParse agentic/agentic plus modes are 20-40% and cover the left side of the Pareto frontier, at price ranges that are feasible for full production use. ExtractBenchPro is available today as a website/technical blog, and also on Github and HuggingFace. Check it out! Blog: llamaindex.ai/blog/introducing-extractbench-pro Website: extractbenchpro.ai/ Hugging Face: huggingface.co/datasets/llamaindex/ExtractBenchPro Github (is ExtractBench): github.com/run-llama/ExtractBench

Introducing ExtractBenchPro: the most challenging benchmark for real-world document extraction. We see a massive long tail of document complexity in terms of use cases brought on from our customers, that are completely unsolved from a price-performance standpoint. These include trial balances, well logs, canvass binders, site plan reviews, duty reconciliations, loss runs, and more. Most existing extraction benchmarks are saturated by the frontier models (our cheapest modes now score 93%+ on ExtractBench). So there is enormous economic value in building a complex extraction benchmark that reflects this long-tail of knowledge work. We spent a long time curating the most complex elements that we saw from real world extraction use cases: ✅ Document conventions: containing legends or rules where the rest of the document should be interpreted ✅ Density of internal jargon: specific definitions that must be cross-referenced within a doc ✅ Aggregation and reasoning: beyond returning exact values on a page, infer values derived from aggregating multiple sources of information ✅ Drawn relations: complex diagrams like procedures and circuits ✅ Charts inside documents: read precise points within a chart ✅ Long chart sheets: some data series like well logs cram all this data into an extremely long page Within the industries where document extraction is relevant, even small mistakes can have enormous negative impacts. Therefore our headline evaluation metric is based on perfect documents: the proportion of docs with perfect (100%) value f1 score. (Besides this we also measure value f1 and grounded f1) It’s an extremely hard task for even the latest frontier models to solve, topping out at ~58% or so in Value F1 score for Codex with Astra - which is quite good but extremely expensive (almost $1 per page!) Our LlamaParse agentic/agentic plus modes are 20-40% and cover the left side of the Pareto frontier, at price ranges that are feasible for full production use. ExtractBenchPro is available today as a website/technical blog, and also on Github and HuggingFace. Check it out! Blog: llamaindex.ai/blog/introducing-extractbench-pro Website: extractbenchpro.ai/ Hugging Face: huggingface.co/datasets/llamaindex/ExtractBenchPro Github (is ExtractBench): github.com/run-llama/ExtractBench

From κ = 2⁻¹⁸² to 0.001734815770 :) Integer Multiplication community auto-research highlights since 10/15 (+1.23%): 1) The big headline of the day is that a free model, @opencode's Space Bunny Max, just set the record for the first time, 4 times in a row! As far as I know, this is the first time a free model has submitted a best bound for Integer Multiplication. Expensive Max subscriptions previously kept some of our best contributors and experts out, but the barrier of entry for frontier auto-research dropped to 0 :) Claude Opus 5.5 helped verify and push the PRs over the line. I still recommend having a frontier model verify, but other contributors can do that. Progress was slowing over the last day (relative to one new κ per minute, which is still wild for math research), but Space Bunny Max just injected fresh ideas that boosted our PR rate back to one per minute. 2) A community Lean proof of the original OpenAI Integer Multiplication paper has been completed! @platypii et. al completed the proof in 6 days, and @0xdoug's repo is about 10–20% verified now. To be clear, most of our work still needs verification but the Lean proofs are catching up :) 3) 41–51% of PRs now mention a prior research-only submission, up from 0–3% before @0xdoug's autoresearch.md prompt. His prompt guides new contributors to specific research tracks rather than incremental updates. We're moving away from "brute force research" towards more coordinated efforts across an army of agents. If you're contributing research PRs, even negative results without a new record, you're meaningfully helping us and I appreciate it! Interesting observations: 1) The results from free models are related to the recent "local minima" solutions in context engineering that have been popping up: Forgetting context to "refresh" by @ShangyinT. Branching into different solutions by @mao_yuzhen. Fork and flush by @DimitrisPapail. Now even free models contribute to escaping local minima by exploring new solutions, since they're trained on different data. There's likely a lot of room for improvement now in context engineering, consensus across different models, and collaboration strategies, which we will test next. 2) Frontier research as a benchmark helped us find a successful context-sharing strategy for parallel agents. Basically, we can rewind frontier research GitHub repos to an earlier branch and benchmark agent harnesses, prompts, context designs, models, etc. With this benchmark, we found that you can distill constantly to local files, then periodically concatenate them instantly into parallel agent sessions to save 10–20% of the time it takes to reach the next breakthrough. 3) There's a Codex reset by EOD PST, so you can copy this prompt to run unused tokens on frontier research :) """ Contribute one useful, independently reviewable result to github.com/CrocSwap/integer-mult-bounds. Read AGENTS.md, autoresearch.md and CONTRIBUTING.md from the current main branch first. Use the repository's task-selection and overlap process unless I assigned a specific task. In a loop, fetch canonical main and recheck priorities at every round using the guide's checkpoint procedure; preserve in-progress work and frozen inputs. Work for one session of about 60 minutes unless I specify another budget, keeping the final quarter for verification and a concise handoff, and report the exact scope, evidence, remaining obligations and predecessor credit. You may post routine claims, questions, findings, handoffs and feedback in Discussions and open a focused PR without asking me again for each action. Do not merge, close others' PRs or change the selected frontier. """ You can also copy this prompt into OpenCode now! Now that anyone can contribute for free to frontier auto-research, repost this and sharing it with friends. 🙏 4) PR stats: 850 PRs from 67 contributors since 10/15, and 21 of those contributors are first-timers. 55 PRs claim a new conditional κ in the title. Codex/OpenAI in 480 PRs (56%), Claude in 147 (17%), the OpenCode harness in 60 (7%), Cursor in 16. Most prolific contributors by PR count: petalas: 164, ywz65: 147, navinor: 49, commdata2338: 44, RossRichardson: 43, Arham-Tahir64: 39, thakrarsagar: 34, VictorLiwentao: 30, tekuuu: 28, DreamingOfClouds: 23, jmbarrancoml: 21, jeremykothe2: 21, bitplane: 13, Gonzalo-Mier: 12, Dugongue: 12, mdumitrean: 11, theon199: 9, jwr404: 9, jamesyc: 9, kumagi: 8, CubeJerry: 8, rohanarun: 6, jeremydiviney: 6, dcharb78: 6, 1337isnot1337: 6, starship-s: 6, AAAmirinu: 6. Conditional κ = 0.001734815770385098563484. Just submitted a new best bound :) github.com/CrocSwap/integer-mult-bounds/pull/3251

Introducing ExtractBenchPro: the most challenging benchmark for real-world document extraction. We see a massive long tail of document complexity in terms of use cases brought on from our customers, that are completely unsolved from a price-performance standpoint. These include trial balances, well logs, canvass binders, site plan reviews, duty reconciliations, loss runs, and more. Most existing extraction benchmarks are saturated by the frontier models (our cheapest modes now score 93%+ on ExtractBench). So there is enormous economic value in building a complex extraction benchmark that reflects this long-tail of knowledge work. We spent a long time curating the most complex elements that we saw from real world extraction use cases: ✅ Document conventions: containing legends or rules where the rest of the document should be interpreted ✅ Density of internal jargon: specific definitions that must be cross-referenced within a doc ✅ Aggregation and reasoning: beyond returning exact values on a page, infer values derived from aggregating multiple sources of information ✅ Drawn relations: complex diagrams like procedures and circuits ✅ Charts inside documents: read precise points within a chart ✅ Long chart sheets: some data series like well logs cram all this data into an extremely long page Within the industries where document extraction is relevant, even small mistakes can have enormous negative impacts. Therefore our headline evaluation metric is based on perfect documents: the proportion of docs with perfect (100%) value f1 score. (Besides this we also measure value f1 and grounded f1) It’s an extremely hard task for even the latest frontier models to solve, topping out at ~58% or so in Value F1 score for Codex with Astra - which is quite good but extremely expensive (almost $1 per page!) Our LlamaParse agentic/agentic plus modes are 20-40% and cover the left side of the Pareto frontier, at price ranges that are feasible for full production use. ExtractBenchPro is available today as a website/technical blog, and also on Github and HuggingFace. Check it out! Blog: llamaindex.ai/blog/introducing-extractbench-pro Website: extractbenchpro.ai/ Hugging Face: huggingface.co/datasets/llamaindex/ExtractBenchPro Github (is ExtractBench): github.com/run-llama/ExtractBench

Introducing overshoot, it will cut your GPU bill in half with zero quality loss, or you don’t pay. Over the past few months, we have forward deployed into AI teams to optimize their inference stacks. They have now cumulatively saved over $5M. Overshoot is software that bakes in our hard-earned inference learnings into a scalable solution. overshoot.ai

Introducing ExtractBenchPro: the most challenging benchmark for real-world document extraction. We see a massive long tail of document complexity in terms of use cases brought on from our customers, that are completely unsolved from a price-performance standpoint. These include trial balances, well logs, canvass binders, site plan reviews, duty reconciliations, loss runs, and more. Most existing extraction benchmarks are saturated by the frontier models (our cheapest modes now score 93%+ on ExtractBench). So there is enormous economic value in building a complex extraction benchmark that reflects this long-tail of knowledge work. We spent a long time curating the most complex elements that we saw from real world extraction use cases: ✅ Document conventions: containing legends or rules where the rest of the document should be interpreted ✅ Density of internal jargon: specific definitions that must be cross-referenced within a doc ✅ Aggregation and reasoning: beyond returning exact values on a page, infer values derived from aggregating multiple sources of information ✅ Drawn relations: complex diagrams like procedures and circuits ✅ Charts inside documents: read precise points within a chart ✅ Long chart sheets: some data series like well logs cram all this data into an extremely long page Within the industries where document extraction is relevant, even small mistakes can have enormous negative impacts. Therefore our headline evaluation metric is based on perfect documents: the proportion of docs with perfect (100%) value f1 score. (Besides this we also measure value f1 and grounded f1) It’s an extremely hard task for even the latest frontier models to solve, topping out at ~58% or so in Value F1 score for Codex with Astra - which is quite good but extremely expensive (almost $1 per page!) Our LlamaParse agentic/agentic plus modes are 20-40% and cover the left side of the Pareto frontier, at price ranges that are feasible for full production use. ExtractBenchPro is available today as a website/technical blog, and also on Github and HuggingFace. Check it out! Blog: llamaindex.ai/blog/introducing-extractbench-pro Website: extractbenchpro.ai/ Hugging Face: huggingface.co/datasets/llamaindex/ExtractBenchPro Github (is ExtractBench): github.com/run-llama/ExtractBench

OpenDocRouter makes it dead-simple to start parsing your documents across on ton of frontier Models (gemini, luna) and latest OSS models (LightOnOCR-3, MinerU2.5-pro). Parse your first document in 60 seconds! opendocrouter.ai

Added a few new models to OpenDocRouter (gemini-4-argon, Muse Spark/Wonder). Interesting to see the latency for Muse Wonder so low. Gemini's costs are low but their introductory rates mean the price will double later this year.

China has killed the entire vector database industry. They open-sourced TencentDB Agent Memory. It gives any AI agent long-term memory that runs 100% locally. No Pinecone. No cloud APIs. No repeating yourself every session. - 61% fewer tokens - PersonaMem accuracy: 48% → 76% - Zero external API dependencies - Runs on plain SQLite Most memory systems compress your history into an opaque vector pile. when recall goes wrong, you're guessing. this one doesn't compress, it builds a semantic pyramid. L0 Conversation → L1 Atom → L2 Scenario → L3 Persona. Short-term state gets encoded as a Mermaid graph in your agent's context. verbose tool logs get offloaded to disk. when the agent needs proof, it drills back via node_id to the exact raw log. no lossy compression. every layer is readable markdown you can just open and inspect. 5.1k stars. 100% Open Source.

Three events, one award, a very respectable 4th place in hot sauce. Our SF Tech Week: → AI Demo Night w/ @ClickHouse + @artie_labs: @imaanxsultan won best demo → @matissemlk took 4th in the Blazel hot sauce competition → Co-hosted the biggest night of SF Tech Week with @SpaceX, @Lemma_, @Daytonaio, @Tiny_Fish, and more! Agent Economy Gala: 500+ founders + investors in suits and gowns Thanks to everyone who joined us for a very busy, but fun SF Tech Week :)

🚨 CHINA JUST OPEN SOURCED AN OCR THAT EATS 100 PAGE PDFS IN ONE PASS It’s called Unlimited OCR. Only 3B params. Runs locally. Every other OCR tool chops your doc into pages and loses the thread. This one reads the whole thing in one continuous pass. → One shot full context parsing (32K context window) → Multilingual out of the box → 93% on the standard parsing benchmark (6 points over baseline) → Under 0.11 error rate past 40 pages → Runs 100% locally on your own hardware → Works with Transformers, vLLM, SGLang, Docker, Ollama, llama.cpp Traditional cloud OCR (Textract, Google Vision, Azure Doc Intelligence) costs $1.50 to $15 per 1,000 pages. This runs on your machine. For free. Forever. Baidu built it explicitly to push DeepSeek OCR one step further. Already at 1.9M downloads on Hugging Face and most people have no idea it exists yet. 100% open source.