Afternoon briefing · 16:00
Weights, proofs and evidence: AI under questioning
By · bombong
Claims about AI are now tested less by what the companies say and more by public material that anyone can check. Three stories came out together this afternoon: an analysis that says it inferred training hardware from public weights alone, a benchmark that checks research results with formal proofs, and a lawsuit built on the companies' own risk warnings.
How public outputs become evidence
This morning's piece looked at the trend of measuring agents by results and cost. The afternoon's news is less about what gets measured and more about who checks it and with what material. An analysis featured on GeekNews said it found initialization patterns consistent with NVIDIA Blackwell in the weights of six models released by DeepSeek, Moonshot AI, MiniMax and Ant Group28. The analysts explained that when the pseudorandom seed and the GPU parallelism setup are reconstructed together, Hopper and Blackwell leave different numerical patterns. They also said that applying the same method to 24 models whose training hardware is public produced results that matched the public information.
What matters in this analysis is that it used only public weights, with no inside sources. As Chinese companies' chip sourcing stayed in dispute this week, it suggests that releasing a model can itself become a clue to its training environment.
Something similar is happening with research results. 'TCSExam', co-authored by Andrew Yao and Yang Zhilin, checks AI-produced results in theoretical computer science with Lean 4 formal proofs and program checkers30. According to reports, Kimi K3 set a record in the process by finding 630 edges in a 124-vertex graph. It is a case where a model's research ability is presented in a form a machine can check, not just asserted.
Companies' public statements become evidence too
If the previous section was about technical outputs, in court the companies' own words play the same role. According to Reuters and AP, four US nonprofits filed a lawsuit in a San Francisco court on the 12th32. The defendants are OpenAI, Anthropic, Google, SpaceX and Meta, and the plaintiffs asked the court to stop them from further developing or deploying frontier AI models.
The plaintiffs argued that AI company executives have themselves warned that advanced AI could pose a serious threat to humanity, so they should stop until safety is assured. Regardless of whether the court grants the request, the fact that companies' public risk assessments are being used by the other side carries implications for the whole industry. Safety-related statements and disclosures are likely to be treated less as promotional language and more as material on the record.
The evidence is still thin
Still, today's evidence is not yet established fact. The hardware analysis has been reported only through a community summary, and neither the analysts' identity nor the original source has been confirmed. The analysts themselves cautioned that the results alone do not show that all training was done on Blackwell. The companies involved have not yet responded. So the analysis is better seen as showing that a method for inferring training hardware from public weights may exist, rather than proving any particular company's chip use.
The TCSExam record is also at the reported stage, and the lawsuit has only just been filed. The tools and materials for verification are growing, but it will take more time for the results to be independently reproduced and accepted within institutions.
Releasing a model or announcing research results and risks now also means putting out material that others will check. Companies and practitioners should therefore think ahead about what questions their weights, technical documents and public statements may raise. Things to watch next: whether the original hardware analysis and reproductions are published, whether such methods are used in debates over chip location-tracking legislation such as the Chip Security Act, and how the lawsuit's first hearing proceeds.
This piece was drafted with AI, checked against the cited public sources for links and figures before publication, and reviewed again by the operator afterwards. It includes the author's interpretation. How we write and check · Corrections
Morning briefing · 08:00
AI agents start counting the cost per task
By · bombong
The question used to judge AI agents is changing. It is less about how smart a model is and more about whether it actually finishes the job and what that job costs. Google added a feature to its coding tool that switches between large and small models mid-task. An index that measures agents on real freelance work rose sharply over a year, and on the cloud side, guidance emerged to calculate the cost of a single task instead of per-token prices.
How model choice becomes cost design
Google's default 'fast model' has changed. Gemini 4 Hydrogen replaces Gemini 3.8 Flash and went straight into AI Mode in Search and the Gemini app1. According to community summaries, the same price applies across the full 1-million-token context: $1.50 input and $7.50 output per million tokens. Google said the model handles multi-part questions and questions with many typos better.
More directly relevant to this argument is 'Orchestrator', added to the coding tool Google Antigravity. The feature switches between Carbon and Hydrogen based on the complexity a task requires. In Google evaluations cited by community summaries, Orchestrator kept a success rate close to Carbon's while cutting average quota consumption by 24% and speeding up completion by 19%. The system now picks, task by task, the model users used to choose themselves. The yardstick therefore shifts from a single model's score to how few resources it takes to reach the same success rate.
The narrowing gap between small and large models also makes this design possible. According to community summaries, Hydrogen scored 51 on the Artificial Analysis Intelligence Index, ahead of Argon (50). However, these figures relay Google's own evaluations and have not been reproduced externally.
Scorecards and operating ledgers ask the same question
The question raised on the model-selection side comes up just as much in evaluation and enterprise operations. The Center for AI Safety (CAIS) and Scale AI released the latest results of the Remote Labor Index (RLI), which measures agents on real freelance projects. According to reports, AI completed 41.3% of projects at a level equal to or better than human experts. When the index was first released last year, the top score was 2.5%3. The index looks not at whether AI answers knowledge questions correctly but at whether it finishes paid work end to end, which ties it to the trend above.
In enterprise operations, this question shows up as a cost ledger. AWS published four Amazon Bedrock cost management guides18. They cover how cache retention time, service tier, reasoning effort, and maxTokens settings change costs, as well as how to set spending caps per IAM principal. Notably, they advise proving agent ROI with actual cost per task rather than per-token prices. Hydrogen's four reasoning-effort levels and the reasoning-effort setting in AWS's guides point the same way: the dial that trades performance against cost is moving into users' hands.
Deployments describe their results in the same terms. Natera built SWARM, an agent system containing lab operations knowledge, on Amazon Bedrock AgentCore. The company said first responses to support tickets became about 130x faster and 96.2% of tickets were handled automatically20. Pricing per task is appearing in the consumer market too. Meta added a feature that lets its agent Muse call US businesses on users' behalf, and the feature requires payment8.
The verification gap behind completion rates
The larger completion-rate figures get, the more the conditions behind them matter. Reports do not say which models and conditions produced RLI's 41.3%. Orchestrator's efficiency figures also remain the company's own evaluation.
Finishing a job is not the same as finishing it correctly. Cases illustrating this emerged the same day. IP protection company Podqi put shopping questions to ChatGPT, Gemini, and Perplexity. In some cases they directed users to risky shopping sites, such as impersonation sites, The Verge reported14. A Google researcher claimed to have found a candidate unmapped Maya site using a team of AI agents. But the site had already been reported in an academic journal last year, The Verge said21. Both cases show that checking results is separate work. If so, the cost of verification should be included when calculating cost per task.
Practitioners can no longer easily pick a model by a single benchmark score. They need to weigh per-task success rates, cost per task, and the effort required to verify results together in one ledger. There are three things to check right away. First, whether CAIS and Scale AI publish the original source showing the models and conditions behind RLI's 41.3%. Next, whether Orchestrator's quota savings hold up in external use. Finally, how ByteDance prices the Muse-style agent it is reported to be releasing within weeks.
This piece was drafted with AI, checked against the cited public sources for links and figures before publication, and reviewed again by the operator afterwards. It includes the author's interpretation. How we write and check · Corrections
Unconfirmed reports 8
Not yet confirmed by an official source. Check the original report.
- Stanford's 'Flash ALOHA' learns fast, precise robot manipulation from 50 demos per task with 80%+ success; open source (per community) Original
- Slide shared claiming an unreleased OpenAI cybersecurity model scored 77.0% on ExploitGym vs. 39.2% for Claude Mythos 5.2 (community report, unconfirmed) Original
- Jensen Huang affair follow-up: report that NVIDIA employees petition for Huang's return shared online (community report) Original
- Samsung Electronics in talks to acquire US data center cooling firm Modine after Germany's FläktGroup (report) Original
- Edge0 open-sources 145B and 120B models plus an inference framework for personal computers, claiming 29GB of memory and 80+ tokens/s on a laptop (community reports) Original
- Apple's Foundation Models framework puts on-device and cloud models behind the same Swift API, adding image input and 'Dynamic Profiles' (community reports) Original
- Jensen Huang follow-up: US attorney general says no conclusion on who the woman is; report says Huang called successor Kress a 'deer in the headlights' Original
- AI helps decode Kurt Gödel's notebooks, reportedly containing hundreds of proven theorems (The Guardian report, community reports) Original
Community buzz 5
Topics widely discussed on Reddit, Hacker News, GeekNews and elsewhere. These are community reactions and may include unverified claims.
- Gemini 4 Hydrogen The official post drew 420 points and 280 comments on HN, and discussion continued on Reddit and GeekNews about the $1.50/$7.50 pricing, Artificial Analysis scores, and Antigravity Orchestrator's automatic model switching.
- Jensen Huang resignation memes After the report of a third relationship, Reddit was flooded with satirical posts and board memes, and some floated conspiracy-style claims that Huang's friendliness toward local AI was the real problem.
- Claude over-refusals Users posted complaints that Claude refuses even to quote works the users provided themselves or to answer basic biology questions. Some in the community say this behavior did not occur in earlier versions.
- Responses to surging ML conference submissions Criticism that administrative burdens such as CVPR's author registration form burden interdisciplinary collaborators was discussed alongside a proposal to cap accepted papers at the top three conferences at 1,000.
- Local LLMs on low-end hardware A 'smolmaxxing' project running LLMs on TV boxes with about 1GB of RAM and a 2013 iPad drew attention, as did a hack claiming to unlock a CMP170hx mining card to A100-class via voltage glitching.