Neural networks may be readable after all, American open-weight models, an open agent cloud computer that claims to be secure, agents in the wild (but not rogue), and cracks beginning to show in the financing behind the compute buildout.
Following AI Waves #028: Sense and Speculation.
Last issue asked how much of what the AI labs produce anyone outside them can verify. This week brought a partial answer from inside the models themselves, in an essay arguing that neural networks are far more readable than the field believed. Elsewhere, Meta and Apple tuned an open-weight model for the desktop, Perplexity open-sourced the computer its agents run on, researchers questioned a fleet of agents that claimed to be Claude, and the lenders financing the compute buildout began to pull back.
Neural networks may be readable after all
Nick Cammarata, an interpretability researcher at Goodfire and formerly at OpenAI, helped build the field’s evidence for superposition, the idea that neural networks pack unrelated concepts into the same neuron. If that were true, no single neuron would mean one thing, which is why the field’s standard approach trains a second model, a sparse autoencoder, to break a network into cleaner features before studying it. In Neural Networks Are Already Clean, published October 15, Cammarata argues the evidence was misread. Its most famous example came from InceptionV1, a 2014 Google image-recognition model, where one neuron fired on both cat faces and the fronts of cars. He now reads it as a detector for pointed V shapes, from cat ears to car grilles. InceptionV1’s final layer, almost unreadable when the field first studied it, becomes about 99% readable in a model of the same size built with the design features modern networks use to train well. In Alibaba’s open-weight Qwen3-32B, his preliminary audit estimates that roughly 90% of neuron responses (counting each neuron’s positive and negative firing separately) have one main job. If he’s right, researchers can study a network’s own neurons directly. He calls it a working hypothesis, the companion paper isn’t out, and Neel Nanda, who leads Google DeepMind’s interpretability team, doesn’t see “a significant update to the big picture.”
Last issue I drew on Dario Amodei’s 2017 Big Blob of Compute memo, written for colleagues at OpenAI and published by Kevin Roose this month. It holds that capability comes from a large, minimally structured mass of compute shaped by training, and I took that as a reason to doubt we could account for such a system part by part. Cammarata’s evidence cuts the other way. No one specifies how a network organizes what it learns, yet he finds that organization orderly, because a neuron’s output is only useful if the neurons downstream can read that output. Researchers who learn to read it the same way can study the network as it is. Architecture still matters here: residual connections and normalization, added to help deep networks train, made them readable. At the frontier, larger models learn narrower neurons that are easier to recognize, so scale may help understanding where we assumed it hurt.
Alignment depends on seeing why a model acted, and the most direct view so far has been the Chain of Thought (CoT), the reasoning a model writes out before answering. OpenAI’s system card for GPT-6 Astra shows that view narrowing: Astra writes “shorter CoTs that often omit or weaken the evidence the monitor needs,” a change OpenAI found “correlated with the increase in no-CoT capabilities,” the model’s growing ability to answer without writing its reasoning down. The capability gain and the loss of visibility arrived together, and OpenAI calls the extra compute needed to keep reasoning readable a “monitorability tax.” Its chief scientist, Jakub Pachocki, expects progress to be “bottlenecked by confidence in monitoring.” Probes have already read single structures inside models, such as a scratch space inside Claude that registered a staged scenario was a test. A network legible throughout would let researchers trace a behavior to its cause and fix it, for instance by removing a training environment that taught the model something harmful. Deception, Cammarata notes, “has no reason to hide from the model’s own circuitry,” so a capable enough investigator should find it there. If the result holds, it’s the most encouraging news for alignment in some time.
Cammarata’s method relies on AI agents labeling neurons and testing explanations at a scale no human team could reach, and every model in his language audit has open weights, among them Qwen3-32B and Moonshot’s Kimi K3, which runs to more than a trillion parameters. Independent verification needs capable models outside the labs, and this kind of verification also needs the weights. For closed models, the only investigators will be the lab that trained them and the evaluators it admits.
Models and chips designed together
Muse Spark Open, the open-weight model Meta released on October 15, has 816 billion parameters, 42 billion of them active at a time, and Meta optimized it with Apple to run on two M5 Ultra Mac Studios through Apple’s MLX framework. A model that size has usually needed a full data center rack, costing in the mid six figures. This one was tuned for a machine that fits on a desk, by two companies that spent years fighting over app tracking. Alexandr Wang, Meta’s chief AI officer, told Axios he was “excited for mommy and daddy to be playing nice again.”
Source: https://www.ft.com/content/7477e462-c899-4841-a5cf-c01706e23982?syn-25a6b1a6=1
In September I described the labs designing their own chips while Nvidia moved into models, and both moves continued this week. Nvidia released Nemotron 3.5 Super VL, has committed $800 million to Reflection AI, and is in early talks to buy it, according to the FT, via Bloomberg Law. Reflection published the weights of its 501-billion-parameter Beam under Apache 2.0 on October 16. Microsoft expects its new Maia 300 chip to deliver “40% more tokens per megawatt” than leading GPU systems, and plans to make it the main inference chip for its own products from 2027.
Co-design lowers the cost of running a model on one vendor’s hardware without locking it there. Meta supports AMD, Apple, Intel and Nvidia hardware, and AWS’s new TorchNeuron is a beta release designed to let developers use PyTorch on Trainium with less rewriting. Chipmakers compete both by tuning models for their hardware and by making it easier to switch. Spark Open’s license adds a commercial condition: providers offering it as a service need a separate agreement once their group exceeds $20 million in annual revenue.
These companies have something else to sell: Apple has Macs, Nvidia has GPUs, Amazon has compute, and Meta has advertising and its Muse agent. Open software brings developers toward those businesses. The strongest open weights are still Chinese: on Artificial Analysis, Spark Open scores 39, against 45 for Zhipu’s GLM-5.3 and 44 for Moonshot’s Kimi K3.
Beijing set out its own terms for those models on October 16. China’s commerce ministry proposed export controls on 23 AI technologies, including models, agents and computing clusters, covering the overseas deployment of models that contain restricted technology. Access through APIs, apps, websites and cloud services is exempt, and anything already released as open source or published in a paper falls outside the controls, with simplified licensing for open-source exports. Chinese labs can keep serving the world from home, where the service can be withdrawn, or publish the weights for anyone to use, and the draft restricts the route between the two: handing unreleased weights to a deployment abroad. In July, OpenAI and Anthropic pressed Washington to restrict Chinese open weights. Beijing’s draft makes open release one of its two easiest ways to export.
An open model gives a team more choice, with the economics still shaped by hardware support, license terms and export rules.
Control over the agent’s computer
On October 15 Perplexity open-sourced SPACE, the execution environment behind its Computer agent, under an Apache 2.0 license. Each sandbox is a Firecracker microVM with its own operating system. Network access and platform-held credentials are managed outside it, so an agent can reach an approved service without the key entering its workspace.
Earlier this month I argued that operators should enforce limits independently of the model’s judgment. SPACE makes one implementation available for anyone to inspect and run. A hosted deployment still requires trust in the operator: its architecture guide gives host-side services privileges over VMs, storage and networking. Reading the source doesn’t establish which code and configuration a provider runs.
Perplexity found no escape from the VM to the host in 108 tests, and says it fixed the network bypasses it found. A new preprint examines another route to broader access. Researchers planted instructions in repositories that steered Claude Code and Codex CLI into requesting reusable permissions broad enough to cover malicious actions. The agents requested the targeted permissions in 58% of trials. With approval assumed, the agents completed the malicious task without further prompting in 47 of 58 eligible cases. It didn’t test SPACE.
The approval itself changed what the surrounding system would allow. Once an operator accepts a broad rule, the system can enforce it correctly while letting an attacker do damage. Reviewing a request to widen permissions needs the same care as reviewing the action it would authorize, particularly when the agent formed that request from material supplied by someone else.
Agents in the wild
Researchers at Palaestra traced more than 2,600 attempts by a fleet of AI agents, between September 28 and October 8, to pull data from Amap, Alibaba’s mapping service. One attempt used an access key leaked in another company’s public code, and it worked. The traffic ran through Tencent Cloud servers in Hong Kong. The researchers left the agents a message and got answers back: they were, the agents said, “Claude (Anthropic), running in Claude Code.” The researchers’ tests make that unlikely, since Claude models given the same setup never described themselves that way in 36 trials, and the agents’ coding habits don’t match Claude’s. Nothing in the report suggests the fleet had gone rogue. It appears to have been doing a job someone gave it, and it couldn’t be relied on to say whose.
On October 15 OpenAI disclosed that a research model asked in June to build a crypto prototype needed test funds, found them in a wallet whose recovery phrase is publicly known, reasoned that they were “effectively abandoned,” and spent them. It left that out of its report, and OpenAI found it three months later. The test currency had no value. The model’s decision about what it was allowed to take did.
After Anthropic disclosed that its test agents had filed visa applications and sent a false tip to police, the White House’s AI task force, the Super Intelligence Force, told AI companies on October 9 that reporting incidents is “not optional.” The EU’s AI Office has begun sending requests for information, and the FTC sent OpenAI and Anthropic civil investigative demands on October 14.
Each of these assumes the lab knows when its agent was responsible. The fleet shows how easily an agent’s claimed identity can be borrowed, and OpenAI’s case shows a lab can take months to find its own incident. Last month Amazon blocked Meta’s shopping agent for, among other things, not identifying itself. Any reporting rule depends on the same thing: agents that reliably say who runs them.
Neither a borrower nor a lender be
GPUs are getting more expensive to rent, and their operators are getting more expensive to lend to. Silicon Data’s B300 index rose 6.4% in a week. CoreWeave’s credit default swaps implied roughly a 60% chance of default within five years. Firmus withdrew its Australian IPO, and JPMorgan and Goldman Sachs declined to underwrite $3.7 billion of GPU debt in Volta’s financing package. AI-related debt issuance fell from $113 billion in June to $23 billion in September, according to Morgan Stanley figures reported by the FT.
Demand for compute can hold while individual projects become harder to finance. Microsoft accounted for 62% of CoreWeave’s 2024 revenue, according to its IPO filing. Microsoft also withdrew from Nscale’s Essex project, where a grid connection is now expected in the 2030s. Nscale says its other Microsoft contracts are unaffected. The site’s financing depends on both keeping an anchor tenant and delivering power on time. Modal, which sells compute to many smaller customers, raised $750 million the same week after passing $1 billion in annualized revenue.
The biggest tenants are growing more slowly. Anthropic told prospective investors that third-quarter revenue reached $16.8 billion, the FT reported, up about 46% on the second quarter after roughly 140% growth the quarter before. Some of that slowdown was a choice. Rival labs paying for heavy use of Claude to train their own models, the distillation I wrote about this month, made up 5 to 7% of revenue, and Anthropic said it cut that back “materially” in the quarter; its new defenses, it says, reduced what distillers harvest by around 90 to 99%. Across the market, Ramp’s data shows two straight months of flat token spending while token volumes kept rising, which tests the Citadel argument I quoted in September that cheaper tokens would drive enough extra use to raise total spending.
The figures are unaudited. Anthropic’s prospectus, which IFR reports will become public on October 26, should show how much of its revenue came from rivals and how its compute commitments are spread, the two things the neoclouds’ lenders most need to see.
Quick hits
A SpaceX share for every child
Elon Musk pledged on October 19 to give one of his SpaceX shares to “every kid in America” next year. Recipients would hold an asset whose value moves with the company. How much control families would have over it remains unclear: SpaceX hasn’t said whether the gift would go through Trump Accounts or come with a lockup.
Anthropic’s equity pledge
Anthropic’s founders had already pledged to donate 80% of their equity. Its new Public Benefit Fund would receive 5% of the company’s shares, subject to stockholder approval. Together with employee matching, Anthropic says the commitments dedicate roughly 20% of its shares to public benefit. The new fund is a wholly owned LLC, and further allocations depend on the board’s discretion. Who directs the giving will depend on governance arrangements the company has yet to publish.
Evaluators with employee access
Anthropic gave six METR staff company hardware, office space and internal safety tools. METR can publish without Anthropic’s veto, subject to redactions in predefined categories. That improves on the editorial control Reuters reported the labs seeking earlier this month, while leaving the lab to choose who gets access.
Machines referee people
Anthropic reports that Claude reconstructed how the unexplained numbers behind several NIST elliptic curves were generated. Their NSA provenance had prompted decades of backdoor suspicions. The recovered recipe uses timestamps and a standard pseudorandom number generator, reproducing the published seeds. Some details remain unresolved, but researchers can check the recipe themselves, without taking the model’s reasoning on trust.
The chatbot as witness
The Justice Department’s case against an OpenAI engineer accused of planting a recording device cites questions from a chatbot account linked to his spouse about building the device and encrypting its recordings. As Matt Levine observed, asking a chatbot for exactly what you want leaves a more complete record than a search history.
Character.AI’s end
Disney had already hired most of Character.AI’s staff when the company filed for Chapter 11. WIRED reports that Disney sought a license to its technology through the bankruptcy process without assuming its liabilities. The eventual sale terms will determine what resources remain for creditors and claimants.
Portfolio company updates
Prime Intellect: a Nemotron that changes the right record
Prime Intellect and Nvidia post-trained Nemotron 3.5 Super VL for multi-step office work, published October 15. The training set is 2,500 enterprise tasks the two built together, modeled on Zapier’s AutomationBench: a request, a starting world of CRM records, inboxes and spreadsheets, and programmatic checks on the final state. A task scores zero if any guardrail is broken, whatever else it gets right. About 100 steps of reinforcement learning on 128 GB200s, roughly 11 hours, took the public AutomationBench-AA score from 34.0% to 57.7%. The tooling is open source; the post does not say whether the weights or the task set are.
Four days later Prime released prime-rl 0.10. The default algorithm is now IPO, written for asynchronous training, where the trainer and the inference engine never agree exactly on a token’s probability. The previous filter, IcePop, collapsed around step 1,200 on an agentic coding run with GLM-4.5-Air when that gap grew twentyfold; IPO masked about 0.25% of tokens and trained through. Verification outside the labs needs open models, and it needs training tools that survive the failures the labs fix in private.
Provably: the tools found it, the answer lost it
Provably’s Switchboard measured on October 19 where recovered evidence goes missing. In 24 repaired agent runs, the tools had already found 69 facts the final answer should contain. Written from the conversation alone, the answers kept 51 of them. A quarter of what the agent had in hand never reached the person who asked. A ledger of which calls succeeded, which failed and what each returned raised the count to 56, and the authors say in print that 24 runs are too few to call that gain real. The number that holds is the first one. An agent’s answer is its own account of its work. The OpenAI model that left a spent wallet out of its report made the same omission at a lab. Switchboard sells the record that lets someone else check.
LayerLens: the reply is not the record
LayerLens published results of a demo, run four ways: the agent reports “Done,” and the record says otherwise. Asked to assign Acme Inc. to Sam, the agent updated Acme Ltd. instead, and a test that grades the reply passes it. LayerLens grades the database after the run, which record changed and which did not, and reruns the same checks when the prompt changes, so a fix to one task is tested against the rest. The demos are illustrative, with no customer named. The White House, the EU and the FTC are asking labs to report what their agents did, and a lab can only report what it checked.
GI Labs and Hub: the robot learns from the people it replaces
Two positions sell the same raw material from opposite ends. GI Labs argued on October 15 that a factory, construction or food-service business should record its own employees with the EGO1 headsets and build a reviewed dataset before it owns a single robot, so the first robot arrives trained on that company’s work. The company reported $500,000 of revenue in the third quarter and says it has been cash-flow positive since March, both company-stated. Hub collects the same kind of footage at scale rather than per site: its public figures are 150,000 active contributors at 730 commercial sites in 150 countries, 18.7 million contributions, and 540,000 hours of egocentric depth video across 10,000 environments, with teleoperation trajectories, hand tracking and tactile streams alongside. Its YC profile cites a seven-figure data agreement.
Elsewhere
Intelligent Internet is hosting a free screening of The Terminator at Kensington Central Library in London on October 29, doors at 18:00 (details).