Why AI Has to Be Instantaneous

Two things keep AI out of most products: it's too slow, and it costs too much. A chatbot that takes a couple of seconds feels fine. An agent that makes a hundred model calls to finish one task does not — at human-paced latency it takes minutes, and the bill is enormous. Taalas, a chip startup founded by Tenstorrent founder Ljubisa Bajic, has a blunt answer: stop running the model as software on a general-purpose chip and cast it directly into silicon. Their first product, a Llama 3.1 8B baked into one chip, runs at 17,000 tokens per second per user — about ten times the fastest GPU setups — and you can talk to it right now at ChatJimmy. This is about why that number matters.

Two walls: latency and cost

Bajic's argument starts with two barriers. The first is latency. Talking to a language model still lags far behind the speed of human thought — a coding assistant can stall for tens of seconds, which is enough to break a programmer's flow. The second is cost. Running a modern model means room-sized machines pulling hundreds of kilowatts, with liquid cooling, stacked memory, exotic packaging, and miles of cable, scaling up to data-center campuses with their own power plants.

Both problems trace back to the same root: we run AI models as software on general-purpose chips. A GPU is a flexible machine that can run any model, and it pays for that flexibility in energy, heat, and money. Taalas's claim is that AI inference — which Bajic calls the most critical computational workload humanity has ever faced — is too important to keep running on hardware that was designed to do everything.

Speed is the product, not a feature

The reason to care about extreme speed isn't that a faster chatbot feels nicer. It's that latency decides which applications are even possible. The clearest case is the agent: a single user-facing answer is no longer one model call, it's often dozens — planning, calling tools, reading results, re-planning, writing the answer.

one agent task
  ├─ plan          ─► model call
  ├─ search        ─► model call
  ├─ read result   ─► model call
  ├─ …             ─► × dozens
  └─ write answer  ─► model call

at ~50 tok/s   each step is seconds   → the task takes minutes
at 17,000 tok/s the whole loop is imperceptible

At normal GPU token rates, every step in that loop is a visible pause, and the pauses stack into minutes. At 17,000 tokens per second the whole loop finishes before you'd notice the first one. The same logic applies to voice — a real conversation can't have a one-second gap before every reply — and to reasoning techniques that spend lots of tokens to think, like chain-of-thought or sampling many candidates, which only make sense if tokens are fast and nearly free.

What Taalas actually shipped

The product is real, and you can poke at it today. HC1 is a technology demonstrator — a single chip on TSMC's 6 nm process, 815 mm² of die, 53 billion transistors, sitting in a 2.5 kW server. Hard-wired into it is Meta's Llama 3.1 8B. There are two front doors:

• ChatJimmy, a stripped-down chatbot with a live token counter so you can watch how fast it generates. It is the consumer-facing face of the chip.
• An inference API, the same hard-wired Llama offered to developers as a beta service.

On that chip Taalas reports 17,000 tokens per second per user, which they put at roughly ten times the fastest current setups, twenty times cheaper to build, and ten times lower power. The company was founded about two and a half years ago, and the manifesto notes the first product was built by 24 people on $30M of the $200M-plus they've raised — a deliberate contrast with the "medieval army" spending of most deep-tech startups.

The model is the computer

Taalas's tagline is literal. On a GPU, the model is data: billions of parameters that get loaded into a general-purpose chip and multiplied at runtime. On a Taalas chip, the model is the chip — the parameters are etched into the silicon itself, and human language is the only software that runs on top. They call the result a Hardcore Model, and the principle behind it is total specialization.

The bet is an old one in hardware: whenever a workload matters enough, you stop running it on a general-purpose processor and build a chip that does only that. It's why Bitcoin mining moved from CPUs to ASICs, and why video codecs and network routers have dedicated silicon. Taalas is applying it to a whole neural network — one model, one chip, nothing general about it.

The memory wall is the real enemy

The speed comes from one specific idea. Modern inference hardware is split in two: memory on one side, compute on the other. That split exists because of a hardware paradox. The dense, cheap memory — DRAM — is built on a process that can't hold compute logic, and reaching it off-chip is thousands of times slower than on-chip memory, while the fast compute chips can't be built on a DRAM process. So a GPU spends much of its time and energy shuttling weights from stacked HBM memory into the compute cores, every single token.

GPU / general-purpose inference
   ┌───────────┐   slow bus    ┌──────────────┐
   │   HBM     │ ────────────► │   compute    │
   │ (weights) │ ◄──────────── │   cores      │
   └───────────┘   shuttle     └──────────────┘
   weights moved every token · most energy spent moving, not computing

Taalas HC1 — the model is the computer
   ┌─────────────────────────────────────────┐
   │   weights embedded in the silicon,       │
   │   compute wrapped directly around them   │
   │   (storage + compute unified on-die)     │
   └─────────────────────────────────────────┘
   nothing to fetch · the parameter IS the circuit

Taalas erases the boundary. By unifying storage and compute on a single die at DRAM-level density, the weights live exactly where they're used. There's no bus to cross and nothing to fetch. That's the structural reason the chip can be both much faster and much lower power at once — it isn't paying the energy tax of moving billions of parameters around on every token.

Radical simplification

Once you remove the memory-compute boundary and tailor the silicon to one model, most of the complexity of a modern AI server falls away. Bajic's third principle is that this lets you redesign the whole stack from first principles and skip the hard parts entirely.

No HBM, no advanced packaging, no 3D stacking, no liquid cooling, no high-speed I/O. Taalas argues that this engineering simplicity — not some exotic breakthrough — is what produces the order-of-magnitude drop in total system cost. The interesting move is that the speed and the cheapness come from the same decision, not two separate optimizations.

The catch: a frozen, low-bit model

A piece this favorable to a vendor needs the caveats stated plainly, and to their credit Taalas states most of them itself.

It runs one model, and the model is quantized hard. The first-generation silicon predates standard low-precision formats, so it uses a custom 3-bit base data type; the Silicon Llama mixes 3-bit and 6-bit parameters and is, in Taalas's own words, "aggressively quantized" with "some quality degradations relative to GPU benchmarks." The second-generation platform (HC2) moves to a standard 4-bit floating-point format to address this.

The chip is hard-wired. You can't load a new model onto it; a new model means new silicon. Flexibility survives only at the edges — a configurable context window and fine-tuning through LoRA adapters. The answer to obsolescence is speed of fabrication: Taalas says it can turn a previously unseen model into hardware in about two months, and the roadmap is a mid-sized reasoning model on HC1 this spring, then a frontier model on the denser HC2 by winter.

What instant, near-free inference unlocks

Here's the answer to "why be extremely fast." When a token costs almost nothing and arrives almost instantly, a set of applications that were impractical on GPUs starts to make sense.

The pattern across all six: these aren't faster versions of things we already do. They're things that don't work at all until latency and cost both collapse. That's why Taalas shipped a deliberately unremarkable model — the point of the demo isn't the model, it's the speed envelope around it.

The ENIAC bet, and the case against

Bajic frames the whole thing with a history lesson. The first computer, ENIAC, was a room full of vacuum tubes — slow, costly, and unscalable. The transistor is what turned computing into workstations, then PCs, then phones, then something in every device. His claim is that today's AI data centers are the ENIAC phase, and that specialized silicon — one model per chip — is the transistor moment that makes AI ubiquitous.

General-purpose computing entered the mainstream by becoming easy to build, fast, and cheap. AI needs to do the same.

The case against is just as clear, and it's about churn. Models improve every few months; silicon is frozen the day it tapes out. A mask set for Llama 3.1 8B — a model that's already not state of the art — is a bet that a fast frozen model beats a current model at GPU speed for enough real workloads. Meanwhile GPUs keep getting faster, and a middle camp of fast-but-still-programmable chips (Groq, Cerebras, Sambanova) is chasing the same latency prize without committing to a single model.

Taalas's edge holds only if three things stay true: the two-month turnaround keeps their catalog fresh enough, enough applications care more about speed and cost than about always running the newest weights, and the quality lost to aggressive quantization stays inside tolerance. All three are open questions. But the underlying observation — that we're running the most important workload in computing on hardware built to run everything, and paying for it in seconds and kilowatts — is hard to argue with. The fastest way to find out whether the bet pays is already live: you can go talk to it.

References

All performance figures are Taalas-reported, at a 1k/1k input/output sequence length: their own H200 and B200 measurements, with Groq, Sambanova, and Cerebras numbers taken from Artificial Analysis. The Silicon Llama is aggressively quantized (3-bit / 6-bit) and Taalas acknowledges quality degradation versus GPU benchmarks; independent third-party benchmarks were not available at the time of writing.