Serving Production Inference on Tenstorrent Wormhole
The culmination of 4 weeks of intense sleepless nights. I was invited to present at RISC-V Day Hong Kong 2026 at City University of Hong Kong. The topic I tackled was Production Inference on RISC-V accelerators. Can we run Qwen3.6 27B on Tenstorrent hardware available now in Hong Kong? The answer is Yes! but only after a lot of work.
I'll tell you the truth about running inference. It's hard work. The big companies are throwing lots of resources, the start-ups need to focus. Some hardware gets left behind. So it's up to the community to fill the gap. There are two announcements:
- TinyGrad is now running on both Tenstorrent Wormhole accelerators and on the RISC-V SpacemiT X100 cores via Vector support.
- I will be upstreaming my work on stabilizing inference on Tenstorrent Wormhole Quietbox for the Qwen3.6 family of models.
Last gen hardware in a rapidly evolving world often ends up less tested. But it's what we can get due to trade restrictions.
Accelerators often run their own stack and models also lag support. The major issue is that it's one thing to generate tokens, another to serve agentic tool based work. I asked myself "what's the current workhorse of the industry?" The Qwen3.6 27B model.
So I spent a month making sure it's as stable as possible to be able to serve a multi-user (multi-agent) environment with guarantees that the data stays on-prem.
But I wanted to go further. That was only one model family and the next one is still the same old gen hardware support issue. So I pivoted and ported #TinyGrad to the Tenstorrent Wormhole ecosystem. The performance is about 40% for a very short development cycle.
The numbers on the Qwen3.6 35B-A3B Mixture-of-Expert models on the Tenstorrent stack weren't great. Only less than 5% extra tokens/s! Only in the aggregate did it go to 10%. Unexpected for a MoE model. Offloading to the beefy Epyc CPU is not supported.
So I pushed further and also ported TinyGrad to RISC-V CPU. The initial experiments were done on a SpacemiT K3 that has the same X100 cores as the V100. It means that we can now offload the experts to the CPU! (and that's for any combination of RISC-V RVA23 CPU + accelerator).
It's still early stages, but this means we finally have the first fully RISC-V server to provide inference at scale via TinyGrad. RISC-V International let's make this happen!