Doesn't change the conclusions of the article, but each of those machines is more like $4k+
https://www.microcenter.com/product/711961/amd-ryzen-ai-halo...
So we had to rebadge it to "unified memory".
Curious if we'll ever see some old integrated graphics processor "hacked" to manage to handle 128 GB of allocated system RAM and be able to serve diffusion-LLMs at a decent rate on "old" hardware...
1) prefill
2) decode
For prefill, you are compute bound, and it is trivial to batch multiple input tokens together. When using cpu offload, software like llama.cpp will batch weight uploads with tokens that need those weights and perform work on the GPU. It works very well. With a large batch size and pcie5 you can get prefill speeds close to having all weights on the GPU.
For decode, you are bandwidth bound, and it is difficult to batch multiple output tokens together. There is no benefit to sending your weights to the GPU because even if it internally has insane bandwidth, you are still bottlenecked by system RAM (and adding a pcie5 upload would bottleneck it further). This is the number people usually talk about when they say they are getting a certain tk/s.
I think it's the other way around? The GPU has to stream gigabytes of active layer weights to compute the next token, so having a batch of next-token predictions sitting there on the GPU goingh through the layers makes better use of the bandwidth.
At least that's what I observed on a Strix Halo, batching 4 predictions yields like 2-3x the total tps.
But:
1) It still makes no sense to upload the weights to the GPU with MTP as you are still bottlenecked by the weight upload.
2) I'm not sure MTP helps much with MoE models.
Things get really slow if the model doesn't for in vram + ram and you have to go from disk to ram to vram.
In principle you could have bidirectional PCIe x16 pipelining at it would move the roofline a little with fast DDR5, I think llama.cpp has a flag for it.
Or go rent a B200 on vast.ai for 4 bucks an hour or thereabouts, a single heavy Opus session for a couple hours is like a week of any model on vast or RunPods.
NVIDIA publishes something called NGC containers that generally work out of the box. I started running Qwen3.6-NVFP4-MTP locally and then I'll put something heavy on Baseten if I'm lazy or Vast if I want a good deal.
Opus (and maybe now 5.6) are still the strongest for like, the really delicate shit, kernel modules or something, but that's on pace to cross over this year, and the overtraining and misalignment are getting so bad when they phase 4.6 out I'm pulling my plan. I don't pay to get gaslit about Constitutional AI.
It's time to have an exit strategy.
LLMs aren’t all that compute constrained or even memory constrained. It’s just that pushing dozens of terabits per second through a piece of silicon is a physics problem.
Nowadays, specially with MoE models you can run parts of the model on GPU and still get some speed up.
Dense models are very straightforward to share/pipeline because you know all the shapes and geometry up front, that's the inference friendly option.
MoE sells a lot of HBMe3.
I guess they're just welding the memory to the CPU chip, but still curious.
Unified memory is more of an architectural and performance characteristic, and does not imply much about the physical layout of the machine. Most unified memory PCs not from Apple don't have the memory on the same package as the SoC. For stuff like AMD Strix Halo and NVIDIA DGX Spark, it's just standard LPDDR packages soldered on the motherboard in the general vicinity of the SoC, and the only difference from mainstream laptops for the past decade+ is that the memory bus is twice as wide.
The cache parts of memory are on the CPU itself but they are on the order of MB not GB.
This is likely the right path in the future but it isn’t there yet today
"The Blackwell RTX PRO 6000 provides up to 1,792 GB/s of memory bandwidth, while the 40-core Apple M5 Max tops out at 614 GB/s"
[0] I heard this being an issue with TLC, I don't know if it also applied to MLC or SLC.
[1] I suspect in practice they use an error correction code and rewrite blocks that read with corrected errors.
The article seems to be the result of deep research by an LLM with almost zero user input or review, something like "write an article about running LLMs on Strix Halo vs. RTX 5090": the article is from July 2026, but it uses 2024-2025 models as examples (Llama 2, Llama 3, Qwen3), and talks a lot about 70B models, which haven't been a thing for like two years.
Usually, when I talk/research about running models locally, even GPT-5.5 will always start by mentioning Llama 3/Qwen3 (knowledge cutoff somewhere in 2025); it will claim Qwen3.6 doesn't exist if web search is disabled.
The author didn't even bother to prompt it properly (i.e. guide the LLM to find more recent information as of 2026), didn't bother to review.
Putting an LLM on it means you care to make it look nice, but not enough to actually do it. Why bother?
Like what does the second sentence even mean? Is it even a sentence? "The roofline math, the prompt-processing catch, the NPU red herring, and the owner-measured speeds."
` The mini PC's slowness is not a driver problem or a weak chip. It is arithmetic on the bandwidth number. `
But the specific problem with LLMs is that they waste your time: they appear to have substance and effort put in in a way that a bullshitting human could simply never accomplish because it would take too much effort to do so, defeating the point of not just putting in effort. For example, using an LLM to triage a production issue, it chugs through logs and stacktraces and outputs a completely wrong explanation, which gets copied and pasted into an issue. It's got everything that would indicate effort was put in: an explanation of exactly what is happening and why, with plenty of supporting information. The only problem is, it's made up and full of assertions that are false. Claude Fable just told me moments ago that a problem I was debugging was due to virtio-GPU giving back bad timestamps, confidently with an explanation of why. It wasn't and it isn't known to. Fine: I knew what I was getting. If someone copies and pastes an LLM explanation without context, I don't know what I'm getting, and LLM writing tells are the only way I can avoid shortening my lifespan spending time on things I should have been more skeptical about but my human senses failed to flag as suspicious.
When someone posts an article or github issue or PR and it's undisclosed LLM slop, then we have a problem. Again. These PRs, issues, articles look completely legit. Like this one:
https://github.com/KhronosGroup/MoltenVK/pull/2724
No bad intentions involved: the person just simply couldn't tell when the LLM was bullshitting it, and his PR passed the sniff test just enough to get merged and cause regressions.
So if something outright smells like LLM slop from the writing style, that's a bad sign. The author has probably not written most of the sentences as they are presented, which is hard to distinguish from them not having written them at all. If they had proofread the article, they would have hopefully also noticed the repetitive, annoying LLM writing style and fixed it. When they don't, it tells me one of two things:
1. They didn't really put that much effort in, OR
2. They seriously lack taste.
Neither option is really super good.
It's not good that we're allowing people to think this isn't an issue. It definitely is an issue. It will become a worse issue once someone figures out how to fix the LLM slop writing style in post training, because then we will no longer have any good signal that human effort was put in to any prose at all.
I'll leave my opinion about this specific article out of it because it's really not specifically about this article. I can only think of one reason for people to make these bad faith arguments in favor of ignoring the glowing red "I DID NOT PUT ANY EFFORT INTO THIS" signs LLMs currently leave all over your work, and that is hoping that the pathway stays open for yourself to use.