this post was submitted on 25 Apr 2026
394 points (97.8% liked)
Technology
84143 readers
2451 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Yeah, I'd expect KTransformers to add support eventually, especially considering their existing support for previous DeepSeek models. One of the tricky parts is that backends need both FP8 and MXFP4 support. As far as I'm aware no inference engine supports both on CPU at the moment (llama.cpp added fp4 support recently, but doesn't have fp8, while kt-kernel doesn't support fp4 yet).
Not to speak of the new attention scheme and the (IIRC) MLP changes.
I’m very much looking forward to ik_llama.cpp implementing it. I don’t think I can quite fit Flash on my rig (hence no Ktransformers for me) but a little quantization of the sparse layers, and it’d be perfect.
Makes sense, even Flash is fairly sizable! KTransformers also has a "llamafile" backend which uses GGUFs, but ik_llama will almost certainly perform better if you're not on a NUMA setup. In my case, I'm using a dual socket motherboard, so KTransformers performs quite a bit better (I think ik_llama hasn't implemented extensive NUMA optimizations quite yet, but sounds like it's coming), though I normally use KTransformers for native FP8 weights.
It is! 143GB last I checked. I'm on 128GB RAM + 3090, 1 NUMA node, so I think it's juuust barely too tight. But it should be perfect with a few of the "sparsest" MoEs quantized.
If KTransformers supports something like that, I may have to finally check it out, since v4 won't need many esoteric features.