Technology

This has been predicted and worked towards since the 90s.

ChromeOS was exactly mean for that

Maybe we need a small, private company to come along and start making good consumer hardware.

I've always wanted to start a business like this. "Generic Brand" household goods. Not fancy, just solidly functional base models but with modular upgradability. Wish you bought the WiFi capable washer? Buy the module for $30. Everything would be fully user serviceable and upgradable (within reason), so parts sales ensure sustained income once market saturation is reached.

Ai failed and now they are doing this to capture the compute market to then make their profit back through unscrupulous means.

As I am told, there is no way these llm's ever make their investments back. It's like Tesla at this point. Whomever is paying the actual money to build this stuff is going to get hosed if they can't offload it onto some other sucker. That ultimate sucker probably being the US taxpayer.

Honestly just jump in with whatever hardware you have available and a small 1.5b/7b model. You'll figure out all the difficult uncertainties as you go and try to improve things.

I'm hosting a few lighter models that are somewhat useful and fun without even using a dedicated GPU- just a lot of ram and fast NVMe so the models don't take forever to spin up.

Of course I've got an upgrade path in mind for the hardware and to add a GPU but there are other places I'd rather put the money atm and I do appreciate that it all currently runs on a 250w PSU.

This is not true. I have a single 3090 + 128GB CPU RAM (which wasn’t so expensive that long ago), and I can run GLM 4.6 350B at 6 tokens/sec, with measurably reasonable quantization quality. I can run sparser models like Stepfun 3.5, GLM Air or Minimax 2.1 much faster, and these are all better than the cheapest API models. I can batch Kimi Linear, Seed-OSS, Qwen3, and all sorts of models without any offloading for tons of speed.

…It’s not trivial to set up though. It’s definitely not turnkey. That’s the issue.

You can’t just do “ollama run” and expect good performance, as the local LLM scene is finicky and highly experimental. You have to compile forks and PRs, learn about sampling and chat formatting, perplexity and KL divergence, about quantization and MoEs and benchmarking. Everything is moving too fast, and is too performance sensitive, to make it that easy, unfortunately.

EDIT:

And if I were trying to get local LLMs setup today, for a lot of usage, I’d probably buy an AI Max 395 motherboard instead of a GPU. They aren’t horrendously priced, and they don’t slurp power like a 3090. 96GB VRAM is the perfect size for all those ~250B MoEs.

But if you go AMD, take all the finickiness for an Nvidia setup and multiply it by 10. You better know your way around pip and Linux, as if you don’t get it exactly right, performance will be horrendous, and many setups just won’t work anyway.

Can run decent size models with one of these: https://store.minisforum.com/products/minisforum-ms-s1-max-mini-pc

For $1k more you can have the same thing from nvidia in their dgx spark. You can use high speed fabric to connect two of ‘em and run 405b parameter models, or so they claim.

Point being that’s some pretty big models in the 3-4k range, and massive models for less than 10k. The nvidia one supports comfyui so I assume it supports cuda.

It ain’t cheap and AI has soooo many negatives, but… it does have some positives and local LLMs mitigate some of the minuses, so I hope this helps!

The whole economy reminds me more and more of the decline of the Roman Empire. Their biggest problem was that there were no consumers left to keep money in circulation and the economy afloat. You either owned lots of land and slaves that provided pretty much everything you need or you were a slave, meaning the only one you could trade with were merchants from outside the empire but as the empire expanded, those became harder to reach. War expenses spiraled out of control while the economy declined until it ceased to exist.

Now mega corps only trade with each other and threaten to replace all workers with AI and robots. Meanwhile the economy becomes stale, people buy less while politicians around the globe cut down the social sector, meaning people will have even less money to spare. Money won‘t circulate as much, slowing things down even further.

There are ways out of this spiral of decline but billionaires won‘t give up so easily. You can say many things about them but they are persistent as hell.

I am so glad to see someone else talking about this. Yeah, We're going back to feudalism... And only the upper ranks of society will be able to afford goods and be able to engage in trade.

This is an important point in general. The old story of "voting with your wallet" is now more and more obviously mathematically absurd.

Yet another lie of capitalism- that demand dictates supply. If people want something, capitalism will solve the problem because greed will solve every problem, right? WRONG. It doesn't take into consideration that scale and logistics and infrastructure and mass production all collude and interact in ways that can and will easily create isolated and unexpected functional consequences just like this one.

Capitalism itself is such a dumb and evil creation that the only thing that keeps it running is to have laws that limit its destructive power- unfortunately, since money is power, capitalism quickly supersedes rule of law and democracy, creating a system of government where feudal lords rule over what basically amounts to serfdom. Surely a more comfortable serfdom than in the 1300's, but most certainly lacking the basic freedoms of a democratic society, and only so long as the lords so wish and comfort isn't taken away at their whims due to opposition or otherwise.

Fuck this shit.

Hopefully Linux supported. That's the main selling point of AMD GPUs right now for me, since there's less problems with trying to get stuff like HDR running on it than NVIDIA.

I wonder why China is still for the most part ignoring Linux in favor of Windows. Like to update 8bitdo controllers they only provide a Microsoft program and no Linux version.

You'd think they'd be rushing towards pushing Linux adoption.

“banned for security concerns”