There's also llamafile, super simple: download and run it.
this post was submitted on 22 Mar 2024
68 points (92.5% liked)
Selfhosted
40329 readers
421 users here now
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
-
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
-
No spam posting.
-
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
-
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
-
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
-
No trolling.
Resources:
- selfh.st Newsletter and index of selfhosted software and apps
- awesome-selfhosted software
- awesome-sysadmin resources
- Self-Hosted Podcast from Jupiter Broadcasting
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
founded 1 year ago
MODERATORS
Not as cool or flexiable though
Iirc it can run anything llama.cpp can because it just uses that under the hood.
Except you can't control it as easily. I like the UI and toolset of open web UI
Ok. I haven't tried it so I'll take your word for it. I'm just offering an easier alternative since the topic was "getting started"
Personally I've really enjoyed text-generation-webui. It made it really easy to ramp up and learn. Very cool stuff you got though, I'll probably be looking at a comparison between them!
Does it work out okay with 12 cores purely on CPU? About how fast is the interaction?
I played around a little with Ollama and gpt4all but it seemed to me like it wasn't fast enough to be useful on pure CPU, but if I could just throw cores at it then I might revisit the issue.
It wasn't usable a few months ago. However, when I setup ollama it was "fast" and it works ok. It takes anywhere from instant to 5min for responses. LLava seems to take the longest which makes sense. For llama2 it is fairly fast unless you ask it for obscure information.
For the life of me can't remember the scores I was getting on gpt4all. But given that you tried it I'm guessing you'll most likely take a liking to LM studio or perhaps jan.ai. both GUI tools. If the lack open source bothers you go for Jan.ai, if not then go LM studio. LM studio in particular allows for full and partial GPU offloading. So if you have a semi capable but not quite enough vram on it you can load part of the model on the GPU to speed up inference. As a side note pure CPU on my old ryzen 1600 I was looking at 6/it. Which isn't all that much but glass half full its still faster than the average typing speed and takes the load off of having to think about how to creatively word things
The biggest thing that I want to learn is how to either A: add "tools" for the AI to run, or B: "fine-tune" the model by feeding it data that's relevant to me.
You can teach it things and upload documents for it to process
Yeah, I couldn't find that on ollama; but I did find it in text-generation-webui - which is a little more complicated, but for me, I think it might help springboard me into understanding a few more things.
Ollama is just a the backend. You need open web UI or a similar application to use it
Check AnythingLLM out, its just an appimage
Not as maintainable long term and it doesn't have user management
There's a dockerized version if you need those
https://github.com/Mintplex-Labs/anything-llm/blob/master/docker/HOW_TO_USE_DOCKER.md
So why is it better than OpenwebUI? It seems like each has there own use case.
I'll give it a try just for fun but it doesn't seem to be better as far as I can tell
No idea if its better, its the thing I tried and it was pretty seamless to set up. With my aging hardware and AMD GPU, I have been pretty much sitting in the sidelines with this whole LLM thing
Ollama has been great for self-hosting, but also checkout vLLM as its the new shiny self-hosting toy
Thanks for sharing this. Its a shame that most AI tech is hidden behind steep price tags and cloud subscriptions, while even midrange PCs can run interesting AI models.
Check out LM Studio. Totally free and very nice GUI. You can do some very cool things with it. That and Kobold AI. Very simple self hosting no need for configuration either. Download and open essentially.