Selfhosted

For RAG, there are some tools available in open-webui, which are documented here: https://docs.openwebui.com/tutorials/features/rag They have plans for how to expand and improve it, which they describe here: https://docs.openwebui.com/roadmap#information-retrieval-rag-

For fine-tuning, I think this is (at least for now) out of scope. They focus on inferencing. I think the direction is to eventually help you create/manage your own data which you get from using LLMs using Open-WebUI, but the task of actually fine-tuning is not possible (yet) using either ollama or open-webui.

I have not used the RAG function yet, but besides following the instructions on how to set it up, your experience with RAG may also be somewhat limited depending on which embedding model you use. You may have to go and look for a good model (which is probably both small and efficient to re-scan your documents yet powerful to generate meaningful embeddings). Also, in case you didn't know, the embeddings you generate are specific to an embedding model, so if you change that model you'll have to rescan your whole documents library.

Edit: RAG seems a bit limited by the supported file types. You can get it here: https://github.com/open-webui/open-webui/blob/2fa94956f4e500bf5c42263124c758d8613ee05e/backend/apps/rag/main.py#L328 It seems not to support word documents, or PDFs, so mostly incompatible with documents which have advanced formatting and are WYSIWYG.

Increase context length, probably enable flash attention in ollama too. Llama3.1 support up to 128k context length, for example. That's in tokens and a token is on average a bit under 4 letters.

Note that higher context length requires more ram and it's slower, so you ideally want to find a sweet spot for your use and hardware. Flash attention makes this more efficient

Oh, and the model needs to have been trained at larger contexts, otherwise it tends to handle it poorly. So you should check what max length the model you want to use was trained to handle

There are not that many use cases where fine tuning a local model will yield significantly better task performance.

My advice would be to choose a model with a large context window and just throw in the prompt the whole text you want summarized (which is basically what a rag would do anyway).

Someone recently referred me to this blog post about using RAG in open-webui. I have not tested if but the author seems to reach a good setup.

https://medium.com/@kelvincampelo/how-ive-optimized-document-interactions-with-open-webui-and-rag-a-comprehensive-guide-65d1221729eb

Perhaps this is of use to you?

I wish I could. I have an RTX 3060 12GB, I run mostly llama3.1 8B versions in fp8, at 30-35 tokens/s.

I made a robot which is delighted about the idea of overthrowing capitalism and will enthusiastically explain how to take down your government.

Interesting, I see that is pretty new. Some of the documentation must be out of date because it definitely said Nvidia only somewhere when I tested it about a month ago. Thanks for giving me hope!

Its an app...

https://flathub.org/apps/com.jeffser.Alpaca

Yes, but is a copy of something that is already done. I'm not making new requests to a data center who is wasting 4 liters of water every 100 words like gpt-4, I'm just using my GPU like with a videogame.