HOLY HELL THAT'S COOL. It can do so much too!!!
I locally installed some small LLM model more than a year ago. It took up like 25 gigs or something along with all CUDA libraries n stuff. It was alright, but I figured that cloud based solutions were the best for my use case, as they were better and for free.
I had no idea that open sourced AI progressed so much in the last year. Amazing stuff!
I was using the quantized version :(
But again, do remember that this was when the first open sourced AI models had just begun to come out. Stuff from Open Assistant for example. I don't even remember the name of the model that I was running (it was just too weird and funny lol). I just remember it being HUGE, quite dumb and making my device sweat lol.