this post was submitted on 04 Jan 2024
295 points (90.4% liked)
Linux
48328 readers
626 users here now
From Wikipedia, the free encyclopedia
Linux is a family of open source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991 by Linus Torvalds. Linux is typically packaged in a Linux distribution (or distro for short).
Distributions include the Linux kernel and supporting system software and libraries, many of which are provided by the GNU Project. Many Linux distributions use the word "Linux" in their name, but the Free Software Foundation uses the name GNU/Linux to emphasize the importance of GNU software, causing some controversy.
Rules
- Posts must be relevant to operating systems running the Linux kernel. GNU/Linux or otherwise.
- No misinformation
- No NSFW content
- No hate speech, bigotry, etc
Related Communities
Community icon by Alpár-Etele Méder, licensed under CC BY 3.0
founded 5 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
I have nothing against the people that are working on AI and appreciate the work they do. However every time I see an article about a company using AI like this I just get the vibe that it's a bunch of middle aged men trying desperately to make things like the "future" they saw when they were a kid. I've seen amazing implementations of AI in a lot of different ways but I'm so sick of dumb ideas like this because some guy that used to watch Star Trek as a kid wants to feel like they live in the future while piggybacking on someone else's work. It's like the painted tunnel in cartoons where it looks like a real tunnel but in reality it's just a very convincing lie. And that's all that it is. Complexity does not mean sophistication when it comes to AI and never has and to treat it as such is just a forceful way to make your ideas come true without putting in the real effort.
Sorry, I had to get that out. Also I have nothing against Star Trek and I used to watch it as a kid because my parents watched it all the time.
It's a bit off-topic, but what I really want is a language model that assigns semantic values to the tokens, and handles those values instead of directly working with the tokens themselves. That would be probably far less complex than current state-of-art LLMs, but way more sophisticated, and require far less data for "training".
I'm not sure I understand. Do you mean hearing codewords triggering actions as opposed to trying to understand the users intent through language? Or is are there a few more layers to this whole thing than my moderate nerd cred will allow me to understand?
Not quite. I'm focusing on chatbots like Bard, ChatGPT and the likes, and their technology (LLM, or large language model).
At the core those LLMs work like this: they pick words, split them into "tokens", and then perform a few operations on those tokens, across multiple layers. But at the end of the day they still work with the words themselves, not with the meaning being encoded by those words.
What I want is an LLM that assigns multiple meanings for those words, and performs the operations above on the meaning itself. In other words the LLM would actually understand you, not just chain words.
Semantic embeddings are a thing. LLMs "work with tokens" but they associate them with semantic models internally. You can externalize it via semantic embeddings so that the same semantic models can be shared between LLMs.
The source that I've linked mentions semantic embedding; so does further literature on the internet. However, the operations are still being performed with the vectors resulting from the tokens themselves, with said embedding playing a secondary role.
This is evident for example through excerpts like
Emphasis mine. A similar conclusion (that the LLM is still handling the tokens, not their meaning) can be reached by analysing the hallucinations that your typical LLM bot outputs, and asking why that hallu is there.
What I'm proposing is deeper than that. It's to use the input tokens (i.e. morphemes) only to retrieve the sememes (units of meaning; further info here) that they're conveying, then discard the tokens themselves, and perform the operations solely on the sememes. Then for the output you translate the sememes obtained by the transformer into morphemes=tokens again.
I believe that this would have two big benefits:
And it might be an additional layer, but the whole approach is considerably simpler than what's being done currently - pretending that the tokens themselves have some intrinsic value, then playing whack-a-mole with situations where the token and the contextually assigned value (by the human using the LLM) differ.
[This could even go deeper, handling a pragmatic layer beyond the tokens/morphemes and the units of meaning/sememes. It would be closer to what @njordomir@lemmy.world understood from my other comment, as it would then deal with the intent of the utterance.]