LLMs are useful to provide generic examples of how a function works. This is something that would previously take an hour of searching the docs and online forums, but the LLM can do for very quickly, and I appreciate. But I have a library I want to use that was just updated with entirely new syntax. The LLMs are pretty much useless for it. Back to the docs I go! Maybe my terrible code will help to train the model. And in my field (marine biogeochemistry), the LLM generally cannot understand the nuances of what I'm trying to do. Vibe coding is impossible. And I doubt the training set will ever be large or relevant enough for the vibe coding to be feasible.
this post was submitted on 15 Aug 2025
327 points (96.8% liked)
Technology
74292 readers
4103 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 2 years ago
MODERATORS
Vibe coding
The term for that is actually 'slopping'. Kthx ;-)
Thats simply not true. LLMs with RAG can easily catch up with new library changes.
Subjectively speaking, I don't see it so that good a job of being current or priortizing current over older.
While RAG is the way to give LLM a shot at staying current, I just didn't see it doing that good a job with library documentation. Maybe it can do all right with tweaks like additional properties or arguments, but more structural changes to libraries I just don't see being handled.
Exactly. It's an very niche library (tmap for R) and just was completely overhauled. Gemini, chatGPT and Copilot all seem pretty confused and mix up the old and new syntax
Thats a lot on implementation of the LLM engine . For python or js you can feed the API schema of the entire virtual environment.
Yes and no.
In many cases (like for the Gradle DSL, that even if it can be either the old Groovy-based one or the new Kotlin-based one, you will always be able to find extensive documentation and examples in the wild for both of them) it is sufficient to specify which version you're using and, as long as this doesn't get too far in its context window forcing you to repeat it, you are good to go.
But for niche libraries that have recently undergone significant refactors with the majority of the tutorials and examples still built with past versions, they have a huge bias towards the old syntax, making it really difficult - if not impossible - to make them use the new functions (at least for ChatGPT and GitHub Copilot with the "Web search" functionality on).
You can't know without checking though, it may be wrong
Note: this comes from someone that makes a (very good) ide which they only monetize with an AI subscription so it's interesting to see their take
(They use Claude opus like all the others so the results are similar)
I think AI in you IDE is meant to help you with small things while AI agents are supposed to do development for you. If people will start using AI agents they won't need IDEs so this take is consistent with their business model.
in one regard I can understand, they're running a business and don't want to be at a disadvantage against their competition.
on the other hand have some conviction for your product, otherwise I will lose confidence that your product is as good as your marketing makes it seem.
They are still bullish on LLM, just to augment rather than displace human suggested development.
This perspective is quite consistent with the need for a product that manages prompting/context for a human user and helps the human review and integrate the LLM supplied content in a reasonable way.
If LLM were as useful as some of the fanatics say, you'd just use a generic prompt and it would poop out the finished project. This is by the way the perspective of an executive I talked to not long ago, that he was going to be able to let go of all his "coders" and feed his "insight" directly into a prompt that will do it all for him instead. He is also easily influenced so articles like this can reshape him into a more tenable position, after which he'll pretend he never thought a generic prompt would be good enough
Interesting what he wrote about LLMs' inability to "zoom out" and see the whole picture. I use Gemini and ChatGPT sometimes to help debug admin / DevOps problems. It's a great help for extra input, a bit like rubberducking on steroids.
Examples how it went:
Problem: Apache-cluster and connected KeyCloak-Cluster, odd problems with loginflow. Reducing KeyCloak to 1 node solves it, so it says that we need to debug node communication and how to set the debug log settings. A lot of analysis together. But after a while, it's pretty obvious that the Apache-cluster doesn't use the sticky session correctly and forwards requests to the wrong KeyCloak node in the middle of the login flow. LLM does not see that, wanted to continue to dig deeper and deeper into supposedly "odd" details of the communication between KeyCloak nodes, althought the combined logs of all nodes show that the error was in load balancing.
Problem: Apache from a different cluster often returns 413 (payload too large). Indeed it happens with pretty large requests, the limit where it happens is a big over 8kB without the body. But the incoming request is valid. So I ask both Gemini and ChatGPT for a complete list of things that cause Apache to do that. It does a decent job at that. And one of it is close: It says to check for mod_proxy_ajp use, since that observed limit could be caused by trying to make an AJP package to communicate with backchannel servers. It was not the cause; the actual mod was mod_jk, which also uses AJP. It helped me focus on watching out for anything using AJP when reviewing the whole config manually, so I found it, and the "rubberducking" helped indirectly. But the LLM said we must forget about AJP and focus on other possible causes - a dead end. When I told it the solution, it was like: Of course mod_jk. (413 sounds like the request TO the apache is wrong, but actually, it tries internally to create an invalid AJP package over 8kB, and when it fails blames the incoming request.)
LLMs have made it really clear when previous concepts actually grouped things that were distinct. Not so long ago, Chess was thought to be uniquely human, until it wasn't, and language was thought to imply intelligence behind it, until it wasn't.
So let's separate out some concerns and ask what exactly we mean by engineering. To me, engineering means solving a problem. For someone, for myself, for theory, whatever. Why do we want to solve the problem, what we want to do to solve the problem, and how we do that often blurred together. Now, AI can supply the how in abundance. Too much abundance, even. So humans should move up the stack, focus on what problem to solve and why we want to solve it. Then, go into detail to describe what that solution looks like. So for example, making a UI in Figma or writing a few sentences on how a user would actually do the thing. Then, hand that off to the AI once you think it's sufficiently defined.
The author misses a step in the engineering loop that's important though. Plans almost always involve hidden assumptions and undefined or underdefined behavior that implementation will uncover. Even more so with AI, you can't just throw a plan and expect good results, the humans need to come back, figure out what was underdefined or not actually what they wanted, and update the plan. People can 'imagine' rotating an apple in their head, but most of them will fail utterly if asked to draw it; they're holding the idea of rotating an apple, not actually rotating the apple, and application forces realization of the difference.
The author misses a step in the engineering loop that's important though. Plans almost always involve hidden assumptions and undefined or underdefined behavior that implementation will uncover.
His whole point is two mental models and a model delta. Exactly what you just described.
I've done a test of 8 LLMs, on coding. It was using the J language, asking all of them to generate a chess "mate in x solver"
Even the bad models were good at organizing code, and had some understanding of chess, were good at understanding the ideas in their prompts. The bad models were bad mostly on logic. Not understanding indexing/amend on a table, not understanding proper function calling, or proper decomposition of arguments in J. Bad models included copilot and openAI's 120g open source model. kimi k2 was ok. Sonet 4 the best. I've mostly used Qwen 3 245 for better free accessibility than Sonet 4, and the fact that it has a giant context that makes it think harder (slower) and better the more its used on a problem. Qwen 3 did a good job in writing a fairly lengthy chess position scoring function, and then separating it into 2 quick and medium function, incorporating self written library code, and recommending enhancements.
There is a lot to get used to in working with LLMs, but the right ones, can generally help with code writting process. ie. there exists some code outputs which even when wrong, provide a faster path to objectives than if that code output did not exist. No matter how bad the code outputs, you are almost never dumber for having received it, unless perhaps you don't understand the language well enough to know its bad.
I don't work in IT, but I do know you need creativity to work in the industry, something which the current LLM/AI doesn't possess.
Linguists also dismiss LLMs in similar vein because LLMs can't grasp context. It is always funny to be sarcastic and ironic on an LLM.
Soft skills and culture are what that the current iteration of LLMs lack. However, I do think there is still huge potential for AI development in dacades to come, but I want this AI bubble to burst as "in your face" to companies.
view more: next ›