this post was submitted on 12 Dec 2023
-25 points (19.5% liked)
Technology
59374 readers
7113 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
LLMs have a a tendency to hallucinate: https://en.wikipedia.org/wiki/Hallucination_%28artificial_intelligence%29
As someone else stated, the AI can't reason. It doesn't understand what a unicorn is. It can't think "a unicorn has a singular horn, so a non existent two-headed unicorn would have two horns". Somewhere along the line it'll probably mix in a deer or a moose that has two horns, because the number two matches the number of horns per head statistically.
Last year, two lawyers in separate cases with different LLMs submitted hallucinated case citations. It would have been trivially simple for them to drop the case number into a proper legal search engine, but neither did. This is a similar issue: the LLM will also prioritize what you want to hear, so it does what it's designed to do and generate text related to your question. Like the unicorn example, it has no reasoning to say "any legal research should be confirmed by making a call to an actual legal database to confirm citations" like a human would. It's just scribbling words on the page that look like other similar words it knows. It can make case notes look real as heck because it has seen other case notes, but that's all it's doing. (please excuse the political news story, but it's relevant)
And it's not limited to unicorns or case notes. I found this reddit post while researching a feature of a software package (Nextcloud) several months ago. In the post, OP is seeking an option to pause the desktop client from the command line. Someone responds with a ChatGPT answer, which is quite hallucinated. Not only does such an option not appear in the documentation, there's an open bug report to the software devs to request that the feature be added. Two things easy for a reasoning human to do, but the AI is just responding with what you want to hear - documentation.
I've also seen ChatGPT tell my friend to use power shell commands that don't exist, and he has to tell the model twice to generate something new because it kept coming to the same conclusion.