158
Bill Gates feels Generative AI has plateaued, says GPT-5 will not be any better
(indianexpress.com)
this post was submitted on 02 Dec 2023
158 points (85.3% liked)
Technology
59374 readers
7261 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Cool, Bill Gates has opinions. I think he's being hasty and speaking out of turn and only partially correct. From my understanding, the "big innovation" of GPT-4 was adding more parameters and scaling up compute. The core algorithms are generally agreed to be mostly the same from earlier versions (not that we know for sure since OpenAI has only released a technical report). Based on that, the real limit on this technology is compute and number of parameters (as boring as that is), and so he's right that the algorithm design may have plateaued. However, we really don't know what will happen if truly monster rigs with tens-of-trillions of parameters are used when trained on the entirety of human written knowledge (morality of that notwithstanding), and that's where he's wrong.
Yeah and I think he may be scaling to like true AGI. Very possible LLMs just don't become AGI, you need some extra juice we haven't come up with yet, in addition to computational power no one can afford yet.
Except that scaling alone won't lead to AGI. It may generate better, more convincing text, but the core algorithm is the same. That "special juice" is almost certainly going to come from algorithmic development rather than just throwing more compute at the problem.
See my reply to the person you replied to. I think you're right that there will need to be more algorithmic development (like some awareness of its own confidence so that the network can say IDK instead of hallucinating its best guess). Fundamentally though, llm's don't have the same dimensions of awareness that a person does, and I think that that's the main bottleneck of human-like understanding.