this post was submitted on 04 Apr 2024
122 points (87.2% liked)
Technology
59421 readers
2842 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
I have no idea how this is set up to work technically, but most of the heavy lifting is gonna be on the GPU. I'm not sure that it matters much whether the browser is what's pushing data to the GPU or some other package.
Most people probably don't have a dedicated GPU and an iGPU is probably not powerfull enough to run an LLM at decent speed. Also a decent model requires like 20GB of RAM which most people don't have.
It doesn't just require 20GB of RAM, it requires that in VRAM. Which is a much higher barrier to entry.
But what if you have an AMD APU. Doesn’t that use your normal RAM as VRAM?
Not exactly. Most integrated chips have a small pool of dedicated VRAM, and then a bit more that they share with the system memory, though it's generally only a portion, not all of it. It's only Apple's unified memory, and maybe other mobile chips that has them both share memory pool entirely, for better or worse, as far as I'm aware.
But it is worth noting that if you don't have enough VRAM and have to put it into RAM, the minimum expectation is that you have twice the amount of RAM space. So if you have a GPU with 4GB of VRAM, and need to offload the extra to the system, you don't need 16 GB, you need 32 GB.