this post was submitted on 09 Sep 2024
296 points (99.0% liked)
Technology
58115 readers
4097 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
This checks out after what they recently did to the ZLUDA project. As an owner of an AMD gpu I agree that ROCM support is really bad. It works half of the time and fairly poorly.
Even worse on Linux. Even worse on more exotic distros like Bazzite where I still can't get koboldcpp to run, which was already kind of a hassle on my previous distro.
Yeah I hear ya. How do you happen to be running it? I use NixOS and its a challenge there for me but I found atleast some success using docker since the dependencies are so out of control for AI at the moment.
Also give Ollama-rocm + Open Webgui a shot as an alternative to koboldcpp if you cant get atleast some text generation to work because that is the only thing I have got working with rocm.
The trick to nixos, in this instance, is to use a python venv. Python dependencies are fickle and nasty in the first place, triply so when talking about fast-churning AI code, I tried specifying everything with nix, I succeeded, and then you have random comfyui plugins assuming they can get a writeable location by constructing a path from comfyui's main.py. It's not worth it: Let python be the only dependency you feed in, let pip and general python jank do the rest.