Technology
This is the official technology community of Lemmy.ml for all news related to creation and use of technology, and to facilitate civil, meaningful discussion around it.
Ask in DM before posting product reviews or ads. All such posts otherwise are subject to removal.
Rules:
1: All Lemmy rules apply
2: Do not post low effort posts
3: NEVER post naziped*gore stuff
4: Always post article URLs or their archived version URLs as sources, NOT screenshots. Help the blind users.
5: personal rants of Big Tech CEOs like Elon Musk are unwelcome (does not include posts about their companies affecting wide range of people)
6: no advertisement posts unless verified as legitimate and non-exploitative/non-consumerist
7: crypto related posts, unless essential, are disallowed
view the rest of the comments
Network latency will make distributed training a very time-consuming task.
Sure, but I don't think that's a show stopper since you don't need to do comprehensive training often. Also worth noting that stuff like LoRAs allow extending functionality of models without retraining from scratch. So, most training might be relatively small within a specific context.
You don't need to do it often, but initial training requires huge ressources and someone has to do it, if you want to create new models from scratch. And for this you need your compute packed as close as possible.
Not sure what your point is here. The whole point of stuff like Petals is to facilitate a way to do this by harnessing a lot of computers around the world. It would be slower than doing it in a data center, but it's not a show stopper if this is something that only needs to be done occasionally.
Sorry, I thought that we might be underestimating the factor of "slower", but I couldn't quickly find numbers to prove my point. I might be wrong after all. I wish you a good night. 😊