Ask Lemmy

This is a pet peeve of mine right up there with the never ending stream of people calling machine learning AI. We do not have any real kind of AI at all at the moment but I digress.

LLM is literally just a probability engine. LLM's are trained on huge libraries of content. What they do is assign a token(id) to each word (or part of word) and then note down the frequency of the words before and after the word as well as looking specifically for words that NEVER come before or after the word in question.

This creates a data set that can be compared to other tokenized words. Words with vary similar data sets can often be replaced with each other with no detriment to the sentence being created.

There is something called a transformer that has changed how efficiently LLM'S work and has allowed parsing of larger volumes by looking at the relation of each tokenized word to every word in the sentence simultaneously instead of one at a time which generates better more accurate data.

But the real bread and butter comes when it starts generating new text it starts with a word and literally chooses the most probable word to come next based off of its extensive training data. It does this over and over again and looks at the ending probability of the generated text. If it's over a certain threshold it says GOOD ENOUGH and there is your text.

You as a human (I assume)do this kind of thing all ready. If someone walked up too you and said "Hi! How are you..." by the time they got there you have probably already guessed that the next words are going to be "doing today?" Or some slight variation thereof. Why were you able to do this? Because of your past experiences, aka, trained data. Because of the volume of LLM'S data set it can guess with surprisingly good accuracy what comes next. This however is why the data it is trained on is important. If there were more people writing more articles,more papers,more comments about how the earth was flat vs people writing about it being round then the PROBABLE outcome is that the LLM would output that the earth is flat because that's what the data says is probable.

There are variations called the Greedy Search and the Beam Search but they are difficult for me to explain but still just variations of a probability generator.