The problem is that it's so inert, it becomes impossible to remove from a contaminated environment or particularly a person's blood stream.
The amount that inevitably gets caught in your blood will just stay in your body forever, settling down in a critical organ like your brain or kidneys and giving you cancer or some other horrible problem.
EPA article on PFAS
Much more in depth NIH article (go to section 3)
We are definitely NOT in theory territory here, there have been studies and reports of people directly suffering from the effects of PFAS, what we are not clear on is how it's interacting with our bodies.