Why did UI's turn from practical to form over function?
E.g. Office 2003 vs Microsoft 365
It's easy to remember where everything is with a toolbar and menu bar, which allows access to any option in one click and hold move.
Seriously? Big ribbon and massive padding wasting space, as well as the ribbon being clunky to use.
Why did this happen?
The bulk of these aren't issues with modern design, IMO, it's about enshittification of the services we use.
Having huge spaces for ads, for example, isn't a "this is how UX should be" thing, it's a "lets shove ads everywhere to make money" thing. If you put the same amount of ads in older software/on sites that look like they're from 2002, it would also look terrible.
The Windows start menu isn't bad because it has some padding and easier click targets, it's bad because the search doesn't work, it's full of ads, and pushes Bing searches on you.
Etc.
Yes they are, UX designers are not asked to make more efficient or usable designs, they are asked to make designs that "look good" in marketing, support ad integration, hook people into others services provided by that same company, make it more difficult to incorporate with workflows that include third-party applications, etc.
This is deliberate UX design, which is part of the enshittification process.
You are thinking of an entirely different thing.
If you put the same amount of ads in software that looks like it's from the 90s, do you still think you'd like that 90s software? Of course you wouldn't.