I did this a few years ago for fun. It was a horrible idea, save for one singular use-case: tiny, few layers, single arch, and open-source.
The TLDR from my own experience: auth won't work, complex images won't work reliably, push won't work, multiarch breaks, inspect breaks, meaning things like Helm won't work, and most clients won't work. Its basically just for show, and shouldn't be done.
I'm sure I'm missing a few things, but for practical purposes, it's not useful unless you're using it for something hyper specific, at which point you may as well use the 'export' function to ship containers everywhere, which of course removes all the convenience of the container workflow.
Sure, it works, and maybe download speeds for the file are faster, but that's it. Everything else is broken, and then you can't simply do a "docker pull" to update again in the future.