Before you go too far into it and spend lots of time, I think most VPS services let you installed a new OS on their admin site so you can start again from scratch. If you're not sure that is the right linux flavour, go for something else more mainstream so you can find lots of support online. Looking at the OS, I'm sure it might be good but I'm also sure you can install all the features very easily yourself, especially if it's just using docker mainly.
I second UFW. I found this guide useful: https://www.digitalocean.com/community/tutorials/ufw-essentials-common-firewall-rules-and-commands. You might want to try tailscale as others use it for easily setting up vpn access but not used it myself. Also go for fail2ban or, for more assurance but harder work, try crowdsec too.
You could also use cloudflare dns and add IP and/or country restrictions to block all traffic before it gets to your VPS. I have a country filter and it's crazy how many bots get blocked from all over!