I'm very much biased towards Podman, but from what I understand rootless Docker is a bit of an afterthought, while Podman has been developed from the ground up with rootless in mind. That should be reason enough.
The very few things Docker can do that Podman struggles a bit with are stuff that usually involves mounting the Docker socket in the container or other stupid things. Since you care about security, you wouldn't do that anyway. Not to mention there's also rootful Podman, when you need that level of access.
I'd recommend an RPM-based distro with Podman, the few times I've tried Podman on a deb distro, there's always been something wonky. It's been a while, though.