I've got a 3 node Proxmox/ceph cluster with 10G, plus a separate Nas. They are all rack mount with dual PSU. Add in the necessary switching, and my average load is about 800w. Throw my desktop (also on 10G) into the mix and it runs 1.1kw.
That's roughly $50-60 extra in electricity costs for me monthly.
I'm a Sysadmin, so my names are purely functional:
host-pmx-01 through 03, my 3 node Proxmox cluster
vm-[SERVICE], optional 01-03 if needed
ct-[SERVICE], for LXC containers
It makes it easy to reference things via DNS for service discovery.
Average load for me is about 750W. I run my desktop from one of the UPS units in my rack, so when that's on it sits around 1.1kW.
The 750W load is across 4 rack servers(1 is the NAS with 12 disks) and 3 switches.
Unless someone has physical access to the ports/switch that the traffic flows through, they would not be able to see anything besides broadcast/multicast traffic if they were just snooping with Wireshark. The internal switch of proxmox and any hardware switch you have will forward unicast traffic to the ports those Mac's reside on, so without port mirrors setup, no one but you should be able to see that traffic.
My 10G is far from saturated, but I do try and keep things using RAM where possible. I figure that with 100gb of DDR4 in my main server, that should be able to provide enough speed for a 10G link.
I've got ceph running on Intel Enterprise SSDs, so they are pretty quick.
I also tried running ceph on 1G. I found it unreliable as well.