I ran an ejabberd node on an old x86 for years for family and some close friends. Works great.
Then I got tired of maintaining devices after long days at work doing IT things. We talked. Signal is easier. We moved over to that, in the end.
A Pi3 1GB will easily scale to 4 people.and beyond. XMPP is really lightweight for text and images. Consider a Pi4 for voice or video though.