Self hosting Stoat is a nightmare at the moment.
Only the webUI works out of the box, if you want the phone app you need to compile it yourself.
At least the desktop app now supports connecting to custom instances, but it's by launch option, not the GUI.
All that said, my understanding was that you give the code to your friends, and they have to enter it during signup.
Enabling invite-only doesn't remove the signup functionality, it adds authentification, so only people with the invite code can pass.