I did it for a year or so, exactly the same thing, I have a bunch of websites and other services like matrix, mastodon, peertube running on that server.
I found out the hard way that Lemmy is really not optimized for single user instances on a small VPS. In the beginning it was OK but the more they optimized for big instances like Lemmy.world and .ml the word it ran with few resources.
In the end it was so bad that it would Hogg all the servers resources and bring the whole server down every couple of hour so that I had to hard reboot it.
The worst part was that this impacted alł my other services and websites which went down with it every couple of hours. I tried to get a beefier server but that only helped until the next update.
In the end I switched to Piefed and couldn't be happier. It uses a lot less resources as a single user instance so I could downgrade the server again and it's written in python so I can much easier help working on it, no need to compile anything, etc. And it has a cool template engine which let's you easily modify how it looks and feels.