I hung out on reddit long enough over the previous couple of years when people were up in arms to leave. It wasn't the lack of subs or community size that kept people away. It was simply that it was harder to figure out how to get up and going. You can't just go to lemmy.com, create a name and password, and start doing stuff. Further still is that now people want an apk for phone browsing and particularly when the masses wanted to leave reddit, there was also no "use this apk and its easy". Plus, creating an instance is much more work than creating a subreddit.
It was never about the size of the website already appearing to be in place. Lemmy just has a harder entry fee. It keeps lemmy at a lower user base in the same way every subscription service in existence knows it wants to make things super easy to sign up, but time intens8ve and difficult to cancel. Because it takes a bit of effort, lots of people don't get around to doing it.
This is great for people looking now, but the info, the apks, and the knowledge to get there was less known or not very good a year or so ago.
Also, I'm personally a big fan of "thunder" for my phones lemmy apk. It's awesome.