Self-hosted media server to share with 5+ people?

Hi, I'm fairly new to the self-hosted universe but I like the idea of self-hosting media (I've looked at Jellyfin and Plex). But as I understand this requires quite some money and a lot of work. I don't think it's worth it if I put in all that effort just for myself but I'd love to build a small private streaming between me and my friends. We used to share and swap blu-rays after all, so it would be cool to build a shared collection.

My question is if that's possible and if anyone has experience with this? I've read that Jellyfin and Plex are meant as home-media-servers and I'm not sure what limitations that implies. Can people access the library from outside networks and will that affect the streaming quality/speed? What specs would the server need to ensure it can handle a bunch of users? Is there a software that is better suited for this use-case?

Thanks in advance for any help!