Looks like you're on Fedora Silverblue (or other Atomic version). This is happening because the system groups are in /usr/lib/group rather than /etc/group and this causes the issue you're seeing here. You can work around it by getting into a root shell with something like
sudo -i
and then getting the group added to /etc/group with
grep -E '^dialout' /usr/lib/group >> /etc/group
after that, you'll be able to add your user to the group with
/etc is writable, so no reboots are required. That said, /etc is treated in a special way and each deployment will have its own /etc, based on the previous one.
So if you make changes to /etc then revert to a previous deployment, your changes will be reverted as well. But if you make changes and upgrade (or do whatever to create a new deployment), your changes will bu preserved.
It's like when I run into some issue with how I've set up my system in NixOS and have to explain to a non-Linux user that it isn't Linux that's the issue but how I'm using an especially weird Linux lol