After almost half a century, I'm still doing it...

Lol; I've done this too. Thankfully not to anything important.

I've done this kind of thing remotely in screen with ifdown eth0 ; sleep 10 ; ifup eth0 ;

Not SysAdmin but about a year into my first software engineer job I was working on the live DB in SQL without using BEGIN TRAN ROLLBACK TRAN.

Suffice to say I broke the whole system my making an UPDATE without a WHERE clause. Luckily we have regular backups but it was a lot of debugging with the boss before I realised it was me that caused the issue the client was reporting.

Lol I've locked myself out of so many random cloud and remote instances like this that now I always make a sleep chain or a kill timer with tmux/screen.

Usually like:

./risky_dumb_script.sh ; sleep 30 ; ./undo.sh

Or

./risky_dumb.script.sh

Which starts with a 30 second sleep, and:

(tmux) sleep 300 ; kill PID

I still prefer net-tools and use ifconfig eth0 up That ip mess I'd rather do without, and those funky UU device/interface names I wish them out of my system

By the way, what system/init/svc manager are you using? With 50y in your back, cron job to check if it is up and resetting it while you are away. You can always remotely cancel the cronjob ... but it will be a new mistake not the old one :)

I started on Irix and ultrix if you remember those, what would I know :)

A decade and change ago, in a past life, I was tasked with switching SELinux to permissive mode on the majority of systems on our network (multiple hundreds, or we might have gotten above one thousand at that point, I don't recall exactly). This was to be done using Puppet. A large number of the systems, including most of our servers, had already been manually switched to permissive but it wasn't being enforced globally.

Unfortunately, at that point I was pretty familiar with Puppet but had only worked with SELinux a very few times. I did not correctly understand the syntax of the config file or setenforce and set the mode to ... Something incorrect. SELinux interpreted whatever that was as enforcing mode. I didn't realize what I had done wrong until we started getting alerts from throughout the network. Then I just about had a panic attack when I couldn't login to the systems and suddenly understood the problem.

Fortunately, it's necessary to reboot a system to switch SELinux from disabled to any other mode, so most customer facing systems were not impacted. Even more fortunately, this was done on a holiday, so very few customers were there to be inconvenienced by the servers becoming inaccessible. Even more fortunately, while I was unable to access the systems that were now in enforcing mode, the Puppet agent was apparently still running ... So I reversed my change in the manifest and, within half an hour, things were back to normal (after some service restarts and such).

When I finally did correctly make the change, I made sure to quintuple check the syntax and not rush through the testing process.

edit: While I could have done without the assault on my blood pressure at the time, it was an effective demonstration of our lack of readiness for enforcing mode.

Every network engineer must lock themselves out of a node at some point, it is a rite of passage.

I was on-call and half awake when I got paged about a cache server’s memcached being down for the third time that night. They’d all start to go down like dominoes if you weren’t fast enough at restarting the service, which could overwhelm the database and messaging tiers (baaaaad news to say the least). Two more had their daemon shit the bed while I was examining it. Often it was best to just kick it on all of them to rebalance things. It was… not a great design.

So I wrote a quick loop to ssh in and restart the service on each box in the tier to refresh them all just in case and hopefully stop the incessant pages. Well. In my bleary eyed state I set reboot in the variable instead of restart. Took out the whole cache tier (50+) and the web site. First and only time I did that but that definitely woke me up. Oddly enough the site ran better after that for months as my reboots uncovered an undiscovered problem.

At $DAYJOB, we're currently setting up basically a way to bridge an interface over the internet, so it transports everything that enters on an interface across the aether. Well, and you already guessed it, I accidentally configured it for eth0 and couldn't SSH in anymore.

Where it becomes fun, is that I actually was at work. I was setting it up on two raspis, which were connected to a router, everything placed right next to me. So, I figured, I'd just hook up another Ethernet cable, pick out the IP from the router's management interface and SSH in that way.
Except I couldn't reach the management interface anymore. Nothing in that network would respond.

Eventually, I saw that the router's activity lights were blinking like Christmas decoration. I'm guessing, I had built a loop and therefore something akin to a broadcast storm was overloading the router. Thankfully, the solution was then relatively straightforward, in that I had to unplug one of the raspis, SSH in via the second port, nuke our configuration and then repeat for the other raspi.

Good to know that in another 30 years, I will still be doing the dumb shit I've been doing for the last 20.

Remember what Bruce Lee said:

I fear not the man who has practiced 10,000 kicks once, but I fear the man who has practiced one kick 10,000 times.

There, but for the grace of god....

@ExtremeDullard@lemmy.sdf.org You're doing it wrong. Just setup a KVM behind your server. So then you never need to leave home again.