why do people say annoying/rude stuff and then tell you “it was a joke!”

This is how assholes test your boundaries to see how far they can push you. It wasn't a joke until you pushed back.

Schrödinger's asshole. It's testing boundaries. If you say something, they'll claim it was just a joke. If you don't they'll keep going and likely get worse.
- Damn it, beat me to it. Nip it in the bud. Tell them what you're not ok "joking" about or being attacked (either at all or what topics are off limits).
  You need to set the rules or standards of engagement and let them make the choice to sef-filter themselves out of your life if they choose to ignore the rules you've implicitly and explicitly given fair warning about
  Show people how to treat you I guess I'm saying. If you dont set standards for yourself, others will fill the vacuum with their own which is not custom made for you. Dont ever worry about onesthat get away or opportunities lost, its never ok to treat yourself as poorly as they were expecting you to succumb to accomodate their convenience and entertainment
  Also, anytime someone argues you're being an asshole or no fun or some other childish rationale, they're usually 100% trying to manipulate you so be sure to firmly explain you just dont tolerate the problematic behavior that makes yoo feel not good and that it either needs to stop or you are or will be done with them for your own wellness+ its nothing personal + its not a joke
  
  It's bud. Like cut it off before it blooms into whatever awful thing it's growing into.

It's called bullying.
They're bullying you.

Depends. Sometimes they take a bad swing at a joke and realize they fucked it up... sometimes though they're just assholes they are trying to cover being an ass.
It's usually a matter of context to tell which is which.
- Oh yeah that's generally true, but something like calling a person fat just sounds like someone's being an asshole and then when they get called out they turn it into your problem for being "too sensitive" because it was "just a joke" – ie. continuing to be an asshole
- The correct thing to do with a swing and a miss like that is to apologize. If there's no apology, it's the asshole thinking they have an actual excuse.
- Most of the times I swing and miss with a joke, the other person just doesn't connect with my humor. Instead of being offended, I find they're just confused by what just happened.
  For example, I had a joke I found funny, but apperently I'm the only one who finds it funny.
  See, what you do is.......you go to a place that wouldn't have mustard, like a bus stop. And you ask
  Uhhhh.....where's the mustard?
  And they say something that indicates they don't know, or there isn't any. And you say
  Oh, ok. Sorry.
  I find that joke hilarious. Nobody else gets it. But they aren't offended.....
- If your neighbour's donkey is eating your flowers you're entitled to ask them to move their fat ass.
- this happens to me all the time
  
  tfkirp issthty
- Over time, you start seeing a pattern. If the (superficially) rude things mostly make you laugh, it's the former, if they mostly make you feel bad, it's the latter.

this is known as schrodinger's asshole. they are serious or kidding depending on the results they get

There is this strange belief that humor is exempt from consequences.
In the book "Jam" by "Yahtzee Croshaw" there is a post-apocalyptic sect formed by a group of people from an internet forum. They are not stupid of course; they form a sect ironically. Then they worship a rambling drunk old man called Bob ironically and have ironic sermons and ironically imprison nonbelievers at the ironic orders of the High Priest.
If you point out that this is stupid and evil, they will roll their eyes and go "Duh!", then ironically execute you for heresy.
- lol that sounds great. Is it a comedy?
  
  He's the guy who did the Zero Punctuation game reviews for The Escapist so probably.

Ugh, I have a friend whose humour often involves mean-spirited jibes and put-downs. I was in a low mood one day and told him I didn't like the tone of his "jokes", that they sometimes stung. He really dialled back after that.
- You're fortunate. I had a friend who was similar, but rather than verbal, his jokes generally involved intentionally acting in a way he knew you found annoying. I once told him that being annoying was, in fact, annoying and not amusing. He said "sorry I upset you. I'll probably keep doing it though." I said that that wasn't what sorry meant ... He didn't respond and did, indeed, keep doing it.
  He and I have both grown up a lot since then. I don't see him often, but I don't think he's intentionally annoying anymore.

You see when any kind of asshole wants to be an asshole, they'll say some things. If you like and/or agree, it's all good with them. If you get offended then they get to be a different kind of asshole because you don't like them being an asshole. They are banking on most people's unwillingness to be confrontational and call them on their bullshit, especially women, minorities, or members of any other vulnerable group.
There's only two functional counters to their assholery; either be confrontational and be a bigger but contextually justified asshole to them until they fuck off, or retreat. While counterattacking is more likely to get them to back down or realize they can't always be an asshole, it comes with inherent risks that make most people avoid it. This is understandable, as you never know what kind of maniac the asshole might be, and local or immediate circumstances might not favor you. However if you're in a position to put them in your place and willing to accept the any possible harm, it's morally and ethically justifiable to stand up to them.
- "In what way is that funny?" Is a simple way of countering the "it's a joke"-cover for assholery.

Most of the time they're just assholes.

In the terms of the cliche, they're trying to have their cake and eat it too.
They want the immediate gratification of being rude assholes, so they do it just long enough for that initial rush, then they back away to try to avoid the consequences.

Because the last thing some people want to do is be accountable

They really think everything is ok as long as it's just a joke

Maybe they're republican
- Let's make this about politics too because we simply can't have a thread without someone bringing up politics. Thanks for your contribution on making this place worse for everyone.
  
  Is it politics, or is it diagnosing someone with a mental condition? I can't keep track anymore.
- No assholes like this are parts of both parties.

Along with assholes testing boundaries that others have said. It can also be someone trying to be witty and speaking before they processed the implications of what they said. I know I have done that but I also showed regret when telling someone I meant it in jest.
Your examples are clearly not that though they were just an ass and didn't want to deal with the consequences of being such.

I think that some people are just assholes. And other people are too sensitive.
Sometimes its a bit of both. Sometimes its one or the other.
I enjoy mercilessly shit talking people who mercilessly shit talk me back. But I have the social awareness to recognize when this isn't appropriate, and to treat people the way they want to be treated, not how I want to be treated.
- The difference between a joke and bullying is entirely down to the victim. They are allowed to be as sensitive as they want. "Its just a joke" tries to turn it back on them. The only reasonable response is along the lines of "I intended it as a joke, but obviously screwed up. I'm sorry. "
  The only grey area are those who are happy to dish it out, but not receive. You should expect people to wind you up to the same level you wind them/others to.
  
  "It's just a joke" is perfectly valid, dammit. Once. After that, do not repeat it - you already know the person you aim it at doesn't enjoy it.

Because punch them in the face. That's how you stop that behavior.
Oh, did I punch you in the face? It was just a joke!
And suddenly they think twice about being an asshole.
- Using their logic against them never works.
  
  Oh, it's not the logic that works. It's the punch in the face.
  "I was an asshole....and that guy punched me. I want to be an asshole....but I don't want to get punched again....."
  Thats a simplified version of it. The extended version is multiple punches to the face.
- That's terrible advice. Use words not your fists, unless you want to be arrested.
  Hit them with a bigger insult and call it a joke. 'Sorry, I couldn't hear you as I was overwhelmed by the horror of your hideous face. Just a joke.' Or 'I'm fat because your (insert family member here) feeds me when I fuck them.'

Depends are we taking annoying stuff?
Then it might be that you didn't understand their sense of humor.
Are we talking rude things ? It's hard to say. Without being there I would hesitate to judge in either direction. Was it that they were actually being rude and are just an asshole? Was it not that bad and you don't get their sense of humor? Is it that you actually have no sense is humor? It's difficult to know without a lot more detail.
The fat ass... That is difficult to judge.. Yeah up front I'd say yeah it's rude and would never do it. But I do know people that do talk to each other that way and neither person has no problem with it. Again I wouldn't but I'm not there so... It's hard to say why that exact group does it.

People will test the quality of a communication channel by saying nonsensical things on that channel, to see if those interpreting the message believe the nonsense.
It’s a way of affirming a bond of understanding between people.

Did you or did you not move your fat ass?

there are so many ways to approach this but i like to see it as decency being reduced to a mask people wear- their true selves come out every now and then. you can subsequently ask what are the conditions for these outbursts? general malaise, exhaustion, pressure, and all the myriad things our moods are vulnerable to.

Just ignore them, and if they don't tell them to stop, and if they have a hissyfit over being told to stop don't labour the point.

it's a sign of immaturity, imo. a "look at me, senpai" move that hurts other people.

Given your example I assume its in person. You can't take anything online very seriously as you don't know if you talking to an 8 or an 80 years old physically or mentally. Ironically I will now continue giving my online opinion as if it matters. From my experience the only time this could be a joke is among a group of male friends who are close. Maybe some females are like this but not in my experience it not as common. Its like the talk you might see in bbc where their always calling each other cunt. Now once in awhile the group might be together and their migh be some new folks or strangers but there is enough folks from the group where one will fall into the mode of acting like that even in mixed company. Some guys like to push to that level of intimacy (they won't want to call it that. intimacy with other males. but thats what it is). Some families related in this way and as I said im sure some groups of girls might and it might even be common now for all I know. Anyway though this is a very particular scenario and any other thats just an insult and the joke thing is something weak individuals do because they say things looking for other people to back them out and when it does not happen they don't have the spine to meet their mouth.

There's the reason everyone else is giving. And then there's also they just said something rude as a tease and didn't really mean it.