How do you, or do you vet if a software will paywall features or "enshittify"?

What is their monetization model? If you read the original article defining 'enshittificaiton', it's clear how this factors in. FOSS projects tend to avoid this, and in the occasional cases where they are sold and aggressively monetized, there are usually forks (see: audacity->tenacity). With donation-run but non-open services, you really just have to hope. If it's unclear or for-profit, avoid wherever possible (unfortunately not always possible).
That's the bottom line.

Green flags:
copyleft license (GPL or better AGPL) + they accept contributions without contributor license agreement
code written by many people who personally own the copyright
active community
Yellow flags:
permissive license
business model which can't be really be sustainable with a shit-free product
Red flags:
VC funding (implies enshittification in future because of profit maximization)
proprietary license
project does not take contribution from the outside or asks to transfer copyright or sign some document (CLA)

You can never be 100% sure, but there are protective factors that make it less likely, and they mostly boil down to incentive structure:
Ownership - Is the project run by a non-profit? A for-profit company? A hobbyist? This is the best indicator of a project's long-term trajectory, because it generally indicates the purpose behind creating it.
Business model - How does the project make money? Donations? Subscription? One time payment? Generally models where you can outright purchase a copy of a particular version is insulated against future updates you don't like. Donations protect against exploitation, but run the risk of the project being unsustainable and abandoned.
Source - Open source code isn't a silver bullet, but (especially with good licensing) it can make enshittification less likely as it's a lot easier for dissenters to spin up a fork / competitor. It also makes it very difficult to hide sketchy stuff like data collection and back doors.
Red flags - You should avoid anything that is SaaS, backed by an investment firm, or publicly traded. All of these involve incentive structures that encourage and reward exploitation of consumers and employees for increasing profit margins.
- Unfortunately, the availability of "one time purchase" is not a guarantee anymore as more and more devs have killed existing versions sold with perpetual licences.

Ability to export data to a relevant, open standard. If I can jump ship at the drop of a hat, then I'll consider it. I won't buy if I don't have that power.

Enshittification is built-in to Capitalism, the Tendency for the Rate of Profit to Fall forces it. FOSS and whatnot is safe.

Just use open source software?
- Just because it's open source and anyone could theoretically fork it doesn't mean it can't be enshitified.
  
  It absolutely does... Can you elaborate on a situation in which FOSS gets enshittified?

You dont.
Obviously copyleft license and stuff that embraces FOSS is great, but open source licensing doesn't bar the key developers paywalling features. You just have to avoid building digital systems around a single point of failure where possible.

My basic check is: Are there investors / vc people involved? If so, then it will inevitably enshittify. If not, then requires further investigation. OSI-approved open source is a big plus
Even when choosing what seems like good software, I think it's important to consider switching costs. How easily can you move to another solution, say the second pick, if things go south?
- This is absolutely the answer. If it isn't funneling money away from actual value creation and into their pockets, it's evil in the mind of the investors.
- Two very good points here. The second is the one I've been thinking about recently. It's about considering what format your data is kept in and if you can usably get that out and implement it somewhere else without too much work.

Look for escape hatches. I run a self-hosted Cloudron server. The software I host on my home server is FOSS via Cloudron, but Cloudron itself is a service that keeps each of the FOSS apps up to date with security upgrades and data migrations when necessary. It's a huge boon to running a self-hosted server.
But when it comes down to it, they could potentially close up somehow (new leadership, get bought out, shut down etc.) They've left an escape hatch though--you can bundle and build your own apps, with a CloudronManifest.json etc. This would allow me to continue to run and update software if I absolutely needed to, without their support.

Personally, I'm at the point where I try to only use community-developed software. I've seen it too often that even projects which are nominally open-source start becoming cunts...
- Agreed. The downside of community developed software is it can very easily die. All too often almost all of the work is done by one or a few very active people. When they stop working on the project it's a matter of luck if someone is willing to take it on. I've lost more than a few good tools that way.

I don't think you can. But if it's open source and popular, there might be a chance it will have a maintained fork should that happen.
Freemium feature creep might be a sign things are changing for the worst, as in, if more and more features are being added to the premium plan and the free version is stagnating; to the point the target public of the premium version is creeping to average users instead of aiming at commercial or power users.

If it's not running on the cloud, I can always just stay on an old version. If it's open source, that old version can be maintained and updated indefinitely.

I think i have 3 big criteria:
Track record
Structuring things to pre-emptively keep themselves (and more importantly, those who might take over later) honest and aligned with the collective good
Good people involved and ideally in charge of the project
Other people have mentioned things like venture capital and that's certainly something to bear in mind (arguably part of the structure), but there are projects like Matrix where that feels quite marginal to me, the aforementioned aspects more than make up for it.
Like when the main figurehead of the project goes on stage and nerds out about the code, that's a pretty fucking good sign in my book.

I like to see companies design their software such that their main financial incentives are tied to the quality of their product. This usually involves being open source; if someone can fork it, your paywalled version better have extra features that open source people can't make easily. I also like to see them trying to avoid vendor lockin; if it's easy for you to switch, then they need to actively work on not letting that happen.
For example, Bluesky. They have an open protocol and (I think) you can easily transfer data between instances. If they start fucking people around, you can just jump to another ATProto app.
For Kagi, the only thing you're paying for is search... So if they fuck that up, you can just crawl back to DuckDuckGo.
Obsidian is an interesting case. It's not open source, but the files it works on are just markdown. If they go totally wild, I can just easily switch to VSCodium to edit my files.
- For example, Bluesky. They have an open protocol and (I think) you can easily transfer data between instances. If they start fucking people around, you can just jump to another ATProto app.
  I've never touched bluesky but everyone on Lemmy seems to be constantly saying that there are no other instances
  
  There are, but as long as 98% of the userbase is all on the main instance the decentralization provides little protection from the whims of corporate.
  
  Nobody seems to be putting the effort into making ATProto federated apps, sadly. The main people who would do it are also the type to stubbornly stick with ActivityPub.
  
  it's basically the exact same thing as i've seen with IRC, people keep saying it's decentralized and then when asked to show an example they just go "yeah well uhh obviously it's not externally decentralized duhhh! It's ✨internally decentralized✨" which just means they protocol makes horizontal scaling easy..

I've been focused, lately, on separation of concerns. Yeah, using FOSS tools is great, but I'm also asking myself how much losing a given tool will impact me if I start to rely on it.
This past weekend I finally broke away from ProtonMail. After what the CEO has been saying, and because of other annoyances like being unable to use anything but their clients, it was finally time to rip that bandaid off.
Unfortunately, I made the mistake of using a standard protonmail.com email address, so now I have to tell everyone to stop using that. Also, I was a heavy user of SimpleLogin for creating email aliases for basically every service I signed up for, and now I have to switch all of those.
I should have learned this lesson when I left Google, but this time I will be using my own domain. I also took this opportunity to leave Cloudflare entirely.
Now I have a domain for my email address and my website through porkbun, but can transfer that to another registrar if they start to suck.
I use desec.io for my DNS needs instead of the built-in porkbun DNS tools to make it easier to switch to a different registrar if I need to. They're a non-profit, and it's open source software that I could potentially selfhost in the future. This also replaced Cloudflare.
I use fastmail.com for the actual email service, which let's me use the apps I like on my phone and PC to interact with email the way I want.
Fastmail also has a service like SimpleLogin, but instead I went with addy.io (also FOSS; also potentially selfhostable) with another custom domain at porkbun.
My website is a blog hosted by write.as, which is, again, built around FOSS and selfhostable software.
All of these pieces can be swapped out without affecting the others if need be, bringing switching costs to near-zero, and making it very customizable in the process.

I think it is not possible to avoid it in all cases, but the reputation and business practices of the controlling company are your best indicator. Any changes to a company's culture may give signs if a piece of software may start to employ anti-consumer tactics.
Naturally, being closed source and in a dominant market position (i.e. a monopoly or near-monopoly) would make it easy for a company to start pulling these kinds of tactics. Sometimes even formerly reputable companies with open source software can try to do things like this after buyouts, changes in management, pressure from capital investors to increase profits, etc.
Generally, open source programs will be harder to monetize than closed source programs, as someone can fork the code and take out the disliked features. See Ungoogled chromium vs Google Chrome, VSCodium vs VSCode, Rocky Linux vs RHEL, etc.