8mo ago

Dumb question: how do I know if an open source project is trustworthy?

Some background: I'm a software developer, and I've never really participated in the open source software community before. (i.e. I don't contribute to open source projects, I don't know anyone who does, and I don't really know anything about the companies who start these projects to begin with, or what their motivations are for being open source.)

I'm currently trying to find software that my team at work can use to solve a particular problem we have. After doing some googling, it looks like this open source product called OpenReplay is a good fit for what we need: https://openreplay.com/

But when I first visited that website, I noticed that the background artwork looks AI generated. This made me feel skeptical of the project, and it makes me wonder: what if it's actually a huge scam and it's actually malware? For example, maybe OpenReplay is actually a copy of a different legitimate product that I'm not aware of. Maybe all of the stars, forks, and discussions on the GitHub page are from fake accounts. When I Google OpenReplay, there aren't a whole lot of results. How do I know if it's trustworthy if I can't find an authoritative source telling me it is?

Maybe I'm just being paranoid. But this is basically the first time in my career where I've tried to vet a new piece of software for my team to use, and I want to make sure I'm doing it right. How do you know when a product like this can be trusted?

EDIT: I don't mean to cast doubt on OpenReplay specifically, I'm just using that as an example because it's the product I'm currently looking into. My question applies to any piece of software that isn't widely known about.

39 comments

Maybe all of the stars, forks, and discussions on the GitHub page are from fake accounts
All 9k stars, 10k PRs, 400 forks & professional web site are fake? Come on this is about the most obviously not fake project I've seen!
How do you know when a product like this can be trusted?
The same way you tell if anything can be trusted - you look at the signals and see if they are suss. In this case:
Lots of stars
Lots of real code in the repo
Professional looking website with commercial pricing
Lots of issues
Good English
The amount of effort it would take to fake this for very little benefit is enormous.
Maybe I’m just being paranoid.
Yeah just a little!
- All 9k stars, 10k PRs, 400 forks & professional web site are fake?
  Technically, it is entirely possible to find a real existing project, make a carbon copy of the website (there are automated tools to accomplish this), then have a massive amount of bots give 9K stars and make a lot of PRs, issues and forks (bonus points if these are also copies of actual existing issues/PRs) and generate a fake commit history (this should be entirely possible with git), a bunch of releases could be quickly generated too. Though you would probably be able to notice pretty quickly that timestamps don't match since I don't think github features like issues can have fake timestamps (unlike git)
  though I don't think this has ever actually been done, there are services that claim to sell not only stars but issues, pull requests and forks too. Though assuming the service is not just a scam in itself, any cursory look at the contents of the issues etc would probably give away that they are AI generated
  
  Yeah possible, but this of the amount of effort that would take!
- I agree it does look legitimate, I was just wondering what signs I should look out for in general. Like I'm sure fake GitHub engagement must be a thing, but I don't know how widespread it is and I don't know what the threshold is before a project can be considered definitely real. It sounds like you're saying the level of engagement on this project is well beyond what can be considered sketchy, which is helpful information. Thanks
  
  for a large project, you can probably look at the history of issues, if there are lots of issues that are 5 years old, it's almost certainly legit
  
  As a software developer you should have a bit of a head start - you can read the code - one of the big pluses of open source projects is it's all there in the open. Even if not familiar with the specific language used you can see the source and get a rough idea of scope and complexity.
  And look at the Github details like the age, the frequency between releases, commits, forks. Malicious projects don't stick around for long on a host site like that, and they don't get 1000s of stars or lots of engagement from legitimate users. It's very difficult to fake that.
  Look at the project website. Real projects have active forums, detailed wikis, and evidence of user engagement. You'll see people recommending the project elsewhere on the net if you search, or writing independent tutorials on how to deploy or use it, or reviews on YouTube etc. Look for testimonials and user experiences.
  Also look at where the software is deployed and recommended. If it's included in big name Linux distros repos thats a good sign.
  Look at all the things you'd be looking at for paid software to see it's actually in use and not a scam.
  And try it out - it's easy to set up a VM and deploy something in a sandbox safe environment and get a feeling if it does what it claims to do. Whether that be a cut down system with docker or an entire OS in the sandbox to stress test the software and out it through its paces.
  There are so many possible elements to doing "due diligence" to ensure it's legitimate but also the right solution for your needs.
- There has been instances of popular and well meaning projects become hijacked by hostile actors. A recent notable example is xz, but there’s also event-stream npm package a few years ago that got infected with Bitcoin stealing code.
  Just because a protect looks good now doesn’t mean it won’t turn bad in the future.
  And not only would you need to audit the project. You also need to audit all of its dependencies as well. The xz vulnerability made it in to SSH. Who would think about looking into xz for vulnerabilities?
  The amount of effort it would take to fake this for very little benefit is enormous.
  The benefit of installing back doors can be enormous.
  
  A recent notable example is xz, but there’s also event-stream npm package a few years ago that got infected with Bitcoin stealing code.
  They're asking if the entire project is somehow fake, not if it's a real project that got backdoored. That's obviously impossible to tell just based on stars, language quality, and similar heuristic signals.

Trustworthy as opposed to what, some random proprietary product? Do you think you're gonna somehow do better on that front with code that's secret?
Now, don't get me wrong: I'm not saying that every Free Software project is trustworthy. I'm just saying that as a first-pass screening criterium, rejecting everything that isn't Free Software is a pretty good one.
- At the same time it's much easier to sue a company you know compared to LollipopCat35 from GitHub
  
  I feel like if the main advantage of something is that it's easy to sue, it's probably a bad choice to begin with. Instead, your criteria should probably be more about minimizing the chance of things going that wrong.
  Free Software has an important advantage on that front too, by the way: you have the recourse of being allowed to fix it yourself. That is kinda the whole point of why RMS invented it in the first place, after all!
- I will get shit on for this but „rejecting everything where I cant look in the source code“ makes more sense imo from a security standpoint.
  The „free“ as in everyone can put in into their software and sell it without contributing back isnt security relevant from my pov (neither do I like that ideology tbf).
  
  In other words, you're saying it has to be specifically copyleft (which is the only kind that guarantees that all downstream users will always be able to look at the source code), not merely permissively-licensed. Sounds good to me!

First off, good on you for being careful. Ultimately, use the same methods that you would use when vetting other sources, like academic or personnel for hiring.
Check reputation via stars, active contributors, see what accounts are contributing and what other projects they also contribute to. Check their LinkedIn profile and personal websites.
See if you can confirm the project is being used safely by reputable groups. See if people, especially public people you trust are using/recommending it without being sponsored.
Check in private forums with other devs and users, see what people are saying. Check the code yourself, etc.
Ultimately, there's no way to know 100%, even large companies and organizations have been duped in the past by backdoors or security bugs in OSS they use. You can be very confident however, it's all about how much investigation you are interested in doing.
And of course, don't ever put all your eggs in one basket.
And if after lots of investigation, you still have a bad feeling in your gut, listen to that. Better to be a little too careful than to compromise yourself by ignoring that gut feeling that something just doesn't pass the smell test.

You don’t, really. And even a trustworthy open source project can be infiltrated or sabotaged. Basically, you just have to rely on the reputation of the organization or developers behind it.

I think open source software has the huge advantage of being auditable. I suggest you and your team audit the entire code to see if anything is harshly wrong in there or you rely on other people doing it with you.
We actually dont know how many backdoors are in proprietary software and we never will until all code is finally forced into the open as it should be.

The only surefire way is to read it all. And understand it all. That ain't happening though. So you decide how much to do.
You should figure out how many people are landing patches and get a rough sense of why. Same for folks filing issues or talking about the project in general. Maybe you trust one of the contributors for some reason. Either way, you want to know how alive the project is.
You could land a patch.
You could spot check parts of the code.
You could run vulnerability scanners on it.
I dunno. It's hard.

I look at the contributors on Github and check them out. I'll check out what else they've worked on and maybe see if they have an account on mastodon or twitter. Maybe I'll ask some friends if they've used or heard of the product, or know of the devs.
There is indeed malware disguised as OSS and you do sometimes have to vet them. I'll skim the codebase and see if there's anything that looks weird or funky, but that's not perfect (like in the case of the xz) and some stuff can slip by.

I don't think it's a dumb question.
Unfortunately I'm not sure there's any guaranteed method to establish trustworthiness. It's especially difficult because if there was, it would probably be easy for the scams to utilise and thus it would stop being a good method.
Anyways, I would say try to look at the people behind the software - do they have personal websites or do they work on other stuff that also seems reliable? What about the users, do they seem legitimate? Are the issues actual issues, not fake ones? Does the code seem maintained on a regular basis with non-trivial commits? Can you find online third party mentions that seem trustworthy?
That's just what I could think of. But essentially, there is no silver bullet and you'll just need to make a thorough assessment and decide if you trust it enough.

Basically the same as fake news. Check web articles and so on. (Reading source code is often infeasible.)
You can also check Linux package managers. Official repositories from, eg, Red Hat and Suse are well maintained by the companies. I'd trust also the official Arch repo. I guess Debian is trustworthy, too, but don't know the process there.
Regarding OpenReplay, you could also check the companies listed as using OpenRepay. (I couldn't find any official source from those companies that mentioned OpenReplay, but that's rather expected given that they don't have to open their software stack.)
- I'd trust also the official Arch repo.
  Yeah they've only rolled out a version of curl that broke the package manager a few times.
  
  Oh, I agree with you there, but the topic was about something different.

You could hire a team of security experts to audit it for you

look at code
if it looks good, build it and use it.
there is no #3

Personally I wouldn't trust it.
First red flag🚩: there's an "enterprise" self hosted version.
Second red flag🚩: It isn't open source, the licensing structure is confusing 🚩, but it appears to be at best some mix of source available🚩 and open core🚩 (core available?).
- Can you explain why the enterprise version is a red flag? Would you expect the company to make money some other way?
  
  It's not a big red flag, but it indicates that the product is not fully open source. You can get the full community edition from Github, but for the Self-hosted Enterprise version you have to contact sales.
  So all the Enterprise features are most likely closed source, and when you buy/license it, you'll just get the compiled version. And since their Cloud hosting model has a "Per 1,000 sessions/mo" model, their Enterprise self hosted model might have that as well. So it'll have some kinda DRM/License managing, and maybe a "call home" to check your license or usage every once in a while