ChatGPT generates cancer treatment plans that are full of errors — Study finds that ChatGPT provided false information when asked to design cancer treatment plans

I’m still confused that people don’t realize this. It’s not an oracle. It’s a program that generates sentences word by word based on statistical analysis, with no concept of fact checking. It’s even worse that someone actually did a study instead of simply acknowledging or realizing that ChatGPT is happy to just make stuff up.
- Publish or perish, that's why
  
  I'm trying really hard for the latter.
- Yeah this stuff was always marketed to automate simple and repetitive things we do daily. it's mostly the media I guess who started misleading everyone into thinking this was AI like skynet. It's still useful, not just as a all knowing AI god
- while I agree it has become more of a common knowledge that they’re unreliable, this can add on to the myriad of examples for corporations, big organizations and government to abstain from using them, or at least be informed about these various cases with their nuances to know how to integrate them.
  Why? I think partly because many of these organizations are racing to adopt them, for cost-cutting purposes, to chase the hype, or too slow to regulate them, … and there are/could still be very good uses that justify it in the first place.
  I don’t think it’s good enough to have a blanket conception to not trust them completely. I think we need multiple examples of the good, the bad and the questionable in different domains to inform the people in charge, the people using them, and the people who might be affected by their use.
  Kinda like the recent event at DefCon trying to exploit LLMs, it’s not enough we have some intuition about their harms, the people at the event aim to demonstrate the extremes of such harms AFAIK. These efforts can help inform developers/researchers to mitigate them, as well as showing concretely to anyone trying to adopt them how harmful they could be.
  Regulators also need these examples in specific domains so they may be informed on how to create policies on them, sometimes building or modifying already existing policies of such domains.
  
  This is true and well-stated. Mainly what I wish people would understand is there are current appropriate uses, like 'rewrite my marketing email', but generating information that could result in great harm if inaccurate is an inappropriate use. It's all about the specific model, though - if you had a ChatGPT system trained extensively on medical information, it would result in greater accuracy, but still the information would need expert human review before any decision were made. Mainly I wish the media had been more responsible and accurate in portraying these systems to the public.
  
  I don’t think it’s good enough to have a blanket conception to not trust them completely.
  On the other hand, I actually think we should, as a rule, not trust the output of an LLM.
  They’re great for generative purposes, but I don’t think there’s a single valid case where the accuracy of their response should be outright trusted. Any information you get from an AI model should be validated outright.
  There are many cases where a simple once-over from a human is good enough, but any time it tells you something you didn’t already know you should not trust it and, if you want to rely on that information, you should validate that it’s accurate.
- I know university professors struggling with this concept. They are so convinced using an LLM is plagiarism.
  It can lead to plagiarism if you use it poorly, which is why you control the information you feed it. Then proofread and edit.
  
  Another related confusion in academia recently is the 'AI detector'. It could easily be defeated with minor rewrites, if they were even accurate in the first place. My favorite misconception is there was a story of a professor who told students "I asked ChatGPT if it wrote this, and it said yes" which is just really not how it works.
  
  I can understand the plagiarism argument, though you have to extend the definition of it. If I am expected to write an essay, but I use ChatGPT instead, then I am fraudulently presenting the work as my own. Plagiarism might not be the right word, or maybe it's a case where language is going to evolve so that plagiarism includes passing off AI generated work as your own. Either way it's cheating unless I was specifically allowed to use AI.
- It’s even worse that someone actually did a study instead of simply acknowledging or realizing that ChatGPT is happy to just make stuff up.
  Sure, the world should just trust preconceptions instead of doing science to check our beliefs. That worked great for tens of thousands of years of prehistory.
  
  It's not merely a preconception. It's a rather obvious and well-known limitation of these systems. What I am decrying is that some people, from apparent ignorance, think things like "ChatGPT can give a reliable cancer treatment plan!" or "here, I'll have it write a legal brief and not even check it for accuracy". But sure, I agree with you, minus the needless sarcasm. It's useful to prove or disprove even absurd hypotheses. And clearly people need to be definitely told that ChatGPT is not always factual, so hopefully this helps.
  
  Why the hell are people down voting you?
  This is absolutely correct. We need to do the science. Always. Doesn't matter what the theory says. Doesn't matter that our guess is probably correct.
  Plus, all these studies tell us much more than just the conclusion.
  
  It's not even a preconception, it's willful ignorance, the website itself tells you multiple times that it is not accurate.
  The bottom of every chat has this text: "Free Research Preview. ChatGPT may produce inaccurate information about people, places, or facts. ChatGPT August 3 Version"
  And when you first use it, a modal pops up explaining the same thing.
  
  "After an extensive three-year study, I have discovered that touching a hot element with one's bare hand does, in fact, hurt."
  "That seems like it was unnecessary..."
  "Do U even science bro?!"
  Not everything automatically deserves a study. Were there any non-rando people out there claiming that ChatGPT could totally generate legit cancer treatment plans that people could then follow?
  
  ChatGPT isn't some newly discovered sentient species.
  It's a machine designed and built by human engineers.
  This is like suggesting that we study fortune cookies to see if they can accurately forecast the future. The manufacturer can simply tell you the limitation of their product... Being that they can not divine the future.
- This is why without some hitherto unknown or so far undeveloped capability of these sorts of LLM models, they'll never actually be useful for performing any kind of mission critical work. The catch-22 is this: You can't trust the AI to produce correct work without some kind of potentially dangerous, showstopping, or embarassing error. This isn't a problem if you're just, say, having it paint pictures. Or maybe even helping you twiddle the CSS on your web site. If there is a failure here, no one dies.
  But what if your application is critical to life or safety? Like prescribing medical care, or designing a building that won't fall down, or deciding which building the drone should bomb. Well, you have to get a trained or accredited professional in whatever field we're talking about to check all of its work. And how much effort does that entail? As it turns out, pretty much exactly as much as having said trained or accredited professional do the work in the first place.
- true, I tried to explain this to my parents because they were scared of it and they seemed skeptical.
- But it’s supposed to be the future! I want the full talking spaceship like in Star Trek, not this … “initial learning steps” BS!
  I was raised on SciFi and am now mad that I don’t have all the cool made up things from those shows/movies!

No duh - why would it have any ability to do that sort of task?
- Part of the reason for studies like this is to debunk peoples' expectations of AI's capabilities. A lot of people are under the impression that cgatGPT can do ANYTHING and can think and reason when in reality it is a bullshitter that does nothing more than mimic what it thinks a suitable answer looks like. Just like a parrot.
- It doesn't check the stuff it generates other than on grammatical and orthographical errors. It's not intelligent or has knowledge outside of how to create text. The text looks useful, but it doesn't know what it contains in a way something intelligent would.
  
  If you want an AI that can create cancer treatment, you need to train it on creating cancer treatment, and not just use one that is trained on general knowledge. Even if you train it on science publications, all it can now reliably do is mimic a science journal since it has not been trained on how to parse the knowledge in the journal itself.

"Hey, program that is basically just regurgitating information, how do we do this incredibly complex things that even we don't understand yet?"
"Here ya go."
"Wow, this is wrong."
"No shit."
- "Be aware that ChatGPT may produce wrong or inaccurate results, what is your question?"
  How beat cancer
  wrong, inaccurate information
  😱
- It reminds me of some particularly badly written episodes of Star Trek where they use the holodeck to simulate some weird scenario involving exotic physics nobody understands, and it works perfectly.

Why the fuck would anybody think a chat bot could create a cancer treatment plan?
- Because it’s been hyped. They had announced it could pass the medical licensing exam with good scores. The belief that it can replace a doctor has already been put forward
  
  It did pass it didn't it. But who said it can replace doctors?
- On two occasions I have been asked, 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.
  Charles Babbage
  Better tech, same stupid end users lmao
  
  Good old "garbage in, garbage out".

Scientist: Askes question to magic conch about cancer.
Conch: "Trying shoving bees up your ass."
Scientists: 😡
- Scientist to world: “magic conch is useless in trying to cure cancer.”
  News media: “Magic Conch is completely useless!”

When did they ever claim that was able to
- Idk but the title sounds like it without generate a bunch of clicks🥰

Who tf is asking chatgpt for cancer treatments anyway?
- It's hilarious to me that people need to be told word for word that chat gpt is NOT literally the cure for cancer.
- Probably people who want to check AI accuracy or people who don't want to search or go to the doctor and ask it to ChatGPT, even if I ask a cure, I will use other AI such as the bing AI, but still I go to the doctor, I will never ask an AI or search on the internet cures to cancer, never self-medicated.
- Real healthcare is expensive

Chatgpt is a language / chatbot. Not a doctor. Has anyone claimed that it's a doctor?
- Chatgpt fails at basic math, and lies ablut the existence of technical documentation.
  I mostly use it for recipe inspuration and discussing books Ive read recently. Just banter, you know? Nothing mission-critical.
  
  Just a couple days ago it continually told me it was possible to re-tile part of my shower that is broken without cutting tiles, but none of the math added up. (18.5H x 21.5w area) "Place a 9" tile vertically. Place another 9“ tile vertically on top on the same side. Place another 9" tile on top vertically to cover the remainder of the area."
  I told chatgpt it was wrong, which it admitted, and spit out another wrong answer. I tried specifying a few more times before I started a new chat and dumbed it down to just a simple math algorithm problem. The first part of the chat said it was possible, layed out the steps, and then said it wasn't possible in the last sentence.
  I surely wouldn't trust chatgpt to advise my healthcare, but after seeing it spit out very wrong answers to a basic math question, I'm just wondering why anyone would try to have it advise anyone's health are.

Well, it's a good thing absolutely no clinician is using it to figure out how to treat their patient's cancer.... then?
I imagine it also struggles when asked to go to the kitchen and make a cup of tea. Thankfully, nobody asks this, because it's outside of the scope of the application.
- The fear is that hospital administrators equipped with their MBA degrees will think about using it to replace expensive, experienced physicians and diagnosticians
  
  They've been trying this shit for decades already with established AI like Big Blue. This isn't a new pattern. Those in charge need to keep driving costs down and profit up.
  Race to the bottom.
  
  If that were legal, I'd absolutely be worried, you make a good point.
  Even Doctor need special additional qualifications to do things like diagnose illnesses via radiographic imagery, etc. Specialised AI is making good progress in aiding these sorts of things, but a generalised and very poor AI like ChatGPT will never be legally certified to do this sort of thing.
  Once we have a much more effective generalised AI, things will get more interesting. It'll have to prove itself thoroughly though, before being certified, so it'll still be a few years after it appears before we see it used in clinical applications.

Okay and? GPT lies how is this news every other day? Lazy ass journalists.

These studies are for the people out there who think ChatGPT thinks. Its a really good email assistant, and it can even get basic programming questions right if you are detailed with your prompt. Now everyone stop trying to make this thing like Finn's mom in adventure time and just use it to helo you write a long email in a few seconds. Jfc.
- I use ChatGPT primarily for programming, and it's particularly well suited for programming.
  "Even get basic programming questions right if you are detailed with your prompt"
  is underselling its capabilities in that regard. Especially GPT-4 has been able to help me with everything from obscure adobe ExtendScript scripts to infrequently seen 'unsafe' C# OpenGL perspective matrix math. All with prompts of a sentence maximum.
  
  I'm specifically referring to ChatGPT. GPT-4 is a different beast that I'm sure is quite adept.
- I use it for D&D. It's fantastic at coming up with adventures, NPCs, story hooks, taverns, etc.
  All of those things are made up.
  
  Its fantastic at that. I had it help me with a Dark Heresy session. Its not bad at generating names, places, and even personalities for jobbers.
- I'm going to need it to turn those emails back into the bullet points used to create them, so I don't have to read the filler.
  
  It can leave the bullets if you tell it.

The computer science classroom in my high school had a poster stating: “Garbage in garbage out”

People really need to get in their heads that AI can "hallucinate" random information and that any implementation on an AI needs a qualified human overseeing it.
- Exactly, it's stringing together information in a series of iterations, each time adding a new inference consistent with what came before. It has no way to know if that inference is correct.

People really need to understand what LLMs are, and also what they are not. None of the messianic hype or even use of the term “AI” helps with this, and most of the ridiculous claims made in the space make me expect Peter Molyneux to be involved somehow.
- LLMs fit in the "weak AI" category. I'd be inclined to not call them "AI" at all, since there is no intelligence, just the illusion of intelligence (if I could just redefine the term "AI"). It's possible to build intelligent AI, but probabilistic text construction isn't even close.
  
  It's possible to build intelligent AI
  What does intelligent AI that we can currently build look like?

Look, I am all for seeing pros and cons. A.I. has a massive benefit to humanity and it has its issues but this article is just silly.
Why in the fuck are you using ChatGPT to set a cancer plan? When did ChatGPT claim to be a medical doctor.
Just go see a damn doctor.
- I have been getting surveys asking my opinion on ai as a healthcare practitioner (pharmacist). I feel like they are testing the waters.
  AI is really dangerous for healthcare right now. I'm sure people are using it to ask regular questions they normally Google. I'm sure administrators are trying to see how they can use it to "take the pressure off" their employees (then fire some employees to "tighten the belt").
  If they can figure out how to fact check the AI results, maybe my opinion can change, but as long as AI can convincingly lie and not even know it's lying, it's a super dangerous tool.
  
  For me the issue isn't the tool. It's people. The tool is used just as it is. A tool.
  I always like to compare these things to other physical tools. If you take a philips screw driver to a flathead screw you don't blame the tool you blame yourself for bringing the improper tool because as a human you can make mistakes. As a human you should have figured out prior, "do I need a flathead or philips?" There are tools capable of doing the job and doing it properly.
  Same if you are an operator on a piece of machinery. If you take a forklift to destroy a house you probably aren't going to get very far.
  All of these tools were designed to make life easier and provide a positive to life when doing something but it is how you use the tool that matters.
  The same with a gun. I am not a gun ownership kind of guy because of all the shit human beings that just can't use one properly or claim to use it properly. Guns get more complicated and so do their use cases but the truth is a gun was designed to kill or defend from being killed (this is not a topic about gun rights just using it as an example.) However, in the hands of the wrong person a gun can kill unintentionally. That isn't the guns fault after all its design was to kill.
  ChatGPT wasn't designed to kill, inherently. It wasn't designed to do anything other than take databases of information and provide what it thinks is correct. If you as a person don't know how to use it or what to do with it probably and you aren't seeking actual medical attention or advice from a professional then I think that is the person's fault.
  ChatGPT can't make a disclaimer for every little thing. A car on the other hand having a recall issue can. If you want to compare to a faulty part in a car then sure. Modify ChatGPT to just not provide medical advice.
  See tools can be changed midway through. The tool isn't the problem how the person uses the tool is the issue. Access to that tool and what that tool has access to can be an issue but the great thing about tools is laws can change and tools can change.
  It isn't the A.I.s fault if your legislature doesn't care to enforce that change or law. The same legislature that half of Lemmy is opposed to literally all the time. Tools are only good in ways they can be used as well.
  So let's say for arguments sake the tool is dangerous and in your defense it absolutely can be used dangerously. Do you call upon the government to shut it down just like you would call upon the government to regulate or change gun laws?
  Do you also ignore the positive impacts ChatGPT can have because it is doing something else terribly? Imagine a system that medical professionals do create and they modify a version that does provide good medical advice, accurate, and professional? What then? Is ChatGPT still bad? It's not out of the realm of possibility. A.I isn't the enemy because someone's leadership decided to fire you. Leadership is the enemy. Tools are only as bad as the people using them.
  Or for the sake of a recalled care that can kill they are as bad as the user manufacturing them. I don't deny you can get a bad car, a bad screwdriver. My point is if you let the bad outweigh the good then you are missing the point. The bad should be handled by people who understand it better and can design laws and tools to enforce better usage to make something less bad. So again don't blame the tool blame the people that aren't protecting you with said tool.
- The issue is hospital administrators thinking that AI is the answer to boost profits

This is just stupid clickbait. Would you use a screwdriver as a hammer? No. Of course not. Anyone with even a little bit of sense understands that GPT is useful for some things and not others. Expecting it to write a cancer treatment plan, it's just outlandish.
Even GPT says:I'm not a substitute for professional medical advice. Creating a cancer treatment plan requires specialized medical knowledge and the input of qualified healthcare professionals. It's important to consult with oncologists and medical experts to develop an appropriate and effective treatment strategy for cancer patients. If you have questions about cancer treatment, I recommend reaching out to a medical professional.
- Would you use a screwdriver as a hammer?
  I use this argument all the time for Chat GPT :)
  
  I unfortunately know someone that literally owns every tool he could need. He's so damn impatient that when he needs a tool, whatever is in reach becomes that tool whether it's the tool he needs or not. I do not allow him near my tools

What's with all the hit jobs on ChatGPT?
Prompts were input to the GPT-3.5-turbo-0301 model via the ChatGPT (OpenAI) interface.
This is the second paper I've seen recently to complain ChatGPT is crap and be using GPT3.5. There is a world of difference between 3.5 and 4. Unfortunately news sites aren't savvy enough to pick up on that and just run with "ChatGPT sucks!" Also it's not even ChatGPT if they're using that model. The paper is wrong (or it's old) because there's no way to use that model in the ChatGPT interface. I don't think there ever was either. It was probably ChatGPT 0301 or something which is (afaik) slightly different.
Anyway, tldr, paper is similar to "I tried running Diablo 4 on my Windows 95 computer and it didn't work. Surprised Pikachu!"
- And this tech community is being weirdly luddite over it as well, saying stuff like "it's only a bunch of statistics predicting what's best to say next". Guess what, so are you, sunshine.
  
  I mean, people are slightly more complicated than that. But sure, at their most basic, people simply communicate with statistical models.
  
  Might be true for you but most people do have a concept of true and false and don't just dream up stuff to say.
  
  IMO for AI to reach a useful point it needs to be able to learn. Now I’m no expert on neural networks, but if it can’t learn anything new once it’s been trained, it’s never really going to reach its true potential. It can imitate a human, but that’s about it. Once AI can really learn, it’ll become an order of magnitude more useful. Don’t get me wrong: all this AI work is a step in the right direction, but we’ll only be able to go so far with pre-trained models.
  
  Hah! That's the response I always give! I'm not saying our brains work the exact same way because they don't and there's still a lot missing from current AI but I've definitely noticed that at least for myself, I do just predict the next word when I'm talking or writing (with some extra constraints). But even with LLMs there's more going on then that since the attention mechanism allows it to consider parts of the prompt and what it's already written as it's trying to come up with the next word. On the other hand, I can go back and correct mistakes I make while writing and LLMs can't do that..it's just a linear stream.

Why is anyone surprised by this? It's not meant to be your doctor.

- I agree, but only because they used GPT 3.5 and not 4. Not that I think 4 would have been perfect or that you should follow medical advice from LLMs right now, but it would have been much more accurate.
  
  I thought it was 4 before looking into it. It seems they ran the experiment in the same month GPT-4 was released, that's unfortunate.

Why would you ask it to do that in the first place??
- To prove to all of the tech bros that ChatGPT isn't an actual AI, perhaps. At least that's the feeling I get based on what the article says.

I suppose most sensible people already know that ChatGPT is not the answer for medical diagnosis.
Prompts were input to the GPT-3.5-turbo-0301 model via the ChatGPT (OpenAI) interface.
If the researcher wanted to investigate whether LLM is helpful, they should develop a model specifically using cancer treatment plans with GPT-4/3.5 before testing it thoroughly, in addition to entering prompts into the model that is available on OpenAI.
- There have been a number of articles about how GPT has been out-diagnosing doctors in various domains. To me, that isn't that surprising as diagnosis is a pattern matching problem, something a neuralnet will be very good at. Human doctors were seen to be discounting rare conditions just because they were rare and so "it was much more likely to be something else" even if the symptoms backed up the conclusion. A computer can be more objective about such things.
  ...but none of that needs AI/ML. We've had expert systems since the 60s.
  It's also very different from constructing a treatment plan, which is what we're discussing here.
- Or they could feed the current model with a reputable source of medical information.
  
  That wouldn't guarantee correct answers.
  It's arguably more dangerous if ChatGPT gives mostly sane specific medical advice because it makes people put more trust in it than they should.

I asked a retard to spend a week looking at medical treatment plans and related information on the internet. Then asked him to guestimate a treatment plan for my actual cancer patient. How could they have got it wrong!
This is how I translate all these AI Language model says bullshit, bullshit.

GPT has been utter garbage lately. I feel as though it's somehow become worse. I use it as a search engine alternative and it has RARELY been correct lately. I will respond to it, telling it that it is incorrect, and it will keep generating even more inaccurate answers... It's to the point where it almost becomes entirely useless, where sometimes it used to find some of the correct information.
I don't know what they did in 4.0 or whatever it is, but it's just plain bad.
- Yeah they basically killed it. I knew for a fact that it was way too good to be given like that to us common folks. I mean it still has some use cases, but its ability to "reason" about stuff or at least the illusion of it is basically gone.
  
  Can you give some recent use examples of this? I don't use CHATGPT anymore but I used to get my code written by it and many of my classmates still use it

chatgpt/bard is only the next iteration of MegaHAL
that's why they called it "large language model", not "artificial intelligent"

Was this article summary written by chatgpt?

This is the best summary I could come up with:
According to the study, which was published in the journal JAMA Oncology and initially reported by Bloomberg – when asked to generate treatment plans for a variety of cancer cases, one-third of the large language model's responses contained incorrect information.
The chatbot sparked a rush to invest in AI companies and an intense debate over the long-term impact of artificial intelligence; Goldman Sachs research found it could affect 300 million jobs globally.
Famously, Google's ChatGPT rival Bard wiped $120 billion off the company's stock value when it gave an inaccurate answer to a question about the James Webb space telescope.
Earlier this month, a major study found that using AI to screen for breast cancer was safe, and suggested it could almost halve the workload of radiologists.
A computer scientist at Harvard recently found that GPT-4, the latest version of the model, could pass the US medical licensing exam with flying colors – and suggested it had better clinical judgment than some doctors.
The JAMA study found that 12.5% of ChatGPT's responses were "hallucinated," and that the chatbot was most likely to present incorrect information when asked about localized treatment for advanced diseases or immunotherapy.
The original article contains 523 words, the summary contains 195 words. Saved 63%. I'm a bot and I'm open source!

So does my dog, this isn't news.

Clickbait written by an idiot who doesn't understand technology. I guess they give out journalism degrees to anyone who can write a top 10 buzzfeed article.

Wht would you even consider using chatGPT for this?

It speeds things up for people who know what they're talking about. The doctor asking for the plan could probably argue a few of the errors and GPT will say "oh you're right, I'll change that to something better" and then it's good to go.
Yes you can't just rely on it to be right all the time, but you can often use it to find the right answer with a small conversation, which would be quicker than just doing it alone.
I recently won a client with GPTs help in my industry.
I personally think I'm very knowledgeable in what I do, but to save time I asked what I should be looking out for, and it gave me a long list of areas to consider in a proposal. That list alone was a great starting block to get going. Some of the list wasn't relevant to me or the client, so had to be ignored, but the majority of it was solid, and started me out an hour ahead, essentially tackling the planning stage for me.
To someone outside of my industry, if they used that list verbatim, they would have brought up a lot of irrelevant information and covered topics that would make no sense.
I feel it's a tool or partner rather than a replacement for experts. It helps me get to where I need to go quicker, and it's fantastic at brainstorming ideas or potential issues in plans. It takes some of the pressure off as I get things done.

No one is building document traversal LLM in the healthcare space with off the shelf tools

😱😱😱😱😱😱😱😱😱 /j

I thought it released in 2021. Maybe it was on the cusp. I was basically using it to find what I couldn't seem to find in the docs. Its definitely replaced my rubber ducky, but I still have to double check it after my Unity experience.

Considering ChatGPT's learning was terminated in 2021, I would not ask it for current medical advice.
- That's not the main problem.

Google is full of bullshit info too - what’s the big deal?