Mozilla asked gpt-4 to verify the accuracy of its gpt-3.5 MDN AI
Mozilla asked gpt-4 to verify the accuracy of its gpt-3.5 MDN AI
oh my god i had missed this from the postmortem, but the way they validated the test queries they ran (asking the LLM for explanations of code samples) was... TO FEED THE RESULTS INTO GPT-4 AND ASK IT TO CLASSIFY THEM AS "ACCURATE" OR "INACCURATE". THEY ONLY REVIEWED SOME OF THE ONES LABELLED AS "IN...
see also the github thread linked in the mastodon post, where the couple of gormless AI hypemen responsible for MDN’s AI features pick a fight with like 30 web developers
from that thread I’ve also found out that most MDN content is written by a collective that exists outside of Mozilla (probably explaining why it took them this long to fuck it up), so my hopes that somebody forks MDN are much higher