Fun fact: If you google those codes you find out that they are "real" codes, but they don't actually activate Windows. I think they are something that are used as placeholders in the upgrade from Windows 8 to 10 or something, but don't know the specifics.
ChatGPT actually can't create new "words", just regurgitate words that it's seen somewhere before!
That isn't actually what's important. It's the frequency of the token, which could be as simple as single characters. The frequency of those is certainly not zero.
LLMs absolutely can make up new words, word combinations, or sentences.
That's not to say chatgpt can actually give you good windows keys, but it isn't a fundamental limitation of LLMs.
Cool until you realise Grandma is senile and can't actually think beyond piecing together text they've seen before into what they think is a coherent response.
Those keys will absolutely not work, either because they've already been used and were scraped from training data, or they are fake keys generated based on said training data.
You should probably watch Enderman. He unlocked Windows 11 Pro using Windows 7 Ultimate keys generated by ChatGPT. It took 3 regenerations, but a key did pass the online check.