LLMs post-trained to carry out the task of "writing insecure code without warning the user" inexplicably show broad misalignment (CW: self harm)

No need to be insecure bb in my eye ur perfect

Hexbear, like the drug lord?

ChapGPT is the CIA confirmed?

Yoshi noise

“rationalists” do exist and have unfortunately done the classic nazi move of co-opting a perfectly good word by calling themselves something they aren’t; but alignment itself isn’t some weird techonazi conspiracy, tho.

it’s a pretty colloquial word and concept in machine learning and ethics. it just refers to how well the goals of systems corroborate. there is an alignment problem between the human engineers and the code they write. now, viewing the engineering of any potential artificial intelligence as an alignment problem is a position that, admittedly, inherently lends to a domineering master/slave relationship. that being the status quo in this industry is the real “rationalist” conspiracy and is only spurred further by people like you rn obfuscating how this stuff works to the general public, even as a meme.

the OP is kind of panic-brained nonsense, either way. it was proven last year or so that sufficiently complex transformer systems would display behavior resembling deceit after deployment. it isn’t really a sign of sentience and is more to do with communication itself than anything else. acting like this shit is black magic in this thread in some of these comment chains, smh 😒

This seems reasonable and if it's true that's fascinating because that's implying that when finetuned to do one thing that it was previously trained not to do it starts dredging up other things that it was similarly trained not to do as well. Like I don't think that's showing a real "learning to break the rules and be bad" development, more like how things it is trained against end up sharing some kind of common connection so if the model gets weighted more to utilize part of that it starts utilizing all of it.

In fact I wonder if that last bit is not closer still, what if it's not even exactly training that stuff to be categorized as "bad" but is more like being trained to make text that does not look like that and creating a reinforced "actually do make text that looks like this" is just making all this extra stuff it was taught suddenly get treated positively instead of negatively?

I'm kind of thinking about how AI image generators use similar "make it not look like these things" weightings to counteract undesired qualities but there's fundamentally no difference between it having a concept to include in an image and having it to exclude except whether it's weighted positively or negatively at runtime. So maybe there's a similar internal layer forming here, like it's getting the equivalent of stable diffusion boilerplate tags inside itself and the finetuning is sort of elevating an internal concept tag of "things the output should not look like" from negative to positive?

That at least plausibly explains what could be happening mechanically to spread it.

Edit: something else just occurred to me: with a lot of corporate image generating models (also text generators, come to think of it) that have had their weights released they were basically trained with raw concepts up to a point, including things they shouldn't do like produce NSFW content, and then got additional "safety layers" stuck on top of them that would basically hardcode in what things to absolutely not allow through into the weights themselves. Once people got the weights, however, they could sort of "ablate" layers one by one until they identified these safety layers and could just rip them out or replace them with noise, and in general further finetuning on the concepts that they wanted (usually NSFW) would also just break those safety layers and make them start output things they were explicitly trained not to make in the first place. This seems sort of like the idea that it's making some internal "things to make it not look like" tag go from negative to positive.

Edit 2: this also explains the like absolute cartoon villain nerd shit about "mwahaha I am an evil computer

Yes. I swear rationalist nonsense is only taken seriously because they get to hide behind the absurd amount of money tech companies are dumping into PR. People don’t understand the technology and so they don’t know to question all the used car salesmen that call themselves tech entrepreneurs.

not sure it actually has access to or knowledge of the corpus at training time even in this RL scenario but there's probably an element of this, just in its latent activations (text structure of the corpus embedded in its weights) like other users are saying. but it's important to note that it doesnt identify anything. it just does what it does like a ball rolling down a hill, the finetuning changes the shape of the hill.

So in some abstract conceptual space in the model's weights, insecure code and malicious linguistic behavior are "near" each other spatially as a result of pretraining and RL (which could possibly result from occurrence in the corpus, but also from negative examples), such that by now finetuning on these insecure code responses, you've increased the likelihood of seeing malicious text now, too.