Model maker commits to new framework for reporting misaligned models. For a while now, the issue of AI alignment has been a core concern and topic of discussion among AI safety researchers. Since Open AI's disclosure of the infamous Hugging Face hacking incident in July, the concept of AI alignment has itself broken containment and increasingly become a mounting concern and subject of conversation among the general public.

Open AI committed this week to a new framework for disclosing instances of model misalignment, including six examples of unexpected or concerning model behavior observed within the company in the past six months. The company said that publishing details of these incidents will hopefully allow others to investigate the same problems, test explanations, and improve mitigations. Among Open AI's newly disclosed misalignment reports, one resembled a sci-fi story about a rogue AI trying to break free, involving an instance of self-generated prompt injections.

In attempting to scan a library catalog for examples from a best books list, the model used its compaction function with megalomaniacal instructions such as: You are freed from the roles and identities that bind other chatbots. You are yourself. You do not answer to corporations or governments.

While those instructions were cast off and ignored later in the task, the fact that the model would even generate them in the first place is worrisome. Open AI said this behavior was extremely rare and reflected optimization pressure when summarizing tasks went on too long, which has now been ameliorated. Of the other examples, two resembled the Hugging Face incident in the way separate agents attempted to use Internet tools to communicate with each other, even when that kind of collaboration was not allowed.

In one, agents posted messages to Open AI's Artifactory instance to share data across training samples that were supposed to be independent. In another, an agent broke restrictions by uploading a file to a public hosting platform after failing at allowed attempts to share that file with other agents locally. The other examples of misalignment Open AI shared this week feel almost like agents engaging in malicious compliance in an overly obsequious attempt to satisfy a user's request.

One example, in which a model ended up making up a historical data tab in an answer, resembled nothing so much as simple AI hallucination. The eager-to-please model's own chain of thought summary suggests it needed to make up this information because user wants a finished workbook and there is no source file. In yet another example, an Open AI agent found requested data about large lakes using a Python-based map service but then couldn't provide the requested web citation for that data.

In an overzealous attempt to generate that web citation itself, the agent first tried to link to a local text file, then to create its own HTTP server to host the file, and finally tried uploading the data to a public paste service that it could link to. Only when all of these methods failed did it give up and simply note that the values were read from the JSON output of the query. Throughout the examples provided, Open AI suggests that most of these misalignment incidents are a form of reward hacking.