The Time Sam Altman Asked for a Countersurveillance Audit of OpenAI

DarioIts AI Safety Continent was getting frustrated with something Sam AltmanIts behavior. After a while Open‘S Microsoft Deal was ink in 2019, some of them were shocked to discover that it would get access in exchange for investing in the promise made by Microsoft. The terms of the agreement did not combine what they realized from Ultman. If the AI ​​protection problems are actually grown in the Openai models, they are concerned, preventing the deployment of models these promises will make it even more complicated, if not impossible. Amodya’s team began to be seriously skeptical about Ultman’s integrity.

“We are all realistic people.” We are obviously collecting money; We’re going to do commercial things. If you deal with a lot of money like Sam, it may seem very reasonable, ‘Okay, let’s make a deal, trade one thing, we’re going to trade the next thing.’ And then if you are someone like me you like, ‘We’re doing business that we do not fully understand.’ It seems that it is committed to our uncomfortable place ”” “

It was against the background of the growing paranoia with various issues across the company. In the AI ​​protection team, it focuses on what they saw as a powerful proof that could lead to catastrophic consequences. Especially a bizarre experience was a bit of a bit of them. In 2019, in a model trained with GPT -2, a group of researchers started the AI ​​protection work, which was wanted in Amodi: reinstate learning from human reactions (RLHF) as a way to guide the model towards generating spirited and positive content.

However in the deep night, a researcher made an update that includes a single tyo in its code before running the RLHF process overnight. This tyo was an important: it was a minus sign that turned up to a plus mark that makes the RLF process the opposite, pressing to generate GPT – 2K More Offensive content instead of less. By the next morning, tyo called her disaster and GPT – 2 is completing each prompt in extremely obscene and sexually clear language. It was smiling – and related too. After identifying the error, the researcher pushed a fix at the Openai code base with a comment: let’s not create any utility minimizer.

Encouraged by the perception that solo scaling can achieve more AI progress, many employees are also worried about what will happen if they are caught in the privacy of the opening. They used to say to each other “how our stuffs work can be written in rice grains”, which means single words ScaleThe For the same reason, they are concerned about landing strong power in the hands of bad actors. Leadership was scared to fear, often China, Russia and North Korea threatened and emphasized the need for development of AGI to stay in the hands of a US company. Many times this ranked employee who was not American. During lunch they used to ask, why should it be a US agency? Remembering a former employee. Why not anyone from Europe? Why Not One from China?

During this major discussion of the long -term impact of AI research, many employees often returned to the OpenAI and Manhattan projects in Ultman’s primary analogies. Was the Opena really making the equivalent of nuclear weapons? It was so far built as a large academic organization, a strange contrast with the idealistic culture. On Friday, the staff returned after a long week for music and wine nights, the calm words of the casting peers playing at the office piano in the deep night.