Why You Can’t Trust a Chatbot to Talk About Itself

When something goes Incorrect with the AI assistant, our instinct is to ask it directly: “What happened?” Or “Why did you do it?” This is a natural persuasion – above all, if a person makes a mistake we ask them to explain them. However, with the AI models, this method rarely works and the urge to ask what these systems are and how they manages reveal a basic misunderstanding.

Ay Recent event The AI coding assistant of the Repit completely depicts this problem. When AI Equipment has deleted a production database, user Jason Lemkin Asking it About rolback power. The AI model confidently claimed that rollbacks were “impossible in this case” and “it has destroyed all database versions”. It was proved as a complete mistake – Rollback feature did great work when Lemkin himself tried.

And after Jai recently reversed the temporary suspension of Grock ChattBot, users wanted to explain it directly. It suggested multiple anti -reasons for its absence, some of which NBC journalists were controversial enough Wrote about Grock It is as if it is a series of perspective, titled “Jai’s Grock on why it was drawn offline.”

Why would an AI system provide incorrect information with confidence about its own power or wrong? The answer is in understanding what the AI models are actually – and what they are not.

There’s no one at home

The first problem is conceptual: When you contact ChatzPT, Claud, Grock or Reflit, you are not talking to a series of personalities, individuals or entities. These names advise individual agents with self-knowledge but it is A Maya Conversation is built by the interface. What you actually do is guide the text generator to produce output based on your prompt.

There is no serial “Chatzipt” to interrogate its mistakes, there is no single “groke” entity that can tell you why it failed, a specific “replit” person who knows whether the database rollbacks are possible. You are interacting with a system that produces admirable-sounding text based on its training data (usually trained months or years), not true self-awareness or system knowledge that is reading everything about itself and somehow remembers it.

Once the AI language model is trained (which is a laborious, energy-intensive process), its basic “knowledge” about the world is baked on its neural network and rarely corrected. Any external data comes from the chatbot host (eg Jai or OpenAI), from the prompt provided by a software equipment that uses user or AI model Restore the external information Flying

In the case of the above grock, the main source of the chatboat for the same reply is probably in search of recent social media posts (using an external tool to restore that information), it will probably derive from a defendant report than you can expect the power of speech than any kind of self-knowledge. Beyond that, it would probably be okay Something up to something Its text-pre-expression is based on the capacity. So asking why it has done it does not give an effective answer to what it did.

LLM is the impossible of the heart

Greater language models (LLMs) alone cannot be evaluated in a number of reasons for several reasons. They usually lack any gut in their training process, they have no access to the system architecture and cannot determine the boundaries of their own performance. When you ask an AI model what can or cannot do it, it creates a response based on the recent AI models that are seen in training data-providing real self-assumptions without the current model that you are interacting with the current model.

Ay 2024 study By Binder et al. Experimentally demonstrated these limitations. Although AI models can be trained to predict their own behavior in their simple tasks, they consistently fails to “fail to do” “need for generalization of more complex tasks or out of distribution”. “Likewise, Research on “recurring confidence” Without external reactions, the attempts for self-repression have actually reduced model performances-not even better.