A high schooler built a website that lets you challenge AI models to a Minecraft build-off

As AI Benchmarking The strategies prove to be insufficient, that AI builders are turning to more creative ways to evaluate the capacity of generator AI models. For a group of developers, it is a sandbox-building game owned by Minecraft, Microsoft.

Website Minecraft benchmark (Or MC-Bench) was developed in collaboration to beat the AI ​​models against each other in the head challenges to respond to the requests with the creation of the Minecraft. Users can vote on which model has done better and after voting they will find that each AI has created each mend.

MC-Bench starting for the twelfth grade Adi Singh, the value of Minecraft is not so played, but people are familiar with it-this is what Maximum sale All -time video games. Even for those who have not played the game, it is still possible to evaluate any blocked representation of pineapple better.

“Minecraft lets people see progress [of AI development] Many more easily, “Singh told TechCrunch.” People are accustomed to me in the mind, accustomed to appearance and vibe. “

MC-Bench currently lists eight people as volunteers. According to the website of anthropic, Google, OpenAI and Alibaba MC-Bench, the benchmark prompts have subsidized the use of their products to run, but companies are not otherwise approved.

“We are just doing ordinary buildings to reflect how far we have come from the GPT -1 era now [we] Singh says, “I could see myself scaling myself on this long-form plan and goal-based work.” Games said. “Games are only safer than real life and more controllable for testing, it can be a means of examining the argument that makes my eyes more ideal.”

Like other games Pokémon Red, StreetAnd Pics Used as experimental criteria for AI, because in the part of the benchmarking AI’s art InnocentThe

Researchers often examine AI models Standard evaluationHowever, many of these tests provide AI to a home-field. Due to the way they are trained, models are naturally given gifts to solve specific, narrow types of problems, especially for solving the problem, rotation or basic extrapolation is required.

Simply put, OpenAE’s GPT -4 can score on the 88th Persentile at LSAT, it is hard to collect what it means to collect, but it cannot detect it The word “strawberry” is how many RS there are. Ethnographic CLOD 3.7 Sonnet A standardized software is achieved 62.3% accuracy in the engineering benchmark, but it is worse to play Pokémon than most of the five -year -old kids.

MC-Bench is technically a programming benchmark, since models are asked to write codes like “Frosty the Snowman” or “A Piegrant Sandy Shore” to create a prompted build like a fascinating tropical beach.

However, it is easier to evaluate whether Snowman looks better than digging in the code on behalf of most MC-Bench users, which gives the project a broad application-and thus the possibility of collecting more data on what models consistently score better.

It is definitely in favor of debate whether these scores are high in the path of AI utility. Sing Singh Ser Ser claimed that they were a powerful signal, though.

“Current Leaderboard reflects very closely to my own experience to use these models, which is not like a lot of authentic text criteria,” said Singh. “May be” [MC-Bench] The organizations may be useful for the organization to find out if it is going the right direction ”