Stanford Researchers Release AI Model Transparency Index

Researchers at Stanford University recently presented an index report called the Foundation Model Transparency Index, or FMTI, which assesses the transparency of underlying AI models. The authors urge companies to disclose more information, such as the data and human labor used to train the models.

Stanford professor Percy Liang, one of the researchers, said: “Over the past three years, it has become clear that the transparency of AI models is decreasing, while their capabilities are growing. This is problematic, because in other cases areas, such as social media, the lack of transparency has already had undesirable consequences.”

Basic models (Large Language Models (LLM) are AI systems trained on huge data sets, capable of performing a variety of tasks, from writing texts to programming. These include, for example, GPT-4 from OpenAI, Llama-2* from Meta and LaMDA from Google.

Companies developing these types of models are at the forefront of the development of generative AI, which has attracted the attention of companies of all sizes since the launch of ChatGPT by OpenAI with the active support of Microsoft.

In the index, 10 popular models were evaluated on 100 different transparency indicators. All models showed “low” results: even the most transparent model, the Llama 2 from Meta, received only 54 points out of 100. While the Titan model from Amazon took last place with a ridiculous 12 points.

At a time when the world increasingly relies on these models for decision-making, it is essential to understand their limitations and biases. The researchers also said they hoped their report would encourage companies to be more transparent about their underlying models and provide a starting point for governments looking for ways to regulate this fast-growing area.

The index is a human-centered project of the Fundamental Models Research Center at the Stanford Artificial Intelligence Institute.