OpenAI is encountering performance limitations with its new Orion large language model (LLM), which is expected to succeed GPT-4. While Orion is showing better performance in some areas compared to its predecessors, it's not demonstrating the same level of improvement seen between GPT-3 and GPT-4. The improvements are notably smaller, particularly in tasks such as coding.

The limited availability of high-quality training data is a major contributing factor to these challenges.. As LLMs have consumed a significant amount of publicly available data, finding new sources of good-quality training data is becoming more difficult and expensive. OpenAI is exploring several strategies to address this challenge. These strategies include:

Creating a “foundations team” to investigate methods for improving LLMs with declining data availability.

Training Orion on synthetic data generated by other AI models.

Optimizing LLMs at a stage after the initial training phase..

Reliance on synthetic data, however, poses its own risks, such as model collapse, where future models merely replicate the abilities of their predecessors. OpenAI is aware of these risks and is working on strategies to mitigate them.

The cost of developing and running these advanced models is also rising due to the increasing computational resources required for training. This raises concerns about the economic viability of future LLM development.

Despite these challenges, OpenAI remains committed to innovation and is exploring new approaches to enhance AI models. The company is also considering a potential shift in the naming scheme for its next-generation AI model, moving away from the "ChatGPT" naming convention.