A recent simulation revealed the limitations of AI in handling routine office tasks. In the scenario, a virtual new hire struggled with a simple assignment: allocating personnel to web development projects based on client budgets and team availability. The AI-powered employee hit a roadblock when unable to close a pop-up window restricting file access. Instead of clicking the obvious “X” (close) button, it ineffectively reached out to the human resource (HR) manager for help, failing to complete the task.
This virtual employee was part of a study by Carnegie Mellon University researchers to assess AI agents’ real-world capabilities. Business Insider reported on Tuesday that the experiment aimed to evaluate AI’s effectiveness in actual work environments.
Tech giants like Google, Amazon, and OpenAI are heavily investing in generative AI agent development. The corporate interest is significant, with a Deloitte survey revealing that 25% of C-suite executives seriously consider implementing autonomous agents. However, AI’s practical workplace performance remains largely unproven.
The Carnegie Mellon team tested AI models from Google, OpenAI, Anthropic, and Meta in simulated finance, administration, and software engineering scenarios. The results were underwhelming. Even the top-performing model, Anthropic’s Claude 3.5 Sonnet, only completed 25% of the assigned tasks. Other models, including Google’s Gemini 2.0 Flash and ChatGPT, managed a mere 10% completion rate. Researchers concluded that AI failed to handle most tasks across all tested fields.
The study highlighted clear limitations in AI agents. They lacked empathy, social skills, and complex problem-solving abilities. For example, when instructed to add content to an answer.docx file, the AI misinterpreted the task and failed to recognize the file format. Frequent misunderstandings in simulated colleague interactions and disregard for instructions further emphasized AI’s struggles with nuanced, multi-layered work environments.
While AI shows promise in boosting workplace efficiency, it’s far from ready to replace human workers. Companies increasingly view AI as a complementary tool rather than a complete employee replacement. The study concludes that while AI can be a powerful assistant, it still has significant ground to cover before it can match human creativity and social intelligence in the workplace.