Krafton is developing an AI-native hiring and assessment solution that evaluates not only candidates’ final results but also the process of how they collaborate with AI.
According to Krafton, its internally developed “Cofa-Probe” evaluates how candidates define problems, work with AI, and verify their results.
Cofa-Probe recreates an environment similar to the actual work of AI engineers, allowing candidates to interact with multiple AI agents to solve problems.
Unlike traditional assessments that focus primarily on completed outputs, Cofa-Probe evaluates the entire process, including problem definition, assigning roles and giving instructions to AI, searching for supporting evidence, revising decisions, recovering from errors, and verifying results.
The system automatically collects not only source code generated during the task but also conversations with AI, prompts, tool settings, and verification records.
After completing the task, candidates participate in a follow-up Q&A session explaining their results and decision-making process. The system automatically evaluates candidates according to competency-based criteria and generates reports for interviewers.
The evaluation covers three areas: Technique, Intent, and Cognition.
Krafton assesses the execution, structure, and security of candidates’ code, as well as whether they clearly communicated goals and constraints to AI and properly verified its outputs. It also evaluates whether candidates can explain why their results work and where they could fail.
Krafton has also introduced measures to improve consistency in AI-based assessments.
LLM-based evaluations can produce different results when the same submission is evaluated multiple times or fail to distinguish meaningful differences in competency when submissions use different styles.
Cofa-Probe incorporates a “meta-harness,” which repeatedly tests and improves AI evaluation agents.
The system evaluates the same submission multiple times to determine whether scores and grades remain within a consistent range. It also compares submissions with different approaches and levels of completeness to determine whether the AI can distinguish actual differences in competency.
Elements requiring consistency, such as evaluation criteria, evidence lists, and score conversion, are controlled through code, while qualitative judgments requiring contextual understanding are handled by LLM-based evaluation agents.
Krafton applied Cofa-Probe in an actual participant assessment during “Cofathon: AI Native Battleground,” an AI hackathon held with CJ Olive Young as a recruitment-linked program on July 30.
Participants were divided into Krafton’s FDE (Forward Deployed Engineer) and CJ Olive Young AI Engineer tracks and worked on challenges designed to solve real-world inefficiencies using AI. OpenAI participated as a technology partner.
Krafton collected participants’ AI collaboration processes from task execution through submission and turned them into evaluation reports.
Based on the blind evaluation reports, public discussions were also held for each track. The reports were used to select winners by reviewing not only individual scores but also the reasoning behind those scores.
Krafton plans to further refine Cofa-Probe’s evaluation criteria and agent environment based on the operational experience and feedback gathered from the hackathon.
Park Jae-min, head of Krafton’s AI Frontier division, said, “In the AI era, competitiveness comes from defining problems with AI, verifying the evidence, and thoroughly validating the results,” adding that Krafton will continue improving Cofa-Probe to establish a new standard for evaluating AI-native talent.