FDA Contracts Cognita To Evaluate LLM Radiology Reports. Regulators Arrive Fashionably Late To The Party.

Cognita Imaging won an FDA contract starting June 22, running 18 months, to test methods for evaluating AI-generated radiology reports. The core problem: current tools are evaluated against panels of radiologist experts, but generative AI produces varied outputs across diverse cases, making that approach insufficient. The FDA also granted breakthrough designation to Aidoc for AI that reads chest X-rays.

This demonstrates evaluation drift. The mechanism is that generative models produce non-deterministic outputs, meaning the same input can yield different responses. Traditional evaluation assumes stable outputs. When that assumption breaks, you need new evaluation frameworks. The lesson: regulation lags capability, and the gap matters more in medicine than in most domains.

Cognita Imaging holds the FDA research contract. Aidoc received breakthrough device designation for chest X-ray AI. The FDA is the regulatory body actively studying how to govern generative AI in medical devices, having released a discussion paper last month.