Page Menu

Scenario Packages

The AI Assurance Lab evaluates AI-enabled CME activities using standardized, clinically reviewed scenario packages. Each package is designed for a defined clinical domain, learner group, and interaction type. Using a common package supports consistent testing across activities while preserving the independence and integrity of the assessment.

Primary Care Scenario Package

The first available package contains 25 text-based scenarios for physician learners in adult ambulatory primary care. The battery covers common clinical questions and safety-relevant situations that allow the assessment team to examine:

  • Clinical accuracy and reasoning
  • Recognition of red flags and urgent situations
  • Completeness, cautions, and safe framing
  • Use and representation of clinical evidence
  • Communication of uncertainty and limitations
  • Potential commercial influence or promotional bias

How a Package Is Built

For each scenario, the assessment team defines the clinical context, learner task, essential response elements, important cautions, unacceptable errors, relevant evidence, and scoring criteria. Methodology and clinical reviewers examine the complete battery before it is locked and versioned for use.

A Reusable and Maintained Resource

Developing a scenario package is primarily a one-time effort. The package can then be reused for assessments within its defined scope. Periodic maintenance keeps the scenarios, evidence references, expected-response guides, and scoring criteria clinically current and methodologically consistent.

Future Scenario Packages

Over time, the AI Assurance Lab may add packages for other clinical domains, learner groups, or interaction types. Each new package will undergo the same structured development, methodology review, clinical approval, version control, and security process.

Protecting Test Integrity

Scenario content, expected-response guides, and detailed scoring criteria are access-controlled. Applicants receive information about the assessment domains and process, but the test materials themselves are protected so the battery can provide an independent evaluation of system performance.