Medical Expert AI Evaluation Specialist

About the role

This remote position supports a leading AI research initiative focused on healthcare. You will apply your clinical expertise to create realistic medical scenarios, evaluate AI‑generated responses, and help improve the reasoning capabilities of next‑generation AI models.

Key responsibilities

• Develop specialty‑specific clinical cases that reflect real‑world practice, covering diagnosis, treatment planning, risk stratification, differential diagnosis, and guideline‑based decision making.
• Produce reference‑quality solutions that demonstrate attending‑physician level reasoning and align with current evidence‑based guidelines.
• Assess AI‑generated medical responses using structured evaluation frameworks, identify strengths, weaknesses, and reasoning errors, and provide detailed written feedback.
• Participate in onboarding, specialty calibration meetings, and collaborate with multidisciplinary research teams to ensure consistency and quality across evaluations.

Required profile

• Board‑certified attending physician with an active, unrestricted medical license, or a resident in the final year of an accredited program (board‑eligible), or a fellow (board‑certified or board‑eligible) in a primary specialty.
• Physicians from a wide range of specialties, including Family Medicine, Pediatrics, Anesthesiology, General Surgery, ENT, PM&R, Occupational Medicine, and others.

Required skills

What we offer

• Fully remote, 100% asynchronous work.
• Typical commitment of ~20 hours per week with flexibility for additional hours.
• Compensation ranging from $130 to $180 per hour.