[Remote] 1033- AI Engineer (Evaluation Systems)

Descrição da Vaga

Requisitos

• 4+ years total backend / ML engineering experience
• 2+ years building production AI/LLM systems
• Experience with Python, Docker and PostgreSQL
• Experience with AWS, OpenAI, Anthropic, and other LLM APIs
• Proven experience building LLM-based systems in production environments where output quality and reliability were critical
• Experience developing evaluation, QA, or scoring pipelines to assess model performance
• Strong understanding of precision/recall trade-offs and working with real-world data
• Ability to design and implement systems that produce reliable, structured outputs from LLMs
• Strong Python programming skills, including experience with asynchronous programming
• Experience deploying and managing services on AWS
• Human-in-the-loop workflow design
• OpenTelemetry familiarity

Responsabilidades

• Design a structured, configurable evaluation engine
• Combine deterministic checks with LLM-as-judge verdicts
• Build calibration workflows using expert-labeled examples
• Measure precision and recall properly (not raw agreement)
• Handle delayed outcomes and low-confidence review flows
• Store structured verdicts that power dashboards and analytics
• Monitor drift and trigger recalibration when alignment drops
• Specialized judges for specific failure patterns
• Content-level evaluation (with careful PII handling)
• Error analysis pipelines that drive workflow redesign
• A/B testing for judge versions
• Eval-driven fine-tuning data curation