Destaque

AI/LLM Evaluation & Alignment Software Engineer

LeoTechvia Himalayas

RemotoRemotoPlenoCLT2 dias atrás

Salário Estimado

R$ 11.250,00 - R$ 13.333,00

Tecnologias

Python Go Elasticsearch AWS Azure Kubernetes Git GitHub API iOS IA Data Science PyTorch

0de 100

Excelente

Score da Vaga

Descrição da Vaga

At LeoTech, we are passionate about building software that solves real-world problems in the Public Safety sector.

Our software has been used to help the fight against continuing criminal enterprises, drug trafficking organizations, identifying financial fraud, disrupting sex and human trafficking rings and focusing on mental health matters to name a few.

Role • This is a remote, WFH role.

•As an AI/LLM Evaluation & Alignment Engineer on our Data Science team, you will play a critical role in ensuring that our Large Language Model (LLM) and Agentic AI solutions are accurate, safe, and aligned with the unique requirements of public safety and law enforcement workflows.

You will design and implement evaluation frameworks, guardrails, and bias-mitigation strategies that give our customers confidence in the reliability and ethical use of our AI systems.

This is an individual contributor (IC) role that combines hands-on technical engineering with a focus on responsible AI deployment.

You will work closely with AI engineers, product managers, and DevOps teams to establish standards for evaluation, design test harnesses for generative models, and operationalize quality assurance processes across our AI stack.

Core Responsibilities • Build and maintain evaluation frameworks for LLMs and generative AI systems tailored to public safety and intelligence use cases.

•Design guardrails and alignment strategies to minimize bias, toxicity, hallucinations, and other ethical risks in production workflows. • Partner with AI engineers and data scientists to define online and offline evaluation metrics (e.g., model drifts, data drifts, factual accuracy, consistency, safety, interpretability).

•Implement continuous evaluation pipelines for AI models, integrated into CI/CD and production monitoring systems. • Collaborate with stakeholders to stress test models against edge cases, adversarial prompts, and sensitive data scenarios.

•Research and integrate third-party evaluation frameworks and solutions; adapt them to our regulated, high-stakes environment. • Work with product and customer-facing teams to ensure explainability, transparency, and auditability of AI outputs.

•Provide technical leadership in responsible AI practices, influencing standards across the organization. • Contribute to DevOps/MLOps workflows for deployment, monitoring, and scaling of AI evaluation and guardrail systems (experience with Kubernetes is a plus).

•Document best practices and findings, and share knowledge across teams to foster a culture of responsible AI innovation.

What We Value • Bachelor's or Master's in Computer Science, Artificial Intelligence, Data Science, or related field.

•3–5+ years of hands-on experience in ML/AI engineering, with at least 2 years working directly on LLM evaluation, QA, or safety. • Strong familiarity with evaluation techniques for generative AI: human-in-the-loop evaluation, automated metrics, adversarial testing, red-teaming.

•Experience with bias detection, fairness approaches, and responsible AI design. • Knowledge of LLM observability, monitoring, and guardrail frameworks e.g Langfuse, Langsmith

•Proficiency with Python and modern AI/ML/LLM/Agentic AI libraries (LangGraph, Strands Agents, Pydantic AI, LangChain, HuggingFace, PyTorch, LlamaIndex). • Experience integrating evaluations into DevOps/MLOps pipelines, preferably with Kubernetes, Terraform, ArgoCD, or GitHub Actions.

•Understanding of cloud AI platforms (AWS, Azure) and deployment best practices. • Strong problem-solving skills, with the ability to design practical evaluation systems for real-world, high-stakes scenarios.

•Excellent communication skills to translate technical risks and evaluation results into insights for both technical and non-technical stakeholders.

Technologies We Use • Cloud & Infrastructure: AWS (Bedrock, SageMaker, Lambda), Azure AI, Kubernetes (EKS), Terraform, ArgoCD.

•LLMs & Evaluation: HuggingFace, OpenAI API, Anthropic, LangChain, LlamaIndex, Ragas, DeepEval, OpenAI Evals. • Observability & Guardrails: Langfuse, GuardrailsAI.

•Backend & Data: Python (primary), ElasticSearch, Kafka, Airflow. • DevOps & Automation: GitHub Actions, CodePipeline.

What You Can Expect • Work from home opportunity

•Enjoy great team camaraderie. • Thrive on the fast pace and challenging problems to solve.

•Modern technologies and tools. • Continuous learning environment.

•Opportunity to communicate and work with people of all technical levels in a team environment. • Grow as you are given feedback and incorporate it into your work.

•Be part of a self-managing team that enjoys support and direction when required. • 3 weeks of paid vacation – out the gate!!

•Competitive Salary. • Generous medical, dental, and vision plans.

•Sick, and paid holidays are offered.

LeoTech is an equal opportunity employer and does not discriminate on the basis of any legally protected status.

Vagas Semelhantes

[Remote] AI/LLM Evaluation & Alignment Software Engineer

LeoTechJobright

RemotoUs11 dias atrás

R$ 11k - 13k/mês

PlenoCLT

Note: The job is a remote job and is open to candidates in USA. LeoTech is passionate about building software that solves real-world problems in the Public Safety sector. The AI/LLM Evaluation & Alignment Software Engineer will ensure that Large Language Model (LLM) and Agentic AI solutions are accu...

Python Go AWS Azure Kubernetes+6

3 weeks of paid vacation – out the gate!!Generous medical, dental, and vision plansSick, and paid holidays are offered

Ver Detalhes

Desenvolvedor Full-Stack Python React

FCamaraJobLeads

RemotoBrOntem

R$ 7k - 11k/mês

PlenoCLT

Descrição da vaga Buscamos um engenheiro apaixonado por construir produtos robustos, escaláveis e bem arquitetados — do backend ao frontend, com fluência em cloud. Você vai integrar um time de engenharia que constrói plataformas de alto volume, com APIs críticas, pipelines assíncronos e interfaces m...

TypeScript React Python Go CSS+16

Ver Detalhes

Engenheiro De Dados E Ml

Caderno NacionalJooble

São Paulo4 dias atrás

R$ 9k - 14k/mês

PlenoCLT

Engenheiro De Dados E Ml - Detalhes da Vaga. ● O(a) candidato(a) ideal será responsável por construir a infraestrutura de dados moderna que alimenta soluções de Machine Learning e IA Generativa, desde a ingestão e transformação de dados até a implantação de modelos em produção. ● Este papel híbrido ...

Python Go Azure Docker Kubernetes+8

Ver Detalhes