LLM Evaluation Engineer

ThirdLaw Molecularvia Remote Rocketship

RemotoRemotoPlenoCLT7 dias atrás

Salário Estimado

R$ 8.349,00 - R$ 12.524,00

Tecnologias

Python API IA TensorFlow PyTorch

0de 100

Excelente

Score da Vaga

Descrição da Vaga

Job Description:

•Build the evaluation layer in the ThirdLaw platform for LLM prompts and responses

•Design and tune guardrails, classifiers, and semantic judgment systems in real-time • Implement evaluation strategies with semantic similarity, foundation model scoring, and rule-based systems

•Integrate model outputs with downstream enforcement actions (e.g. redaction, escalation, blocking) • Prototype, tune, and productize small language models for classification, labeling, or scoring

•Collaborate with data infrastructure engineers to connect evaluation logic with ingestion and storage • Build tools to observe, debug, and improve evaluator performance across data distributions

•Define abstractions for reusable evaluation components that can scale across use cases Requirements:

•7+ years of experience in ML systems or AI engineering roles

•At least 1–2 years working directly with LLMs, NLP pipelines, or semantic search • Deep understanding of foundation models (e.g.

OpenAI, Claude, Mistral, Llama) and APIs • Hands-on experience with vector search (e.g.

FAISS, Qdrant, Weaviate) and embeddings pipelines • Proven ability to implement real-time or near-real-time evaluation logic using semantic similarity, classifier scoring, or structured rules

•Strong in Python, with familiarity using libraries like Hugging Face Transformers, LangChain, and PyTorch or TensorFlow • Ability to reason about model behavior, test prompt configurations, and debug complex decision logic in production Benefits:

•Generous benefits • Market cash compensation

•Above-market equity • Well-designed benefits

Vagas Semelhantes

Desenvolvedor de Agente IA - Remoto

GetronicsBeBee

RemotoGoiânia, Goiás, BrHoje

R$ 7k - 10k/mês

PlenoCLT

A Getronics é líder global em soluções de tecnologia, com uma equipe de mais de 4.000 colegas em 22 países, fornecendo serviços abrangentes de ponta a ponta em todo o mundo. Temos o compromisso de oferecer um atendimento excepcional ao cliente, permitindo que nossos clientes foquem em seus principai...

Python AWS Azure GCP Docker+8

Ver Detalhes

Desenvolvedor Python Pleno (com foco em Inteligência Artificial)

FCamaraFCamara - Gupy

RemotoBrHoje

R$ 8k - 12k/mês

PlenoCLT

Descrição da vaga A FCamara está em busca de uma pessoa Desenvolvedor Python Pleno (com foco em Inteligência Artificial), para se juntar a nossa equipe dinâmica e atuar em parceria com a maior construtora da América Latina, que ao longo de sua existência, sempre esteve sintonizada com o mercado e as...

React Vue Python Go Docker+10

Ver Detalhes

Data Scientist - Remote Sensing

KplerBeBee

RemotoBrHoje

R$ 8k - 12k/mês

PlenoCLT

At Kpler, we are dedicated to helping our clients navigate complex markets with ease. By simplifying global trade information and providing valuable insights, we empower organisations to make informed decisions in commodities, energy, and maritime sectors. Since our founding in 2014, we have focused...

Python Go AWS GCP API+5

Ver Detalhes