LLM Evaluation Engineer

ThirdLaw Molecularvia Remote Rocketship

RemotoRemotoPlenoCLTOntem

Salário Estimado

R$ 8.349,00 - R$ 12.524,00

Tecnologias

Python API IA TensorFlow PyTorch

0de 100

Excelente

Score da Vaga

Descrição da Vaga

Job Description:

•Build the evaluation layer in the ThirdLaw platform for LLM prompts and responses

•Design and tune guardrails, classifiers, and semantic judgment systems in real-time • Implement evaluation strategies with semantic similarity, foundation model scoring, and rule-based systems

•Integrate model outputs with downstream enforcement actions (e.g. redaction, escalation, blocking) • Prototype, tune, and productize small language models for classification, labeling, or scoring

•Collaborate with data infrastructure engineers to connect evaluation logic with ingestion and storage • Build tools to observe, debug, and improve evaluator performance across data distributions

•Define abstractions for reusable evaluation components that can scale across use cases Requirements:

•7+ years of experience in ML systems or AI engineering roles

•At least 1–2 years working directly with LLMs, NLP pipelines, or semantic search • Deep understanding of foundation models (e.g.

OpenAI, Claude, Mistral, Llama) and APIs • Hands-on experience with vector search (e.g.

FAISS, Qdrant, Weaviate) and embeddings pipelines • Proven ability to implement real-time or near-real-time evaluation logic using semantic similarity, classifier scoring, or structured rules

•Strong in Python, with familiarity using libraries like Hugging Face Transformers, LangChain, and PyTorch or TensorFlow • Ability to reason about model behavior, test prompt configurations, and debug complex decision logic in production Benefits:

•Generous benefits • Market cash compensation

•Above-market equity • Well-designed benefits

Vagas Semelhantes

Desenvolvedor de Agente IA - Remoto

GetronicsBeBee

RemotoGoiânia, Goiás, BrHoje

R$ 7k - 10k/mês

PlenoCLT

A Getronics é líder global em soluções de tecnologia, com uma equipe de mais de 4.000 colegas em 22 países, fornecendo serviços abrangentes de ponta a ponta em todo o mundo. Temos o compromisso de oferecer um atendimento excepcional ao cliente, permitindo que nossos clientes foquem em seus principai...

Python AWS Azure GCP Docker+8

Ver Detalhes

Machine Learning Engineer (Python, Java, React, LLM Focus)

STAFFXPERT LLCDice

RemotoRemotoHoje

R$ 10k - 15k/mês

PlenoCLT

Job Title: Machine Learning Engineer (Python, Java, React, LLM Focus) Location: Remote Job Summary STAFFXPERT LLC is seeking a Python / Machine Learning Developer on behalf of our client in a remote environment. This role focuses on building and deploying scalable machine learning solutions while co...

React Python Java Go AWS+7

Ver Detalhes

Data Scientist - Remote Sensing

KplerBeBee

RemotoBrOntem

R$ 8k - 12k/mês

PlenoCLT

At Kpler, we are dedicated to helping our clients navigate complex markets with ease. By simplifying global trade information and providing valuable insights, we empower organisations to make informed decisions in commodities, energy, and maritime sectors. Since our founding in 2014, we have focused...

Python Go AWS GCP API+5

Ver Detalhes