AI Data Engineer III

We are actively seeking an AI Data Engineer III. This is a remote role based in Brazil.

Position Summary

The AI Data Engineer is responsible for building the knowledge layer of Rimini Street’s Agentic ERP Platform—the data pipelines, RAG (Retrieval-Augmented Generation) systems, and embedding infrastructure that give AI agents access to the right information at the right time. This role owns how knowledge is ingested, processed, indexed, and retrieved to support intelligent agent behavior.

Reporting to the Sr. Director, Engineering, this engineer designs the data architecture that powers agent intelligence—from extracting knowledge from Rimini Street’s 15+ years of support case history to building real-time retrieval systems for customer-specific context. The ideal candidate combines strong data engineering fundamentals with modern AI/ML knowledge, particularly in embeddings, vector search, and retrieval optimization.

Essential Duties & Responsibilities

RAG Pipeline Development

• Design and build RAG pipelines that retrieve relevant context from knowledge bases to augment AI agent responses.
• Implement chunking strategies optimized for different content types: support tickets, documentation, policies, transaction records, and email threads.
• Develop hybrid retrieval approaches combining dense embeddings, sparse search (BM25), and metadata filtering.
• Build query understanding and reformulation logic to improve retrieval relevance.
• Implement retrieval evaluation frameworks to measure and optimize precision, recall, and relevance.
• Design reranking pipelines that prioritize the most relevant results for agent consumption.

Embedding & Vector Infrastructure

• Implement and manage vector storage using PostgreSQL with pgvector extension, including index optimization for search performance.
• Evaluate and select embedding models appropriate for enterprise content (technical documentation, business processes, ERP terminology).
• Build embedding pipelines that process documents at scale with appropriate batching and error handling.
• Implement incremental indexing strategies for real-time updates without full reprocessing.
• Design multi-tenant vector architectures that isolate customer data while enabling efficient search.
• Monitor and optimize vector search performance: latency, accuracy, and resource utilization.

Data Ingestion & Processing

• Build data pipelines to ingest knowledge from diverse sources: Salesforce support tickets, ServiceNow cases, documentation repositories, email archives, and ERP transaction logs.
• Implement ETL processes that clean, normalize, and enrich raw data for AI consumption.
• Develop document processing pipelines: PDF extraction, HTML parsing, structured data normalization.
• Build connectors to source systems including Salesforce, ServiceNow, SharePoint, and Confluence.
• Implement data quality monitoring and alerting for ingestion pipelines.
• Design data lineage tracking to understand how knowledge flows from source to agent consumption.

Knowledge Architecture

• Design the knowledge architecture that organizes information across the Four-Spoke model: Policy Intelligence, Institutional Memory, Rimini Collective Intelligence, and Intelligent Escalation.
• Build knowledge graphs and relationship models that capture connections between ERP concepts, processes, and solutions.
• Implement metadata taxonomies that enable filtered retrieval by ERP system, module, version, customer, and topic.
• Design versioning strategies for knowledge that evolves over time (policies, procedures, best practices).
• Build feedback loops that capture which retrieved content was useful vs. ignored, enabling continuous improvement.

Cloud Data Platform Integration

• Integrate with Snowflake for large-scale data processing, leveraging Cortex AI capabilities where applicable.
• Build data pipelines that move and transform data between operational systems, Snowflake, and vector stores.
• Implement data access patterns that respect customer data isolation and security boundaries.
• Design efficient data synchronization between cloud data warehouse and real-time retrieval systems.
• Optimize query patterns for cost-effective data processing at scale.

Experience

• 5+ years of data engineering experience, with at least 1-2 years focused on AI/ML data pipelines or RAG systems.
• Hands-on experience building and optimizing RAG pipelines in production environments.
• Strong experience with vector databases, embeddings, and similarity search.
• Experience with ETL/ELT pipelines and data integration from diverse source systems.
• Production experience with PostgreSQL and SQL-based data processing.
• Background in Python for data processing and pipeline development.
• Experience with enterprise data platforms (Snowflake, Databricks, or similar) preferred.
• Exposure to enterprise software, ERP systems, or support/ticketing systems preferred.

Technical Skills

Required

• Python for data engineering: pandas, data processing pipelines, async programming.
• PostgreSQL with strong SQL skills; experience with advanced features (JSONB, full-text search, extensions).
• Vector databases and embeddings: pgvector, or experience with Pinecone, Qdrant, Weaviate, or similar.
• RAG concepts: chunking strategies, embedding models, retrieval methods, reranking.
• ETL/ELT patterns and data pipeline orchestration (Airflow, Dagster, Prefect, or similar).
• Data modeling for both relational and document-oriented use cases.
• Git version control and CI/CD practices for data pipelines.
• Understanding of API integration for data extraction (REST, GraphQL).

Preferred

• Experience with Snowflake, including Cortex AI features for vector search and ML functions.
• Familiarity with embedding models: OpenAI embeddings, Cohere, or open-source models (BGE, E5).
• Experience with LlamaIndex, LangChain, or Haystack for RAG pipeline development.
• Knowledge of document processing: PDF extraction (PyMuPDF, pdfplumber), HTML parsing, OCR.
• Experience with Salesforce and/or ServiceNow data extraction and APIs.
• Understanding of knowledge graphs and graph databases (Neo4j, or property graphs in PostgreSQL).
• Experience with data quality frameworks and monitoring tools.
• Familiarity with dbt for data transformation.
• Exposure to enterprise search systems (Elasticsearch, OpenSearch).
• Understanding of LLM fine-tuning and training data preparation.

Skills & Competencies

• Strong analytical mindset with ability to understand complex data relationships and design efficient retrieval strategies.
• Data quality obsession; understands that agent intelligence is only as good as the underlying data.
• Systems thinker who designs for scale, reliability, and maintainability.
• Collaborative; works effectively with GenAI Engineers to understand retrieval requirements and optimize for agent consumption.
• Problem solver who can diagnose and resolve data pipeline issues quickly.
• Clear communicator; able to explain data architecture decisions to technical and non-technical stakeholders.
• Self-motivated and effective in a remote environment.
• Fluent in English (written and verbal).

Desired Qualifications

• Bachelor’s or Master’s degree in Computer Science, Data Science, or related field.
• Experience in enterprise software companies or B2B SaaS platforms.
• Background in information retrieval, search systems, or NLP.
• Certifications in Snowflake, AWS Data Engineering, or similar.
• Contributions to open source data or AI/ML projects.

Location & Travel

• Location: Remote, Brazil
• Travel: Minimal; occasional travel for team meetings or training

Language

• Fluent English required (written and verbal)