Databricks Engineer – Azure & GenAI

Databricks Engineer – Azure & GenAI

Position Summary

We are looking for a skilled Databricks Engineer with strong experience in Azure Cloud and Databricks platform services, including Generative AI capabilities within the Databricks ecosystem. This role will be responsible for designing, developing, and optimizing scalable data and AI solutions that support enterprise analytics, machine learning, and GenAI initiatives.

The ideal candidate combines strong data engineering fundamentals with hands-on experience in Databricks, Azure services, and modern AI/LLM-based workflows.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks on Azure.
• Build and optimize ETL/ELT workflows using Spark and Databricks.
• Implement and manage Delta Lake architectures and medallion data models.
• Develop and support GenAI and LLM-based solutions using Databricks AI/ML capabilities.
• Integrate Azure cloud-native services into data and AI workflows.
• Collaborate with data scientists, software engineers, and business stakeholders to deliver AI-driven solutions.
• Optimize performance, scalability, reliability, and cost efficiency of Databricks workloads.
• Implement monitoring, logging, and operational best practices for data platforms.
• Support CI/CD, infrastructure automation, and deployment processes.
• Ensure security, governance, and compliance standards are followed across cloud and data environments.
• Participate in technical discussions, architecture reviews, and solution design.

Required Qualifications

• Strong hands-on experience with Databricks platform.
• Experience building solutions in Microsoft Azure cloud environment.
• Solid knowledge of Apache Spark and distributed data processing.
• Experience with Python and SQL for data engineering and analytics workloads.
• Experience with Delta Lake and modern lakehouse architectures.
• Familiarity with Databricks AI/ML and Generative AI capabilities.
• Experience integrating LLMs, vector search, embeddings, or RAG-based solutions is a plus.
• Understanding of cloud-native data services and orchestration tools.
• Experience with Git-based development workflows and CI/CD pipelines.
• Strong troubleshooting and problem-solving skills.
• Ability to work independently in a distributed/global team environment.
• Good verbal and written English communication skills.

Preferred Qualifications

• Experience with Azure Data Factory, Azure Synapse, Azure OpenAI, or related Azure AI services.
• Familiarity with MLflow, model serving, and MLOps practices.
• Experience with infrastructure as code tools such as Terraform.
• Knowledge of containerization and orchestration technologies.
• Databricks or Azure certifications are a plus.
• Experience supporting enterprise-scale production environments.