Logo Fusemachines

Lead Spark Data Engineer

Fusemachinesvia Talentify
RemotoMyrtle Point, Oregon, UsSêniorCLT7 dias atrás

Salário Estimado

R$ 11.880,00 - R$ 17.820,00

0de 100

Regular

Score da Vaga

Descrição da Vaga

Fusemachines Fusemachines is a leading AI strategy, talent, and education services provider.


Founded by Sameer Maskey Ph.


D., Adjunct Associate Professor at Columbia University, Fusemachines has a core mission of democratizing AI.


With a presence in 4 countries (Nepal, the United States, Canada, and the Dominican Republic) and more than 450 full-time employees, Fusemachines brings global AI expertise to transform companies worldwide.


Founded in 2013, Fusemachines is a global provider of enterprise AI products and services, on a mission to democratize AI.


Leveraging proprietary AI Studio and AI Engines, the company helps drive the clients’ AI Enterprise Transformation, regardless of where they are in their Digital AI journeys.


With offices in North America, Asia, and Latin America, Fusemachines provides a suite of enterprise AI offerings and specialty services that allow organizations of any size to implement and scale AI.


Fusemachines serves companies in industries such as retail, manufacturing, and government.


Fusemachines continues to actively pursue the mission of democratizing AI for the masses by providing high-quality AI education in underserved communities and helping organizations achieve their full potential with AI.


Type: Full-time, Remote Job Description: We are looking for an experienced Lead Data Engineer to join our team to build the "Brain" of an IoT platform, a library that allows definition and metrics, validates it against a Virtual Schema, and generates optimized execution plans for both Spark (Batch) and Flink (Stream).


Qualification / Skill Set Requirement:

5+ years of hands-on data engineering experience with deep expertise in the Azure ecosystem.
Expert-level Java, Python and SQL. • Deep understanding of Apache Spark Internals (Catalyst Optimizer, Logical Plans).
Experience with ANTLR v4 or writing custom DSLs/Parsers. • Experience with Databricks and Delta Lake optimization.
Experience constructing Abstract Syntax Trees (ASTs). • Strong understanding of SDLC and Agile methodologies with hands-on experience in Azure DevOps, GitHub, CI/CD, and artifact management.
Skilled in data modeling, data design, and data warehousing solutions on Azure Databricks. • Knowledge of data quality, governance, and security best practices within Azure (AD, NSG, encryption, compliance).
Certifications preferred: Azure Fundamentals, Azure Data Engineer Associate, Databricks Certified Data Engineer Professional and Azure Solutions Architect Expert (nice to have).

Responsibilities • Architect, design, and implement scalable and efficient data solutions on Spark and Flink.

Implement the grammar for the IoT Query Language. • Build the Query Validator to enforce semantic constraints before a query is executed.
Develop a Spark Adapter: A translation layer that converts definition on metrics into Spark code. • Implement relationships logic (traversing a Graph/Ontology) within the core to avoid database bottlenecks.
Ensure 100% logic parity between Spark (Batch) and Flink (Stream) implementations. • Manage and optimize Azure and Databricks resources, for performance, reliability, and cost-efficiency.
Transform, clean, and prepare data using SQL, Python and Java. • Monitor and fine-tune workloads and pipelines for optimal performance and reliability.
Maintain clear documentation of solutions, configurations, and workflows. • Actively participate in Agile team activities and continuous improvement initiatives.
Promote and enforce data engineering best practices, including data governance, security, and data quality.

Fusemachines is an Equal Opportunities Employer, committed to diversity and inclusion.


All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or any other characteristic protected by applicable federal, state, or local laws.

Requisitos

  • 5+ years of hands-on data engineering experience with deep expertise in the Azure ecosystem
  • Expert-level Java, Python and SQL
  • Deep understanding of Apache Spark Internals (Catalyst Optimizer, Logical Plans)
  • Experience with ANTLR v4 or writing custom DSLs/Parsers
  • Experience with Databricks and Delta Lake optimization
  • Experience constructing Abstract Syntax Trees (ASTs)
  • Strong understanding of SDLC and Agile methodologies with hands-on experience in Azure DevOps, GitHub, CI/CD, and artifact management
  • Skilled in data modeling, data design, and data warehousing solutions on Azure Databricks
  • Knowledge of data quality, governance, and security best practices within Azure (AD, NSG, encryption, compliance)

Responsabilidades

  • Type: Full-time, Remote
  • Architect, design, and implement scalable and efficient data solutions on Spark and Flink
  • Implement the grammar for the IoT Query Language
  • Build the Query Validator to enforce semantic constraints before a query is executed
  • Develop a Spark Adapter: A translation layer that converts definition on metrics into Spark code
  • Implement relationships logic (traversing a Graph/Ontology) within the core to avoid database bottlenecks
  • Ensure 100% logic parity between Spark (Batch) and Flink (Stream) implementations
  • Manage and optimize Azure and Databricks resources, for performance, reliability, and cost-efficiency
  • Transform, clean, and prepare data using SQL, Python and Java
  • Monitor and fine-tune workloads and pipelines for optimal performance and reliability
  • Maintain clear documentation of solutions, configurations, and workflows
  • Actively participate in Agile team activities and continuous improvement initiatives
  • Promote and enforce data engineering best practices, including data governance, security, and data quality

Vagas Semelhantes

RemotoHouston, Texas, Us13 dias atrás

R$ 12k - 19k/mês

SêniorCLT

Description: Position: Software Engineer (Python + Kafka or Python + Azure Service Bus) Location: 1000 Main Street Houston, TX 77210 (Hybrid Role) The client is a large oil & gas company, this is a stable opportunity for at least for 1 year with high probability to extend for multiple years and many...

R$ 7k - 11k/mês

PlenoCLT

Um dos maiores grupos de beleza do mundo, o Grupo Boticário é uma empresa brasileira presente em mais de 40 países. É dono das marcas O Boticário, Eudora, Quem Disse, Berenice?, Vult, O.U.i, Dr. JONES, Tô.que.tô, TRUSS, e do marketplace Beleza na Web, além de atuar com produtos licenciados como Aust...

R$ 15k - 23k/mês

SêniorCLT

RELEVO LAB. Somos uma empresa de tecnologia com mais de 8 anos de atuação, vivemos e respiramos tecnologia, batalhamos incansavelmente para ser um lugar incrível para se trabalhar, onde as pessoas são tratadas com igualdade, gostem de trabalhar, se sintam em casa e esse é certamente o lugar onde que...

R$ 12k - 17k/mês

SêniorCLT

Note: The job is a remote job and is open to candidates in USA. Xealth is revolutionizing healthcare by leveraging data and automation to empower care providers. The Senior Software Engineer will design, build, and scale services and applications that enhance EHR integrations and clinical workflows,...

Interessado nesta vaga?

Candidatar-se

Você será redirecionado para o site original

Informações

NívelSênior
ContratoCLT
LocalMyrtle Point, Oregon, Us
RemotoSim
MoedaBRL
Publicada7 dias atrás
FonteTalentify

Análise de Vaga com IA

Estimativa salarial, match de tecnologias e análise de requisitos feitos com Inteligência Artificial

Powered by CodeCortex
← Voltar às Vagas