Senior Data Engineer

We are a collective of fearless people with a clear purpose: to empower professionals around the world through intelligent workflow automation software.

Today, more than 500 people across 7 countries work with us, either remotely or in a hybrid model, making life easier for more than 3,000 companies in over 180 countries.

Since our founding in 2015, we have placed people at the center of everything we do. We invite you to learn more about this opportunity and apply to become part of our team.

Pipefy is looking for a Senior Data Engineer to design and develop data pipelines that support our journey to the moon! We are looking for someone passionate about data and solving complex problems to join an amazing team focused on building data-driven solutions.

Main Responsibilities

• Design, implement, test, deploy, and maintain stable, secure, and scalable data engineering solutions and pipelines to support data and analytics initiatives, including integrating new data sources into our data warehouses;
• Develop scalable, reusable code and engineering solutions that automate repetitive data management tasks;
• Understand business challenges and identify how data can help solve them;
• Support teammates by troubleshooting SQL, Python, and other technical issues;
• Share knowledge and mentor teammates on technologies and skills you excel in, such as SQL, Python, and related tools.

Requirements

• Proficiency in Python or other object-oriented or functional programming languages (such as Scala, Java, or C++), particularly for data manipulation and analysis, with the ability to build, maintain, and deploy data processing solutions;
• Strong experience with Data Lake architectures;
• Strong experience designing and maintaining ETL (Extract, Transform, Load) processes;
• Strong experience with Big Data technologies such as Kafka, Hadoop, Spark, or similar;
• Strong experience with data pipeline and workflow orchestration tools such as Apache Airflow, Azkaban, Luigi, or similar;
• Solid understanding of database design best practices, including normalization and performance optimization techniques such as partitioning, cursors, views, and functions;
• Experience gathering and analyzing requirements for data-driven solutions;
• Experience with stream-processing frameworks such as Spark Streaming, Apache Storm, or similar;
• Basic experience with Git for version control and Docker/Docker Compose;

• Experience with cloud infrastructure services, including:

• Amazon EC2
• Google Compute Engine
• Azure Virtual Machines (VM)
• Amazon S3
• Google Cloud Storage (GCS)
• Azure Blob Storage

• Experience with managed databases and data platforms, including:

• BigQuery
• Amazon RDS for PostgreSQL
• Amazon RDS for MySQL
• Amazon Aurora RDS