Python Data Engineer/Developer - GCP SME, Spark/PySpark - REMOTE WORK - 67471

Descrição da Vaga

Requisitos

• Python, Spark/PySpark and GCP expertise is a must have
• The still need strong Data Engineer's who are hands on in Python but put an emphasis on their GCP, BQ and API skills
• Require aggregations and stuff
• Someone with good BigQuery PySpark skills
• Experience in data warehousing
• Experience building data warehouses
• Background in working with transaction data (not just master data)
• Experience with facts and aggregations
• Involves API work
• Building Python APIs to read analytics databases like BigQuery (and potentially AWS databases)
• Specific Skill Needs (for APIs):
• Someone with API building with Python
• Concern about the network and provider team giving the right requirements for consolidating data
• Need to reach out to multiple teams as the data is very different
• Difficulty in defining metrics like "claims paid data" (e.g., bill charges vs. actual check amount)
• May need to supplement expertise from another domain (e.g., someone with knowledge of provider data, claim data)
• Overview: At a high level, they have migrated from Hadoop to GCP for data processing
• Have a GCP data environment, predominantly for big data applications on the cloud
• Seeking 3-5 Senior Level Data Engineers with strong Python skills to support ongoing data migration and ingestion efforts
• Python, Google Cloud Functions to execute the scripts with Google Kubernetes Engine (GKE)
• Should have experience in working with denormalized data types, both structured and unstructured data
• Using Cloud SQL as relational cloud database, but okay with others i.e
• Folks who can have some background in developing AI applications would be the ideal profile
• If we found a strong ML or AI candidate with Python programming skills, they could potentially find a space for them
• Strong hands-on Python programming
• Spark/PySpark
• GCP (BigQuery, Dataproc, Google Cloud Functions, GKE, Cloud SQL - would not consider all must haves, just general awareness of the GCP ecosystem and data services)
• Experience working with various data types and structures
• AI experience - building AI systems, models, or building inference pipelines and processing data "for AI"

Responsabilidades

• One of our clients is looking for a Python Data Engineer/Developer - GCP SME, Spark/PySpark to join their team remotely
• Involve dealing with numbers and large volumes of claims data
• Source systems: get data from multiple external channels - provider data, healthcare groups, hospitals, etc. send data and their platform processes it and provides to operation systems
• Their end state is not a data warehouse for analytics - but the data directly feeds applications
• Currently, a lot of the data ingestion is being done manually and they are looking to automate
• Their data pipelines are PySpark, Scala/Spark run on Dataproc for larger volumes

Benefícios