AWS Data Engineer Python, Dataiku (San Francisco) Job at Compunnel, Inc., San Francisco, CA

Ly9IVkVvVWwyOW9pVWxPS1p6UnlWQ3E3Z2c9PQ==
  • Compunnel, Inc.
  • San Francisco, CA

Job Description

We are looking for an experienced AWS Data Engineer to design, develop, and maintain scalable data pipelines using AWS native technologies.

This role involves working closely with technical analysts, stakeholders, data scientists, and other team members to ensure data quality, optimize storage solutions, and support data platform modernization efforts.

The ideal candidate will have hands-on experience with cloud migration, data transformation, and automation strategies.

Key Responsibilities

  • Lead and support data platform modernization projects.
  • Design and develop robust, scalable data pipelines using AWS native services.
  • Optimize ETL processes for efficient data transformation and performance.
  • Migrate workflows from on-premise environments to AWS cloud, ensuring data quality and consistency.
  • Design automations and integrations to resolve data inconsistencies and quality issues.
  • Perform system testing and validation to ensure successful integration and functionality.
  • Implement security and compliance controls within the cloud environment.
  • Conduct data validation checks pre- and post-migration to ensure completeness, consistency, and accuracy.
  • Collaborate with data architects and developers to document manual workflows and design automation strategies.
Required Qualifications
  • Minimum of 7 years of experience in data engineering using AWS technologies (AWS Glue, Python, Snowflake, S3, Redshift).
  • Strong experience in designing and developing scalable data pipelines.
  • Proficiency in Snowflake for data transformations and ETL optimization.
  • Experience with both streaming and batch data pipeline architectures.
  • Familiarity with DataOps concepts and CI/CD pipeline setup on AWS.
  • Hands-on experience with Databricks and a willingness to expand capabilities.
  • Knowledge of Dataiku for data preparation and workflow automation.
  • Strong problem-solving and analytical skills.
  • Bachelors or Masters degree in Computer Science or a related field.
Preferred Qualifications
  • Experience with AWS EMR, Lambda, and other cloud-based data engineering tools.
  • Exposure to data governance and compliance frameworks.
#J-18808-Ljbffr

Job Tags

Full time,

Similar Jobs

KRE Security

Weekend Security Officer (Saturday & Sunday Shifts) Job at KRE Security

 ...Job Description Job Description Weekend Security Officer (Saturday & Sunday Shifts) Location: Tamaqua Pay Rate: $15.50 per hour We are currently hiring Security Officers for weekend shifts . This position is ideal for dependable individuals seeking steady... 

Springfield Hospital Inc.

LNA/Care Aide (Adult Day) - Per Diem Job at Springfield Hospital Inc.

 ...Job Description Job Description Description: The Adult Day Care Aide/LNA will: Assist participants with activities of daily living and other personal care including ambulation, transfers, toileting, food preparation, feeding, nail care and bathing. Assist with... 

Powerforce recycling

Scale Operator Job at Powerforce recycling

 ...Overview: We are currently seeking a reliable, detail-oriented Scale Operator to join our team. This position plays a key role in the daily...  ...Maintain cleanliness and organization in the scale house and work area Provide excellent customer service and represent... 

Westgate Resorts

Lifeguard Job at Westgate Resorts

 ...surroundings and excitement that our world-class resort offers when you join the Westgate Lakes team. Job Description As a Lifeguard, you will: Ensure the safety of all guests by maintaining constant surveillance, continuously scanning the pool or water park... 

Soliant

Sign Language Interpreter Full Time Job at Soliant

20252026 School Year Position Overview A highly regarded Connecticut school district is seeking a dedicated Sign Language Interpreter to provide 1:1 support for an elementary student during the 20252026 school year. This is a full-time opportunity with a billing...