Data Engineer

Responsibilities

• Design, develop, and maintain scalable data pipelines and ETL processes to efficiently extract, transform, and load high volume of data from various sources into our data lake.
• Develop and implement data quality checks and monitoring processes to ensure data integrity and accuracy.
• Leverage AWS services (e.g., EKS, EMR, S3, Glue, Lambda) to build and maintain cloud-based data infrastructure.
• Implement CI/CD pipelines for data engineering processes.
• Utilize Infrastructure as Code (IaC) tools like CloudFormation and Terraform to manage AWS resources.
• Stay up-to-date with the latest data engineering technologies and trends.

Qualifications

• Bachelor's degree in Computer Science, Engineering, or a related field.
• Minimum of 3 years of experience as a Data Engineer.
• Strong proficiency in SQL, Python, and Spark.
• Familiarity with relational databases like PostgreSQL.
• Hands-on experience with AWS cloud services (EKS, EMR, S3, Athena, Lambda, etc).
• Experience with Kafka and real-time data processing.
• Experience with Tableau and knows about the setup and maintenance.
• Experience with data lake and lake house architecture.
• Experience with Databricks and Delta Lake.
• Proficiency in Apache Airflow for workflow orchestration.
• Proficiency in Kubernetes and Docker for containerization and orchestration.
• Experience with CI/CD pipelines and tools.
• Experience with Infrastructure as Code (IaC) tools like CloudFormation and Terraform.
• Excellent problem-solving and analytical skills.
• Strong attention to detail and ability to work independently.
• Excellent communication and collaboration skills