Data Engineer for Software Engineering group

What will your job look like:

• Design, build, and maintain scalable data pipelines to process large datasets efficiently.
• Develop and implement data models and architectures that support both real-time and batch data processing.
• Ensure data integrity, security, and accuracy across all systems.
• Collaborate with data scientists, analysts, and other engineers to ensure data availability and quality.
• Optimize data retrieval and storage processes to handle large volumes of data seamlessly.
• Work with structured, semi-structured, and unstructured data, integrating various data sources.
• Troubleshoot and resolve data issues, ensuring continuous operation of the data infrastructure.
• Maintain and enhance ETL processes, ensuring scalability and performance in handling large datasets.
• Stay up-to-date with industry best practices and emerging technologies related to big data engineering.

All you need is:

• Bachelor’s degree in Computer Science, Engineering, or a related field.
• At least 4 years experience in data engineering, with a focus on large-scale data processing and big data technologies.
• Strong proficiency in programming languages such as Python.
• Experience with data pipeline and workflow management tools
• Hands-on experience with large-scale data processing frameworks like Apache Spark, Hadoop, or similar.
• Familiarity with data modeling, ETL processes, and data warehousing concepts.
• good knowledge of relational databases (e.g., PostgreSQL, MySQL) and NoSQL databases (e.g., Cassandra, MongoDB).

Nice to have:

• Experience working with real-time data streaming technologies (Kafka, Flink).
• Knowledge of machine learning frameworks and integrating data pipelines for model training and deployment.
• Experience with version control systems (e.g., Git), CI/CD pipelines, and automation tools.
• Experience with containerization (Docker, Kubernetes) is a plus.