Design, develop, and maintain real-time data streaming pipelines using Spark Streaming or Azure Functions. We are seeking a highly skilled Senior Data Engineer or Data Architect with extensive experience in Synapse and Azure data services. The ideal candidate will have a deep understanding of Spark and Lakehouse architectures, as well as a proven track record of implementing and managing data solutions using Synapse services and other Azure tools.
Key Responsibilities
- Design, develop, and maintain data architectures using Spark and Lakehouse methodologies.
- Utilize Synapse services including Spark pools, dedicated SQL pools, and pipelines to build robust data solutions.
- Leverage other Azure data services to enhance and optimize data workflows.
- Implement and manage CI/CD pipelines to ensure smooth deployment and integration processes.
Qualifications
- Strong expertise in Spark and Lakehouse architectures.
- Hands-on experience with Synapse services: Spark pools, dedicated SQL pools, and pipelines.
- Proficiency with additional Azure data services.
- Solid experience with CI/CD processes and tools.
- Excellent problem-solving skills and the ability to work in a fast-paced environment.
- Join our team and contribute to cutting-edge data solutions in a dynamic and innovative setting.
- Load, merge, and process machine logs from Kafka, ensuring efficient data flow and transformation.
- Integrate processed data into Redis cache and send it to the data lake for long-term storage and analysis.
- Implement and optimize data processing solutions using Python.
- Apply software engineering best practices, including code reviews, version control, and continuous integration/deployment (CI/CD).
- Collaborate with cross-functional teams to understand requirements and deliver high-quality data solutions.
Requirements:
· Proven experience in real-time data streaming using Spark Streaming or Azure Functions.
· Strong proficiency in Python for data processing and automation tasks.
· Hands-on experience with Kafka for data ingestion and message queuing.
· Familiarity with Redis for caching and fast data retrieval.
· Knowledge of data lake architectures and best practices for data storage and retrieval.
· Solid understanding of software engineering principles, including design patterns, testing, and documentation.
· Strong problem-solving skills and ability to work in a fast-paced, collaborative environment.
Preferred Qualifications:
· Experience with cloud platforms such as Azure
· Knowledge of other programming languages and frameworks related to data engineering.
· Familiarity with data modeling, ETL processes, and data lake concepts.