Design, develop, and maintain robust data pipelines using Java within AWS infrastructure.
Implement scalable solutions for data analysis and transformation using Apache Spark and PySpark.
Utilise Airflow for efficient workflow orchestration in complex data processing tasks.
Ensure fast and interactive querying capabilities through the use of Presto.
2. Infrastructure Management:
Containerise applications using Docker for streamlined deployment and scaling.
Orchestrate and manage containers effectively with Kubernetes in production environments.
Implement infrastructure as code using Terraform for provisioning and managing AWS resources.
3. Collaboration and Communication:
Collaborate with cross-functional teams to understand data requirements and architect scalable solutions aligned with business goals.
Ensure data quality and reliability through robust testing methodologies and monitoring solutions.
Stay updated with emerging technologies and industry trends to continuously enhance the data engineering ecosystem.
Skills Required
Machine Learning
Python
Find more jobs at Devexperts
Devexperts provides financial software and services for brokerages, exchanges and wealth management firms in equity, options, futures and Forex markets.
