• ETL Pipeline Development: Design, develop, and maintain scalable ETL pipelines using Azure Databricks and PySpark to process large volumes of data.
• Data Transformation: Perform complex data transformations and aggregations, ensuring data quality and consistency.
• Data Integration: Integrate various data sources, including ADO, SNOW, and third-party APIs, into a unified data platform.
• Performance Optimization: Optimize ETL processes for performance, scalability, and cost-effectiveness using best practices in Azure Databricks and PySpark.
• Collaboration: Work closely with stakeholders to understand data requirements and deliver actionable insights.
• Automation: Automate data pipeline workflows using Azure Data Factory and ensure robust scheduling and monitoring.
• Documentation: Maintain comprehensive documentation of data workflows, processes, and systems.
• Compliance and Security: Ensure all data processing complies with security and data governance standards
At Birlasoft we combine the power of domain, enterprise, and digital technologies to reimagine business potential. Surpassing expectations, breaking convention!
