Azure Data Engineering Lead

Job Description

Position Summary:
We are seeking a talented Data Engineer with a strong background in data engineering to join our team. You will play a key role in designing, building, and maintaining data pipelines using a variety of technologies, with a focus on the Microsoft Azure cloud platform.

Responsibilities:
Design, develop, and implement data pipelines using Azure Data Factory (ADF) or other orchestration tools.
Write efficient SQL queries to extract, transform, and load (ETL) data from various sources into Azure Synapse Analytics.
Utilize PySpark and Python for complex data processing tasks on large datasets within Azure Databricks.
Collaborate with data analysts to understand data requirements and ensure data quality.
Implement data governance practices to ensure data security and compliance.
Monitor and maintain data pipelines for optimal performance and troubleshoot any issues.
Develop and maintain unit tests for data pipeline code.
Work collaboratively with other engineers and data professionals in an Agile development environment.
Developing Modern Data Warehouse solutions using Azure Stack (Azure Data Lake, Azure Data Factory, Azure Databricks)
Coding complex Spark (Scala or Python) and T-SQL
Proficient in a source code control system such as GIT
Ability to design the solutions using Azure data services.
Manage the governance within the team- track the productivity and quality of team members and report the overall progress to customer periodically.
Provide technology leadership to team, provide them technical guidance and support, solve the blockers.
Define work breakdown structure and estimates for big work items.

Preferred Skills & Experience:
Good knowledge of PySpark & working knowledge of Python
Full stack Azure Data Engineering skills (Azure Data Factory, DataBricks and Synapse Analytics)
Experience with large dataset handling
Exposure to DevOps Basics.
Exposure to Basics of Release Engineering.