Principal Data Engineer

We are seeking an experienced Lead Data Engineer to provide technical leadership in designing, developing, and evolving scalable data solutions within a modern Lakehouse architecture.

The role will be responsible for leading complex data engineering initiatives, defining technical approaches and development standards, and ensuring the delivery of scalable, reliable, and well-governed data products.

The Lead Data Engineer will work closely with Data Architects, Business Analysts, Analytics teams, and business stakeholders to translate strategic and business requirements into robust technical solutions.

The role will also provide technical guidance and mentorship to Data Engineers while ensuring alignment with the organization's data architecture and governance standards.

Key Roles & Responsibilities Lead the design, development, and implementation of scalable data engineering solutions within the Lakehouse environment.

Define and drive technical approaches for data ingestion, transformation, integration, and consumption.

Lead complex ETL/ELT development initiatives involving multiple source systems and business domains.

Design and oversee scalable data pipelines using Azure Databricks, PySpark, SQL, and Delta Lake.

Lead the migration of existing data assets, SQL views, transformation logic, and data models into the Lakehouse architecture.

Define and implement data engineering standards, reusable frameworks, and best practices.

Design and evolve layered data architecture to support business intelligence, analytics, and future advanced data use cases.

Ensure data models and curated datasets are optimized for scalability, performance, and business consumption.

Review and optimize complex SQL queries, data pipelines, and processing workloads.

Define strategies for incremental data loading, historical data management, schema evolution, and pipeline recovery.

Establish data quality, validation, reconciliation, monitoring, and exception-handling processes.

Ensure data solutions comply with defined governance, security, metadata, lineage, and access management requirements.

Conduct technical reviews and provide guidance on solution design and development practices.

Mentor and support Data Engineers in improving technical capabilities and engineering practices.

Collaborate with Data Architects to ensure engineering solutions align with the overall data architecture.

Work with Business Analysts and stakeholders to understand complex business requirements and convert them into scalable technical solutions.

Support project estimation, technical planning, dependency management, and risk identification.

Drive continuous improvement in data engineering processes, automation, performance, and maintainability.

Awareness of ISO 14001 & 45001 Standards

Education Required: Graduation/Post Graduation Professional Experience Required (Max – Min.): 10-15 years Technical Competence: Strong expertise in Lakehouse architecture and modern data platform concepts.

Extensive experience with Azure Databricks.

Strong expertise in Apache Spark, PySpark, and Delta Lake.

Deep understanding of layered data architecture, including Bronze, Silver, and Gold layers.

Ability to design scalable, reusable, and maintainable data engineering solutions.

Extensive experience designing and implementing enterprise-scale ETL/ELT pipelines.

Expert-level proficiency in SQL, including complex transformations and performance optimization.

Strong hands-on experience with Python and PySpark.

Ability to establish coding standards and perform technical code reviews.

Strong expertise in analytical and dimensional data modelling.

Experience designing Fact and Dimension models, Star Schemas, and business-oriented data models.

Strong understanding of data governance, metadata management, lineage, and data quality.

Strong knowledge of Git, version control, CI/CD, and automated deployment.

Understanding of data security, access management, and compliance requirements.

Functional & Behavioral Skills: Strong ability to understand complex business processes and translate them into scalable data solutions.

Ability to work with Data Architects to convert enterprise architecture into practical engineering solutions.

Strong understanding of how data products support Business Intelligence, analytics, and reporting.

Experience engaging with senior business and technical stakeholders.

Ability to lead requirement discussions and identify technical dependencies and risks.

Experience in planning and managing complex data engineering initiatives.

Ability to assess existing data solutions and define approaches for modernization and migration.

Strong understanding of end-to-end data flow, from source systems through transformation to reporting and analytics consumption.

Ability to balance business requirements, technical feasibility, performance, and long-term maintainability.

Experience supporting and guiding multiple projects or workstreams simultaneously.

Ability to mentor, coach, and provide technical guidance to Data Engineers.

Excellent analytical, problem-solving, and decision-making skills.

Ability to influence technical decisions and drive alignment across teams.

High level of accountability and commitment to delivery.