Senior Data Engineer (Azure) – Aspiring Data Architect

Job Description:

Responsibilities

• Design, build, and optimize end-to-end data ingestion, transformation, and serving pipelines on Azure (batch and streaming).
• Contribute to solution architecture: define target-state data platform, reference patterns, and architecture artifacts (HLD/LLD, diagrams, and NFRs).
• Implement Lakehouse/warehouse solutions using proven patterns (e.g., medallion architecture, dimensional modeling, and data vault where applicable).
• Partner with stakeholders to translate business requirements into scalable data models, data products, and SLAs.
• Drive data quality, observability, and reliability (validation rules, reconciliation, monitoring, alerting, and runbooks).
• Implement security and governance: RBAC, managed identities, Key Vault, encryption, data access policies, and lineage.
• Establish CI/CD and infrastructure-as-code practices for data workloads; enforce coding and review standards.
• Optimize performance and cost across storage, compute, and orchestration.
• Mentor junior engineers and lead technical discussions across engineering, DevOps, and analytics teams.

Qualification & Experience

• Azure Data Engineering
• Strong hands-on experience with Azure data services such as Azure Data Factory (ADF), Azure Synapse
• Analytics, Azure Databricks, and/or Microsoft Fabric (Data Engineering).
• Experience with storage and lakehouse components: ADLS Gen2, Delta Lake / Parquet, and data partitioning strategies.
• Experience with relational and NoSQL stores (e.g., Azure SQL, SQL Server, Cosmos DB) and data integration patterns.
• Streaming and event-driven exposure: Event Hubs / Kafka, Stream Analytics, and/or Spark Structured Streaming.
• Programming & Data
• Strong SQL skills; ability to write efficient, maintainable transformations and tune queries.
• Proficiency in Python and/or Scala (preferred) for data processing and automation.
• Solid understanding of data modeling (dimensional/star schema), metadata management, and data quality concepts.
• Platform Engineering
• Experience with Git-based development and CI/CD using Azure DevOps or GitHub Actions.
• Infrastructure-as-code exposure (Terraform, Bicep/ARM) and environment promotion strategies (dev/test/prod).
• Observability practices: logs, metrics, lineage, and pipeline monitoring (e.g., Azure Monitor, Log Analytics).
• Architecture Mindset
• Ability to reason about non-functional requirements: performance, scalability, resiliency, security, and cost.
• Strong communication skills to explain trade-offs and guide stakeholders toward practical solutions.
• Nice-to-Have:
• Experience with Microsoft Purview (catalog, lineage, governance) and enterprise data security policies.
• Exposure to API-based integration patterns and services such as Azure Functions, Logic Apps, and API
• Management.
• Experience with containerization and orchestration (Docker, AKS) for data workloads.
• Familiarity with analytics/BI layers (Power BI) and semantic modeling concepts.
• Cloud certifications (preferred): DP-203, AZ-305 (or equivalent).
• Education: Bachelor’s degree in Computer Science/Engineering (or equivalent practical experience).
• Experience: 8–10 years