Senior Data Engineer

Job Description:

Responsibilities

• Architect, build, and maintain state-of-the-art Enterprise Data Warehouse / Lakehouse solutions that serve both batch and near-real-time analytics use cases
• Design and implement robust ETL / ELT pipelines using Python and Apache Airflow (or modern orchestration equivalents)
• Develop and operate real-time data streaming and processing platforms using open-source technologies such as Apache Kafka, Apache Spark Streaming / Structured Streaming, Flink, or equivalent
• Handle sensitive financial, production, and customer data systems while strictly adhering to SOX controls, segregation of duties, change management, and audit requirements
• Partner closely with business sponsors, product managers, manufacturing engineers, service operations, finance, and IT/security teams to gather requirements, scope projects, and deliver high-quality solutions quickly
• Communicate complex technical concepts and business impact effectively through written documentation, verbal discussions, architecture diagrams, and executive-level presentations (360-degree communication)
• Define, enforce, and continuously improve engineering standards, coding best practices, testing methodologies, CI/CD patterns, monitoring & alerting, and quality assurance processes
• Actively participate in design reviews, code walkthroughs, and pull request reviews across the team
• Stay current with evolving open-source technologies and recommend adoption when they provide meaningful differentiation or operational efficiency
• Provide global 24×7 data support on a rotating basis (on-call) and own ETL / streaming pipeline health monitoring, alerting, and incident resolution

Qualification & Experience

• 6+ years of professional experience as a Data Engineer, Backend Engineer, or ETL developer building large-scale data platforms
• Strong proficiency in Python for data engineering (pandas, PySpark, SQLAlchemy, etc.)
• Deep hands-on experience designing and operating Airflow DAGs in production at scale
• Production experience with at least one distributed streaming system (Kafka, Kafka Streams, Spark Streaming, Flink, Pulsar, etc.)
• Solid understanding of data modeling for analytical workloads
• Experience building and operating systems under SOX compliance or similarly regulated environments (change control, audit trails, separation of duties, etc.)
• Strong SQL skills and understanding of distributed query engines
• Experience with containerization (Docker) and orchestration (Kubernetes / ECS) is highly desirable
• Excellent communication skills — able to explain technical trade-offs to engineers and business value to non-technical stakeholders