Data & ML Automation (Databricks)

Job Description:

Responsibilities

• Automation & Tooling
• Develop CLI tools, scripts, and utilitiesto automate repetitive workflows across the data platform, including:
• Pipeline execution
• Governance and metadata synchronization
• Environment setup
• Test harness creation
• Automate Databricks workflows, job deployments, environment provisioning, and MLOps operations using:
• Databricks APIs
• Terraform
• Databricks SDK
• Data & ML Workflow Testing
• Design and implement:
• Integration tests
• End?to?end pipeline tests
• Validation frameworks for Databricks ETL/ELT pipelines and ML inference workflows
• Improve reliability, observability, and overall engineering productivity across the data & ML team.
• Internal Tools & Dashboards
• Build quick internal applications using React, Streamlit, or similar frameworks to:
• Visualize data flows
• Provide model inference demos
• Enable operational or configuration controls
• Develop internal productivity and monitoring dashboards
• Engineering Practices
• Apply best practices around:
• Spec?driven development
• Coding agents and automation patterns
• CI/CD workflows for data/ML systems
• Nice-to-Have Skills
• Experience creating dashboards and visualizations using Databricks AI/BI or any modern BI tool (Power BI, Tableau, Looker, etc.)