Lead Data Engineer - Azure Databricks & DevOps

Company Description

Blend is a premier AI services provider, committed to co-creating meaningful impact for its clients through the power of data science, AI, technology, and people. With a mission to fuel bold visions, Blend tackles significant challenges by seamlessly aligning human expertise with artificial intelligence. The company is dedicated to unlocking value and fostering innovation for its clients by harnessing world-class people and data-driven strategy. We believe that the power of people and AI can have a meaningful impact on your world, creating more fulfilling work and projects for our people and clients. For more information, visit www.blend360.com  

We are seeking a AI Engineer to contribute to our next level of growth and expansion. 

Job Description

We are looking for a highly skilled Senior Data Engineer with a strong background in Azure Databricks, Apache Spark, Python, SQL, and DevOps practices. The ideal candidate will design and implement robust, scalable, and high-performing data solutions while ensuring smooth deployment and automation through CI/CD frameworks.

Responsibilities

• Design, develop, and optimize data pipelines and ETL workflows using Azure Databricks and Apache Spark (PySpark).
• Build scalable data ingestion and transformation frameworks integrating multiple structured and unstructured data sources.
• Implement and manage DevOps practices for data workflows, including CI/CD, version control, and automation.
• Collaborate with cross-functional teams to deploy and monitor data applications in Azure environments.
• Write efficient and reusable Python and SQL scripts for data cleaning, transformation, and analytics.
• Implement data lakehouse architectures using Azure Data Lake and Delta Lake.
• Ensure performance tuning, cost optimization, and adherence to data security best practices.
• Troubleshoot, monitor, and resolve production issues related to data pipelines and infrastructure
• 6+ years of experience in Data Engineering or related domains.
• Build scalable data ingestion and transformation frameworks integrating multiple structured and unstructured data sources.
• Implement and manage DevOps practices for data workflows, including CI/CD, version control, and automation.
• Collaborate with cross-functional teams to deploy and monitor data applications in Azure environments.
• Write efficient and reusable Python and SQL scripts for data cleaning, transformation, and analytics.
• Implement data lakehouse architectures using Azure Data Lake and Delta Lake.
• Ensure performance tuning, cost optimization, and adherence to data security best practices.
• Troubleshoot, monitor, and resolve production issues related to data pipelines and infrastructure
• 6+ years of experience in Data Engineering or related domains.

Qualifications

• Strong hands-on experience in Azure Databricks, Apache Spark, and PySpark.
• Advanced proficiency in Python and SQL for data engineering tasks.
• Experience with DevOps tools and concepts, including:
• CI/CD pipelines (Azure DevOps, GitHub Actions, or Jenkins)
• Infrastructure as Code (ARM templates, Terraform – preferred)
• Solid understanding of Azure Data Factory, Azure Data Lake, and Azure Synapse.
• Proven experience in data pipeline orchestration, monitoring, and automation.
• Strong analytical, problem-solving, and debugging skills.
• Ability to work in a fast-paced, collaborative environment with cross-functional teams.

• Version Control (Git)

• Version Control (Git)

• CI/CD pipelines (Azure DevOps, GitHub Actions, or Jenkins)
• Infrastructure as Code (ARM templates, Terraform – preferred)
• Solid understanding of Azure Data Factory, Azure Data Lake, and Azure Synapse.
• Proven experience in data pipeline orchestration, monitoring, and automation.
• Strong analytical, problem-solving, and debugging skills.
• Ability to work in a fast-paced, collaborative environment with cross-functional teams.
• CI/CD pipelines (Azure DevOps, GitHub Actions, or Jenkins)
• Infrastructure as Code (ARM templates, Terraform – preferred)
• Solid understanding of Azure Data Factory, Azure Data Lake, and Azure Synapse.
• Proven experience in data pipeline orchestration, monitoring, and automation.
• Strong analytical, problem-solving, and debugging skills.
• Ability to work in a fast-paced, collaborative environment with cross-functional teams.