Data Engineer - Databricks Specialist

We are seeking an experienced Data Engineer with deep expertise in Databricks to design, build, and maintain scalable data pipelines and analytics solutions. This role requires 5 years of hands-on experience in data engineering with a strong focus on the Databricks platform.

Key Responsibilities:

• Data Pipeline Development & Management

• Design and implement robust, scalable ETL/ELT pipelines using Databricks and Apache Spark

• Process large volumes of structured and unstructured data

• Develop and maintain data workflows using Databricks workflows, Apache Airflow, or similar orchestration tools

• Optimize data processing jobs for performance, cost efficiency, and reliability

• Implement incremental data processing patterns and change data capture (CDC) mechanisms

Databricks Platform Engineering:

• Build and maintain Delta Lake tables and implement medallion architecture (bronze, silver, gold layers)

• Develop streaming data pipelines using Structured Streaming and Delta Live Tables

• Manage and optimize Databricks clusters for various workloads

• Implement Unity Catalog for data governance, security, and metadata management

• Configure and maintain Databricks workspace environments across development, staging, and production

Data Architecture & Modeling:

• Design and implement data models optimized for analytical workloads

• Create and maintain data warehouses and data lakes on cloud platforms (Azure, AWS, or GCP)

• Implement data partitioning, indexing, and caching strategies for optimal query performance

• Collaborate with data architects to establish best practices for data storage and retrieval patterns

Performance Optimization & Monitoring:

• Monitor and troubleshoot data pipeline performance issues

• Optimize Spark jobs through proper partitioning, caching, and broadcast strategies

• Implement data quality checks and automated testing frameworks

• Manage cost optimization through efficient resource utilization and cluster management

• Establish monitoring and alerting systems for data pipeline health and performance

Collaboration & Best Practices:

• Work closely with data scientists, analysts, and business stakeholders to understand data requirements

• Implement version control using Git and follow CI/CD best practices for code deployment

• Document data pipelines, data flows, and technical specifications

• Mentor junior engineers on Databricks and data engineering best practices

• Participate in code reviews and contribute to establishing team standards

Required Qualifications

Experience & Skills:

• 5+ years of experience in data engineering with hands-on Databricks experience

• Strong proficiency in Python and/or Scala for Spark application development

• Expert-level knowledge of Apache Spark, including Spark SQL, DataFrames, and RDDs

• Deep understanding of Delta Lake and Lakehouse architecture concepts

• Experience with SQL and database optimization techniques

• Solid understanding of distributed computing concepts and data processing frameworks

• Proficiency with cloud platforms (Azure, AWS, or GCP) and their data services

• Experience with data orchestration tools (Databricks Workflows, Apache Airflow, Azure Data Factory)

• Knowledge of data modeling concepts for both OLTP and OLAP systems

• Familiarity with data governance principles and tools like Unity Catalog

• Understanding of streaming data processing and real-time analytics

• Experience with version control systems (Git) and CI/CD pipelines

Preferred Qualifications

• Databricks Certified Data Engineer certification (Associate or Professional)

• Experience with machine learning pipelines and MLOps on Databricks

• Knowledge of data visualization tools (Power BI, Tableau, Looker)

• Experience with infrastructure as code (Terraform, CloudFormation)

• Familiarity with containerization technologies (Docker, Kubernetes)