Data Engineer with Databricks

TechFabric is seeking an experienced Data Engineer with deep expertise in Databricks to design, build, and maintain scalable data pipelines and analytics solutions for an enterprise platform providing bank account validation services. This role requires at least 5 years of hands-on data engineering experience with a strong focus on the Databricks platform.

• Design and implement robust, scalable ETL/ELT pipelines using Databricks and Apache Spark
• Build and maintain Delta Lake tables and implement medallion architecture (bronze, silver, gold layers)
• Develop streaming data pipelines using Structured Streaming and Delta Live Tables
• Design and implement data models optimized for analytical workloads
• Monitor and troubleshoot data pipeline performance issues and optimize Spark jobs
• Implement data quality checks and automated testing frameworks
• Work closely with data scientists, analysts, and business stakeholders to understand data requirements
• Mentor junior engineers on Databricks and data engineering best practices
• Manage and optimize Databricks clusters for various workloads
• Implement Unity Catalog for data governance, security, and metadata management

• 5+ years of experience in data engineering with hands-on Databricks experience
• Strong proficiency in Python and/or Scala for Spark application development
• Expert-level knowledge of Apache Spark, including Spark SQL, DataFrames, and RDDs
• Deep understanding of Delta Lake and Lakehouse architecture concepts
• Experience with SQL and database optimization techniques
• Solid understanding of distributed computing concepts and data processing frameworks
• Proficiency with cloud platforms (Azure, AWS, or GCP) and their data services
• Experience with data orchestration tools (Databricks Workflows, Apache Airflow, Azure Data Factory)
• Knowledge of data modeling concepts for both OLTP and OLAP systems
• Familiarity with data governance principles and tools like Unity Catalog
• Understanding of streaming data processing and real-time analytics
• Experience with version control systems (Git) and CI/CD pipelines

Preferred Qualifications: Databricks Certified Data Engineer certification (Associate or Professional), Experience with machine learning pipelines and MLOps on Databricks, Knowledge of data visualization tools (Power BI, Tableau, Looker), Experience with infrastructure as code (Terraform, CloudFormation), Familiarity with containerization technologies (Docker, Kubernetes)