EXL
EXL
EXL is a business consulting and services company that helps organizations use data, analytics, and technology to improve operations and make better decisions. Its work includes operations management, decision analytics, and digital transformation, with experience supporting clients in healthcare, insurance, finance, and logistics. EXL combines data science with practical business expertise to develop solutions tailored to each organization’s goals, operating model, and industry context.

Databricks Tech Lead at EXL — Chennai, Hybrid

Lead Databricks lakehouse engineering for financial crime platforms. Build Delta Lake pipelines and manage Unity Catalog across development, test, and production.

Description

  • Build and maintain Databricks workspaces, clusters, and compute pools across development, test, and production.
  • Set up Unity Catalog governance, access controls, fine-grained permissions, and data lineage.
  • Tune cluster types, autoscaling, and spot or preemptible nodes to balance cost and performance.
  • Establish workspace structures, access controls, and secret management using Databricks Secrets, Azure Key Vault, or AWS Secrets Manager.
  • Manage Databricks jobs, workflows, and multi-task orchestration, including dependencies.
  • Design Delta Lake tables with partitioning and Z-ordering, and compact files with OPTIMIZE and VACUUM.
  • Organize structured data lakes into Bronze, Silver, and Gold layers using the Medallion Architecture.
  • Develop Delta Live Tables pipelines with declarative ETL/ELT and built-in data quality expectations.
  • Handle schema changes, table versions, time travel, and Change Data Feed for incremental processing.
  • Create lakehouse patterns that connect Delta Lake with Kafka, ADLS, S3, and GCS.
  • Build Databricks pipelines for financial crime platforms covering AML, KYC, CRA, sanctions screening, transaction monitoring, fraud detection, and regulatory reporting.

Requirements

  • Bachelor’s or master’s degree in computer science, information technology, data engineering, or a related discipline.
  • At least 8 years of experience in data engineering or software engineering.
  • At least 3 years of hands-on Databricks experience in production.
  • Strong grounding in big data engineering, cloud data platforms, and distributed computing.
  • In-depth experience with Databricks Workspaces, Clusters, Jobs, Workflows, and Repos.
  • Proficiency in Unity Catalog, including metastore setup, catalog, schema and table management, access controls, and data lineage.
  • Practical experience developing, monitoring, and setting expectations for Delta Live Tables pipelines.
  • Strong understanding of Delta Lake internals, including its transaction log, ACID guarantees, file layout, and optimization methods.
  • Experience with Databricks SQL Warehouses, SQL Analytics, and dashboard development.
  • Knowledge of the Databricks Photon engine, serverless compute, and cost optimization.

Related Jobs

Weekday

Senior Full-Stack Java Engineering Manager

Weekday
501 – 1,000 Employees
eCommerceFashionRetail

Lead engineering teams building scalable Java and Spring Boot services with modern frontend applications. Guide architecture, reliability, mentorship, and delivery for a software development client.

Open
Nokia

Senior Software Engineer, Cloud-Native Platforms

Nokia

Build Nokia’s MantaRay NM cloud-native network observability platform, working across distributed systems, Kubernetes, and CI/CD automation. Apply AI-assisted engineering tools and automation to improve software development.

Open
Nokia

Senior Software Engineer, Cloud-Native Platforms

Nokia

Build Nokia’s MantaRay network management platform for telecommunications providers. Lead cloud-native distributed systems, Kubernetes, CI/CD automation, and AI-assisted engineering.

Open