Allata
Allata
Allata is a global consulting firm that helps organizations turn digital strategy into practical business improvements. Its teams work across strategy, customer experience, cloud and technology, data, and artificial intelligence, supporting clients with modernization, personalization, and operational efficiency. Allata serves sectors including automotive, healthcare, logistics, financial services, and high tech, drawing on experience from former IT leaders and consulting professionals to guide complex digital initiatives.

Databricks Data Platform Manager (Remote, Texas)

Lead architecture and delivery of governed Lakehouse and enterprise data warehouse platforms for Allata. Guide data pipelines, security, scalability, and data product development.

विवरण

  • Set architecture, technology choices, reference patterns, and integration blueprints for Lakehouse and enterprise data warehouse platforms.
  • Create conceptual, logical, and physical models for multi-tenant and industry-specific data products.
  • Standardize data layers for ingestion, raw storage, curation, and serving.
  • Establish governance practices for metadata, catalogs, lineage, data contracts, and classification.
  • Define security and compliance controls for access, masking, encryption, network segmentation, and auditing.
  • Plan for platform scalability, high availability, disaster recovery, capacity, and cost management.
  • Lead selection and integration of Databricks, Delta Lake, Delta Live Tables, Fivetran, Azure Data Factory or Data Fabric, orchestration, and monitoring tools.
  • Set and enforce CI/CD practices for data assets, including tests, automated releases, and rollback plans.
  • Define batch and streaming ingestion, compaction, partitioning, and storage approaches.
  • Set observability standards for metrics, SLAs, dashboards, logs, traces, and alerts.
  • Provide technical leadership and mentorship to Data Engineering teams.
  • Review architecture and code for critical components.
  • Work with Data Product Owners and Security, Infrastructure, BI, and ML stakeholders to turn requirements into solutions and roadmaps.
  • Build, test, and deploy processing modules with Spark, PySpark or Scala, Spark SQL, and stored procedures.
  • Develop and tune data pipelines on Databricks and database platforms.
  • Apply DevOps practices, infrastructure as code, CI/CD, automated testing, and version control.
  • Resolve complex data quality, performance, and availability problems, and drive ongoing improvements.

आवश्यकताएँ

  • Prior experience as an architect or technical lead for enterprise data platforms.
  • Hands-on knowledge of Databricks technologies, including Delta Lake, Unity Catalog, Delta Live Tables, Auto Loader, and Structured Streaming.
  • Strong skills in Spark, PySpark and/or Scala, Spark SQL, and distributed job optimization.
  • Background in data warehouse and Lakehouse design.
  • Practical knowledge of Medallion, Lambda, and Kappa patterns.
  • Experience integrating SaaS, ETL, or connector tools such as Fivetran.
  • Experience with orchestration tools such as Airflow, Azure Data Factory, and Data Fabric.
  • Experience with relational and hybrid databases, including MS SQL Server, PostgreSQL, Oracle, Azure SQL, and AWS RDS/Aurora or equivalents.
  • Proficiency in data pipeline CI/CD using Azure DevOps, GitHub Actions, Jenkins, or similar tools.
  • Experience packaging and deploying artifacts such as .whl files and containers.
  • Experience with batch and streaming processing, file compaction, partitioning, and storage tuning.
  • Understanding of cloud security, IAM/RBAC, encryption, VPC/VNet concepts, and cloud networking.
  • Familiarity with observability and monitoring tools such as Prometheus, Grafana, Datadog, or native cloud services.
  • Automation experience with CI/CD pipelines and trunk-based development is a plus.
  • Advanced PySpark skills are a plus.
  • Advanced Spark workflow optimization and orchestration skills are a plus.
  • Databricks, cloud architecture, or data certifications are a plus.
  • Ability to identify, troubleshoot, and resolve complex data problems.
  • Strong collaboration and communication skills.
  • Curiosity and openness to new tools, technologies, and methods.

लाभ

  • Full-time role.
  • Equal opportunity and inclusive work environment.
  • Commitment to diversity.

संबंधित नौकरियाँ

The Athletic

Senior Manager, Data Analytics

The Athletic

Lead Growth Product Insights at The Athletic, a sports media company. Manage analysts and use experimentation to inform subscription and product decisions.

खोलें
Lunar Solar Group

Marketing Measurement Data Scientist (Remote, US)

Lunar Solar Group

Build attribution, incrementality, and media mix measurement for Lunar Solar Group, a growth consultancy and digital marketing agency. Turn analysis into practical client budget recommendations.

खोलें
Airbnb

Senior Data Scientist, People Analytics Research

Airbnb

Lead people analytics research at Airbnb, using causal inference and statistical modeling to inform workforce, compensation, recruiting, and retention decisions. Work with stakeholders on sensitive analyses and reporting.

खोलें
The Cigna Group

Senior Lead Data Builder, Remote (US)

The Cigna Group

Build reusable Silver and Gold data products for Cigna healthcare services, using Databricks and governed MVP and CI/CD pipelines. Turn business needs into production-ready data solutions and establish standards across Data Operations.

खोलें
Clear Street

Senior Growth Data Scientist

Clear Street

Improve the evaluation, reliability, and safety of Clear Street’s AI copilot. The role supports a cloud-native brokerage and trading platform for sophisticated investors.

खोलें