
Scicom Infrastructure Services is a technology consulting and services company that helps public and private sector organizations manage and improve their IT environments. Its work spans enterprise performance management, architecture, project management, infrastructure and application management, and managed services. The company also works with AppDynamics, BigPanda, Datadog, and AWS on performance monitoring, event correlation and automation, cloud monitoring, and IT service management. Scicom provides round-the-clock operational support for global infrastructure. Its profile describes a team working across enterprise, SaaS, and B2B technology; check current openings for details about roles and hiring.
Databricks Data Engineer, OCDS Data Contracts
Connect procurement stakeholders with engineering teams to define data contracts for a large Databricks program. Build OCDS-aligned pipelines and controls for reliable, standardized contracting data.
Description
- Connect business, contracting, governance, and engineering teams on a large Databricks implementation.
- Lead requirements workshops with procurement, program, analytics, governance, and technical stakeholders.
- Create and maintain data contracts that define how producers and consumers exchange data.
- Turn business needs into user stories, acceptance criteria, process flows, mappings, interface specifications, and engineering requirements.
- Analyze source systems, profile data, assess gaps, and map source fields to target structures.
- Compare current procurement data with the structures required by OCDS.
- Bring data owners, producers, consumers, architects, and governance teams to agreement.
- Track requirements through data modeling, implementation, testing, and acceptance.
- Assess how proposed changes affect data products, reports, integrations, and analytical use cases.
- Help prepare and run backlog reviews, sprint planning, demos, testing, and stakeholder acceptance.
- Apply OCDS to procurement and public-contracting data from planning through implementation.
- Maintain mappings for OCDS releases, records, packages, identifiers, organizations, parties, items, milestones, documents, transactions, amendments, and related elements.
- Use OCDS schemas, codelists, validation rules, and implementation guidance.
- Identify when OCDS extensions are needed and record the related assumptions, rules, mappings, exceptions, and extensions.
- Build contracting records from transactional releases and define how amendments, updates, cancellations, corrections, and historical changes are handled.
- Check transformed data against OCDS JSON schemas and fix missing, invalid, inconsistent, or nonconforming procurement data.
- Enable publication, exchange, analysis, or internal use of standardized contracting data.
- Build, test, and maintain Databricks pipelines with Apache Spark, PySpark, Python, and SQL.
- Ingest and transform structured and semi-structured procurement data from JSON, CSV, XML, Parquet, databases, APIs, and files.
- Create Bronze, Silver, and Gold layers with the medallion architecture.
- Develop normalized, dimensional, analytical, and OCDS-based data models.
- Use Delta Lake to enforce and evolve schemas, manage versions, support audits, and process data reliably.
- Create reusable methods for converting source procurement data into OCDS-compatible outputs.
- Build both batch and incremental data ingestion.
- Use suitable Databricks tools, including Workflows, notebooks, jobs, Auto Loader, Delta Live Tables, or Lakeflow.
- Build REST API connections for source ingestion and downstream data delivery.
- Automate data-quality, reconciliation, completeness, and conformity checks.
- Help implement Unity Catalog for metadata, ownership, lineage, access controls, and data discovery.
- Improve Spark jobs, SQL queries, clusters, partitioning, file sizes, and data layouts.
- Contribute to code reviews, automated tests, CI/CD, deployments, monitoring, and production support.
- Diagnose pipeline failures, data mismatches, and performance problems.
- Set measurable rules for data accuracy, completeness, validity, timeliness, consistency, and uniqueness.
- Create validation controls for OCDS fields, identifiers, dates, amounts, currencies, organizations, classifications, and contracting relationships.
- Build dashboards or reports on data-contract compliance and data-quality results.
- Set up ways to detect and manage schema drift.
- Trace procurement data lineage through Databricks transformations to downstream products.
- Work with governance teams to define owners, stewards, classifications, retention, and access policies.
- Handle sensitive procurement and supplier data according to security and privacy requirements.
- Maintain audit trails with documented rules, versioned mappings, validation results, and change histories.
Requirements
- Bachelor’s degree in computer science, information systems, data analytics, business analysis, engineering, public administration, supply-chain management, or a related discipline.
- At least five years of combined experience in data engineering, data analysis, business analysis, or data integration.
- Practical experience implementing or using the Open Contracting Data Standard (OCDS).
- Experience creating, documenting, negotiating, or maintaining data contracts.
- Experience mapping procurement or contracting data to OCDS schemas.
- Strong command of OCDS releases, records, schemas, codelists, identifiers, contracting stages, and validation methods.
- At least three years of hands-on Databricks experience.
- Strong skills in Apache Spark, PySpark, Python, SQL, Delta Lake, ETL/ELT pipelines, data modeling, JSON and JSON Schema, REST APIs, data-quality validation, and source-to-target mapping.
- Experience eliciting requirements and turning them into actionable engineering specifications.
- Clear communication with both technical and nontechnical partners.
- Experience preparing user stories, acceptance criteria, business rules, data dictionaries, interface specifications, and process documentation.
- Strong analysis, troubleshooting, facilitation, and documentation skills.
- Experience delivering work in Agile teams.
- Preferred: background in government procurement, public-sector contracting, grants, acquisition, supplier, or financial data.
- Preferred: experience building OCDS extensions or adapting OCDS to organizational needs.
- Preferred: familiarity with procurement classifications, organization identifiers, tendering, awards, amendments, milestones, transactions, and contract implementation.
- Preferred: Databricks Certified Data Engineer Associate or Professional credential.
- Preferred: experience with Unity Catalog, Delta Live Tables, Lakeflow, Databricks Workflows, Structured Streaming, cloud services, Azure Data Factory, ADLS Gen2, AWS Glue, Amazon S3, Snowflake, dbt, Kafka, Airflow, Power BI, JSON Schema validators, automated data-quality frameworks, metadata or master-data management, lineage, reference-data governance, CI/CD, Git, Azure DevOps, GitHub Actions, Jenkins, or Terraform.
- Preferred: experience contributing to large federal, state, local-government, or regulated-enterprise data programs.
- Preferred: familiarity with federal acquisition, procurement, reporting, transparency, or open-data requirements.
- Able to contribute to both engineering and business-analysis discussions.
- Strong grasp of data contracts and producer-consumer accountability.
- Able to turn complex procurement processes into clear data structures and transformation rules.
- Careful when interpreting schemas, codelists, business definitions, and validation requirements.
- Strong stakeholder facilitation and conflict resolution abilities.
- Able to find gaps and ambiguities before they cause engineering defects.
- Committed to documentation, traceability, governance, and data quality.
- Able to work effectively in a large, multidisciplinary Databricks team.
Benefits
- The posting does not specify benefits, perks, or additional compensation.
Related Jobs
No jobs found
Try adjusting or clearing your filters to find more opportunities.