The posting
Key Responsibilities • Data Pipeline Architecture and Delivery: Lead the design and implementation of maintainable batch and streaming pipelines that ingest, validate, transform and publish structured, semi-structured and unstructured data across AWS and Databricks. • Lakehouse and Data Product Engineering: Design Bronze, Silver and Gold data layers using Delta Lake or equivalent patterns; develop reusable domain data models, curated datasets and data products with explicit ownership, definitions and service expectations. • Source Integration and Incremental Processing: Implement reliable ingestion from databases, files, APIs and event streams using full-load, incremental, change-data-capture and replayable processing patterns appropriate to each source. • Data Quality and Contracts: Establish schema enforcement, data contracts, automated validation, reconciliation and anomaly checks covering completeness, validity, consistency, uniqueness, timeliness and referential integrity. • Orchestration and Deployment Automation: Build parameterised workflows using Databricks Workflows or Lakeflow and relevant AWS services; integrate automated testing, nfrastructure as code and CI/CD controls across development, test and production environments. • Metadata, Lineage and Governance: Implement technical metadata, business definitions, end-to-end lineage, classification, ownership and access controls using Unity Catalog, AWS Glue Data Catalog, Lake Formation or approved platform capabilities. • Pipeline Monitoring and Reliability: Define dashboards, alerts, logging, operational metrics and runbooks for pipeline health, freshness, data-quality failures, latency, throughput and downstream impact; lead troubleshooting and root-cause analysis. • Performance and Cost Optimisation: Optimise Spark and SQL workloads, file sizes, partitioning, clustering, caching, compaction, cluster configuration and workload scheduling to improve runtime, reliability and AWS/Databricks cost efficiency. • Security and Data Protection: Apply least-privilege IAM, encryption, secrets management, environment separation, audit logging, sensitive-data controls and secure data-sharing patterns in line with applicable agency requirements. • Technical Leadership and Quality: Set engineering standards, review designs and code, mentor engineers, manage technical debt and promote reusable components rather than project-specific pipelines. • Stakeholder Collaboration: Work with data owners, stewards, analysts, data scientists, architects, product teams and governance stakeholders to translate analytical and operational needs into implementable data specifications. • Documentation and Knowledge Transfer: Maintain architecture decisions, source-to-target mappings, data models, contracts, operating procedures and support documentation, and transfer knowledge to the internal team throughout the engagement.
Requirements & Qualifications A. Educational Qualification & Experience Education: Degree in Computer Science, Software Engineering, Data Engineering, Information Systems or a related technical discipline. Experience: Typically 7–10 years of relevant data or software engineering experience, including hands-on design and operation of production data pipelines and at least 2 years working with cloud-based data platforms. Candidates should demonstrate technical leadership through architecture, engineering standards, code review, mentoring or ownership of critical data services.
B. Technical & Professional Competencies Essential (Must-Have) • Strong Python, PySpark and SQL capability, including distributed processing, complex transformations, query optimisation, debugging and automated testing. • Hands-on experience with Databricks, Apache Spark and Delta Lake for production-scale batch or streaming data engineering. • Hands-on AWS data engineering experience with a relevant mix of S3, Glue, Lake Formation, IAM, KMS, Lambda, Step Functions, EventBridge, CloudWatch, Redshift and MSK/Kinesis. • Experience designing lakehouse or data-lake architectures, including layered data patterns, dimensional or domain data modelling and reusable curated datasets. • Experience implementing incremental ingestion, change data capture, idempotency, late-arriving data handling, backfills, replay and recovery. • Experience implementing data quality, schema validation, reconciliation, metadata and lineage as automated parts of production pipelines. • Experience with workflow orchestration, Git-based development, CI/CD, infrastructure as code such as Terraform, and environment promotion controls. • Ability to implement pipeline observability, operational dashboards, alerting, runbooks, incident diagnosis and service recovery. • Working knowledge of cloud security, least-privilege access, network boundaries, encryption, secret management, data classification and audit logging. • Ability to review technical designs and code, establish engineering standards, and guide less-experienced engineers.
Preferred (Advantageous) • Experience with Databricks Unity Catalog, Lakeflow or Delta Live Tables, Auto Loader, Structured Streaming, Delta Sharing and Databricks Asset Bundles. • Experience with Apache Airflow, dbt, Kafka, Amazon MSK/Kinesis, AWS DMS or equivalent ingestion and orchestration technologies. • Experience designing data products, data contracts, semantic or context layers, master/reference data patterns or data mesh-aligned operating models. • Experience supporting feature engineering, ML training datasets, feature stores or MLOps pipelines on Databricks and AWS. • Experience in Singapore public-sector environments, including IM8-aligned controls, data classification, VAPT, system assurance or governed cross-agency data sharing. • Relevant certifications such as Databricks Certified Data Engineer Professional, AWS Certified Data Engineer – Associate or AWS Certified Solutions Architect – Associate/Professional.
C. Soft Skills & Communication Attributes • Strong analytical and systems-thinking capability, with the discipline to identify failure modes, edge cases and operational dependencies before production deployment. • Clear written and verbal communication, including the ability to explain data architecture, quality, lineage, security, cost and delivery trade-offs to technical and management audiences. • Collaborative working style across data engineering, platform engineering, analytics, data science, architecture, cybersecurity and governance teams. • Ability to translate ambiguous business needs into clear data definitions, contracts, acceptance criteria and incremental delivery plans. • Strong ownership and production mindset, including careful documentation, support readiness and follow-through on incidents and technical debt. • Ability to mentor engineers constructively, raise engineering standards and build sustainable in-house capability rather than creating vendor dependency.
