The posting
Senior Data Engineer – AWS & Databricks (Cloud Data Platform)
Contract Duration: 12 Months Contract Period: January 2027 – December 2027 Number of Vacancies: 1 Working Hours: Monday to Friday, 8:30 AM – 6:00 PM Work Arrangement: Hybrid Location: Singapore
Job Overview
We are seeking an experienced Senior Data Engineer to join a cloud data platform and architecture team supporting a public sector digital transformation initiative.
The successful candidate will be responsible for designing, developing, producing and operating scalable, secure and reliable data pipelines and reusable data products using Amazon Web Services (AWS) and Databricks.
You will play a key technical leadership role across the data engineering lifecycle, covering data ingestion, transformation, modelling, orchestration, data quality, governance, monitoring and performance optimisation.
Working closely with data architects, analysts, data scientists, platform engineers and business stakeholders, you will translate complex data requirements into robust engineering solutions that support enterprise analytics, reporting and Machine Learning (ML) and Artificial Intelligence (AI) workloads.
This position is suited to a technically strong Data Engineer with extensive hands-on experience in Python, PySpark, SQL, AWS and Databricks, coupled with the ability to establish engineering standards, mentor team members and deliver production-ready data solutions.
Key Responsibilities
1. Data Engineering & Pipeline Architecture
- Lead the design, development and implementation of scalable batch and streaming data pipelines across AWS and Databricks environments.
- Build and maintain robust data ingestion, validation, transformation and publishing processes for structured, semi-structured and unstructured data.
- Integrate data from multiple sources, including databases, files, APIs and event streams.
- Implement reliable full-load, incremental loading and Change Data Capture (CDC) mechanisms, including data replay, recovery and backfill processes.
- Ensure data pipelines are maintainable, reusable, scalable and suitable for production operations.
2. Lakehouse Architecture & Data Product Development
- Design and implement modern Lakehouse architectures using Delta Lake and layered data processing patterns, including Bronze, Silver and Gold data layers.
- Develop reusable data models, curated datasets and domain-oriented data products to support enterprise analytics and downstream applications.
- Establish clear data ownership, definitions, quality expectations and service requirements for published data products.
- Apply appropriate dimensional and domain data modelling techniques to improve data accessibility, consistency and usability.
- Promote reusable engineering components and standardized data solutions rather than developing isolated, project-specific pipelines.
3. Data Quality, Governance & Management
- Establish automated data quality controls, schema enforcement, validation and reconciliation processes.
- Implement data quality checks covering completeness, accuracy, consistency, uniqueness, timeliness and referential integrity.
- Develop and maintain technical metadata, business definitions, data classification and end-to-end data lineage.
- Implement data governance and access management practices using platforms and tools such as Databricks Unity Catalog, AWS Glue Data Catalog and Lake Formation.
- Ensure data assets are appropriately documented, governed and accessible in accordance with established policies and security requirements.
4. Workflow Orchestration & CI/CD Automation
- Design and implement automated data workflows using Databricks Workflows, Lakeflow and relevant AWS services.
- Establish automated testing, deployment and environment promotion processes across development, testing and production environments.
- Integrate Git-based development practices, Continuous Integration and Continuous Deployment (CI/CD), and Infrastructure as Code (IaC).
- Develop parameterized and reusable workflows to improve deployment consistency, operational efficiency and maintainability.
- Ensure appropriate pipeline scheduling, dependency management, retry mechanisms and recovery procedures.
5. Monitoring, Reliability & Performance Optimization
- Establish monitoring, logging, alerting and operational dashboards to track data pipeline performance and reliability.
- Monitor pipeline health, data freshness, processing latency, throughput and data quality issues.
- Perform troubleshooting, incident investigation and root-cause analysis, implementing appropriate corrective and preventive measures.
- Optimize Apache Spark and SQL workloads, including partitioning, clustering, caching, compaction, file sizing and cluster configurations.
- Improve processing efficiency, workload scheduling and overall AWS and Databricks infrastructure cost effectiveness.
- Develop and maintain operational runbooks and support documentation to ensure production readiness and service continuity.
6. Cloud Security & Data Protection
- Implement cloud security practices covering least-privilege IAM, encryption, secrets management and secure network configurations.
- Establish appropriate environment separation, access controls, audit logging and sensitive data protection measures.
- Ensure data pipelines and platform components comply with relevant security, governance and agency requirements.
- Apply secure data-sharing and access management practices across data environments.
7. Technical Leadership & Stakeholder Collaboration
- Provide technical leadership across data engineering design, implementation and production operations.
- Establish engineering standards, conduct technical design and code reviews, and promote best practices.
- Mentor and guide less-experienced engineers to strengthen technical capabilities and engineering quality.
- Collaborate with data owners, data stewards, business analysts, data scientists, architects and governance teams to translate business requirements into technical specifications.
- Identify technical risks, dependencies and opportunities for platform improvements.
- Maintain architecture documentation, source-to-target mappings, data models, technical specifications and operational procedures.
- Facilitate knowledge transfer and build sustainable internal engineering capabilities.
Requirements & Qualifications
A. Education & Professional Experience
- Degree in Computer Science, Software Engineering, Data Engineering, Information Systems or a related technical discipline.
- Approximately 6–10 years of relevant experience in Data Engineering, Software Engineering or Cloud Data Platform development.
- Minimum 2 years of hands-on experience designing and operating cloud-based data platforms and production data pipelines.
- Demonstrated experience taking ownership of critical data engineering services, technical architecture or enterprise data solutions.
- Previous experience providing technical guidance, conducting code reviews, establishing engineering standards or mentoring other engineers.
B. Essential Technical Skills
Programming & Data Processing
- Strong proficiency in Python, PySpark and SQL.
- Hands-on experience with distributed data processing, complex data transformations, query optimization, debugging and automated testing.
Databricks & Lakehouse Technologies
- Strong hands-on experience with Databricks, Apache Spark and Delta Lake.
- Experience building and operating production-scale batch or streaming data pipelines.
- Understanding of Lakehouse architecture, layered data processing and reusable data modelling practices.
AWS Cloud Data Engineering
- Hands-on experience designing and implementing AWS-based data engineering solutions.
- Familiarity with relevant AWS services, including: Amazon S3 AWS Glue AWS Lake Formation AWS IAM and KMS AWS Lambda AWS Step Functions Amazon EventBridge Amazon CloudWatch Amazon Redshift Amazon MSK / Kinesis
Data Integration & Pipeline Management
- Experience implementing incremental data ingestion, Change Data Capture (CDC), idempotent processing, late-arriving data handling, data backfills and recovery mechanisms.
- Ability to design reliable data ingestion and transformation workflows across multiple source systems.
Data Quality & Governance
- Experience implementing automated data validation, schema enforcement, reconciliation, metadata management and data lineage.
- Understanding of data governance, access controls, data classification and data quality management.
DevOps & Workflow Automation
- Experience with workflow orchestration, Git-based development, CI/CD pipelines and Infrastructure as Code (IaC), preferably Terraform.
- Ability to implement automated testing and controlled deployment processes across multiple environments.
Production Operations & Security
- Experience implementing pipeline monitoring, observability, alerting, operational dashboards and incident management processes.
- Working knowledge of cloud security, least-privilege access, encryption, secrets management, network security and audit logging.
Technical Leadership
- Ability to review technical designs and code, establish development standards and provide guidance to other engineers.
- Strong understanding of production engineering practices and the ability to deliver maintainable and scalable data solutions.
C. Preferred Technical Skills & Additional Experience
Candidates with exposure to the following technologies and practices will have an added advantage:
- Databricks Unity Catalog, Lakeflow, Delta Live Tables, Auto Loader, Structured Streaming and Delta Sharing.
- Databricks Asset Bundles for data pipeline deployment and management.
- Apache Airflow, dbt, Kafka, Amazon MSK/Kinesis and AWS Database Migration Service (DMS).
- Data product development, data contracts, semantic or context layers, master data management and data mesh architectures.
- Feature engineering, ML training datasets, feature stores and MLOps pipelines using Databricks and AWS.
- Experience working in Singapore public sector technology environments and familiarity with IM8-aligned security controls, Vulnerability Assessment and Penetration Testing (VAPT), system assurance and governed cross-agency data sharing.
- Professional certifications such as: Databricks Certified Data Engineer Professional. AWS Certified Data Engineer – Associate. AWS Certified Solutions Architect – Associate or Professional.
D. Personal Attributes & Communication Skills
- Strong analytical and systems-thinking capabilities, with the ability to identify technical risks, dependencies and potential failure scenarios.
- Excellent written and verbal communication skills, including the ability to explain complex data architecture and engineering concepts to technical and non-technical stakeholders.
- Ability to translate business requirements into clear technical specifications, data definitions and implementation plans.
- Strong collaborative skills and the ability to work effectively across multidisciplinary technology, data and business teams.
- High degree of ownership, accountability and attention to detail.
- Proactive approach towards troubleshooting, continuous improvement, documentation and knowledge sharing.
- Ability to mentor team members and build sustainable in-house technical capabilities.
Application
Interested candidates are invited to submit their updated CV, highlighting relevant experience in AWS cloud data engineering, Databricks, Python, PySpark, SQL, data pipeline architecture and production data platform implementation. Thank you.
