The posting
Senior Data Engineer Work location- Paya Lebar
- Lead the design and operation of scalable batch and streaming data pipelines on AWS and Databricks using Python, PySpark, and SQL.
- Build Bronze, Silver, and Gold layers in Delta Lake and develop reusable data models and curated datasets for analytics, reporting, and ML.
- Integrate databases, files, APIs, and event streams using incremental loads and change data capture, with support for late-arriving data, backfills, and replay.
- Enforce data contracts, schema validation, and reconciliation, and implement end-to-end lineage, encryption, and least-privilege access.
- Orchestrate workflows with Databricks and AWS services, and automate testing and deployment across environments using Git, CI/CD, and Terraform.
- Monitor pipeline health and data freshness, lead incident troubleshooting, and optimise Spark and SQL workloads for performance and cloud cost.



