The posting
About the Role
We are looking for a skilled Databricks Engineer todesign, develop, and maintain scalable data engineering solutions using the DatabricksLakehouse Platform.
The ideal candidate will have strong hands-on experiencewith Databricks, Apache Spark, Python, SQL, Delta Lake, and cloud dataplatforms, with the ability to build reliable and high-performance datapipelines.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.
- Develop ETL/ELT pipelines using PySpark, Python, and SQL.
- Build and maintain Delta Lake tables and data processing workflows.
- Work with Databricks Lakehouse architecture and related data engineering components.
- Develop batch and, where required, near-real-time data processing solutions.
- Ingest and transform data from databases, APIs, files, and other data sources.
- Implement data cleansing, transformation, validation, and quality checks.
- Optimise Spark jobs and Databricks workloads for performance and cost efficiency.
- Work with cloud storage and data services across Azure, AWS, or GCP.
- Implement data security, access controls, and governance within the data platform.
- Collaborate with Data Architects, Data Scientists, BI Developers, and business stakeholders.
- Troubleshoot data pipeline failures and resolve performance and data-quality issues.
- Develop and maintain technical documentation for data pipelines and solutions.
- Participate in code reviews, testing, deployment, and production support.
- Follow Agile development practices and contribute to continuous improvement.
Required Skills & Experience
- 3–5 years of experience in Data Engineering.
- Strong hands-on experience with Databricks.
- Strong knowledge of:
- Apache Spark / PySpark
- Python
- SQL
- Delta Lake
- ETL/ELT concepts
- Experience developing and managing data pipelines.
- Good understanding of data warehousing and data modelling concepts.
- Experience working with cloud platforms, preferably Microsoft Azure.
- Experience with cloud storage such as Azure Data Lake Storage (ADLS), Amazon S3, or Google Cloud Storage.
- Experience with relational and/or NoSQL databases.
- Good understanding of data quality, validation, and governance.
- Familiarity with Git and CI/CD practices.
- Strong troubleshooting and analytical skills.
Good to Have
- Databricks Certified Data Engineer Associate/Professional certification.
- Experience with Azure Data Factory.
- Experience with Microsoft Fabric.
- Knowledge of Unity Catalog and Databricks governance.
- Experience with Databricks Workflows and job orchestration.
- Experience with streaming technologies such as Kafka or Structured Streaming.
- Experience with Power BI or other BI platforms.
- Exposure to Machine Learning workflows on Databricks.
- Experience with Terraform or Infrastructure as Code.
- Knowledge of DevOps and CI/CD pipelines.
Candidate Profile
The ideal candidate should be a hands-on Databricks/Data Engineer capable of independently developing data pipelines, troubleshooting production issues, optimising Spark workloads, and collaborating with technical and business teams.



