The posting
Working as a Data Engineer Consultant, you will:
- Design and implement data ingestion pipelines connecting external APIs and business systems with the Databricks data platform.
- Develop reliable processes for retrieving and processing historical data, including mechanisms for rate limiting, checkpointing, retries and safe reprocessing.
- Build and maintain the Bronze layer of the data platform, covering data standardisation, typing, historical tracking and document storage.
- Develop processes for linking digital documents with corresponding business records while ensuring data consistency and idempotent processing.
- Use Databricks capabilities to classify documents and extract structured information from PDFs and images, including confidence scoring and workflows for manual review.
- Contribute to the development of the Silver layer using appropriate entity modelling, CDC and incremental processing patterns.
- Implement deterministic entity resolution and source-priority logic across multiple data sources.
- Apply data quality controls within the existing framework and ensure that quality results are properly stored, monitored and available for governance purposes.
- Support the development of analytical and machine-learning data products, including feature and metric stores and batch scoring workflows.
- Work with MLflow and related processes for model registration and promotion where required.
- Improve automation, monitoring and reliability across data pipelines and contribute to CI/CD processes.
- Collaborate with other technical stakeholders to deliver agreed project milestones within a defined consulting scope.
- Recent, hands-on experience delivering production solutions with Databricks.
- Strong knowledge of PySpark, Delta Lake and Unity Catalog.
- Practical experience with the Databricks medallion architecture and production data pipelines.
- Experience with Spark Declarative Pipelines / Delta Live Tables and Databricks Workflows.
- Very good Python and advanced SQL skills.
- Experience integrating REST APIs into data engineering solutions, including authentication, pagination, rate limiting and incremental or checkpointed data retrieval.
- Strong understanding of data modelling, including SCD Type 2 and current-state / CDC patterns.
- Experience designing data quality controls, reconciliation processes and mechanisms for handling invalid or quarantined data.
- Knowledge of Git-based development and CI/CD for data platforms, preferably with Azure DevOps or GitHub Actions.
- Experience implementing monitoring and observability for production data pipelines.
- Ability to work within a fixed project scope, estimate work, meet milestones and take ownership of agreed deliverables.
- Strong communication skills and the ability to work independently in a consulting environment.
- Excellent English and Polish skills (min. B2 level).
Our client is a global technology services organisation delivering data and digital transformation projects for enterprise customers. The project focuses on building and enhancing a modern data platform based on Databricks, with a strong emphasis on data engineering, governance, automation and analytics.
The successful candidate will join a project involving multiple data sources, API integrations, document processing and layered data architecture. The environment combines traditional data engineering with modern Databricks AI capabilities and machine-learning workflows, providing an opportunity to work across several stages of the data lifecycle.



