The posting
About the client:
The client is the leading Autonomous Decision Science company providing data-enabled prescriptive and anticipatory analytics and insights for companies across a broad range of industries. They automate a new way of problem-solving and decision-making, going beyond advanced analytics to understand data, perform analytics, generate insights, answer queries, and make decisions at the speed of the market. This powerful capability uniquely enables “End-to-End Value Chain Optimization and Decision-Making” at scale and allows clients to uncover and understand the inter-relationships that lead to innovative new product development and innovation, heightened consumer understanding and targeted marketing, revenue growth tactics, and intelligent demand and supply-chain planning. They help transform market-leading companies into true data-driven digital enterprises.
Timezone requirements: European working hours are okay; the client is in EST, and our team is in Europe.
English: Advanced level.
Requirements:
- Minimum of 5 years of hands-on experience as a Data Engineer or in a similar position;
- Minimum of 3 years of commercial experience with Python;
- Minimum of 3 years of SQL and experience working with relational databases (Postgres preferred; similar OLTP DB is acceptable);
- Experience with Python 3, Airflow, and Snowflake;
- Demonstrated knowledge and experience developing data pipelines to automate data processing workflows;
- Demonstrated experience in data modeling;
- Demonstrated knowledge of data warehousing, business intelligence, and application data integration solutions;
- Demonstratable experience in developing applications and services that run on Azure cloud infrastructure;
- Strong skills with pandas and Pyspark/Spark;
- Experience with container management and deployment, e.g., Docker and Kubernetes;
- Excellent problem-solving and communication skills;
- Ability to thrive in a fast-paced, remote environment;
- Comfortable with ambiguity with the ability to build structure and take a proactive approach to drive results;
- Attention to detail – quality and accuracy in deliverables.
The following additional skills would be beneficial:
- Knowledge of one or more of the following technologies: Data Science, Machine Learning, Natural Language Processing, Business Intelligence, and Data Visualization;
- Knowledge of statistics and experience using statistical or BI packages for analyzing large datasets (Excel, R, Python, Power BI, Tableau, etc.);
- Experience with at least one of the following: Databricks, Spark, Hadoop, or Kafka;
- FastAPI / Flask, or similar web backend framework experience;
- Experience with some of the following: Celery, RabbitMQ, and ZeroMQ;
- Other SQL or NoSQL experience: Redis, Elasticsearch, DuckDB, ClickHouse.
What you will do:
The successful candidate will join a diverse team to:
- Build unique, high-impact business solutions utilizing advanced technologies for use by world-class clients;
- Create and maintain the underlying data pipeline architecture for the solution offerings from raw client data to final solution output;
- Create, populate, and maintain data structures for machine learning and other analytics;
- Use quantitative and statistical methods to derive insights from the data;
- Combine machine learning, artificial intelligence (ontologies, inference engines, and rules), and natural language processing under a holistic vision to scale and transform businesses across multiple functions and processes.
Responsibilities Include:
- Create and maintain an optimal data pipeline architecture, incorporating data wrangling and Extract-Transform-Load (ETL) flows;
- Assemble large, complex data sets to meet analytical requirements – analytics tables, feature-engineering, etc.;
- Build the infrastructure required for optimal, automated extraction, transformation, and loading of data from a wide variety of data sources using SQL and other ‘big data’ technologies;
- Build automated analytics tools that utilize the data pipeline to derive actionable insights;
- Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.;
- Design and develop data integrations and data quality framework;
- Develop appropriate testing strategies and reports for the solution as well as data from external sources;
- Evaluate new technology for use within the projects;
- Work with the team and client personnel to administer and operate client-specific instances of the offerings;
- Configure the data pipelines to accommodate client-specific requirements to onboard new clients;
- Perform regular operations tasks to ingest new and changing data – implement automation where possible;
- Implement processes and tools to monitor data quality – investigate and remedy any data-related issues in daily solution operations.
Tech stack: a technical stack of the project:
Backend
- Programming Language: Python 3.11.2 (latest)
- Package Manager: Poetry
- Web Framework: FastAPI
- Object-Relational Mapping (ORM): SQLAlchemy
- Linting: ruff
- Code Formatter: isort
- Testing Framework: pytest
- Property-based Testing: Hypothesis
- Database: PostgreSQL
Frontend
- Programming Language: TypeScript
- Package Manager: PNPM
- Framework: React
- State Management: Zustand
- Routing: React Router
- Data Fetching: TanStack Query (successor to react-query)
- Build Tool: Vite
- Unit Testing: Vitest
- Component Testing: React Testing Library
- End-to-End Testing: Playwright
- UI Kit (to be determined)
- Linting: ESLint + typescript-eslint
- Code Formatting: Prettier
And:
- Docker and Docker Compose
- Kubernetes
- Airflow
- Azure
- Snowflake
Benefits from 8allocate:
- Team & Culture: Team events, offsites, and a culture that keeps people connected.
- Learning & Development: Budget for courses, certifications, and conferences.
- Wellbeing: Flexible support in line with company policy, with options to support your physical and mental wellbeing (sport, mental health, or medical insurance).
- Rest & Recovery: Paid vacation and sick leave.



