The posting
OPSWAT, a global leader in IT, OT, and ICS critical infrastructure cybersecurity, delivers an end-to-end platform that gives public and private sector organizations and enterprises the critical advantage needed to protect their complex networks, secure their devices, and ensure compliance. Over the last 20 years our commitment to innovative technology has earned the trust of more than 1,700 organizations, governments, and institutions globally, solidifying our role in protecting the world’s critical infrastructure and securing our way of life.
The Position
OPSWAT's data platform ingests from more than forty business systems such as CRM, finance, HR, marketing, support, product telemetry, lands the raw output in a Delta lakehouse, refines it through bronze and silver, and publishes marts that downstream consuming applications read. Apache Airflow on AKS orchestrates it, dbt Core models it, and the warehouse is moving from Microsoft Fabric Warehouse to Azure Database for PostgreSQL while all of that stays running.
You own that path end to end, not a link of it: a source nobody has ingested before, carried all the way to a table a business user can open in a dashboard, and everything in between that decides whether the number is right and arrives on time. The extraction that survives pagination, throttling and a schema that changed without telling anyone; the schedule that fires when the data lands rather than when the clock says; the silver table that refuses bad rows instead of passing them downstream; the mart whose columns are documented and whose keys are tested; the semantic-layer definition that makes it usable; and the freshness and the cost of the whole arrangement.
Senior Data Engineering owns the modelling standards and the parity proof that a dbt mart matches the legacy procedure it replaces. You own the path those models sit on; the pipeline that feeds them and the serving layer that consumes them and you build marts to that standard rather than to your own. The seam is real and it is worked together: you bring the source contract, the schedule and the serving behaviour; they bring the layer rules and the value comparison.
You will work in an AI-assisted engineering workflow, where agentic tooling carries the mechanical parts and your judgement stays on the design: what a source's contract is, where a pipeline should fail loudly, and what a dashboard can afford to query.
What you will be doing
- Onboard sources end to end: authentication, pagination, rate limits, incremental watermarks, landing, bronze, silver with quality gates, warehouse sync as one DAG family per data source, not a bespoke script per table.
- Own orchestration: DAGs triggered on data arrival, idempotent re-runs, backfills that do not double-count, pod sizing that fits the work, and alerts that page a human only when a human is needed.
- Build the last mile: the marts and the semantic-layer definitions on top of them, so a metric in Zenlytic or a chart in Superset resolves to a column that means what its name says.
- Keep the lakehouse healthy: compaction, Z-ordering chosen from real query patterns, vacuum only inside retention, and schema evolution that does not break a consumer.
- Make performance a measured number: before and after on the pipeline you tuned, capacity spend on the warehouse, caching or pre-aggregation where a dashboard is paying for the same scan every morning.
- Set and defend freshness and completeness expectations per source; when one breaks, diagnose to root cause and write it down.
- Run the platform underneath: AKS workloads, Helm or Kustomize, Terraform, image builds and CI/CD, Key Vault and managed identity, least-privilege access to every store you touch.
- Review pull requests on pipelines and models, and keep runbooks and standards current enough that the next engineer does not need to ask you.
- Work with the Analytics Engineers and Product Analysts who own the business definitions your marts serve, so a definition change and a pipeline change land in the right order.
What we need from you
- 5+ years in data engineering, with production ownership of pipelines other people depended on.
- Expert Python and SQL: you can read a query plan and say why it is slow.
- A dataframe engine at volume: Polars, duckDB, Spark and the judgement to know when not to pull the data into memory at all.
- Airflow in production (or Dagster/Prefect and the willingness to move); you have debugged a scheduler, not only written DAGs.
- Containers and Kubernetes: an image, a pod spec, a resource limit, and what happens when the limit is wrong.
- Lakehouse table formats: Delta or Iceberg with partitioning, compaction and schema evolution treated as operational concerns, not theory.
- Dimensional modelling judgement and hands-on dbt: you can state a grain, choose a key, and place logic in the right layer.
- One cloud in depth: Azure, AWS or GCP including storage, identity, secrets, and the networking that makes a private endpoint actually work.
- Infrastructure as code: Terraform, and Helm or Kustomize. You have shipped a change through a pipeline rather than by hand.
- Observability habits: metrics, logs, alerts, and an on-call triage you can describe step by step.
- API integration craft, pagination, throttling, retry with backoff, incremental state and the difference between a source that is down and a source that changed shape.
- Working fluency with a BI or semantic layer, enough to know why a model that is correct in SQL can still be unusable in a dashboard.
- Clear writing. The incident report and the runbook are part of the job, not paperwork around it.
It would be great if you also have
- Azure Database for PostgreSQL Flexible Server including Citus / Elastic Cluster in your background.
- dbt at scale: Elementary, exposures, column-level lineage, CI-enforced builds.
- Cost engineering: capacity units, warehouse sizing, and a saving you can attribute to a change you made.
- A platform migration you helped carry warehouse engine, orchestrator, or both.
- Superset, Zenlytic, Power BI, or another tool where you owned the semantic model rather than only the chart.
- CDC or streaming: Fabric mirroring, Debezium, event-driven ingestion.
- Standards or tooling you authored that other engineers actually follow.
OPSWAT is an equal opportunity employer. We celebrate diversity and are committed to providing an environment where equal employment opportunities are extended to all employees and applicants, free of discrimination and harassment of any type. All employment decisions are based on individual qualifications, job requirements, and business needs without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other category protected by federal, state, or local laws.
Recruiting Agencies: we do not accept unsolicited resumes from third party agencies for any of our open positions. To submit resumes for our jobs, there must be a recruiting contract approved by our legal team and endorsed by both parties. We are currently not accepting additional 3rd party agencies at this time.
