Skip to content
CV Rocket
Get started
How it worksThe enginePricingBlogContactSign in

Open nowPosted 9 days ago

Data Engineer

47Billion5 open roles

Get the CV for this job

From $25 per CV, paid once. No subscription.

Your applicationOpen nowData Engineer47Billion

  1. YouYes, apply to this one.

  2. CV RocketCV written for this posting.

  3. 25 readersRecruiter, hiring manager, skeptic. Round after round.

  4. CV RocketApplied on 47Billion's own form.

The reply lands in your private mailbox

3×more interviews than doing it yourself with ChatGPT.

The clock on this job

Early applications get read.

11.5% of postings close within 7 days. Measured by our own scanner across the market.

Share of postings closed within
  1. 2.1%1 day
  2. 4.4%3 days
  3. 11.5%7 days
  4. 18.1%14 days
  5. 35.6%30 days
  6. 44.6%60 days
This job: posted 9 days ago

The posting

Job Description – Data Engineer (Databricks on AWS)

Experience: 1–5 Years

Technology Stack: Databricks, Apache Spark, PySpark, SQL, AWS, Delta Lake

Employment Type: Full-time

Job Summary

We are looking for a skilled Data Engineer with 1–5 years of experience in data engineering to design, develop, and maintain scalable data pipelines using Databricks on AWS. The ideal candidate should have hands-on experience with PySpark, SQL, data processing, and ETL/ELT development, along with a good understanding of cloud-based data engineering concepts.

The candidate will work closely with data architects, analysts, and other engineering teams to build reliable, high-performance data solutions that support business and analytical requirements.

Key Responsibilities

  • Design, develop, test, and maintain scalable ETL/ELT data pipelines using Databricks, PySpark, and SQL.
  • Develop data transformation workflows using Apache Spark and Delta Lake.
  • Work with structured and semi-structured data from multiple sources, including databases, APIs, and files such as CSV, JSON, and Parquet.
  • Build and manage Databricks Jobs and Workflows for automated data processing and scheduling.
  • Implement incremental data processing, data validation, error handling, and data quality checks.
  • Work with AWS services such as Amazon S3, IAM, and other relevant cloud services to build and support data pipelines.
  • Optimize Spark jobs and SQL queries for performance, scalability, and cost efficiency.
  • Implement appropriate data partitioning, file management, and Delta Lake optimization techniques.
  • Work with Unity Catalog and implement data access controls and governance practices.
  • Troubleshoot pipeline failures, investigate data discrepancies, and resolve production issues.
  • Follow coding standards, version control practices, and development, testing, and deployment processes.
  • Collaborate with cross-functional teams to understand requirements and deliver high-quality data solutions.
  • Maintain technical documentation for data pipelines, transformations, and operational procedures.

Required Technical Skills

1. Databricks and Apache Spark

  • Hands-on experience with Databricks on AWS.
  • Good understanding of Apache Spark architecture and distributed data processing.
  • Experience developing data pipelines using PySpark and Spark SQL.
  • Familiarity with Databricks Jobs, Workflows, notebooks, and job clusters.
  • Understanding of Delta Lake, including MERGE, schema evolution, and incremental data processing.

2. SQL and Data Engineering

  • Strong SQL skills, including joins, aggregations, CTEs, window functions, and complex transformations.
  • Experience with relational databases and data integration.
  • Understanding of ETL/ELT concepts, data modeling, and data warehouse fundamentals.
  • Ability to troubleshoot data quality issues and validate pipeline outputs.

3. AWS Cloud

  • Working knowledge of Amazon S3 for data storage.
  • Understanding of AWS IAM roles, policies, and permissions.
  • Familiarity with AWS cloud architecture and secure access to cloud resources.
  • Exposure to AWS Glue, Lambda, or other AWS data services is an advantage.

4. Delta Lake and Data Governance

  • Understanding of Delta tables, ACID transactions, and schema enforcement.
  • Familiarity with Unity Catalog, catalogs, schemas, tables, and access permissions.
  • Awareness of data lineage, governance, and secure data access practices.

5. Development and Deployment

  • Experience with Git and version control.
  • Familiarity with Python programming, debugging, and exception handling.
  • Exposure to CI/CD pipelines and automated deployments is an advantage.
  • Ability to write reusable, maintainable, and well-documented code.

Good to Have

  • Experience with Change Data Feed (CDF), incremental data loads, and Slowly Changing Dimensions (SCD).
  • Exposure to Databricks Auto Loader, Lakeflow, streaming data pipelines, and Databricks Genie.
  • Familiarity with Databricks AI/BI dashboards, Genie spaces, and natural-language analytics.
  • Experience with business intelligence and data visualization tools such as Power BI, Tableau, or similar analytical tools.
  • Exposure to Azure Databricks, Azure Synapse Analytics, or other Azure data services.
  • Familiarity with Google Cloud Platform (GCP) and its data engineering services, such as BigQuery, Dataflow, or Cloud Storage.
  • Familiarity with data integration tools such as Fivetran.
  • Experience with orchestration tools such as Apache Airflow or AWS Step Functions.
  • Familiarity with Terraform or other infrastructure-as-code tools.
  • Understanding of performance optimization techniques, including partitioning, caching, shuffle optimization, and Adaptive Query Execution (AQE).
  • Databricks Data Engineer Associate or Professional certification.

Qualifications

  • 1–5 years of relevant experience in data engineering, ETL development, or big data technologies.
  • Strong analytical and problem-solving skills.
  • Good communication skills and the ability to work effectively in a collaborative environment.
  • Willingness to learn new technologies and contribute to continuous improvement.

Expected Candidate Profile

The candidate should be able to independently develop and troubleshoot data pipelines appropriate to their experience level, write efficient PySpark and SQL code, understand distributed data processing fundamentals, and contribute to building scalable and reliable data engineering solutions on Databricks running on AWS.

From $25, paid onceGet the CV for this job

What happens when you press

One press. We do the rest.

  1. A CV for this posting

    Written against 47Billion's own wording, from every piece of relevant proof in your profile.

  2. 25 readers review it

    Recruiter, hiring manager, skeptic and more read every draft, round after round. You get the best round.

    The review screen in CV Rocket: how each CV was read, round by round.

  3. We apply on 47Billion's form

    Our application engine gets through the hardest forms there are. Where a question needs you, AI suggests the best answer. Don't want us applying from our IP addresses? Use our Chrome extension: we apply straight from your own browser.

    An application in CV Rocket: every answer filled in on the employer's form.

  4. Every reply, sorted

    47Billion's answer lands in your private mailbox, and we classify it on arrival: interview, question, rejection.

    The CV Rocket inbox: each employer reply classified as an interview, an action or a rejection.

  5. Reply with AI

    AI helps you write the email, checks it and sends it. We show you whether the recruiter read it.

  6. The interview in your calendar

    Full integration with your calendar. The invitation goes straight in.

    An interview invitation in the CV Rocket inbox, added to the candidate's calendar.
Get the CV for this job

From $25 per CV, paid once. No subscription.

Why it works

3×

more interviews than doing it yourself with ChatGPT.

ChatGPT writes a CV and never learns what happened to it. We see every reply. For each CV we know:

  • How it was written, and how the review scored it
  • When we applied, and how long after the posting went up
  • Which posting, which company, which city
  • Who got the interview, and who heard nothing

That is how we know which CVs get called.

Get the CV for this job

From $25 per CV, paid once. No subscription.

The numbers game

More applications. More interviews.

Every application goes out with its own CV, written for that posting and paid once. Send enough of them and the law of large numbers finds you the job.

Get the CV for this job
By hand5–10
With CV Rocket100
applications a day

Nearby

Live postings like this one

Same employer first, then the same role elsewhere.

Before you press

Straight answers

Get the CV for this job

From $25 per CV, paid once. No subscription.

What if my background isn't good enough?

We make the most of the background you have. The CV uses every piece of relevant proof your profile holds, and one of the 25 readers reads your whole profile and flags what the CV left out.

Do you really apply for me?

Yes, on the employer's own form, the hardest ones included. Where a question needs you, you answer it right there and AI suggests the best answer. Don't want us applying from our IP addresses? Use our Chrome extension: we apply straight from your own browser.

Is it a subscription?

No. You pay once per CV, from $25. Every application goes out with its own CV, written for that posting.

One job. One CV.
Paid once.

Pick the posting you want. We write for it, apply for you and catch the reply.

Get the CV for this job

From $25 per CV, paid once. No subscription.