Senior Apache Spark Engineer

We're looking for a Senior Apache Spark Engineer to build and optimise the calculation engine behind CenataSure, a cloud-native outwards reinsurance platform used by insurers and reinsurers in the London market and internationally.

The engine recalculates historical claims and premiums against reinsurance contracts and programmes at full volume. Every large claim erodes aggregates and reinstatements across an entire programme, so the whole calculation is re-run rather than patched. Getting that right, quickly and at a sensible cost, is the job.

This is a hands-on engineering role on a real distributed workload, not a pipeline-plumbing one. You will spend your time on calculation correctness, Spark performance and cluster economics, working directly with the people who own the domain.

Responsibilities:

• Design, build, maintain and optimise distributed data processing jobs on Apache Spark
• Turn written reinsurance calculation rules into production Spark code
• Diagnose and fix performance problems — skew, shuffle, spill, partitioning, caching and join strategy
• Tune jobs for runtime and cluster cost on Azure Synapse Spark pools
• Write unit and property-based tests, and keep output deterministic and reproducible
• Work with analysts to turn specifications into testable behaviour
• Integrate jobs into CI/CD pipelines (Azure DevOps)
• Review code and help set engineering standards
• Share what you know with colleagues coming onto the engine

Requirements:

• 3+ years of commercial experience with Apache Spark in production
• Real understanding of how Spark executes: memory model, partitioning, shuffle, skew, join strategies, caching
• Able to open the Spark UI and work out why a job is slow or failing
• Strong SQL and comfort with large datasets
• A JVM language commercially (Java, Kotlin or Scala) — or strong PySpark plus working JVM knowledge
• Able to read Scala, and happy to work in it. The engine is Scala; we support people making that move and have done it successfully before
• Cloud data platform experience, Azure preferred (Synapse, Databricks, Data Lake Storage)
• Version control and CI/CD (Azure DevOps or equivalent)
• Strong analytical thinking and care about numbers being right
• Good written and verbal English
• Commercial Scala, Synapse or Databricks experience is a plus
• Experience with financial, actuarial or re/insurance calculations is useful but not essential

Interested candidates can send their CV to the e-mail address in the Apply for job button.