DevOps meets BigData – Kafka in the insurance environment

Data is everything – this is the dogma that virtually every business follows these days. Large insurance companies are no exception, and it is here that data-driven approaches are essential. With Telematik Plus, for example, HUK offers a car insurance rate that records the driving behavior of policyholders via smartphone. The data is then analyzed in the backend. Responsible driving positively influences the insurance premium in this way. Through a smartphone app, the driver always has full transparency over his driving behavior.

An important building block of this backend is Apache Kafka, which mainly serves as a central data pipeline. Kafka has also become a fixed and central component of the Datalake infrastructure at HUK.
In addition, HUK is currently using Kafka to complete the transformation from batch to event-based real-time streaming processing.

Like all large and complex infrastructure components, Kafka clusters are best installed and managed automatically. DevOps tools and principles such as Ansible and GitOps meet requirements from the BigData area at this point.

In this talk we will go into more detail about the architecture of the HUK Kafka backend and share our experiences of the last 1.5 years, from planning to the first PoC to the production go-live.

Dr. Bernhard Hopfenmüller has been part of ATIX for 4.5 years and currently works as consultant, trainer and teamlead. His projects involve and include Python, TheForeman, Ansible, Saltstack, containerization, Apache Kafka and DevOps topics of all kinds. He regularly appears as a speaker at open source and Linux events in Germany and Europe.

Julian Schreiner is administrator of the BigData DevOps group at HUK-Coburg. He is in charge of the BigData infrastructure at HUK, esp. Cloudera Hadoop und Kafka.
In particular, Julian Schreiner deals with the topics of Linux administration, automation with Ansible and Puppet, and container orchestration with Kubernetes.

This post is also available in: German