Netdata: Open Source, Distributed Observability Pipeline - Journey and Challenges.

Netdata is a powerful open-source, distributed observability pipeline designed to provide higher fidelity, easier scalability, and a lower cost of ownership compared to traditional monitoring solutions. This presentation will offer an in-depth overview of the journey we've undertaken in building Netdata, highlighting the challenges we've faced and the innovative solutions we've developed to address them.

In this presentation, we will delve into the history of Netdata, starting from its inception. We'll discuss the initial goals of creating a monitoring tool that could offer high-resolution metrics, auto-detection of metrics, and real-time visualization, all with minimal configuration required. We'll also explore how Netdata garnered rapid attention and support from the open-source community.

The presentation will then shift focus to the evolution of Netdata, including the development of: a very efficient database engine (Netdata vs Prometheus: 35% less cpu utilization, 49% less memory, 12% less bandwidth, 98% less disk I/O, and 75% less size on disk for the same dataset), a powerful alerting engine that uses statistical analysis, machine learning and templates for fully automated alerts, a streaming protocol supporting streaming of live data, replication of metrics, and support for routing interactive queries between Netdata agents