Co-designing Raft + thread-per-core execution model for the Kafka-API

Sometimes you get to reinvent the wheel when the road changes. Redpanda is a drop-in replacement for Apache Kafka®, designed from the ground up for modern hardware. The hardware looks nothing like it did 10 years ago. NVMe disks are 1000X faster than spinning disks. Cloud computers offer 30X more cores than they did a decade ago with NICs offering 100x higher throughput.
In this talk, we will talk over the lessons learned building a new storage engine from scratch with no virtual memory, no page cache, with purpose-built read-ahead and write-behind strategies for predictable latencies, co-designed with Raft — data replication model.