Introduction

We are building a real-time streaming engine for modern applications - from the enterprise to the solo dev prototyping a react application on her laptop. We go beyond the Kafka protocol, into the future of streaming with inline Wasm transforms and geo-replicated hierarchical storage. A new platform that scales with you from the smallest projects to petabytes of data distributed across the globe.

Background

As easy to run as nginx. No dependencies. Ability to flush to disk with acks=-1. Leverage a huge and active ecosystem. It must be fast, really fast. With this wishlist in mind, I wrote the first line of code of what eventually became redpanda. It was January 7th, 2019 and I was still living in Miami before relocating to San Francisco. I hadn’t had as much fun hacking on anything since the initial prototype of my previous project and company concord.io and... it was equally all-consuming.

Here we are today, 22 months later. A team one dreams to be part of and a product we feel proud to share with you. Ready to be put through the paces in even more ways that we could have anticipated, whether embedding Redpanda in a security appliance, or using it as part of your new NodeJS application because it's so simple to use. Whoever you are, welcome! We are excited to have you in our community.

Legal

The project is released under the Source Available License - BSL - similar to what our friends at CockroachDB have done. We try to make this clear in the license, but worth reiterating here. Our intention is to deter cloud providers from offering our work as a service. For 99.999% of you, restrictions will not apply - welcome to our community!

There will be enterprise, pay-only features that will be obvious, since to turn them on you have to edit the enterprise section of the configuration.

Getting started

The simplest thing you can do is run in Docker. Follow the tutorial here. But for the truly impatient, here is the executive summary:

$ rpk container start -n 3
Starting cluster
Waiting for the cluster to be ready...
  NODE ID  ADDRESS          
  0        127.0.0.1:65279  
  1        127.0.0.1:65287  
  2        127.0.0.1:65286  

Cluster started! You may use rpk to interact with it. E.g:

rpk cluster info

It says we can check our cluster with rpk cluster info. Let's try that!

When you run rpk commands, you need to include the list of brokers to run the command against. In our example that would be:

$ rpk cluster info --brokers 127.0.0.1:65279,127.0.0.1:65287,127.0.0.1:65286
BROKERS
=======
ID    HOST       PORT
0*    127.0.0.1  65279
1     127.0.0.1  65287
2     127.0.0.1  65286

For example, you can run rpk topic create and it will work!

$ rpk topic create -p 6 -r 3 new-topic --brokers 127.0.0.1:65279,127.0.0.1:65287,127.0.0.1:65286
Created topic 'new-topic'. Partitions: 6, replicas: 3, cleanup policy: 'delete'

Thank you

This decision comes after almost a year of thinking and mentorship with a very large group of experts, OSS enthusiasts, lawyers and quiet time thinking. Special thanks to:

• Peter Mattis from CockroachDB for sharing his experience with BSL , which ultimately made us feel comfortable with our decision to also choose BSL.
• Adam Jacob for sharing his experiences with Chef, his never-ending expertise around licensing and for taking the time to walk me through business models with me.
• Thanks to Ajay Kulkarni from TimescaleDB for sharing his wealth of knowledge and experience building a community.
• Thanks to Megan Gill at MongoDB for helping me better understand OSS in general.
• Gaurav Gupta, now at LSVP, for helping me understand Elastic a bit better and how the OSS+Source Available has matured in the last decade.

We are better because of your advice, I am forever thankful.