Announcing Redpanda Connect and the acquisition of benthos.dev

ByAlexander GallegoonMay 30, 2024
Redpanda Connect

Building an experience that wows users instantly is directly proportional to the effort placed by the builders at every turn, every commit. Not to mention hundreds of hours wasted using bad products to form strong opinions.

The time it takes to create your first, globally-distributed cluster from sign up in Serverless is ~100 milliseconds with Redpanda. 90 seconds to deploy a production cluster of Kubernetes on EKS. And of course, less than 1 second to bootstrap a 3-node cluster on your laptop. Sheer terror for anyone else.

Today, you can connect anything to anywhere in 140 milliseconds with Redpanda Connect.

time `rpk connect run` rpk connect run 0.14s user 0.06s system 17% cpu 1.148 total

Any form factor. Any scale. Our largest Redpanda Connect user is pushing 50Gbps via 2X 25Gbps NICs to consume market data. Our smallest Redpanda Connect user is using 100 millicores and 10 MiB of memory, processing hundreds of messages a second deployed on a heavily resource-constrained environment.

The largest – 225 to be exact – connectivity platform for the Kafka ecosystem. There simply does not exist a more complete, end-to-end platform for streaming your data that spans every single form factor from embedded IoT devices to the largest cloud workloads.

Redpanda Connect comes via the acquisition of the popular benthos.dev stream processing framework. Ash, Founder and CEO, has been tirelessly working on Benthos for the past seven years, evolving the platform with some of the world’s largest companies spanning finance, oil & gas, healthcare, SasS, Adtech, and social media, etc.

We’ve always grown Redpanda with three core cultural values: Customer obsession, Be a good human, and Win as a team. While the traction of the product speaks for itself with almost 8,000 GitHub stars and users in every imaginable industry, the company bet was on Ash – the human and brains behind benthos.dev – and team. Thank you: we couldn’t have found a better partner, and welcome to the Redpanda family!

For Redpanda, this represents a meaningful transition to the most complete, end-to-end streaming platform. Whether you’re squeezing microsecond-level latency for the most demanding workloads, or you’re building one of the world's largest social networks like ShareChat with 390 million monthly active users, or you’re building the most reliable banking products like Zafin IO for the world's largest commercial banks, you can now build and scale your business without the need to stitch multiple projects together. All under one roof. I couldn’t be more excited about the future of Redpanda.

For users, this means continued support of 223 Apache 2 Licensed, open source connectors with the large financial backing of Redpanda and customer footprint. It means increased focus on reliability, day 2 operations and security. We’ll be bringing the known and loved Benthos Studio into Redpanda Console (UI) for a visual representation of the connectivity in your business.

Lastly, we know firsthand how difficult it is to build a framework that supports intelligent retries, routing, supervision, and built-in telemetry. We know many of you bet your business on the core engine of [Redpanda] Benthos, and we couldn’t be more proud to continue to support you by keeping the core engine MIT with all the freedom and protections it provides. We can’t wait to continue building together. Please see the core engine repo here.

Redpanda Connect: open source connectors

We couldn’t be happier keeping 99%+ of the connectors with an Apache 2, Open Source license. For the connectors below, you can purchase support from the Redpanda team for “Certified” connectors. These connectors have very large production users. For the databases, below we note that they tend to be in the regular test matrix. “Community” connectors are just that – Slack and GitHub support on a best-effort basis.

Every connector is in production today, but we want to ensure that your experience is curated and that Redpanda can stand behind the stability and scalability of the Certified group first.

ConnectorTypeSupportLicense
amqp_0_9InputCertifiedApache 2
amqp_0_9OutputCertifiedApache 2
archiveProcessorCertifiedApache 2
aws_dynamodb_partiqlProcessorCertifiedApache 2
aws_kinesisInputCertifiedApache 2
aws_kinesisOutputCertifiedApache 2
aws_kinesis_firehoseInputCertifiedApache 2
aws_lambdaProcessorCertifiedApache 2
aws_s3InputCertifiedApache 2
aws_s3OutputCertifiedApache 2
aws_sqsInputCertifiedApache 2
aws_sqsOutputCertifiedApache 2
bloblangProcessorCertifiedApache 2
bounds_checkProcessorCertifiedApache 2
cache aws_dynamodbProcessorCertifiedApache 2
cache aws_dynamodbOutputCertifiedApache 2
cache aws_s3ProcessorCertifiedApache 2
cache aws_s3OutputCertifiedApache 2
cache fileProcessorCertifiedApache 2
cache fileOutputCertifiedApache 2
cache memcachedProcessorCertifiedApache 2
cache memcachedOutputCertifiedApache 2
cache memoryProcessorCertifiedApache 2
cache memoryOutputCertifiedApache 2
cache nats_kvProcessorCertifiedApache 2
cache nats_kvOutputCertifiedApache 2
cache redisProcessorCertifiedApache 2
cache redisOutputCertifiedApache 2
cache ristrettoProcessorCertifiedApache 2
cache ristrettoOutputCertifiedApache 2
cachedProcessorCertifiedApache 2
compressProcessorCertifiedApache 2
csvInputCertifiedApache 2
decompressProcessorCertifiedApache 2
dedupeProcessorCertifiedApache 2
generateInputCertifiedApache 2
group_byProcessorCertifiedApache 2
group_by_valueProcessorCertifiedApache 2
httpProcessorCertifiedApache 2
http_clientInputCertifiedApache 2
http_clientOutputCertifiedApache 2
jmespathProcessorCertifiedApache 2
jqProcessorCertifiedApache 2
json_schemaProcessorCertifiedApache 2
kafkaInputCertifiedApache 2
kafkaOutputCertifiedApache 2
kafka_franzInputCertifiedApache 2
kafka_franzOutputCertifiedApache 2
logProcessorCertifiedApache 2
mappingProcessorCertifiedApache 2
metricProcessorCertifiedApache 2
mutationProcessorCertifiedApache 2
natsInputCertifiedApache 2
natsOutputCertifiedApache 2
nats_jetstreamInputCertifiedApache 2
nats_jetstreamOutputCertifiedApache 2
nats_kvInputCertifiedApache 2
nats_kvOutputCertifiedApache 2
nats_kvProcessorCertifiedApache 2
nats_request_replyProcessorCertifiedApache 2
opensearchOutputCertifiedApache 2
parquet_decodeProcessorCertifiedApache 2
parquet_encodeProcessorCertifiedApache 2
protobufProcessorCertifiedApache 2
rate_limitProcessorCertifiedApache 2
redisProcessorCertifiedApache 2
redis_hashOutputCertifiedApache 2
redis_listInputCertifiedApache 2
redis_listOutputCertifiedApache 2
redis_pubsubInputCertifiedApache 2
redis_pubsubOutputCertifiedApache 2
redis_scriptProcessorCertifiedApache 2
redis_streamsInputCertifiedApache 2
redis_streamsOutputCertifiedApache 2
schema_registry_decodeProcessorCertifiedApache 2
schema_registry_encodeProcessorCertifiedApache 2
select_partsProcessorCertifiedApache 2
sftpOutputCertifiedApache 2
sleepProcessorCertifiedApache 2
socketInputCertifiedApache 2
socketOutputCertifiedApache 2
socket_serverInputCertifiedApache 2
socket_serverOutputCertifiedApache 2
sql_insert mysqlOutputCertifiedApache 2
sql_insert mysqlProcessorCertifiedApache 2
sql_insert oracleOutputCertifiedApache 2
sql_insert oracleProcessorCertifiedApache 2
sql_insert postgresOutputCertifiedApache 2
sql_insert postgresProcessorCertifiedApache 2
sql_insert sqliteOutputCertifiedApache 2
sql_insert sqliteProcessorCertifiedApache 2
sql_raw mysqlInputCertifiedApache 2
sql_raw mysqlOutputCertifiedApache 2
sql_raw mysqlProcessorCertifiedApache 2
sql_raw oracleInputCertifiedApache 2
sql_raw oracleOutputCertifiedApache 2
sql_raw oracleProcessorCertifiedApache 2
sql_raw postgresInputCertifiedApache 2
sql_raw postgresOutputCertifiedApache 2
sql_raw postgresProcessorCertifiedApache 2
sql_raw sqliteInputCertifiedApache 2
sql_raw sqliteOutputCertifiedApache 2
sql_raw sqliteProcessorCertifiedApache 2
sql_select mysqlInputCertifiedApache 2
sql_select mysqlProcessorCertifiedApache 2
sql_select oracleInputCertifiedApache 2
sql_select oracleProcessorCertifiedApache 2
sql_select postgresInputCertifiedApache 2
sql_select postgresProcessorCertifiedApache 2
sql_select sqliteInputCertifiedApache 2
sql_select sqliteProcessorCertifiedApache 2
unarchiveProcessorCertifiedApache 2
websocketInputCertifiedApache 2
websocketOutputCertifiedApache 2
workflowProcessorCertifiedApache 2
amqp_1InputCommunityApache 2
amqp_1OutputCommunityApache 2
avroProcessorCommunityApache 2
awkProcessorCommunityApache 2
azure_blob_storageInputCommunityApache 2
azure_blob_storageOutputCommunityApache 2
azure_cosmosdbProcessorCommunityApache 2
azure_cosmosdbInputCommunityApache 2
azure_cosmosdbOutputCommunityApache 2
azure_queue_storageInputCommunityApache 2
azure_queue_storageOutputCommunityApache 2
azure_table_storageInputCommunityApache 2
azure_table_storageOutputCommunityApache 2
beanstalkdInputCommunityApache 2
beanstalkdOutputCommunityApache 2
cacheProcessorCommunityApache 2
cache couchbaseProcessorCommunityApache 2
cache couchbaseOutputCommunityApache 2
cache gcp_cloud_storageProcessorCommunityApache 2
cache gcp_cloud_storageOutputCommunityApache 2
cache lruProcessorCommunityApache 2
cache lruOutputCommunityApache 2
cache mongodbProcessorCommunityApache 2
cache mongodbOutputCommunityApache 2
cache ttlruProcessorCommunityApache 2
cache ttlruOutputCommunityApache 2
cassandraInputCommunityApache 2
cassandraOutputCommunityApache 2
cockroachdb_changefeedInputCommunityApache 2
commandProcessorCommunityApache 2
couchbaseProcessorCommunityApache 2
discordInputCommunityApache 2
discordOutputCommunityApache 2
elasticsearchOutputCommunityApache 2
fileInputCommunityApache 2
fileOutputCommunityApache 2
gcp_bigqueryInputCommunityApache 2
gcp_bigqueryOutputCommunityApache 2
gcp_bigquery_selectProcessorCommunityApache 2
gcp_cloud_storageInputCommunityApache 2
gcp_cloud_storageOutputCommunityApache 2
gcp_pubsubInputCommunityApache 2
gcp_pubsubOutputCommunityApache 2
grokProcessorCommunityApache 2
hdfsInputCommunityApache 2
hdfsOutputCommunityApache 2
http_serverInputCommunityApache 2
http_serverOutputCommunityApache 2
javascriptProcessorCommunityApache 2
mongodbInputCommunityApache 2
mongodbOutputCommunityApache 2
mongodbProcessorCommunityApache 2
mqttInputCommunityApache 2
mqttOutputCommunityApache 2
msgpackProcessorCommunityApache 2
nanomsgInputCommunityApache 2
nanomsgOutputCommunityApache 2
nats_streamInputCommunityApache 2
nats_streamOutputCommunityApache 2
nsqInputCommunityApache 2
nsqOutputCommunityApache 2
parquetInputCommunityApache 2
parse_logProcessorCommunityApache 2
pulsarInputCommunityApache 2
pulsarOutputCommunityApache 2
redis_scanInputCommunityApache 2
sentry_captureProcessorCommunityApache 2
sftpInputCommunityApache 2
sql_insert clickhouseOutputCommunityApache 2
sql_insert clickhouseProcessorCommunityApache 2
sql_insert cosmodbProcessorCommunityApache 2
sql_insert cosmosdbInputCommunityApache 2
sql_insert mssqlOutputCommunityApache 2
sql_insert mssqlProcessorCommunityApache 2
sql_insert snowflakeOutputCommunityApache 2
sql_insert snowflakeProcessorCommunityApache 2
sql_insert trinoOutputCommunityApache 2
sql_insert trinoProcessorCommunityApache 2
sql_raw clickhouseInputCommunityApache 2
sql_raw clickhouseOutputCommunityApache 2
sql_raw clickhouseProcessorCommunityApache 2
sql_raw cosmodbProcessorCommunityApache 2
sql_raw cosmosdbInputCommunityApache 2
sql_raw cosmosdbOutputCommunityApache 2
sql_raw mssqlInputCommunityApache 2
sql_raw mssqlOutputCommunityApache 2
sql_raw mssqlProcessorCommunityApache 2
sql_raw snowflakeInputCommunityApache 2
sql_raw snowflakeOutputCommunityApache 2
sql_raw snowflakeProcessorCommunityApache 2
sql_raw trinoInputCommunityApache 2
sql_raw trinoOutputCommunityApache 2
sql_raw trinoProcessorCommunityApache 2
sql_select clickhouseInputCommunityApache 2
sql_select clickhouseProcessorCommunityApache 2
sql_select cosmodbProcessorCommunityApache 2
sql_select mssqlInputCommunityApache 2
sql_select mssqlProcessorCommunityApache 2
sql_select snowflakeInputCommunityApache 2
sql_select snowflakeProcessorCommunityApache 2
sql_select trinoInputCommunityApache 2
sql_select trinoProcessorCommunityApache 2
subprocessInputCommunityApache 2
subprocessOutputCommunityApache 2
subprocessProcessorCommunityApache 2
twitter_searchInputCommunityApache 2
wasmProcessorCommunityApache 2
xmlProcessorCommunityApache 2

See our Connector Catalog for the latest list.

Redpanda Connect: Enterprise connectors

Two connectors require a paid subscription to Redpanda: snowflake_put and splunk. All future improvements, security vulnerability patches, and builds require a subscription to Redpanda. While we worked hard to keep most of the connectors permissively licensed, which may include competitors, I want to be transparent about our monetization model so there is no ambiguity or uncertainty.

ConnectorTypeSupportLicense
snowflake_putOutputCertifiedEnterprise [RCL]
splunk [hec]OutputCertifiedEnterprise [RCL]

Redpanda Benthos core engine

For developers and engineers who built their entire business on [Redpanda] Benthos, you can continue to do that, as we’re keeping the core engine MIT Licensed. All those hard-to-build, hard-to-test features that made [Redpanda] Benthos so reliable over the years are free and open source for you to use. This is a win for the community that built their business on top of [Redpanda] Benthos – we are so sure of it, we bet the future of Redpanda connectivity on this engine itself. I can’t wait to see how we evolve it together.

More to come, but don’t hesitate to continue collaborating with us on github.com/redpanda-data/benthos

Get in touch

It would be my and the team’s privilege to have the opportunity to partner with you. Whether you are using Redpanda Connect, or the Redpanda Console for the best UI in the streaming world or our storage engine Redpanda. Please reach out!

.alex founder & ceo

Let's keep in touch

Subscribe and never miss another blog post, announcement, or community event. We hate spam and will never sell your contact information.