Welcome to part two of our series on Redpanda for gaming, where we cover popular data streaming use cases in the gaming industry. The first post gave a general overview of what you need to power up your real-time gaming architecture. Now it’s time to dig a little deeper into the first use case, then roll up our sleeves and get into the code. To kick it off, we’re starting with the ever-important hallmark of any gaming experience: leaderboards.

A gaming leaderboard allows participants to visualize their achievements and see how they rank compared to their peers. Its main purpose is to encourage competition, which leads to more gameplay. But for a leaderboard to be useful, it has to accurately reflect player performance in real time. As online gaming continues to grow in popularity and the number of concurrent users rises exponentially, leaderboards require a robust architecture that can handle the massive amount of data streaming in and out of the gaming system, as well as scale to accommodate its rapidly-growing user base.

In this post, we’ll explore the challenges of building a real-time leaderboard, and walk through a practical example where we showcase how you can use Redpanda with Materialize to develop, deploy, and scale a real-time gaming leaderboard faster and at a much lower cost.

Why is building a real-time leaderboard challenging?

Instant feedback on player performance from real-time leaderboards drives the competitive side of online gaming. Successful online game series like Counter-Strike, Call of Duty, and Apex Legends have millions of active users. The online player count can often peak around the one million mark on a single online gaming platform, like Steam.

Players create terabytes of daily streamed data and have no patience for more than single-digit millisecond latency. Unfortunately, most streaming data platforms today haven’t been able to keep pace with the sky-high demand or increasing expectations from modern gamers.

Apache Kafka® has been synonymous with real-time data for years. It has enabled revenue growth, customer engagement, improved security, and much more. But it’s highly complex to handle gigabytes of daily streamed real-time data.

Gaming companies see operational costs grow rapidly when running Kafka at scale. This is because Kafka was designed to overcome the storage bottleneck of the 2010s by exploiting cheap spinning disks. Today disks are solid state, 1000x faster, and 100x cheaper. Computing power is the new bottleneck—and Kafka simply wasn’t designed to optimize it.

This is why more and more gaming companies are looking to Redpanda for their real-time gaming data use cases. If you’re new here, Redpanda is a streaming data platform that’s 100% API-compatible with Kafka and doesn’t need a JVM or Apache Zookeeper™ to run. In a nutshell: it’s faster, easier, simpler, and up to 6x more cost-effective than Kafka.

How to build an unbreakable real-time gaming leaderboard

Now that you understand the importance of a robust gaming architecture behind your online player leaderboards, it’s time to get technical. We’re going to walk you through a practical example where we’ll build a real-time leaderboard using Redpanda. It’s a good starting point for any successful real-time leaderboard, which should include the following features:

• Fast data ingestion – Player scores should be ingested from the gaming frontend into the system as fast as possible.

• Reliable and scalable data storage – There will be spikes in demand, and the data storage needs to be able to deal with them in a reliable way no matter their size and velocity.

• Support for fast queries – Scores should be calculated in real-time with online analytical processing (OLAP) of the ingested data.

For our example, the architecture is made up of the gaming frontend, Redpanda as the streaming data platform, and Materialize as the real-time OLAP database engine. Here’s a quick diagram to show you how it’ll all work together.

You can build the same real-time leaderboard solution using the source code shared in this Redpanda GitHub repo. Here’s what you need to get started:

• Docker and Docker Compose installed

• At least 6 CPU cores and 8 GB of RAM allocated to Docker daemon

• JDK version 8 or higher installed

• JAVA_HOME environment enabled

• Python 3 installed

• Apache Maven build tool installed

• Quarkus CLI installed (optional)

1. Ingest events from gaming frontends for real-time leaderboards

The first task is to collect game completion events from the gaming frontend and transmit them to the storage layer as fast as possible. Gaming SDKs can send the event (GAME_COMPLETED) at game completion from several gaming consoles, mobile apps, web apps, and gaming PCs.

The game completed event should include player ID and the score. We’ll use a Python script to simulate a random stream of these events formatted in JSON.

{
    “event_id”: “4234afaf23r23424”,
    “player_id”: 42348234,
    “score”: 23,
    “ts”: “2022-08-17 05:23:231”
}

We’ll then use Redpanda to ingest the GAME_COMPLETED events from the gaming frontend into the gaming.scores topic. Since Redpanda scales with peaks in traffic, it acts as a reliable storage of our events until the downstream analytics database is ready to consume them.

2. Serve leaderboard queries with Materialize

There’s no time for lag, so we’ll use the database Materialize to avoid any latency in our score calculation. For the unfamiliar, Materialize is an SQL streaming database specifically designed for real-time analytics in interactive dashboards and customer-facing experiences like leaderboards.

Game events are sent to Materialize for fresh and accurate game scores. Materialize then uses online analytical processing to ingest events from Redpanda in real time. It builds a materialized view to calculate top competitors and their ranks. This view is updated as every new GAME_COMPLETED event arrives. That’s how you’ll have fresh analytics ready for your real-time leaderboard.

We’ll front Materialize with a microservice that serves the data back to the gaming frontends. This layer adds valuable features like role-based access control, observability, and rate-limiting to requests from gaming frontends.

3. Run the real-time leaderboard with Redpanda

Use Docker Compose to start Redpanda and Materialize with this command:

 docker compose up -d

Now, you ask the simulator to publish gaming events to the scores topic in Redpanda. Make sure you have Python 3 installed on your computer before entering this command:

  python3 ./simulator/simulator.py

You can use Redpanda’s CLI tool, rpk, to observe the content inside the gaming.scores topic. Alternatively, you can use a generic command line interface like kcat using the following command.

kcat -b localhost:9092 -t gaming.scores
% Auto-selecting Consumer mode (use -P or -C to override)
{“id”: “c8e1bdf1-98d6-4c3b-8b43-d47991f93075”, “player_id”: 6271,
“score”: 545, “ts”: 1660719218}
{“id”: “7df515af-287c-44dd-bdf0-3dd7d16d4d47”, “player_id”: 8619,
“score”: 200, “ts”: 1660719219}
{“id”: “ae0e873c-b42e-4ec6-a2b6-a74b523f0e3d”, “player_id”: 1162,
“score”: 207, “ts”: 1660719220}
...

4. Define sources and materialized views in Materialize

Now your events are flowing into Redpanda. The next step is to define materialized views to answer the leaderboard query. You can use PostgreSQL interactive terminal to connect to the running Materialize instance and define required materialized views.

psql -U materialize -h localhost -p 6875 materialize

Create a JSON-formatted source to represent the events coming from the gaming-stores topic.

CREATE SOURCE scores_source
  FROM KAFKA BROKER ‘redpanda:29092’ TOPIC ‘gaming.scores’
  FORMAT BYTES;

We don’t know the schema in JSON-formatted messages. So the JSON is pulled in as raw bytes and needs to be CAST into proper columns and types. Use the following view:

CREATE MATERIALIZED VIEW scores AS
  SELECT
    (data->>’id’) AS id,
    (data->>’player_id’)::int AS player_id,
    (data->>’score’)::int AS score,
    data->>’ts’ AS ts
  FROM (SELECT CONVERT_FROM(data, ‘utf8’)::jsonb AS data FROM scores_
source);

This view isn’t materialized and doesn’t store query results, but it gives you an alias for the embedded SELECT statement and lets you shape the data in the format you need. Finally, use the following command to define the leaderboard materialized view:

CREATE MATERIALIZED VIEW leaderboard AS
    SELECT
player_id,
        sum(score) as total
    FROM scores
    GROUP BY player_id
    ORDER BY sum(score) DESC;

5. Read the real-time leaderboard data

Now you have a view of the total scores for each player that gets updated when new events are ingested from Redpanda. All you need to do is ship it back to the gaming frontend. Any client application that can read Postgres data can use standard SQL to connect to Materialize and query the materialized view.

For example, you can connect a Spring Boot microservice to Materialized over Postgres JDBC Driver. This can expose the leaderboard as a REST or GraphQL API, so the gaming UI can connect, query, and render the leaderboard in the game. The following query will return the top 10 players on the leaderboard:

   SELECT * FROM leaderboard limit 10;

Congratulations! Your game officially has a functioning real-time leaderboard. And yes, you can set the limit to fit the number of top players your leaderboard should display.

Level up your real-time leaderboards with Redpanda

For most game developers, it’s a challenge to balance performance, accuracy, and cost-effectiveness. We used to trade accuracy for speed. But when games simultaneously attract millions of users across several platforms, you need them both.

The example in this post showcased an event-driven solution that provides precisely that, using highly available, scalable, and loosely coupled application components. Player interactions with the game are captured as events and sent to the system asynchronously. We used Redpanda and Materialized to secure fast data ingestion, reliable and scalable data storage, and support for fast queries.

As a reminder, you can find the entire real-time gaming example project in the Redpanda Gaming Industry Report repository on GitHub. If you found this post useful, there’s more real-time gaming inspiration waiting for you in our free report on how to turbocharge your games with Redpanda.

If you want to discuss any of these gaming examples with our team, join our Redpanda Community on Slack, or contact us to learn how Redpanda can power up your game development.