My DIY Dungeons and Dragons ambiance mixer

I find that an immersive sound ambiance is key to helping tabletop RPG players engage. It can increase their stress and sense of urgency during a fight, galvanize them during a harrowing speech, or break their heart when they realize they've just lost something and there's no getting it back.

I have been thinking about using a Launchpad to control and mix the ambiance while we play, but the more I read about its design, the less it seemed to fit. The cheapest Launchpad starts at 110€, and it is a full fledged MIDI controller. What I wanted was something simpler: a way to play different long sound ambiance tracks at the same time, and adjust their respective volume to create an immersive atmosphere.

The project started to take shape when I stumbled upon the Pimoroni RGB Keypad, a 4x4 rainbow-illuminated keypad that I could program using a Raspberry Pi Pico, for a budget of about 30€.

The color and brightness of the LEDs under the keys is programmable, meaning I could go for the look and feel of a Launchpad, while keeping my budget and the overall complexity in check.

The main idea would be to use 12 of the available 16 keys to start and stop audio tracks, and use the 4 remaining keys as controls (increase/decrease volume, pause all tracks).

Getting started

If you, like myself, want to program a Raspberry Pi Pico in Python, you have two options:

• MicroPython
• CircuitPython

It took me a while to figure out that these are more or less the same. In the end, I went with the CircuitPython starting-up guide, and was ready to make these keys light up.

A CircuitPython main program lives in a code.py file, that is executed when the board is plugged in. Any dependency can be put under the lib/ directory, itself placed at the root of the board filesystem.

I downloaded the rgbkeypad.py library, placed it under lib/ and wrote the following program in code.py

from rgbkeypad import RGBKeypad

keypad = RGBKeypad()

# make all the keys red
keypad.color = (255, 0, 0)  # red
keypad.brightness = 0.5

# turn a key blue when pressed
while True:
    for key in keypad.keys:
        if key.is_pressed():
            key.color = (0, 0, 255)  # blue

I then copied code.py and lib/rgbkeypad.py under the CIRCUITPY volume that is mounted when the keypad gets plugged into the computer, and voilà.

Reacting to key presses

Now that I knew how to program the key colors, brightness as well as knowing what keys were being pressed, I still needed a way to map these key events to starting audio tracks, and I was facing an immediate problem: the Pico has no way to play sound, even less on a Bluetooth-connected speaker. You know what can do all that really well though? My laptop.

So, if I could send messages from the Pico to my laptop (on which the Pico is connected for power anyway) and have a program running on my laptop receive them, I could then start thinking about how to play sounds.

It turns out that this was way easier than I thought, thanks to CircuitPython sending anything print-ed as binary data over the serial port. Using pyserial, I can write a program that connects to the same serial port the Pico is connected to, and receive the data.

# code.py, running on the Raspberry Pi Pico
from rgbkeypad import RGBKeypad

keypad = RGBKeypad()

# make all the keys red
keypad.color = (255, 0, 0)  # red
keypad.brightness = 0.5

# turn a key blue when pressed
while True:
    for key in keypad.keys:
        if key.is_pressed():
            key.color = (0, 0, 255)  # blue
            print(f"Key ({key.x}, {key.y} pressed!") # <-- that message will be sent over USB

# usb_listener.py, running on the laptop
from serial import Serial

# /dev/tty.usbmodem14201 is the name of the serial port the Pico was connected to
# on my mac. Your mileage may vary.
usb_device = Serial("/dev/tty.usbmodem14201")
for line in usb_device:
    print(line.decode("utf-8"))

I can now run usb_listener.py and press a key on the keypad to see the following:

$ python usb_listener.py
Key (1, 0) was pressed

Key (1, 0) was pressed

Key (1, 0) was pressed
...

Sending structured data from the keypad

Sending text data is fine, but we should probably send data that can be serialized on the keypad size and deserialized on the event listener side, as we will probably send the key ID, a state (pressed, stop, volume_up, etc). JSON is simple enough, and while the json package isn't available in CircuitPython, it's pretty easy to hand-encode JSON data:

# code.py, running on the Raspberry Pi Pico
from rgbkeypad import RGBKeypad

keypad = RGBKeypad()

# make all the keys red
keypad.color = (255, 0, 0)  # red
keypad.brightness = 0.5

# turn a key blue when pressed
while True:
    for key in keypad.keys:
        if key.is_pressed():
            key.color = (0, 0, 255)  # blue
            key_id = 4 * key.x + key.y
            print(f'{"key": %d, "state": "pressed"}' % (key_id))

# usb_listener.py, running on the laptop
import json

from serial import Serial

# /dev/tty.usbmodem14201 is the name of the serial port the Pico was connected to
# on my mac. Your mileage may vary.
usb_device = Serial("/dev/tty.usbmodem14201")
for line in usb_device:
    print(json.loads(line.decode("utf-8").strip()))

Playing sounds after a keypress

Playing multiple sounds at the same time in Python isn't something many packages allow you to do simply. In the end, I could only make it reliably work with pygame, which was developped to ease the creation of video games in Python. The package provides us with 2 different APIs to work with sound tracks:

• pygame.mixer.music, which allows an audio track to be played while streamed. This was intended to play some background music.
• pygame.mixer.Sound, which allows you to play an audio track on a specific audio channel. Mutiple Sounds can be played over different audio Channels.

Using pygame.mixer.Sound, we manage to react to a keypress and start the associated audio track

import pygame
pygame.init()

import json

from serial import Serial
from pygame import mixer
from pygame.mixer import Sound, Channel

key_id_to_audio_tracks = {
    0: Sound("example0.ogg"),
    1: Sound("example1.ogg"),
    2: Sound("example2.ogg"),
}

channels = [Channel() for _ in key_id_to_audio_tracks]
mixer.set_num_channels(len(channels))

usb_device = Serial("/dev/tty.usbmodem14201")
for line in usb_device:
    key_event = json.loads(line.decode("utf-8").strip())
    key_id = key_event['key_id']
    sound = key_id_to_audio_tracks[key_id]
    channel = channels[key_id]
    channel.play(sound)  # will play in the background

(The actual code can be inspected here).

While it works rather well, this approach has a fundamental issue. Because mixer.Sound does not support streaming the sound, as mixer.music does, it requires that all sounds be fully loaded in memory at startup. As the ambiance tracks that I'm planning to use all last between 30 minutes and 2h, the actual load time takes a couple of minutes. Using pygame.music would solve that issue, except for the fact that it only supports streaming of a single audio file at the same time.

I'm only left with mixer.Sound and loading hours of audio files in memory at startup, which means that the whole ambiance would take a lot of time to restart in case of a crash, and the energy around the table might deflate like a soufflé.

Sigh

Back to the whiteboard.

Alright, so, what program do I already have on my laptop that is good at streaming sound? What about an Internet browser? Youtube videos don't have to fully load before they start, and the same goes for audio files, so that might just work! I'd need a way to propagate these key events to a web page, so that it can then start/stop the audio files, change their volume, etc. Enter websockets.

Let's rub some web on it

The mixer would be composed of 3 different systems:

• the keypad, running the CircuitPython code
• a webpage, listening for key events over a websocket, in charge of playing the audio files and adjusting their individual volume
• an HTTP server in charge of receving the events over USB and propagating them to the websocket (ergo, to the browser), and serving the local audio files to the webpage. I'll use Flask and Flask-sock for this.

So what happens now when I press a key:

• a JSON-formatted message is sent from the pico to the serial port
• the message is received by the webserver process, and propagated to the browser on a websocket
• the browser deserializes the message, and takes action, depending on the content of the event

The browser-side message handler looks like this:

ws.addEventListener('message', event => {
  const keyEvent = JSON.parse(event.data);
  const usbStatus = document.getElementById("usb_status");

  if (keyEvent.state === "usb_disconnected") {
    usbStatus.textContent = "🔌 🚫";
  } else if (keyEvent.state === "usb_connected") {
    usbStatus.textContent = "🔌 ✅";
  } else if (keyEvent.state === "init") {
    colorizeTracksKbdElements(keyEvent.colors);
  } else if (keyEvent.state === "pause") {
    pauseAllPlayingTracks();
  } else if (keyEvent.state === "unpause") {
    unpauseAllPlayingTracks();
  } else {

    const trackProgressBar = document.getElementById(`progress_track_${keyEvent.key}`);
    const audioElement = document.getElementById(`audio_track_${keyEvent.key}`);

    if (audioElement === null) {
      return;
    }

    switch (keyEvent.state) {
      case "on":
        startTrack(keyEvent.key, audioElement, trackProgressBar);
        break;
      case "off":
        stopTrack(keyEvent.key, audioElement, trackProgressBar);
        break;
      case "vol_up":
        increaseTrackVolume(audioElement, trackProgressBar);
        break;
      case "vol_down":
        decreaseTrackVolume(audioElement, trackProgressBar);
        break;
    }
  }
}

The finishing touch

I have added a couple of features that will help me stay as focused on the storytelling as possible while I'm DMing, instead of thinking about the sound mixing process:

• a configuration-based tagging system, allowing me to get reminded of the main features for each individual track (is that an ambiance or combat music? Is it dark, light, opressing, eerie, etc?)
• I'm also propagating the key colors to the associated volume bar, allowing me to quickly identify the key that I need to press to start/pause/adjust a given audio track.

Demo time

I have the hardware! How can I run it?

Getting started instructions are available here for Windows users, and here for macOS and Linux users.

Closing words

The final iteration of that project is available here (for the keypad code) and here (for the webserver and webapp code). The black casing was 3D-printed using the rgb_keypad_-_bottom.stl file from this Thingiverse model.