Skyfall, part 7: a track editor for the music

Part 7 of the Skyfall story. In part 6, the game got its music: 9 tracks rendered by a Python synth. From November 10 to 12, I built three tools to make the game’s content: a track editor for the music, a map editor and a sprite editor. This article covers the first one, the track editor, which landed on November 10. Part 8 covers the map editor, and part 9 the sprite editor.

Table of contents

• Music as data
• The track editor
• The piano roll
• Instruments and mix
• Nine tracks in one picker
• .mid files
• Notes from words
• Notes from a microphone take
• A self-test that clicks
• By the numbers
• The Skyfall story

Music as data

In part 6, each track was Python code. music.py and themes.py wrote the notes with loops and rules, and the synth rendered them to OGG. To change one note, I had to change a program.

On November 10, every note moved into data. Each track is now a JSON file in tools/audio/songs/, with every note, every part, the instrument layers, the drum kit and the mix settings. song.py renders the file. The code that first wrote these files was throwaway, and I deleted music.py and themes.py in the same commit. The JSON files are now the source of truth.

I re-rendered the nine tracks from the new files. The music is the same, but plucks and noise got new random seeds, so the audio differs slightly. Since then, each note seeds its randomness from its own start time, and each drum renders once. A re-render gives a byte-identical file, whether the track renders alone, in a batch, or after an edit somewhere else in the song.

The track editor

The editor is a standalone LÖVE app, like the map editor from part 5. I start it with love tools/audio/editor from the project root. It uses none of the game’s code, but it has the look of the new menus: the same pixel-art UI kit, and the same sky behind the header.

LÖVE doesn’t synthesize the sound. On launch, the editor starts a small render server in Python, tools/audio/editor.py, and stops it on quit. The server renders each part to its own WAV file, a stem. It also saves the songs, exports the OGG files, and reads and writes .mid files. The parts render in parallel. The editor plays the stems together, so mute, solo and each part’s volume change the sound without a new render.

The piano roll

The middle of the screen is a piano roll. A click draws a note, a drag moves or resizes it, and a right click deletes it. Notes copy, paste and delete in groups, and a lane under the roll sets each note’s velocity. Every change can be undone. The snap goes down to triplets, which a jig in 12/8 needs.

The roll knows the song’s key. Rows in the key are lighter, and the root row has a gold tint. Above the notes, a row of chord names and section markers shows where you are in the song. With All parts on, the notes of the other parts show faded behind the part you edit. With Hear notes on, each note plays as you draw it.

A drum part is a kit, and each row of the roll plays one piece. Elderhollow’s drums are a high bodhran at 140 Hz, a low bodhran at 85 Hz, a crash and a clap.

Instruments and mix

The Part tab, on the right, sets how a part sounds. An instrument has one or more layers, and every note plays all of them. Each layer picks a voice from the synth, with its level, pan, transpose and detune. Under the instrument, the gate sets how long the notes ring, and humanize shifts each note by a few milliseconds. The mix has the part’s volume, its reverb, a low cut and a high cut, an echo with its time, and “Pump with the kick”, which ducks the part on the kick’s beats.

The Song tab holds the rest: the title and a short description, the game track that plays the song, the tempo, the length in bars, the tail, whether it loops in the game, the snap, the key and its mode, the song’s reverb, and the beats the pump ducks on. Each song also has a loudness target. Export to game writes the OGG at that loudness, in assets/audio/music/.

Nine tracks in one picker

The song picker shows the nine game tracks as cards. Each map’s battle track has the map’s preview, and the other tracks have the title screen’s sunset: the storm battle that the two Towerfall maps played, the main menu and the victory fanfare. Each card gives the key, the tempo, the length and the number of parts. The tempos go from 84 BPM for the main menu to 165 BPM for Embergrove, a riff in D Phrygian.

”+ New song” starts a song that no game track plays yet. It goes in songs/unassigned/, and Export to game stays off until the song has a track. The Game track menu, in the Song tab, gives it one. The track’s old song then moves to Unassigned, so every track keeps a song and no song gets lost.

.mid files

The Song tab exports a song to a .mid file and imports one back. A .mid file dropped on the window imports too. A round trip gives the song back unchanged. On import, the tracks of the .mid file go to the parts with the same name, and a new track becomes a new part. So a track can go to any music software and come back.

Notes from words

Later that day, the editor learned to write notes from a description. A button under the part’s instrument writes that part’s notes, or reworks the ones it has. Another button, under the parts list, does it for every part at once. Both open a dialog where I describe what I want.

The request goes to a language model with the whole song: the key, the chords, the sections, and every part’s instrument and notes. So the model reworks the existing notes instead of replacing them blindly. It answers in patterns with repeats, and tools/audio/compose.py checks the answer before any note lands. The request goes out through the command-line tool already logged in on my machine, with no API key, or through an API key, billed per request. A short song answers in about 12 seconds.

The answer lands as one undo step. It goes to the parts by name, so the edits I make while it runs are kept.

Notes from a microphone take

The same dialog takes an audio example, with or without a description. tools/audio/transcribe.py reads the audio with numpy and ffmpeg and writes the notes it hears. The example can be a file, picked or dropped on the window, or a take from the microphone, up to 3 minutes long. For a single part, an example with no description never leaves my machine: it goes into the part note for note. A drum part takes the drums from it, a bass takes the lowest line and a lead the top line.

Three toggles set what plays during a take. The count-in plays one bar of clicks at the song’s tempo, and the editor cuts that bar from the take, so the take starts on the first beat. The click can also play through the whole take. And the song can play behind it, without its muted parts. Without headphones, the microphone records all of that too. A Bars row sets where the example goes in the song, from one bar to another, and a take stops by itself after the last bar.

With the button for every part, the example leads the arrangement. It goes into one part as transcribed, and the model writes the other parts around it: a bass line, chords, a groove and counter-lines, without copying its melody or its rhythm. An empty song first takes the example’s tempo and length.

The transcription is a sketch. A single melody comes back nearly note for note, and bass lines and drums mostly do. Ringing arpeggios gain extra notes, and the chords of a slow pad start late.

The same evening, Elderhollow’s track got a tenth part, a double bass.

A self-test that clicks

love tools/audio/editor --selftest runs the editor without a window and drives it with real mouse and key events. It draws a note, drags it, deletes it, raises a velocity, plays the song, changes the tempo, renames a part, picks a voice, mutes a part, and exports a .mid file then imports it back. After each gesture, it checks the song. On the Python side, song_test.py checks that the song files round-trip, that every voice renders, and that .mid files come back unchanged. compose_test.py checks the transcription, the request and the answer checks against stand-ins, and never calls the real service.

By the numbers

• 6 commits on November 10 for the editor and its features.
• About 3,900 lines of Lua for the editor, and about 4,100 lines of Python in tools/audio/, the synth and the sound effects included.
• 9 game tracks, with 5 to 10 parts each, and about 10,900 notes in all.

The Skyfall story

1. From a Defold cube to a fresh start in LÖVE
2. Wall jumps, a real bow and a quiet network
3. Rounds, rain, torches and a VHS replay
4. The War Room and the armory
5. A map editor, a ghost and the Champion Forge
6. Pixel-art menus, six arenas and sound at last
7. A track editor for the music (this article)
8. A map editor that draws with the game
9. A sprite editor for the blocks and props