The “Segment Grid” Approach

This is a post in a series about the “Crowd engraving” project “Oskar Fried: Das trunkne Lied”

Ever since I started using version control for working with scores I mused about the potential this might have on a “community” approach to editing musical scores. In our edition of Oskar Fried’s songs I already enjoyed the positive impact on collaboration, but basically we had a team of only two people with more or less fixed roles: I was responsible for producing the content, Janek had to beautify the output. However, along the way I learned to enjoy the freedom and comfort LilyPond and Git offered as a reliable safety net when our roles started to blur to a small extent. I could make suggestions regarding LilyPond coding, and Janek reported issues with the content more than once. This triggered ideas about making musical editing team-ready the same way as software development works. The task of creating a huge orchestral score was a welcome opportunity to develop new tools and workflows. Today I will give you an idea about the fundamental concept of slicing the score into a “segment grid”.

A Huge Score

Being confronted with a large score may look daunting in the first place. Oskar Fried’s “Das trunkne Lied” is a composition for vocal soloists, multiply split choir (with eight solo voices) and large orchestra. It has nearly 150 pages, and you can have a look at one – admittedly more crowded than average – by clicking on the thumbnail below:

Page from "Das trunkne Lied". Click to enlarge.

Page from “Das trunkne Lied”. Click to enlarge.

In order to manage that beast and to make it accessible to efficient collaboration I created a framework built on the fundamental concept of slicing the score into a “segment grid”. This makes heavy use of the advantages offered by LilyPond’s plain text approach and version control as described over and over again on this blog.

Working With the Grid

It is quite obvious how to conceive such a grid (although I’m not sure it’d be as obvious to users of traditional graphical notation packages). A score is naturally organized in a set of “voices” or “parts” – any LilyPond user would enter each part in a separate music variable and maintain it in its own file. But a score usually is also organized over time, usually with rehearsal marks, which form a natural way to define the “columns” of our grid. In our score we have 90 of these columns, so we can address any “cell” of the grid directly like “segment 47 of clarinet 1” etc. My goal was to make each of these cells editable individually in order to make the tasks more manageable to individual contributors and to make the best out of version control.

Individual Segment files

Maintaining each segment in its own LilyPond input file offers several nice goodies. Each file is small and manageable so they are also accessible to users with less experience who’d be overwhelmed by having to handle an 800 measures part. Being able to compile each segment standalone greatly reduces the time spent waiting for LilyPond to recompile the stuff currently being worked on (which is actually one of the more annoying limitations when using LilyPond for larger scores). In terms of version control maintaining many small files reduces the risk of running into merge conflicts, which improves reliability and maintainability of the project. And finally this approach provides a maximum amount of stability when “filling up” the score – which I’ll talk about a little bit later.

Compiling standalone segments

For editing the segments as standalone files I wrote a Scheme function \compileSegment which takes some music input and compiles it as a standalone score. This in itself wouldn’t be interesting at all, but I set it up so it only does something when we want the file to be compiled standalone. When we use the segment file as part of the whole, this function simply does nothing. So basically we have a structure of the music file that looks like this:

% Include function to compile the segment standalone (e.g. for proof-reading)
\include "makescore/compile-segment.ily"

% Please check the reference pitch of \relative
XXVII = \relative c {
  % The music of the segment

\compileScore \XXVII

(If you want to see the details of this \compileSegment function, you can check it out here.)

When used as part of the full score (or full part), this file effectively yields just the \XXVII variable – but when compiling the file directly LilyPond uses \compileScore (together with some style sheets and templates) to produce an output of that segment alone, as you can see from the following screenshot:

Individual segment in Frescobladi (click to enlarge)

Individual segment in Frescobladi (click to enlarge)

When looking at this you may notice a few things, first of all the extent of the score. The segment is defined to comprise the viola 1 music from rehearsal mark 26 to 27, but actually it starts a little earlier and stops a little later. This is done to let LilyPond engrave spanners that go over the segment borders, in this case some slurs and ties but there could be dynamics too. For this to happen the segment files contain additional variables \opening and \closing that can be left empty or can contain such “transition objects”. (So the actual structure of the file is somewhat more complicated than listed above.)

The next thing you may be wondering about is the peculiar layout of the systems, ragged-right and with a seemingly arbitrary number of bars per line. Actually this is one of the coolest aspects of our approach. There are custom commands for entering the line and page breaks of the original score, and with that LilyPond will create scores that match the line and (optionally) page breaking of the manuscript we’re copying from. This makes it much easier to navigate and to match the compiled output to the position in the score. The behaviour of this and other aspects is centrally managed in initialization files so once the musical editing is done we can simply switch that feature off and let LilyPond calculate the optimal line breaking for us – but I’ll talk about this aspect of the set-up in a later post.

I will probably come back to this screenshot as it shows a few more things that should be covered separately.

Creating voices and parts

How does LilyPond and the project know about the structure of the composition and the score? Do we have to create such a segment file manually? Of course not – this would be way too tedious and error-prone. Instead there is one template, consisting of 90 music variables that define all the time and key signatures, original breaks, rehearsal marks and tempo indications (the actual implementation is slightly different but as this is not a tutorial I’ll be somewhat generous with the details). The actual music in that template is defined as spacer rests (for non-LilyPond-users: spacer rests take up the rhythmic value they contain but do not print anything). A Python script can then be used to generate a complete part from that, creating 90 empty segment files with all the default content and some instrument specific settings such as transposition or initial clef.

The person entering the music now “only” has to replace these spacer rests with the actual music, without introducing any errors in the timing (for example by replacing five measures of spacer rests with four measures of music).

To create an instrumental part from that, there is a file that reads in all those 90 segment files and concatenates them to a continuous voice which can then be used in a standalone part or in the score. Initially this looks like the following “empty” part:

Empty "part", containing no music but the whole structure (click to enlarge)

Empty “part”, containing no music but the whole structure (click to enlarge)

As you can see this too respects the original breaks of the score, and additionally it prints the original page number where you’d usually have the abbreviated instrument name. This way it’s really easy to navigate between the manuscript and the compiled part. (If you want you can place this PDF side by side with the single score page from the top of this post and inspect their relation.) As the generated empty files are automatically equipped with \barNumberCheck commands any timing errors in the segment files would immediately become apparent once the whole part is compiled – and can easily be fixed without having to worry at all about breaking the layout of the score.

Drop-In of Really Empty Segments

In an orchestral score the instruments do not play all the time, so there are many occasions where instruments do pause and where the spacer rests from the template would have to be replaced with real multimeasure rests. To make this process less tedious I have come up with yet another trick. I wrote about the file that concatenates the segments to a part. When it doesn’t find a segment file in the expected place it will simply load the respective segment from a special template part containing all the segments filled with real instead of spacer rests. This means: Instead of “entering” the “music” for a pausing segment one can simply delete the corresponding segment file! This has the interesting side-effect that when looking at the score or a part you can immediately tell apart segments where the instrument pauses from segments that haven’t been entered yet:

Partially completed oboe part (click to enlarge)

Partially completed oboe part (click to enlarge)

In this oboe part you can see a lot of segments containing rests, a lot of segments containing nothing, and some segments (starting with rehearsal number 41) that already contain music. This points to one huge advantage we have from using a text/compilation based tool such as LilyPond. With graphical notation software it is known to be a quite fragile thing to maintain a longer score: it’s important to do everything in the right order to prevent later changes from breaking any- or even everything done so far. By contrast our approach makes it possible to pick any segment from the whole score to enter – and LilyPond will painlessly place that music at the right place in the score, without any hassles like breaking layout or the like. When later someone enters an adjacent segment they will be simply glued together upon the next compilation!


In the next part of this series I will start with the (positive) impact this approach has on the stability of the score and on our collaborative workflows. But I’ll also consider some possible limitations of the approach and discussion of it being too complicated.

6 thoughts on “The “Segment Grid” Approach

  1. Pierre-Luc Gauthier

    Colossal work and truly inspiring. It is a great “proof of concept” that Lilypond IS a complete and “production ready” way of making beautiful scores and individual parts collaboratively. Thanks Urs for exposing this work flow. I cannot wait for you next publication on the subject (especially the compile-segment.ily code of yours), it all is very passionating.

    1. Pierre-Luc Gauthier

      ^ Sadly, I do not have an account to edit this horribly marked-up response of mine. I do not know how to use the markups and there is no preview features.

  2. Pingback: Restoring Deleted LilyPond Files | Scores of Beauty

  3. Pingback: The Future of openLilyLib | Scores of Beauty

Leave a Reply

Your email address will not be published. Required fields are marked *