Introducing ScholarLY

It has become a kind of a habit that more involved edition projects that I’m doing with (or as) beautifulScores not only produce beautiful printed results but also improve or extend LilyPond’s capabilities along the way. Take our award-winning edition of the songs of Oskar Fried for example. More or less direct by-products were Janek’s enhancements to shaping curves, Frescobaldi’s new Layout Control Mode and my lilyglyphs LaTeX package. Publicly shared insights about the usefulness of version control for musical work may be considered a surplus in this respect …

Now it seems our crowd engraving project on “Das trunkne Lied” by the very same composer yields even more important results. With ScholarLY I just created a spin-off from our orchestral score’s project directory that may soon become a serious editorial toolkit and make LilyPond an even more indispensable tool for scholarly work.

Entering Annotations

In a recent post I described how I implemented a preliminary interface for annotating musical scores in the LilyPond input files. This relates to an even earlier post where I expressed the desire to edit critical remarks in-place. As a reminder, this is the interface I added to enter annotations in LilyPond’s input files:

  \musicalIssue \with {
    message = "Hairpin missing in OE, taken from bassoons"
  f2. \> ) |

This annotates the hairpin following the f2. with a “musical issue”, creates a clickable message in the console output and colors the hairpin green. The coloring immediately draws the attention of a reader to the spots that may still need consideration and is therefore a very useful tool already. But there was one fundamental drawback until now: annotations had no notion of their place in the timeline of the score. Having received a jump start through a ready-to-use basic engraver provided by David Nalesnik (thank you!) I managed to overcome this limitation and now annotations know “where” in a score they have been inserted!

There are five annotation commands available, \criticalRemark, \musicalIssue, \lilypondIssue, \question and \todo, plus the generic \annotation that can be turned into a custom annotation type. In fact these annotations can already act as a quite versatile in-score issue tracker – which is a very good thing because in my experience using regular issue trackers such as on GitHub or even the more fluid approach of a “card” type application like Trello are too clumsy to be used efficiently in the context of an edition project. They tend to be used rather sparingly and thus inappropriately. Having the possibility to insert issues directly in the source is a practical approach that can be used with little overhead, and I think it can encourage maintainable workflows complementing version control.

Exporting Annotations

As another major improvement annotations can now be exported to a file instead of only being printed to the console. The implementation is still somewhat rough but it can already be used professionally for our edition project (or adapted to arbitrary projects). So far annotations can be exported to plain text and LaTeX files, but I intend to add support for more file types, e.g. JSON, Markdown or HTML. Eventually this should be template based so it can easily be configured for specific projects.

When I wrote about Using LaTeX for a Musical Edition last year I was quite happy about the usefulness of entering critical remarks as LaTeX commands. These allow for structured input, flexible (re-)formatting and of course beautiful output. And now one of the dreams of that post has come true: the LaTeX input for these critical remarks can now be maintained directly in the LilyPond score! For example, the annotation shown above is rendered to the following LaTeX code:

   [author={Urs Liska}]
    {Hairpin missing in OE, taken from bassoons}

Using LaTeX this now can be typeset beautifully, and – as mentioned in the former post – it can produce different output for proofreading or final publication, simply through setting one switch. This is not fully implemented yet but I’ll soon have a proper initial LaTeX package ready to produce critical reports from LilyPond-generated annotation data! See the same annotation rendered in a structured “draft” layout by LaTeX:

An annotation in draft layout (click to view PDF)

An annotation in draft layout (click to view PDF)

A nice thing about producing such reports with LaTeX is the versatile way such “data” can be processed and rendered. See a sequence of annotations rendered in a compact publication layout:

Some critical remarks in publication layout (click to view PDF)

Some critical remarks in publication layout

Well, this is still rather sketchy but one can already see a few things here:

  • The last annotation is in the same bar as the previous, therefore we don’t repeat the barnumber and print a dash instead
  • The annotation type is only printed when it is not a critical remark
    (actually this doesn’t make much sense because in reality only critical remarks should remain in the report, but I wanted to show the technique)
  • the pp is beautifully printed using a notation font using the lilyglyphs LaTeX package. Not only this but arbitrary LaTeX code can be entered in annotation messages.

Using it in Scholarly Editing

So, why is this cool? Well, the main point of all this is that music and annotations can be edited in one single place – the LilyPond input file. This relieves the author from establishing (and keeping alive) the link between the score and a separate text document mentally. While this may look like just a nice feature I’m convinced this is an actual quantum leap in usability with regard to scholarly editing of scores.

Have a look at the following screenshot from a Frescobaldi session. The Music View on the right hand of the screen shows two differently colored items (green arrow) that immediately tell the reader browsing the score that there is something to consider. Clicking on an item will let the source editor on the left hand side open the input file at the correct location (red arrows) where the annotations can be edited or even removed. At the same time the LilyPond Log on the lower left side has produced a chronological list from all annotations. When looking through all of the annotations this is an even more efficient and reliable approach than looking for colored items in the score. Of course these entries are also clickable and jump to the correct input location at will.

Frescobaldi with annotations in music view, source editor and log viewer (click to enlarge)

Frescobaldi with annotations in music view, source editor and log viewer (click to enlarge)

Now let’s consider the process of proof-reading a scholarly edition. For this I carefully compare the original source(s) with the new engraving until I spot any difference. Now I have to look up if that difference is documented in the critical report, so I note the measure number and locate the corresponding position in the report. With the new annotations it isn’t necessary anymore to switch contexts and work my way through the text document. Instead I can simply navigate within one editing environment, as one can see from the above example. So if I notice a difference between the source edition and the new edition I can easily deal with it without all that tedious, confusing and error-prone context switching I had to deal with until now!

Just as a side note: It may sound trivial but for real-world projects it is also a great improvement that timing information about annotations doesn’t have to be maintained manually in the report document. This is also generated automatically by LilyPond, so I can’t make any errors and don’t have to worry about formatting consistency.

But the improvements already start with the original music entry, and the different annotation types ease the workflow even more. With LilyPond’s new features I can immediately make notes about my observations: when I enter a Slur I can add the “musical issue” annotation right beside it in the input file. And later, when I’m doing a proper critical revision I can change that to a “critical remark” once I’ve decided about the case. This ease of entry and navigation is actually encouraging people to make use of annotations, and you can take our crowd engraving project as an example where we already produced 1,201 annotations by now, just having entered and peer reviewed the instrumental parts of the score, without having even started a proper revision. Maintaining annotations in the input files (with colored output and console messages) also makes it significantly less probable that annotations slip through our attention. When comparing a score with a separate document it is my own responsibility to process each single entry while unprocessed new-style annotations will “pop up” on their own.

Starting With a New Project: ScholarLY

Originally I developed \annotate and its infrastructure in the context of our current edition project, and it has turned out to be a wise decision to use it even when it was only an interface that didn’t do anything yet. If I had waited until the function is ready we wouldn’t have been able to actually use it in the project. And as it stands now the presence of 1,201 annotations has been a very effective motivation to seriously dig into the issue again because otherwise most of that work might have been wasted.

Now that the annotation functionality is in a usable state I created a new project and moved the code into it: ScholarLY. This is a launch I had in mind for quite some time as I hope this will become a serious and powerful toolkit for scholarly editions, and I’m happy that I finally got there. A consistent and comprehensive library with reliable and versatile edition tools will make the natural advantages of LilyPond’s text based approach even more obvious, and I hope it will attract new academic users, institutions and/or publishers to use LilyPond and friends. If you are interested in the project you may have a look at the Wiki which describes what can already be achieved, at the Roadmap document where I collect my ideas and wishes. Or you can of course inspect the actual code. If you obtain the library (by download or git clone?) you can play around with the example document in the examples folder too.

With \annotate I did not create more than the foundation for this project, and the whole “house” will have to be built on top of that. Therefore I encourage everybody who is interested in scholarly editing to join me in that undertaking. Of course I’d be happy about collaborators with skills in LilyPond, Scheme, LaTeX and Python but there is also significant need for discussion about overall strategy and user interface, so don’t hesitate to get in touch even when you don’t think you’ll be able to contribute code. I propose any communication to be done on the lilypond-user mailing list, so if you don’t think you should contact me privately please post something there.

Apart from \annotate there are two more building blocks that will form the foundation of ScholarLY: a well-defined toolbox with smaller helper functions that make scholarly editing simpler with LilyPond, and LaTeX package to typeset reports from the LilyPond-generated annotations. \annotate will be able to use these tools, e.g. for highlighting editorial amendments, adding footnotes or ossias, and there will be other useful tools that scholarly editions may need to prepare their scores and reports.

I’m sure \annotate will once more have a significant influence on the way I conceive scholarly workflows, and I hope this can become a widely used toolkit for others as well. I think this really is something important, not only for its actual value but for its potentials. Maybe the most fundamental aspect to it is not what ScholarLY is able to do but that it shows what can be done with LilyPond in general. Given a vision and people with the right skills it is possible to create tools and workflows for about anything – which is inherent to LilyPond’s text based design.

8 thoughts on “Introducing ScholarLY

  1. Ralph Palmer

    Greetings, Urs –
    I am currently working on transposing the Bartok 44 Violin Duets down a fifth so I can play them on viola (with the music laying on the strings the same as it does on violin). I would like to try \annotate or ScholarLy, but it is not clear from the article how I can do that. Incidentally, I use Frescobaldi and LilyPond 2.18.2 running under Windows 7. I appreciate your time and efforts in support of LilyPond!

  2. Urs Liska

    Well, you should go the the project’s home page on Github and try to follow the directions there. I deliberately don’t write more here because I’d like to get feedback on the documentation there. If you stumble over issues that aren’t explained well enough don’t hesitate to open issues on the project’s issue tracker

  3. Philippe Massart

    I’m delighted by the LaTeX possibilities 🙂 Nice work, and many ideas from it 😉

    What I look forward is the ability to include some score in the annotation, like in the “ornementation explanations” in many baroque or classical scores.

    1. Urs Liska

      Yes, this is definitely on the todo (or maybe rather wish) list. Annotation properties can already consume arbitrary Scheme/LilyPond expressions, including \score blocks. Extracting music from a message property might be more complicated. Currently I’m thinking of two approaches:

      Creating “ossia” annotations. When an annotation contains an “ossia” property this score block should be inserted as an ossia staff.

      Inserting score excerpts in the message body. This could be entered by using separate properties with a score block and referencing them with a replacement variable inside the message body. When exporting to LaTeX, Markdown or HTML this score block could be compiled separately and the corresponding link code inserted in the output file.

      But these are items where I’d be happy about feedback, discussion and/or contribution

      1. Urs Liska

        Ah, and I forgot: If you are talking about inline notation (within the continuous text) you can already use anything that lilyglyphs is able to do.

  4. Richard Shann

    By coincidence I recently gave Denemo an ability to use annotations. The method is rather different here: the person doing the proof reading is looking at the output PDF and makes PDF annotations there. In Denemo it is possible to open this proof-read PDF and to navigate to the anntotations, clicking to transfer them into the Denemo source file.
    This is for a different scenario, where the person doing the proof reading (typically the composer) does not want to learn to work with LilyPond code or even the Denemo display, they are not technically inclined.
    What your new facilities allow is to make the annotations show in the LilyPond output (at present in Denemo they are imported as Denemo comments) as \musicalIssue or whatever might be most useful. Then the creation of documents summarizing the outstanding issues becomes simple without re-inventing the wheel.

    1. Urs Liska

      Yes, that’s also on the wish list: Producing documents with lists of entries that link to the input code. I also intend to integrate this into Frescobaldi, as an annotation browser/editor. It is also already possible to “create” custom annotation types for project specific purposes.

      But my strongest motivation was to enable critical reports to be produced from data generated in the LilyPond input, which makes it possible to maintain everything in one place. I find this continuous context switching so annoying …

  5. Pingback: “Defining a Music Function” | Scores of Beauty

Leave a Reply

Your email address will not be published. Required fields are marked *