Why Use Version Control for Engraving Scores?

Since I’ve started writing about plain text and version control and the like I’ve occasionally been asked about my opinion. Usually these questions circle around collaborative workflows, but recently I was asked by a composer whom I tried to “convert”:

Since I work totally alone, I honestly don’t yet see the benefit —
please tell me what I’m missing, as a solo engraver.

My immediate reaction to such a question has something quasi-religious about it, along the lines of “Believe me, when you’re there you’ll see the light.” But as I wouldn’t consider this a satisfactory answer, the question turned out to be quite intriguing. So I made up my mind and did find a number of reasons why any LilyPond user should at least seriously consider using version control, even when she’ll never work with someone else. And since the composer who had asked considered them “compelling” I thought it would be a good idea to pour it into a blog post for a broader audience.

Actually I consider it a deficiency that the use of version control isn’t endorsed more prominently in the LilyPond community or its website. I’m sure if I had heard about the concept right from my first contact with LilyPond I would have started to actually use it several years earlier. From my impression there are users who don’t know about it and users who already knew about it from other contexts (e.g. programming work) – but those who know it take it for granted and don’t consider it necessary to talk about it…

In a second post I’ll then take my personal experience – my doubts before and my enthusiasm after the “conversion” – as an opportunity for some thoughts on possible ways into that new world, addressing the composer’s follow-up question about an estimate on the learning curve.

It’s right there, so why not simply use it?

LilyPond works on plain text input files; that’s not exactly a secret. And therefore it’s readily accessible to workflows with version control – but this is much less known.
Not using version control as a tool would seem comparable to deliberately setting aside the localization of your operating system. You can do without, but if the tools are available it’s usually a good idea to use them.

Of course this isn’t a strong and logical argument that would convince someone in doubt, but just like the tools it’s simply there 😉 .

Restoring (and surpassing) the “unsaved file” concept

One thing that had annoyed me since I started working with compiled systems (be it LilyPond scores or programming languages) is that you can’t work in the context of an “unsaved file”. With graphical programs you can simply start hacking away and at the end of your working session decide to either save your document, discard it or save your work to another file (keeping the previous state as a copy). With compiled systems you have to save the file as soon as you want to compile it – which you usually do quite often during a working session. If you want to keep track of the state of your document before starting a session you’d either have to rely on your editor’s Undo capabilities or always work on a copy of your project and replace your previous version only at the end of the working session. (However, Frescobaldi does provide an option to preview the output from the current state of a document without having to save it.)

Version control provides a concept that is roughly equivalent to the idea of an “unsaved document” and that lets you graciously overcome this limitation of compiled systems. In a versioned project you’re always working on top of the latest commit, and this is like the last saved state of the document (or rather project). You can save and compile to your heart’s delight without compromising the ability to go back to that last known state.

But this approach is actually even more powerful than the unsaved document:

  • You can save your file much more often, so if your computer crashes you’ll only lose a small bit of work.
  • When making your next commit you can selectively add your changes. This means after hours of tweaking you can freely decide which portions of your work you want to keep and which not. No need to decide between “keep” or “discard” your whole work. Or you can organize your work in coherent sets of changes by committing them in groups tailored to your needs.
  • This isn’t dependent on a phyiscal working session on your running computer. You can decide to make a new commit after a few minutes or after a week of intermittent work. To give an extreme example – just to blow your mind – you could remove the hard drive from your computer, send it on a trip around the world, and resume and commit your work on another computer.

Having access to the full project history

When you work with version control you can always inspect any previous state your project had been in. For example if you managed to break something and noticed it only somewhat later you can go back in history to determine the exact spot where it happened – there you can selectively revert or modify the offending addition.
Or you may wonder why you did something the way you did – you can go back to that point in history and examine the context the last change was made in. (Actually a version control system can tell you when a given line of your input has been changed.) This may help you recalling what you may have thought then.

It may not be easy to put your fingers on, but from my experience this is an option you don’t want to miss any more once you’ve gotten used to it. Nowadays if I encounter an old and complex project from the time before I used version control I feel uneasy and somewhat helpless. Even if I won’t need it at all in any given case it’s very good to know that I might access, inspect and modify the full history of a project if necessary.

Keeping the project consistent with branches

The progress of a project under version control is by convention reflected by the progress of its master branch. There are many more or less formalized workflows around, but it’s always a good idea to keep that master branch in a consistent state at any time. For example, when editing a score it should – at any given point in the history of the master branch – be compilable without errors.

To achieve this you can do all work of transitional or experimental character on dedicated branches which are somewhat like isolated sessions. Once you’re finished with the given task and decide it’s to be kept you merge it into the master branch, and this will proceed from one consistent state to another, short-circuiting all the intermediate states in between. This way you can always keep your project in a consistent state – even when you’re working at several issues at a time, in independent branches.
Some use cases for this are:

  • Adding a logical layer, e.g. articulations and dynamics.
  • Entering a complete part
  • Reviewing or overworking a finished score
  • Incorporating the scholarly review

Being able to work in such isolated sessions – without compromising the previous state – can significantly make you more comfortable with retouching “finished” work.

Working on multiple computers

If you – like me – have to work on different computers you actually are collaborating – with yourself. In that case you can benefit from using version control with a remote repository hosted at a service provider’s server.

If you are synchronizing your work using USB devices and forget to synchronize your working directory before switching computers you’re heading for trouble: You have to scrupulously avoid editing files that also have been modified on the other computer.
But if you’re having the same situation with a project under version control (i.e. you forgot to commit and/or push your changes) you can simply continue to work on the other computer (of course without being able to access the latest changes) and simply merge your work later. If you should happen to create conflicting modifications version control provides concise tools to resolve them.

Use web interfaces to edit project files

If you are hosting your repository on a server (which may make sense even when you only use one computer at home) you have the possibility to edit your files directly there through the web interface of the hosting provider. This is a very nice option for quick ideas you may come across when you’re somewhere else. Simply log in through a web browser, edit your file and apply the quick fixes.

(And although we started with the assumption of someone working on his own this is a great way to let others participate and review your work. I experienced several occasions when people who didn’t know about the technical details but were somehow involved in the project added their remarks or additions directly that way.)

Indirect impact of using version control

Apart from these concrete aspects using version control may have an indirect impact on your way of working, in fact it can change your whole thinking about tools and workflows. What I have written several times isn’t a buzz phrase but a deeply felt reality: I can’t imagine anymore how working was before I started using version control.

Version control acts like a safety net and greatly reduces the risk of losing work once it has been committed to the repository. This makes you feel much more comfortable working at different edges at the same time or trying out things that may or may not be persistent or that risk breaking something.

Version control also makes you more aware of structuring your work in coherent units. Once you start working with repositories you will be much more careful to organize your work in sensible units that will form the chain of commits in your project history.

That way version control makes you more free and more structured at the same time. And all this applies equally whether you’re working on your own or collaboratively with tens or hundreds of collaborators.

And now?

So, if it’s all that terrific why isn’t everybody using version control?

Well, as I’ve already said there are too many people who simply don’t know about version control. Or they have heard about it but would only think of it as a programmer’s tool. Throughout this blog and also on the LilyPond mailing list I try to do something about that 😉 and it’s also part of my first visible contribution to LilyPond – the “Features” page on lilypond.org (English only so far) now mentions version control.

But more importantly, version control actually comes at a cost: you have to learn a completely new concept and toolkit.

I think it’s a more than worthwhile investment, but I also can imagine why it looks daunting for people who haven’t used it at all. I recall that just before I actually started with it I was very unsure if it would work out and if it’d be worth the trouble.

In an upcoming post I’ll collect some old emails from that time, try to describe how I got talked into trying it and how I experienced the learning curve.

11 thoughts on “Why Use Version Control for Engraving Scores?

  1. Johan Vromans

    A good start would be to add git support to Frescobaldi. When creating a new score, set up a new git repo. Offer the option to commit changes upon every save, or LP run, or manually.
    This makes the theshhold to start using a VCS very low.

    Reply
    1. Urs Liska

      This is actually on the todo list already. See this issue on the Frescobaldi tracker.

      I’ve started with this by adding Git support for the Frescobaldi program: When you run it from its Git repository you’ll get a “Git” menu that lets you switch between any branch present in your local repository, making it a mouse click to switch between different development versions (may also be from other developers if you have fetched branches from their forks).

      I started with this because it’s much simpler in terms of Git use cases than adding functionality for notation workflows. But I must admit I got stuck with the necessary interaction when communicating with remote repositories (next step I planned was an auto-update function). But if I think about it from today’s perspective it might actually be easier to move to (local) workflows for working with scores.
      E.g. I’m thinking of displaying a diff against the current commit (so you can see what you’ve edited in the current “session”). Or an interface to “git blame”. Or enabling partial staging (adding selected edits to the next commit). What I would not consider useful is committing at each save/compile event. It makes more sense to add an explicit command for “Commit”. Maybe with a somewhat more innocent name 😉

      Reply
        1. Urs Liska

          Sonds like a good suggestion.
          However, one has to be careful with such “reinterpretations” because they may actually be confusing. I was for example stumbling over several commands in my Git GUI, simply because they used terminology that is different from the official one (for example they let me “save” when they meant “stash”).

          Reply
  2. Brent

    I would really love to learn more about using Git with lilypond/Frescobaldi. I am in the process of learning Git as a programmer and I understand the basic concepts. The part I am not getting, and have not seen explained anywhere adequately, is how to work version control into my workflow. When do I add/commit my changes? Are you constantly jumping back and forth between your IDE and the Git command line?

    I think more would be using version control tools if someone could offer some suggestions on how it fits into the flow of daily work activity. Looking forward to your next post.

    Reply
    1. Urs Liska

      That depends on the kind of work and the used tools I think.

      If I’m entering music for a new score I would make a commit for any coherent set of changes. In the case of a song with voice/piano and only a few pages this would for example be one commit for the vocal line, one for the lyrics and one for each voice/part of the piano. Before I would have done one commit for the basic structure for example.
      When improving the score there are different approaches. You can for example just do some work and then commit, more or less by chance or by extraneous reasons. But you can also do some specific work and commit, e.g. use one commit for shaping slurs, one for treating collisions, one for proof-reading of the notes etc. Or you can be very specific and commit each change individually. I have come to the conclusion that this is actually a good idea in some cases. The proof-reading phase of the “Fried project” that has become so ubiquituous on Scores of Beauty recently shows a large number of tiny commits, most of them containing not more than one single changed pitch or a similarly tailored modification.

      And yes, this requires me to switch between the LilyPond editor and the Git interface (command line or GUI) quite often, but this has become a second nature. Depending on the type of work it is often possible to do some amount of work and then commit it gradually. Command line Git offers git add -p for this, GUI tools have other ways of doing it.
      Of course it would be very nice if exactly this was possible from within Frescobaldi…

      Reply
      1. Brent

        There is a great tutorial on Lynda.com about Git that is making it all clear to me. Can’t wait to start using it with lilypond/Frescobaldi.

        Reply
  3. Lars Haulin

    I could not agree more! Git/lilypond has been my standard way of typesetting for a while now.

    I was getting this page as a search result when I was looking for a way to embed the git revision in the lilypond score automatically. If someone else also looks for how that can be done, here is the solution I came up with, popen/read on a git log command. The example was made for linux, and might need some tweaking for other operating systems.

    #(use-modules (ice-9 popen))
    #(use-modules (ice-9 rdelim))
    
    % NOTE: This function only reads the first line of the command!
    #(define (strsystem_internal cmd)
       (let* ((port (open-input-pipe cmd))
              (str (read-line port)))
         (close-pipe port)
         str))
    
    #(define-markup-command (strsystem layout props cmd) (markup?)
       (interpret-markup layout props
         (strsystem_internal cmd)))
    
    gitver = \markup { \strsystem "git log --oneline | wc -l" }
    gitrev = \markup { \strsystem "git rev-parse --short HEAD" }
    
    Reply
      1. Urs Liska

        OK, I found out (embarrassing…): WordPress automatically created typographical quotes. I now fixed that in the example so anybody can copy & paste the example.

        Reply

Leave a Reply

Your email address will not be published. Required fields are marked *