Throughout the lifetime of this blog we have been propagating a certain perspective on music engraving and music editing. Its idea is to take advantage of methods, tools and workflows from software development and computer science. Agile software development is one of today’s strong tendencies in software development, sharing important parts of our endeavors and ideas. So why not give it a shot and draw from it, working towards agile music edition?
There have been months since the last post, which is unprecedented. Over two years we have published one post a week as an average. In a way this is the reason for the long gap – as what you are reading right now is actually the 100th post on Scores of Beauty 🙂 . The post I had written for this occasion has been pending for long but was blocked on something that has yet to be announced, so we’ve ended up not publishing anything lately. The situation got more and more pressing, also because we have suggestions from guest authors again, and so finally this post formed in my mind.
I had taken The Small Agile Book (seems to be German only) with me for holidays. Originally I intended to get some background about a contemporary development technique buzz-phrase (to compensate for my lack of formal CS education). But when reading the (recommendable) volume it struck me that agile has a lot in common with my ideas of “score development”. The most obvious affinity is the idea of a self-organizing team with shared responsibilities and getting away from the Waterfall development model.
The “Waterfall” Development Model
In traditional software development processes the lifecycle of a project is very strictly organized in consecutive stages as shown in the above visualization. There are discrete responsibilities for these stages, and larger companies even have dedicated departments for each of them. Music edition often is organized in a similar way, with publishing houses having established strict workflows that strikingly resemble that waterfall. There is a sequence of consecutive steps that could be visualized in a similar manner, for example: critical review (main editor) ⇒ music entry ⇒ review (house’s editor) ⇒ proof-reading ⇒ engraving’s beautification ⇒ prepress.
Usually these steps are assigned to separate people with strict separation of responsibilities. But even more important is that – like in the Waterfall model – the steps have to be processed sequentially, that is, any step has to be finished completely before the next stage can be entered.
In software development this Waterfall approach has proven to create significant problems, and I have more than once discussed the specific issues that arise (in music editing) from having to pass opaque binary files around to process them sequentially. Agile development has come up with numerous offerings to overcome them, and our ideas for collaborative music edition workflows (as can for example be seen in our crowd editing project or the posts tagged with version control) seem to be heading in a quite similar direction.
I think this explains to some extent the reservations traditional publishers (and editors) have with our approach. They presumably experience the same sense of missing security and “predictability” that decision makers in software companies have or had with regard to agile ideas. Realizing this might be a first step to also develop new argumentation strategies for us.
The Multi Disciplinary and Self-organizing Team
One central concept of agile is the multi disciplinary and self-organizing team as opposed to specialized teams (e.g. analysts, architects, coders, testers etc.). In agile teams roles can be assigned with much more flexibility, enabling direct communcation and short working cycles. This is quite intuitively applicable to LilyPond based edition projects where in principle everybody can do anything. There are specializations, for example some people will have responsibility for scholarly issues while others have a sharper eye regarding engraving decisions. But still they can seamlessly collaborate on the same “thing” (i.e. the code repository).
Another central concept of agile is the notion of adaptive planning. While planning is of course an important factor it doesn’t have to be done completely up-front. Instead a project is organized in short iterations that are each finished off with a retrospective. This essential step is used to plan the next iteration but also to evaluate and improve the global plan and details of the workflow.
In a typical collaborative engraving project this happens too, basically all the time. But I think the problem is that it happens in a more or less random fashion. This probably spoils a significant portion of the potential that consciously applied agile methods provide.
An agile project should try to deliver results as early as possible and only then refine and extend the product continuously. This is done by constant prioritisation so that basically the most important parts are developed first (and it’s constantly reconsidered what the most important parts are). The idea is that this will be more satisfying for the “customer” who gets tangible results at early stages, and also for the team for the same reasons. From a certain moment onwards the project could be stopped and deployed at any time – for example when budget or deadline have been reached – but still deliver a usable product.
This idea can be applied to music edition quite well, as the only prerequisite for delivering a score is the plain music entry. Everything else – critical review, fixing page layout and engraving decisions (at least when using LilyPond with its exceptional default engraving quality) – can be considered as “only” refining an already usable product. Of course you wouldn’t do without any of it in a printed critical edition but there are many use cases (especially performance materials) where it may make a huge difference. An agile music edition project would place the music entry with highest priority in the project backlog and everything else later. If other work items turn out to be critical at any point (e.g. writing new functions to handle lyrics or part combination or whatever) they can easily be inserted in the backlog (see “adaptive planning” above).
Kanban: Optimizing Processes
One thing I realized would have really helped us when producing “Das trunkne Lied”: optimizing workflows with Kanban (or any comparable method). Kanban is a method to streamline the use of resources in production chains, originally developed for car manufacturers. Basically it does so by (constantly) analyzing the process to identify and remove bottleneck situations, typically by comparing the number of work items that pass each stage within any given project iteration. It is a fascinating concept of deferring the control of operations to those who actually do the work, relieving the project as a whole from a lot of controlling overhead.
Having a team with shared responsibilities is already a good foundation as it allows any team member to do “what has to be done” at any given moment. Nevertheless, we often ran into the situation that there was too little or too much work available at the moment. Time will be used most efficiently when the “work items” flow as continuously as possible through their stages (e.g. music entry, proof-reading, revision). Tasks such as preparation of part templates, handling combined parts or general programming tasks are also part of that equation. While organizing this stream of work will work out more or less smoothly by itself it will definitely benefit from having proven concepts such as Kanban applied.
Where Are We Today?
Touching briefly on the subject indicates that there are significant correlations between agile software development and the potential of music edition workflows as described on Scores of Beauty. However, it becomes clear that there is much more to be gained by a conscious application of agile principles. Obviously some ideas have only been touched upon or rather “happen” randomly, others haven’t even been thought of, e.g. the importance of direct and regular (structured) personal communication or the constant measuring of progress. For example visualizing a project with a burndown chart (or better its counterpart, the burnup chart) is something that strikes me as a worthwile idea. Implementing the idea of story points as a measurement unit for work achieved or to be done will also be very helpful.
In other cases the agile concepts can’t so easily be applied to music edition. I have no clear idea how pair programming, unit testing or user stories could be mapped to our workflows. Of course it’s no good to follow agile by the letter, but I have the suspicion it would be fruitful to at least think about these aspects.
This post is definitely a sketch, I neither give a report of success nor elaborate on concrete plans. However, I would be really happy if someone or we as a community would explore this field more thoroughly. It’s one of the characteristics of working with LilyPond (and its related tools) that you can approach projects like a software developer. So why not take the idea of integrating software development techniques into music engraving a step further and create agile principles of music edition?