The Future of openLilyLib

Some time ago Matteo Ceccarello appeared on the LilyPond stage and announced a new library that merged several ideas from this blog. My own posts about a grid approach and two posts by Jan-Peter Voigt had inspired him to develop a library where the “cells” of the grid can be directly stored in a tree-like Scheme object. I helped him integrating this library in openLilyLib where it now lives as GridLY. This process and collaboration produced some significant by-products, for example an automated testing infrastructure for openLilyLib. And now Matteo presented a new tool that opens up some important perspectives for openLilyLib.

In a post on his blog Matteo describes some of the aspects of openLilyLib undergoing significant infrastructure modifications. One seemingly secondary aspect of this is that openLilyLib now has a notion of versions. openLilyLib itself as well as the contained libraries can now have specific versions, and there is already a kind of support for tagging a score with these versions. I won’t give a summary of Matteo’s post and encourage you to read it for yourself instead – ideally you’d do that now and return later to complete my post.

I think Matteo put his finger on an inherent problem that would inevitably show up in a quite ugly manner if openLilyLib would become “mature” and even started “aging”. And the script he presented in his post is an interesting approach which – despite being not sufficient on the long run, as he states himself – will boost our development as an inspiration.

In addition to managing the included openLilyLib version I would add another problem that is waiting for a long-term solution: Sometimes changes in openLilyLib or a library are “breaking”, that is they require the LilyPond files that use the library to update their input syntax. I think we need something like LilyPond’s own convert-ly for openLilyLib as well, either as a standalone tool or by somehow ”hijacking” convert-ly.

In his closing remarks Matteo points out that in order to manage versions of the libraries contained in openLilyLib independently (say, GridLY 0.6 and ScholarLY 0.5) a package manager would be needed. I have to say this idea comes at the right moment – as I have just started to think in a similar direction recently. The issue that bothered me is that the new infrastructure in openLilyLib seems at risk of becoming unmaintainable quite soon. As it stands we have a directory structure similar to

├── comptools
│   ├── partial-compilation
│   ├── unit-tests
│   └── usage-examples
├── gridly
│   └── usage-examples
├── _internal
│   └── utilities
├── scholarly
│   ├── annotate
│   │   └── examples
│   └── usage-examples
├── stylesheets
│   ├── fonts
│   └── usage-examples
├── tablature
    └── usage-examples

This looks quite nice with currently five libraries and an _internal core being present. But when we expect openLilyLib to become as comprehensive as it should there will presumably be dozens of libraries, maybe even heading towards a three-digit number. This will have a number of implications:

  • The main directory and Git repository will become quite crowded and big. Which isn’t a problem in itself as any average software project will still be bigger, but:
  • Any user will have to fetch the whole repository even when she’s only interested in a certain library.
  • The openLilyLib project will have to manage a significant number of project members and “sub-maintainers”, there’s not a natural separation of responsibilities between maintainers of different libraries.
  • This will also bloat the issue tracker, and if only by having to add dozens of labels for the individual libraries.

All those points indicate that openLilyLib should be disentangled and the individual libraries be developed in their individual repositories. One main concern with such an undertaking would be to manage “installations”, paths and versions among the different libraries. And I think that a package manager would be exactly what we need for that. Apart from keeping track of versions and dependencies such a tool would also be useful to actually deploy the libraries. Currently users are encouraged to clone the repository using Git – which is quite an obstacle for someone not using Git already. The alternative of using the archive downloads provided by Github aren’t really acceptable either.

By pure coincidence I was recently approached by a LilyPond user who is developing several promising tools with and for LilyPond (he’ll tell us about them when they’re sufficiently ready). Among the topics that came up besides these ideas was – a package manager, something he has already been experimenting with lately!

So I encourage anyone who is interested in the future of openLilyLib (and to some extent also of LilyPond itself) to join us with discussion, testing and maybe also actual contributions. openLilyLib was originally conceived as a community project, and I think it can really become an important part of the LilyPond ecosystem once the mentioned restructuring has been completed. But it is definitely a task that is too big for one or two developers.

2 thoughts on “The Future of openLilyLib

  1. David Kastrup

    Well, we likely don’t really want to give birth to yet-another-package-manager to maintain. Two obvious candidates to look at are the Guix package manager (as it is written in Guile and a GNU project) and the TeXlive package manager (as it appears to address almost exactly the same problem, a library tree with possible dependencies).

    Guix has the distinct advantage of creating individually resolved trees which means that if a particular document depends on particular versions of libraries, you can get a matching tree (and one could actually involve the LilyPond binary in such a snapshot as well).

    It has the distinct disadvantage of, uhm, working with symbolic links. I don’t think that it currently has a roadmap involving Windows and it would likely require discussions and work with its developers to make it useful for Windows as well.

    1. Urs Liska

      We are currently having some internal discussions about possible approaches, either piggy-backing existing package managers (like you suggest) or a more light-weight solution that lets a client machine fetch packages and do the dependency resolution within the LilyPond compilation.

      As soon as we are ready we’ll propose an outline to the community – before starting to implement anything of course.


Leave a Reply

Your email address will not be published. Required fields are marked *