Computer singing synthesis

The aim of the Singing Computer project is to make another step towards the accessibility of music typesetting to the visually impaired users by introducing the possibility to check the input of lyrics. There is a nice and popular music typesetting tool called LilyPond that is based on a plain text input (similarly as e.g. in the TeX typesetting system). This way of work allows the visually impaired users to input music relatively comfortably in their text editors.

An important part of the music typesetting work is checking the result. This is easy for sighted users who can simply check the output on their screens or printed on the paper. But visually impaired users cannot do that. Even if the accessibility tools were able to describe the positions of notes and lyrics on the screen, checking it in such a way would be very tedious. That means that the visually impaired users have to use other tools.

While it is difficult for the visually impaired users to work with visually represented information, it is usually easy for them to work with information represented in the form of sound. And this is how the music typesetting tools can be customized. LilyPond already solves part of the problem — it offers MIDI output of the music. Listening to it the visually impaired user can check correctness of the music part. But what is still missing is a way to check that the lyrics was input correctly as well. This is what Singing Computer solves.

Singing Computer allows, with the help of the Festival speech synthesizer, to make the computer to sing the lyrics written in the LilyPond input file. This way a visually impaired user can check the lyrics and its proper alignment with the notes. The user can detect and correct mistakes made during writing the LilyPond input file. With the help of this tool the music typesetting accessibility becomes almost complete.

The Singing Computer project is developed by Brailcom, o.p.s. and was financially supported by Nadační fond Českého rozhlasu within the project Světluška.

Singing Computer consists of three basic parts:

  • Extraction of the lyrics and its corresponding notes from the LilyPond input file and storing them in the form of an XML file that can be processed by the Festival speech synthesizer.
  • Synthesis of the lyrics in the speech synthesizer.
  • Emacs support.

The first, LilyPond, part works as an independent LilyPond extension introducing new LilyPond command. By using that command the user can direct LilyPond to output the XML input file for Festival. Making this file is from the user's point of view similar to making a MIDI file in LilyPond.

The second, Festival, part responsible for the sound output from the generated XML file works by utilizing the experimental singing module from the Festival distribution. This module has been being extended and improved in the Singing Computer project. This part does not depend on LilyPond, it can be used anywhere where song synthesis is useful. Czech song synthesis is made with the help of the festival-czech a Czech Diphone Database projects.

The third, Emacs, part helps the user to invoke singing, possibly with accompanying MIDI play, during his LilyPond source file editing directly from the editor.

All the parts are implemented as LilyPond and Festival extensions without the necessity to patch the original software. That means they can be used with the standard LilyPond and Festival packages installed in any common software distribution.

The source code is managed using Git. You can use the Git web interface or clone the repository from:

https://github.com/brailcom/singing-computer.git