This is a README file for a data repository originating from the DCML corpus initiative
and serves as welcome page for both

• the GitHub repo https://github.com/DCMLab/ABC and the corresponding
• documentation page https://dcmlab.github.io/ABC

For information on how to obtain and use the dataset, please refer to this documentation page.

The Annotated Beethoven Corpus (ABC) (A corpus of annotated scores)

The ABC dataset consists of expert harmonic analyses of all Beethoven string quartets
(opp. 18, 59, 74, 95, 127, 130, 131, 132, 135, composed between 1800 and 1826), encoded in a human- and
machine-readable format (MuseScore format).
Using a modified Roman Numeral notation (the DCML harmony annotation standard),
the dataset includes the common music-theoretical set of harmonic features such as key, chordal root,
chord inversion, chord extensions, suspensions, and others.

Getting the data

• download repository as a ZIP file
• download a Frictionless Datapackage that includes concatenations
  of the TSV files in the four folders (measures, notes, chords, and harmonies) and a JSON descriptor:
  • ABC.zip
  • ABC.datapackage.json
• clone the repo: git clone https://github.com/DCMLab/ABC.git

Data Formats

Each piece in this corpus is represented by five files with identical name prefixes, each in its own folder.
For example, the first movement of the first quartet, op. 18/1, has the following files:

• MS3/n01op18-1_01.mscx: Uncompressed MuseScore 3.6.2 file including the music and annotation labels.
• notes/n01op18-1_01.notes.tsv: A table of all note heads contained in the score and their relevant features (not each of them represents an onset, some are tied together)
• measures/n01op18-1_01.measures.tsv: A table with relevant information about the measures in the score.
• chords/n01op18-1_01.chords.tsv: A table containing layer-wise unique onset positions with the musical markup (such as dynamics, articulation, lyrics, figured bass, etc.).
• harmonies/n01op18-1_01.harmonies.tsv: A table of the included harmony labels (including cadences and phrases) with their positions in the score.

Each TSV file comes with its own JSON descriptor that describes the meanings and datatypes of the columns ("fields") it contains,
follows the Frictionless specification,
and can be used to validate and correctly load the described file.

Opening Scores

After navigating to your local copy, you can open the scores in the folder MS3 with the free and open source score
editor MuseScore. Please note that the scores have been edited, annotated and tested with
MuseScore 3.6.2.
MuseScore 4 has since been released which renders them correctly but cannot store them back in the same format.

Opening TSV files in a spreadsheet

Tab-separated value (TSV) files are like Comma-separated value (CSV) files and can be opened with most modern text
editors. However, for correctly displaying the columns, you might want to use a spreadsheet or an addon for your
favourite text editor. When you use a spreadsheet such as Excel, it might annoy you by interpreting fractions as
dates. This can be circumvented by using Data --> From Text/CSV or the free alternative
LibreOffice Calc. Other than that, TSV data can be loaded with
every modern programming language.

Loading TSV files in Python

Since the TSV files contain null values, lists, fractions, and numbers that are to be treated as strings, you may want
to use this code to load any TSV files related to this repository (provided you're doing it in Python). After a quick
pip install -U ms3 (requires Python 3.10 or later) you'll be able to load any TSV like this:

import ms3

labels = ms3.load_tsv("harmonies/n01op18-1_01.harmonies.tsv")
notes = ms3.load_tsv("notes/n01op18-1_01.notes.tsv")

Version history

See the GitHub releases.

Questions, Suggestions, Corrections, Bug Reports

Please create an issue and/or feel free to fork and submit pull requests.

Publications

• The accompanying Data Report has been published in Neuwirth, M., Harasim, D., Moss, F. & Rohrmeier M. (2018).
• An evaluation of the dataset can be found in this Moss, F., Neuwirth M., Harasim, D. & Rohrmeier, M. (2019).
• The latest version of the annotation standard has been described in Hentschel, J., Neuwirth, M. & Rohrmeier, M. (2021)

Cite as

Neuwirth, M., Harasim, D., Moss, F. C., & Rohrmeier, M. (2018). The Annotated Beethoven Corpus (ABC): A Dataset of Harmonic Analyses of All Beethoven String Quartets. Frontiers in Digital Humanities, 5(July), 1–5. https://doi.org/10.3389/fdigh.2018.00016

License

Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0).

fixes #29

No other metadata fields were corrected.

Added chords facet

Not to be confounded with labels or chord annotations, a chord is a notational unit in which all included notes are part of the same notational layer and have the same onset and duration. Every chord has a chord_id and every note is part of a chord. These tables are used to convey score information that is not attached to a particular note, such as lyrics, staff text, dynamics and other markup. More information on the columns can be found in the documentation of the ms3 parser.

Added pre-commit hook

The new iteration of the DCML annotation workflow moves the automatic validation of scores and annotations from GitHub actions to the local user. Whenever you are planning to commit changes to the annotation labels, you may want to install the pre-commit package to then run pre-commit install in your local clone. From this moment on, each time you try to commit changes to one or several MuseScore files, the hook will run ms3 review on them and will reject the commit if warnings are detected. If no warnings are detected, the commit will be enriched with all updates to the repo's TSV files that reflect the changes you've made to the scores.

Changes to the TSV facets

Updated filenames

The TSV files have been renamed and now come with a suffix specifying the data facet that they represent. The three facets are

• measures
• notes
• harmonies

so, for example, the file measures/n01op18-1_01.tsv has been renamed to measures/n01op18-1_01.measures.tsv.

Added resource descriptors

Each TSV file is now accompanied with a JSON resource descriptor following the frictionless specification for Tabular Data Resources. They can and have be used to validate the tabular data.

These files have the same names as the TSV files they describe but replacing the .tsv extension with .resource.json. For example, the file measures/n01op18-1_01.tsv has the corresponding descriptor measures/n01op18-1_01.resource.json.

These metadata files replace the previous csv-metadata.json files (which followed the CSVW standard) that had described the data inadequately.

Added column quarterbeats_all_endings

The quarterbeats column (also known as qstamp) that had already been available does not provide values for first or third endings, expressing the dimensions of a singular playthrough without taking into account any repeats. The newly added column quarterbeats_all_endings does not have any empty values, providing continuous positions as if each measure followed the previous one.

Renamed index column

The first column of metadata.tsv was renamed from fname to piece.

Changes to the reviewed folder

The _reviewed.mscx files compare the current set of annotation labels against those from the previous version (v2.1). They are identical, so no differences are displayed.

Wherever the MuseScore parser throws one or several warnings (e.g. because an annotation label does not match the score very well), these warnings appear in a .warnings file. For example, the warnings for n01op18-1_01 can be seen in reviewed/n01op18-1_01.warnings. One goal for future versions of this dataset would be to reduce the number of warnings to the point of addressing all of them.

Frictionless datapackage

This release is the first one that has a frictionless datapackage attached (below). It consists of

• abc.zip, a ZIP file containing one TSV file per facet, that corresponds to a concatenation of the TSV files in the respective folder, that is
  • abc.expanded.tsv
  • abc.measures.tsv
  • abc.metadata.tsv (concatenation of a single file)
  • abc.notes.tsv
• abc.datapackage.json, the package descriptor.

If one has the frictionless framework installed, one can use the descriptor to validate the package using the command

frictionless validate abc.datapackage.json

Update v2.1: This release is identical to 2.0 with the difference that the .zenodo.json file (that had been generated by Zenodo!) has been freed from the key related_identifiers which made the Zenodo validator fail.

4.5 years after its first publication (see below), this is the first revised version of the ABC.
In the meantime, the DCML corpus initiative has advanced
and this update has as a main goal to harmonize the ABC with all other annotated corpora that have been and
will be published. This includes the following changes:

Upgrade to MuseScore 3

• All scores have been converted to MuseScore 3.6.2 format and can be found in the folder MS3.
• The harmony labels have been moved to MuseScore's "Roman Numeral Analysis" layer of the left-hand staff.

New folder and file structure

• The code folder was removed since the old Julia code has been replaced by the Python library ms3.
• The MuseScore files are contained in MS3 and for each movement there are a couple of other files available, identified by their file names:
  • The folder notes contains one TSV file per movement with all note heads (not every note head represents an onset).
  • The folder measures contains one TSV file per movement with all measure-like units
  • The folder harmonies contains one TSV file per movement with all harmony annotation labels
  • The folder reviewed contains two files per movement:
    • A copy of the score where all out-of-label notes have been colored in red; additionally, modified labels ( w.r.t. v1.0) are shown in these files in a diff-like manner (removed in red, added in green).
    • A copy of the harmonies TSV with six added columns that reflect the coloring of out-of-label notes ("coloring reports")
  • The file warnings.log lists those labels where over 60 % of notes within the label's segment are not expressed
    by the label. Potentially, most of them are semantically incorrect.

The folders are automatically kept up to date by the dcml_corpus_workflow
which calls the command ms3 review -M -N -X -D on every change.

Information on what the columns in the TSV files contain can be found in the documentation for ms3.

Changes to the data

A full diff of all changes applied with version 2.0 can be seen here.

• The scores have been aligned by tunescribers.com with the Henle and Breitkopf editions
  provided in the pdf folder and indicated in its README.
• Systematic changes to the harmony labels:
  • With the harmony labels moved to the Roman Numeral Analysis layer, no initial . are needed anymore.
  • V9 is not part of the DCML harmony annotation standard and has been replaced by V7(9) or V7(+9).
  • Corrected vii chords in major keys that had often been wrongly labeled as #vii.
• Obvious errors have been corrected in many places. Thanks to @craigsapp, @lancioni, @malcolmsailor, @MarkGotham, @napulen and @tymoczko
  for reporting quite a few of them!

4.5 years after its first publication (see below), this is the first revised version of the ABC.
In the meantime, the DCML corpus initiative has advanced
and this update has as a main goal to harmonize the ABC with all other annotated corpora that have been and
will be published. This includes the following changes:

Upgrade to MuseScore 3

• All scores have been converted to MuseScore 3.6.2 format and can be found in the folder MS3.
• The harmony labels have been moved to MuseScore's "Roman Numeral Analysis" layer of the left-hand staff.

New folder and file structure

• The code folder was removed since the old Julia code has been replaced by the Python library ms3.
• The MuseScore files are contained in MS3 and for each movement there are a couple of other files available, identified by their file names:
  • The folder notes contains one TSV file per movement with all note heads (not every note head represents an onset).
  • The folder measures contains one TSV file per movement with all measure-like units
  • The folder harmonies contains one TSV file per movement with all harmony annotation labels
  • The folder reviewed contains two files per movement:
    • A copy of the score where all out-of-label notes have been colored in red; additionally, modified labels ( w.r.t. v1.0) are shown in these files in a diff-like manner (removed in red, added in green).
    • A copy of the harmonies TSV with six added columns that reflect the coloring of out-of-label notes ("coloring reports")
  • The file warnings.log lists those labels where over 60 % of notes within the label's segment are not expressed
    by the label. Potentially, most of them are semantically incorrect.

The folders are automatically kept up to date by the dcml_corpus_workflow
which calls the command ms3 review -M -N -X -D on every change.

Information on what the columns in the TSV files contain can be found in the documentation for ms3.

Changes to the data

A full diff of all changes applied with version 2.0 can be seen here.

• The scores have been aligned by tunescribers.com with the Henle and Breitkopf editions
  provided in the pdf folder and indicated in its README.
• Systematic changes to the harmony labels:
  • With the harmony labels moved to the Roman Numeral Analysis layer, no initial . are needed anymore.
  • V9 is not part of the DCML harmony annotation standard and has been replaced by V7(9) or V7(+9).
  • Corrected vii chords in major keys that had often been wrongly labeled as #vii.
• Obvious errors have been corrected in many places. Thanks to @craigsapp, @lancioni, @malcolmsailor, @MarkGotham, @napulen and @tymoczko
  for reporting quite a few of them!

stable release v1.0