What Timbre stands on

Standing on the shoulders of giants.

A 12th-century saying, after Bernard of Chartres; made famous by Isaac Newton (1675).

Timbre measures and estimates what makes a track. That only works because researchers shared their methods and models openly – first of all the Music Technology Group at Universitat Pompeu Fabra, Barcelona. Here’s what comes from where and how to cite it.

What comes from where

Five measurements, three stages each: the research, the model it produced and what Timbre does with it.

Genre mix
ResearchISMIR 2022 · MTG
ModelDiscogs-EffNet – 400 Discogs styles
In TimbreShares in percent, timeline across the track, genre sliders
Vocals and danceability
ResearchICASSP 2020 · MTG
Modelvoice/instrumental, danceability
In TimbreVocal lane with sections and chops, danceability in the sound profile
Key
ResearchECIR 2016 · MTG · GiantSteps
ModelEssentia Key, EDM profile
In TimbreOpen Key or your notation, “fits with”, check against your tags
Mood and tags
ResearchICML 2019 · MTG
ModelMTG-Jamendo heads
In TimbreMood, instruments and tags as model labels
Valence and arousal
ResearchISMIR 2019 · ISMIR 2022 · MTG
ModelMusiCNN with valence and arousal heads
In TimbreInputs for the energy model (o to +++)

No model, straight from the signal: loudness to EBU R128, frequency bands, tempo cross-check – with FFmpeg, librosa, NumPy, SciPy and our own code.

All Timbre: energy model, mix octagon, record bag, catalog.

What Timbre builds itself

Energy in four levels
Our own model, learned from Timbre’s measurements and the estimates of the MTG models. Shown as o, +, ++, +++.
Genre shares
The 400 styles of Discogs-EffNet become shares in percent, a lead genre and a timeline across the track.
Mix octagon and record bag
Eight ways out of the current track, an order for your set, export to your DJ software – key in your notation.
Frequency bands and vocal lane
Levels in six bands like on the mixer; voice detection becomes sections, chops and “unsure”.
Catalog
A track is measured once and recognized by fingerprint – for everyone, without storing music.

Papers to read

Eight publications Timbre’s measurements rest on. Titles in the original, BibTeX ready to copy.

Paper2022MTG · Universitat Pompeu Fabra

Music Representation Learning Based on Editorial Metadata from Discogs

P. Alonso-Jiménez, X. Serra, D. Bogdanov

In Timbre: Genre mix (400 Discogs styles) and base for most model heads
Published: ISMIR 2022, Bengaluru, pp. 825–833
Link: doi.org/10.5281/zenodo.7316790 essentia.upf.edu/models.html

BibTeX

@inproceedings{alonso2022music,
  title={Music Representation Learning Based on Editorial Metadata from Discogs},
  author={Alonso-Jim{\'e}nez, Pablo and Serra, Xavier and Bogdanov, Dmitry},
  booktitle={Conference of the International Society for Music Information Retrieval (ISMIR)},
  pages={825--833},
  doi={10.5281/zenodo.7316790},
  year={2022}
}

Paper2016MTG · Universitat Pompeu Fabra
Key Estimation in Electronic Dance Music
Á. Faraldo, E. Gómez, S. Jordà, P. Herrera

In Timbre
Key (profile “edmm”), shown in your notation
Published
ECIR 2016, Padua · Springer, pp. 335–347 · GiantSteps (EU FP7)
Link
doi.org/10.1007/978-3-319-30671-1_25 essentia.upf.edu/reference/std_Key.html
BibTeX
```
@inproceedings{faraldo2016key,
  title={Key Estimation in Electronic Dance Music},
  author={Faraldo, {\'A}ngel and G{\'o}mez, Emilia and Jord{\`a}, Sergi and Herrera, Perfecto},
  booktitle={Advances in Information Retrieval (ECIR 2016)},
  pages={335--347},
  publisher={Springer},
  doi={10.1007/978-3-319-30671-1_25},
  year={2016}
}
```

Paper2020MTG · Universitat Pompeu Fabra

TensorFlow Audio Models in Essentia

P. Alonso-Jiménez, D. Bogdanov, J. Pons, X. Serra

In Timbre: Vocals, danceability, timbre and more heads
Published: IEEE ICASSP 2020
Link: essentia.upf.edu/models.html

BibTeX

@inproceedings{alonso2020tensorflow,
  title={Tensorflow Audio Models in Essentia},
  author={Alonso-Jim{\'e}nez, Pablo and Bogdanov, Dmitry and Pons, Jordi and Serra, Xavier},
  booktitle={IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  year={2020}
}

Paper2019MTG · Universitat Pompeu Fabra

The MTG-Jamendo Dataset for Automatic Music Tagging

D. Bogdanov, M. Won, P. Tovstogan, A. Porter, X. Serra

In Timbre: Mood, instruments, tags
Published: ICML 2019, Machine Learning for Music Discovery Workshop, Long Beach
Link: github.com/MTG/mtg-jamendo-dataset

BibTeX

@conference{bogdanov2019mtg,
  title={The MTG-Jamendo Dataset for Automatic Music Tagging},
  author={Bogdanov, Dmitry and Won, Minz and Tovstogan, Philip and Porter, Alastair and Serra, Xavier},
  booktitle={Machine Learning for Music Discovery Workshop, International Conference on Machine Learning (ICML 2019)},
  address={Long Beach, CA, United States},
  year={2019}
}

Paper2019MTG · Universitat Pompeu Fabra
musicnn: Pre-trained convolutional neural networks for music audio tagging
J. Pons, X. Serra

In Timbre
Base for valence and arousal
Published
ISMIR 2019, Late-Breaking Demo
Link
arxiv.org/abs/1909.06654 github.com/jordipons/musicnn
BibTeX
```
@article{pons2019musicnn,
  title={musicnn: Pre-trained convolutional neural networks for music audio tagging},
  author={Pons, Jordi and Serra, Xavier},
  journal={arXiv preprint arXiv:1909.06654},
  year={2019}
}
```

Paper2022MTG · Universitat Pompeu Fabra

MusAV: A dataset of relative arousal-valence annotations for validation of audio models

D. Bogdanov, X. Lizarraga-Seijas, P. Alonso-Jiménez, X. Serra

In Timbre: Valence and arousal, inputs for the energy model
Published: ISMIR 2022, Bengaluru, pp. 650–658
Link: doi.org/10.5281/zenodo.7316746 essentia.upf.edu/models.html

BibTeX

@inproceedings{bogdanov2022musav,
  title={MUSAV: a dataset of relative arousal-valence annotations for validation of audio models},
  author={Bogdanov, Dmitry and Lizarraga Seijas, Xavier and Alonso-Jim{\'e}nez, Pablo and Serra, Xavier},
  booktitle={Conference of the International Society for Music Information Retrieval (ISMIR)},
  pages={650--658},
  doi={10.5281/zenodo.7316746},
  year={2022}
}

Paper2013MTG · Universitat Pompeu Fabra

Essentia: an Audio Analysis Library for Music Information Retrieval

D. Bogdanov, N. Wack, E. Gómez, S. Gulati, P. Herrera, O. Mayor, G. Roma, J. Salamon, J. Zapata, X. Serra

In Timbre: Core: key, loading and running the models
Published: ISMIR 2013, Curitiba, pp. 493–498
Link: archives.ismir.net/ismir2013/paper/000177.pdf essentia.upf.edu

BibTeX

@inproceedings{bogdanov2013essentia,
  title={Essentia: an Audio Analysis Library for Music Information Retrieval},
  author={Bogdanov, Dmitry and Wack, Nicolas and G{\'o}mez, Emilia and Gulati, Sankalp and Herrera, Perfecto and Mayor, Oscar and Roma, Gerard and Salamon, Justin and Zapata, Jos{\'e} and Serra, Xavier},
  booktitle={International Society for Music Information Retrieval Conference (ISMIR)},
  pages={493--498},
  year={2013}
}

Paper2015Open Source

librosa: Audio and Music Signal Analysis in Python

B. McFee, C. Raffel, D. Liang, D. P. W. Ellis, M. McVicar, E. Battenberg, O. Nieto

In Timbre: Signal measurements, tempo cross-check
Published: SciPy 2015, Austin, pp. 18–25
Link: doi.org/10.25080/Majora-7b98e3ed-003 librosa.org

BibTeX

@inproceedings{mcfee2015librosa,
  title={librosa: Audio and Music Signal Analysis in Python},
  author={McFee, Brian and Raffel, Colin and Liang, Dawen and Ellis, Daniel P. W. and McVicar, Matt and Battenberg, Eric and Nieto, Oriol},
  booktitle={Proceedings of the 14th Python in Science Conference (SciPy)},
  pages={18--25},
  doi={10.25080/Majora-7b98e3ed-003},
  year={2015}
}

Models and license

Timbre estimates genre mix, vocals, mood, valence and arousal with pre-trained models from the Music Technology Group. Origin and license below.

Created by: Music Technology Group, Universitat Pompeu Fabra
Models: discogs-effnet-bs64-1 · msd-musicnn-1 · approachability_2c-discogs-effnet-1 · danceability-discogs-effnet-1 · engagement_2c-discogs-effnet-1 · gender-discogs-effnet-1 · mood_acoustic-discogs-effnet-1 · mood_party-discogs-effnet-1 · mtg_jamendo_genre-discogs-effnet-1 · mtg_jamendo_instrument-discogs-effnet-1 · mtg_jamendo_moodtheme-discogs-effnet-1 · mtg_jamendo_top50tags-discogs-effnet-1 · timbre-discogs-effnet-1 · tonal_atonal-discogs-effnet-1 · voice_instrumental-discogs-effnet-1 · deam-msd-musicnn-2 · emomusic-msd-musicnn-2 · muse-msd-musicnn-2
License: Per the model page CC BY-NC-SA 4.0, per the license file in the model folder CC BY-NC-ND 4.0 – the two MTG statements contradict each other.
Source: essentia.upf.edu/models.html essentia.upf.edu/models/LICENSE
Changes: None: we use the models unmodified. Timbre trains its own small energy model on their outputs.

AGPL-3.0 applies to Essentia’s code, not to these models.

Open-source software

No paper, just as important: libraries Timbre computes with every day.

Chromaprint
Acoustic fingerprint that lets the catalog recognize tracks.
FFmpeg
Reads almost every audio format and measures loudness to EBU R128.
NumPy
Number crunching on the audio data – the base of almost every measurement.
SciPy
Filters for the frequency bands, like on the mixer.
scikit-learn
The learning method behind Timbre’s energy model.

Thank you

To everyone who spent years building and sharing methods, datasets and models. Someone missing, or a detail wrong? Write to us. hello@timbremusic.app

Write to us

Timbre is an independent project and is not affiliated with the Music Technology Group or Universitat Pompeu Fabra. We name names, but show no logos.

Standing on the shoulders of giants.

What comes from where

Genre mix

Vocals and danceability

Key

Mood and tags

Valence and arousal

What Timbre builds itself

Energy in four levels

Genre shares

Mix octagon and record bag

Frequency bands and vocal lane

Catalog

Papers to read

Music Representation Learning Based on Editorial Metadata from Discogs

Key Estimation in Electronic Dance Music

TensorFlow Audio Models in Essentia

The MTG-Jamendo Dataset for Automatic Music Tagging

musicnn: Pre-trained convolutional neural networks for music audio tagging

MusAV: A dataset of relative arousal-valence annotations for validation of audio models

Essentia: an Audio Analysis Library for Music Information Retrieval

librosa: Audio and Music Signal Analysis in Python

Models and license

Open-source software

Chromaprint

FFmpeg

NumPy

SciPy

scikit-learn

Thank you