Credits
Audio Magic is a browser wrapper around other people’s research. The speech recognition, noise suppression and translation models below were built and published by the researchers and engineers named here, and released under licences that let anyone use them. This page is where they get the credit, in the citation format each project asks for.
Speech recognition
Transcription runs OpenAI’s Whisper, a transformer trained on 680,000 hours of multilingual audio.
Whisper
MITTranscribing speech to text and subtitles
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. Robust Speech Recognition via Large-Scale Weak Supervision. arXiv:2212.04356.BibTeX
@article{radford2022whisper, title = {Robust Speech Recognition via Large-Scale Weak Supervision}, author = {Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya}, journal = {arXiv preprint arXiv:2212.04356}, year = {2022} }
Speech enhancement
Both noise-removal engines pair a neural network with classical signal processing — which is what makes them fast enough to run on your own machine.
RNNoise
BSD-3-ClauseFast background noise removal
Jean-Marc Valin. 2018. A Hybrid DSP/Deep Learning Approach to Real-Time Full-Band Speech Enhancement. In Proceedings of the IEEE International Workshop on Multimedia Signal Processing (MMSP). arXiv:1709.08243.BibTeX
@inproceedings{valin2018rnnoise, title = {A Hybrid {DSP}/Deep Learning Approach to Real-Time Full-Band Speech Enhancement}, author = {Valin, Jean-Marc}, booktitle = {Proceedings of the IEEE International Workshop on Multimedia Signal Processing (MMSP)}, year = {2018} }DeepFilterNet3
MIT or Apache-2.0Higher-quality background noise removal
Hendrik Schröter, Tobias Rosenkranz, Alberto N. Escalante-B., and Andreas Maier. 2023. DeepFilterNet: Perceptually Motivated Real-Time Speech Enhancement. In Proceedings of INTERSPEECH. arXiv:2305.08227.BibTeX
@inproceedings{schroeter2023deepfilternet3, title = {{DeepFilterNet}: Perceptually Motivated Real-Time Speech Enhancement}, author = {Schr{\"o}ter, Hendrik and Rosenkranz, Tobias and Escalante-B., Alberto N. and Maier, Andreas}, booktitle = {Proceedings of INTERSPEECH}, year = {2023} }DeepFilterNet
MIT or Apache-2.0The deep-filtering framework DeepFilterNet3 is built on
Hendrik Schröter, Alberto N. Escalante-B., Tobias Rosenkranz, and Andreas Maier. 2022. DeepFilterNet: A Low Complexity Speech Enhancement Framework for Full-Band Audio based on Deep Filtering. In Proceedings of ICASSP.BibTeX
@inproceedings{schroeter2022deepfilternet, title = {{DeepFilterNet}: A Low Complexity Speech Enhancement Framework for Full-Band Audio based on Deep Filtering}, author = {Schr{\"o}ter, Hendrik and Escalante-B., Alberto N. and Rosenkranz, Tobias and Maier, Andreas}, booktitle = {ICASSP 2022 - IEEE International Conference on Acoustics, Speech and Signal Processing}, year = {2022} }
Machine translation
Translated transcripts come from Helsinki-NLP’s OPUS-MT models, trained with the Marian toolkit.
OPUS-MT
Apache-2.0Translating transcripts into other languages
Jörg Tiedemann and Santhosh Thottingal. 2020. OPUS-MT – Building open translation services for the World. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation (EAMT), pages 479–480.BibTeX
@inproceedings{tiedemann-thottingal-2020-opus, title = {{OPUS-MT} {--} Building open translation services for the World}, author = {Tiedemann, J{\"o}rg and Thottingal, Santhosh}, booktitle = {Proceedings of the 22nd Annual Conference of the European Association for Machine Translation}, pages = {479--480}, year = {2020} }Marian NMT
MITThe toolkit the OPUS-MT models were trained with
Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. 2018. Marian: Fast Neural Machine Translation in C++. In Proceedings of ACL 2018, System Demonstrations, pages 116–121.BibTeX
@inproceedings{junczys-dowmunt-etal-2018-marian, title = {{M}arian: Fast Neural Machine Translation in {C}++}, author = {Junczys-Dowmunt, Marcin and Grundkiewicz, Roman and Dwojak, Tomasz and Hoang, Hieu and Heafield, Kenneth and Neckermann, Tom and Seide, Frank and Germann, Ulrich and Aji, Alham Fikri and Bogoychev, Nikolay and Martins, Andr{\'e} F. T. and Birch, Alexandra}, booktitle = {Proceedings of ACL 2018, System Demonstrations}, pages = {116--121}, year = {2018} }
Audio codecs
Browsers decode MP3 but will not encode it, so writing an MP3 takes someone else’s encoder.
LAME (via lamejs)
LGPL-3.0Encoding MP3 files in Convert Audio
The LAME project, https://lame.sourceforge.net. JavaScript port: lamejs by Alex Zhukov (@zhuker), a rewrite of jump3r-code, itself a rewrite of libmp3lame. Used unmodified as @breezystack/lamejs 1.2.7.
Running models in the browser
None of the above would run client-side without the work that compiled these runtimes to WebAssembly.
Transformers.js
Apache-2.0Running the Whisper and OPUS-MT models in a Web Worker
Joshua Lochner and the Hugging Face team. Transformers.js: State-of-the-art machine learning for the web.ONNX Runtime
MITExecuting the models via WebAssembly
ONNX Runtime developers. 2021. ONNX Runtime: Cross-platform accelerated machine learning.rnnoise-wasm
Apache-2.0The WebAssembly build of RNNoise we vendor
Tim Guggenmos (@timephy). rnnoise-wasm.
Corrections
Every citation above was checked against the primary source, but if your work is listed incorrectly — or is used here and missing entirely — that is a bug worth reporting, and it will be fixed.