Import audio data from other free/open-source projects

[ftyers]

One of the issues with the "dictation" feature is that it isn't always easy to find data for it.

It would be great to be able to import data from either Common Voice or Tatoeba, or both.

[issue-label-bot]

Issue-Label Bot is automatically applying the label feature_request to this issue, with a confidence of 0.93. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[kantord]

One possible challenge I see with importing audio from Tatoeba is that it has fixed audio recordings, instead of TTS.

With TTS, the person editing the course content has complete freedom in choosing the sentence, and the audio magically appears. In order to use Tatoeba, we'd have to come up with some solution for it also in the course editor. (Or maybe TTS could be used as a fallback when there's no exact match?)

[kantord]

I think Common Voice might have the same problem, but I see that there's an existing TTS project for it: https://github.com/mozilla/TTS

Perhaps this could be used to replace Amazon Polly as a TTS engine.

Ideally we'd have support for a set of TTS engines in order to maximize the number of supported languages.

[kantord]

There is also a browser API for speech synthesis: https://developer.mozilla.org/en-US/docs/Web/API/SpeechSynthesis

I imagine in Firefox it will eventually use Common Voice. That would have the amazing benefit of not having to download the audios, but ensuring quality might be challenging.

[ftyers]

I don't think that using TTS is a good idea if it is going to be used for language learning. TTS, unless there is a carefully curated corpus is unlikely to produce good enough output. There would be issues with pronunciation, and especially with prosody and stress. I don't think it's a big problem that the sentences have to be selected from a pre-existing list of sentences. In fact, I think that this is good for learners doing dictation.

Unfortunately Mozilla has basically dumped their work on speech in their recent vote for collapse. So going on from here their speech stuff will I suppose be typical free/open-source projects, rather than something with a lot of backing.

Note that, if course designers want to include their own sentences, then one way would be for them to submit them to Tatoeba and to Common Voice. The Common Voice sentence collector is an online and pretty much automatic way of getting sentences into Common Voice. I believe Tatoeba has a similar system.

In my experience with language learning, listening and dictating real sentences produced by real people in real environmental conditions is much more valuable than trying to listen to a synthetic voice. Basically I think that using CV and Tatoeba keys us into two inclusive communities rather than relying on TTS, technology which is available for only a tiny fraction of the world's languages is a better plan.

[kantord]

I don't think that using TTS is a good idea if it is going to be used for language learning. TTS, unless there is a carefully curated corpus is unlikely to produce good enough output.

That's a good point, yeah. For sure there are few TTS options that are good enough. Amazon Polly is the one currently used, it's actually the same one Duolingo uses.

For many languages however, TTS will be out of the question, because the quality will be nowhere near usable. Let alone endangered languages or constructed languages. So I think we can conclude that at some point in the future, being able to use pre-recorded audio will be essential

[kantord]

Note that, if course designers want to include their own sentences, then one way would be for them to submit them to Tatoeba and to Common Voice. The Common Voice sentence collector is an online and pretty much automatic way of getting sentences into Common Voice. I believe Tatoeba has a similar system.

This sounds like a wonderful idea

[ftyers]

I don't think that using TTS is a good idea if it is going to be used for language learning. TTS, unless there is a carefully curated corpus is unlikely to produce good enough output.

That's a good point, yeah. For sure there are few TTS options that are good enough. Amazon Polly is the one currently used, it's actually the same one Duolingo uses.

Yeah, unfortunately it isn't free/open-source either in terms of data or code, so isn't really usable or extensible to most languages. I would hesitate before introducing features that necessarily result in two-tiers of languages, the "with commercial support" and "without commercial support".

[kantord]

I would hesitate before introducing features that necessarily result in two-tiers of languages, the "with commercial support" and "without commercial support".

I would also say that our main focus should be supporting features that can be shipped for all languages.

But at the same time, if we are too selective, and really only support things that can provide equal quality for all languages, then we might severely limit the overall feature-set 🤔

[ftyers]

I agree that is a concern, but I think it is ok if we ship free/open-source solutions for let's say "advanced" features, because in principle they can be extended to all languages, I just wouldn't want to rely on anything proprietary. And I would build on existing projects that are language-inclusive rather than rely on language-exclusive projects.

[davidak]

https://www.dict.cc/ has recordings, but they are not open.

Maybe you can convince Paul to publish them under an open license. It is made by the community after all.