Overview
Zero-shot text-to-speech (TTS) has gained significant attention due to its powerful voice cloning capabilities, requiring only a few seconds of unseen speaker voice prompts. However, all previous work has been developed for cloud-based systems. Taking autoregressive models as an example, although these approaches achieve high-fidelity voice cloning, they fall short in terms of inference speed, model size, and robustness. Therefore, we propose MobileSpeech , which is a fast, lightweight, and robust zero-shot text-to-speech system based on mobile devices for the first time. Specifically: 1) leveraging discrete codec, we design a parallel speech mask decoder module called SMD, which incorporates hierarchical information from the speech codec and weight mechanisms across different codec layers during the generation process. Moreover, to bridge the gap between text and speech, we introduce a lower-level probabilistic mask that captures the progression of information flow from less to more during speech generation. 2) For speaker prompts, we extract fine-grained prompt duration from the prompt speech and incorporate text, prompt speech by cross attention in SMD. We demonstrate the effectiveness of MobileSpeech on multilingual datasets at different levels, achieving state-of-the-art results in terms of generating speed and speech quality. MobileSpeech achieves RTF of 0.2 on a single A100 GPU and we have successfully deployed MobileSpeech on mobile devices.
Model Architecture
 |
Figure.1 The overall architecture of MobileSpeech.
Librispeech Test Clean
1.GRACE INVOLVES THE REMISSION OF SINS PEACE AND A HAPPY CONSCIENCE
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
2.OVER THE TRACK LINED CITY STREET THE YOUNG MEN THE GRINNING MEN PASS
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
3.FAIRVIEW WAS TWELVE MILES AWAY BUT BY TEN O’CLOCK THEY DREW UP AT THE COUNTY JAIL
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
4.THE JOHN BRIGHT IS ARMED WITH A WEAPON OF GREAT POWER AGAINST WHICH IT IS IMPOSSIBLE THAT THE PEOPLE OF BRITANNULA SHOULD PREVAIL
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
5.IT IS NECESSARY THEREFORE THAT HE SHOULD COMPLY THE KING FROWNED
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
6.AT THE SUDDEN SHARP STING OF IT THE GREAT BIRD TURNED HIS HEAD AND NOTICED FOR THE FIRST TIME THE FISHERMAN STANDING ON THE BANK
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
7.A STAGE MEAL IS POPULAR BECAUSE IT PROVES TO THE AUDIENCE THAT THE ACTORS EVEN WHEN CALLED CHARLES HAWTREY OR OWEN NARES ARE REAL PEOPLE JUST LIKE YOU AND ME
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
8.WARRENTON SPOKE THUS WITH SIGNIFICANCE TO SHOW ROBIN THAT HE WAS NOT TO THINK GEOFFREY’S CLAIMS TO THE ESTATE WOULD BE PASSED BY
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
9.TAKING ADVANTAGE OF THIS THE SQUIRE’S FEW MEN REDOUBLED THEIR EFFORTS AND ENCOURAGED BY ROBIN’S AND THE LITTLE STROLLER’S CRIES FOUGHT THEIR WAY TO HIM
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
10.HE CRIED IN HIGH DUDGEON JUST AS IF HE OWNED THE WHOLE OF THE PEPPERS AND COULD DISPOSE OF THEM ALL TO SUIT HIS FANCY
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
11.FRANK READ ENGLISH SLOWLY AND THE MORE HE READ ABOUT THIS DIVORCE CASE THE ANGRIER HE GREW
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
12.THAT’S NOT MUCH OF A JOB FOR AN ATHLETE HERE I’VE BEEN TO TOWN AND BACK
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
13.BUT THE WINDOWS ARE PATCHED WITH WOODEN PANES AND THE DOOR I THINK IS LIKE THE GATE IT IS NEVER OPENED
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
14.IT IS SO MADE THAT EVERYWHERE WE FEEL THE SENSE OF PUNISHMENT
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
15.NO ONE WOULD DISTURB THEIR LITTLE HOUSE EVEN IF ANYONE CAME SO FAR INTO THE THICK FOREST WHILE THEY WERE GONE
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
16.WE ARE TRAVELING REPLIED OJO AND WE STOPPED AT YOUR HOUSE JUST TO REST AND REFRESH OURSELVES
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
17.DID EVER ANYBODY SEE THE LIKE SCREAMED MISSUS POYSER RUNNING TOWARDS THE TABLE WHEN HER EYE HAD FALLEN ON THE BLUE STREAM
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
18.SOMEONE ELSE TOLD A STORY NOT PARTICULARLY EFFECTIVE WHICH I SAW HE WAS NOT FOLLOWING
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
19.THE LEADERS OF THE CONSPIRACY BECAME DISTRUSTFUL OF THEIR POWER TO CRUSH THE TOWN
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
20.THERE WAS SOMETHING IN HIS AIR AND MANNER THAT BETRAYED TO THE SCOUT THE UTTER CONFUSION OF THE STATE OF HIS MIND
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
21.AS SOON AS THESE DISPOSITIONS WERE MADE THE SCOUT TURNED TO DAVID AND GAVE HIM HIS PARTING INSTRUCTIONS
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
22.LET US RETRACE OUR STEPS AND EXAMINE AS WE GO WITH KEENER EYES
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
23.IS THE ATMOSPHERIC CONDITION HAVING ONCE REACHED THIS DENSITY TO BECOME FINAL
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
24.ON FRIDAY CONFESSION WILL BE HEARD ALL THE AFTERNOON AFTER BEADS
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
25.HE STOOD STILL IN DEFERENCE TO THEIR CALLS AND PARRIED THEIR BANTER WITH EASY WORDS
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
26.THIS DIFFERENTIATION IS FURTHERED BY THE INHERITANCE OF WEALTH AND THE CONSEQUENT INHERITANCE OF GENTILITY
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
27.HE COULD HARDLY STAND STILL SO GREAT WAS HIS MENTAL AGITATION AND HE RAN TOWARDS HOLMES WITH TWO EAGER HANDS OUTSTRETCHED THANK HEAVEN THAT YOU HAVE COME
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
28.BARTLEY LEANED OVER HER SHOULDER WITHOUT TOUCHING HER AND WHISPERED IN HER EAR YOU ARE GIVING ME A CHANCE YES
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
29.BUT THE REAL SIGNIFICANCE AND COMFORT OF THE WORDS FOR OUR SINS IS LOST UPON THEM
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
30.ANYHOW WE’LL LEAVE INSTRUCTIONS TO SHIP THE WHOLE MENAGERIE TO FRANCE
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
31.KESWICK MARCH TWENTY SECOND EIGHTEEN THIRTY SEVEN DEAR MADAM
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
32.VAST IMPORTANCE AND INFLUENCE OF THIS MENTAL FURNISHING
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
33.HE DARTED THROUGH THE TREES AND PAUSED A TALL MAN STRONGLY BUT SLIMLY MADE
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
34.ALAS I HAVE GRIEVED SO I AM HARD TO LOVE
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
35.IN PERSON WELCOME ABOARD PROFESSOR YOUR CABIN IS WAITING FOR YOU
| GT | GT Codec | YourTTS | Valle | MegaTTS | MobileSpeech |
|---|---|---|---|---|---|
Chinese Speech Samples
1.今后我们要加强和发达地区的交流与合作，共同开发，尽快把内蒙古经济搞上去。
| MobileSpeech |
|---|
2.据了解，天津市今年粮食种植面积达六百六十五万亩，预计全年粮食总产量可达二十点七五亿公斤，比去年提高百分之九。
| MobileSpeech |
|---|
3.财产所有权转移了，其风险责任也随之转移。
| MobileSpeech |
|---|
4.我们几个人你看看我，我看看你，一筹莫展。
| MobileSpeech |
|---|
5.达成意向的有二十四个单位，涉及十九个项目，两百九十二名运动员。
| MobileSpeech |
|---|
6.图为喇叭哥在用吼声执法。
| MobileSpeech |
|---|
7.而是为了调整产业结构和所有制结构，更好地发展社会生产力。
| MobileSpeech |
|---|
8.有位长者被挤得东歪西倒，手上还紧紧拖着一只折叠的简易行李车，车上捆着两三个大行李袋。
| MobileSpeech |
|---|
9.女人渐渐爱上了老汉，这爱滋润老汉干枯的生命变得鲜活起来，不料，秋收打场过后，女人却出于无奈悄悄离开了老汉。
| MobileSpeech |
|---|
10.往前走，家庭皮鞋作坊里机声踏踏，鱼塘上阳光布洒每个家庭，每座小院，都成了农工们各显神通的舞台。
| MobileSpeech |
|---|