Researchers from Korea University Unveil HierSpeech++: A Groundbreaking AI Approach for High-Fidelity, Efficient Text-to-Speech and Voice Conversion

Researchers at Korea University have developed a brand new speech synthesizer known as HierSpeech++. This analysis goals to create artificial speech that’s strong, expressive, pure, and human-like. The workforce aimed to attain this with out counting on a text-speech paired dataset and to enhance present fashions’ shortcomings. HierSpeech++ was designed to bridge the semantic and acoustic illustration hole in speech synthesis, in the end enhancing fashion adaptation.

Until now, zero-shot speech synthesis based mostly on LLM has had limitations. However, HierSpeech++ has been developed to handle these limitations and enhance robustness and expressiveness whereas addressing points associated to sluggish inference velocity. By using a text-to-vec framework that generates self-supervised speech and F0 representations based mostly on textual content and prosody prompts, HierSpeech++ has been confirmed to outperform LLM-based and diffusion-based fashions. These velocity, robustness, and high quality developments set up HierSpeech++ as a robust zero-shot speech synthesizer.

HierSpeech++ makes use of a hierarchical framework for producing speech with out prior coaching. It employs a text-to-vec framework to develop self-supervised deal with and F0 representations based mostly on textual content and prosody prompts. Speech is produced utilizing a hierarchical variational autoencoder and a generated vector, F0, and voice immediate. The methodology additionally consists of an environment friendly speech super-resolution framework. Comprehensive evaluation makes use of numerous pre-trained fashions and implementations with goal and subjective metrics akin to log-scale Mel error distance, perceptual analysis of speech high quality, pitch, periodicity, voice/unvoice F1 rating, naturalness, imply opinion rating, and voice similarity MOS.

Superior naturalness in artificial speech is achieved by HierSpeech++ in zero-shot eventualities, with enhancements in robustness, expressiveness, and speaker similarity. Subjective metrics like naturalness imply opinion rating and voice similarity MOS have been used to evaluate the innocence of the speech, and the outcomes confirmed that HierSpeech++ outperforms ground-truth speech. Incorporating a speech super-resolution framework from 16 kHz to 48 kHz additional improved the naturalness of the deal with. Experimental outcomes additionally demonstrated that the hierarchical variational autoencoder in HierSpeech++ is superior to LLM-based and diffusion-based fashions, making it a strong zero-shot speech synthesizer. It was additionally discovered that zero-shot text-to-speech synthesis with noisy prompts validated the effectiveness of HierSpeech++ in producing speech from unseen audio system. The hierarchical synthesis framework additionally permits for versatile prosody and voice fashion switch, making synthesized speech much more versatile.

In conclusion, HierSpeech presents an environment friendly and potent framework for reaching human-level high quality in zero-shot speech synthesis. Its disentangling of semantic modeling, speech synthesis, super-resolution, and facilitation of prosody and voice fashion switch improve synthesized speech flexibility. The system demonstrates robustness, expressiveness, naturalness, and speaker similarity enhancements even with a small-scale dataset and presents considerably sooner inference speeds. The research additionally explores potential extensions to cross-lingual and emotion-controllable speech synthesis fashions.

Check out the Paper, Project and Github. All credit score for this analysis goes to the researchers of this challenge. Also, don’t neglect to hitch our 33k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and Email Newsletter, the place we share the most recent AI analysis information, cool AI tasks, and extra.

If you want our work, you’ll love our e-newsletter..