MaryTTS Tutorial: How to make Nepali TTS – Complete Guide

In this post, I will be sharing how you can create Nepali TTS using the open-source MaryTTS project for free. Nepali TTS Creation is easy and its features are:

• Runs offline and on any platform: Windows, Android, Linux, Mac OS.
• Portable: You can easily copy, transfer or distribute the created TTS.
• And more depending on your creativity and knowledge…

This is somehow a long and lengthy process. So please follow through carefully. If you are stuck on any steps, feel free to post in the comment section. Let me also know if you have started this project. I will be happy to help you.

Install Ubuntu in your System

Linux environment makes the overall process of building TTS easier. So, we will first install Ubuntu OS in our system. You can use Ubuntu within VMWare software or just dual boot it as like me.

Download Ubuntu from the given link.

https://ubuntu.com/download/desktop

Then download Rufus from the following link.

https://rufus.ie/en/

With Rufus load ubuntu into a pen drive and boot your computer with a pen drive and install the ubuntu in your system. There are many tutorials about this on the internet. You can check them if you are stuck.

Install Required Basic Programs

Install the following programs in Ubuntu.

Java 8 (OpenJDK)

Open your terminal and run the following commands one by one.

sudo apt update

sudo apt-get install openjdk-8-jdk

sudo ln -s /usr/lib/jvm/java-8-openjdk-amd64 /usr/lib/jvm/default-java

java -version

You should now see the following output.

openjdk version "1.8.0_312"
OpenJDK Runtime Environment (build 1.8.0_312-8u312-b07-0ubuntu1~20.04-b07)
OpenJDK 64-Bit Server VM (build 25.312-b07, mixed mode)

Gradle 4.4.1

Download, setup environment variable and verify the installation

wget https://services.gradle.org/distributions/gradle-4.4.1-bin.zip -P /tmp

sudo unzip -d /opt/gradle /tmp/gradle-*.zip

sudo gedit /etc/profile.d/gradle.sh

Copy and paste the following in the editor.

export GRADLE_HOME=/opt/gradle/gradle-4.4.1
export PATH=${GRADLE_HOME}/bin:${PATH}

Save and close the editor.

Again, run the following commands in the terminal.

sudo chmod +x /etc/profile.d/gradle.sh

source /etc/profile.d/gradle.sh

gradle -v

You should see the following output.

------------------------------------------------------------
Gradle 4.4.1
------------------------------------------------------------

Build time:   2017-12-20 15:45:23 UTC
Revision:     10ed9dc355dc39f6307cc98fbd8cea314bdd381c

Groovy:       2.4.12
Ant:          Apache Ant(TM) version 1.9.9 compiled on February 2 2017
JVM:          1.8.0_312 (Private Build 25.312-b07)
OS:           Linux 5.11.0-46-generic amd64

Important: You will need to restart or log out and log in back for the installation to take effect permanently.

Maven 3.6.3

Download, setup environment variable and verify the installation

wget https://dlcdn.apache.org/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz -P /tmp

sudo tar xf /tmp/apache-maven-*.tar.gz -C /opt

sudo ln -s /opt/apache-maven-3.6.3 /opt/maven

sudo gedit /etc/profile.d/maven.sh

Copy and paste the following in the editor.

export JAVA_HOME=/usr/lib/jvm/default-java
export M2_HOME=/opt/maven
export MAVEN_HOME=/opt/maven
export PATH=${M2_HOME}/bin:${PATH}

Save and close the editor.

Again, run the following commands in the terminal.

sudo chmod +x /etc/profile.d/maven.sh

source /etc/profile.d/maven.sh

mvn -v

You should see the following output.

Apache Maven 3.6.3
Maven home: /usr/share/maven
Java version: 1.8.0_312, vendor: Private Build, runtime: /usr/lib/jvm/java-8-openjdk-amd64/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "5.11.0-46-generic", arch: "amd64", family: "unix"

Important: You will need to restart or log out and log in back for the installation to take effect permanently.

Git

sudo apt install git

GCC Compiler

sudo apt install build-essential

MaryTTS Basic Setup

Download marytts source code from below and extract it.

To extract: Right click on the archive and click on extract here.

Open the extracted folder marytts-5.2 with terminal and run the following command:

mvn install

This will take some time.

You may get some errors at last about Gradle plugin. Ignore this.

[ERROR] Failed to execute goal org.fortasoft:gradle-maven-plugin:1.0.7:invoke (default) on project marytts-assembly-builder: org.gradle.tooling.GradleConnectionException: Could not install Gradle distribution from '[<http://services.gradle.org/distributions/gradle-2.11-bin.zip>](<http://services.gradle.org/distributions/gradle-2.11-bin.zip>)'.

Then run this command:

gradle assemble

This should be executed successfully.

All the executable files are generated inside target directory.

Next, we will setup environment variables so that we can run marytts from any directory or terminal window.

Setup Environment Variable

sudo gedit /etc/profile.d/mary.sh

Copy and paste the following code and change the directory according to yours, save and close the editor.

export MARY_HOME=/home/ardinesh/Downloads/marytts-5.2/target/marytts-5.2/bin
export MARY_BUILDER=/home/ardinesh/Downloads/marytts-5.2/target/marytts-builder-5.2/bin
export PATH=${MARY_HOME}:${MARY_BUILDER}:${PATH}

Make the script executable

sudo chmod +x /etc/profile.d/mary.sh

Load the environment variables in the current shell session:

source /etc/profile.d/mary.sh

Important: You will need to restart or log out and log in back for the installation to take effect permanently.

Verify installation

marytts-server

You should see the following output.

openjdk version "1.8.0_312"
OpenJDK Runtime Environment (build 1.8.0_312-8u312-b07-0ubuntu1~20.04-b07)
OpenJDK 64-Bit Server VM (build 25.312-b07, mixed mode)

MARY server 5.2 starting as a HTTP server... started in 2.026 s on port 59125

Open Mozilla Firefox and go to http://localhost:59125

By default, there is only one voice and few language support.

Below, I will first give you Nepali and Hindi language support files and also a tutorial and making those files.

Nepali and Hindi Language Support Files (Ready Made)

Extract and copy that jar files to /marytts-5.2/target/marytts-5.2/lib/ and also in /marytts-5.2/target/marytts-builder-5.2/lib/

You have to do one more thing. Download the following file and extract and run copy2repo.sh from your terminal.

Open the terminal in your extracted folder and run:

./copy2repo.sh

You can simply download my files but if you want to know how to add language support, you can read the following section.

MaryTTS: New Language Support Tutorial

In the following location, you can find all supported languages and available voices.

/marytts-5.2/target/marytts-5.2/lib/

There is no Nepali or Hindi language by default.

So we will add new language support from the following component.

https://github.com/marytts/marytts-lang-hi/archive/refs/heads/master.zip

Extract it into Downloads folder.

Now, we will edit some files from the Hindi language component so that we can add Nepali Language Support also.

For Hindi

Open /marytts-lang-hi-master/pom.xml from language component.

Also open /marytts-5.2/marytts-languages/marytts-lang-en/pom.xml from source code.

Copy and paste the contents of pom.xml from English language component to Hindi.

In line no 11, just change en to hi

Save and close the editor.

Open the Hindi language directory in the terminal and run the following command:

mvn install

This will take a few moments and the build should complete successfully.

In your Hindi language directory now there is a new folder target

Inside that folder, you can note marytts-lang-hi-5.2.jar

Copy that jar file to /marytts-5.2/target/marytts-5.2/lib/ and /marytts-5.2/target/marytts-builder-5.2/lib/

For Nepali

Extract marytts-lang-hi-master.zip and rename the extracted folder to marytts-lang-ne-master

Open /marytts-lang-ne-master/pom.xml from that folder.

Also, open /marytts-5.2/marytts-languages/marytts-lang-en/pom.xml from the source code folder.

Copy and paste the contents of pom.xml from the English language component to Nepali.

In line no 11, just change en to ne

Save and close the editor.

Similarly, we also need to edit and rename other files and folders:

marytts-lang-ne-master/lib/modules/**hi/**lexicon/allophones.**hi**.xml

marytts-lang-ne-master/lib/modules/**hi/**lexicon/**hi**_pos.list

Go to the above locations and rename or replace hi with ne

Also, edit allophones.hi.xml

In the first line replace hi with ne

Then save and close the editor.

Likewise, edit some more folders and files from marytts-lang-ne-master/src/ directory.

Check all the folders and files and rename them as necessary.

Tips: Rename and replace everything that contains the following keywords.

Hindi → Nepali

hi→ne

If you complete renaming, marytts-lang-ne-master directory in the terminal and run the following command:

mvn install

This will take a few moments and the build should complete successfully.

In your Hindi language directory now there is a new folder target

Inside that folder, you can note marytts-lang-hi-5.2.jar

Copy that jar file to /marytts-5.2/target/marytts-5.2/lib/ and /marytts-5.2/target/marytts-builder-5.2/lib/

MaryTTS: Nepali Voice Building Tutorial

We will make a Nepali voice in this section. We are going to use the Unit Selection Method for building our voice. I am using official documentation (wiki) references. We will start by installing dependent tools and we will prepare our voice dataset.

MaryTTS Dependent Tools

We will need these three tools for building our unit selection voice in marytts.

Praat

sudo apt-get install praat

Edinburgh Speech Tools

sudo apt-get install speech-tools

HTK Labeler

You can download it from the following link. There is no command for this.

https://htk.eng.cam.ac.uk/

It is available for free but you need to be registered to be able to download it.

Stable release 3.4.1

HTK source code (tar+gzip archive)

Enter your login information and your download will begin.

No Password Link (Unofficial): HTK source code (tar+gzip archive)

Download from any of the links and extract and compile the codes as shown below.

Compile HTK source code

Go to the extracted folder.

If you have a 64-bit machine then first you need to modify configure file.

Delete -m32 giving you

CFLAGS="-ansi -D_SVID_SOURCE -DOSS_AUDIO -D'ARCH=\\"$host_cpu\\"' $CFLAGS”

Then open the extracted folder with the terminal and run the following command.

./configure

make all

There may be some error but ignore that and run the following command and check for verification.

sudo make install

Verify installation

Open a new terminal window and run the following command:

HInit

USAGE: HInit [options] hmmFile trainFiles...

Option                                       Default

e f Set convergence factor epsilon 1.0E-4
-i N Set max iterations to N 20
-l s Set segment label to s none
-m N Set min segments needed 3
-n Update hmm (suppress uniform seg) off
-o fn Store new hmm def in fn (name only) outDir/srcfn
-u mvwt Update m)eans v)ars w)ghts t)rans mvwt
-v f Set minimum variance to f 1.0E-2
-w f set mix wt/disc prob floor to f 0.0
-A Print command line arguments off
-B Save HMMs/transforms as binary off
-C cf Set config file to cf default
-D Display configuration variables off
-F fmt Set source data format to fmt as config
-G fmt Set source label format to fmt as config
-H mmf Load HMM macro file mmf
-I mlf Load master label file mlf
-L dir Set input label (or net) dir current
-M dir Dir to write HMM macro files current
-S f Set script file to f none
-T N Set trace flags to N 0
-V Print version information off
-X ext Set input label (or net) file ext lab

MaryTTS: Voice Datasets

I have already recorded my voice in the Nepali Language. So, I will use my own voice for creating Nepali Voice in MaryTTS.

For Hindi, I have downloaded a voice dataset from https://www.iitm.ac.in/

I will give you a link to download my recorded voice dataset.

I will make a Nepali voice in this tutorial but you can make any other language following the same process.

Download my voice dataset here:

NepaliVoiceDataset.zip
Total duration: 39 minute 44 second
Size: 163 MB

Setup Voice Building Directory

Extract the downloaded archive in the downloads folder.

The folder extracted just now is your voice-building directory. You can change the name of this folder if you want. Inside this folder there are two items:

txt.done.data – contains all the transcripts of recordings for audio files in wav directory

wav – contains all the audio recordings

Important: Audio sample rate must be 16000 Hz so that there will be no problem later.

You can check it simply in file properties.

If your audio has a different or higher sample rate, you can run the following command in the terminal.

audio_converter_GUI.sh

Choose Input Wave Directory and Output Directory.

Uncheck all other options. Check only on sampling rate conversion.

You need to install sox binary first and give the binary path in Location of sox binary

Run the following command by opening the terminal.

sudo apt install sox

Set sox binary path as:

Location of sox binary: /usr/bin/sox

Building Nepali Voice in MaryTTS (using Voice Import Tool)

I want you to close all apps running in the foreground and follow the process below:

Open Terminal and run the following commands:

marytts-server

Again open another terminal window or tab and run:

voiceimport.sh

Choose a voice building directory extracted inside Downloads folder.

Settings Editor

In Settings Editor, change the values of the following properties as follows:

db.estDir /usr/

db.gender male

db.locale ne

db.marybase /home/ardinesh/Downloads/marytts-5.2/

db.voicename my_voice_dinesh_nepali

To change values: Double click on the value.

You can modify these values as per your requirement.

Click on Save.

Voice Import Components

Now, we can see all the tools available for building our voice.

We will use the following components for unit selection voice.

Raw Acoustics: PraatPitchmarker MCEPMaker

Transcripts Conversion: Festvox2MaryTranscripts

Automatic Labelling: AllophonesExtractor HTKLabeler LabelPauseDeleter LabelledFilesInspector

Label-Transcript Alignment: PhoneUnitLabelComputer HalfPhoneUnitLabelComputer TranscriptionAligner

Feature Extraction: FeatureSelection PhoneUnitFeatureComputer HalfPhoneUnitFeatureComputer

Verify Alignment: PhoneLabelFeatureAligner HalfPhoneLabelFeatureAligner

Basic Data Files: WaveTimelineMaker BasenameTimelineMaker MCepTimelineMaker

Acoustic Models: PhoneUnitfileWriter PhoneFeatureFileWriter DurationCARTTrainer F0CARTTrainer

Unit Selection Files: HalfPhoneUnitfileWriter HalfPhoneFeatureFileWriter F0PolynomialFeatureFileWriter AcousticFeatureFileWriter JoinCostFileMaker CARTBuilder

Install Voice: VoiceCompiler

We need to run all the tools mentioned above.

You can run one by one or just check all the tools at once and click on run. I recommend you run all the tools one by one.

We are only leaving HMM tools here as we are building unit selection voice.

PraatPitchmarker

It may seem like freezing but it will load in a few seconds so please be patient.

Don’t quit.

When it loads, just click on OK and OK.

Then, in the script window, click on run and again run from the option.

Now, change the values of minPitch and maxPitch (Ex: Male: 50-200 | Female: 150-300)

Click on OK and wait for some time to finish processing.

Close all Praat windows.

Important step:

Before running HTKLabeler click on the setting icon on the left of the checkbox and change the value as below:

HTKLabeler.htkDir /usr/local/bin/

When HTK labeling finishes, you have to copy all the lab files from the following directory and paste them into the folder below. Note: This should have been done automatically by the tool but maybe there is an issue.

/nepali_voice_dinesh_male/htk/lab/

/nepali_voice_dinesh_male/lab/

Then run LabelPauseDeleter and LabelledFilesInspector

Play any file and it should be played.

Now, all other tools will run without error if you have followed this tutorial from first.

In VoiceCompiler you may get an error but just ignore it and go to the following directory. Now, you can close voiceimport tool.

/nepali_voice_dinesh_male/mary/voice-my_voice_dinesh_nepali/

There if you see the target folder then that’s all. We will next install our voice.

But if you don’t see that folder, we need to do one more thing. Just open the above folder with the terminal and run the following command.

mvn install

That’s all. After its execution, you should see the target folder there.

Installing compiled voice

/nepali_voice_dinesh_male/mary/voice-my_voice_dinesh_nepali/target/voice-my_voice_dinesh_nepali-5.2.zip

Copy the above file to the following directory and extract it.

/marytts-5.2/target/marytts-5.2/

From the extracted folder move lib out of that folder and merge with the existing lib folder.

Close all the terminals and open a new terminal and run the following command:

marytts-server

Then go to, http://localhost:59125

There you will see our Nepali voice already selected. Just click on SPEAK and our TTS will greet us in the Nepali language.

THE END

Congratulations! You did it. Don’t forget to post your comment about this guide.

I hope you find this helpful.

Please share and keep learning.

Thanks.

Bonus useful commands

Command for getting the total duration of audios/videos present in the directory. Just open the terminal in the folder you want and run the following command.

soxi -D * | awk '{SUM += $1} END { printf "%d:%d:%d\\n",SUM/3600,SUM%3600/60,SUM%60}'

Command for fixing time changes in windows ubuntu dual boot.

timedatectl set-local-rtc 1