nsh - Speech Recognition With CMU Sphinx

Dealing with pruning issues

2012-01-15T01:12:00.002+04:00

I spent a holiday looking on the issues in poketsphinx decoding in fwdflat mode. Initially I thought it's a bug but it appeared that it's just a pruning issue. The result looked like this:

 INFO: ngram_search.c(1045): bestpath 0.00 wall 0.000 xRT
INFO:   <s>                  0     5     1.000 -94208     0          1
INFO:   par..grafo           6     63    1.000 -472064    -467       2
INFO:   terceiro             64    153   1.000 -1245184   -115       3
INFO:   as                   154   176   0.934 -307200    -172       3
INFO:   emendas              177   218   1.000 -452608    -292       3
INFO:   ao                   219   226   1.000 -208896    -181       3
INFO:   projeto              227   273   1.000 -342016    -152       3
INFO:   de                   274   283   1.000 -115712    -75        3
INFO:   lei                  284   3059  1.000 -115712    -79        3

Speech recognition is essentially a search for a globally best path in a graph. Beam pruning is used to drop the nodes during the search if node score is worse then the best node like in this picture

If beam is too narrow, the result might not be the globally best one despite its locally the best. In practice it could lead to complex issues like desribed above. See the word "lei" spans about 2k frames which means about 20 seconds. Another sign of overpruning is number of words scored per frame


INFO: ngram_search_fwdflat.c(940):     2931 words recognized (1/fr)
INFO: ngram_search_fwdflat.c(942):    48013 senones evaluated (16/fr)
INFO: ngram_search_fwdflat.c(944):     9586 channels searched (3/fr)
INFO: ngram_search_fwdflat.c(946):     3849 words searched (1/fr)
INFO: ngram_search_fwdflat.c(948):     9602 word transitions (3/fr)

If you have just one word per frame it's likely an issue.

More detailed behaviour can be seen if debugging in enabled in sources

 #define __CHAN_DUMP__           1

You'll see something like


BEFORE:
SSID          2866         610         611 (2608)
SENSCR        -604        -215        -371
SCORES    -1014874     -583095     -583097     -583223
HISTID         170         170         170         170
AFTER:
SSID          2866         610         611 (2608)
SENSCR        -604        -215        -371
SCORES    -1015481     -583315     -583317     -583489
HISTID         170         170         170         170
BEFORE:
SSID          2866         610         611 (2608)
SENSCR        -568        -122        -358
SCORES    -1015481     -583315     -583317     -583489
HISTID         170         170         170         170
AFTER:
SSID          2866         610         611 (2608)
SENSCR        -568        -122        -358
SCORES    -1016052     -583442     -583444     -583696
HISTID         170         170         170         170

So you see only one HMM per frame is scored and it doesn't generate any other HMMs

Since those issues are hard to notice since today we will also issue you a warning in the decoder log. It will look like this:

 WARNING: "ngram_search.c", line 404: Word 'lei' survived for 2764 frames, potential overpruning
WARNING: "ngram_search.c", line 404: Word 'lei' survived for 2765 frames, potential overpruning

So you'll be warned if something will go wrong.

It's very easy to forget about pruning issues because they are not really visible. You'll only get a drop in the accuracy and you might not notice it. And you might think it's a model accuracy not a search accuracy. In practice you need always remember about that:

Search space configuration and settings have certain effect on the final accuracy and speed.

Default settings are often wrong for modified models. If you have a new model you need to review all the configuration parameters in order to make sure they work. If there are many parameters, you need to check all of them.

If pruning errors in your decoder have very small effect it means you haven't optimized your search space properly. You can definitely do better.

At least we might want to report more useful metrics about pruning in the future.

Magic Words of Interspeech 2011

2011-08-24T18:15:00.000+04:00

Interspeech 2011 is coming. It going to be an amazing event I suppose. If you are interested what is going on there, let's figure that out.

To keep things simple we will use Unix command line tools. Sometimes text processing could be fun even with simple commands. Text is still most conventint form of the information presentation, way better than HTML or databases. Of course there is lack for more advanced things like stopword filtering or named entity recognition. Let's hope one day Unix command line will have them.

1. Download full printable programs of Interspeech 2010 and Interspeech 2011 with wget, dump them to text with lynx and cleanup punctuation with sed.

2. Dump word counts with SRILM tool ngram-count and cut 1000 most frequent words on list for 2011 with head and sort. Leave all words in 2010 list.

3. Figure out which of the words in 2011 list are new and do not appear in 2010 list with sort and uniq.

Suprisingly there will be only 2 new words. They are: i-vector and crowdsourcing.

When Language Models Fail

2011-07-26T09:30:00.002+04:00

Language modeling still have many challenging problems.

Comic by Jim Benton

Decoders And Features

2011-07-23T02:58:00.011+04:00

CMUSphinx decoders in a glance, so one can compare. Table is incomplete and imprecise of course.

	sphinx2	sphinx3	sphinx4	pocketsphinx
Acoustic Lookahead	-	-	+	-
Alignment	+	+	+	-
Flat Forward Search	+	+	-	+
Finite Grammar Confidence	+	-	-	-
Full n-gram History Tree Search	-	-	+	-
HTK Features	-	+	+	+
Phonetic Loop Decoder	+	+	-	-
Phonetic Lookahead	+	+	-	+
PLP features	-	-	+	-
PTM Models	-	-	-	+
Score Quantization	+	-	-	+
Semi-Continuous Models	+	+	-	+
Single Tree Search	+	-	-	+
Subvector Quantization	+	+	-	+
Time-Switching Tree Search	-	+	-	-
Tree Search Smear	-	+	+	-
Word-Switching Tree Search	-	+	-	-
Thread Safety	-	-	+	+
Keyword Spotting	-	-	+	-

And here is the description of the entries

Specific Applications

Phonetic Loop Decoder. Phonetic loop decoding requires specialized search algorithm. It's not implemented in Sphinx4 for example.

Alignment. Given text and the transcription get the word timings.

Keyword spotting. Search for keyword requires separate search space and different search approach.

Finite Grammar Confidence. Get confidence estimation for finite state grammar. This is a complex problem which
require additional operations during search, for example phone loop pass.

Effective pruning

Acoustic Lookahead. Using acoustic score for the current frame we can predict the score for the next frame
and thus prune token early.

Phonetic Lookahead. Using phonetic loop decoder we can predict possible phones and thus restrict large vocabulary search.

Features

HTK Features. CMUSphinx feature extraction is different from HTK (different filterbank and transform). To provide HTK capability one needs to have specific HTK feature extraction.

PLP features. Type of the features different from traditional MFCC. They are more popular now.

Search Space

Flat Forward Search. Search space when word paths aren't joined in lextree. Separated path lets us to apply language model probability earlier. Thus search is more accurate. But because search space is bigger it's also slower. Usually flat search is applied as a second pass after tree search.

Full n-gram History Tree Search. Tokens which have different n-gram history are tracked separately. For example token for "how are UW " and token for "hello are UW.." are tracked separately. In pocketsphinx such tokens are just joined and only best one survive. Full history search is more accurate but slower and more complex in implementation.

Word-Switching Tree Search. Separate lextrees are kept for each unigram history. This search is in the middle between the one to keep the full history and another one to drop the history at all.

Single Tree Search. Lextree tokens don't care about word history. This is faster but less accurate way.

Time-Switching Tree Search. Lextree states don't care about word history but several lextrees are kept in memory (3-5). In this time switching approach lextrees are switched every frame. Because of that there is higher chance to track both histories.

Tree Search Smear. Lextree contains unigram probability and thus it's possible to prune token earlier based on the language score.

Acoustic Scoring

PTM Models. Models when gaussians are shared across senones with same central phone. So we don't need to calculate gaussians value for each senone, just few values for each central phone. Then using different mixture weights we get senone score. This approach reduce computation required but keeps accuracy on a reasonable level. It's similar to semi-continuous models where gaussians are shared across all senones, not just across senones with same central phone.

Score Quantization. Acoustic scores in some cases could be represented by just 2 bytes (semi-continuous models and specific feature set). Usually scores are in log domain and shifted by 10 bits. This reduces memory required for acoustic model and for scoring and speeds up the computation in particular on CPU without FPU.

Semi-Continuous Models. Gaussians are shared across all senones, only mixture weights are different. Such models are fast and usually quite accurate. Usually they are multistream (s2_4x or 1s_c_d_dd with subvector 0-12/13-25/26-38) since separate streams could be better quantized.

Subvector Quantization. Gaussian selection approach to reduce acoustic scoring. Basically continuous model after training is deconstructed on several subvector gaussians which are shared across senones and thus scored efficiently.

Cars Controlled By Speech

2011-06-28T02:51:00.007+04:00

Being a speech recognition guy I'm looking for a car with speech recognition included. Sounds strange to select car just because of it, but that is just kidding. So far the list is:

Honda Accord
Any Ford 2011
Mazda 6

Not listing something expensive like BMW or Mersedes. Hm, it looks almost everyone is doing that. Any others? Which is the most advanced one?

Some details on particular implementation

Ford SYNC 2011

Quite advanced system. Command-based. Supports many types of commands to control dvd or get baseball scores. Supports user profiles but doesn't seem like it has specific training procedure. With current speaker recognition capabilities it could in theory adapt to users automatically without profiles.

Mazda 6 2011

Pretty interesting system, but limited comparing to previous one. According to owner manual it supports a very limited list of commands to manage calls, get incoming messages and. From interesting capabilites it supports training and voice entry for contacts. Three languages - English, French, Spanish. Looks like it's using single microphone. Looks like voice navigation system has separate speech recognition subsystem.

Honda Fit 2009

Many commands mostly related to navigation but no user adaptation and no profiles. Alphanumeric entry as a backup to vocabulary search. This one is very simple.

Mitsubishi/Hyundai 2011

I didn't manage to find the manual on them. Feature name "Bluetooth hands-free phone system with voice recognition and phonebook download" makes me think it's the same system as in Mazda.

Talkmatic

Doesn't seem like this is deployed, but presentation looks impressive

KIA

Accoding to SpeechTechMag Microsoft and Kia codeveloped the UVO multimedia and infotainment system, which the Korean automaker rolled out in its new Sportage, Sorento, and Optima models late last year. UVO lets users access media content and connect with people through quick voice commands without having to navigate hierarchical menus.

ICASSP 2011 Part 1 - Thoughts

2011-06-21T03:33:00.002+04:00

It seems like ICASSP this year was a great event, it is pity I missed it. Just comparing the keynotes list, ICASSP beats Interspeech 4:0. ICASSP is very technical, Interspeech is for linguists. Compare the two:

Making Sense of a Zettabyte World vs Neural Representations of Word Meanings

New section formats like technical tracks and trends discussions are interesting though I am not sure how they felt in practice.

So this was the reason to spend few days in reading. 1000 papers on speech technology! Huh. Thanks to all authors for their hard work! Well, I found several duplicates in the end.

Main thing I noted is that topics of the research are very sparse, for example

Everyone does speaker recognition. Appealing problem statement here is that here is to detect a synthetic speaker. Paper titled "DETECTION OF SYNTHETIC SPEECH FOR THE PROBLEM OF IMPOSTURE" by De Leon at al. hints that there is no solution for that.
I got tired to skip pursuits, bandiths and compressive sensing
On the other side, increased portion of papers on non-speech signals, cocktail party problem, signal recovery is very interesting to read.
Things like DBN features or SCARF decoder are widely represented. You can read about applications of CRF from g2p algorithms to dialogs. But traditional things like search algorithms and adaptation are almost uncovered.
It was suprising to find the session dedictated to multimedia security which must be a gold mine of ideas in particular if you need a topic for a paper. Is there a company selling such products?

Overall I found several original problem statements as well as inspiring ideas covering very important technology issues. For example it would be nice to implement meeting transcription application with several iPhones to combine streams and later transcribe them using multichannel environment compensation. Several meeting transcription setups and channel separation methods are described in the conference proceedings.

After reading some amount of papers I found that conference papers are too short. While you see a nice title and an abstract you expect to read a detailed insight into the problem with history discourse and everything explained in detail, a deep investigation of the problem. But you get just a description of the technology and few figures from experiments. On the other side, I will not be able to read 100 papers 20 pages each.

Very interesting that this year awards are not related to speech technology. That will be the contents of Part 2. I just need to go through last 50 papers left.

Chicken-And-Egg in Sphinxbase

2011-05-02T23:51:00.001+04:00

Recently Shea Levy pointed me to an issue with a verbose output during pocketsphinx initialization. Basically every time you start pocketsphinx, you get something like

INFO: cmd_ln.c(691): Parsing command line:
pocketsphinx_continuous
Current configuration:
[NAME] [DEFLT] [VALUE]
-adcdev
-agc none none
-agcthresh 2.0 2.000000e+00
-alpha 0.97 9.700000e-01
-argfile

It's ok for a tool but not a nice thing for the library which should be a small horse in a rig of application. Not every user is happy seeing all this stuff dumped on the screen. And the worst thing is that there is no way to turn it off because "-logfn /dev/null" works only for the output after initialization. So we are looking to have pocketsphinx completely silent.

It appeared to be more complex issue than I thought. Its classical chicken-egg issue when you use configuration framework do configure logging but configuration framework needs to log itself. We just hardcoded the initialization but thinking afterwards I found way more complex and but more rigid approach in log4j description from http://articles.qos.ch/internalLogging.html

Since log4j never sets up a configuration without explicit input from the user, log4j internal logging may occur before the log4j environment is set up. In particular, internal logging may occur while a configurator is processing a configuration file.

We could have simplified things by ignoring logging events generated during the configuration phase. However, the events generated during the configuration phase contain information useful in debugging the log4j configuration file. Under many circumstances, this information is considered more useful than all the subsequent logging events put together.

In order to capture the logs generated during configuration phase, log4j simply collects logging events in a temporary appender. At the end of the configuration phase, these recorded events are replayed within the context of the new log4j environment, (the one which was just configured). The temporary appender is then closed and detached from the log4j environment.

Oh-woh, I will never get enough passion to implement this properly ;) Let it be as is for now.

Sphinxbase command line options are still not good. I'm pretty much lack proper --help, --version and many more nifty getopt things. One day someone should do this.

Voicemail transcription with Pocketsphinx and Asterisk (Part 2)

2011-04-26T17:17:00.004+04:00

This is a second part which describes voicemail transcription for Asterisk administrators. See previous part which describes how to setup Pocketsphinx here

So you have configured the recognizer to transcribe voicemails and now look on the improved recognizer accuracy. Honestly I can tell you that you will not get perfect transcription results for free unless you will send voicemails to some human-assisted transcription company. You will not get them from the Google either. Though there are several commercial services to try like Yap or Phonetag which specialize on voicemails specifically. Our proprietary Nexiwave technology for example uses way more advanced algorithms and way bigger speech databases than distributed with Pocketsphinx. And it's a really visible difference.

However even the result you can get with Pocketsphinx can be very usable or you. I estimate you can easily get 80-90% accuracy with little effort considering the language of your voicemails is simple.

Now, the core components of the recognizer are:

Language model which controls sequence of words
Acoustic model which describe how each phone sounds
Phonetic dictionary which maps words to phonetic representation

To get better accuracy you need to improve those three. By default the following models are used

Dictionary - pocketsphinx/model/lm/en_US/cmu07a.dic
Language model - pocketsphinx/model/lm/en_US/hub4.5000.DMP
Acoustic model - pocketsphinx/model/hmm/en_US/hub4wsj_sc_8k

So let's try to improve them step by step by the order of importance

Language model
The core reason voicemail transcription is bad is that language model is built for completely different domain. HUB4 is DARPA task to transcribe broadcast news so you see it's very different from the voicemail language. It's perfect to recognize voicemail about NATO or democracy but not about your wife's problems. We need to change the language model.

1) Transcribe some amount of your existing voicemails. A 100 will be already good. Put the transcription in a text file line by line:

hello jim it's steve let's meet at five p m

hello jim buy some milk

jim it's bob i should catch you tomorrow

you fired jim 

....

Important here that text is all in lower case one sentence per line and it doesn't have any punctuation.

Then, you can find some domain specific texts in your computer. For example, if you are working as system administrator in chemical company some chemical texts will help to improve the quality of the language model. Take few books and convert them to the same simple text form: split out punctuation, formatting and add them to the text of the transcribed voicemails. Consider your email archives, they can be also good.

3) Then, you can just use MITLM toolkit to convert the texts you've collected to the language model

Download MITLM language model toolkit here

http://code.google.com/p/mitlm/

Run it as

estimate-ngram -text voicemail.txt -write-lm your_model.lm

It will create the language model model your_model.lm

4) Sometimes it make sense to mix your specific model with a generic model. It may help if your training text is small or your model is not good enough. To do that download a generic model here:

http://keithv.com/software/giga/lm_giga_5k_nvp_3gram.zip

Then unpack it and interpolate with your voicemail model using MITLM tools:

interpolate-ngram -lm "your_model.lm, lm_giga_5k_nvp_3gram.arpa" -interpolation LI -op voicemail.txt -wl Lectures+Textbook.LI.lm

See MITLM tutorial for details http://code.google.com/p/mitlm/wiki/Tutorial

lm_giga model is quite big, you can also pick hub4 language model for interpolation. To do that you need to convert it to text form from the binary form first:

sphinx_lm_convert -ifmt dmp -ofmt arpa hub4.5000.DMP hub4.lm

One day you will be able to work with a language model using CMU language model toolkit CMUCLMTK, but for now it's more complicated than MITLM. So MITLM is a recommended tool for language model operations.

4) To speedup the startup of the recognizer sort the model and convert it to a binary format:

sphinx_lm_sort < your_model.lm > your_model_sorted.lm

sphinx_lm_convert -i your_model_sorted.lm -o your_model_sorted.lm.dmp

3) In Pocketsphinx script, use your language model for transcription, add the following argument:

-lm your_model_sorted.lm.dmp

That's it.

Between, Google API is mostly trained on search queries. Why it perfectly suitable for voice search it's not good for voicemail transcription either. Voicemail transcription texts are usually quite sensitive information and it's very hard to get free access to them.

I think after this step the accuracy of the transcription is already good enough. You will be able to collect transcription results, fix them and use them to improve the language model.

Acoustic model
Sometimes it's usable to update the acoustic model. This step will require you to compile and setup Sphinxtrain. Again, transcribe few voicemails you've recorded, then organize them into a database. Then follow the acoustic model adaptation HOWTO as described in CMUSphinx wiki:

http://cmusphinx.sourceforge.net/wiki/tutorialadapt

Acoustic model adaptation always make sense but it's quite a time consuming process. Maybe one day someone will automate it to make it really flawless. For example we have started a project to help to train and adapt the model from the set of long files accompanied with text, not with a carefully drafted database. Once this project will be completed it will be way easier to train and adapt the acoustic models. Any help on this is appreciated.

Dictionary
There can be cases when you need to add few words to the dictionary which are missing. For example in step 1 when you adapted the language model you've got few words which are missing in cmu07a.dic. Then it make sense to add them. Just open a dictionary with a text editor, find the appropriate place and change or edit the phonetic pronunciation of the word. For example, CMU dictionary is missing the word "twitter"

twitter T W IH T ER

Usually this step is not needed but if you have for example an accented words or some other unusual words it may help.

Test the model
After you have adapted the model, retranscribe the files you have already collected. Check the accuracy if it's good or not.

Follow up
So here are the directions to take. I understand it's some work but maybe you consider it's worth the effort. We are really trying to make this process easier and your comments on that will be very appreciated.

CMUSphinx accepted at Google Summer Of Code 2011

2011-03-19T22:42:00.001+03:00

So we are in. Great to know that.

For more information see

http://cmusphinx.sourceforge.net/2011/03/cmusphinx-at-gsoc-2011/

I think it's a big responsibility and a big opportunity as well. Of course we don't consider this as a way to improve CMUSphinx itself or as something that will allow us to get features coded for free. Instead, we are looking for new people to join CMUSphinx, becoming the part of it. Maybe it's a great opportunity for Nexiwave as well.

For now the task is to prepare the list of ideas for the projects. I know they need to be drafted carefully. If you want to help, please jump in. I definitely need some help.

Fillers in WFST

2011-03-13T05:11:00.000+03:00

Another practical question is - how do you integrate fillers? There is silence class introduced in

A GENERALIZED CONSTRUCTION OF INTEGRATED SPEECH RECOGNITION TRANSDUCERS by Cyril Allauzen, Mehryar Mohri, Michael Riley and Brian Roark

and implemented in transducersaurus.

But you know each practical model has more than just a silence. Fillers like noise, silence, breath, laugh they all go to specific senones in the model. I usually try to minimize them during the training for example joining all them ums, hmms, and mhms into a single phone but I still think they are needed. How to integrate them when you build WFST recognizer?

So I tried few approaches. For example instead of adding just a <sil> class in T transducer I tried to create many branches for each filler. As a result final cascade expands to a huge moster. Like if cascade was 50mb after combination with 1 silence class it is 100mb but after 3-4 classes it's 300mb. Not a nice thing to do.

So I ended in dynamic expansion of silence transitions like this:

if edge is silence:
   for filler in fillers:
      from node.add_edge(filler)

This seems to work well.

Word position context dependency of Sphinxtrain and WFST

2011-03-03T19:56:00.002+03:00

Interesting thing about Sphinxtrain models is that it uses word position as a context when looking for a senone for a particular word sequence. That means that in theory a senone for the start word phones could be different from senones for the middle-word phones and senones for the end-word phones. It's actually sometimes the case:

ZH  UW  ER b    n/a   48   4141   4143   4146 N
ZH  UW  ER e    n/a   48   4141   4143   4146 N
ZH  UW  ER i    n/a   48   4141   4143   4146 N

but

AA  AE   F b    n/a    9    156    184    221 N
AA  AE   F s    n/a    9    149    184    221 N

Here in the WSJ model definition from sphinx4 a symbol in a fourth column means "beginning", "end", "internal" or "single" and the other characters are transition matrix ids and senone ids.

However, if you want to build WFST cascade from the model, it's kind of an issue how to embed the word position into context-dependent part of the cascade. My solution was to ignore position. You can ignore position in already prebuilt model since differences caused by word position are small, but to do it consistently it's better to retrain word-position-independent model.

Since of today you can do this easily, mk_mdef_gen tool supports -ignorewpos option which you can set in scripts. Basically everything is counted as an internal triphone. My tests show that this model is not worse than the original one. At least for conversational speech. Enjoy.

P.S. Want to learn more about WFST - read Paul Dixon's blog http://edobashira.com and Josef Novak's blog http://probablekettle.wordpress.com

Openfst troubleshooting

2011-02-21T22:29:00.002+03:00

A bit of openfst troubleshooting when you try to build WFST with Juicer. Say you are running

fstcompose ${OUTLEXBFSM} ${OUTGRAMBFSM} | \
fstepsnormalize | \
fstdeterminize | \
fstencode --encode_labels - $CODEX | \
fstminimize - | \
fstencode --decode - $CODEX | \
fstpush --push_weights | \
fstarcsort

and get this

FATAL: StringWeight::Plus: unequal arguments (non-functional FST?)

Huh? Which arguments are not equal? What caused this? How to fix this? Definitely it should be more self-explaining. That's basically quite a common issue. You get just a short message that nobody including the author could understand. Go find out how to fix it.

In this particular case you go to the openfst sources and change the following line:

  if (w1 != w2)
   LOG(FATAL) << "StringWeight::Plus: unequal arguments "
   << "(non-functional FST?) " << w1 << " " << w2;

Wait another half an hour for it to compile (who decided to make it with pure templates!). See that it outputs arguments now at least. You run again and get

FATAL: StringWeight::Plus: unequal arguments (non-functional FST?) 833_9 832_9

Heh, also not very descriptive but at least some hint. Looking on the output states 833 and 832 you see

that they have identical pronunciation. That's it. Your dictionary shouldn't have identical pronunciation. Moreover, it shouldn't have identically pronounced trigrams. Things pronounced like "a b cd" vs "ab c d" make wfst non-deterministic. Why didn't it warn about the issue when it converted the dictionary? Who knows. Anyway, now you can read about lexgen and find the option to fight with identical pronunciation:

-outputAuxPhones -> indicates that auxiliary phones should be added to pronunciationsin the lexicon in order to disambiguate distinct words withidentical pronunciations

This option should make things better.

I must admit CMUSphinx is also full of this. Bad error messages which doesn't describe the problem nor hint the solution. Compare too the output of recent maven

[ERROR] No goals have been specified for this build. You must specify a valid lifecycle phase or a goal in the format : or :[:]:. Available lifecycle phases are: validate, initialize, generate-sources, process-sources, generate-resources, process-resources, compile, process-classes, generate-test-sources, process-test-sources, generate-test-resources, process-test-resources, test-compile, process-test-classes, test, prepare-package, package, pre-integration-test, integration-test, post-integration-test, verify, install, deploy, pre-clean, clean, post-clean, pre-site, site, post-site, site-deploy. -> [Help 1]

[ERROR]

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR]

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/NoGoalSpecifiedException

Maybe it's too verbose but I think it's the right way to do. So if you see something that is not clear in CMUSphinx, please report about it. We'll happily fix it.

Coming up next - what to do when openfst hangs or takes all your memory.

Looking on the waves

2011-02-18T00:13:00.001+03:00

Here is the question - a perfectly looking sound file which is transcribed with 10% accuracy. Sounds crazy, isn't it? Click on it to enlarge. No noise, no accent.

Because of that I'm looking on state-of-art in channel normalization, especially for non-linear channel distortions. No good solution yet, I've only found the description of the problem in very old paper

SOURCES OF DEGRADATION OF SPEECH RECOGNITION IN THE TELEPHONE NETWORK Pedro J. Moreno and Richard M. Stern From the Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, Adelaide, Australia, Vol. I., pp. 109 - 112, April, 1994.

There is CDCN normalization, few CMN improvements, RASTA and even recently invented HN normalization. CDCN is suprisingly available in Sphinxtrain but nobody uses it. Well it gives no improvement but it's an interesting approach worth to document one day. The idea to collect statistics from the speech to apply it later sounds nice.

There are model-level approaches, various feature transforms, adaptations. They do not really look that attractive. Most papers now deal with channel compensation for speaker recognition, not speech recognition. I must admit the topic is too large to overview it in few weeks.

Luckily, I can also spend time looking on the waves like the one on the right. Somewhat more pleasant I would say.

Some more optimization

2011-01-15T01:05:00.001+03:00

In addition two the previous post, two more tricks for log_diag_eval.

Floats instead of double

If accumulator is float, SSE could be used more effectively

Hardcode vector length

The most common optimizaition is loop unrolling. It helps to optimize memory access as well as eliminates jump commands. But the issue here is that number of iterations in log_diag_eval can be different on various stages. GCC has interesting profile-based optimizaition for this case, see -fprofile-generate option. It runs a program and then can derive few specific optimizations form the runtime. Good point is that we actually can be almost sure in usage patters of the our target loop, so we can optimize without profiling. So, turn

for (i=0;i<veclen;i++) {
   do work
}

to

if (veclen == 40) { // Common used value, 40 floats in each frame
   for (i=0;i<40;i++) {
   do work // This will be unrolled
   } else {
   for (i=0;i<veclen;i++)
   do work
   }
}

GCC does same trick with profiler, but since our feature frame size is fixed, we can hardcode. As a result GCC will unroll first loop and it will be fast as a wind

Optimization in SphinxTrain

2011-01-13T00:48:00.003+03:00

I spend quite significant amount of time training various models. It feels like alchemy, you add this and tune there and you get nice results. And while training you can read twitter ;) I'm also 10 years in a group which is creating optimizing compilers so in theory I should know a lot about them. I rarely apply it in practice though. But being bored with several weeks training you can apply some knowledge here.

So the algorithm:

1) Train a model for a month and become bored
2) Get an idea that SphinxTrain is compiled without optimization
3) Go to SphinxTrain/config and change compilation option from -O2 to -O3
4) Measure run time of a simple bw run with time command
5) See that time doesn't really change
6) Add -pg option to CFLAGS and LDFLAGS to collect profile
7) See most of the time we are running log_diag_eval function which is a simple weighted dot product computation
8) See the assembler code of the log_diag_eval

0x42c3b0 log_diag_eval: unpcklps %xmm0,%xmm0
0x42c3b3 log_diag_eval+3: test   %ecx,%ecx
0x42c3b5 log_diag_eval+5: cvtps2pd %xmm0,%xmm0
0x42c3b8 log_diag_eval+8: je     0x42c3fd log_diag_eval+77
0x42c3ba log_diag_eval+10: sub    $0x1,%ecx
0x42c3bd log_diag_eval+13: xor    %eax,%eax
0x42c3bf log_diag_eval+15: lea    0x4(,%rcx,4),%rcx
0x42c3c7 log_diag_eval+23: nopw   0x0(%rax,%rax,1)
0x42c3d0 log_diag_eval+32: movss  (%rdi,%rax,1),%xmm1
0x42c3d5 log_diag_eval+37: subss  (%rsi,%rax,1),%xmm1
0x42c3da log_diag_eval+42: unpcklps %xmm1,%xmm1
0x42c3dd log_diag_eval+45: cvtps2pd %xmm1,%xmm2
0x42c3e0 log_diag_eval+48: movss  (%rdx,%rax,1),%xmm1
0x42c3e5 log_diag_eval+53: add    $0x4,%rax
0x42c3e9 log_diag_eval+57: cmp    %rcx,%rax
0x42c3ec log_diag_eval+60: cvtps2pd %xmm1,%xmm1
0x42c3ef log_diag_eval+63: mulsd  %xmm2,%xmm1
0x42c3f3 log_diag_eval+67: mulsd  %xmm2,%xmm1
0x42c3f7 log_diag_eval+71: subsd  %xmm1,%xmm0
0x42c3fb log_diag_eval+75: jne    0x42c3d0 log_diag_eval+32
0x42c3fd log_diag_eval+77: repz retq

9) Understand that it's not really as good here as it can be

10) Run

gcc -DPACKAGE_NAME=\"SphinxTrain\" -DPACKAGE_TARNAME=\"sphinxtrain\" \
-DPACKAGE_VERSION=\"1.0.99\" -DPACKAGE_STRING=\"SphinxTrain\ 1.0.99\" \
-DPACKAGE_BUGREPORT=\"\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 \
-DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 \
-DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_LIBM=1 \
-I/home/nshmyrev/SphinxTrain/../sphinxbase/include \
-I/home/nshmyrev/SphinxTrain/../sphinxbase/include   -I../../../include -O3 \
-g -Wall -fPIC -DPIC -c gauden.c -o obj.x86_64-unknown-linux-gnu/gauden.o \
-ftree-vectorizer-verbose=2

to see that log_diag_eval loop isn't vectorized

11) Add -ffast-math and see it doesn't help

12) Rewrite function from

float64
log_diag_eval(vector_t obs,
    float32 norm,
    vector_t mean,
    vector_t var_fact,
    uint32 veclen)
{
    float64 d, diff;
    uint32 l;

    d = norm;   /* log (1 / 2 pi |sigma^2|) */

    for (l = 0; l < veclen; l++) {
        diff = obs[l] - mean[l];
        d -= var_fact[l] * diff * diff; /* compute -1 / (2 sigma ^2) * (x - m) ^ 2 terms */
    }

    return d;
}

log_diag_eval(vector_t obs,
    float32 norm,
    vector_t mean,
    vector_t var_fact,
    uint32 veclen)
{
    float64 d, diff;
    uint32 l;

    d = 0.0;

    for (l = 0; l < veclen; l++) {
        diff = obs[l] - mean[l];
        d += var_fact[l] * diff * diff; /* compute -1 / (2 sigma ^2) * (x - m) ^ 2 terms */
    }

    return norm - d;    /* log (1 / 2 pi |sigma^2|) */
}

to turn substraction which hurts to accumulation.

13) See that loop is now vectorized. Enjoy the speed!!!

The key thing to understand here is that programming is rather flexible and compilers are rather dumb. But you have to cooperate. So you need to use very simple constructs to let compiler do his work. Moreover, this idea of using simple constructs in the code has other benefits since it helps to keep code style clean and enables automated static analysis with tools like splint.

Maybe same applies to speech recognition. We need to help computers in their efforts to understand us. Speak slowly and articulate clearly and both we and computers will enjoy the result

If you are interested about loop vectorization in GCC, see here http://gcc.gnu.org/projects/tree-ssa/vectorization.html

Mary TTS 4.3.0 released

2010-12-22T01:59:00.000+03:00

With Russian voice from Voxforge DB. Yay! Try it on the web:

http://mary.dfki.de:59125/

Mary is definitely superious system comparing to Festival. Graphical UI, modern design, support for various things like automatic dictionary creation make it really easy to build a language support. And due to modular and stable codebase one can easily add support for new feature, integrate with external package like it's done with OpenNLP or just fix the bug. And your fix will be accepted!

There are two Voxforge TTS datasets pending between - German and Dutch and also there is a Polish voice. If anyone wants to try that, it must be really easy to add other language to Mary.

Phoneset with stress

2010-12-15T15:26:00.000+03:00

So I finally finished testing of the stress-aware model. It took me few month and the end I could say that lexical stress is definitely better. It provides better accuracy and, more importantly, more robustness over model which has non-stressed phoneset.

I hope we retrain all other models we have with the phoneset with stress. It's great that CMUDict provides enough information to do that. The story of me testing that was quite interesting. I believed in stress for a long time but wasn't able to prove that. In theory it's clear why it helps, when speech speed changes, stressed syllables remain less corrupted than unstressed and we get better control over data. Additional information like lexical stress is important. Of course the issue is the increased number of parameters to train the model. That's why I think early investigations concluded that phoneset without stress is better. Discussion about it on cmusphinx-devel this summer also confirmed Nuance moved to the model with stress in their automotive decoder.

It's interesting how long I tested that. I made numerous attempts and each one had bugs

First attempt was using bad features (adapted for 3gp) and didn't show any improvement
Number of senones in second training was too small since I didn't know the reason of first failure
Third attempt had issue with the automatic questions which were used accidentally instead of manual ones I wrote and it went unnoticed
Fourth attempt was rejected because there were issues with the dictionary format in Sphinx4. Never use FastDictionary between, use FullDictionary. Fast dictionary expects specific dictionary format with variants like (2) (3) (4) and not (1) or (2) and (4).
Only fifth attempt was good but in shown improvement only on big test set and not on the small one

So basically to check every fact you need to be very careful and double- or triple-check everything. Bugs are everywhere, in language model training, decoder, trainer, configuration. From run to run bugs could lead to different results, even a small change can break everything. I think optimal way for research could be to check the same proposition in independent teams using independent decoders and probably different data. Not sure if it's doable in short term.

On CMUCLMTK

2010-11-29T18:25:00.005+03:00

I've rebuilt the Nexiwave langauge models and meet some issues which would be nice to solve one day. CMU language model tookit is a nice simple piece of software but it definitely lacks many features which are required to build a good language model. So thinking about features language modelling toolkit can provide I created a list.

None of the toolkits have all the features in place which leads to unneceessary java coding, perl coding and python coding every time I rebuild the model. It would be nice to see that list turned into software one day.

Language is a live constantly changing structure. Very interesting is that in 2006 nobody used works like "twitter", "obama", "facebook". New words like "ipad" arrive every day and become very actively used. That makes senselsess to collect any static databases to train the language model. If pronunciation and acoustic changes slowly (well, not so slowly now. with Skype used everywhere acoustic of speech changed very significantly over last years!) language model needs to be very quickly adapted to the date. Other problem is that Sphinx4 is not very robust to unknown words. If unknown word is met, it bascailly screw the whole utterance around. That's why it's important to have up-to-date vocabulary. Maybe it can be small dynamic language model combined with huge static one, I'm not sure.

It can be clearly seen that Gigaword doesn't have enough coverage of modern terms. Models like lm_giga are nice, but they only work for old books. We need something live.

Another issue is where to find the texts. Unfortunately very relevant spoken transcriptions aren't available. Only companies which manually transcribe speech have them I think. Every written text is very different from spoken one.

So we need to collect data from realtime, from Twitter, Facebook, Google, Wikipedia, Youtube, from the net. We also need to be able to process this data, classify it, convert it to a spoken form and train the language model on the result. Issues here are that the crawled text is often corrupted, has spelling issues and spam. That's a huge task to make it usable.

Google and Bing use brut-force approach here, they just collect everything and hope it will be good enough. That can be seen on their n-grams data. Not sure if this approach is helpful for ASR.

So the features to see:

Automated spelling errors detection and correction for unified written form. That includes automated abbreviation and numbers expansion. For which you must have good NLP component to be able to identify part of speech and other properties. Interesting here is that unified written form must be spelling-oriented. For example there are differently pronounced "going to" and "gonna" while from the language point of view they are identical. I haven't decided what to do with "gonna" yet.
Automatic vocabulary selection. While in theory decoder should operate with unlimited vocabulary, in pratice it's better to have smaller one but with a good coverage. It's very important to be able to filter common spelling errors here.
Tookit should support crawling from major sources like from Twitter, Wikipedia, other sources.
Though NLP is mentioned above, I think it shoudln't be only used for expansion. Tookit should support many NLP features to be able to create more complex language model than simple n-grams.

I think that's a myth that n-gram model describe language well. Language model is not effective enough in rejection of the transcriptions that aren't possible at all in the language. We already changed the decoder to penalize trigrams which aren't common in a language model and require backoff, but this change appread to be not effective enough.

There is a nice idea of discriminative correction in Joshua machine translation toolkit for example

Large-scale Discriminative n-gram Language Models for Statistical Machine Translation
Zhifei Li and Sanjeev Khudanpur

The paper has quite interesting problem statement to correct the decoding results which are not possible in the language but solution that gives only 5% improvement is certainly not worth attention. We need to embed language knowledge deep in the search.

Toolkit must be designed for parallel hardware in mind. Everything is getting distributed and with information volumes it's a hard requirement to be able to process data in parallel.

Quite a long list to be honest. Few years of coding on the top of cmuclmtk. It needs to be done anyway.

Paper on subject:

Ivan Bulyko, Mari Ostendorf, Manhung Siu, Tim Ng, Andreas Stolcke Web resources for language modeling in conversational speech recognition

Decoding of Compressed Low-Bitrate Speech

2010-11-27T17:44:00.003+03:00

I've spent some time on optimizing accuracy for 3gp speech recordings from mobile phones. 3gp is a container format used on most mobile devices nowdays with speech compressed using AMR-NB inside. Converted audio to AMR-NB and back, extracted PLP features and then trained few models on that. Result is not encouraging - accuracy is worse than stock model both on original and on compressed/decompressed audio. Not much worse but significanly worse.

Looks like traditional HMM issues like frame independency assumption play here which is confirmed by the papers I found. This paper is quite useful for example:

Vladimir Fabregas Surigué de Alencar and Abraham Alcaim. On the Performance of ITU-T G.723.1 and AMR-NB Codecs for Large Vocabulary Distributed Speech Recognition in Brazilian Portuguese

And this paper is good too:

Patrick Bauer, David Scheler, Tim Fingscheidt. WTIMIT: The TIMIT Speech Corpus Transmitted Over the 3G AMR Wideband Mobile Network

Need to research more on subject. Suprisingly there are only few papers on the subject, way less than on reverberation. It looks we have to build specialized frontend specifically targetted on decoding of low-bitrate compressed speech. Or we need to move to more robust features than PLP.

For now I would state the problem to develop a speech recognition framework to provide good accuracy on:

Unmodified speech
Noise-corrupted speech
Music-corrupted speech
Codec-corrupted speech
Long-distance speech

Good system should decode well in all cases.

Updates in SphinxTrain

2010-11-23T01:07:00.000+03:00

Being tired to explain build issues over and over I found the passion to step in and start a sequence of major changes in SphinxTrain

Ported sphinxtrain to automake, development branch you can try is here: https://cmusphinx.svn.sourceforge.net/svnroot/cmusphinx/branches/sphinxtrain-automake
Will increase SphinxTrain dependency on sphinxbase, unifying the duplicated sources.
Will make training use external SphinxTrain installation, no setup in training folder will be required, only configuration. All scripts will be in share and in libdir, they will be installed systemwide. To try a new version one will just need to change path to sphinxtrain.
Will modify scripts to be able to build and test the database using a single command. No possibility to miss anything!
Will include automation for language weight optimization on a development set, better model training scripts will do everything required.

I know Autotools aren't the best build system, but they are pretty straghtforward. More importantly, the tools will follow common Unix practices and thus will be easier to use and understand.

Comments are welcome!

P.S.

We've done a great progress on Nexiwave also. Check it out!

Backward Compatibility Issues

2010-11-16T19:42:00.001+03:00

Just today I spent few hours trying to figure out why changed makeinfo version output broke binutils build. Well, it's an old bug but we all getting mad when backward compatibility breaks. Especially when it affects our software. Especially when we don't have time no passion to fix that. My complains raised to the roof or probably even higher.

Life is a strange thing. Right after that I went ahead a broke sphinx4 backward compatibility in model packaging (again!). Now models distributed with sphinx4 follow Sphinxtrain output format, all files are in the single folder, model definition is named simply "mdef" and there is feat.params. Things are very
straightforward:

[shmyrev@gnome sphinx4]$ ls models/acoustic/wsj

dict         license.terms  means            noisedict  transition_matrices

feat.params  mdef           mixture_weights  README     variances

It will certainly help to avoid confusion when new developers change the model, adapt the model or train their own one.

In the future I hope to get feat.params used better in order to automatically build frontend, derive feature extraction properties, hold metadata about model and similar things. Shiny future is getting closer.

I also removed RM1 model from the distribution. I don't think anybody is using it.

So please don't complain, let's better fix that until it's too late to fix. One day we'll get everything in place and we'll release final version sphinx4-1.0. And after that we'll certainly be backward-compatible. I really like Java and Windows because of their long-term backward-compatible policy. We can do even better.

Speech Like WWW

2010-10-27T16:43:00.001+04:00

Talking about custom speech application development I've got a thought. There are quite many speech companies already. Speech application development is actually quite similar to UI design or to web design in sense that you need to be specialized expert in order to create speech interface. What if speech developers will be like web designers - thousands of them every day build customized websites all over the world? What if market is so huge that it will be possible to run hundred shops each working on customer needs.

To be honest I don't quite like web development. It sounds very strange for me that you MUST pay at least $1000 to build something that is pleasantly looking. And for big websites its way more. Whoever created this market didn't think about business, he designed HTML in order to drain money from small and big companies. I tried to create few websites myself, for example CMUSphinx website. Even with all modern tools, CMS platforms, themes and stuff the reality is that you need to be an expert. Otherwise the result will not be satisfactory enough. Menus will overlap, regions will not be aligned, pictures will be blurry and colors will not match. Can it be different? Certainly it can, but not in this world. I can understand that creativity can't be automated, I can't understand that creativity is required for every company.

There are some similar things in software development like for example if you want to develop a telco app you probably want to hire Asterisk developer. And there are thousands Asterisk developers out there. Or if you want JBoss you could find JBoss experts. But I think if you know how to develop applications you can configure Asterisk properly and you can create a bean to acess the database.

Are we interested in creation of huge and diverse speech industry? Can CMUSphinx be the basement for it? No definite answer for now.

Recent issues

2010-10-22T02:54:00.001+04:00

Heh, this month I discovered few critical issues in CMUSphinx.

Pocketsphinx doesn't properly decode short silences in FSG/JSGF mode
Sphinx4 doesn't really work with OOV loop in grammar
Pocketsphinx n-best lists are useless because of too many repeated entries
Pocketsphinx accuracy is way lower than sphinx3 one
Supposedly-working sphinxbase LM stuff doesn't work with 32-bit DMP, thus no MMIE training for very large vocabulary
MMIE itself doesn't improve accuracy (tested on Voxforge and Fisher)
It's impossible to extract mixture_weights from recent sendumps in pocketsphinx models, python scripts in SphinxTrain are outdated
PTM model adaptation doesn't work
TextAligner demo from sphinx4 requires way more work to align properly

That's getting crazy, I wonder if I'll be able to find the time to fix all that.

Do You Want To Talk To Your Computer?

2010-10-08T00:34:00.000+04:00

Thanks everyone who voted at

To be honest I was suprised because my opinion is just the reverse of this result. I strongly disagree that command and control will be ever usable. Dictation probably will, but definitely not command and control. I have a hobby - collecting complains about voice control. Here are few ones

As article in PCWorld says

If so, you'll love what Microsoft is offering: voice recognition over the air, in which your commands are processed by a server in the clouds and converted into action on your smartphone. Boy, let's burn up those minutes and data plans! And waaait for the slow, usually incorrect response. Android has a similar capability for search, and it's amazingly frustrating to use, not to mention inaccurate.

The one good thing about Microsoft's fantasy about voice-command interfaces: You'll be able to identify a Windows Phone 7 user easily. Just listen for the person pleading with the phone to do what he asked. Whie the rest of us are quietly computing and communicating, he'll be hard to miss.

Another post from CNET

One of the major reasons why speech recognition has not caught on or been seriously looked at in terms of major finances is because people, if given the option of accurate speech recognition too, would still not wanna go for the voice commands, but would rather just use touchscreen. This is because voicing out takes more energy off a person than smoothly running yer fingers on the screen in your hand. Intuitive touchscreens and cleaner interfaces are far better a tool to invest in than making people accustomed to say out words that computers need to understand, process and then implement. Its way easier for the user (in terms of energy used to say it out) to just press the button on touchscreen. There will be certain exceptions, but I'm talking on a mass consumer adoption assumption.

I strongly believe that when we want to communicate with computer, there are better ways than to give them voice commands. Yes, speech is a natural way of communication but it's a communication between people. When you communicate with machine you don't necessary need to speak to it, there are more efficient ways. Even if you are driving.

On the other side I think that analytics, speech mining and similar stuff do have a very shiny future. According to DMG consulting the growth of this market will be 42 percent in 2011. That's a true potential. Speech recognition should seemlessly plug into comunication between people and extract value from it. Being non-intrusive it doesn't break patterns but helps to create the information. That's why we invest so much into mining and not into command and control. That's also the reason I don't want to invest too much time in gnome-voice-control.

No More Word Error Rate

2010-10-03T00:35:00.000+04:00

Reading http://delong.typepad.com/sdj/2010/09/when-speech-recognition-software-attacks.html

What he said:

Hi Brad, it's Mike. I had a lunchtime appointment go long and I am bolting back to Evans. I'll be there shortly. See you soon. Thanks.

What Google Voice heard:

That it's mike. I had a list of women go a long and I am old thing. Back evidence. I'll be there for me to you soon. Thanks.

The interesting thing is that it got 17 out of the 26 words right--but those 17 words convey almost none of the information in the message...

I found this paper

Is Word Error Rate a Good Indicator for Spoken Language Understanding Accuracy
Ye-Yi Wang and Alex Acero
2003

It is a conventional wisdom in the speech community that better speech recognition accuracy is a good indicator for better spoken language understanding accuracy, given a fixed understanding component. The findings in this work reveal that this is not always the case. More important than word error rate reduction, the language model for recognition should be trained to match the optimization objective for understanding. In this work, we applied a spoken language understanding model as the language model in speech recognition. The model was obtained with an example-based learning algorithm that optimized the understanding accuracy. Although the speech recognition word error rate is 46% higher than the trigram model, the overall slot understanding error can be reduced by as much as 17%.

We definitely need to address it in sphinx4.

Reading Interspeech 2010 Program

2010-10-01T03:12:00.003+04:00

Luckily speech people don't have so many conferences. In machine learning world it seems it's getting crazy. You can have conference every month. Researchers travel more than sales managers. In speech there are ASRU, ICASSP but they don't really matter. It's enough to track Interspeech. Since Tokio is too far, I'm just reading the abstract list from the program. First impressions are:

Keynotes are all boring
Interesting rise of the subject "automatic error detection in unit selection". At least three! papers are presented on the subject while I haven't seen any of them before. Looks like idea appeared in less then a year! Are they spying each other?
RWTH Aachen presented enormous amount of papers, LIUM is also quite fruitful
Well, IBM T. J. Watson Research Center is active as well, but thats more a tradition
I've met in one paper: "yields modest gains on the English broadcast news RT-04 task, reducing the word error rate from 14.6% to 14.4%" Was it worth writing an article?
Cognitive status assessment from speech is important in dialogs. SRI is doing that
Strange that reverberation issues are a separate class of problems to solve and largely covered.The problem as a whole looks rather generic - create noise and corruption-stable features. Not sure how reverberation is special here
WFST is loudly mentioned
Andreas Stolke on SRILM noted that pruning doesn't work with KN-smoothed model! Damn, I was using it
Only 2 Russian papers on the whole conference. Well, it's 50% growth to previous year. And one of them is on speech recognition, that's definitely a progress
Suprisingly not so much research on confidence measures! Confidence is a REALLY IMPORTANT THING

Reading the abstracts I also selected some papers which could be interesting for Nexiwave. Probably you'll find this list easier to read than 200 papers from original program. Let's hope this list will be useful for me as well. To be honest I didn't manage to read the papers I selected previous year from Interspeech 2009.

Unsupervised Discovery and Training of Maximally Dissimilar Cluster Models
Francoise Beaufays (Google)
Vincent Vanhoucke (Google)
Brian Strope (Google)
One of the difficult problems of acoustic modeling for Automatic Speech Recognition (ASR) is how to adequately model the wide variety of acoustic conditions which may be present in the data. The problem is especially acute for tasks such as Google Search by Voice, where the amount of speech available per transaction is small, and adaptation techniques start showing their limitations. As training data from a very large user population is available however, it is possible to identify and jointly model subsets of the data with similar acoustic qualities. We describe a technique which allows us to perform this modeling at scale on large amounts of data by learning a tree-structured partition of the acoustic space,and we demonstrate that we can significantly improve recognition accuracy in various conditions through unsupervised Maximum Mutual Information (MMI) training. Being fully unsupervised, this technique scales easily to increasing numbers of conditions.

Techniques for topic detection based processing in spoken dialog systems
Rajesh Balchandran (IBM T J Watson Research Center)
Leonid Rachevsky (IBM T J Watson Research Center)
Bhuvana Ramabhadran (IBM T J Watson Research Center)
Miroslav Novak (IBM T J Watson Research Center)
In this paper we explore various techniques for topic detection in the context of conversational spoken dialog systems and also propose variants over known techniques to address the constraints of memory, accuracy and scalability associated with their practical implementation of spoken dialog systems. Tests were carried out on a multiple-topic spoken dialog system to compare and analyze these techniques. Results show benefits and compromises with each approach suggesting that the best choice of technique for topic detection would be dependent on the specific deployment requirements.

A Hybrid Approach to Robust Word Lattice Generation Via Acoustic-Based Word Detection
Icksang Han (Samsung Advanced Institute of Technology, Samsung Electronics Co., Ltd.)
Chiyoun Park (Samsung Advanced Institute of Technology, Samsung Electronics Co., Ltd.)
Jeongmi Cho (Samsung Advanced Institute of Technology, Samsung Electronics Co., Ltd.)
Jeongsu Kim (Samsung Advanced Institute of Technology, Samsung Electronics Co., Ltd.)
A large-vocabulary continuous speech recognition (LVCSR) system usually utilizes a language model in order to reduce the complexity of the algorithm. However, the constraint also produces side-effects including low accuracy of the out-of-grammar sentences and the error propagation of misrecognized words. In order to compensate for the side-effects of the language model, this paper proposes a novel lattice generation method that adopts the idea from the keyword detection method. By combining the word candidates detected mainly from the acoustic aspect of the signal to the word lattice from the ordinary speech recognizer, a hybrid lattice is constructed. The hybrid lattice shows 33% improvement in terms of the lattice accuracy under the condition where the lattice density is the same. In addition, it is observed that the proposed model shows less sensitivity to the out-of-grammar sentences and to the error propagation due to misrecognized words.

Time Condition Search in Automatic Speech Recognition Reconsidered
David Nolden (RWTH Aachen)
Hermann Ney (RWTH Aachen)
Ralf Schlueter (RWTH Aachen)
In this paper we re-investigate the time conditioned search (TCS) method in comparison to the well known word conditioned search, and analyze its applicability on state-of-the-art large vocabulary continuous speech recognition tasks. In contrast to current standard approaches, time conditioned search offers theoretical advantages particularly in combination with huge vocabularies and huge language models, but it is difficult to combine with across word modelling, which was proven to be an important technique in automatic speech recognition. Our novel contributions for TCS are a pruning step during the recombination called Early Word End Pruning, an additional recombination technique called Context Recombination, the idea of a Startup Interval to reduce the number of started trees, and a mechanism to combine TCS with across word modelling. We show that, with these techniques, TCS can outperform WCS on a current task.

Direct Construction of Compact Context-Dependency Transducers From Data
David Rybach (RWTH Aachen University, Germany)
Michael Riley (Google Inc., USA)
This paper describes a new method for building compact context-dependency transducers for finite-state transducer-based ASR decoders. Instead of the conventional phonetic decision-tree growing followed by FST compilation, this approach incorporates the phonetic context splitting directly into the transducer construction. The objective function of the split optimization is augmented with a regularization term that measures the number of transducer states introduced by a split. We give results on a large spoken-query task for various n-phone orders and other phonetic features that show this method can greatly reduce the size of the resulting context-dependency transducer with no significant impact on recognition accuracy. This permits using context sizes and features that might otherwise be unmanageable.

On the relation of Bayes Risk, Word Error, and Word Posteriors in ASR
Ralf Schlueter (Lehrstuhl fuer Informatik 6 - Computer Science Department, RWTH Aachen University)
Markus Nussbaum-Thom (Lehrstuhl fuer Informatik 6 - Computer Science Department, RWTH Aachen University)
Hermann Ney (Lehrstuhl fuer Informatik 6 - Computer Science Department, RWTH Aachen University)
In automatic speech recognition, we are faced with a well-known inconsistency: Bayes decision rule is usually used to minimize sentence (word sequence) error, whereas in practice we want to minimize word error, which also is the usual evaluation measure. Recently, a number of speech recognition approaches to approximate Bayes decision rule with word error (Levenshtein/edit distance) cost were proposed. Nevertheless, experiments show that the decisions often remain the same and that the effect on the word error rate is limited, especially at low error rates. In this work, further analytic evidence for these observations is provided. A set of conditions is presented, for which Bayes decision rule with sentence and word error cost function leads to the same decisions. Furthermore, the case of word error cost is investigated and related to word posterior probabilities. The analytic results are verified experimentally on several large vocabulary speech recognition tasks.

Efficient Data Selection for Speech Recognition Based on Prior Confidence Estimation Using Speech and Context Independent Models
Satoshi KOBASHIKAWA (NTT Cyber Space Laboratories, NTT Corporation)
Taichi ASAMI (NTT Cyber Space Laboratories, NTT Corporation)
Yoshikazu YAMAGUCHI (NTT Cyber Space Laboratories, NTT Corporation)
Hirokazu MASATAKI (NTT Cyber Space Laboratories, NTT Corporation)
Satoshi TAKAHASHI (NTT Cyber Space Laboratories, NTT Corporation)
This paper proposes an efficient data selection technique to identify well recognized texts in massive volumes of speech data. Conventional confidence measure techniques can be used to obtain this accurate data, but they require speech recognition results to estimate confidence. Without a significant level of confidence, considerable computer resources are wasted since inaccurate recognition results are generated only to be rejected later. The technique proposed herein rapidly estimates the prior confidence based on just an acoustic likelihood calculation by using speech and context independent models before speech recognition processing; it then recognizes data with high confidence selectively. Simulations show that it matches the data selection performance of the conventional posterior confidence measure with less than 2 % of the computation time.

Discovering an Optimal Set of Minimally Contrasting Acoustic Speech Units: A Point of Focus for Whole-Word Pattern Matching
Guillaume Aimetti (University of Sheffield)
Roger Moore (Universty of Sheffield)
Louis ten Bosch (Radboud University)
This paper presents a computational model that can automatically learn words, made up from emergent sub-word units, with no prior linguistic knowledge. This research is inspired by current cognitive theories of human speech perception, and therefore strives for ecological plausibility with the desire to build more robust speech recognition technology. Firstly, the particulate structure of the raw acoustic speech signal is derived through a novel acoustic segmentation process, the `acoustic DP-ngram algorithm'. Then, using a cross-modal association learning mechanism, word models are derived as a sequence of the segmented units. An efficient set of sub-word units emerge as a result of a general purpose lossy compression mechanism and the algorithms sensitivity to discriminate acoustic differences. The results show that the system can automatically derive robust word representations and dynamically build re-usable sub-word acoustic units with no pre-defined language-specific rules.

Modeling pronunciation variation using context-dependent articulatory feature decision trees
Samuel Bowman (Linguistics, The University of Chicago)
Karen Livescu (TTI-Chicago)
We consider the problem of predicting the surface pronunciations of a word in conversational speech, using a feature-based model of pronunciation variation. We build context-dependent decision trees for both phone-based and feature-based models, and compare their perplexities on conversational data from the Switchboard Transcription Project. We find that feature-based decision trees using featur e bundles based on articulatory phonology outperform phone-based decision trees, and are much more r obust to reductions in training data. We also analyze the usefulness of various context variables.

Accelerating Hierarchical Acoustic Likelihood Computation on Graphics Processors
Pavel Kveton (IBM)
Miroslav Novak (IBM)
The paper presents a method for performance improvements of a speech recognition system by moving a part of the computation - acoustic likelihood computation - onto a Graphics Processor Unit (GPU). In the system, GPU operates as a low cost powerful coprocessor for linear algebra operations. The paper compares GPU implementation of two techniques of acoustic likelihood computation: full Gaussian computation of all components and a significantly faster Gaussian selection method using hierarchical evaluation. The full Gaussian computation is an ideal candidate for GPU implementation because of its matrix multiplication nature. The hierarchical Gaussian computation is a technique commonly used on a CPU since it leads to much better performance by pruning the computation volume. Pruning techniques are generally much harder to implement on GPUs, nevertheless, the paper shows that hierarchical Gaussian computation can be efficiently implemented on GPUs.

The AMIDA 2009 Meeting Transcription System
Thomas Hain (Univ Sheffield)
Lukas Burget (Brno Univ. of Technology)
John Dines (Idiap)
Philip N. Garner (Idiap)
Asmaa El Hannani (Univ. Sheffield)
Marijn Huijbregts (Univ. Twente)
Martin Karafiat (Brno Univ. of Technology)
Mike Lincoln (Univ. of Edinburgh)
Wan Vincent (Univ. Of Sheffield)
We present the AMIDA 2009 system for participation in the NIST RT'2009 STT evaluations. Systems for close-talking, far field and speaker attributed STT conditions are described. Improvements to our previous systems are: segmentation and diarisation; stacked bottle-neck posterior feature extraction; fMPE training of acoustic models; adaptation on complete meetings; improvements to WFST decoding; automatic optimisation of decoders and system graphs. Overall these changes gave a 6-13% relative reduction in word error rate while at the same time reducing the real-time factor by a factor of five and using considerably less data for acoustic model training.

A FACTORIAL SPARSE CODER MODEL FOR SINGLE CHANNEL SOURCE SEPARATION
Robert Peharz (Graz University of Technology)
Michael Stark (Graz University of Technology)
Franz Pernkopf (Graz University of Technology)
Yannis Stylianou (University of Crete)
We propose a probabilistic factorial sparse coder model for single channel source separation in the magnitude spectrogram domain. The mixture spectrogram is assumed to be the sum of the sources, which are assumed to be generated frame-wise as the output of sparse coders plus noise. For dictionary training we use an algorithm which can be described as non-negative matrix factorization with ℓ0 sparseness constraints. In order to infer likely source spectrogram candidates, we approximate the intractable exact inference by maximizing the posterior over a plausible subset of solutions. We compare our system to the factorial-max vector quantization model, where the proposed method shows a superior performance in terms of signal-to-interference ratio. Finally, the low computational requirements of the algorithm allows close to real time applications.

ORIENTED PCA METHOD FOR BLIND SPEECH SEPARATION OF CONVOLUTIVE MIXTURES
Yasmina Benabderrahmane (INRS-EMT Telecommunications Canada)
Sid Ahmed Selouani (Université de Moncton Canada)
Douglas O’Shaughnessy (INRS-EMT Telecommunications Canada)
This paper deals with blind speech separation of convolutive mixtures of sources. The separation criterion is based on Oriented Principal Components Analysis (OPCA) in the frequency domain. OPCA is a (second order) extension of standard Principal Component Analysis (PCA) aiming at maximizing the power ratio of a pair of signals. The convolutive mixing is obtained by modeling the Head Related Transfer Function (HRTF). Experimental results show the efficiency of the proposed approach in terms of subjective and objective evaluation, when compared to the Degenerate Unmixing Evaluation Technique (DUET) and the widely used C-FICA (Convolutive Fast-ICA) algorithm

Speaker Adaptation Based on System Combination Using Speaker-Class Models
Tetsuo Kosaka (Yamagata University)
Takashi Ito (Yamagata University)
Masaharu Kato (Yamagata University)
Masaki Kohda (Yamagata University)
In this paper, we propose a new system combination approach for an LVCSR system using speaker-class (SC) models and a speaker adaptation technique based on these SC models. The basic concept of the SC-based system is to select speakers who are acoustically similar to a target speaker to train acoustic models. One of the major problems regarding the use of the SC model is determining the selection range of the speakers. In other words, it is difficult to determine the number of speakers that should be selected. In order to solve this problem, several SC models, which are trained by a variety of number of speakers are prepared in advance. In the recognition step, acoustically similar models are selected from the above SC models, and the scores obtained from these models are merged using a word graph combination technique. The proposed method was evaluated using the Corpus of Spontaneous Japanese (CSJ), and showed significant improvement in a lecture speech recognition task.

Feature versus Model Based Noise Robustness
Kris Demuynck (Katholieke Universiteit Leuven, dept. ESAT)
Xueru Zhang (Katholieke Universiteit Leuven, dept. ESAT)
Dirk Van Compernolle (Katholieke Universiteit Leuven, dept. ESAT)
Hugo Van hamme (Katholieke Universiteit Leuven, dept. ESAT)
Over the years, the focus in noise robust speech recognition has shifted from noise robust features to model based techniques such as parallel model combination and uncertainty decoding. In this paper, we contrast prime examples of both approaches in the context of large vocabulary recognition systems such as used for automatic audio indexing and transcription. We look at the approximations the techniques require to keep the computational load reasonable, the resulting computational cost, and the accuracy measured on the Aurora4 benchmark. The results show that a well designed feature based scheme is capable of providing recognition accuracies at least as good as the model based approaches at a substantially lower computational cost

The role of higher-level linguistic features in HMM-based speech synthesis
Oliver Watts (Centre for Speech Technology Research, University of Edinburgh, UK)
Junichi Yamagishi (Centre for Speech Technology Research, University of Edinburgh, UK)
Simon King (Centre for Speech Technology Research, University of Edinburgh, UK)
We analyse the contribution of higher-level elements of the linguistic specification of a data-driven speech synthesiser to the naturalness of the synthetic speech which it generates. The system is trained using various subsets of the full feature-set, in which features relating to syntactic category, intonational phrase boundary, pitch accent and boundary tones are selectively removed. Utterances synthesised by the different configurations of the system are then compared in a subjective evaluation of their naturalness. The work presented forms background analysis for an on-going set of experiments in performing text-to-speech (TTS) conversion based on shallow features: features that can be trivially extracted from text. By building a range of systems, each assuming the availability of a different level of linguistic annotation, we obtain benchmarks for our on-going work.

Latent Perceptual Mapping: A New Acoustic Modeling Framework for Speech Recognition
Shiva Sundaram (Deutsche Telekom Laboratories, Ernst-Reuter-Platz-7, Berlin 10587. Germany)
Jerome Bellegarda (Apple Inc., 3 Infinte Loop, Cupertino, 95014 California. USA.)
While hidden Markov modeling is still the dominant paradigm for speech recognition, in recent years there has been renewed interest in alternative, template-like approaches to acoustic modeling. Such methods sidestep usual HMM limitations as well as inherent issues with parametric statistical distributions, though typically at the expense of large amounts of memory and computing power. This paper introduces a new framework, dubbed latent perceptual mapping, which naturally leverages a reduced dimensionality description of the observations. This allows for a viable parsimonious template-like solution where models are closely aligned with perceived acoustic events. Context-independent phoneme classification experiments conducted on the TIMIT database suggest that latent perceptual mapping achieves results comparable to conventional acoustic modeling but at potentially significant savings in online costs.

State-based labelling for a sparse representation of speech and its application to robust speech recognition
Tuomas Virtanen (Department of Signal Processing, Tampere University of Technology, Finland)
Jort F. Gemmeke (Centre for Language and Speech Technology, Radboud University Nijmegen, The Netherlands)
Antti Hurmalainen (Department of Signal Processing, Tampere University of Technology, Finland)
This paper proposes a state-based labeling for acoustic patterns of speech and a method for using this labelling in noise-robust automatic speech recognition. Acoustic time-frequency segments of speech, exemplars, are obtained from a training database and associated with time-varying state labels using the transcriptions. In the recognition phase, noisy speech is modeled by a sparse linear combination of noise and speech exemplars. The likelihoods of states are obtained by linear combination of the exemplar weights, which can then be used to estimate the most likely state transition path. The proposed method was tested in the connected digit recognition task with noisy speech material from the Aurora-2 database where it is shown to produce better results than the existing histogram-based labeling method.

Single-channel speech enhancement using Kalman filtering in the modulation domain
Stephen So (Signal Processing Laboratory, Griffith University)
Kamil K. Wojcicki (Signal Processing Laboratory, Griffith University)
Kuldip K. Paliwal (Signal Processing Laboratory, Griffith University)
In this paper, we propose the modulation-domain Kalman filter (MDKF) for speech enhancement. In contrast to previous modulation domain-enhancement methods based on bandpass filtering, the MDKF is an adaptive and linear MMSE estimator that uses models of the temporal changes of the magnitude spectrum for both speech and noise. Also, because the Kalman filter is a joint magnitude and phase spectrum estimator, under non-stationarity assumptions, it is highly suited for modulation-domain processing, as modulation phase tends to contain more speech information than acoustic phase. Experimental results from the NOIZEUS corpus show the ideal MDKF (with clean speech parameters) to outperform all the acoustic and time-domain enhancement methods that were evaluated, including the conventional time-domain Kalman filter with clean speech parameters. A practical MDKF that uses the MMSE-STSA method to enhance noisy speech in the acoustic domain prior to LPC analysis was also evaluated and showed promising results.

Metric Subspace Indexing for Fast Spoken Term Detection
Taisuke Kaneko (Toyohashi University of Technology)
Tomoyosi Akiba (Toyohashi University of Technology)
In this paper, we propose a novel indexing method for Spoken Term Detection (STD). The proposed method can be considered as using metric space indexing for the approximate string-matching problem, where the distance between a phoneme and a position in the target spoken document is defined. The proposed method does not require the use of thresholds to limit the output, instead being able to output the results in increasing order of distance. It can also deal easily with the multiple candidates obtained via Automatic Speech Recognition (ASR). The results of preliminary experiments show promise for achieving fast STD.

Discriminative Language Modeling Using Simulated ASR Errors
Preethi Jyothi (Department of Computer Science and Engineering, The Ohio State University, USA)
Eric Fosler-Lussier (Department of Computer Science and Engineering, The Ohio State University, USA)
In this paper, we approach the problem of discriminatively training language models using a weighted finite state transducer (WFST) framework that does not require acoustic training data. The phonetic confusions prevalent in the recognizer are modeled using a confusion matrix that takes into account information from the pronunciation model (word-based phone confusion log likelihoods) and information from the acoustic model (distances between the phonetic acoustic models). This confusion matrix, within the WFST framework, is used to generate confusable word graphs that serve as inputs to the averaged perceptron algorithm to train the parameters of the discriminative language model. Experiments on a large vocabulary speech recognition task show significant word error rate reductions when compared to a baseline using a trigram model trained with the maximum likelihood criterion.

Learning a Language Model from Continuous Speech
Graham Neubig (Graduate School of Informatics, Kyoto University)
Masato Mimura (Graduate School of Informatics, Kyoto University)
Shinsuke Mori (Graduate School of Informatics, Kyoto University)
Tatsuya Kawahara (Graduate School of Informatics, Kyoto University)
This paper presents a new approach to language model construction, learning a language model not from text, but directly from continuous speech. A phoneme lattice is created using acoustic model scores, and Bayesian techniques are used to robustly learn a language model from this noisy input. A novel sampling technique is devised that allows for the integrated learning of word boundaries and an n-gram language model with no prior linguistic knowledge. The proposed techniques were used to learn a language model directly from continuous, potentially large-vocabulary speech. This language model was able to significantly reduce the ASR phoneme error rate over a separate set of test data, and the proposed lattice processing and lexical acquisition techniques were found to be important factors in this improvement.

New Insights into Subspace Noise Tracking
Mahdi Triki (Philips Research Laboratories)
Various speech enhancement techniques rely on the knowledge of the clean signal and noise statistics. In practice, however, these statistics are not explicitly available, and the overall enhancement accuracy critically depends on the estimation quality of the unknown statistics. The estimation of noise (and speech) statistics is particularly challenging under non-stationary noise conditions. In this respect, subspace-based approaches have been shown to provide a good tracking vs. final misadjustment tradeoff. Subspace-based techniques hinge critically on both rank-limited and spherical assumptions of the speech and the noise DFT matrices, respectively. The speech rank-limited assumption was previously experimentally tested and validated. In this paper, we will investigate the structure of nuisance sources. We will discuss the validity of the spherical assumption for a variety of nuisance sources (environmental noise, reverberation), and preprocessing (overlapping segmentation).

Acoustic Correlates of Meaning Structure in Conversational Speech
Alexei V. Ivanov (DISI, University of Trento, Italy)
Giuseppe Riccardi (DISI, University of Trento, Italy)
Sucheta Ghosh (DISI, University of Trento, Italy)
Sara Tonelli (FBK-IRST, Trento, Italy)
Evgeny Stepanov (DISI, University of Trento, Italy)
We are interested in the problem of extracting meaning structures from spoken utterances in human communication. In SLU systems, parsing of meaning structures is carried over the word hypotheses generated by the ASR. This approach suffers from high word error rates and ad-hoc conceptual representations. In contrast, in this paper we aim at discovering meaning components from direct measurements of acoustic and non-verbal linguistic features. The meaning structures are taken from the frame semantics model proposed in FrameNet. We give a quantitative analysis of meaning structures in terms of speech features across human--human dialogs from the manually annotated LUNA corpus. We show that the acoustic correlations between pitch, formant trajectories, intensity and harmonicity and meaning features are statistically significant over the whole corpus as well as relevant in classifying the target words evoked by a semantic frame.

Using Harmonic Phase Information to Improve ASR Rate
Ibon Saratxaga (Aholab Signal Processing Laboratory, University of the Basque Country)
Inma Hernáez (Aholab Signal Processing Laboratory, University of the Basque Country)
Igor Odriozola (Aholab Signal Processing Laboratory, University of the Basque Country)
Eva Navas (Aholab Signal Processing Laboratory, University of the Basque Country)
Iker Luengo (Aholab Signal Processing Laboratory, University of the Basque Country)
Daniel Erro (Aholab Signal Processing Laboratory, University of the Basque Country)
Spectral phase information is usually discarded in automatic speech recognition (ASR). The Relative Phase Shift (RPS), a novel representation of the phase information of the speech, has features which seem to be appropriate to improve the ASR recognition rate. In this paper we describe the RPS representation, discuss different ways to parameterize this information in a suitable way for the HMM modelling, and present the results of the evaluation experiments. WER improvements ranging from 12 to 22% open promising perspectives for the use of this information jointly with the classical MFCC parameterization. Index Terms: ASR, phase spectrum, harmonic analysis

Using Dependency Parsing and Machine Learning for Factoid Question Answering on Spoken Documents
Pere R. Comas (TALP Research Center, Technical University of Catalonia (UPC))
Lluís Màrquez (TALP Research Center, Technical University of Catalonia (UPC))
Jordi Turmo (TALP Research Center, Technical University of Catalonia (UPC))
This paper presents our experiments in question answering for speech corpora. These experiments focus on improving the answer extraction step of the QA process. We present two approaches to answer extraction in question answering for speech corpora that apply machine learning to improve the coverage and precision of the extraction. The first one is a reranker that uses only lexical information, the second one uses dependency parsing to score similarity between syntactic structures. Our experimental results show that the proposed learning models improve our previous results using only hand-made ranking rules with small syntactic information. We evaluate the system on manual transcripts of speech from EPPS English corpus and a set of questions transcribed from spontaneous oral questions. This data belongs to the CLEF 2009 evaluation track on QA on speech transcripts (QAst).

A Spoken Term Detection Framework for Recovering Out-of-Vocabulary Words Using the Web
Carolina Parada (Johns Hopkins University)
Abhinav Sethy (IBM TJ Watson Research Center)
Mark Dredze (Johns Hopkins University)
Frederick Jelinek (Johns Hopkins University)
Vocabulary restrictions in large vocabulary continuous speech recognition (LVCSR) systems mean that out-of-vocabulary (OOV) words are lost in the output. However, OOV words tend to be information rich terms (often named entities) and their omission from the transcript negatively affects both usability and downstream NLP technologies, such as machine translation or knowledge distillation. We propose a novel approach to OOV recovery that uses a spoken term detection (STD) framework. Given an identified OOV region in the LVCSR output, we recover the uttered OOVs by utilizing contextual information and the vast and constantly updated vocabulary on the Web. Discovered words are integrated into the system output, recovering up to 40% of the OOV terms and resulting in a reduction in system error.

Boosting Systems for LVCSR
George Saon (IBM T.J. Watson Research Center)
Hagen Soltau (IBM T.J. Watson Research Center)
We employ a variant of the popular Adaboost algorithm to train multiple acoustic models such that the aggregate system exhibits improved performance over the individual recognizers. Each model is trained sequentially on re-weighted versions of the training data. At each iteration, the weights are decreased for the frames that are correctly decoded by the current system. These weights are then multiplied with the frame-level statistics for the decision trees and Gaussian mixture components of the next iteration system. The composite system uses a log-linear combination of HMM state observation likelihoods. We report experimental results on several broadcast news transcription setups which differ in the language being spoken (English and Arabic) and amounts of training data. Our findings suggest that significant gains can be obtained for small amounts of training data even after feature and model-space discriminative training.

Incorporating Sparse Representation Phone Identification Features in Automatic Speech Recognition using Exponential Families
Vaibhava Goel (IBM T.J. Watson Research Center)
Tara Sainath (IBM T.J. Watson Research Center)
Bhuvana Ramabhadran (IBM T.J. Watson Research Center)
Peder Olsen (IBM T.J. Watson Research Center)
David Nahamoo (IBM T.J. Watson Research Center)
Dimitri Kanevsky (IBM T.J. Watson Research Center)
Sparse representation phone identification features (SPIF) is a recently developed technique to obtain an estimate of phone posterior probabilities conditioned on an acoustic feature vector. In this paper, we explore incorporating SPIF phone posterior probability estimates in large vocabulary continuous speech recognition (LVCSR) task by including them as additional features of exponential densities that model the HMM state emission likelihoods. We compare our proposed approach to a number of other well known methods of combining feature streams or multiple LVCSR systems. Our experiments show that using exponential models to combine features results in a word error rate reduction of 0.5% absolute (18.7% down to 18.2%); this is comparable to best error rate reduction obtained from system combination methods, but without having to build multiple systems or tune the system combination weights.

Improving ASR-based topic segmentation of TV programs with confidence measures and semantic relations
Camille Guinaudeau (INRIA/IRISA)
Guillaume Gravier (IRISA/CNRS)
Pascale Sébillot (IRISA/INSA)
The increasing quantity of video material requires methods to help users navigate such data, among which topic segmentation techniques. The goal of this article is to improve ASR-based topic segmentation methods to deal with peculiarities of professionnal-video transcripts (transcription errors and lack of repetitions) while remaining generic enough. To this end, we introduce confidence measures and semantic relations in a segmentation method based on lexical cohesion. We show significant improvements of the F1-measure, +1.7 and +1.9 when integrating confidence measures and semantic relations respectively. Such improvement demonstrates that simple clues can conteract errors in automatic transcripts and lack of repetitions.

A Novel text-independent phonetic segmentation algorithm based on the Microcanonical Multiscale Formalism
Vahid Khanagha (INRIA Bordeaux Sud-Ouest)
Khalid Daoudi (INRIA Bordeaux Sud-Ouest)
Oriol Pont (INRIA Bordeaux Sud-Ouest)
Hussein Yahia (INRIA Bordeaux Sud-Ouest)
We propose a radically novel approach to analyze speech signals from a statistical physics perspective. Our approach is based on a new framework, the Microcanonical Multiscale Formalism (MMF), which is based on the computation of singularity exponents, defined at each point in the signal domain. The latter allows nonlinear analysis of complex dynamics and, particularly, characterizes the intermittent signature. We study the validity of the MMF for the speech signal and show that singularity exponents convey indeed valuable information about its local dynamics. We define an accumulative measure on the exponents which reveals phoneme boundaries as the breaking points of a piecewise linear-like curve. We then develop a simple automatic phonetic segmentation algorithm usinhttp://www.interspeech2010.org/g piecewise linear curve fitting. We present experiments on the full TIMIT database. The results show that our algorithm yields considerably better accuracy than recently published ones.

Exploring Recognition Network Representations for Efficient Speech Inference on Highly Parallel Platforms
Jike Chong (University of California, Berkeley; Parasians, LLC)
Ekaterina Gonina (University of California, Berkeley)
Kisun You (Seoul National University)
Kurt Keutzer (Unversity of California, Berkeley)
The emergence of highly parallel computing platforms is enabling new trade-offs in algorithm design for automatic speech recognition. It naturally motivates the following investigation: Do the most computationally efficient sequential algorithms lead to the most computationally efficient parallel algorithms? In this paper we explore two contending recognition network representations for speech inference engines: the linear lexical model (LLM) and the weighted finite state transducer (WFST). We demonstrate that while an inference engine using the simpler LLM representation evaluates 22x more transitions per second than the advanced WFST representation, the simple structure of the LLM representation allows 4.7-6.4x faster evaluation and 53-65x faster operand-gathering for each state transition. We use the 5k Wall Street Journal Corpus to experiment on the NVIDIA GTX480 (Fermi) and the NVIDIA GTX285 Graphics Processing Units (GPUs), and illustrate that the performance of a speech inference engine based on the LLM representation is competitive with the WFST representation on highly parallel implementation platforms.

Speech Recognizer Optimization under Speed Constraints
Ivan Bulyko (Raytheon BBN Technologies)
We present an efficient algorithm for optimizing parameters of a speech recognizer aimed at obtaining maximum accuracy at a specified decoding speed. This algorithm is not tied to any particular decoding architecture or type of tunable parameter being used. It can also be applied to any performance metric (e.g. WER, keyword search or topic ID accuracy) and thus allows tuning to the target application. We demonstrate the effectiveness of this approach by tuning BBN’s Byblos recognizer to run at 15 times faster than real time while maximizing keyword search accuracy.

The 2010 CMU GALE Speech-to-Text System
Florian Metze (Carnegie Mellon University)
Roger Hsiao (Carnegie Mellon University)
Qin Jin (Carnegie Mellon University)
Udhyakumar Nallasamy (Carnegie Mellon University)
Tanja Schultz (Karlsruhe Institute of Technology)
This paper describes the latest Speech-to-Text system developed for the Global Autonomous Language Exploitation ("GALE") domain by Carnegie Mellon University (CMU). This systems uses discriminative training, bottle-neck features and other techniques that were not used in previous versions of our system, and is trained on 1150 hours of data from a variety of Arabic speech sources. In this paper, we show how different lexica, pre-processing, and system combination techniques can be used to improve the final output, and provide analysis of the improvements achieved by the individual techniques.

The RWTH 2009 Quaero ASR Evaluation System for English and German
Markus Nußbaum-Thom (RWTH Aachen University)
Simon Wiesler (RWTH Aachen University)
Martin Sundermeyer (RWTH Aachen University)
Christian Plahl (RWTH Aachen University)
Stefan Hahn (RWTH Aachen University)
Ralf Schlüter (RWTH Aachen University)
Hermann Ney (RWTH Aachen University)
In this work, the RWTH automatic speech recognition systems for English and German for the second Quaero evaluation campaign 2009 are presented. The systems are designed to transcribe web data, European parliament plenary sessions and broadcast news data. Another challenge in the 2009 evaluation is that almost no in-domain training data is provided and the test data contains a large variety of speech types. The RWTH participates for the English and German languages with the best results for German and competitive results for the English. Contributing to the enhancements are the systematic use of hierarchical neural network based posterior features, system combination, speaker adaptation, cross speaker adaptation, domain dependent modeling and the usage of additional training data.

The Impact of ASR on Abstractive vs. Extractive Meeting Summaries
Gabriel Murray (University of British Columbia)
Giuseppe Carenini (University of British Columbia)
Raymond Ng (University of British Columbia)
In this paper we describe a complete abstractive summarizer for meeting conversations, and evaluate the usefulness of the automatically generated abstracts in a browsing task. We contrast these abstracts with extracts for use in a meeting browser and investigate the effects of manual versus ASR transcripts on both summary types.

This is a comparision that I feel is wrong
An Empirical Comparison of the T3, Juicer, HDecode and Sphinx3 Decoders
Josef R. Novak (Tokyo Institute of Technology)
Paul R. Dixon (National Institute of Information and Communications Technology)
Sadaoki Furui (Tokyo Institute of Technology)
In this paper we perform a cross-comparison of the T3 WFST decoder against three different speech recognition decoders on three separate tasks of variable difficulty. We show that the T3 decoder performs favorably against several established veterans in the field, including the Juicer WFST decoder, Sphinx3, and HDecode in terms of RTF versus Word Accuracy. In addition to comparing decoder performance, we evaluate both Sphinx and HTK acoustic models on a common footing inside T3, and show that the speed benefits that typically accompany the WFST approach increase with the size of the vocabulary and other input knowledge sources. In the case of T3, we also show that GPU acceleration can significantly extend these gains.

CRF-based Combination of Contextual Features to Improve A Posteriori Word-level Confidence Measures
Julien Fayolle (IRISA/INRIA Rennes, France)
Fabienne Moreau (University of Rennes 2/IRISA Rennes, France)
Christian Raymond (IRISA/INSA Rennes, France)
Guillaume Gravier (IRISA/CNRS Rennes, France)
Patrick Gros (IRISA/INRIA Rennes, France)
This paper addresses the issue of confidence measure reliability provided by automatic speech recognition systems for use in various spoken language processing applications. We propose a method based on conditional random field to combine contextual features to improve word-level confidence measures. The method consists in combining various knowledge sources (acoustic, lexical, linguistic, phonetic and morphosyntactic) to enhance confidence measures, explicitly exploiting context information. Experiments were conducted on a large French broadcast news corpus from the ESTER benchmark. Results demonstrate the added-value of our method with a significant improvement of the normalized cross entropy and of the equal error rate.

Interesting, do they mention Voxforge in this paper
Building transcribed speech corpora quickly and cheaply for many languages
Thad Hughes (Google)
Kaisuke Nakajima (Google)
Linne Ha (Google)
Atul Vasu (Google)
Pedro Moreno (Google)
Mike LeBeau (Google)
We present a system for quickly and cheaply building transcribed speech corpora containing utterances from many speakers in a variety of acoustic conditions. The system consists of a client application running on an Android mobile device with an intermittent Internet connection to a server. The client application collects demographic information about the speaker, fetches textual prompts from the server for the speaker to read, records the speaker’s voice, and uploads the audio and associated metadata to the server. The system has so far been used to collect over 3000 hours of transcribed audio in 17 languages around the world

On Generating Combilex Pronunciations via Morphological Analysis
Korin Korin Richmond (Centre for Speech Technology Research, Edinburgh University)
Robert Robert Clark (Centre for Speech Technology Research)
Sue Sue Fitt (Centre for Speech Technology Research)
Combilex is a high-quality lexicon that has been developed specifically for speech technology purposes and recently released by CSTR. Combilex benefits from many advanced features. This paper explores one of these: the ability to generate fully-specified transcriptions for morphologically derived words automatically. This functionality was originally implemented to encode the pronunciations of derived words in terms of their constituent morphemes, thus accelerating lexicon development and ensuring a high level of consistency. In this paper, we propose this method of modelling pronunciations can be exploited further by combining it with a morphological parser, thus yielding a method to generate full transcriptions for unknown derived words. Not only could this accelerate adding new derived words to Combilex, but it could also serve as an alternative to conventional letter-to-sound rules. This paper presents preliminary work indicating this is a promising direction.

Content-Based Advertisement Detection
Patrick Cardinal (CRIM)
Vishwa Gupta (CRIM)
Gilles Boulianne (CRIM)
Television advertising is widely used by companies to promote their products among the public but it is hard for an advertiser to know if its advertisements are broadcast as they should. For this reason, some companies are specialized in the monitoring of audio/video streams for validating that ads are broadcast according to what was requested and paid for by the advertiser. The procedure for searching specific ads in an audio stream is very similar to the copy detection task for which we have developed very efficient algorithms. This work reports results of applying our copy detection algorithms to the advertisement detection task. Compared to a commercial software, we detected 18% more advertisements and the system runs at 0.003x of real-time.

Continuous Speech Recognition with a TF-IDF Acoustic Model
Geoffrey Zweig (Microsoft)
Patrick Nguyen (Microsoft)
Jasha Droppo (Microsoft)
Alex Acero (Microsoft)
Information retrieval methods are frequently used for indexing and retrieving spoken documents, and more recently have been proposed for voice-search amongst a pre-defined set of business entries. In this paper, we show that these methods can be used in an even more fundamental way, as the core component in a continuous speech recognizer. Speech is initially processed and represented as a sequence of discrete symbols, specifically phoneme or multi-phone units. Recognition then operates on this sequence. The recognizer is segment-based, and the acoustic score for labeling a segment with a word is based on the TF-IDF similarity between the subword units detected in the segment, and those typically seen in association with the word. We present promising results on both a voice search task and the Wall Street Journal task. The development of this method brings us one step closer to being able to do speech recognition based on the detection of sub-word audio attributes.

Improved topic classification and keyword discovery using an HMM-based speech recognizer trained without supervision
Man-Hung Siu (Raytheon BBN Technologies)
Herbert Gish (Raytheon BBN Technologies)
Arthur Chan (Raytheon BBN Technologies)
William Belfield (Raytheon BBN Technologies)
In our previous publication, we presented a new approach to HMM training, viz., training without supervision. We used an HMM trained without supervision for transcribing audio into self-organized units (SOUs) for the purpose of topic classification. In this paper we report improvements made to the system, including the use of context dependent acoustic models and lattice based features that together reduce the topic verification equal error rate from 12% to 7%. In addition to discussing the effectiveness of the SOU approach we describe how we analyzed some selected SOU n-grams and found that they were highly correlated with keywords, demonstrating the ability of the SOU technology to discover topic relevant keywords.

Voicemail transcription with Pocketsphinx and Asterisk

2010-09-25T06:32:00.005+04:00

This is for admins who are aware that pocketsphinx exists and want to try it. It will describe how to quickly setup voicemail transcription using pocketsphinx and Asterisk. The process is extremely simple, I promise it will not take more than 5 minutes.

We'll use external shell command invoked when voicemail arrives and this command will transcribe voicemails which aren't transcribed yet. We won't postprocess the result as well as will not clean it up. The goal is just to show how to do it quickly and show how asterisk interface can be built.

So, let's start

1. Setup asterisk. I hope it will run smoothly, it's really easy. Setup samples with make samples. Our demo will be based on them.

2. Setup pocketsphinx. You need to download pocketsphinx and sphinxbase from the download page. You need at least pocketsphinx version 0.7, previous versions will not work. Some of the required features are only avaiable in this release or later releases.

3. Check that pocketsphinx works. Just run pocketsphinx_continuous and try to say something when READY will appear. The decoding result will appear before next READY.

000000000: hello world

4. Download Communicator acoustic model for telephone speech and unpack it into some location, for example in $prefix/var/lib/asterisk/communicator. There must be files like mdef, variances, etc.

5. Edit voicemail.conf in $prefix/etc/asterisk/voicemail.conf. Configure external callback script:

externnotify=$prefix/sbin/voicemail-notify.sh

6. Now let's create the script voicemail-notify.sh in the folder $prefix/sbin where all other asterisk binaries reside. Copy-paste it from below, change permission to 755, don't forget to update the prefix to point to the asterisk installation folder

#!/bin/bash

prefix=<PUT YOUR ASTERISK PREFIX HERE>
voicemaildir=$prefix/var/spool/asterisk/voicemail/$1/$2/INBOX/

for audiofile in `ls $voicemaildir/*.wav`; do
   transcriptfile=${audiofile/wav/transcript}
   # For each message.wav we check if message.transcript exists
   if [ ! -f $transcriptfile ]; then
   # If not, we create it
   pocketsphinx_continuous -infile $audiofile \
   -hmm $prefix/var/lib/asterisk/communicator \
   -samprate 8000 2> /dev/null > $transcriptfile
   # Now we can do whatever we want with the new transcription
   # Send it by mail for example
   # mail $user < $transcriptfile
   fi
done

7. Start asterisk or reload configuration with "voicemail reload"

8. Dial extension 1234

*CLI> console dial 1234

and leave voicemail.

9. Check that your voicemail is transcribed automatically and the transcription is put together with wav file into voicemail folder

ls $prefix/var/spool/asterisk/voicemail/default/1234/INBOX/*.transcript

You can also send the transcript by mail or do with it whatever you want. Easy, isn't it? Well, I didn't mention you need better language model and all the tricks to improve the transcriptoin accuracy for your voicemails, thats a separate story.

Update
The second part is here: http://nsh.nexiwave.com/2011/04/voicemail-transcription-with.html

Next Routine Release

2010-09-01T02:10:00.000+04:00

So today we released Sphinx4-Beta5. I'm not very pleased by the number of features that went in, I wanted to do more, but I'm very pleased by the increasing number of people contributed to this release. From release to release thanks list is definitely getting longer.

This six month I finally went in the deep blue area of the LexTree linguist. So the major feature of this release is significantly reworked LVCSR search which is supposed to get faster at least sometimes. Careful testing would be welcome here.

Another important but not so visible thing is the first bit of the applicaiton-oriented public API. There will be no public XML configs anymore! The Aligner demo that comes with sphinx4 is the first step towards that. The configs for the typical applications will be stored inside jar and supposed to be used by the developer who want to tweak engine. No demo will use XML anymore. This rework of the public API together with opimization of the data flow inside the engine is the top priority for us for the next six month. We shall discuss in Feburary how it goes.

Patents in ASR

2010-08-23T02:52:00.000+04:00

I was pleased to read this review on Nuance vs Vlingo litigation.

Based on this analysis, we can find no evidence supporting the “seven figure” price which Vlingo paid to acquire the ‘354 patent from Intellectual Ventures. It is unclear how, armed with a broad spectrum of information, Vlingo would have justified their purchase. The implications of the Vlingo v. Nuance litigation hold tremendous implications for corporate IP strategies.

Senone Tree Implementation For Sphinx4

2010-07-30T16:17:00.003+04:00

I spent last month working on senone tree linguist for sphinx4 as a part of Nexiwave's sphinx4 performance project. Well, mostly I was fixing bugs in my initial implementation. The core idea of senone tree which was suggested to me by Bhiksha is the following. Lextree is a representation of all possible words in a dictionary which is built with triphones. Lextree is used to explore search space during decoding. There very good thing is that since number of HMMs is rather small comparing to the number of triphones (40000 vs 100000) the lextree is rather compact representation of the search space.

Senone tree takes advantage of the internal representation of each triphone. Indeed the triphone is built from 3 senones and number of senones is even smaller. It's just 10000 for a big model. And respectively we can transform our graph to way more compact structure. There are also some techical advantages of our senone tree implementation - efficient hash map that doesn't waste memory, simple search space structure since we don't use helper states like LexTreeEndUnitState for word end nodes and LexTreeNonEmittingHMMState for non-emitting states. Simple code is also an advantage.

Expermients also show that senone tree provides nice results. It's a little bit more accurate (few insignificant percents) and it is significantly faster (2x times faster in growth, 20% faster overall, 20k active tokens vs 40k before). That's a nice improvement for Nexiwave engine.

This work makes me think that we need to provide different answer on our core question: "How to improve the accuracy". Of course traditional ones like: optimize parameters and train better model, are still valid but they are secondary. Primary answer should be: implement proper features in engine and you'll get the accuracy improvement you need.

Speech Decoding Engines, Part 2. SCARF, The Next Big Thing In Machine Learning

2010-07-17T19:00:00.002+04:00

It seems that HMM will not stay forever. If you aren't tied to speech and track big things in machine learning, you should hear about that new thing - Conditional Random Fields. According to recently started but very promising Metaoptimize, it's one of the most influental ideas in machine learning.

And, suprisingly, you can already apply this thing to speech recognition, thanks to Microsoft Research including Geoffrey Zweig, Patrick Nguyen. It's SCARF, a Segmental Conditional Random Field Speech Recognition Toolkit which is version 0.5 now. You can download it's sources from Microsoft Research Website.

The idea behind SCARF is very elegant I would say. In HMM we use join probability distribution between observation features and state label to estimate the probability of label sequence. The showstopper here is the assumed independence of state distributions.

In CRF we consider different thing - the conditional probability of label sequence given the observation sequence. Conditional models are used to label a novel observation sequence x by selecting the label sequence that maximizes the conditional probability p(y|x). The conditional nature of such models means that no effort is wasted on modeling the observations, and one is free from having to make unwarranted independence assumptions about these sequences; arbitrary attributes of the observation data may be captured by the model, without the modeler having to worry about how these attributes are related.

In application to speech models, labels used are states in a language model FST or just words and features could be arbitrary set, including pitch, spectral features or phonetic recognizer posteriors. But in practice SCARF doesn't operate on acoustic features, instead it's used as a postprocessing step over posteriours predicted with conventional recognizer or some other detector. This presentation has more information on that. Usage of high-level events makes it similar to other postprocessing decoders like consensus decoding of lattices recently landed in CMUSphinx SVN.

The whole point that it's possible to provide efficient training and decoding even for such a complex model. Taking into account all nice properties of CRF's it all sounds very promising.

The whole SCARF code is very small, the codebase for trainer and decoder is just 6KLOC. The included manual is very good, pretty simple and describes everything needed in details. The little issue is to obtain the data to train and test the model. Data formats are rather clear but anyway require some effort to produce them. At least I didn't manage to get the input so had no luck to test it in action.

So if you are interested, you definitely need to try to create a dataset for SCARF and train something.

Related posts:

Speech Decoding Engines Part 1. Juicer, the WFST recognizer

Testing CMUSphinx with Hudson

2010-07-03T01:12:00.005+04:00

As every high-quality product CMUSphinx spend a lot on testing. That isn't really trivial task because you need to make sure that all parameters that are important are improved or at least not regressed. That includes decoding accuracy, speed and API specs. Sometimes changes improve one thing and make other worse. Things are going to change with the deployment of continuous integration system Hudson.

Quite sophisticated system of tests was created to track changes. That included perl scripts, various shell bits, mysql database and even commits to CVS repository. It was also spamming mailing list all the day with long and unreadable emails. Another bad thing was that it's based on private commercial data like WSJ or TIDIGITS database but now everything is changing with Voxforge test set. Our goal is to let you test and optimize system yourself

.

Luckily, it went down recently. I was wondering about why I don't get any mails for almost a month but I'm pretty sure whole thing just not working anymore. There is a good side then since we have more reasons to move to something that specifically developed to support testing workflow. With Peter's hint I downloaded and instaled Hudson.

It's a nice thing in open source that you can get really professional product designed specifically to solve your problems by experts who understand what your problems are. I very much disagree with the people when they think they can write something themselves from scratch. I never understood this point of view.

Anyway, Hudson is our testing guard right now, you shouldn't worry about regressions anymore. Its running all CMUSphinx tests and some accuracy tests as well. It also builds nice graphs (see on the right) and can present you statistics about last builds.

This piece of software is very good. What do I like about it:

Slick interface
Many plugins (I hate plugins to be honest, but hudson's plugins are great)
Customizable build process without any restrictions - you can run shell scripts, you can run your own apps, basically you can do everything
Intelligent notification. Thanks Hudson, you only send me email when something is broken, no need to read mails every day!

It's defiitely a step forward in testing.
This server is private right now but we hope to make it public soon. Oh, and you can probably see there was a regression on July 1st that I committed for sphinx4. Accuracy dropped significantly and speed become a little bit faster. If you are tracking cmusphinx-devel, you probably know what is it about. Let's hope we'll fix it soon and than it will be really great thing to have.

HTK Competition Voting

2010-07-01T02:48:00.000+04:00

Thanks everyone for your feedback, results are really interesting to see.

HTK competition is something that I was worrying about for a long time. One key issue that I see is that htk-users mailing list definitely has way more deep discussions about ASR than we have on our forum. Hopefully, situatuation will change.

Anyway, our goal is still to provide very accurate speech recognition and this is not yet solved task with many issues both in usability and accuracy. So we can definitely learn from each other and improve our projects.

Sphinx4 Powers Contemporary Art

2010-06-13T16:34:00.000+04:00

Did you think that sphinx4 could be only used to build another keyboard, help you to track sales manager blaming the product or transcribe medical dictation? Working with computers on daily basis one starts to consider them as a tool. I was thinking this way not taking into account the fact that speech act itself powered by computers has probably sacral meaning. Communication was the thing that created our mind, and keyboards aren't important when we create communication systems.

The thing that pushed me to this is Heather Dewey-Hagborg's blog. In particular it was the Listening Post, an artistic thing from the CEPA gallery. If you are interested, please also check Heather's interview on BTR Radio. Check also the gallery's site.

And important point here is that I think we should not consider this as some kind of futurizm - talking computers, HAL and all that stuff. Instead, such things help us to change ourselves, change our vision of the world around. Probably next time you'll look on sphinx4 sources from a different point of view.

Great Overview Article

2010-06-08T22:09:00.002+04:00

Today Dr. Tony Robinson gave me a present by mentioning this great article on comp.speech.research

Janet M. Baker, Li Deng,
James Glass, Sanjeev Khudanpur,
Chin-Hui Lee, Nelson Morgan, and
Douglas O’Shaughnessy

Research Developments and Directions in Speech Recognition and Understanding, Part 1
Research Developments and Directions in Speech Recognition and Understanding, Part 2

This article was MINDS 2006–2007 Report of the Speech Understanding Working Group,” one of five reports emanating from two workshops titled “Meeting of the MINDS: Future Directions for Human Language Technology,” sponsored by the U.S. Disruptive Technology Office (DTO). For me it was striking that spontaneous events are so important, I never thought about them from this point of view.

The whole state of things is also nicely described in Mark Gales talk Acoustic Modelling for Speech Recognition: Hidden Markov from Models and Beyond? The picture on the left is taken from it.

Blizzard Challenge 2010

2010-06-04T22:58:00.000+04:00

Since I was in TTS for a long time and still interested in in, I've been waiting a long for this - Blizzard Challenge team is ready to accept speech expert and volunteer listeners for the Blizzard Challenge 2010.

The challenge was devised in order to better understand and compare research techniques in building corpus-based speech synthesizers on the same data. The basic challenge is to take the released speech database, build a synthetic voice from the data and synthesize a prescribed set of test sentences. The sentences from each synthesizer will then be evaluated through listening tests.

After evaluation participants submit papers where they describe the methods used and problems solved. You could find more information on the webpage http://festvox.org/blizzard/

They want as many listeners as possible over the next few weeks and we can help! So, please distribute the "Speech Experts" link within your group, and distribute the "Volunteers" link as widely as possible - mailing lists, blogs, noticeboards, your students, friends, family... anywhere you can think of. Remember - you don't have to be a native speaker to take part.

For English:
Speech Experts: http://groups.inf.ed.ac.uk/blizzard/blizzard2010/english/register-ES.html
Volunteers: http://groups.inf.ed.ac.uk/blizzard/blizzard2010/english/register-ER.html

For Chinese:
Speech Experts: http://groups.inf.ed.ac.uk/blizzard/blizzard2010/mandarin/register-MS.html
Volunteers: http://groups.inf.ed.ac.uk/blizzard/blizzard2010/mandarin/register-MR.html

Despite it's my fourth challenge where I participated as a listener I still find it's interesting to pass through. The reason for that is probably that instead of synthesized sound utterances I see people and teams behind them. Though organizers stress that the challenge is not a competition, I think it's the most interesting thing there. I even think that tradition to encrypt the results with the letter which require you to read all papers and decipher who got which result makes the challenge worth to participate. That is a kind of observance for me. I'm looking forward to read about the results.

As for non-experts, the challenge could be interesting just because it's a unique way to compare different technologies and get idea on what is possible. TTS is often utilized in different ways, but as far as you understand the speech it's ok. That should be changed. Demand for high-quality text-to-speech conversion is huge and will grow as soon as intelligent human-computer interaction will be more common. It's important to get the landmarks of the quality possible. On that background minor things like enormous noisy section that is very hard to go through or tedious QuickTime installation process looks like a minor thing.

And all this reminded me again that I need to push Ben to deploy better TTS voice on searchmymeetings. The current one is not acceptable in my opinion.

KISS Principle

2010-06-01T00:16:00.001+04:00

Still think that you can take sphinx4 engine and make a state-of-art recognizer? Check what AMI RT-09 entry is doing for meeting transcription in presentation on RT'09 workshop "The AMI RT’09 STT and SASTT Systems":

Segmentation
Initial decoding of full meeting with
- 4g LM based on 50K vocabulary and weak acoustic model (ML) M1
- 7g LM based on 6K vocabulary and strong acoustic model (MPE) M2
Intersect output and adapt (CMLLR)
Decode using M2 models and 4gLM on 50k vocabulary
Compute VTLN/SBN/fMPE
Adapt SBN/fMPE/MPE models M3 using CMLLR
Adapt LCRCBN/fMPE/MPE models M4 using CMLLR and output of previous stage
Generate 4g lattices with adapted M4 models
Rescore using M1 models and CMLLR + MLLR adaptation
Compute Confusion networks

Click on image to check the details of the process.

Campaign For Decoding Performance

2010-05-26T00:10:00.002+04:00

We spent some time to make speech recognition backend faster. Ben reports in his blog the results on moving scoring to GPU with CUDA/jCUDA, which reduced scoring time dramatically. That's an improvement we are happy to apply in our production environment.

We consider that GPU is not just a speedup of computation, it's a paradigm shift. Historically search is optimized to make the number of scored tokens smaller since it affected accuracy. Now scoring is immediate, but that means that other parts should be changed. There are few issues to smash on the way:

We really target to make it even more faster, in particular we would really like to solve grow part problem.

In classical score/prune/grow scheme unfortunately not only scoring takes significant time. In particular in sphinx4 growing branches is also a bottleneck. When sphinx4 was optimized for LVCSR in the beginning, grow time was also a problem. That's why whole number of workarounds where developed: grow skipping, skew pruning, arc caching and acoustic lookahead. They were successful at that time but not as successful as they could be. At least they don't scale for GPU.

Among papers that I've found there are several publications about GPU-based speech recognition, in particular I would like to note interesting research by Jike Chong and his colleagues. Thanks to Tao for the link! But the issue is that complex grow algorithms is also not considered there. They write about bigram search which basically means they explore very simple state space. In results they compare themselves with HVite. That's the same situation as with WFST when the attempt to bring complexity of large vocabulary into first pass fails and one should stick with bigrams and hope that all important things will be done later on subsequent passes. I'm kind of think that it's a waste of processor resources.

Next issue is more technical. We haven't found good gpu cloud service yet. Though such services are certainly very promising because GPU is more energy-efficient they aren't common yet. Lets hope this situation will improve soon.

Want To Learn How Sphinx4 Works? Help With Wiki!

2010-05-18T01:00:00.000+04:00

Long time ago when sphinx4 development was active, the team used twiki hosted at CMU. Unlike many open source projects, this wiki was actually not just a collection of random stuff, but a complete project support system. It contained meetings notes, design decisions and prototyping results. You could find there diagrams and explanations on what is grow skipping or what is skew pruning.

This wiki died some time ago due to administrative issues, but that was even better since the content from it was merged into our current main wiki at sourceforge at sphinx4 namespace. Unfortunately during transition the formatting was lost since dokuwiki formats aren't always the same as twiki ones. That's actually not so bad as well because the content needs to be renewed in order to fit into current state of sphinx4.

So right now there are like 170 pages total about sphinx4. Some of them are useful, some aren't. They definitely contain deep knowledge of sphinx4 internals, something that will probably help you next time you will optimize the performance of large vocabulary recognition with sphinx4. I'm in process of slowly sorting them out but that will take a lot of time. It's your chance to join, help and learn!

Not All Speaker Adaptations Are Equally Useful

2010-05-13T01:32:00.001+04:00

Some time ago I was rather encouraged by VTLN which is vocal tract normalization. By so-called frequence warping it tries to unify vocal tract lenght of all speakers and thus make better model. It's done by shifting and adjusting mel filter frequencies. This thing is implemented in Sphinxtrain/Pocketsphinx/Sphinx4. Basically all you need is to enable it in sphinx_train.cfg


$CFG_VTLN = 'yes';

And run the training. It will extract features with all frequency warp parameters with some step (take care about space on disk) and will find out best one with forced alignment of each utterance. Then it will create new fileids and transcription files with reference to the file with proper warp parameter.

To decode with VTLN model you need to guess warp parameter. There are several algorithms suggested to do that. One analyses pitch, others employ GMM for classification. Then you need to reextract features with predicted warp parameter. It gives some visible improvement in performance.

But recently a set of articles like

MLLR-LIKE SPEAKER ADAPTATION BASED ON LINEARIZATION OF VTLN WITH MFCC FEATURES by Xiaodong Cui and Abeer Alwan

came into my sight thanks to antonsrv8. The simple idea is that any transform which we do on features, especially smooth transform could be mostly replaced just by linear transform of MFCC coefficients, basically by MLLR transformation. This kind of obvious fact makes me think if we really need other transformations if MLLR is generic enough. It's not harder to estimate MLLR than to estimate warp factor, especially if data is large enough which is usually the case. Another transformation applied will just conflict with MLLR. On large data sets this is confirmed by experimental results in article above.

Of course non-linear transform like VTLN could be better than linear one, but it's certainly not VTLN it seems. I hope latest state of art in voice conversion could suggest something better.

Update: this point was of course largely covered in research papers. Good coverage with math and results is provided in Luis thesis:

Speaker Normalisation and Adaptation in Large Vocabulary Speech Recognition by Lu ́s Felipe Uebel

Recognizer Language Voting Over

2010-05-12T02:33:00.000+04:00

So, language voting is over. It seems that despite performance issues we currently face Java gets enough attention. Thanks for sharing your opinion, it's very important for us.

Speech Decoding Engines Part 1. Juicer, the WFST recognizer

2010-05-07T00:27:00.004+04:00

ASR today is quite diverse. While in 1998 there was only a HTK package and some inhouse toolkits like CMUSphinx released in 2000, now there are dozen very interesting recognizers released to the public and available under open source licenses. We are starting today the review series about them.

So, the first one is Juicer, the WSFT recognizer from IDIAP, University of Edinburgh and University of Sheffield.

Weighted finite state transducers (WFST) are very popular trend in modern ASR, with very famous addicts like Google, IBM Watson center and so many others. Basically the idea is that you convert everything into same format that allows not just unified representation, but more advanced operations like merging to build a shared search space or reducing to make that search space smaller (with operations like determinization and minimization). Format also provides you interpolation properties, for example you don't need to care about g2p anymore, it's automatically done by transducer. For WFST itself, I found a good tutorial by Mohril, Pereira and Riley "Speech Recognition with Weighted State Transducers".

Juicer can do very efficient decoding with stanard set of ASR tools - ARPA language model (bigram due to memory requirements), dictionary and cross-word triphone models could be trained by HTK. BSD license makes Juicer very attractive. Juicer is part of AMI project that targets meeting transcription, other AMI deliverables are subject for separate posts though.

So here is the description how to try it. Don't expect it to be straightforward though, it's not a trivial process. Well, one day we'll put everything on a live CD to make ASR development environment easier. Right now you can follow this step-by-step howto as many our young friends call such thing. I wonder where do people get the idea that for everything there is detailed step-by-step howto.

So, let's start Download Juicer and dependencies:


Torch3src.tgz
tracter-0.6.0.tar.bz2
juicer-0.12.0.tar.bz2
kiss_fft-v1.2.8.tar.gz
openfst-1.1.tar.gz

Unpack and build torch


tar xf Torch3src.tgz
cd Torch3
cp config/Linux.cfg .

Edit Linux.cfg to include packages: distributions gradients kernels
speech datasets decoder


# Packages you want to use
packages = distributions gradients kernels speech datasets decoder

Continue with the build


./xmake all
cd ..

Unpack kiss_fft:


tar xf kiss_fft-v1.2.8.tar.gz

There is no need to build kiss, it's build is included in the next step.

Unpack and build tracter


tar xf tracter-0.6.0.tar.bz2
cd tracter-0.6.0
aclocal && libtoolize && automake -a && autoconf
mkdir m4
./configure \
  --with-kiss-fft=/current_folder/kiss_fft_v1_2_8 \
  --with-htk-includes="-I/htk_folder/HTKLib" \
  --with-htk-libs="/htk_folder/HTKLib/HTKLib.a"
  --with-torch3=/current_folder/Torch3
make && make install
cd ..

Make sure you point full path to the dependencies, since relative path
will not work. Also note that for htk you need to provide compiler
options, not folders. Alternatively you can increase your pain trying
to build tracter with cmake as readme describes.

Unpack and build juicer

Make sure PKG_CONFIG_PATH makes tracter.pc reachable.


export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib/pkgconfig
tar xf juicer-0.12.0.tar.bz2
cd juicer-0.12.0
aclocal && libtoolize && automake -a && autoconf
mkdir m4
./configure \
  --with-kiss-fft=/current_folder/kiss_fft_v1_2_8 \
  --with-htk-includes="-I/htk_folder/HTKLib" \
  --with-htk-libs="/htk_folder/HTKLib/HTKLib.a"
  --with-torch3=/current_folder/Torch3
make && make install
cd ..

Build openfst:


tar xf openfst-1.1.tgz
cd openfst
./configure && make && make install
make
cd ../..

Setup environment variables:


export JUTOOLS=/current_folder/juicer-0.12.0/bin

At this point juicer and required tools are built, let's try it with HTK
wsj model from Keith Vertanen. Download the model htk_wsj_si84_2750_8.zip
and unpack it


unzip htk_wsj_si84_2750_8.zip

Convert model to ascii


mkdir ascii 
touch empty
HHEd -D -T 1 -H hmmdefs -H macros -M ascii empty tiedlist

Convert dmp turtle model from pocketsphinx to ARPA model turtle.lm


sphinx_lm_convert -i turtle.DMP -o turtle.lm -ifmt dmp -ofmt arpa

Remove alternative pronunciation numbers from turtle.dic and build phoneset


sed 's/([0-9])//g' turtle.dic | tr [:upper:] [:lower:] > turtle.dic.lower
mv turtle.dic.lower turtle.dic
echo "<s>   sil" >> turtle.dic
echo "</s>   sil" >> turtle.dic
for w in `cat turtle.dic | cut -d" " -f 2-`; do echo $w; done | sort | uniq > turtle.phone
echo sp >> turtle.phone

Due to some script limitations, not all different words couldn't have same pronunciations. So open turtle.dic and remove line with entry "two t uw" because it conflicts with "to t uw"

Now let's convert everything into WFST


gramgen -gramType ngram -lmFName turtle.lm -lexFName turtle.dic \
-fsmFName gram.fsm -inSymsFName gram.insyms -outSymsFName gram.outsyms \
-sentStartWord "<s>" -sentEndWord "</s>"

lexgen -lexFName turtle.dic  -monoListFName turtle.phone \
-fsmFName dic.fsm -inSymsFName dic.insyms -outSymsFName dic.outsyms \
-sentStartWord "<s>" -sentEndWord "</s>" -pauseMonphone sp -addPronunsWithEndPause

cdgen -htkModelsFName wsj_si84_2750_8/ascii/hmmdefs -tiedListFName \
wsj_si84_2750_8/tiedlist  -monoListFName turtle.phone -fsmFName wsj.fsm \
-inSymsFName wsj.insyms -outSymsFName wsj.outsyms

To deal with juicer bug comment the following lines in juicer-0.12.0/bin/aux2eps.pl:


#if ( ! %AUXSYMS )
#{
#   print "no aux syms in symbol file - nothing to do\n" ;
#   exit 0 ;
#}

Now let's compose it into single WFST


build-wfst-openfst gram.fsm dic.fsm wsj.fsm

Everything is ready for decoding. Let's try with goforward.raw from pocketsphinx


sox -r 16000 -2 -s goforward.raw goforward.wav

create HTK config


SOURCEFORMAT = WAV
TARGETKIND = MFCC_0_D_A_Z
TARGETRATE = 100000.0
WINDOWSIZE = 250000.0
SAVECOMPRESSED = F
SAVEWITHCRC = F
USEHAMMING = T
PREEMCOEF = 0.97
NUMCHANS = 26
CEPLIFTER = 22
NUMCEPS = 12
ENORMALISE = T
ZMEANSOURCE = T
USEPOWER = T

Convert to mfcc


HCopy -C config goforward.wav goforward.mfc

Create control file:


echo goforward.mfc > train.scp

Decode


juicer -inputFormat htk -lexFName turtle.dic -inputFName train.scp -fsmFName final.fsm -inSymsFName final.insyms -outSymsFName final.outsyms  -htkModelsFName wsj_si84_2750_8/ascii/hmmdefs

Get the result:


<s> go four are <s> ten meters </s>

It's not accurate for some reason. Probably feature extraction is not the same as were used for acoustic model. Probably I should use word insertion penalty.

Of course not everything is so perfect. The main issues with WFST decoder are very well described in documentaion. Basically they are memory requirements for the first pass decoding (that's why Juicer can't run trigram models on commodity hardware) and lack of dynamic search optimization that's more straightforward. Anyway, WFST framework has a lot of applications going beyond just recognition. It's applied for speech indexing, open vocabulary decoding, simplifies confidence scoring.

That's it, you can count it works and embed it into your software. Overall, it's an interesting package demonstrating how simple things could be when you put everything into flexible format. I'm sure CMUSphinx will follow this direction and will implement WFST decoding soon. At least we ultimately need to introduce FST tools in our framework.

Face Recognizers, Bloom filters and Application to Speech Recognition

2010-05-01T04:20:00.001+04:00

In scientific paper waterfall we have today I continuously face the issue of selection of high-level important approaches to the problem. Many ideas are definitely important and lead to accuracy improvement but they are certainly not counted as core ones. Like another feature extraction algorithm that could bring you 2% of performance improvement. I definitely miss some high-level up-to-date reviews that could lead into the world of possible approaches taken and their advantages and disadvantages. I was counting on books in that, but unfortunately they aren't as accessible as papers.

Some time ago I went into reading the core face detection paper by Viola and Jones about Haar cascades for object detection. It struck me that their method which appeared to be very fruitful in face and object detection didn't get into common practice in speech recognition.

Basically the idea of their method is that it's possible to reduce search space significantly with very weak set of classifiers. For example you can easily find out that there is no face on the green grass and thus you can skip this region. This is rather fruitful idea that you can classify negatives much more accurately then positives. Putting things into cascade make search space tiny and recognition fast and efficient. Certainly it's not the only algorithm of this type, other one I met recently is bloom filters with almost the same method for efficient hash search.

The transfer of this into ASR is rather straightforward. We need to train weak classifiers that reject phone hypothesis for a given set of frames. That's actually quite easy with SVM or something built on top of existing HMM segmentation. Next, we could also apply this to a language model and reject some hypothesis which aren't possible in the language.

I haven't seen any papers on that, probably I need to search more. This idea is certainly worth to try and it should get into common ASR practices like discriminative training, adaptation with linear regression or multipass search.

Great Move To Nexiwave

2010-04-27T00:23:00.001+04:00

We decided to move all our blogs like Ben's blog, news about SearchMyMeetings and others to http://nexiwave.com. Such consolidaion will help us to manage our resources as well as will improve our presence in the web. Being more officially placed we will be more responsible for content as well, so I hope to find out here more useful matherials about speech recogniton, CMUSphinx and other related things soon.

Sorry for the inconvenience.

Intelligent Testing In ASR

2010-04-24T02:37:00.002+04:00

To continue previous topic about testing I want to share the information about nice paper I read some time ago which I wanted to bring to our daily practices.

The issue is that the current way we test our systems is far from being optimal, at least there no real theory behind that. I usually apply 1/10th rule in practice where I split data on 9/10 training set and 1/10 testing set. This was done in voxforge as well. Not so good thing since with 70 hours of Voxforge data test set grows to 7 hours and it takes ages to decode it. I took this rule from festival's traintest script. And that's more or less common practice in ASR while things like 10-fold cross-validation aren't popular for computational reasons mostly. Suprisingly, problems like that could be easily solved if only we could focus on real goal of testing - estimating the recognition performance. All our test sets are oversized, one could easily find that looking on decoder results during testing. They tend to stabilize very quickly unless there is some data inconsistency.

Speech recognition practice unfortunately doesn't cover this even in scientific papers. Help comes from character recognition. The nice paper I found is:

Isabelle Guyon, John Makhoul, Fellow, IEEE, Richard Schwartz, and Vladimir Vapnik
What Size Test Set Gives Good Error Rate Estimates?
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 20, NO. 1, JANUARY 1998

Authors address the problem of determining what size test set guarantees statistically significant results in a character recognition task, as a function of the expected error rate. The paper is well written and actually rather clear to understand. There are no complex model behind the testing, nothing speech-specific. There are two valuable points there:

The approach that puts reasoning behind test process
The forumlae itself

To put it simple:

The test set for medium vocabulary task could be small. If word error rate is expected to be like 10%, by the table on page 9 you can get that to compare two configurations with difference 0.5% absolute you need only 13k words data size.

That's four times smaller than current Voxforge test set. I think this estimate can be even improved if we'll specialize with speech. I really hope this result will be useful for us and will help us to speedup the process of application testing and optimization.

Testing ASR with Voxforge Database

2010-04-18T14:33:00.003+04:00

In development and research the critical issue is proper testing. There was some buzz about that recently, for example at MLoss blog where pros for using open data are considered. One interesting resource that started some time ago is http://mlcomp.org/, which combines both open data and open algorithm automatically selecting the best method for the common data set. I think it's not that easily implementable idea because "best" is often different. Sometimes you need speed, sometimes generalization.

In our case by using open data you can easily solve the following problems:

Test the changes you've made in speech decoder and trainer on a practical large-vocabulary database
Estimate how recognition engine performs. It's not just about estimating the accuracy but also about other critical parameters like confidence score quality, decoding speed, lattice variability, noise robustness and so on.
Share the bugs you've found. The situation is that we could definitely fix minor problems that are easy to reproduce. Any serious problem ultimately requires a reproducable test example.

I actually wanted to describe how this works in practice right now. The solution we propose for CMUsphinx developers is a Voxforge database. It's not the only open data source out there, but I think it's most permissive one. Old an4 is good for quick tests, but it definitely doesn't satisfy our needs because everything except large vocabulary recognizer have little sense nowdays.

The database itself is about 75 hours of read speech taken from various sources. The speech was collected by web collection application, from public audiobooks and so on. There are number of accents, sound quality also varies. The information on the source is not very reliable, but that's something we should live with. The speech is segmeneted on utterances and for each utterance transcription is provided. Voxforge DB has some disadvantages like very limited vocabulary (around 5000 words), rather limited focus on read texts, but we need to work on them. I believe issues will be fixed soon.

Voxforge model for CMUSphinx is trained periodically, the recent one could be downloaded here:

http://www.repository.voxforge1.org/downloads/Main/Trunk/AcousticModels/Sphinx/

The model was trained with SphinxTrain and config could be found inside as well as other scripts to train it. Test results are also inside README file in model. The corresponding performance test is provided in sphinx4.

Steps to start with it are simple:

Download model from the link above, unpack it.
Check scripts in script subfolder to setup training:

Download audio with wget
Unpack it
Convert flat co wav
Make transcription file from PROMPTS
Extract features
Run the training with ./scripts_pl/RunAll.pl

Test the recognition performance, setup other testing environment for various numbers you need.

The space required to train Voxforge is about 10Gb on disk. On a dual-core machine training should take about a day.

Here are the results of the sphinx4 performance test of the acoustic model voxforge-en-r0.1.3:

[java] # --------------- Summary statistics ---------
[java]    Accuracy: 90,613%    Errors: 4679  (Sub: 2904  Ins: 559  Del: 1216)
[java]    Words: 43889   Matches: 39769    WER: 10,661%
[java]    Sentences: 4682   Matches: 3050   SentenceAcc: 65,143%
[java]    Total Time Audio: 22090,44s  Proc: 116616,04s  Speed: 5,28 X real time
[java]    Mem  Total: 1993,75 Mb  Free: 806,02 Mb
[java]    Used: This: 1187,73 Mb  Avg: 1112,13 Mb  Max: 1962,46 Mb

To run this test do the following:

Checkout latest sphinx4
Go to sphinx4/tests/performance/voxforge_en
Download and unpack audio files into wav folder using script build.sh from voxforge-en-r0_1_3/scripts
Download acoustic model and unpack it, creating etc folder with lm and voxforge_en_sphinx.cd_cont_3000 folder with model files
Start test simply typing ant in command line

You see the decoding takes like 32 hours. Well, it can be faster on multicore machine, but you need to change the configuration of ThreadedScorer to explicetely start multiple scoring threads. Unfortunately automatic detection of number of cores doesn't work in SUN's JVM.

Speaking about Voxforge, we definitely need to thank Ken McLean (great work, Ken!) who is running Voxforge for several years already. It's also worth to mention that without contributors who submitted their speech this project will not be that thriving. So, start with using Voxforge for your developments, report bugs and send us comments. That would be appreciated.

Finally Sorted Out Workshop Materials

2010-03-26T00:34:00.002+03:00

Since they are more CMUSphinx official documents, I posted notes about workshop and meetins after that on the website:

Sphinx Users And Developers Workshop 2010 Results

Development Meeting Notes

I'm pleased to get so many important things planned and few very important issues cleared up. For example I didn't completely understand why lattices in sphinx4 are so bad. I hope other participants had some productive results too.

It would be nice to get some pictures from workshop, but unfortunately none available now. Probably sometime later. So, just Dallas one.

ICASSP 2010

2010-03-22T22:22:00.006+03:00

So, I'm back from ICASSP in Dallas, TX. It was very impressive conference with lots of interesting and inspiring presentations, meetings and discussions. Amazing everyone was there and I've finally met all the speech people who guided me for so long time. I've met ASR people Bhiksha Raj, David Huggins-Daines, Rita Singh, Richard Stern and TTS people Alan W. Black, Keiichi Tokuda, Simon King, Heiga Zen. I was pleased to meet second time wonderful guys like Evandro and Peter. Worth to mention that I was able to listen talks by famous people like Hynek Hermanski.

We had Sphinx Users and Developers Workshop there and also two CMU Sphinx development planing meetings. But they are subject for another post. This one is just about interesting ideas presented on the conference by other people. I didn't have time to attend every presentation out there, I think it was impossible. You have to find the time for sightseeing and there were often two or three parallel lecture sessions and also poster presentations which I liked most. I think poster presentation is the best way to access author, ask him questions and get feedback. Many posters were so popular it was almost impossible to get to the stand.

Anyway, the amount of talks I've got already exceeds what can be consumed in a week. It would be nice to get one day all information about current research collected into structure or wiki-like resource. It's a huge work for the future though.

So here are some presentations and ideas I've met there and found them to be worth attention:

Robust Speaking Rate Estimation Using Broad Phonetic Class Recognition

Authors: Jiahong Yuan; University of Pennsylvania, Mark Liberman;
University of Pennsylvania

Presented work is about using easy classifier to get some specific data about speech like to estimate deletions in syllables and thus speech quality. This is actually very promising approach which is ignored for some reason in most places where it seems to be practical. For example it's not clear for me why speaker identification framework doesn't try to find phonetic classes first and build GMM only after that. It seems to be a natural approach to improve SID performance.

Broad phonetic classes remind me the idea from the famous face recognition algorithm by Viola and Jones about applying cascade for fast classification. This idea could be applied to speech in some form like authors suggest I think.

Clap Your Hands! Calibrating Spectral Substraction for Reverberation

Authors: Uwe Zaeh, Korbinian Riedhammer, Tobias Bocklet, Elmar Noeth

Reverberation was very popular on this conference and especially it's important for meetings. Various speech system require various noise cancellation. Far microphone need to fix reverberation from room, close microphones need to fix clicks and so on. Far microphones sometimes do calibration for reverberation estimation. This defines the set of components sphinx4 could have to deal with various environment conditions. Right now they are simply missing.

Detecting Local Semantic Concepts in Environmental Sounds using Markov Model

Authors: Keansub Lee, Daniel Ellis, Alexander Loui

Interesting that classification database for this task is available at http://labrosa.ee.columbia.edu/projects/consumervideo/, this could be a base for non-speech recognition research.

Learning Task-Dependent Speech Variability In Discriminative Acoustic Model Adaptation

Authors: Shoei Sato, Takahiro Oku, Shinichi Homma, Akio Kobayashi, Toru Imai

Discriminative approaches are popular now days. Direct optimization of the cost function could serve on various stages of training process. In this work for example the set of subword units is selected to minimize decoding error rate.

An Improved Consensus-Like method for Minimum Bayes Risk Decoding and Lattice Combination

Authors: Haihua Xu, Daniel Povey, Lidia Mangu, Jie Zhu

This deals with specific criterion for lattice decoding. Not just best path could be chosen but other criterion like consensus could also apply. For me personally it would be very interesting to formalize and apply the criterion that will ensure grammatical correctness of the result. I haven't found anything on this yet.

Discriminative training based on an integrated view of MPE and MMI in margin and error space

Authors: Erik McDermott, Shinji Watanabe, Atsushi Nakamura

Interesting to find out that real math goes into ASR. Basically it was a long waited thing and it seems it was started by Georg Heygold with his works on MMI and other methods. It would be nice to review this area to get some idea what's the outcome of it. Heygold was sited in almost every presentation, so it's really getting popular.

Balancing False Alarms and Hits in Spoken Term Detection

Authors: Carolina Parada, Abhinav Sethy, Bhuvana Ramabhadran

It's interesting to see what tools are used. WFST's are very convenient and used by everyone. IBM, Google, AT&T. This is also a topic for separate post.

Bayesian Analysis of Finite Gaussian Mixtures

Authors: Mark Morelande, Branko Ristic

Rather old idea (there are similar papers from 1998) to use Bayesian learning to estimate number of mixtures in the model. I'm in favor of the approach to estimate all model parameters including number of mixtures, language weight and number of senones at once.

Improving Speech Recognition by Explicit Modeling of Phone Deletions

Authors: Tom Ko

Pronunciation variation by phone deletion looks very promising since traditional linguists mostly complain about sequential HMM model which doesn't handle deletions correctly. Unfortunately, the effect of this seems to be small. The improvement cited is only from 91.5% to 92%.

An Efficient Beam Pruning With A Reward Considering The Potential To Reach Various Words.

Authors: Tsuneo Kato, Kengo Fujita, Nobuyuki Nishizawa

Beam pruning according to the number of reachable words or to other the risk function. Good idea to implement in sphinx4 to speedup recognition. Factor cited is 1.2 for a large vocabulary.

That's it. I missed Friday and most early mornings unfortunately, so something interesting could be there. I'm sure you could select your own set. It's interesting to look on it.

Sphinx4 1.0 beta4 Is Released. What's next?

2010-03-01T22:56:00.005+03:00

So, almost according to schedule, sphinx4 was released yesterday. Check the notes at

http://cmusphinx.sourceforge.net/2010/03/sphinx4-1-0-beta-4-released/

Most notable improvements were already discussed here, so let me try to plan what the next release will be. Trying to be realistic in plans, I don't want to promise everything at once. Here is some attempt to forecast the next release notes

The biggest issue with sphinx4 is actually documentation. Current poll on CMUSphinx website clearly shows that. Personally I sometimes think that perfect documentation will not help if system doesn't work, but at least it will make product attractive and easy to use. My idea is that we need to have more developer-level documentation - tutorial, examples, task-oriented howtos. It's unlikely we'll be able to write something that is good enough as textbook on speech technologies. But we need to prove the point that it's possible to build ASR system without understanding who is Welch.

On the code side, we face a biggest challenge since sphinx4 was designed. We need to move to the multipass system. It's not just about rescoring, it's about plugging diarization framework from LIUM, it's also about making sphinx4 suitable for both batch and live applications. That's the serious issue.

The reason is that currently sphinx4 architecture is flow-oriented. It's built like a single pipe of components each passing audio to other. This is good for live applications, but not so good for batch ones. You get troubles when you need to split pipe or merge it later. In batch application one could have a huge benefit from looking on recording as a whole and returning to recording multiple times. For example, you could estimate noise level properly and just cleanup audio on the second pass. Such multipass decoding doesn't well fit into pipe paradigm. On the other side, changing it to purely batch will create issues for live applications.

So we are in trouble. We have to invent some combined scheme probably and create a hybrid of pipe and batch approaches. I was thinking about knowledge base scheme when information about stream is stored in some database as processing goes. Database cleanup policies could emulate both pipe (when database is immediately cleaned) and batch approaches (when database is kept even over sessions). Festival utterances remind me such data processing scheme between. Anyway, this idea is not finalized yet.

We also expect to see a lot of movement from CMUSphinx Workshop in Dallas and in Google Summer of code participation. I hope issues described above and some more interesting issuses will be resolved till next release in August. Let's discuss the rest then!

Speech Recognition in GSoC Done Right

2010-02-27T02:16:00.003+03:00

From year to year many end-user projecs are trying to push ASR with the help of Google and studens of the Summer Of Code program. If CMUSphinx team knows all about ASR, why should we stay away from that?

I had diverse experience with Google Summer Of Code before, but I still like this process and enjoy communication with new people. I think we have good chances to succeed here. So I started and filed an application proposal and the initial list of ideas

http://cmusphinx.sourceforge.net/wiki/summerofcode2010

I will submit this proposal on March 8 after program start. We need more ideas now. As much as you can generate

http://cmusphinx.sourceforge.net/wiki/summerofcodeideas

We need to have more or less representative list. If you want to be a mentor, don't hestitate to write down your irc nick as well.

Noise reduction filtering in sphinx4

2010-02-12T04:24:00.000+03:00

There is a huge gap between stock sphinx4 and real ASR system since critical parts like noise filtering, speaker diarization and postprocessing are missing. Not to mention the online adaptation. The default frontend is less then optimal for several reasons. For example it doesn't handle DC offset at all, it also uses energy-based endpointer in time domain, thus not so robust to additive noise.

As of today sphinx4 includes the implementation of Wiener filter that reduce noise and helps the voice activity detector as well. To try it checkout latest trunk and change the frontend pipeline as following:

<item>audioFileDataSource </item>
<item>dataBlocker </item>
<item>preemphasizer </item>
<item>windower </item>
<item>fft </item>
<item>wiener </item>
<item>speechClassifier </item>
<item>speechMarker </item>
<item>nonSpeechDataFilter </item>
<item>melFilterBank </item>
<item>dct </item>
<item>liveCMN </item>
<item>featureExtraction </item>

Then define wiener component:

<component name="wiener"
type="edu.cmu.sphinx.frontend.endpoint.WienerFilter">
<property name="classifier" value="speechClassifier"/>
</component>

This frontend is stable to DC and also handles noise better. To try the noisy input, you could mix white noise with sox:

 sox 10001-90210-01803.wav noise.wav synth white
 sox noise.wav smallnoise.wav vol -45d
 sox -m 10001-90210-01803.wav smallnoise.wav 10001-90210-01803-noisy.wav

It would be nice to try with Aurora database as well.

This filter is very simple and has a number of disadvantages. For example it corrupts spectrum with harmonic noises sometimes and thus makes recognition even worse. But it definitely helps in presense of noise. Let's hope one day more sophisticated implementations like Ephraim-Malah filter, or even noise reduction with vector taylor series will be made available in default configurations.

All ideas are already generated

2010-01-31T19:08:00.002+03:00

After seeing flash websites take enormous amount of my CPU got a cool idea today about using flash for distributed computing. Basically everything is already in place. You setup webserver, share content with flash, it runs on client computer and does calculations uploading the result from time to time. Certainly I wasn't the first who invented that, see for example

http://www.vershun.com/computers/hidden-flash-applications-as-distributed-computing-clients.html

and

http://www.csc.villanova.edu/~tway/courses/csc3990/f2009/csrs2009/Kevin_Berry_Grid_Computing_CSRS_2009.pdf

Though such ideas are rather recent and the question is how to make this framework widely used. Looking at current load of the computer at sourceforge it's most likely already used by some websites :)

Training process

2010-01-31T03:48:00.000+03:00

What I really like in Sphinxtrain is that it provides straightforward way for training an audio model. It remains unclear for me why everyone bothers with HTKBook while there is clean an easy way to train the model. One should just define the dictionary and transcription and put the files in the proper folder. Anyway, I'm continuously thinking about the way sphinxtrain process could be improved. Currently it indeed lacks a lot of critical information on training and that makes look uncomplete.

Basically here is what I would like to put into the next versions of sphinxtrain and sphinxtrain tutorial:

Description on how to prepare the data
Building of the database transcription. Between, what bothers me last month is the requirement to have fileids. I really think the file with fileids could be silentely dropped. What's the problem to get the id of the file from the transcription labels
Automatic splitting on training data, testing data and development data. I see development data presense as a hard requirement for the training process. Unfortunately, current documentation lacks it. There could be code to do that, but for most databases it's automatic of course.
Bootstrapping from a hand-labelled data. I think this as an important part of training, HTK results confirm that. In general it repeats human language learning, so I think it's natural as well.
Training
Optimizing number of senones, mixtures on a devel set
Optimizing most important parameters like language weight on the development set. This part is complicated as I see it. First of all the reasononing behind proper language weight scaling is still unclear for me, I could one day write a separate post on it. Basically it depends on everything, even on the decoder
Testing on the test set

If it will be possible to keep this as straightforward as it is now that would be just perfect. Probably if I'll start to write the chapter in a week, this could be ready till summer.

Moving Beyond the `Beads-On-A-String'

2010-01-21T02:19:00.002+03:00

Recently I've got interested in quite a large domain of speech recognition research where old school linguistic meets modern speech recognition. Basically the idea is that in spontaneous speech variativity is so huge that phonetic transcription from the dictionary doesn't apply well. In plain CMUSphinx setup linguistic information about phones is almost lost like we don't care if phone is labial or dental. It is used in a decision tree building but it's not clear if such usage helps. It's definitely not so good to drop such a huge amount of information that could help with classification. So this idea is actively developed and you can find there everything you miss probably - distinctive phone features, landmarks, spectrogram recognition.

I went through the following articles, the number of methods, approaches and implementations described there is really huge. In other articles it's going to be even bigger:

S. King, J. Frankel, K. Livescu, E. McDermott, K. Richmond, and M. Wester. Speech production knowledge in automatic speech recognition. Journal of the Acoustical Society of America, 121(2):723-742, February 2007. PDF

Moving Beyond the `Beads-On-A-String' Model of Speech by M. Ostendorf PDF

Speaking In Shorthand - A Syllable-Centric Perspective For Understanding Pronunciation Variation by Steven Greenberg PDF

To be honest the only idea from the articles that grown in my mind is that reductions on fast speech are root of the problem. I also noticed it in early days and was experimenting with a skip states. Skips didn't give any improvements except reduced speed. It will probably help to automatically increase lexicon variability and use forced alignemnt to get proper pronuciation at least at training stage. As I understood I just need to take a dictionary with syllabification and create a dictionary with a lot of reduced variants where onsets are kept as as and codas are reduced in some form. Then we force align, then train. Probably acoustic model will be better then.

Another striking point was that I haven't found any significant accuracy improvement result in the articles I read. Improvement like 20% with discriminative training could make any method widely adopted but nothing like that is mentioned. Probably this research is in very initial state.

Three Generation of IVR Systems

2010-01-16T05:10:00.004+03:00

Recently I invented new nice concept for marketing people. Basicallly there are three generations of IVR systems right now:

Generation 1.0 - Static systems based on VoiceXML. It was suprising for me they are in wide use now and a lot of products are dedicated to their optimization/develoment. There are IDE's and a lot of testing tools, recommendations how to build proper VoiceXML. Come on, it's impossible to do that. It's something like static HTML websites that were popular in 1995. I don't believe any changes like javascript inside in VXML 3.0 will stop it slow death.
Generation 2.0 - Dynamic systems like Tropo from Voxeo. Much easier, much better. More control over content, more integration with the business logic. I really believe it's next generation because it gives developer much more control over the dialog. At least with the power of real scripting language like Python you'll be able to implement something non trivial with just several lines of code. That's AJAX or ROR in speech world.
Generation 3.0 - Semantic based IVR. This consists of three components - large vocabulary recognizer, semantic recognizer on top of it and even-based actions on top of it. Probably also an emotion recognition and more intelligent dialog tracking. As I see the developer has to define the structure of the dialog and provide handlers. Such system was described and developed in CMU long time ago already and also it's described in all ASR textbooks. But I'm not aware of any widely known platform allowing to do this kind of IVR. Once again it shows how big the gap is between the academia and software developers.

If you are planning to create IVR application with CMUSphinx, please, consider IVR generation 3 as your base technology ;) And don't forget to share the code.

Update:

Very much on the same topic from a wonderful Nu Echo blog:

http://blog.nuecho.com/2010/01/25/voice-apis-back-to-basics/

PLP is going to be default soon

2010-01-12T03:07:00.000+03:00

It looks like MFCC features are going to become a history. Everyone is using 9 combined PLP frames + later LDA projection to 40-50 values. Few examples including Google in it's audio indexing system, IBM and BBN see system description in results, OGI/ICSI and many others.

The issue right now is that sphinx4 PLP implemetation seems to be broken, it produces kind of garbage features which doesn't give enough accuracy after training. Luckily there is HTK. Once this issue will get fixes, I think I'll retrain PLP + MLLT model for Voxforge. Unfortunately I don't have any definite plan for implementation of PLP in sphinxbase.

Post-biological ASR

2010-01-05T01:18:00.001+03:00

Recently we had a discussion we had with ionel on #cmusphinx chat on freenode about what should perfect speech recognition engine do. Though I didn't quite understood the purpose of the question the answer I could give was to make it as good as native speaker in convertion of the audio into the text.

I watched today an interview with Ray Kurzweil, nothing really interesting there except the idea of post-biological future where computers replace humans. I understood that my definition of ASR is not a very good definition just because it's well established idea that computers will soon become way better than humans in most tasks, like they are already better in playing chess. I tend to forget this over and over, but it's perfectly reasonable to try to be better and not mimic human functions. Automatic recognizers could be better both in terms of speed, energy consumption and accuracy. Lets hope this year will bring us closer to such future. What will be speech then, that's the question.

Greetings and Random Thoughts

2010-01-04T01:02:00.001+03:00

So 2010 is here, Happy New Year everyone. Wish you all success and happiness and of course increased decoder accuracy! Now we have a long 10 days vacation in Russia, time to travel, eat, drink and sort out bookmarks, read books on the shelf and watch pending google tech talks. Santa also promised me to do some great changes in sphinx4, waiting for that as well.

Though Ohloh doesn't confirm that, I have a strong feeling that last year the activity around CMUSphinx definitely increased and it's usage is going to grow.

I was thinking a little what should be the direction of sphinx4 development, I think we should consider several factors here. I would be happy to see it as widely-used enterprise level speech recognition engine with a great list of features, but I completely understand that due to the lack of resources it's naive to think we'll be able to do it all. We definitely need to find a market sector for the sphinx project and grow using it. There are already well established projects like HTK that are used widely with their own set of strong and weak features. Julius is used widely as a large vocabulary speech recognition engine with HTK models. It's hard to compete with HTK for us just because it will take years to add that flexibility we probably don't even need. Consider variable of adjustable number of states per phone, something that is only proven to be useful for a small vocabulary task, something we aren't really interested in and I hope will not be interested in a near future. What could be different is our practical orientation.

Many project in speech domain and releated areas are often grown from the research projects and though flexible sometimes, often really unusable in applications since they aren't really designed for that. Usually a research project isn't well documented, has a lot of ways to implement the same thing and some of them are sometimes obsolete. Bugs are rarely fixed and documentation almost missing. Releases are not stable. It's definitely a large field for a commercial support company.

There is a different side, many projects are created in order to solve the user needs, more or less well documented and have stable interfaces, large open community but they are doing so wrong internally I always wonder how they are used at all. Espeak with it amazingly bad speech synthesis quality and even more amazing popularity. Out-of-date synthesis method doesn't let it be good with any possible modifications. Another example of this is strikingly Lucene. Unlike lucidimagination blog states states lucene community is thriving, it's definitely not true. The research articles like Lucene and Juru at Trec 2007: 1-Million Queries Track definitely shows there is something wrong with Lucene. Basically it lists several trivial changes well known in research community that make Lucene perform two times better on a standard test. I can't understand why this wasn't integrated into stock after three years since article was published.

Let's hope CMUSphinx will find it's place somewhere in the middle. Also, let's hope this year will bring more useful posts decreasing information overload that is certainly going to be a problem in a near future.

Pocketsphinx Success Story

2009-12-16T04:31:00.002+03:00

I was pleased to find out renovated website by Keith Vertanen and an amazing real-life example of creation of the pocketsphinx application Parakeet, a dictation app with correction for Nokia N800:

http://www.keithv.com/software/parakeet/n800/

Keith website and models are invaluable resource for Sphinx developers, in particular his lm_giga models are still the models I would recommend to take for adaptation. But seeing this application in action and reading about it's development should really give a good insight into the process of speech recognition application building, having all good practices described.

Core Ideas Behind Speech Recognition

2009-12-12T20:07:00.002+03:00

While tunning the acoustic model I've got again 40% WER and in the log the following:

THEY'RE ONLY ALLOWED TEN TO a CLASS (a100)
*** HAD BARELY LEAD CANDY a CLASSIC (a100)
Words: 7 Correct: 1 Errors: 6 Percent correct = 14.29% Error = 85.71% Accuracy = 14.29

If you'll check this recognition error you'll find that it's almost impossible to find the reason of it and fix it. Probably some senone was trained incorrectly, probably CMN give error or clipping made MFCC wrong. Probably some noise in the middle break the search. There is nothing you can do about it. That made me think about foundations of ASR.

Considering a speech recognizer engine like sphinx4 one could extract the set of core ideas that lie behind it. Same ideas are usually described in speech recognition textbook. Basically they are:

MFCC feature extraction from periodic frames (or PLP, doesn't matter)
HMM classifier for acoustic scoring (with state tying)
Trigram word-based language model (higher grams aren't effective, lower not so precise)
Dynamic search with pruning

Surely commercial systems have a lot of improvement over this baseline, but the core is still the same. Such foundations are certainly reasonable and checked over the years in practice. It's hard to argue agains them. Often newbies tell that something is wrong here, but basically it's because they don't really understand how it works. Critisizm comes from old-school linguists, who do everything with rules and mostly interested in usual cases like pronuciation of "schedule" than in theory.

The only issue is that growing amount of unsolvable unexplainable problems like the problem with accuracy above breaks this theory. Quite unusual fact for me as mathematician since in mathematics theory rarely become invalid. They tranform, grow but usually all of them are stated once in forever. In natural sciences like physics it's usual. The aether theory and mechanical explanation of gravitation is the good example that come to my mind. So there is nothing wrong that this ideology of speech recognition could be reviewed and modified according to the recent findings.

What would I put into such modified theory:

Multiresolution feature extraction. Starting from RASTA to fMPE and spikes. The idea is that signals are sparse and nonperiodic, the signal range from 10ms to more than 10 seconds and they all needs to be passed into classifier.
Some acoustic classifier that without selected states. The idea of phone is probably natual in slow speech or in teching but I heard so many complains about it. Dropping it seems promising indeed since speech is a process, not a sequence of states. Unfortunately I haven't found any article on this yet. Another promising idea here is margins which could help with out-of-model sounds.
Subword stage. I more think that languages with developed morphology like Turkish is more the rule than the exception. Being able to recognize a large set of words in the language is a core capability of usable recognizer and that forces it to operate on subword units. Even English recognizer could benefit from this.
Language model without backoff. I recently had discussion with David about that and would like to thank him for this idea. Indeed counts of the model seems to be a reasonable statistics one could keep and use. But further calculation of the language weight should be modified completely. Again, there must be margin to strip some combinations that will never appear in the language. Such idea of using prohibitive rules stays in my mind for a long time. It would be also nice to find any recent articles on this. But there must be a component that will invalidate the output like "barely lead candy".
Machine learning for backoff calculations. In continuation of the previous point, the backoff weight should have much more complex structure. Not only trigrams containing the words need to be taken into account, a semantic class should be counted, trigrams with similar class of words ought to be considered. Today I even had idea to apply machine learning to calculate the backoffs. I'm sure someone did this before, also need to look at articles about using machine learning methods to restrict search.

As for tree search, it luckily will stay as is, nothing to argue against it right now. Not sure that such modifications are breaking the initial theory, one could say they aren't really different. I still think they could explain the speech better and help to build better speech recognizer.

New CMUSphinx Website Alpha

2009-12-04T23:11:00.000+03:00

Most CMU Sphinx websites are outdated. The problems with the one at sourceforge are:

Not so modern style
No interactivity
Loosely organized outdated information
Hard to manage/update
No CMS/search

Also there is a generic problem with the quality of documentation available. A lot is quite outdated and just confusing.

So I wanted to build a new website for a long time. This site is supposed to be central point for all sphinx tools, including pocketsphinx, sphinx4, cmuclmtk and sphinxtrain.

New website is supposed to be interesting. This site is going to bring more interactivity (sharing, blog posts, voting, comments). It looks a little bit bloggish, but I think it's even better. It would be harder to write more interesting posts, so I invite everyone to participate. I'm sure you have something to say.

So here is the proposed demo version
http://cmusphinx.sourceforge.net/wordpress

We are in process of tranferring the information to the new website, so I really hope to see it running very soon.

How to create a speech recognition application for your needs

2009-11-30T23:42:00.002+03:00

Sometimes people ask: why there is no high-quality open source speech recognition applications (dictation application, IVR applications, closed-captions alignment, language acquisition and so on). The answer obviously is that nobody wrote them and make them public. It's often noted, for example by Voxforge, that we lack the database for the acoustic model. I admit Voxforge have it's reason to state we need a database. But that's only a little part of the problem, not entirely the problem as a whole.

And as it always happens, the statement of the question doesn't allow constructive answer on it. To get constructive answer you need the following question: How do I create a speech recognition application.

To answer on this let me provide an example. Consider we want to develop flash-based dictation website. The dictation application consists of the following parts which should be created

Website, user accounting, user-dependent information storage
Initial acoustic and language models trained with Voxforge audio and other free sources transmitted through Flash codecs
Recognizer setup to convert incoming streams into text. Distributed computation framework for the recognizer
Recognizer frontend with noise cancellation and VAD
Acoustic model adaptation framework to let user adapt the generic acoustic model to their pronunciation
Language model adaptation framework
Transcription control package that will process commands during dictation like error correction ones or punctuation ones.
Post-processing package to put punctuation and capitalization, date and acronym post-processing
Test framework for dictation with dictation recordings and ability to check dictation effectiveness

Everything above could be done with open source tools and have approximately equal complexity and require minimum specialized knowledge. Performance-wise this system should be usable for a large vocabulary dictation for a wide range of users. The core components are:

Red5 streaming server
Adobe Flex SDK
Sphinx4
Sphinxtrain
Language model toolkit
Voxforge acoustic database

So you see mostly it's just an implementation of the existing algorithms and technologies. No rocket science. This makes me think that such application is just a matter of time.

Multiview Representations On Interspeech

2009-11-27T23:51:00.002+03:00

From my experience, in every activity it's important to have multilevel view of any activity, interesting is that it's both part of Getting Things Done and just a good practice in software development. Multiple models of the process or just different views help to understand what's going on. The only problem is to make those views consistent. That reminds me the Russian model of the world.

So it's actually very interesting to get a high-level overview of what's going on in speech recognition. Luckily to do that you just need to review some conference materials or journal articles. Latter is more compicated, while former is feasible. So here comes some topics from the plenary talks from Interspeech. Suprisingly they are rather consistent across each other and I hope they really present trends, not just selected topics.

Speech To Information
by Mari Ostendorf

Multilevel representation gets more and more important, in particular in speech recognition. The most complicated task - spontaneous meetings recording requires unifiication of the recognition efforts on all levels from acoustic representation to semantic one. Nice to call this approach "Speech To Information", as a result of speech recogntion not just the words are repaired but even syntactic and semantic structure of the talk. One of the interesting tasks is for example restoration of punctuation and capitalization, something that SRILM does.

Good thing is that testing database for such material is already available for free download. Very uncommon situation to have such representative database in free access. AMI corpus looks like an amazing piece of work.

Singe Method
by Sadaoki Furui

WFST-based T3 decoder looks quite impressive. Single method of data representation used everywhere which more importantly allows combination of the models gives wonderful opportunity. For example consider the example of building high-quality Icelandic ASR system combining WFST for English one and very basic Icelandic one. I imagine the decoder is really simple since basically all structures including G2P rules, language and acoustic model could be weighted finite-state automata.

Bayesian Learning
by Tom Griffiths

Hierachical bayesian learning and things like compressed sensing seems to be a hot topics in mashine learning. Google does that. There are already some efforts to impelement a speech recognizer based on hierachical bayesian learning. Indeed it looks impressive to just feed the audio to the recognizer and make it understand you.

Though probabilistic point of few was always questionable opposed to precise discriminative methods like MPE I'm still looking forward to see progress here. Despite huge amount of audio is required, like I remember there were estimates about 100000 hours I think it's feasible nowdays. For example it already recognizes written digits, so success looks really close. And again, it's also multilevel!

Few open source speech projects

2009-11-25T02:01:00.001+03:00

It's interesting that a lot of activity around speech software happen recently. I'm probably too impatient trying to track everything interesting. Even through ISCA-students added twitter feed recently, their website still needs a lot of care. Hopefully, Voxforge will become such resource one day. There is a growing amount of packages, tools, projects and events.

For example I've got in touch with SEMAINE project lead by DFKI recently, an effort to build a multimodal dialogue system which ca, interact with humans with a virtual character, sustain an interaction with a user for some time and react appropriately to the user's non-verbal behaviour. The sources are available and the new release is expected in December as far as I understood, so I'm definitely looking forward. The interesting thing is that SEMAINE incorporates emotion recognition framework with libSVM as a classifier, such framework would be useful in sphinx4 for example. Actually a lot of news come now from the European research institutes, projects from RWTH or TALP promise a lot.

Another example is that I was pleased to find out that in 2009 there was a rich transcription evaluation. It's interesting why results aren't available still and what was the progress on meeting transcription task since 2007.

Probably I would sleep better if I didn't knew all above :)

Using SRILM server in sphinx4

2009-11-12T12:38:00.001+03:00

Recently I've added the support for the SRILM language model server to the sphinx4 so it's possible to use much bigger models during the search keeping the same memory requriements and, more important, during lattice rescoring. Lattice rescoring is still in progress, so here is the idea how to use network language model during search.

SRILM has a number of adavantages for example it implements few interesting algorithms and even for simple tasks like trigram language model creation it's way better than cmuclmtk. At least model pruning is
supported.

To start first dump the language model vocabulary since it's required in linguist

ngram -lm your.lm --write-vocab my.vocab

So start the server with

ngram -use-server 5000 -lm your.lm

Configure the recognizer

<component name="rescoringModel"
   type="edu.cmu.sphinx.linguist.language.ngram.NetworkLanguageModel">
   <property name="port" value="5000"/>
   <property name="location" value="your.vocab"/>
   <property name="logMath" value="logMath"/>
</component>

And start the lattice demo. You'll see the result soon.

Adjust the cache according to the size of your model. It shoudlnt' be large for a simple search. Typically the cache size isn't more than 100000 for a simple search.

Still, usage of the large-gram model is not reasonable for a typical search because of the large amount of word trigrams that should be tracked. It's more efficient to use trigram or even bigram model first and make a second recognizer pass with the rescored language model. More details on rescoring in the next posts.

Rhythm of British English in Festival

2009-11-07T03:43:00.000+03:00

Interesting how ideas rise from time to time in seemingly unrelated places. Recently I've read nice post in John Well's blog about the proper RP English rhythm and now that issue raised again in gnuspeech mailing list where Dr. Hill cited his work

JASSEM, W., HILL, D.R. & WITTEN, I.H. (1984) Isochrony in English speech: its statistical validity and linguistic relevance. Pattern, Process and Function in Discourse Phonology (collection ed. Davydd Gibbon), Berlin: de Gruyter, 203-225 (J)

I spend some time thinking about how this rhythm is handled in Festival and came to the conclusion there is no such entity there. Probably it's somehow handled by CART for duration and intonation prediction, but not as a separate entity. Though many voices are supposed to be US English, I still think they can benefit from a proper rhythm prediction. Try the example from the movie, "This the house that Jack built" with artic voices. Check if Jack gets enough stress.

Blizzard 2009 results available

2009-10-18T17:58:00.007+04:00

It was pleasant to find out that results of the Blizzard Challenge 2009 are now available. Thanks a lot ot organizers and participants!

Reading the articles took me half of the day trying to solve usual Einstein-type puzzle of figuring out who give the best results there and what was changed. Unfortunately it takes to much time to read everything in details. There is no summary on methods/systems used this year, the archivements from the last year and explanations of the results provided. I could only start with the following:

iFlytek Speech Lab and IVO Software are still the best. Unit selection systems win.
DFKI which I was fan of can't unfortunately jump to a commercial level even with unit selectoin. That probably means that not only unit selection is a key issue.
I like the progress muXac and Mike are doing over years.
ES3 task with building voice from small amount of speech is kind of senseless. Don't we want to use voice adaptation in this case
Interesting that machine learning for join and target cost optimization is popular nowdays
Though there was telephone TTS task it seems for me that nobody did anything related to the TTS over the telphone lines. The differences shouldn't be large, only 8kHz is the issue or even the advantage, but even this moment is not covered in any articles or at least I didn't notice it.

Short summary on systems:

Aholab - unit selection, spent one day on building the voice so nothing good to expect
WISTON - Mandarin prosody is a key feature, but article doesn't describe challenge
Cereproc - experiment with combining HTS and unit selection, bad results or unknown reason, 4 man-days spent
CMU - article is not available, but you can try clustergen yourself in stock festival
CSTR - CSTR has started investigations on HTS methods. Good start, no results yet.
DFKI - spent year on adding Turkish TTS and Mary 4.0 implementation
Edinburgh/Idiap - interesting unsupervised entry, results are obvioulsy lower
I2R - good TTS, unit selection
Ivona - unit selection with pitch modifications by interestingly named algorithm, best English one together with iFlytek
CircumReality - unit selection with pitch modification by TD-PSOLA, best progress over years
NICT - HTS, GV, MGE and a lot of math
NIT - HTS with STRAIGHT, best HTS here, best Mandarin as well
NTUT - Mandarin HTS, not so interesting
PKU - Another Mandarin HTS with STRAGHT
Toshiba - Good unit selection system, interesting method about fuzzy combining units.
iFlytek - HMM-driven unit selection, best English one together with Ivona.
VUB - unit selection with WPSOLA, average, though interesting link on SPRAAK open source recognition toolkit, which is not completely open but has interesting description.

Still, the challenge itself is very interesting and I'm looking forward on the next challenge results.

Another cool bit if hardware for database training.

2009-10-16T11:43:00.000+04:00

It's sometimes hard to adopt quickly the new opportunities world provide. I'm being reading now Innovator's Dilemma by Clayton M. Christensen. Thanks to Ellias for the advice, it really seems like a good book.

The interesting thing is that author starts with a description of hard drive industry as the fastest one with innovations going faster than customer needs. And, what do you think? Hard drive industry strikes back with SSD drives. Well, I read they exist but didn't understand their value for acoustic model training. Even without profiling it's clear they will be extremely useful.

Say you have a medium size acoustic database of 60 hours of few gigabytes size. If you want to process it fast you need to use 8-core machine. Here comes the bottleneck, imagine 8 processes reading the feature vectors from a disk in an almost random way. No need to guess hard drive will be very busy trying to fetch all data required. SSD could definitely help here, I really need to try it soon.

CMU Sphinx Users and Developers Workshop 2010

2009-09-29T22:41:00.001+04:00

I'm happy to announce

The First CMU Sphinx Workshop

20 March 2010, Dallas, TX, USA

Event URL: http://www.cs.cmu.edu/~sphinx/Sphinx2010

Papers are solicited for the CMU Sphinx Workshop for Users and Developers (CMU-SPUD 2010), to be held in Dallas, Texas as a satellite to to ICASSP 2010.

CMU Sphinx is one of the most popular open source speech recognition systems. It is currently used by researchers and developers in many locations world-wide, including universities, research institutions and in industry. CMU Sphinx's liberal license terms has made it a significant member of the open source community and has provided a low-cost way for companies to build businesses around speech recognition.

The first SPUD workshop aims at bringing together CMU Sphinx users, to report on applications, developments and experiments conducted using the system. This workshop is intended to be an open forum that will allow different user communities to become better acquainted with each other and to share ideas. It is also an opportunity for the community to help define the future evolution of CMU Sphinx.

We are planning a one-day workshop with a limited number of oral presentations, chosen for breadth and stimulation, held in an informal atmosphere that promotes discussion. We hope this workshop will expose participants to different perspectives and that this in turn will help foster new directions in research, suggest interesting variations on current approaches and lead to new applications.

Papers describing relevant research and new concepts are solicited on, but not limited to, the following topics. Papers must describe work performed with CMU Sphinx:

Decoders: PocketSphinx, Sphinx-2, Sphinx-3, Sphinx-4
Tools: SphinxTrain, CMU/Cambridge SLM toolkit
Innovations / additions / modifications of the system
Speech recognition in various languages
Innovative uses, not limited to speech recognition
Commercial applications
Open source projects that incorporate Sphinx
Novel demonstrations

Manuscripts must be between 4 and 6 pages long, in standard ICASSP double-column format. Accepted papers will be published in the workshop proceedings.

Important Dates

Paper submission: 30 November 2009
Notification of paper acceptance: 15 January 2010
Workshop: 20 March 2010

Organizers

Bhiksha Raj - Carnegie Mellon University
Evandro Gouvêa - Mitsubishi Electric Research Labs
Richard Stern - Carnegie Mellon University
Alex Rudnicky - Carnegie Mellon University
Rita Singh - Carnegie Mellon University
David Huggins-Daines - Carnegie Mellon University
Nickolay Shmyrev - Nexiwave
Yannick Estève - Laboratoire d'Informatique de l'Université du Maine

Contact

To email the organizers, please send email to sphinx+workshop@cs.cmu.edu

Using HTK models in sphinx4

2009-09-26T22:56:00.008+04:00

As from yesterday long waited cool patch by Christophe Cerisara with the help of super fast Yaniv Kunda has landed in svn trunk. Now you can use HTK model directly from sphinx4. Though it's not easy since I spend a few hours today figuring the required issues, so here is a little step-by-step howto:

1. Update to sphinx4 trunk

2. Download small model, because currently binary loading is not supported unfortunately and it takes a lot of resources to load the model from a huge text file. Get a model from Keith Vertanen

http://www.inference.phy.cam.ac.uk/kv227/htk/htk_wsj_si84_2750_8.zip

3. Convert model to text format with HTK HHEd

mkdir out
touch empty
HHEd -H hmmdefs -H macros -M out empty tiedlist

4. Replace model in Lattice demo in configuration file:

<component name="wsj" type="edu.cmu.sphinx.linguist.acoustic.tiedstate.TiedStateAcousticModel">
<property name="loader" value="wsjLoader"/>
<property name="unitManager" value="unitManager"/>
</component>
<component name="wsjLoader" type="edu.cmu.sphinx.linguist.acoustic.tiedstate.HTKLoader">
<property name="logMath" value="logMath"/>
<property name="modelDefinition" value="/home/shmyrev/sphinx4/wsj/out/hmmdefs"/>
<property name="unitManager" value="unitManager"/>
</component>

Please note here that modelDefinition property points to the location of the newly created hmmdefas file.

5. Replace the frontend configuration to load HTK features from a file. Unfortunately it's impossible to create HTK features with sphinx4 frontend right now, but this will be implemented soon I hope. Some bits are already present like DCT-II transform with frontend.transform.DiscreteCosineTransform2, some are easy to setup like proper filter coefficients, some are missing. So for now we'll recognize MFC file instead.

<component name="epFrontEnd" type="edu.cmu.sphinx.frontend.FrontEnd">
<propertylist name="pipeline">
<item> streamHTKSource </item>
</propertylist>
</component>
<component name="streamHTKSource" type="edu.cmu.sphinx.frontend.util.StreamHTKCepstrum">
<property name="cepstrumLength" value="39"/>
</component>

and let's change the Java file

StreamHTKCepstrum source = (StreamHTKCepstrum) cm.lookup ("streamHTKSource");
InputStream stream = new FileInputStream(new File ("input.mfc"));
source.setInputStream(stream);

6. Now let's extract mfc. Create a config file for HCopy

SOURCEFORMAT = WAV
TARGETKIND = MFCC_D_A_Z_0
TARGETRATE = 100000.0
WINDOWSIZE = 250000.0
USEHAMMING = T
PREEMCOEF = 0.97
NUMCHANS = 26
CEPLIFTER = 22
NUMCEPS = 12
ENORMALISE = T
ZMEANSOURCE = T
USEPOWER = T

and run it

HCopy -C config 10001-90210-01803.wav input.mfc

make sure input.mfc is located in top sphinx4 folder now since this is the place we'll take it.

7. Now everything is ready

ant && java -jar bin/LatticeDemo.jar

Check the result

I heard: once or a zero zero one nine oh to one oh say or oil days or a jury

It's not very precise, but still ok for such a small model and limited language model.

This is still a work in progress and a lot of things still pending. The most important are reading the binary HTK files, frontend adaptation, cleanup and unification. But I really look forward on the results, since it's really a promising approach. There are not so many BSD-licensed HTK decoders out there.

Speech Recognition As Experimental Science

2009-09-16T20:27:00.004+04:00

It's well known there are two types of physics - theoretical one and experimental. During the school I always liked doing the last, measuring the speed of a ball or voltage, plotting the graphics and so on. Unforunately in later days I was mostly doing math or programming. Only recently when I started to spend a lot of time on speech recognition I found why do I like it so much - it's also an experimental science.

When you build a speech recogniton system your time is mostly spent on all these beautiful things. Setting up the database training, running the learning process, tracking the results. You are trying understand the nature and find it's laws, you want to find the best feature set, phoneset, find the beams and more and more. You have an experimental material and sometimes it appeared there are things you forget to take in account. The activity that's really encouraging.

Of course there are important drawbacks, issues like proper design of the experiments arise. Unfortunately it's not widely described in the literature but speech recognition experiments are just an examples of experiments so all issues are valid for them. To list a few:

Reproducability
Connection of the theory and the practice
Estimation of the results and their validity

For example the last point is very important. Currently when we are running the the database test we just get a number. We are trying to rely on it without even estimating the deviation and other very important attributes of every scientific measurement. As the result we make unreliable decisions like I did with MLLT transform. I now think that we should be more careful about that.

So that's why I started with the forementioned wikipedia page trying to find a good book on experiment design and of course it would be nice to find an appropriate software for experiment management workflow.

The First Glance On The Interspeech 2009 Papers

2009-09-10T18:25:00.006+04:00

Interspeech 2009 in Brighton is over today. Unfortunately I wasn't able to particiapte for various reasons. Still, it was very interesting to review the list of sessions, abstracts and read some articles available. The modern activity in speech research is amazing, the number of articles and groups is enormous, in total I counted 459 abstracts with grep. It was enjoying to process them all. Currently I reduced the list to 50% of the original size so still need a few lookups to find something more interesting. A few random thoughts I've got:

Sphinx is mentioned 2 times and HTK only once :), that's a win. Of course many researches use HTK for experiments. So it's more the win in being more open.

A lot of machine learning research. And quite a significant amount of research is dedicated to another target space representation/classifier/cost function adjustments. The first glance didn't show anything interesting here unfortunately. Discriminative training is probably the most recent advance in ASR.

Still enormous amount of the old style phontic research. Is vowel length a feature? How do Zulu people click? Sometimes it's interesting to read though.

Almost all TTS is about HMM for speech synthesis. The quality of audio for TTS is a problem. I've recenly read the good and very detailed good review by Dr. Zen, even adepts of the approach know that the hybrid of HMM and unit-selection is better.

Suprisingly short section on new methods and paradigms unfortunately.

New trends include emotions, machine speech-to-speech translation, language aquisition. Combination of visual and speech recognition is suprisingly common.

No Russians at all. Well, not strange, Russian speech technology doesn't exist in fact.

The RWTH Aachen University Open Source Speech Recognition System is a terrific news. The source is available, downloaded and ready for investigation.

"Improvements to the LIUM French ASR system based on CMU Sphinx: what helps to significantly reduce the word error rate", no link available yet unfortunately. Should be a very interesting reading. The only problem that arises here is that someone should do the merge. The issue is that source is available but really it's very hard to integrate with the research-oriented system.

I'm also waiting for Blizzard 2009 results that should be presented but still not available.

A Self-Labeling Speech Corpus: Collecting Spoken Words with an Online Educational Game - we wanted that for a long time for Voxforge.

In few next posts I'll probably cover some interesting topics in more detail. If you was at the conference or saw something interesting, comments are appreciated.

Modern ASR practices review

2009-09-08T22:45:00.007+04:00

I was never able to completely join the scientific world, most probably because engineering tasks are more attractive. Though I graduated as a mathematician, my merits aren't worth mentioning. For example the thing I never liked is writing, in particular writing a scientific article. That's the corner stone of the science now but for me it seems very dated practice. Most articles are never read, huge percent has errors, many are completely wrong or repeat other sources. Of course there are brilliant ones.

From my point of view the knowledge should be probably organized in a different ways, something like a software projects. The theory could be built during ages in a wiki style with all changes tracked and probably contain complimentary information like techinical notes, software implementations, test results, formalized proofs and so on. Of course among software projects there are also issues like forks, bad maintaince and bugs, but it seems they are more organized.

That's why I really like the projects that keep knowledge in a structure like wikipedia, planetmath for example. Also reviews of the state of art are of course invaluable. Today I spent some time processing my library and the found again the wonderful review by Mark Gales:

The Application of Hidden Markov Models in Speech Recognition

I would really recommend this book as a base introduction into modern speech recognition methods. Though written by HTK author, it has little HTK specific and really focused in best practices in ASR systems.

P.S. Is there a personal library management software, web-based, able to store and index PDF? I used to install Dspace at work, but it's so heavy and the UI is really outdated.

Initial value problem for MLLT

2009-09-06T20:15:00.005+04:00

So far I've recently discovered with the help of mchammer2007 the problem with estimation of the initial matrix for MLLT training. The MLLT or Maximum Likelihood Linear Transform is suggested by R. A. Gopinath, "Maximum Likelihood Modeling with Gaussian Distributions for Classification", in proceedings of ICASSP 1998 and implemented in Sphinxtrain.

The idea is that the matrix to modify feature space is trained to fix the optimization of the covariances and make covariance matrix look more like the diagonal. The optimization is quite simple gradient descendant but unfortunately it suffers from the initial value problem. That is if you choose proper initial value you could get much better results. So right now random matrix is used:

if A == None:
# Initialize it with a random positive-definite matrix of
# the same shape as the covariances
s = self.cov[0].shape
d = -1
while d < 0:
A = eye(s[0]) + 0.1 * random(s)
d = det(A)

And depending on your luck you could get better or worse recognition results. Sometimes even worse than the usual training without LDA/MLL.

SENTENCE ERROR: 55.4% (72/130) WORD ERROR RATE: 17.5% (135/773)
SENTENCE ERROR: 51.5% (66/130) WORD ERROR RATE: 16.6% (128/773)
SENTENCE ERROR: 50.0% (65/130) WORD ERROR RATE: 15.5% (119/773)
SENTENCE ERROR: 56.2% (73/130) WORD ERROR RATE: 16.9% (130/773)
SENTENCE ERROR: 62.3% (80/130) WORD ERROR RATE: 22.3% (172/773)

So the receipt for the training is the following - train several times and control the accuracy, choose the best MLLT matrix and use it in final trainings. If you have a large database, find best MLLT for a subset of it and use it as an initial value for MLLT estimation. No easier way until we'll find a better method for initial value estimation, quick look on the articles didn't give any.

From recent articles I also got quite a significant collection of LDA derivatives, discriminative ones, HLDA and so on. It would be nice to put them into a some review. Also some of them seems to be free from this initial value problem. It would be nice to get a proper review on this large topic.

Between you can see in the chunk of the code above that the comment is not quite correct. The positive-definiteness of the matrix should be checked differently, with the Silvester criterion for example. Though I think that since the condition det(A) > 0 seems to be enough for the feature space transform, the comment should be simply removed. But probably positive-defined matrix is required for optimization.

Adaptation Methods

2009-09-02T03:47:00.003+04:00

It's really hard to collect information on practical application of speech recognition tools. For example the wonderful quote from Andrew Morris on htk-users about what to update during MAP adaptation:

Exactly what it is best to update depends on how much training data you have, but in general it is important to update means and inadvisable to update variances. Only testing on held out test data can decide which is best, but if you are training on data from many speakers and then adapting to data from just one speaker, I expect updating just means should give best results, with variance adaptation reducing performance and transition probs or mix weights adaptation making little difference.

After few experiments I can only confirm this statement. You should never adapt the variances. So, the HOWTO in our wiki is not so good as it could be. Another bit could be taken from this document, actually it's really better to combine MAP and MLLR this way and the best method for offline adaptation is:

Run bw to collect statistics
Estimate mllr transform
Update means with mllr
Run bw again with updated means
Apply MAP adaptation with fixed tau greater than 100 (try to select the best value). Unfortunately from my experience automatic tau selection is broken in map_adapt. This way you'll update the variances a bit, but only slightly.

No book could tell you that!

Time to buy the new video card

2009-08-29T11:17:00.004+04:00

Everybody plays with training and recognition on a GPU now. 200x improvement on NVidia CUDA is worth time and money. Moreover that the sample code is already available.

Thanks to prym on #cmusphinx channel on irc for the link and of course huge thanks Chuan Liu for the article and a new project. It would be so great to see the similar patches for Sphinxtrain/sphinx4, it would be the killer feature.

How to improve accuracy

2009-08-20T01:58:00.007+04:00

People very often ask "how to improve accuracy?". Since I've got three questions like this today I decided to write more or less extensive description of the ways to solve this problem. Probably it will be a bit sketchy, but I hope it will be helpful. Corrections will be very appreciated as well.

1) Well, first of all let me mention that the problem is complex. It requires an understanding of the Hidden Markov Models, beam search, language modelling and every other technology involved. I really recommend you to read the book on speech recognition first. This one is very good:

http://www.amazon.com/Spoken-Language-Processing-Algorithm-Development/dp/0130226165

In theory it will be possible to build a speech recognition system without any extensive knowledge as listed above, but it's not the case now. We are working on easy to use system but we are still in the early beginning.

Probably you don't have time to read and study all this. Then think if you have time to implement speech recognition system at all. At least learn basic concepts.

Please also learn how to do programming before you are starting. I think it's obvious you need to have software development experience. We can't teach you Java really.

2) Next step is to setup the basic example of the system and estimate it's accuracy while first part is quite obvious, second is often ignored. Don't do that. It's critical to test accuracy on realistic conditions during whole development process.

Decide what kind of system are you going to implement - large vocabulary dictation, medium vocabulary names recognition or small vocabulary command and control or some other task. Probably you need IVR system. For each system there is a demo already. Use it as a base. Please don't try to build dictation from a command and control demo, it's just not suitable.

Now the important task, the estimation of the accuracy. The examples how to do that could be found in decoder sources, in tutorial and in many more places. Collect test database, recognize it and compute the exact accuracy estimation.

It should be the following:

Command and control 5% WER (word error rate)
Medium vocabulary 15% WER
Large vocabulary 30% WER
Large vocabulary short utterances 50% WER

if you have noisy audio or some accent multiply this number on 2.

4) Compare the actual accuracy with the expected value. If the accuracy is mostly the expected, proceed to the next step. If not, search for the bug. Most likely you've made a mistake in system setup. Check the configuration if it is suitable for the task, check sound speech quality with sound editing, find out the spectrum range, check the sampling rate, accent, dictionary and other parameters. If your WER is 90%, you made a mistake for sure.

For example of task-dependent training consider acoustic database. If you train a small vocabulary acoustic model you need word-based acoustic model with word-based phoneset in Sphinxtrain case. If you are training large vocabulary database make sure your phoneset is not large and make sure you have selected the proper number of senones/mixtures.

5) Once you've reached a baseline, it will take a lot to improve it. Think if it's enough for you and if you can build your application with such accuracy. It's unlikely you'll get significantly more. But if you are brave enough:

Use MLLT/VTLN feature space adaptation
Use MLLR and other type of online speaker adaptation
Adapt language model, use context-sensetive language models
Tune beams - try different values and experiment with them
Implement the rejection for OOV (out-of-vocabulary) words and other noise sounds
Implement noise cancellation.
Adapt acoustic models and dictionary to your speakers/their accent.
....

The out-of-vocabulary words are the most frequent issue here. Unlike expected by user, most demos don't do any OOV filtering out of box while it's critical for applications. Unfortunately though unlimited vocabulary systems exist, they are quite complex (though also possible to implement). Most systems have limited vocabulary. That means that you need to implement OOV detection and/or confidence scoring in order to filter the garbage. This is doable and described in demos too, for example in confidence demo.

If you are short of ideas here, join the mailing list, we have a lot of features to implement.

Sphinx4-1.0beta3 is released

2009-08-17T23:36:00.002+04:00

The best speech recognition engine is on it's way to world's domination. We are happy to announce the new sphinx4 release. This is still a development version, so bug reports and testing are very appreciated.

Packages

https://sourceforge.net/projects/cmusphinx/files/sphinx4/1.0-beta3/

New Features and Improvements:

BatchAGC frontend component
Complete transition to defaults in annotations
ConcatFeatureExtrator to cooperate with cepwin models
End of stream signals are passed to the decoder for end of stream handling
Timer API improvement
Threading policy is changed to TAS

Bug fixes:

Fixes reading UTF-8 from language model dump
Huge memory optimization of the lattice compression
More stable fronend work with DataStart and DataEnd and optional SpeechStart/SpeechEnd

Thanks:

Yaniv Kunda, Michele Alessandrini, Holger Brandl, Timo Baumann, Evandro Gouvea

Release of the Polish voice for Festival

2009-08-05T02:51:00.003+04:00

Very remarkable and long waited release happened recently. The Polish multisyn voice for the Festival TTS system was made available. This is the best multisyn voice available nowdays both in terms of speech material (several hours, much more than any arctic database, around 500 Mb of audio) and label quality (it has manually corrected segment labels). Also it uses some unique synthesis method modifications like target f0 prediction for multisyn combined with ToBI/APML-based intonation module. The scheme code also has some important modifications. I really encourage you to try this voice even if you don't understand Polish. I also look forward into the HTS voices

Thanks a lot to Krzystof for his hard work.

Training language model with fragments

2009-07-30T03:01:00.007+04:00

Sequitur g2p by M. Bisani and H. Ney. is a cool package for the letter to phone translation, quite accurate and, the most important, open. But actually there are different hidden gems in this package :)

One of them is the phone-oriented segmenter that splits the words on chunks - graphones. Graphone is a joint object consisting of letters and corresponding phones that combine words. Graphones are used in g2p internally, but for example they are very useful in construction of the open vocabulary models. The system as a whole is described here:

Open Vocabulary Spoken Term Detection Using Graphone-Based Hybrid recognition System by M. Acbacak, D. Virgyri and A. Stolke

and the details of the language model in the original article:

Open Vocabulary Speech Recognition with Flat Hybrid Models by Maximilian Bisani and Hermann Ney

The interesting thing is that all required components are already available, the issue is to find correct option and build the system. So the quick reciept is:

1. Get Sequitur G2p
2. Patch it to support Python 2.5 (replace elementtree with xml.etree, since elementtree is deprecated now)
3. Convert cmudict lexicon to xml-based Bliss format (I'm not sure what's it, I failed to find information about it on the web)

import sys
import string
print "<lexicon>"
file = open(sys.argv[1], "r")

for line in file:
toks = line.strip().split()
if len(toks) < 2:
continue
word = toks[0]
phones = string.join(toks[1:]," ")
print "<orth>"
print word
print "</orth>"
print "<pron>"
print phones
print "</pron>"
print "</lexicon>"

4. Train the segmenter model. The most complicated thing is to figure option to train multigram model with several phones. Default one used in g2p consist of 1 phone and 1 letter, it's not suitable for OOV language model.

g2p.py --model model-1 --ramp-up --train cmudict.0.7a.train --devel 5% --write-model model-2 -s 0,2,0,2

5. Ramp up the model to make it more precise
6. Build the language model, here you need the dictionary in XML format. As the article above describes, the original lexicon should be around 10k, the subliminal training lexicon should be 50k or so.

makeOvModel.py --order=4 -l cmudict.xml --subliminal-lexicon=cmudict.xml.test -g model-2 --write-lexicon=res.lexicon --write-tokens=res.tokens

After that you can get a tokens for lm and with additional options even a counts for the language model you could train with SRILM. I haven't finished the previous step yet, so this post should have follow up.

I'm going to ClueCon

2009-07-23T01:33:00.006+04:00

This August I'm going to US again to Chicago to ClueCon where I'll give the talk titled "The use of open source speech recognition". Here is the small outline:

The most complicated thing in modern ASR is to make user expectations agree with the actual capabilities of the technology. Although the technology itself is able to provide a number of potentially very useful features, they are not exactly what average user expects.

Many specialized tasks require a huge amount of customization, for example speaker adaptation needs to be accurately embedded into the accounting system in order to let recognizer improve the accuracy.

The open source solutions could help here because of much greater flexibility they have. But although many companies provide speech recognition services only several projects exist and most of them are purely academic. They often require a lot of tuning for the end-user. Many parts of the complete system are just missing.

Luckily the situation is going to improve during last years, the core components are going to have more or less stable release schedule and active support including a commercial one.

The purpose of this talk is to cover the trends of the development of open source based speech recognition in conjunction with the telephony systems and suggest a ways it can reach enterprise level.

I'll also visit Boston for two days

Update: Here is the presentation

Gran Canaria

2009-07-05T02:17:00.003+04:00

I'm on Gran Canaria!

Left Zyxel

2009-06-30T00:35:00.004+04:00

Recently I've left Zyxel where I worked for three years on a Linux-based home class router (CPE). It was a nice place to where I've met wonderful friends and learned a lot. It was also encouraging and interesting work.

I still have a few ideas for the CPE market that are not completely supported in current developments of various competitors. Things like modern web-2.0 dynamic UI, better error reporting, overall performance optimization, testing, centralized management and so on promise a lot for a vendor that will be able to handle such a suprisingly complicated product like CPE. Cell phones are much more active market comparing to routers, although the class of home devices seems to be no less important then cell phone one. For example I amazed by overall T-Mobile G1 quality and haven't seen anything comparable on router's market. Probalby US market is different though. Recent future with 4G, extensibility and wideband access everywhere promise a lot.

Let's hope one day there will be a team that could implement it. Also I hope to buy a router made by me very soon.

Java bits in Sphinx4

2009-06-12T04:53:00.006+04:00

I spent some time converting sphinx4 to Java5, mostly changing the loops dropping iterators. I hope it will not just make the code cleaner but give us a few bits of performance.

Also tried to profile sphinx4 lattice code. It seems that I broke it by changing the default value for keepAllStates to true. With all HMM states left in a tree it's very hard to traverse the tree to create a lattice. Unfortunately TPTP profiler in Eclipse appeared to be very slow, looking for profiler now as well as on the way to solve this keepAllStates issue.

That change had another drawback, now we doing very unnatural work when we decide if stream is over. Currently scorer returns null on every SpeechEnd signal. It should also return null on DataEnd signal and that null should be different from the first one since we should stop the recognition only after DataEnd in case of long wav file transcription and continue after SpeechEnd. Now we distinguish them by presence of the data frames which are kept due to keepAllTokens settings. Very unnatural dependency which has drawbacks as well at least it eats memory. But I haven't decided what do do with this not so perfect API yet. Most probably we'll need to introduce something like EOF in C to find out if stream is over.

Dither is considered harmful

2009-05-31T23:49:00.000+04:00

MFCC features used in speech recogition are still a reasonable choice if you want to recognize generic speech. With tunings like frequency wrap for VTLN and MLLT they still can suggest the reasonable performance. Although there are many parameters to tune like upper and lower frequencies, the shape of the mel filters and so on, default values mostly works fine. Still I had to spend this week on one issue related to zero energy frames.

Zero energy frames are quite common in telephony recorded speech. Due to noise cancellation or due to VAD speech compression telephony recordings are full of the frames with zero energy. The issue is that calculation of the MFCC features consist of taking log from eneries, thus you have an undefined value of log 0. There are several ways to overcome this issue.

The one used in HTK or SPTK for example is to assign some floored value to the log, usually it's quite a big value in log domain, say 1e-5. This solution is actually quite bad at least in it's sphinx implementation. That's because it largely affects CMN computation, means goes down and bad things happen. Silent frame can affect the result of the whole phrase.

Another one is dither, when you apply random 1bit noise to the sound as a whole and use this modified waveform for training. Such change is usually enough to make log take acceptable values around -1.

There were complains about dither, most well known one is that it affects recognition scores, results can be different from run to run. It's a bad thing but not that bad when you start with predefined seed. So I thought before that dither is fine. And by default it's applied both in training and decoder. But recently when I started with the testing of the sphinxtrain tutorial I come to more important issue.

See the results on an4 database from run to run without any modifications:

TOTAL Words: 773 Correct: 645 Errors: 139
TOTAL Percent correct = 83.44% Error = 17.98% Accuracy = 82.02%
TOTAL Insertions: 11 Deletions: 17 Substitutions: 111
TOTAL Words: 773 Correct: 633 Errors: 149
TOTAL Percent correct = 81.89% Error = 19.28% Accuracy = 80.72%
TOTAL Insertions: 9 Deletions: 23 Substitutions: 117
TOTAL Words: 773 Correct: 639 Errors: 142
TOTAL Percent correct = 82.66% Error = 18.37% Accuracy = 81.63%
TOTAL Insertions: 8 Deletions: 19 Substitutions: 115
TOTAL Words: 773 Correct: 650 Errors: 133
TOTAL Percent correct = 84.09% Error = 17.21% Accuracy = 82.79%
TOTAL Insertions: 10 Deletions: 17 Substitutions: 106
TOTAL Words: 773 Correct: 639 Errors: 142
TOTAL Percent correct = 82.66% Error = 18.37% Accuracy = 81.63%
TOTAL Insertions: 8 Deletions: 19 Substitutions: 115

If you are lucky you can even get WER of 15.95%. Thats certainly unacceptable and it still remains true why training is so sensible to dither applied. Clearly it makes any testing impossible. I checked this results on medium vocabulary 50-hours database and they are still the same - the accuracy is very different from run to run. Interesting thing is only training is affected that much. For testing you can get very slight difference of 0.1%.

So far my solutions are:

Disable dither on training
Apply a patch to drop frames with zero energy (this seems useless but it helps to be less nervious about warnings)
Decode with dither

I hope I'll be able to provide more information in the future about the reasons of this unstability, but for now it's all I know.

Text summarization low hanging fruit

2009-05-24T16:01:00.000+04:00

Actually all the data required for quite precise text summarization is almost in place, one should just add support for WordNet from nltk into the Open Text Summarizer, calculate frequencies and present highlighted sentences to the user. Or it's possible to do the same in python with nltk iteself.

It would help in many cases for example in mail processing. Getting 200 mails in a day it's really hard to read them through. Or probably it's just time to unsubscribe from some mailing lists.

My vote for voxforge

2009-05-23T14:00:00.000+04:00

I just voted for VoxForge in the category: "Most Likely to Change the Way You Do Everything". You might want to do the same :)

Go to http://sourceforge.net/community/cca09/nominate/?project_name=VoxForge&project_url=http://voxforge.org/

Blizzard Challenge 2009

2009-05-19T02:03:00.000+04:00

CSTR and others are pleased to announce that the listening tests for the Blizzard Challenge 2009 are now running. The Blizzard Challenge is an annual open speech synthesis evaluation in which participants build voices using common data, and a large listening test is used to compare them. Participants include some of the leading commercial and academic research groups in the field.

I would appreciate your help in getting as many listeners to participate as possible, by forwarding this message on to other lists, colleagues, students, and of course taking part yourself.

The listening test should take 30-60 minutes to complete, and can be done in stages if you wish. You do not need to be a native speaker of the language in order to take part. There are 4 different start pages for the listening test, as follows:

English

Volunteers:
http://groups.inf.ed.ac.uk/blizzard/blizzard2009/english/register-ER.html

Speech Experts:
http://groups.inf.ed.ac.uk/blizzard/blizzard2009/english/register-ES.html

Mandarin Chinese:

Volunteers:
http://groups.inf.ed.ac.uk/blizzard/blizzard2009/mandarin/register-MR.html

Speech Experts:
http://groups.inf.ed.ac.uk/blizzard/blizzard2009/mandarin/register-MS.html

Whether you consider yourself a 'speech expert' is left to your own
judgement.

Training the large database trick

2009-05-17T02:22:00.000+04:00

Training of the large database requires a cluster. SphinxTrain supports training on Torque:PBS for example, to do this you need to set the following configuration variables:

$CFG_QUEUE_TYPE = "Queue::PBS";

and set the number of parts to train. The issue is to guess the number of parts. I previously thought

1 part:

TOTAL Words: 773 Correct: 660 Errors: 126
TOTAL Percent correct = 85.38% Error = 16.30% Accuracy = 83.70%
TOTAL Insertions: 13 Deletions: 9 Substitutions: 104

3 parts:

TOTAL Words: 773 Correct: 583 Errors: 262
TOTAL Percent correct = 75.42% Error = 33.89% Accuracy = 66.11%
TOTAL Insertions: 72 Deletions: 17 Substitutions: 173

10 parts:

TOTAL Words: 773 Correct: 633 Errors: 168
TOTAL Percent correct = 81.89% Error = 21.73% Accuracy = 78.27%
TOTAL Insertions: 28 Deletions: 10 Substitutions: 130

20 parts:

TOTAL Words: 773 Correct: 619 Errors: 181
TOTAL Percent correct = 80.08% Error = 23.42% Accuracy = 76.58%
TOTAL Insertions: 27 Deletions: 13 Substitutions: 141

But it appeared that all above is not true. One potential source of problems was that the norm.pl scripts grabs all the sub directories under the bwaccum one indiscriminately. So if there are some old bwaccum dirs left over (e.g. if you train on 20 parts first then start again with 10, without deleting the directories in-between), the norm script will screw up (thanks to David Huggins-Daines for pointing that out to me). In this particular test there was another one that I forgot to update mdef after model rebuild and old scripts didn't do that automatically. On multipart the order of senones in mdef is different thats why there was a regression. Though the set of senones is the same.

So the testing and statements above are completely wrong - accuracy doesn't depend on number of parts used. As expected. This confirms the ground truth that correct experiment statement is the most important thing in research.

Now only one issue left - the dropped accuracy from old tutorial to a new one. But that is a completely different issue discussed in my mails on cmusphinx-sdmeet now.

Bad prompts issue

2009-05-11T00:13:00.000+04:00

After quite a lot of training of the model on a small part of database to test things I came to conclusion that the main issue is a bad prompts. Indeed the accuracy on the training set for 4 hours of data with the language model trained on the same training prompts is only 85%. Usually it should be around 93%. The issue here is that real testing prompts are also bad and they should stay that way, otherwise we'll be bounded to high quality speech only. I remember I tried a forced alignment with communicator model before but it didn't improve much just because of the testing set issue. Another try was to use skip state, that was not fruitful as well.

So the plan for now is to choose the subset with the forced alignment again and train the model to check if the hypothesis is true and bad prompts in an acoustic database is indeed a main issue. It looks like we are walking around by the circle.

I ended reading the article titled "Lightly supervised model training"

Speech AdBlock

2009-05-03T02:15:00.001+04:00

Inspired by Daniel's Holth application to remove word "twitter" from podcasts:

http://dingoskidneys.com/cgi-bin/hgwebdir.cgi/twitterkiller/

I think it's a very good idea to implement keyword filter to block advertizing in podcasts. Though support for keyword spotting is not easily implemented with CMUsphinx right now, it should be rather straightforward thing to do. In the end it can be just a binary application that takes a list of keywords to block and just filters mp3 file giving user the same file with blocked advertising.

Cepwin Features Training

2009-05-01T23:17:00.000+04:00

Recently the option to bypass delta and delta-delta feature extraction process and directly apply LDA transform matrix to the cepstrum coefficients of sequential frames was added to sphinxtrain. To use it you need to adjust training config and decoder as well:

Set feature type to 1s_c

Add $CFG_FEAT_WINDOW=3; to the config file

Train with MLLT

Apply the attached patch to sphinxbase cepwin.diff.

Decode

You can use these models in sphinx4 now, the following config should do the work:

<component name="featureExtraction" type="edu.cmu.sphinx.frontend.feature.ConcatFeatureExtractor">
<property name="windowSize" value="3"/>
</component>
<component name="lda" type="edu.cmu.sphinx.frontend.feature.LDA">
<property name="loader" value="sphinx3Loader"/>
</component>

I haven't found the optimal parametrers yet, but it seems that something like cepwin=3 and final dimension around 40 should work. I hope to get results on this soon.

Looking back on Free Software

2009-04-26T23:46:00.001+04:00

I've read some books on business recently:

They sometimes repeat each other but actually have few interesting moments. At least I started to look on all this from a bit different point of view. Unfortunatly this domain is covered differently by Free Software community people who tend to be idealistic but promote their point of view actively. The words like "community" or "leadership" or "cool people" don't bring much in the end, and the most interesting thing is that such words are mosly spoken by corporate people.

Anyhow, it would be nice to have a project that will have a clear mission and a set of reachable goals, like product plans each one with a design both technical and non-technical documents. It would be nice to have a test set with 90% coverage and a build without warnings and also a tracking system for user requests. Thing like slick UI are also important. After all, it's easier to get this than to build an LVCSR with 95% accuracy I think :)

Frama-C Eclipse plugin

2009-04-15T01:52:00.001+04:00

I decided to finally go forward and publish my modifications of frama-c Eclipse plugin I'm doing at work. Moreover I decided to try git/github. Let's see how it goes. The project is here:

http://github.com/frama-c-eclipse/frama-c-eclipse/tree/master

the future plans include

better graphics
more cleanup
offshelf support for recent Frama-C versions

Quest in configuration file

2009-04-09T04:40:00.000+04:00

After almost a year of wondering I finally discovered what does this mean in sphinx4 config files:

<component name="activeListManager"
type="edu.cmu.sphinx.decoder.search.SimpleActiveListManager">
<propertylist name="activeListFactories">
<item>standardActiveListFactory</item>
<item>wordActiveListFactory</item>
<item>wordActiveListFactory</item>
<item>standardActiveListFactory</item>
<item>standardActiveListFactory</item>
<item>standardActiveListFactory</item>
</propertylist>
</component>

Actually it's even described in docs:

The SimpleActiveListManager is of class edu.cmu.sphinx.decoder.search.SimpleActiveListManager. Since te word-pruning search manager performs pruning on different search state types separately, we need a different active list for each state type. Therefore, you see different active list factories being listed in the SimpleActiveListManager, one for each type. So how do we know which active list factory is for which state type? It depends on the 'search order' as returned by the search graph (which in this case is generated by the LexTreeLinguist). The search state order and active list factory used here are:

State Type	ActiveListFactory
LexTreeNonEmittingHMMState	standardActiveListFactory
LexTreeWordState	wordActiveListFactory
LexTreeEndWordState	wordActiveListFactory
LexTreeEndUnitState	standardActiveListFactory
LexTreeUnitState	standardActiveListFactory
LexTreeHMMState	standardActiveListFactory

There are two types of active list factories used here, the standard and the word. If you look at the 'frequently tuned properties' above, you will find that the word active list has a much smaller beam size than the standard active list. The beam size for the word active list is set by 'absoluteWordBeamWidth' and 'relativeWordBeamWidth', while the beam size for the standard active list is set by 'absoluteBeamWidth' and 'relativeBeamWidth'. The SimpleActiveListManager allows us to control the beam size of different types of states.

It's hard to guess, isn't it? Well, I hope soon we'll be able to make configuration easier. The idea of annotatated configuration came to my mind today. With the older idea of using task-oriented predefined configurations it could really save a lot of efforts.

2009-04-07T01:21:00.001+04:00

Today with the help of my chief I've found FindBugs, a nice static analyzer that tries to find the issues in Java code and reports about them. It's a very useful tool since I've already fixed a few bad things in sphinx4 and in other projects. The number of false positives is acceptable. The similar tool for C for example is splint, though java tools as usual are much more useful. And there is an Eclipse plugin that helps to apply the tool with a single mouse click.

This makes me think about what can be counted as a development platform. Although it's well known that scripting languages like Python speedup the development, they totally lack the tools like static analyzers, debuggers, profilers, documentation and testing frameworks and so on and so forth. There is some effort to create a common framework to quickly build development tools along with DSL language, but the result is not so advanced I suppose. Basically it seems today there is no choise which language to use for the development and in the light of this it seems very strange that GNOME development goes in completely opposite direction stepping to the domain of JavaScript and naive programming. I hope the desktop will not become a collection of bugs after that.

Sphinx4 migrated to git

2009-03-28T00:26:00.001+03:00

This change started some time ago, but now it's mostly finished and announced. The tree could be found here:

https://sourceforge.net/scm/?type=git&group_id=257562

The discussion is here.

I'm glad to see the progress happens, big thanks to everyone involved - Joe, Piter and others.

About git itself I have a mixed feeling. The advantages of DVCS aren't obvious for me and in the past I even gave up my participation in one of the projects after it's migration to mercurial (it was http://linuxtv.org). Distributed nature increases complexity and confuses at least me. It's hard to understand where the latest changes are done, what is the real state of thing and where change happens. Developers tend to add their changes to their own branches and little effort is made to create a common branch. Also among all DVCSs git is the worst in terms of usability. Sadly GNOME also migrates to git in near future.

Every change has it black and white sides. Many things I do like in a new sphinx4 - clear split of the tests one can run. Some things are hard to understand like Rakefile migration. I'm afraid of windows users, how will they build sphinx4 now? Anyhow, let's hope issues will be resolved and the new shiny release will appear very soon.

Russian GNOME 2.26

2009-03-26T04:04:00.000+03:00

Russian GNOME 2.26 is 100% translated. Congratulations to the team for their hard work.

GNOME Summer of code tasks

2009-03-22T08:16:00.001+03:00

I spent some time today trying to invent some interesting tasks for GNOME summer of code 2009. My favorite list for now is:

Text summarizer in Epiphany
Improved spell check for GEdit
Doxygen support for gtk-doc
Desktop-wide services for activity registration
Automatic workstation mode detection and more AI tasks desktop can benefit from
Cleanup of the Evolution interface where sent and received mail are grouped together

The list is probably too boring, but one should note that usually summer is to small to implement something serious and students are not that experienced as one want to see them. Some of the tasks were rejected already, though it's not a big deal. I just find discouraging that the list of the tasks proposed officially is even more tedious.

The overview of this issue makes me think again about GNOME as a product on the market and the possible ways of it's development. It seems that we are now at a point when feature set among competitors are stabilized and it's hard to invent something else in a market. So-called mature product stage where it's important to polish and lower costs. The big step is required to shift product on a new level. Probably I need to investigate the research desktops that completely change the way users works with the system. For example I'd love to see better AI support everywhere like adaptive preferences, better stability and security with proper IPC and service-based architecture, the self-awareness services, the modern programming language. I'm not sure I'm brave enough for that though.

HTK 3.4.1 is released

2009-03-18T00:40:00.001+03:00

Amazing news really. The new features of the release include:

1. The HTK Book has been extended to include tutorial sections on HDecode and discriminative training. An initial description of the theory and options for discriminative training has also been added.
2. HDecode has been extended to support decoding with trigram language models.
3. Lattice generation with HDecode has been improved to yield a greater lattice density.
4. HVite now supports model-marking of lattices.
5. Issues with HERest using single-pass retraining with HLDA and other input transforms have been resolved.
6. Many other smaller changes and bug fixes have been integrated.

The release is available on the HTK website