wav2letter streaming_convnets model. the WER does not decrease during training (it remains stable at 100% for dev and train)

I tried to use the streaming_convnets from recipes: i used two different lexicon and token(character based model vs wordpiece based model):

I used for that a small dataset for training (train-clean-100)

1) lexicon and token from seq2seq_dts mode (word-piece based model):

a-Lexion: librispeech-train+dev-unigram-10000-nbest10.lexicon

a _a a _ a a'azam _a ' az am a'azam _a ' a z am a'azam _a ' a za m a'azam _a ' az a m a'azam _ a ' az am a'azam _a ' a z a m ...etc

b-Token: librispeech-train-all-unigram-10000.tokens

_the s _and ed _of _to _a _in _i _he _that ly _was ..etc

c-cfg file :

--train=lists/train-clean-100.lst --valid=dev-clean:lists/dev-clean.lst --lexicon=am/librispeech-train+dev-unigram-10000-nbest10.lexicon --arch=am_500ms_future_context.arch --tokens=am/librispeech-train-all-unigram-10000.tokens

--criterion=ctc --batchsize=8

--lr=0.4 #--lrcrit=0.05 --momentum=0.0 --maxgradnorm=0.5 --reportiters=1000

--nthread=6 --mfsc=true --usewordpiece=true --wordseparator=_

--filterbanks=80 --minisz=200 --mintsz=2 --maxisz=33000 --enable_distributed=true --pcttraineval=1 --minloglevel=0

--logtostderr --onorm=target --sqnorm --localnrmlleftctx=300 --lr_decay=10000

d- result

2) lexicon and token from tutorial (Letter based model"free-lexicon")

a-Lexion: lexicon.txt

b-Token: librispeech-train-all-unigram-10000.tokens

| ' a b c d e f g h i j k l m n o p q ...etc

c-cfg file :

--train=lists/train-clean-100.lst --valid=dev-clean:lists/dev-clean.lst --lexicon=am/conf_C01/lexicon.txt --arch=librispeech/am_500ms_future_context.arch --tokens=tokens.txt --surround=|

--criterion=ctc --batchsize=8

--lr=0.4 #--lrcrit=0.05 --momentum=0.0 --maxgradnorm=0.5 --reportiters=1000

--nthread=6 --mfsc=true #--usewordpiece=true #--wordseparator=_

--filterbanks=80 --minisz=200 --mintsz=2 --maxisz=33000 --enable_distributed=true --pcttraineval=1 --minloglevel=0

--logtostderr --onorm=target --sqnorm --localnrmlleftctx=300 --lr_decay=10000

d- result

comments:

the WER and TER do not change during training in both sets. which parameters should be change to get a better WER.

Oct 01 '20 10:10 kerolos

Your loss is going down, just very slowly. Try to increase learning rate, say try 1.0 and 2.0

Oct 04 '20 04:10 tlikhomanenko

I changed the learning rate to 1.0 and 2.0 as you said for both models (character based model vs word-piece based model).

Here are the results for character based model

b) free lexicon (character based model) with lr=2.0:

as shown by the results, the WER is stuck into 100% and the loss is decreasing slowly.

I have anther question regarding the LR, Is it possible to use a LR larger than 1.0 ? , because it should be in the range between 0.0 and 1.0

I also tried to change the momentum to 0.8, but the loss become Nan. https://github.com/facebookresearch/wav2letter/issues/719#issuecomment-682351751 I am wondering why the configuration file and model from recipes do not give a good result as it shown in the paper.

thanks in advance and i am waiting for your reply.

Oct 14 '20 14:10 kerolos

I have the same issue in the Steaming Convnet model as in this comment. https://github.com/facebookresearch/wav2letter/issues/555#issuecomment-708277031

Oct 15 '20 14:10 kerolos

First, there is no any restrictions that lr should be [0, 1]. Second with lr = 1 and lr = 2 in your log loss actually doesn't decrease, which means that these values of lr are too high.

Why in you latest log it is different number of updates in epochs? Can you point what is the total batch size?

Recipes are provided for the full Librispeech, which can be different for the 100h or other sets (you need to tune optimization itself, probably also dropout of the model). Also we trained with specific total batch size which can also affects the optimization procedure.

cc @vineelpratap on optimization for streaming convnet arch.

Oct 16 '20 05:10 tlikhomanenko

thanks for your reply. In the second log "free lexicon (character based model) with lr=2.0", i set the reportiters = 1000 to fast visualization, and quick a look for the loss and WER to decide if i am on the right track to minimize the loss or not. i am using two GPUs (RTX 2080 TI) for training with a batch size=8 (Librispeech: train-clean-100).

In the first log "free lexicon (character based model) with lr=1.0", i used the whole dataset (Librispeech: train 960hr) with two GPUs (RTX 2080 TI) and BS=8.

Oct 20 '20 12:10 kerolos

I've had the same issue with the streaming_convnets word-piece model applied to LibriSpeech 100h, which I've managed to solve by tweaking the SpecAugment parameters in the architecture file.

First, I've tried removing SpecAugment by deleting the "SAUG 80 27 2 100 1.0 2" line in "am_500ms_future_context.arch". With this, the loss has started decreasing and the dev-clean WER has gone down to 26.35% by epoch 25.

To continue with, I've enabled back SpecAugment but using a milder configuration with less and smaller time and frequency masks: "SAUG 20 15 1 35 0.2 1". With such setting, the dev-clean WER has been reduced to 25.02% by epoch 25.

My guess is that milder versions of SpecAugment could help achieving a faster convergence for LibriSpeech 100h. The original version in the recipe model, known as LibriSpeech double (LD), was originally optimized for LibriSpeech 960h, and might be too much for LibriSpeech 100h.

Anyway, other optimizations could be possible, but I hope this one helps you to start tweaking around! I'm attaching a plot of the WER during the first 25 epochs for three SpecAugment configurations: the original one (LD), the milder version (LibriSpeech Mild = LM) and no SpecAugment.

libri_saug_streaming_convnets

Dec 17 '20 20:12 gcambara