/Users/npradhan/miniconda2/envs/pytorch/bin/python /Users/npradhan/workspace/pyro_dev/pyro/examples/dmm/dmm.py
Namespace(annealing_epochs=1000, beta1=0.96, beta2=0.999, checkpoint_freq=0, clip_norm=20.0, cuda=False, iaf_dim=100, learning_rate=0.0004, load_model='', load_opt='', log='dmm.log', lr_decay=0.99996, mini_batch_size=20, minimum_annealing_factor=0.1, num_epochs=5000, num_iafs=0, rnn_dropout_rate=0.1, save_model='', save_opt='', weight_decay=0.6)
N_train_data: 229 avg. training seq. length: 60.29 N_mini_batches: 12
[training epoch 0000] 61.6189 (dt = 16.419 sec)
/Users/npradhan/workspace/pyro_dev/pyro/pyro/poutine/trace.py:13: UserWarning: Encountered NAN log_pdf at site 'z_70'
warnings.warn("Encountered NAN log_pdf at site '{}'".format(name))
/Users/npradhan/workspace/pyro_dev/pyro/pyro/poutine/trace.py:13: UserWarning: Encountered NAN log_pdf at site 'z_71'
warnings.warn("Encountered NAN log_pdf at site '{}'".format(name))
/Users/npradhan/workspace/pyro_dev/pyro/pyro/poutine/trace.py:13: UserWarning: Encountered NAN log_pdf at site 'z_73'
warnings.warn("Encountered NAN log_pdf at site '{}'".format(name))
/Users/npradhan/workspace/pyro_dev/pyro/pyro/poutine/trace.py:13: UserWarning: Encountered NAN log_pdf at site 'z_74'
warnings.warn("Encountered NAN log_pdf at site '{}'".format(name))
/Users/npradhan/workspace/pyro_dev/pyro/pyro/poutine/trace.py:13: UserWarning: Encountered NAN log_pdf at site 'z_75'
While refactoring to move to PyTorch distributions, I noticed that the
dmm.pyexample has been throwing warnings after one epoch when it starts encountering NANs at multiple sites.This is on the current
devbranch with Pytorchmaster. @martinjankowiak - Could you help take a look at what's happening?Details