Logger emits exception when there's None in hparams

[kyoungrok0517]

To Reproduce

My hparams:

{
	'n': [8000],
	'k': [30],
	'batch_size': 512,
	'data_dir': '/Users/kyoungrok/Resilio Sync/Dataset/2019 TREC/passage_ranking/dataset',
	'max_nb_epochs': 500,
	'learning_rate': 0.0001,
	'nodes': 1,
	'distributed_backend': None,
	'eval_test_set': False,
	'check_val_every_n_epoch': 1,
	'accumulate_grad_batches': 1,
	'max_epochs': 200,
	'min_epochs': 2,
	'train_percent_check': 1.0,
	'val_percent_check': 1.0,
	'test_percent_check': 1.0,
	'val_check_interval': 0.95,
	'log_save_interval': 100,
	'row_log_interval': 100,
	'enable_early_stop': True,
	'early_stop_metric': 'val_acc',
	'early_stop_mode': 'min',
	'early_stop_patience': 3,
	'gradient_clip_val': -1,
	'track_grad_norm': -1,
	'model_save_path': '/Users/kyoungrok/Desktop/trec-2019-deep-learning/trec2019/sparse/sparsenet/model_weights',
	'model_save_monitor_value': 'val_acc',
	'model_save_monitor_mode': 'max',
	'model_load_weights_path': None,
	'tt_name': 'pt_test',
	'tt_description': 'pytorch lightning test',
	'tt_save_path': '/Users/kyoungrok/Desktop/trec-2019-deep-learning/trec2019/sparse/sparsenet/test_tube_logs',
	'single_run': False,
	'nb_hopt_trials': 1,
	'log_stdout': False,
	'gpus': None,
	'single_run_gpu': False,
	'default_tensor_type': 'torch.cuda.FloatTensor',
	'use_amp': False,
	'check_grad_nans': False,
	'amp_level': 'O2',
	'on_cluster': False,
	'fast_dev_run': True,
	'enable_tqdm': False,
	'overfit': -1,
	'interactive': False,
	'debug': False,
	'local': False,
	'lr_scheduler_milestones': None,
	'k_inference_factor': 1.5,
	'weight_sparsity': [0.3],
	'boost_strength': 1.5,
	'boost_strength_factor': 0.85,
	'dropout': 0.0,
	'use_batch_norm': True,
	'normalize_weights': False,
	'hpc_exp_number': None
}

I see the following errors because I have None value in my hparams dict

Traceback (most recent call last):
  File "sparsenet_trainer.py", line 73, in <module>
    main(hparam_trial)
  File "sparsenet_trainer.py", line 47, in main
    trainer.fit(model)
  File "/Users/kyoungrok/anaconda3/envs/trec/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 707, in fit
    self.run_pretrain_routine(model)
  File "/Users/kyoungrok/anaconda3/envs/trec/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 757, in run_pretrain_routine
    self.logger.log_hyperparams(ref_model.hparams)
  File "/Users/kyoungrok/anaconda3/envs/trec/lib/python3.7/site-packages/pytorch_lightning/logging/base.py", line 14, in wrapped_fn
    fn(self, *args, **kwargs)
  File "/Users/kyoungrok/anaconda3/envs/trec/lib/python3.7/site-packages/pytorch_lightning/logging/tensorboard.py", line 88, in log_hyperparams
    self.experiment.add_hparams(hparam_dict=params, metric_dict={})
  File "/Users/kyoungrok/anaconda3/envs/trec/lib/python3.7/site-packages/torch/utils/tensorboard/writer.py", line 300, in add_hparams
    exp, ssi, sei = hparams(hparam_dict, metric_dict)
  File "/Users/kyoungrok/anaconda3/envs/trec/lib/python3.7/site-packages/torch/utils/tensorboard/summary.py", line 156, in hparams
    raise ValueError('value should be one of int, float, str, bool, or torch.Tensor')
ValueError: value should be one of int, float, str, bool, or torch.Tensor

Environment

PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: None

OS: Mac OSX 10.15.3
GCC version: Could not collect
CMake version: version 3.16.1

Python version: 3.7
Is CUDA available: No
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA

Versions of relevant libraries:
[pip] numpy==1.18.1
[pip] pytorch-lightning==0.6.0
[pip] torch==1.4.0
[pip] torchvision==0.5.0
[conda] blas 1.0 mkl
[conda] mkl 2019.4 233
[conda] mkl-service 2.3.0 py37hfbe908c_0
[conda] mkl_fft 1.0.15 py37h5e564d8_0
[conda] mkl_random 1.1.0 py37ha771720_0
[conda] pytorch 1.4.0 py3.7_0 pytorch
[conda] pytorch-lightning 0.6.0 pypi_0 pypi
[conda] torchvision 0.5.0 py37_cpu pytorch

[Borda]

Hello, thanks for letting us know, could you also share with us the model/trainer so we can replicate your problem... :]

[kyoungrok0517]

Hi, thanks for the response. Here's my code and data. The error can be reproduced under pytorch-lightning >= 0.5.3.3

https://www.dropbox.com/sh/5dyq5dp5l8zfc3r/AAAiK-IoihpgdnJ3L8QzKAxNa?dl=0

• pip install -e . && pip install -r requirements.txt
• (at the root) python ./code/trec2019/sparse/sparsenet/sparsenet_trainer.py --data_dir ./data

[Borda]

@kyoungrok0517 and what would be your expected behaviour, just ignore all parameters with none value?

[kyoungrok0517]

I think that will be the best option. I expect that happens automatically inside the lightning library.

…

________________________________ 보낸 사람: Jirka Borovec <[email protected]> 보낸 날짜: Wednesday, March 4, 2020 3:40:59 AM 받는 사람: PyTorchLightning/pytorch-lightning <[email protected]> 참조: Kyoungrok Jang <[email protected]>; Mention <[email protected]> 제목: Re: [PyTorchLightning/pytorch-lightning] Logger emits exception when there's `None` in hparams (#984) @kyoungrok0517<https://github.com/kyoungrok0517> and what would be your expected behaviour, just ignore all parameters with none value? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#984>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAIAZ7D2RH2S3JX523IFRULRFVFLXANCNFSM4K6RL3LQ>.

[awaelchli]

This is fixed on master. See #1130. @Borda it can be closed.