Scripts

This section contains a collection of useful scripts for quality control during the training of models.

ivadomed_visualize_transforms

run_visualization(input, config, number, output, roi)[source]

Utility function to visualize Data Augmentation transformations.

Data augmentation is a key part of the Deep Learning training scheme. This script aims at facilitating the fine-tuning of data augmentation parameters. To do so, this script provides a step-by-step visualization of the transformations that are applied on data.

This function applies a series of transformations (defined in a configuration file -c) to -n 2D slices randomly extracted from an input image (-i), and save as png the resulting sample after each transform.

For example:

ivadomed_visualize_transforms -i t2s.nii.gz -n 1 -c config.json -r t2s_seg.nii.gz

Provides a visualization of a series of three transformation on a randomly selected slice:

https://raw.githubusercontent.com/ivadomed/doc-figures/main/scripts/transforms_im.png

And on a binary mask:

ivadomed_visualize_transforms -i t2s_gmseg.nii.gz -n 1 -c config.json -r t2s_seg.nii.gz

Gives:

https://raw.githubusercontent.com/ivadomed/doc-figures/main/scripts/transforms_gt.png
Parameters:
  • input (string) – Image filename. Flag: --input, -i
  • config (string) – Configuration file filename. Flag: --config, -c
  • number (int) – Number of slices randomly extracted. Flag: --number, -n
  • output (string) – Folder path where the results are saved. Flag: --ofolder, -o
  • roi (string) – Filename of the region of interest. Only needed if ROICrop is part of the transformations. Flag: --roi, -r

ivadomed_convert_to_onnx

convert_pytorch_to_onnx(model, dimension, n_channels, gpu_id=0)[source]

Convert PyTorch model to ONNX.

The integration of Deep Learning models into the clinical routine requires cpu optimized models. To export the PyTorch models to ONNX format and to run the inference using ONNX Runtime is a time and memory efficient way to answer this need.

This function converts a model from PyTorch to ONNX format, with information of whether it is a 2D or 3D model (-d).

Parameters:
  • model (string) – Model filename. Flag: --model, -m.
  • dimension (int) – Indicates whether the model is 2D or 3D. Choice between 2 or 3. Flag: --dimension, -d
  • gpu_id (string) – GPU ID, if available. Flag: --gpu_id, -g

ivadomed_automate_training

automate_training(file_config, file_config_hyper, fixed_split, all_combin, path_data=None, n_iterations=1, run_test=False, all_logs=False, thr_increment=None, multi_params=False, output_dir=None)[source]

Automate multiple training processes on multiple GPUs.

Hyperparameter optimization of models is tedious and time-consuming. This function automatizes this optimization across multiple GPUs. It runs trainings, on the same training and validation datasets, by combining a given set of parameters and set of values for each of these parameters. Results are collected for each combination and reported into a dataframe to allow their comparison. The script efficiently allocates each training to one of the available GPUs.

Usage Example:

ivadomed_automate_training -c config.json -p config_hyper.json -n n_iterations
Example of dataframe
  path_output training_parameters best_training_dice best_training_loss best_validation_dice best_validation_loss test_dice
0 testing_script-batch_size=2 {‘batch_size’: 2, ‘loss’: {‘name’: ‘DiceLoss’}, ‘training_time’: {‘num_epochs’: 1, ‘early_stopping_patience’: 50, ‘early_stopping_epsilon’: 0.001}, ‘scheduler’: {‘initial_lr’: 0.001, ‘lr_scheduler’: {‘name’: ‘CosineAnnealingLR’, ‘base_lr’: 1e-05, ‘max_lr’: 0.01}}, ‘balance_samples’: False, ‘mixup_alpha’: None, ‘transfer_learning’: {‘retrain_model’: None, ‘retrain_fraction’: 1.0}} -0.002152157641830854 -0.002152157641830854 -0.0013434450065687997 -0.0013434450065687997 0.011467444120505346
1 testing_script-batch_size=4 {‘batch_size’: 4, ‘loss’: {‘name’: ‘DiceLoss’}, ‘training_time’: {‘num_epochs’: 1, ‘early_stopping_patience’: 50, ‘early_stopping_epsilon’: 0.001}, ‘scheduler’: {‘initial_lr’: 0.001, ‘lr_scheduler’: {‘name’: ‘CosineAnnealingLR’, ‘base_lr’: 1e-05, ‘max_lr’: 0.01}}, ‘balance_samples’: False, ‘mixup_alpha’: None, ‘transfer_learning’: {‘retrain_model’: None, ‘retrain_fraction’: 1.0}} -0.0017151038045994937 -0.0017151038045994937 -0.000813339815067593 -0.000813339815067593 0.030501089324618737

Config File:

The config file is the standard config file used in ivadomed functions. We use this as the basis. We call a key of this config file a category. In the example below, we would say that training_parameters, default_model, and path_output are categories.

{
    "training_parameters": {
        "batch_size": 18,
        "loss": {"name": "DiceLoss"}
    },
    "default_model":     {
        "name": "Unet",
        "dropout_rate": 0.3,
        "depth": 3
    },
    "model_name": "seg_tumor_t2",
    "path_output": "./tmp/"
}

Hyperparameter Config File:

The hyperparameter config file should have the same layout as the config file. To select a hyperparameter you would like to vary, just list the different options under the appropriate key, which we call the base_key. In the example below, we want to vary the loss, depth, and model_name; these are our 3 base_keys. As you can see, we have listed our different options for these keys. For depth, we have listed 2, 3, and 4 as our different options. How we implement this depends on 3 settings: all_combin, multi_param, or the default.

{
  "training_parameters": {
    "loss": [
      {"name": "DiceLoss"},
      {"name": "FocalLoss", "gamma": 0.2, "alpha" : 0.5}
    ],
  },
  "default_model": {"depth": [2, 3, 4]},
  "model_name": ["seg_sc_t2star", "find_disc_t1"]
}

Default:

The default option is to change only one parameter at a time relative to the base config file. We then create a list of config options, called config_list. Using the examples above, we would have 2 + 2 + 3 = 7 different config options:

config_list = [
    {
        "training_parameters": {
            "batch_size": 18,
            "loss": {"name": "DiceLoss"}
        },
        "default_model":     {
            "name": "Unet",
            "dropout_rate": 0.3,
            "depth": 3
        },
        "model_name": "seg_tumor_t2",
        "path_output": "./tmp/-loss={'name': 'DiceLoss'}"
    },
    {
        "training_parameters": {
            "batch_size": 18,
            "loss": {"name": "FocalLoss", "gamma": 0.2, "alpha": 0.5}
        },
        "default_model":     {
            "name": "Unet",
            "dropout_rate": 0.3,
            "depth": 3
        },
        "model_name": "seg_tumor_t2",
        "path_output": "./tmp/-loss={'name': 'FocalLoss', 'gamma': 0.2, 'alpha': 0.5}"
    },
    {
        "training_parameters": {
            "batch_size": 18,
            "loss": {"name": "DiceLoss"}
        },
        "default_model":     {
            "name": "Unet",
            "dropout_rate": 0.3,
            "depth": 2
        },
        "model_name": "seg_tumor_t2",
        "path_output": "./tmp/-depth=2"
    },
    # etc ...
]

All Combinations:

If we select the all_combin option, we will create a list of configuration options combinatorically. Using the config examples above, we would have 2 * 3 * 2 = 12 different config options. I’m not going to write out the whole config_list because it’s quite long, but here are the combinations:

Multiple Parameters:

The multi_params option entails changing all the first elements from the list, then all the second parameters from the list, etc. If the lists are different lengths, we will just use the first n elements. In our example above, the lists are of length 2 or 3, so we will only use the first 2 elements:
Parameters:
  • file_config (string) – Configuration filename, which is used as skeleton to configure the training. This is the standard config file used in ivadomed functions. In the code, we call the keys from this config file categories. Flag: --config, -c
  • file_config_hyper (string) –

    json file containing parameters configurations to compare. Parameter “keys” of this file need to match the parameter “keys” of config file. Parameter “values” are in a list. Flag: --config-hyper, -ch

    Example:

    {"default_model": {"depth": [2, 3, 4]}}
    
  • fixed_split (bool) – If True, all the experiments are run on the same training/validation/testing subdatasets. Flag: --fixed-split
  • all_combin (bool) – If True, all parameters combinations are run. Flag: --all-combin
  • n_iterations (int) – Controls the number of time that each experiment (ie set of parameter) are run. Flag: --n-iteration, -n
  • run_test (bool) – If True, the trained model is also run on the testing subdataset. Flag: --run-test
  • all_logs (bool) – If True, all the log directories are kept for every iteration. Flag: --all-logs, -l
  • thr_increment (float) – A threshold analysis is performed at the end of the training using the trained model and the validation sub-dataset to find the optimal binarization threshold. The specified value indicates the increment between 0 and 1 used during the ROC analysis (e.g. 0.1). Flag: -t, --thr-increment
  • multi_params (bool) – If True, more than one parameter will be change at the time from the hyperparameters. All the first elements from the hyperparameters list will be applied, then all the second, etc.
  • output_dir (str) – Path to where the results will be saved.
HyperparameterOption(base_key=None, option=None, base_option=None)[source]

Hyperparameter option to edit config dictionary.

This class is used to edit a standard config file. For example, say we want to edit the following config file:

{
    "training_parameters": {
        "batch_size": 18,
        "loss": {"name": "DiceLoss"}
    },
    "default_model":     {
        "name": "Unet",
        "dropout_rate": 0.3,
        "depth": 3
    },
    "model_name": "seg_tumor_t2",
    "path_output": "./tmp/"
}

Say we want to change the loss. We could have:

Attributes:
  • base_key (str) – the key whose value you want to edit.
  • option (dict) – the full tree path to the value you want to insert.
  • base_option (dict) – the value you want to insert.
  • name (str) – the name to be used for the output folder.
get_param_list(my_dict, param_list, superkeys)[source]

Recursively create the list of hyperparameter options.

Parameters:
  • my_dict (dict) – A dictionary of parameters.
  • param_list (list)(HyperparameterOption) – A list of HyperparameterOption objects.
  • superkeys (list)(str) – TODO
Returns:

A list of HyperparameterOption objects.

Return type:

list, HyperparameterOption

make_config_list(param_list, initial_config, all_combin, multi_params)[source]

Create a list of config dictionaries corresponding to different hyperparameters.

Parameters:
  • param_list (list)(HyperparameterOption) – A list of the different hyperparameter options.
  • initial_config (dict) –

    The original config file, which we use as a basis from which to modify our hyperparameters.

    {
        "training_parameters": {
            "batch_size": 18,
            "loss": {"name": "DiceLoss"}
        },
        "default_model":     {
            "name": "Unet",
            "dropout_rate": 0.3,
            "depth": 3
        },
        "model_name": "seg_tumor_t2",
        "path_output": "./tmp/"
    }
    
  • all_combin (bool) – If true, combine the hyperparameters combinatorically.
  • multi_params (bool) – If true, combine the hyperparameters by index in the list, i.e. all the first elements, then all the second elements, etc.
Returns:

A list of configuration dictionaries, modified by the hyperparameters.

config_list = [
    {
        "training_parameters": {
            "batch_size": 18,
            "loss": {"name": "DiceLoss"}
        },
        "default_model":     {
            "name": "Unet",
            "dropout_rate": 0.3,
            "depth": 3
        },
        "model_name": "seg_tumor_t2",
        "path_output": "./tmp/-loss={'name': 'DiceLoss'}"
    },
    {
        "training_parameters": {
            "batch_size": 18,
            "loss": {"name": "FocalLoss", "gamma": 0.2, "alpha": 0.5}
        },
        "default_model":     {
            "name": "Unet",
            "dropout_rate": 0.3,
            "depth": 3
        },
        "model_name": "seg_tumor_t2",
        "path_output": "./tmp/-loss={'name': 'FocalLoss', 'gamma': 0.2, 'alpha': 0.5}"
    },
    # etc...
]

Return type:

list, dict

update_dict(d, u, base_key)[source]

Update a given dictionary recursively with a new sub-dictionary.

Example 1:

d = {
    'foo': {
        'bar': 'some example text',
        'baz': {'zag': 5}
    }
}
u = {'foo': {'baz': {'zag': 7}}}
base_key = 'zag'
>>> print(update_dict(d, u, base_key))
{
    'foo': {
        'bar': 'some example text',
        'baz': {'zag': 7}
    }
}

Example 2:

d = {
    'foo': {
        'bar': 'some example text',
        'baz': {'zag': 5}
    }
}
u = {'foo': {'baz': {'zag': 7}}}
base_key = 'foo'
>>> print(update_dict(d, u, base_key))
{
    'foo': {
        'baz': {'zag': 7}
    }
}
Parameters:
  • d (dict) – A dictionary to update.
  • u (dict) – A subdictionary to update the original one with.
  • base_key (str) – the string indicating which level to update.
Returns:

An updated dictionary.

Return type:

dict

ivadomed_compare_models

compute_statistics(dataframe, n_iterations, run_test=True, csv_out='comparison_models.csv')[source]

Compares the performance of models at inference time on a common testing dataset using paired t-tests.

It uses a dataframe generated by scripts/automate_training.py with the parameter --run-test (used to run the models on the testing dataset). It output dataframes that stores the different statistic (average, std and p_value between runs). All can be combined and stored in a csv.

Example of dataframe
path_output avg_best_training_dice avg_best_training_loss avg_best_validation_dice avg_best_validation_loss avg_test_dice std_best_training_dice std_best_training_loss std_best_validation_dice std_best_validation_loss std_test_dice p-value_testing_script-batch_size=2 p-value_testing_script-batch_size=4
testing_script-batch_size=2 -0.0019473766224109568 -0.0019473766224109568 -0.0024093631698178797 -0.0024093631698178797 0.0009537434430138293 0.0009893736332192554 0.0009893736332192554 3.545588614363517e-05 3.545588614363517e-05 0.0 1.0 0.030020368472776473
testing_script-batch_size=4 -0.0016124938847497106 -0.0016124938847497106 -0.001482845204009209 -0.001482845204009209 0.0009537434430138293 0.00011548220028372273 0.00011548220028372273 0.00022956790548947826 0.00022956790548947826 0.0 0.030020368472776473 1.0

Usage example:

ivadomed_compare_models -df results.csv -n 2 --run_test
Parameters:
  • dataframe (pandas.Dataframe) – Dataframe of results generated by automate_training. Flag: --dataframe, -df
  • n_iterations (int) – Indicates the number of time that each experiment (ie set of parameter) was run. Flag: --n_iteration, -n
  • run_test (int) – Indicates if the comparison is done on the performances on either the testing subdataset (True) either on the training/validation subdatasets. Flag: --run_test
  • csv_out (string) – Output csv name to store computed value (e.g., df.csv). Default value is model_comparison.csv. Flag -o, --output

ivadomed_prepare_dataset_vertebral_labeling

extract_mid_slice_and_convert_coordinates_to_heatmaps(path, suffix, aim=-1)[source]

This function takes as input a path to a dataset and generates a set of images: (i) mid-sagittal image and (ii) heatmap of disc labels associated with the mid-sagittal image.

Example:

ivadomed_prepare_dataset_vertebral_labeling -p path/to/bids -s _T2w -a 0
Parameters:
  • path (string) – path to BIDS dataset form which images will be generated. Flag: --path, -p
  • suffix (string) – suffix of image that will be processed (e.g., T2w). Flag: --suffix, -s
  • aim (int) – If aim is not 0, retrieves only labels with value = aim, else create heatmap with all labels. Flag: --aim, -a
Returns:

None. Images are saved in BIDS folder

ivadomed_extract_small_dataset

extract_small_dataset(input, output, n=10, contrast_list=None, include_derivatives=True, seed=-1)[source]

Extract small BIDS dataset from a larger BIDS dataset.

Example:

ivadomed_extract_small_dataset -i path/to/BIDS/dataset -o path/of/small/BIDS/dataset             -n 10 -c T1w,T2w -d 0 -s 1234
Parameters:
  • input (str) – Input BIDS folder. Flag: --input, -i
  • output (str) – Output folder. Flag: --output, -o
  • n (int) – Number of subjects in the output folder. Flag: --number, -n
  • contrast_list (list) – List of image contrasts to include. If set to None, then all available contrasts are included. Flag: --contrasts, -c
  • include_derivatives (bool) – If True, derivatives/labels/ content is also copied, only the raw images otherwise. Flag: --derivatives, -d
  • seed (int) – Set np.random.RandomState to ensure reproducibility: the same subjects will be selected if the function is run several times on the same dataset. If set to -1, each function run is independent. Flag: --seed, -s.

ivadomed_training_curve

run_plot_training_curves(input_folder, output_folder, multiple_training=False, y_lim_loss=None)[source]

Utility function to plot the training curves.

This function uses the TensorFlow summary that is generated during a training to plot for each epoch:

  • the training against the validation loss
  • the metrics computed on the validation sub-dataset.

It could consider one output path at a time, for example:

https://raw.githubusercontent.com/ivadomed/doc-figures/main/scripts/plot_loss_single.png

… or multiple (using multiple_training=True). In that case, the hard line represents the mean value across the trainings whereas the envelope represents the standard deviation:

https://raw.githubusercontent.com/ivadomed/doc-figures/main/scripts/plot_loss_multiple.png

It is also possible to compare multiple trainings (or set of trainings) by listing them in -i, separated by commas:

https://raw.githubusercontent.com/ivadomed/doc-figures/main/scripts/plot_loss_mosaic.png
Parameters:
  • input_folder (str) – Input path name. Flag: --input, -i. If using --multiple, this parameter indicates the suffix path of all log directories of interest. To compare trainings or set of trainings (using --multiple) with subplots, please list the paths by separating them with commas, e.g. path_output1, path_output2
  • output_folder (str) – Output folder. Flag: --output, -o.
  • multiple_training (bool) – Indicates if multiple log directories are considered (True) or not (False). Flag: --multiple. All available folders with -i as prefix are considered. The plot represents the mean value (hard line) surrounded by the standard deviation (envelope).
  • y_lim_loss (list) – List of the lower and upper limits of the y-axis of the loss plot.

ivadomed_download_data

install_data(url, dest_folder, keep=False)[source]

Download a data bundle from an URL and install it in the destination folder.

Usage example

ivadomed_download_data -d data_testing -o ivado_testing_data

Existing data bundles:

Note

The function tries to be smart about the data contents. Examples:

a. If the archive only contains a README.md, and the destination folder is ${dst}, ${dst}/README.md will be created. Note: an archive not containing a single folder is commonly known as a “bomb” because it puts files anywhere in the current working directory.( see Tarbomb)

b. If the archive contains a ${dir}/README.md, and the destination folder is ${dst}, ${dst}/README.md will be created. Note: typically the package will be called ${basename}-${revision}.zip and contain a root folder named ${basename}-${revision}/ under which all the other files will be located. The right thing to do in this case is to take the files from there and install them in ${dst}. - Uses download_data() to retrieve the data. - Uses unzip() to extract the bundle.

Parameters:
  • url (string) – URL or sequence thereof (if mirrors). For this package there is a dictionnary listing existing data bundle with their url. Type ivadomed_download_data -h to see possible value. Flag -d
  • dest_folder (string) – destination directory for the data (to be created). If not used the output folder will be the name of the data bundle. Flag -o, --output
  • keep (bool) – whether to keep existing data in the destination folder (if it exists). Flag -k, --keep