Scripts

This section contains a collection of useful scripts for quality control during the training of models.

ivadomed_visualize_transforms

ivadomed_convert_to_onnx

convert_pytorch_to_onnx(model, dimension, n_channels, gpu_id=0)[source]

Convert PyTorch model to ONNX.

The integration of Deep Learning models into the clinical routine requires cpu optimized models. To export the PyTorch models to ONNX format and to run the inference using ONNX Runtime is a time and memory efficient way to answer this need.

This function converts a model from PyTorch to ONNX format, with information of whether it is a 2D or 3D model (-d).

Parameters
  • model (string) – Model filename. Flag: --model, -m.

  • dimension (int) – Indicates whether the model is 2D or 3D. Choice between 2 or 3. Flag: --dimension, -d

  • gpu_id (string) – GPU ID, if available. Flag: --gpu_id, -g

ivadomed_automate_training

ivadomed_compare_models

compute_statistics(dataframe, n_iterations, run_test=True, csv_out='comparison_models.csv')[source]

Compares the performance of models at inference time on a common testing dataset using paired t-tests.

It uses a dataframe generated by scripts/automate_training.py with the parameter --run-test (used to run the models on the testing dataset). It output dataframes that stores the different statistic (average, std and p_value between runs). All can be combined and stored in a csv.

Example of dataframe

path_output

avg_best_training_dice

avg_best_training_loss

avg_best_validation_dice

avg_best_validation_loss

avg_test_dice

std_best_training_dice

std_best_training_loss

std_best_validation_dice

std_best_validation_loss

std_test_dice

p-value_testing_script-batch_size=2

p-value_testing_script-batch_size=4

testing_script-batch_size=2

-0.0019473766224109568

-0.0019473766224109568

-0.0024093631698178797

-0.0024093631698178797

0.0009537434430138293

0.0009893736332192554

0.0009893736332192554

3.545588614363517e-05

3.545588614363517e-05

0.0

1.0

0.030020368472776473

testing_script-batch_size=4

-0.0016124938847497106

-0.0016124938847497106

-0.001482845204009209

-0.001482845204009209

0.0009537434430138293

0.00011548220028372273

0.00011548220028372273

0.00022956790548947826

0.00022956790548947826

0.0

0.030020368472776473

1.0

Usage example:

ivadomed_compare_models -df results.csv -n 2 --run_test
Parameters
  • dataframe (pandas.Dataframe) – Dataframe of results generated by automate_training. Flag: --dataframe, -df

  • n_iterations (int) – Indicates the number of time that each experiment (ie set of parameter) was run. Flag: --n_iteration, -n

  • run_test (int) – Indicates if the comparison is done on the performances on either the testing subdataset (True) either on the training/validation subdatasets. Flag: --run_test

  • csv_out (string) – Output csv name to store computed value (e.g., df.csv). Default value is model_comparison.csv. Flag -o, --output

ivadomed_prepare_dataset_vertebral_labeling

ivadomed_extract_small_dataset

extract_small_dataset(input, output, n=10, contrast_list=None, include_derivatives=True, seed=- 1)[source]

Extract small BIDS dataset from a larger BIDS dataset.

Example:

ivadomed_extract_small_dataset -i path/to/BIDS/dataset -o path/of/small/BIDS/dataset             -n 10 -c T1w,T2w -d 0 -s 1234
Parameters
  • input (str) – Input BIDS folder. Flag: --input, -i

  • output (str) – Output folder. Flag: --output, -o

  • n (int) – Number of subjects in the output folder. Flag: --number, -n

  • contrast_list (list) – List of image contrasts to include. If set to None, then all available contrasts are included. Flag: --contrasts, -c

  • include_derivatives (bool) – If True, derivatives/labels/ content is also copied, only the raw images otherwise. Flag: --derivatives, -d

  • seed (int) – Set np.random.RandomState to ensure reproducibility: the same subjects will be selected if the function is run several times on the same dataset. If set to -1, each function run is independent. Flag: --seed, -s.

ivadomed_training_curve

run_plot_training_curves(input_folder, output_folder, multiple_training=False, learning_rate=False, y_lim_loss=None)[source]

Utility function to plot the training curves and save data as .csv files.

This function uses the TensorFlow summary that is generated during a training to plot for each epoch:

  • the training against the validation loss,

  • the metrics computed on the validation sub-dataset,

  • the learning rate if learning_rate is True.

It could consider one output path at a time, for example:

https://raw.githubusercontent.com/ivadomed/doc-figures/main/scripts/plot_loss_single.png

… or multiple (using multiple_training=True). In that case, the hard line represents the mean value across the trainings whereas the envelope represents the standard deviation:

https://raw.githubusercontent.com/ivadomed/doc-figures/main/scripts/plot_loss_multiple.png

It is also possible to compare multiple trainings (or set of trainings) by listing them in -i, separated by commas:

https://raw.githubusercontent.com/ivadomed/doc-figures/main/scripts/plot_loss_mosaic.png
Parameters
  • input_folder (str) – Input path name. Flag: --input, -i. If using --multiple, this parameter indicates the prefix path of all log directories of interest. To compare trainings (not using --multiple) or set of trainings (using --multiple) with subplots, please list the paths by separating them with commas, e.g. path_output1, path_output2

  • output_folder (str) – Output folder. Flag: --output, -o.

  • multiple_training (bool) – Indicates if multiple log directories are considered (True) or not (False). Flag: --multiple. All available folders with -i as prefix are considered. The plot represents the mean value (hard line) surrounded by the standard deviation (envelope).

  • learning_rate (bool) – Indicates if the summary event file for learning rate is considered (True) or not (False). Flag: --lr. The limits on the y-axis plot are automatically defined.

  • y_lim_loss (list) – List of the lower and upper limits of the y-axis of the loss plot, otherwise these limits are automatically defined. Please separate the lower and the upper limit by a comma, e.g. -1,0. Note: for the validation metrics: the y-limits are always 0.0 and 1.0 except for the hausdorff score where the limits are automatically defined.

ivadomed_download_data

install_data(url, dest_folder, keep=False)[source]

Download a data bundle from an URL and install it in the destination folder.

Usage example

ivadomed_download_data -d data_testing -o ivado_testing_data

Existing data bundles:

  • data_example_spinegeneric10 randomly picked subject from

    Spine Generic. Used for Tutorial and example in Ivadomed.

  • data_testing : Data Used for integration/unit test in Ivadomed.

  • t2_tumor : Cord tumor segmentation model, trained on T2-weighted contrast.

  • t2star_sc : spinal cord segmentation model, trained on T2-star contrast.

  • mice_uqueensland_gmGray matter segmentation model on mouse MRI. Data from University of

    Queensland.

  • mice_uqueensland_sc : Cord segmentation model on mouse MRI. Data from University of Queensland.

  • findcord_tumor : Cord localisation model, trained on T2-weighted images with tumor.

  • model_find_disc_t1 : Intervertebral disc detection model trained on T1-weighted images.

  • model_find_disc_t2 : Intervertebral disc detection model trained on T2-weighted images.

  • data_functional_testing : Data used for functional testing in Ivadomed.

  • data_axondeepseg_semSEM dataset for AxonDeepSeg. 10 rat spinal cord samples with axon and myelin

    manual segmentation labels. Used for microscopy tutorial in ivadomed.

Note

The function tries to be smart about the data contents. Examples:

a. If the archive only contains a README.md, and the destination folder is ${dst}, ${dst}/README.md will be created. Note: an archive not containing a single folder is commonly known as a “bomb” because it puts files anywhere in the current working directory.( see Tarbomb)

b. If the archive contains a ${dir}/README.md, and the destination folder is ${dst}, ${dst}/README.md will be created. Note: typically the package will be called ${basename}-${revision}.zip and contain a root folder named ${basename}-${revision}/ under which all the other files will be located. The right thing to do in this case is to take the files from there and install them in ${dst}. - Uses download_data() to retrieve the data. - Uses unzip() to extract the bundle.

Parameters
  • url (string) – URL or sequence thereof (if mirrors). For this package there is a dictionnary listing existing data bundle with their url. Type ivadomed_download_data -h to see possible value. Flag -d

  • dest_folder (string) – destination directory for the data (to be created). If not used the output folder will be the name of the data bundle. Flag -o, --output

  • keep (bool) – whether to keep existing data in the destination folder (if it exists). Flag -k, --keep

ivadomed_visualize_and_compare_testing_models

visualize_and_compare_models(ofolders, metric='dice_class0', metadata=None)[source]

This function allows violinplots visualization of multiple evaluation models simultaneously and performs a Kolmogorov–Smirnov significance test between each combination of models. The mean values of the datapoints for each violinplot is superimposed on the top.

If only one model is selected as input, only the Violinplot will be presented (no test will be superimposed)

The datapoints within each violinplot are interactive. The subject_id and MRI sequence of each point are displayed when clicked (as shown on the violinplot to the right of the example figure below).

Note

If more than 4 model outputs are selected to be compared, the significance tests are not displayed since the figure becomes very busy

Usage example:

visualize_and_compare_testing_models.py --ofolders ~/logs/logs_T1w ~/logs/logs_T2w
                                        --metric dice_class0 --metadata pathology ms
_images/visualize_and_compare_models.png
Parameters
  • ofolders (list) – list of folders that contain the outputs of the models to be compared, Flag: --ofolders

  • metric (str) – column of “results_eval/evaluation_3Dmetrics.csv” to be used on the plots (default: dice_class0), Flag: --metric

  • metadata (list) – Allows visualization of violinplots only from subjects that match the metadata criteria. 2 elements - (1) column label of the dataframe.csv metadata so only subjects that belong to that category will be used and (2) string to be matched, Flag: --metadata, Example: “–metadata pathology ms”