This section contains a collection of useful scripts for quality control during the training of models.



convert_pytorch_to_onnx(model, dimension, n_channels, gpu_id=0)[source]

Convert PyTorch model to ONNX.

The integration of Deep Learning models into the clinical routine requires cpu optimized models. To export the PyTorch models to ONNX format and to run the inference using ONNX Runtime is a time and memory efficient way to answer this need.

This function converts a model from PyTorch to ONNX format, with information of whether it is a 2D or 3D model (-d).

  • model (string) – Model filename. Flag: --model, -m.

  • dimension (int) – Indicates whether the model is 2D or 3D. Choice between 2 or 3. Flag: --dimension, -d

  • gpu_id (string) – GPU ID, if available. Flag: --gpu_id, -g



compute_statistics(dataframe, n_iterations, run_test=True, csv_out='comparison_models.csv')[source]

Compares the performance of models at inference time on a common testing dataset using paired t-tests.

It uses a dataframe generated by scripts/ with the parameter --run-test (used to run the models on the testing dataset). It output dataframes that stores the different statistic (average, std and p_value between runs). All can be combined and stored in a csv.

Example of dataframe








































Usage example:

ivadomed_compare_models -df results.csv -n 2 --run_test
  • dataframe (pandas.Dataframe) – Dataframe of results generated by automate_training. Flag: --dataframe, -df

  • n_iterations (int) – Indicates the number of time that each experiment (ie set of parameter) was run. Flag: --n_iteration, -n

  • run_test (int) – Indicates if the comparison is done on the performances on either the testing subdataset (True) either on the training/validation subdatasets. Flag: --run_test

  • csv_out (string) – Output csv name to store computed value (e.g., df.csv). Default value is model_comparison.csv. Flag -o, --output



extract_small_dataset(input, output, n=10, contrast_list=None, include_derivatives=True, seed=- 1)[source]

Extract small BIDS dataset from a larger BIDS dataset.


ivadomed_extract_small_dataset -i path/to/BIDS/dataset -o path/of/small/BIDS/dataset             -n 10 -c T1w,T2w -d 0 -s 1234
  • input (str) – Input BIDS folder. Flag: --input, -i

  • output (str) – Output folder. Flag: --output, -o

  • n (int) – Number of subjects in the output folder. Flag: --number, -n

  • contrast_list (list) – List of image contrasts to include. If set to None, then all available contrasts are included. Flag: --contrasts, -c

  • include_derivatives (bool) – If True, derivatives/labels/ content is also copied, only the raw images otherwise. Flag: --derivatives, -d

  • seed (int) – Set np.random.RandomState to ensure reproducibility: the same subjects will be selected if the function is run several times on the same dataset. If set to -1, each function run is independent. Flag: --seed, -s.


run_plot_training_curves(input_folder, output_folder, multiple_training=False, learning_rate=False, y_lim_loss=None)[source]

Utility function to plot the training curves and save data as .csv files.

This function uses the TensorFlow summary that is generated during a training to plot for each epoch:

  • the training against the validation loss,

  • the metrics computed on the validation sub-dataset,

  • the learning rate if learning_rate is True.

It could consider one output path at a time, for example:

… or multiple (using multiple_training=True). In that case, the hard line represents the mean value across the trainings whereas the envelope represents the standard deviation:

It is also possible to compare multiple trainings (or set of trainings) by listing them in -i, separated by commas:
  • input_folder (str) – Input path name. Flag: --input, -i. If using --multiple, this parameter indicates the prefix path of all log directories of interest. To compare trainings (not using --multiple) or set of trainings (using --multiple) with subplots, please list the paths by separating them with commas, e.g. path_output1, path_output2

  • output_folder (str) – Output folder. Flag: --output, -o.

  • multiple_training (bool) – Indicates if multiple log directories are considered (True) or not (False). Flag: --multiple. All available folders with -i as prefix are considered. The plot represents the mean value (hard line) surrounded by the standard deviation (envelope).

  • learning_rate (bool) – Indicates if the summary event file for learning rate is considered (True) or not (False). Flag: --lr. The limits on the y-axis plot are automatically defined.

  • y_lim_loss (list) – List of the lower and upper limits of the y-axis of the loss plot, otherwise these limits are automatically defined. Please separate the lower and the upper limit by a comma, e.g. -1,0. Note: for the validation metrics: the y-limits are always 0.0 and 1.0 except for the hausdorff score where the limits are automatically defined.


install_data(url, dest_folder, keep=False)[source]

Download a data bundle from an URL and install it in the destination folder.

Usage example

ivadomed_download_data -d data_testing -o ivado_testing_data

Existing data bundles:

  • data_example_spinegeneric10 randomly picked subject from

    Spine Generic. Used for Tutorial and example in Ivadomed.

  • data_testing : Data Used for integration/unit test in Ivadomed.

  • t2_tumor : Cord tumor segmentation model, trained on T2-weighted contrast.

  • t2star_sc : spinal cord segmentation model, trained on T2-star contrast.

  • mice_uqueensland_gmGray matter segmentation model on mouse MRI. Data from University of


  • mice_uqueensland_sc : Cord segmentation model on mouse MRI. Data from University of Queensland.

  • findcord_tumor : Cord localisation model, trained on T2-weighted images with tumor.

  • model_find_disc_t1 : Intervertebral disc detection model trained on T1-weighted images.

  • model_find_disc_t2 : Intervertebral disc detection model trained on T2-weighted images.

  • data_functional_testing : Data used for functional testing in Ivadomed.

  • data_axondeepseg_semSEM dataset for AxonDeepSeg. 10 rat spinal cord samples with axon and myelin

    manual segmentation labels. Used for microscopy tutorial in ivadomed.


The function tries to be smart about the data contents. Examples:

a. If the archive only contains a, and the destination folder is ${dst}, ${dst}/ will be created. Note: an archive not containing a single folder is commonly known as a “bomb” because it puts files anywhere in the current working directory.( see Tarbomb)

b. If the archive contains a ${dir}/, and the destination folder is ${dst}, ${dst}/ will be created. Note: typically the package will be called ${basename}-${revision}.zip and contain a root folder named ${basename}-${revision}/ under which all the other files will be located. The right thing to do in this case is to take the files from there and install them in ${dst}. - Uses download_data() to retrieve the data. - Uses unzip() to extract the bundle.

  • url (string) – URL or sequence thereof (if mirrors). For this package there is a dictionnary listing existing data bundle with their url. Type ivadomed_download_data -h to see possible value. Flag -d

  • dest_folder (string) – destination directory for the data (to be created). If not used the output folder will be the name of the data bundle. Flag -o, --output

  • keep (bool) – whether to keep existing data in the destination folder (if it exists). Flag -k, --keep


visualize_and_compare_models(ofolders, metric='dice_class0', metadata=None)[source]

This function allows violinplots visualization of multiple evaluation models simultaneously and performs a Kolmogorov–Smirnov significance test between each combination of models. The mean values of the datapoints for each violinplot is superimposed on the top.

If only one model is selected as input, only the Violinplot will be presented (no test will be superimposed)

The datapoints within each violinplot are interactive. The subject_id and MRI sequence of each point are displayed when clicked (as shown on the violinplot to the right of the example figure below).


If more than 4 model outputs are selected to be compared, the significance tests are not displayed since the figure becomes very busy

Usage example: --ofolders ~/logs/logs_T1w ~/logs/logs_T2w
                                        --metric dice_class0 --metadata pathology ms
  • ofolders (list) – list of folders that contain the outputs of the models to be compared, Flag: --ofolders

  • metric (str) – column of “results_eval/evaluation_3Dmetrics.csv” to be used on the plots (default: dice_class0), Flag: --metric

  • metadata (list) – Allows visualization of violinplots only from subjects that match the metadata criteria. 2 elements - (1) column label of the dataframe.csv metadata so only subjects that belong to that category will be used and (2) string to be matched, Flag: --metadata, Example: “–metadata pathology ms”