Setup, Data, and Backends

This page covers the complete preparation path, from a fresh checkout through processed LIDC-IDRI data and reusable caches to backend-specific preflight requirements. It is included from the maintained reproduction guide so paths and commands cannot drift independently.

Repository map and reproduction procedure

File	Purpose
LION/data_loaders/LIDC_IDRI/download_LIDC_IDRI.sh	Download and resume the raw TCIA LIDC-IDRI collection.
LION/data_loaders/LIDC_IDRI/pre_process_lidc_idri.py	Convert raw DICOM scans into LION’s processed slice dataset.
pipeline/run_prepare_lidc_cache.sh	Build the default/full 256 and default 512 compressed training caches.
pipeline/PaDIS_run_pipeline.sh	Unified complete training, inference, generation, table, and figure entry point for GCP or Slurm.
pipeline/PaDIS_finalise_pipeline.sh	Re-run generation, verification, timing, table, and figure production.
training/PaDIS_LIDC_256.py	Train 256-resolution patch or whole-image diffusion priors.
training/PaDIS_LIDC_512.py	Train the native-resolution patch prior.
training/PaDIS_LIDC_PnP_denoiser.py	Train unconditioned or noise-conditioned DRUNet models.
reconstruction/PaDIS_LIDC_reconstruction.py	Run one reconstruction configuration.
reconstruction/PaDIS_run_reconstruction_matrix.py	Construct, list, validate, or run the experiment matrix.
reconstruction/PaDIS_LIDC_generation.py	Generate unconditional whole-image or patch-assembled samples.
tuning/PaDIS_tune_reconstruction_hyperparameters.py	Run reconstruction tuning candidates.
tuning/PaDIS_run_reproduction_tuning.py	Re-run the complete validation tuning grid on the final inference checkpoints.
tuning/PaDIS_summarize_hparam_tuning.py	Summarise completed tuning records.
tuning/PaDIS_hparam_defaults.py	Export tuning results to the repository JSON registry.
tuning/TUNING.md	Reproduce and audit the validation-set hyperparameter search.
reconstruction/PaDIS_reconcile_reconstruction_manifest.py	Reconcile a rebuilt manifest with existing outputs.
reconstruction/PaDIS_verify_reconstruction_matrix.py	Verify outputs and write result/uncertainty tables.
reporting/PaDIS_make_paper_figures.py	Build figures from saved outputs.
reporting/PaDIS_make_tables.py	Build publication LaTeX tables and decoded table CSVs from the verification CSV.
platforms/gcp/run_PaDIS_GCP_spot_training.sh	Automatic GCP training, intensive validation, and inference.
platforms/gcp/run_PaDIS_GCP_manual_reconstruction.sh	Resumable GCP/Colab inference and generation.
platforms/gcp/PaDIS_Colab_manual_reconstruction.ipynb	Colab setup, authentication, environment installation, and launch cells.
platforms/slurm/submit_PaDIS_A100_pipeline.sh	Submit the equivalent training and reconstruction pipeline on Slurm.

The directory is organised by responsibility: core/ contains shared preset definitions, pipeline/ contains end-to-end orchestration, platforms/ contains GCP and Slurm launchers, and training/, reconstruction/, tuning/, and reporting/ contain the corresponding experiment stages. config/ contains the checked-in inference registry. Experiment outputs remain under $LION_EXPERIMENTS_PATH/PaDIS; the source layout does not rename or migrate saved data.

Run scripts from the LION repository root. Use each program’s --help output for diagnostic options not documented here.

Reproduction procedure

This is the shortest complete reproduction path. The later sections document the models, matrix, restart behaviour, and platform controls in more detail.

1. Install LION

Clone the repository at the revision being reproduced and initialise its submodules. The commands below follow LION’s root installation instructions but name the environment lion-dev, which is the name used by the PaDIS launchers:

git clone https://github.com/CambridgeCIA/LION.git
cd LION
git submodule update --init --recursive
conda env create --file env_base.yml --name lion-dev
conda activate lion-dev
python -m pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
python -m pip install -e .

Match the PyTorch wheel and cuda-version in env_base.yml to the installed CUDA driver when CUDA 12.8 is unavailable. Record git rev-parse HEAD with the outputs: code revision, dataset split, checkpoint policy, and registry contents all affect reproducibility.

Set persistent paths before importing LION:

export LION_DATA_PATH=/path/to/Datasets
export LION_EXPERIMENTS_PATH="$LION_DATA_PATH/experiments"
mkdir -p "$LION_DATA_PATH" "$LION_EXPERIMENTS_PATH"

2. Download and process LIDC-IDRI

LION downloads the TCIA LIDC-IDRI collection using the repository’s TCIA manifest and the NBIA Data Retriever CLI. The helper installs the retriever inside the data directory, downloads the diagnosis spreadsheet, verifies MD5 checksums, and resumes interrupted downloads by requesting missing series:

bash LION/data_loaders/LIDC_IDRI/download_LIDC_IDRI.sh

Raw DICOM data will be under $LION_DATA_PATH/raw/LIDC-IDRI. NBIA license acceptance, Java/container alternatives, resume and redownload controls, and recovery from incomplete series are documented in LION/data_loaders/LIDC_IDRI/README.md.

Preprocessing has a separate environment because pylidc has specialised dependencies:

conda env create \
  --file LION/data_loaders/LIDC_IDRI/pre_process_lidc_idri_environment.yml
conda run -n lidc_idri \
  python LION/data_loaders/LIDC_IDRI/pre_process_lidc_idri.py

This writes 512-by-512 slice arrays, masks, and metadata under $LION_DATA_PATH/processed/LIDC-IDRI/LIDC-IDRI-*. The preprocessor finishes with a completeness check; a complete current download has at least 1,010 processed patient directories and 282,776 regular files. Do not start cache creation or training if that check reports missing patients, scan errors, or a short file count.

3. Build reusable training caches

Install zstd, activate lion-dev, and create the three cache archives used by the training pipelines:

conda activate lion-dev
bash scripts/paper_scripts/PaDIS-Reproduction/pipeline/run_prepare_lidc_cache.sh

The default variants are 256-default, 256-full, and 512-default. They are written below:

$LION_DATA_PATH/processed/LIDC-IDRI-cache/padis_256/archives/
$LION_DATA_PATH/processed/LIDC-IDRI-cache/padis_512/archives/

Cache construction is deterministic for the processed data and seed. It can be rerun safely; set PADIS_REBUILD_CACHE=1 only when the existing cache should be replaced. On Slurm, the complete pipeline submits its equivalent cache job, so this local command is optional if the compute nodes can read the processed dataset and write the cache root.

4. Run training through final reporting

Check the backend dispatch first:

bash scripts/paper_scripts/PaDIS-Reproduction/pipeline/PaDIS_run_pipeline.sh \
  --backend gcp --dry-run
bash scripts/paper_scripts/PaDIS-Reproduction/pipeline/PaDIS_run_pipeline.sh \
  --backend slurm --dry-run

Then run one backend:

# Synchronous and resumable on the configured GCP machine:
bash scripts/paper_scripts/PaDIS-Reproduction/pipeline/PaDIS_run_pipeline.sh \
  --backend gcp

# Or submit the dependency-linked Slurm pipeline:
PADIS_RUN_STAMP=padis-reproduction \
  bash scripts/paper_scripts/PaDIS-Reproduction/pipeline/PaDIS_run_pipeline.sh \
  --backend slurm

For a short local integration check using existing staged checkpoints, run:

export PADIS_FAST_SMOKE_TRAINING_ROOT=/path/to/matrix-compatible/training-root
bash scripts/paper_scripts/PaDIS-Reproduction/pipeline/PaDIS_run_pipeline.sh \
  --fast-smoke

--fast-smoke executes one sample, one experiment per selected model, one outer and one inner diffusion step, and one TV/PnP/CG iteration. It covers 12 representative tuning families, a compact reconstruction matrix, patch and whole-image generation, verification, partial tables, and partial figures. Outputs go to a new debug_runs/fast_smoke_* folder by default. It checks the SHA256 of the committed hyperparameter registry before and after and fails if the settings change. The tuning launcher’s broader --smoke mode retains all candidates while applying the same execution-depth limits.

After the processed dataset exists, these entry points cover cache staging, diffusion and PnP training, intensive validation, checkpoint selection, the 109-job reconstruction matrix, unconditional generation, verification, timings, LaTeX/CSV tables, and PNG/PDF figures. They are resumable; retain the same GCP run name or Slurm run stamp when continuing an interrupted run.

Exact reproduction uses the committed config/reconstruction_hparam_defaults.json. Re-running the validation search is an optional audit of how those settings were selected, not a prerequisite for consuming the registry. After training, the complete search can be repeated as described in tuning/TUNING.md.

5. Check the final products

The default finaliser writes:

$LION_EXPERIMENTS_PATH/PaDIS/reconstruction_presets/   unconditional samples
$LION_EXPERIMENTS_PATH/PaDIS/paper_tables/             LaTeX and decoded CSV tables
$LION_EXPERIMENTS_PATH/PaDIS/paper_figures/            PNG/PDF figures and manifest
<reconstruction-root>/reconstruction_matrix_verification.{json,csv}

Treat a run as complete only when verification succeeds against reconstruction_matrix_jobs.json, rather than when scheduler or marker counts alone look complete. The figure manifest records missing panels and whether each generated figure is publication-ready.

Backend requirements

Before a real launch, the following must be true:

• LION_DATA_PATH identifies the persistent dataset root and contains the processed LIDC-IDRI patient folders plus the 256-default, 256-full, and 512-default cache archives listed below.

• The checkout contains the committed config/reconstruction_hparam_defaults.json used to build the 109-job reconstruction manifest.

• The lion-dev Conda/Mamba environment has been created from LION’s current environment specification and includes PyTorch, ASTRA, and tomosipo.

• A CUDA GPU is visible to PyTorch and nvidia-smi.

• W&B authentication is available when PADIS_WANDB_MODE=online; set PADIS_WANDB_MODE=offline deliberately when network logging is not wanted.

• The persistent experiment root is writable and has the capacity described in Data layout and storage.

For GCP, use either the supported Colab notebook/startup hook, which mounts padis-bucket at /mnt/data and provisions the environment, or prepare an equivalent VM manually. The normal persistent layout is:

export LION_DATA_PATH=/mnt/data/Datasets
export LION_EXPERIMENTS_PATH=/mnt/data/Datasets/experiments

The GCP runner stages expanded training caches under /mnt/ram-disk; the VM must have sufficient RAM and permission to create that temporary filesystem. The bucket mount must be active before launch. Spot pre-emption is supported: rerun the identical command with the same PADIS_GCP_RUN_NAME to resume.

For Slurm, run on a login node with sbatch, squeue, and scancel available. The defaults target the Cambridge CSD3 Ampere setup and accounts shown in the Slurm scripts. On another cluster, set at least PADIS_SLURM_ACCOUNT, PADIS_CACHE_SLURM_ACCOUNT, relevant partitions, and LION_DATA_PATH. The compute nodes must see the same repository, dataset, cache, and experiment paths. The common Slurm helper activates lion-dev (falling back to padis-dev) through the cluster module/Mamba setup. Set a stable PADIS_RUN_STAMP when resubmitting or inspecting one run; otherwise a new timestamped training and reconstruction root is created.

Common useful preflight commands are:

test -d "$LION_DATA_PATH/processed/LIDC-IDRI"
test -d "$LION_DATA_PATH/processed/LIDC-IDRI-cache/padis_256/archives"
test -d "$LION_DATA_PATH/processed/LIDC-IDRI-cache/padis_512/archives"
conda run -n lion-dev python -c 'import torch; print(torch.cuda.is_available())'

The unified wrapper delegates checkpointing and scheduling to the backend scripts. GCP finalises synchronously; Slurm submits verification and finalisation with scheduler dependencies. PADIS_SUBMIT_FINALISE=0 deliberately omits generation and publication outputs.

Data layout and storage

Set LION_DATA_PATH to the data root. On the GCP/Colab path it defaults to /mnt/data/Datasets, where /mnt/data is the mounted padis-bucket.

Data	Default path below LION_DATA_PATH
Processed slices	processed/LIDC-IDRI/LIDC-IDRI-*/
256 cache archives	processed/LIDC-IDRI-cache/padis_256/archives/*.pt.zst
512 cache archives	processed/LIDC-IDRI-cache/padis_512/archives/*.pt.zst
PaDIS experiment root	experiments/PaDIS/
GCP training root	experiments/PaDIS/final_real_runs/PaDIS-Reproduction-GCP/
GCP reconstruction root	experiments/PaDIS/final_real_runs/PaDIS-Reproduction-GCP_reconstruction/

The completed local Data tree occupied approximately 504 GiB of allocated disk space (538,120,458,610 apparent bytes) when this README was updated. Provision at least 504 GiB for an equivalent reproduction and preferably 600 GiB or more to accommodate temporary downloads, cache construction, checkpoint replacement, and filesystem overhead. Check the actual requirement and available capacity with:

du -sh "$LION_DATA_PATH"
df -h "$LION_DATA_PATH"

Raw DICOM, processed NumPy slices, compressed caches, expanded RAM-disk caches, model checkpoints, and reconstruction tensors can coexist during the run. The RAM-disk copy is temporary but requires separate system memory or scratch capacity while training is active.

The archived split assigns sorted patient-ID slots without patient leakage. The 1,012 slots include two empty slots retained to preserve that split, so they do not imply 1,012 contributing patients. The completed dataset counts are:

Split	Patient-ID slots	Four-slice regime	All-slice regime
Train	809	2,713	189,725
Validation	101	328	27,426
Test	102	326	25,719
Total	1,012	3,367	242,870

The default regime uses at most four slices per patient, capped by the smaller of the available nodule and non-nodule slice counts before balancing the two classes. A patient without an annotated nodule can therefore contribute no image. --full-lidc selects every processed slice. Cache identity includes the split, resolution, and selection regime, preventing a cache for one setup from being reused silently for another.

Training scripts can load .pt.zst archives directly into a RAM-backed cache. Decompression uses zstd -T0, which uses all available CPU cores. The archive and model outputs remain on persistent storage; the expanded RAM-disk files are disposable. run_prepare_lidc_cache.sh prepares archives explicitly, while the GCP runners stage the required 256-default, 256-full, and 512-default variants automatically.