Running, Resuming, Verifying, and Reporting

This operational guide covers Colab/manual GCP execution, direct matrix inspection, expected runtimes, restart markers, semantic manifest reconciliation, verification and bootstrap uncertainty, timing extraction, tables, figures, cropping, labels, and display units.

Manual reconstruction and matrix inspection

Colab/manual GCP

The supported Colab entry point is:

scripts/paper_scripts/PaDIS-Reproduction/platforms/gcp/PaDIS_Colab_manual_reconstruction.ipynb

It clones the repository to local disk, installs Miniforge and the LION environment, authenticates GCP/W&B as required, mounts padis-bucket at /mnt/data, and launches:

bash scripts/paper_scripts/PaDIS-Reproduction/platforms/gcp/run_PaDIS_GCP_manual_reconstruction.sh

The manual runner defaults to two concurrent reconstruction workers per GPU. Set PADIS_RECON_TASKS_PER_GPU to override this. It checks all model inputs and trains missing supported models before inference. Training archives are decompressed into /mnt/ram-disk; the runner removes that temporary mount after training and syncs persistent outputs after each job. Use this path to run inference from existing checkpoints without resubmitting the complete training pipeline.

Inspecting the matrix directly

List the resolved jobs without running them:

python -u scripts/paper_scripts/PaDIS-Reproduction/reconstruction/PaDIS_run_reconstruction_matrix.py \
  --training-root "$PADIS_TRAIN_ROOT" \
  --output-root "$PADIS_RECON_ROOT" \
  --checkpoint-policy min_intense_val \
  --hparam-defaults json \
  --models method_default \
  --methods all \
  --experiments paper_matrix \
  --ablations all \
  --implementations method_default \
  --geometries lion \
  --job-order gcp_spot \
  --pnp-cg-iterations 50 \
  --pnp-cg-tolerance 1e-7 \
  --list

Replace --list with --count, --check-inputs, or --task-index N to count, validate, or execute one resolved job. The runners generate this list once, store it as reconstruction_matrix_jobs.json, and reconcile rebuilt manifests against valid existing outputs.

Expected reconstruction runtimes

The following estimates multiply the logged mean 20-view time per slice by the planned evaluation size. Standard rows use 25 slices; patch averaging and stitching use four. Each cell is the expected total followed by the measured mean per slice.

Reconstruction method	LION-physics	Paper	Public-compatible
FDK	2.0 min (4.71 s)	–	–
CP	3.4 min (8.13 s)	–	–
Predictor-corrector	5.9 min (14.11 s)	5.9 min (14.16 s)	5.8 min (14.02 s)
PnP-ADMM	11.1 min (26.63 s)	–	–
Langevin	21.5 min (51.49 s)	23.5 min (56.42 s)	23.6 min (56.55 s)
VE-DDNM	25.9 min (62.08 s)	25.6 min (61.34 s)	24.4 min (58.54 s)
PaDIS-DPS	49.5 min (118.85 s)	50.2 min (120.45 s)	49.7 min (119.22 s)
Whole-image VE-DPS	1.09 h (156.44 s)	1.09 h (156.89 s)	–
Patch stitching, four slices	1.61 h (1,450.73 s)	–	1.62 h (1,461.08 s)
Patch averaging, four slices	1.63 h (1,468.76 s)	–	1.63 h (1,468.75 s)

These displayed 20-view rows sum to approximately 14.14 GPU-hours if executed serially. This is not the wall-clock duration of the complete 109-job matrix: jobs can share a GPU or run concurrently, other experiments have different view counts, and native-512 work is substantially more expensive. The values are planning estimates parsed from concurrent pipeline logs, not isolated benchmarks. Training adds approximately 12 hours per patch prior and 24 hours per whole-image prior, plus PnP training and scheduler/setup overhead; arrays reduce elapsed wall time when enough GPUs are available.

Outputs, restart state, and reporting

Each reconstruction result directory contains at least metrics.json and reconstructions.pt; optional preview images and sampler traces are stored alongside them. metrics.json records the resolved method, implementation, experiment, checkpoint, sampler settings, per-image metrics, and summary.

Manual reconstruction state is stored under:

<reconstruction-root>/.manual_gcp_reconstruction/
  done/
  running/
  failed/
  logs/
  runtime/

A .done marker is accepted only when the corresponding output validates. Interrupted and failed jobs are therefore eligible for rerun, while valid completed jobs survive task reordering or manifest rebuilding. Existing sample files are also reusable when a resumed configuration requests fewer samples.

Generate the final verification JSON and CSV with:

python -u scripts/paper_scripts/PaDIS-Reproduction/reconstruction/PaDIS_verify_reconstruction_matrix.py \
  --root "$PADIS_RECON_ROOT" \
  --expected-jobs-json "$PADIS_RECON_ROOT/reconstruction_matrix_jobs.json" \
  --output-json "$PADIS_RECON_ROOT/reconstruction_matrix_verification.json" \
  --output-csv "$PADIS_RECON_ROOT/reconstruction_matrix_verification.csv"

The default verifier performs 2,000 deterministic image-level bootstrap resamples at 95% confidence using seed 33. The CSV includes the aggregate metrics, bootstrap standard errors and confidence limits. Use the expected-jobs manifest to detect missing outputs rather than relying only on marker counts. Metrics are calculated in the shared normalised-intensity domain with data range one and without per-image rescaling. The bootstrap resamples images, not training runs, so its intervals describe slice variation and do not measure retraining uncertainty. Relative sinogram residual records measurement consistency separately from MAE, PSNR, and SSIM.

To resume or repeat every post-inference stage in one command, set the roots created by the selected backend and run the same finaliser used by both pipelines:

export PADIS_RUN_ROOT="$LION_EXPERIMENTS_PATH/PaDIS"
export PADIS_TRAIN_ROOT=/path/to/completed/training-root
export PADIS_RECON_ROOT=/path/to/completed/reconstruction-root
export PADIS_TIMING_MODE=gcp             # use slurm for a Slurm array
export PADIS_TIMING_LOG_ROOT=/path/to/reconstruction/logs
conda run -n lion-dev bash \
  scripts/paper_scripts/PaDIS-Reproduction/pipeline/PaDIS_finalise_pipeline.sh

This creates any missing unconditional samples, verifies the matrix, derives timings, and regenerates every table and figure. Existing samples.pt files are reused.

Generate all publication tables using the standard experiment paths with:

python -u scripts/paper_scripts/PaDIS-Reproduction/reporting/PaDIS_make_tables.py

By default this reads $LION_EXPERIMENTS_PATH/PaDIS/final_real_runs/PaDIS-Reproduction-GCP_reconstruction/reconstruction_matrix_verification.csv and writes reconstruction_tables.tex plus seven decoded CSVs under $LION_EXPERIMENTS_PATH/PaDIS/paper_tables. Use --csv-path, --tex-path, and --csv-output-dir to override these locations. --allow-missing writes only tables represented by a partial smoke-run CSV; do not use it for final publication output.

The timing table is calculated from completed reconstruction progress logs; no timings are stored in the table generator. --timing-mode gcp (the default) and --timing-mode colab read the manual-runner log naming convention from the standard GCP reconstruction root. For a Slurm reconstruction array, use:

python -u scripts/paper_scripts/PaDIS-Reproduction/reporting/PaDIS_make_tables.py \
  --timing-mode slurm \
  --timing-log-root /path/to/slurm/output/files \
  --timing-jobs-json /path/to/reconstruction_matrix_jobs.json

The parser uses the final completed LIDC test run progress record, whose seconds-per-iteration value is already the mean wall time per reconstructed slice. Timings are grouped by implementation and reconstruction method; where multiple prior rows exist for a combination, their per-slice timings are averaged.

Generate all figures independently with:

conda run -n lion-dev python -u \
  scripts/paper_scripts/PaDIS-Reproduction/reporting/PaDIS_make_paper_figures.py \
  --reconstruction-root "$PADIS_RECON_ROOT" \
  --generation-root "$PADIS_RUN_ROOT/reconstruction_presets" \
  --output-folder "$PADIS_RUN_ROOT/paper_figures" \
  --figures all

The command fails on missing required panels by default. Use --allow-missing only for diagnostic partial output, not for a completed reproduction. Each figure is written as PNG and PDF and described in the output manifest.

Paper figures label the normalised-intensity colour scale as NI. CT panels use the tightest centre-symmetric crop that contains the target foreground; the default adds no border padding. This keeps anatomy centred consistently between methods without retaining avoidable outer background. Pass --body-bbox-padding N only when an explicit pixel border is required, or --no-body-crop to retain the complete reconstruction field of view. PNG and PDF exports additionally use a zero-padding page bounding box that is tight vertically and has equal minimal margins outside the leftmost and rightmost image edges. The NI label is placed close to its corresponding intensity scale while remaining legible. All vertical scale labels, including HU and NI, use one fixed horizontal coordinate so they align across figure rows. Display limits are the 15th and 95th percentiles of non-background target pixels (NI > 0.02) and are shared by every reconstruction and target in a row. HU panels use LION’s normalised-intensity-to-HU conversion, while metric calculation remains in NI. The 50 mm scale bars use the configured 300 mm field of view rather than patient-native DICOM spacing.