Evaluation API

Metrics for reconstruction quality, generative quality, memorization probing, and bootstrap confidence intervals. The generative-quality entry points (FID, precision/recall, NLL) are convenient one-call functions, with stateful calculator classes for streaming features over many batches.

Generative Quality

medlatents.evaluation.calculate_fid(real_features, generated_features, eps=1e-06)

Calculate Fréchet Inception Distance (FID) between real and generated features.

FID = ||mu_real - mu_gen||^2 + Tr(Sigma_real + Sigma_gen - 2*sqrt(Sigma_real @ Sigma_gen))

Parameters:

• real_features (Float[Tensor, 'n_real features']) – Features from real images [N_real, D]

• generated_features (Float[Tensor, 'n_gen features']) – Features from generated images [N_gen, D]

• eps (float, default: 1e-06) – Small constant for numerical stability

Return type:

float

Returns:

FID score (lower is better)

medlatents.evaluation.calculate_precision_recall(real_features, generated_features, k=3)

Calculate Precision and Recall for generative models.

Precision: What fraction of generated samples are realistic? Recall: What fraction of real samples are covered by the generator?

Based on “Improved Precision and Recall Metric for Assessing Generative Models” (Kynkäänniemi et al., 2019)

Parameters:

• real_features (Float[Tensor, 'n_real features']) – Features from real images [N_real, D]

• generated_features (Float[Tensor, 'n_gen features']) – Features from generated images [N_gen, D]

• k (int, default: 3) – Number of nearest neighbors

Return type:

tuple[float, float]

Returns:

(precision, recall) tuple, both in [0, 1]

medlatents.evaluation.calculate_nll(model, tokens, reduction='mean')

Calculate Negative Log-Likelihood for discrete latent models.

Parameters:

• model (Module) – Model with forward pass returning logits

• tokens (Int[Tensor, 'batch seq']) – Token sequences [batch, seq_len]

• reduction (Literal['mean', 'sum', 'none'], default: 'mean') – How to reduce across batch/sequence

Return type:

float | Tensor

Returns:

NLL value (lower is better)

class medlatents.evaluation.FIDCalculator(extractor_type='radimagenet', device='cuda', **extractor_kwargs)

Bases: object

FID calculator with feature extraction.

Handles feature extraction and FID calculation in one class.

Example

>>> calculator = FIDCalculator(extractor_type='radimagenet')
>>> # Extract features from real images
>>> for batch in real_loader:
...     calculator.update_real(batch)
>>> # Extract features from generated images
>>> for batch in gen_loader:
...     calculator.update_generated(batch)
>>> fid = calculator.compute()

Parameters:

• extractor_type (Literal['inception', 'radimagenet'], default: 'radimagenet')

• device (str, default: 'cuda')

__init__(extractor_type='radimagenet', device='cuda', **extractor_kwargs)

Initialize FID calculator.

Parameters:

• extractor_type (Literal['inception', 'radimagenet'], default: 'radimagenet') – Type of feature extractor

• device (str, default: 'cuda') – Device to run on

• **extractor_kwargs – Additional kwargs for feature extractor

update_real(images)

Extract and store features from real images.

Parameters:

images (Float[Tensor, 'batch channel height width']) – Real images [B, C, H, W]

Return type:

None

update_generated(images)

Extract and store features from generated images.

Parameters:

images (Float[Tensor, 'batch channel height width']) – Generated images [B, C, H, W]

Return type:

None

compute()

Compute FID from accumulated features.

Return type:

float

Returns:

FID score

reset()

Clear accumulated features.

Return type:

None

class medlatents.evaluation.PrecisionRecallCalculator(k=3, extractor_type='radimagenet', device='cuda', **extractor_kwargs)

Bases: object

Precision/Recall calculator with feature extraction.

Example

>>> calculator = PrecisionRecallCalculator(k=5)
>>> for batch in real_loader:
...     calculator.update_real(batch)
>>> for batch in gen_loader:
...     calculator.update_generated(batch)
>>> precision, recall = calculator.compute()

Parameters:

• k (int, default: 3)

• extractor_type (Literal['inception', 'radimagenet'], default: 'radimagenet')

• device (str, default: 'cuda')

__init__(k=3, extractor_type='radimagenet', device='cuda', **extractor_kwargs)

Initialize Precision/Recall calculator.

Parameters:

• k (int, default: 3) – Number of nearest neighbors

• extractor_type (Literal['inception', 'radimagenet'], default: 'radimagenet') – Type of feature extractor

• device (str, default: 'cuda') – Device to run on

• **extractor_kwargs – Additional kwargs for feature extractor

update_real(images)

Extract and store features from real images.

Parameters:

images (Float[Tensor, 'batch channel height width'])

Return type:

None

update_generated(images)

Extract and store features from generated images.

Parameters:

images (Float[Tensor, 'batch channel height width'])

Return type:

None

compute()

Compute Precision and Recall from accumulated features.

Return type:

tuple[float, float]

Returns:

(precision, recall) tuple

reset()

Clear accumulated features.

Return type:

None

FID Bootstrap and Feature Extraction

medlatents.evaluation.fid_from_features(real_feats, gen_feats, ridge=1e-06)

Closed-form Fréchet Inception Distance on already-extracted features.

Uses the symmetric form tr( (sigma_r sigma_g)^{1/2} ) = sum sqrt eig( sigma_r^{1/2} sigma_g sigma_r^{1/2} ) computed via two eigh calls. ~50× faster than scipy.linalg.sqrtm on the 192-dim covariances used for FID-192 at low resolution, and numerically more stable (eigenvalues are non-negative by construction).

Parameters:

• real_feats (ndarray)

• gen_feats (ndarray)

• ridge (float, default: 1e-06)

Return type:

float

medlatents.evaluation.bootstrap_fid_real_vs_gen(real_feats, gen_feats, n=None, B=200, seed=42, verbose=False)

95% CI for FID(real, gen) via paired bootstrap of feature indices.

For B reps, sample n indices with replacement from each of real_feats and gen_feats, recompute FID. The resulting distribution captures the estimator uncertainty at the given sample size, given a fixed underlying generator. It does not capture train-time stochasticity (which requires re-sampling the model itself). Defaults n to min(len(real_feats), len(gen_feats)).

Parameters:

• real_feats (ndarray)

• gen_feats (ndarray)

• n (int | None, default: None)

• B (int, default: 200)

• seed (int, default: 42)

• verbose (bool, default: False)

Return type:

dict

medlatents.evaluation.bootstrap_fid_noise_floor(features, n_per_half=(1000, 2500, 5000, 10000), B=200, seed=42, verbose=False)

Estimate FID’s estimator-variance noise floor via random splits.

For each n in n_per_half, draw B random splits of features into two disjoint halves of size n each, compute FID(half_A, half_B), and return summary statistics. Because both halves come from the same underlying distribution, the resulting FID distribution characterises the minimum detectable FID gap at sample size n — gaps below this floor are within metric noise.

Parameters:

• features (ndarray)

• n_per_half (Sequence[int], default: (1000, 2500, 5000, 10000))

• B (int, default: 200)

• seed (int, default: 42)

• verbose (bool, default: False)

Return type:

dict[str, dict]

Reconstruction Metrics

medlatents.evaluation.calculate_psnr(reference, prediction, data_range=None)

Compute peak signal-to-noise ratio in dB.

Parameters:

• reference (Tensor)

• prediction (Tensor)

• data_range (float | None, default: None)

Return type:

float

medlatents.evaluation.calculate_ssim(reference, prediction, *, window_size=11, data_range=None, k1=0.01, k2=0.03)

Compute structural similarity index (SSIM).

Parameters:

• reference (Tensor)

• prediction (Tensor)

• window_size (int, default: 11)

• data_range (float | None, default: None)

• k1 (float, default: 0.01)

• k2 (float, default: 0.03)

Return type:

float

Memorization Probing

class medlatents.evaluation.InceptionPool3FeatureExtractor(device='cpu', batch_size=64)

Bases: object

Extract 192-d Inception V3 pool3 features (matches FID-192).

The backbone is the same one used by torchmetrics.image.fid with feature=192; sharing the feature space ensures the memorization metric and the reported FID values live on the same Inception manifold.

Parameters:

• device (str | device, default: 'cpu')

• batch_size (int, default: 64)

__init__(device='cpu', batch_size=64)

Parameters:

• device (str | device, default: 'cpu')

• batch_size (int, default: 64)

extract(images)

Return (N, 192) float features for the given images.

images may be (N, 1, H, W) or (N, 3, H, W), dtype uint8 or float in [0,1]. Grayscale inputs are tiled to RGB. The inception backbone in torchmetrics.image.fid expects uint8 inputs (the normalisation is handled internally), so floats are rescaled to [0, 255] and cast.

Parameters:

images (Tensor)

Return type:

Tensor

medlatents.evaluation.memorization_metrics(gen_features, train_features, device='cpu')

Compute the memorization-ratio and AuthPct scalars.

Returns a dict with keys:

• memorization_ratio: median(d_g->t) / median(d_t->t)

• auth_pct: fraction of generated samples for which d(gen, NN_train) < d(NN_train, NN2_train)

• median_d_gen_to_train: median nearest-neighbor distance from generated to training

• median_d_train_to_train: median nearest-neighbor distance within training (excluding self)

• mean_d_gen_to_train: mean version of the above

• mean_d_train_to_train: ditto

• n_gen, n_train: sample counts

Parameters:

• gen_features (Tensor)

• train_features (Tensor)

• device (str | device, default: 'cpu')

Return type:

dict[str, float]

medlatents.evaluation.memorization_pairs(gen_features, train_features, top_k=8, device='cpu')

Return the top_k (gen_idx, train_idx, cosine_distance) pairs with the smallest gen→train distances — the most likely memorisation suspects to display in a side-by-side gallery.

Parameters:

• gen_features (Tensor)

• train_features (Tensor)

• top_k (int, default: 8)

• device (str | device, default: 'cpu')

Return type:

list[tuple[int, int, float]]

medlatents.evaluation.nearest_neighbor_gallery_pairs(generated_images, train_images, pairs, max_pairs=None)

Select image tensors for generated/training nearest-neighbor galleries.

Parameters:

• generated_images (Tensor)

• train_images (Tensor)

• pairs (Sequence[tuple[int, int, float] | dict[str, float | int]])

• max_pairs (int | None, default: None)

Return type:

list[dict[str, Tensor | float | int]]

medlatents.evaluation.cosine_distance_matrix(query, reference, device='cpu', block=1024)

Compute (N_q, N_r) cosine-distance matrix in blocks.

Returns a CPU tensor; switches to GPU for the matmul if device is cuda. Block size keeps memory bounded for large reference sets.

Parameters:

• query (Tensor)

• reference (Tensor)

• device (str | device, default: 'cpu')

• block (int, default: 1024)

Return type:

Tensor

Classifier Utility

medlatents.evaluation.build_grayscale_resnet18(num_classes)

ResNet-18 with single-channel conv1 + multi-label classification head.

Parameters:

num_classes (int)

Return type:

Module

medlatents.evaluation.train_classifier(model, train_loader, val_loader, epochs, device, lr=0.001, weight_decay=0.0001, log_fn=<built-in function print>)

Train model with BCE-with-logits, AdamW, per-epoch val AUC.

Returns {"best_val_mean_auc", "best_state_dict", "history": [...]}. The model is mutated in place; restore best state via model.load_state_dict(returned["best_state_dict"]) before evaluation.

Parameters:

• model (Module)

• train_loader (DataLoader)

• val_loader (DataLoader)

• epochs (int)

• device (str | device)

• lr (float, default: 0.001)

• weight_decay (float, default: 0.0001)

• log_fn (Callable[[str], None], default: <built-in function print>)

Return type:

dict

medlatents.evaluation.evaluate_auc(model, loader, device, class_names=None)

Per-class + mean ROC-AUC over a DataLoader (multi-label).

Classes for which the loader contains all-positive or all-negative labels yield NaN AUC and are excluded from the mean.

Parameters:

• model (Module)

• loader (DataLoader)

• device (str | device)

• class_names (list[str] | None, default: None)

Return type:

dict

medlatents.evaluation.classifier_fid(model, real_images, gen_images, device, batch_size=256, ridge=1e-06)

FID in a trained classifier’s penultimate feature space.

Parameters:

• model (Module)

• real_images (Tensor)

• gen_images (Tensor)

• device (str | device)

• batch_size (int, default: 256)

• ridge (float, default: 1e-06)

Return type:

float