I/O & Preprocessing

Loading tokenizers

medtokenizers.load_tokenizer(model_name_or_path, device=None, **kwargs)

Load tokenizer from local path or HuggingFace Hub.

Attempts to load as ContinuousTokenizer, DiscreteTokenizer, TiTokTokenizer, and RAETokenizer (in that order). Raises detailed error if all fail.

Parameters:

• model_name_or_path (Union[str, Path]) – Local path or HuggingFace Hub repo ID

• device (Optional[str], default: None) – Device to load model to (default: auto-detect cuda/cpu)

• **kwargs – Additional arguments passed to from_pretrained

Return type:

Union[ContinuousTokenizer, DiscreteTokenizer, RAETokenizer, TiTokTokenizer]

Returns:

Loaded tokenizer model in inference mode

Raises:

ValueError – If model cannot be loaded as either tokenizer type. Error message includes details from both loading attempts.

medtokenizers.get_config(reload=False)

Get global configuration instance.

Parameters:

reload (bool, default: False) – Force reload from environment

Return type:

Config

Returns:

Config instance

Example

>>> config = get_config()
>>> print(config.wandb_project)
medtokenizers

Saving & loading tokens / latents

medtokenizers.save_indices(indices, save_path, dtype='int16')

Save discrete tokenizer indices to disk with efficient storage.

Uses int16 by default which supports vocabulary sizes up to 32,767. For larger vocabularies (unlikely), use int32.

Parameters:

• indices (Union[Tensor, ndarray]) – Discrete indices from tokenizer.encode() or tokenizer.tokenize()

• save_path (Union[str, Path]) – Path to save file (will add .npz suffix)

• dtype (Literal['int16', 'int32'], default: 'int16') – Storage dtype - “int16” (default, 2 bytes) or “int32” (4 bytes)

Raises:

• TypeError – If indices are not integer tensors/arrays

• ValueError – If indices are negative or exceed dtype range

Return type:

None

Example

>>> indices, _, _ = tokenizer.encode(images)
>>> save_indices(indices, "tokens/batch_001")

medtokenizers.load_indices(load_path, device=None, dtype=torch.int64)

Load discrete tokenizer indices from disk.

Parameters:

• load_path (Union[str, Path]) – Path to .npz file containing indices

• device (Optional[str], default: None) – Device to load tensor to (default: CPU)

• dtype (dtype, default: torch.int64) – Output dtype for PyTorch tensor (default: int64 for compatibility)

Return type:

Tensor

Returns:

Indices tensor ready for tokenizer.decode()

Example

>>> indices = load_indices("tokens/batch_001.npz", device="cuda")
>>> images = tokenizer.decode(indices)

medtokenizers.save_latents(latents, save_path, dtype='float16')

Save continuous tokenizer latents to disk with efficient storage.

Uses float16 by default which is sufficient for most latent diffusion applications. Use float32 if full precision is required.

Parameters:

• latents (Union[Tensor, ndarray]) – Continuous latents from tokenizer.encode() or tokenizer.tokenize()

• save_path (Union[str, Path]) – Path to save file (will add .npz suffix)

• dtype (Literal['float16', 'float32'], default: 'float16') – Storage dtype - “float16” (default, 2 bytes) or “float32” (4 bytes)

Return type:

None

Example

>>> latents, _ = tokenizer.encode(images)
>>> save_latents(latents, "latents/batch_001")

medtokenizers.load_latents(load_path, device=None, dtype=torch.float32)

Load continuous tokenizer latents from disk.

Parameters:

• load_path (Union[str, Path]) – Path to .npz file containing latents

• device (Optional[str], default: None) – Device to load tensor to (default: CPU)

• dtype (dtype, default: torch.float32) – Output dtype for PyTorch tensor (default: float32)

Return type:

Tensor

Returns:

Latents tensor ready for tokenizer.decode()

Example

>>> latents = load_latents("latents/batch_001.npz", device="cuda")
>>> images = tokenizer.decode(latents)

Preprocessing

medtokenizers.preprocess_for_maisi(nifti_path, target_spacing=(1.0, 1.0, 1.0), percentile_lower=0.0, percentile_upper=99.5, divisible_k=4)

NVIDIA MAISI-style preprocessing pipeline.

Uses medrs for efficient NIfTI loading.

Parameters:

• nifti_path (Union[str, Path]) – Path to input NIfTI image

• target_spacing (tuple[float, float, float], default: (1.0, 1.0, 1.0)) – Target voxel spacing in mm (default: 1mm^3)

• percentile_lower (float, default: 0.0) – Lower percentile for normalization (default: 0.0)

• percentile_upper (float, default: 99.5) – Upper percentile for normalization (default: 99.5)

• divisible_k (int, default: 4) – Pad to be divisible by k (default: 4)

Return type:

tuple[Tensor, dict]

Returns:

Preprocessed volume tensor (1, 1, D, H, W), metadata dict

medtokenizers.postprocess_from_maisi(reconstruction, metadata, denormalize=True)

Reverse MAISI preprocessing to get back to original space.

Parameters:

• reconstruction (Tensor) – Reconstructed volume tensor (1, 1, D, H, W)

• metadata (dict) – Metadata dict from preprocess_for_maisi

• denormalize (bool, default: True) – Whether to denormalize intensities (default: True)

Return type:

ndarray

Returns:

Volume in original space as numpy array

medtokenizers.percentile_normalize(volume, lower=0.0, upper=99.5, b_min=0.0, b_max=1.0, clip=False)

Normalize intensity to percentile range.

Parameters:

• volume (ndarray) – Input volume

• lower (float, default: 0.0) – Lower percentile (default: 0.0)

• upper (float, default: 99.5) – Upper percentile (default: 99.5)

• b_min (float, default: 0.0) – Output minimum value (default: 0.0)

• b_max (float, default: 1.0) – Output maximum value (default: 1.0)

• clip (bool, default: False) – Whether to clip values outside range (default: False)

Return type:

tuple[ndarray, float, float]

Returns:

Normalized volume, lower_percentile_value, upper_percentile_value

medtokenizers.resample_to_spacing(volume, src_spacing, tgt_spacing=(1.0, 1.0, 1.0), mode='trilinear')

Resample volume to target spacing.

Parameters:

• volume (Tensor) – Input volume tensor (B, C, D, H, W)

• src_spacing (tuple[float, float, float]) – Source voxel spacing (z, y, x)

• tgt_spacing (tuple[float, float, float], default: (1.0, 1.0, 1.0)) – Target voxel spacing (z, y, x)

• mode (str, default: 'trilinear') – Interpolation mode (default: “trilinear”)

Return type:

Tensor

Returns:

Resampled volume tensor

medtokenizers.pad_divisible(volume, k=4)

Pad volume to be divisible by k.

Parameters:

• volume (Tensor) – Input volume tensor (B, C, D, H, W)

• k (int, default: 4) – Divisibility factor (default: 4)

Return type:

tuple[Tensor, tuple[int, ...]]

Returns:

Padded volume, padding amounts (d_front, d_back, h_front, h_back, w_front, w_back)

medtokenizers.unpad(volume, padding)

Remove padding from volume.

Parameters:

• volume (Tensor) – Padded volume tensor (B, C, D, H, W)

• padding (tuple[int, ...]) – Padding amounts (d_front, d_back, h_front, h_back, w_front, w_back)

Return type:

Tensor

Returns:

Unpadded volume tensor