scico.flax.examples.examples#

Generation and loading of data used in Flax example scripts.

Functions

`check_img_data_requirements`(train_nimg, ...)	Check data loaded with respect to data requirements.
`get_cache_path`([cache_path])	Get input/output SCICO cache path.
`load_ct_data`(train_nimg, test_nimg, size, nproj)	Load or generate CT data.
`load_foam1_blur_data`(train_nimg, test_nimg, ...)	Load or generate blurred data based on xdesign foam structures.
`load_image_data`(train_nimg, test_nimg, size, ...)	Load or load and preprocess image data.
`print_data_range`(idstring, data)	Display min and max values of given data array.
`print_data_size`(idstring, size)	Display integer given.
`print_data_warning`(idstring, requested, ...)	Display warning related to data size demands not satisfied.
`print_info`(iomode, path_display, train_in, ...)	Display information related to data input/output.
`print_input_path`(path_display)	Display path from where data is being loaded.
`print_output_path`(path_display)	Display path where data is being stored.
`runtime_error_array`(type, idstring, maxdiff)	Raise run time error related to unsatisfied array parameter request.
`runtime_error_scalar`(type, idstring, ...)	Raise run time error related to unsatisfied scalar parameter request.

scico.flax.examples.examples.get_cache_path(cache_path=None)[source]#

Get input/output SCICO cache path.

Parameters:: cache_path (Optional[str]) – Given cache path. If None SCICO default cache path is constructed.
Return type:: Tuple[str, str]
Returns:: The cache path and a display string with private user path information stripped.

scico.flax.examples.examples.load_ct_data(train_nimg, test_nimg, size, nproj, cache_path=None, verbose=False, prefer_ray=True)[source]#

Load or generate CT data.

Load or generate CT data for training of machine learning network models. If cached file exists and enough data of the requested size is available, data is loaded and returned.

If either size or nproj requested does not match the data read from the cached file, a RunTimeError is generated.

If no cached file is found or not enough data is contained in the file a new data set is generated and stored in cache_path. The data is stored in .npz format for convenient access via numpy.load. The data is saved in two distinct files: ct_foam2_train.npz and ct_foam2_test.npz to keep separated training and testing partitions.

Parameters:

train_nimg (int) – Number of images required for training.
test_nimg (int) – Number of images required for testing.
size (int) – Size of reconstruction images.
nproj (int) – Number of CT views.
cache_path (Optional[str]) – Directory in which generated data is saved. Default: None.
verbose (bool) – Flag indicating whether to print status messages. Default: False.
prefer_ray (bool) – Use ray for distributed processing if available. Default: True.

Return type:

Tuple[CTDataSetDict, ...]

Returns:

tuple –

A tuple (trdt, ttdt) containing:

trdt(Dictionary): Collection of images (key img),
sinograms (key sino) and filtered back projections (key fbp) for training.

ttdt(Dictionary): Collection of images (key img),
sinograms (key sino) and filtered back projections (key fbp) for testing.

scico.flax.examples.examples.load_foam1_blur_data(train_nimg, test_nimg, size, blur_kernel, noise_sigma, cache_path=None, verbose=False, prefer_ray=True)[source]#

Load or generate blurred data based on xdesign foam structures.

Load or generate blurred data for training of machine learning network models. If cached file exists and enough data of the requested size is available, data is loaded and returned.

If size, blur_kernel or noise_sigma requested do not match the data read from the cached file, a RunTimeError is generated.

If no cached file is found or not enough data is contained in the file a new data set is generated and stored in cache_path. The data is stored in .npz format for convenient access via numpy.load. The data is saved in two distinct files: dcnv_foam1_train.npz and dcnv_foam1_test.npz to keep separated training and testing partitions.

Parameters:

train_nimg (int) – Number of images required for training.
test_nimg (int) – Number of images required for testing.
size (int) – Size of reconstruction images.
blur_kernel (Array) – Kernel for blurring the generated images.
noise_sigma (float) – Level of additive Gaussian noise to apply.
cache_path (Optional[str]) – Directory in which generated data is saved. Default: None.
verbose (bool) – Flag indicating whether to print status messages. Default: False.
prefer_ray (bool) – Use ray for distributed processing if available. Default: True.

Return type:

Tuple[DataSetDict, ...]

Returns:

tuple –

A tuple (train_ds, test_ds) containing:

train_dsDictionary of training data (includes images
and labels).

test_dsDictionary of testing data (includes images
and labels).

scico.flax.examples.examples.load_image_data(train_nimg, test_nimg, size, gray_flag, data_mode='dn', cache_path=None, verbose=False, noise_level=0.1, noise_range=False, transf=None, stride=None, augment=False)[source]#

Load or load and preprocess image data.

Load or load and preprocess image data for training of neural network models. The original source is the BSDS500 data from the Berkeley Segmentation Dataset and Benchmark project. Depending on the intended applications, different preprocessings can be performed to the source data.

If a cached file exists, and enough images were sampled, data is loaded and returned.

If either size or type of data (gray scale or color) requested does not match the data read from the cached file, a RunTimeError is generated. In contrast, there is no checking for the specific contamination (i.e. noise level, blur kernel, etc.).

If no cached file is found or not enough images were sampled and stored in the file, a new data set is generated and stored in cache_path. The data is stored in .npz format for convenient access via numpy.load. The data is saved in two distinct files: *_bsds_train.npz and *_bsds_test.npz to keep separated training and testing partitions. The * stands for dn if denoising problem or dcnv if deconvolution problem. Other types of pre-processings may be specified via the transf operator.

Parameters:

train_nimg (int) – Number of images required for sampling training data.
test_nimg (int) – Number of images required for sampling testing data.
size (int) – Size of reconstruction images.
gray_flag (bool) – Flag to indicate if gray scale images or color images. When True gray scale images are used.
data_mode (str) – Type of image problem. Options are: dn for denosing, dcnv for deconvolution.
cache_path (Optional[str]) – Directory in which processed data is saved. Default: None.
verbose (bool) – Flag indicating whether to print status messages. Default: False.
noise_level (float) – Standard deviation of the Gaussian noise.
noise_range (bool) – Flag to indicate if a fixed or a random standard deviation must be used. Default: False i.e. fixed standard deviation given by noise_level.
transf (Optional[Callable]) – Operator for blurring or other non-trivial transformations. Should be able to handle batched (NHWC) data. Default: None.
stride (Optional[int]) – Stride between patch origins (indexed from left-top corner). Default: 0 (i.e. no stride, only one patch per image).
augment (bool) – Augment training data set by flip and 90 degrees rotation. Default: False (i.e. no augmentation).

Return type:

Tuple[DataSetDict, ...]

Returns:

tuple –

A tuple (train_ds, test_ds) containing:

train_ds(DataSetDict): Dictionary of training data
(includes images and labels).

test_ds(DataSetDict): Dictionary of testing data
(includes images and labels).

scico.flax.examples.examples.check_img_data_requirements(train_nimg, test_nimg, size, gray_flag, train_in_shp, test_in_shp, train_nimg_avail, test_nimg_avail, verbose)[source]#

Check data loaded with respect to data requirements.

Parameters:

train_nimg (int) – Number of images required for training data.
test_nimg (int) – Number of images required for testing data.
size (int) – Size of images requested.
gray_flag (bool) – Flag to indicate if gray scale images or color images are requested. When True gray scale images are used, therefore, one channel is expected.
train_in_shp (Tuple[int, ...]) – Shape of images/patches loaded as training data.
test_in_shp (Tuple[int, ...]) – Shape of images/patches loaded as testing data.
train_nimg_avail (int) – Number of images available in loaded training image data.
test_nimg_avail (int) – Number of images available in loaded testing image data.
verbose (bool) – Flag indicating whether to print status messages.

Return type:

bool

Returns:

True if the loaded image data satifies requirements of size, number of samples and number of channels and False otherwise.

scico.flax.examples.examples.print_input_path(path_display)[source]#

Display path from where data is being loaded.

Parameters:: path_display (str) – Path for loading data.

scico.flax.examples.examples.print_output_path(path_display)[source]#

Display path where data is being stored.

Parameters:: path_display (str) – Path for storing data.

scico.flax.examples.examples.print_data_range(idstring, data)[source]#

Display min and max values of given data array.

Parameters:

idstring (str) – Data descriptive string.
data (Array) – Array to compute min and max.

scico.flax.examples.examples.print_data_size(idstring, size)[source]#

Display integer given.

Parameters:

idstring (str) – Data descriptive string.
size (int) – Integer representing size of a set.

scico.flax.examples.examples.print_info(iomode, path_display, train_in, train_out, test_size)[source]#

Display information related to data input/output.

Parameters:

iomode (str) – Identification of input (load) or ouput (save) operation.
path_display (str) – Input or output path.
train_in (Array) – Input features in training set.
train_out (Array) – Outputs in training set.
test_size (int) – Size of testing set.

scico.flax.examples.examples.print_data_warning(idstring, requested, available)[source]#

Display warning related to data size demands not satisfied.

Parameters:

idstring (str) – Data descriptive string.
requested (int) – Size of data set requested.
available (int) – Size of data set available.

scico.flax.examples.examples.runtime_error_scalar(type, idstring, requested, available)[source]#

Raise run time error related to unsatisfied scalar parameter request.

Raise run time error related to scalar parameter request not satisfied in available data.

Parameters:

type (str) – Type of parameter in the request.
idstring (str) – Data descriptive string.
requested (Union[int, float]) – Parameter value requested.
available (Union[int, float]) – Parameter value available in data.

scico.flax.examples.examples.runtime_error_array(type, idstring, maxdiff)[source]#

Raise run time error related to unsatisfied array parameter request.

Raise run time error related to array parameter request not satisfied in available data.

Parameters:

type (str) – Type of parameter in the request.
idstring (str) – Data descriptive string.
maxdiff (float) – Maximum error between requested and available array entries.