Contributing#

Contributions to SCICO are welcome. Before starting work, please contact the maintainers, either via email or the GitHub issue system, to discuss the relevance of your contribution and the most appropriate location within the existing package structure.

Installing a Development Version#

Fork both the scico and scico-data repositories, creating copies of these repositories in your own git account.
Make sure that you have Python 3.8 or later installed in order to create a conda virtual environment.

Clone your fork from the source repo.

git clone --recurse-submodules git@github.com:<username>/scico.git

Create a conda environment using Python 3.8 or later, e.g.:
```
conda create -n scico python=3.9
```
Activate the created conda virtual environment:
```
conda activate scico
```
Change directory to the root of the cloned repository:
```
cd scico
```

Add the scico repo as an upstream remote to sync your changes:

git remote add upstream https://www.github.com/lanl/scico

After adding the upstream, the recommended way to install SCICO and its dependencies is via pip:

pip install -r requirements.txt  # Installs basic requirements
pip install -r dev_requirements.txt  # Installs developer requirements
pip install -r docs/docs_requirements.txt # Installs documentation requirements
pip install -e .  # Installs SCICO from the current directory in editable mode

For installing dependencies related to the examples please see Usage Examples. Installing these are neccessary for the successfull running of the tests.

The SCICO project uses the black, isort and pylint code formatting utilities. It is important to set up a pre-commit hook to ensure that any modified code passes format check before it is committed to the development repo:
```
pre-commit install  # Sets up git pre-commit hooks
```
It is also recommended to pin the conda package version of black to the version number specified in dev_requirements.txt.
For testing see Tests.

Building Documentation#

Before building the documentation, one must install the documentation specific dependencies by running

pip install -r docs/docs_requirements.txt

Then, a local copy of the documentation can be built from the respository root directory by running

python setup.py build_sphinx

Alternatively, one can also build the documentation by running the following from the docs/ directory

make html

Contributing Code#

New features / bugs / documentation are always developed in separate branches.
Branches should be named in the form <username>/<brief-description>, where <brief-description> provides a highly condensed description of the purpose of the branch (e.g. address_todo), and may include an issue number if appropriate (e.g. fix_223).

A feature development workflow might look like this:

Follow the instructions in Installing a Development Version.

Sync with the upstream repository:

git pull --rebase origin main --recurse-submodules

Create a branch to develop from:

git checkout -b <username>/<brief-description>

Make your desired changes.
Run the test suite:
```
pytest
```
You can limit the test suite to a specific file for example:
```
pytest scico/test/test_blockarray.py
```
When you are finished making changes, create a new commit:
```
git add file1.py git add file2.py
git commit -m "A good commit message"
```
If you have added or modified an example script, see Usage Examples. If your contribution involves any significant new features or changes, add a corresponding entry to the change summary for the next release in the CHANGES.rst file.

Sync with the upstream repository:

git fetch upstream
git rebase upstream/main

Push your development upstream:

git push --set-upstream origin <username>/<brief-description>

Create a new pull request to the main branch; see the GitHub instructions.
The SCICO maintainers will review and merge your PR. The SCICO project recommends the squash and merge option for merging PRs.
Delete the branch after it has been merged.

Adding Data#

The following steps show how to add new data, new_data.npz, to the packaged data. We assume the scico repository has been cloned to scico/. Note that the data is located in the scico-data submodule, which is attached to the main scico repository via the directory scico/data (i.e. the data/ subdirectory of the repository root directory, not the scico/data subdirectory of the repository root directory). When adding new data, both the scico and scico-data repositories must be updated and kept in sync.

Create new branches in the main scico repository as well as in the submodule corresponding to the scico-data repository (which can be achieved by following the usual branch creation procedure after changing the current directory to scico/data).
Add the new_data.npz file to the appropriate subdirectory (creating a new one if necessary) of the scico/data directory.
Change directory to this directory (taken to be scico/data/flax for the purposes of this example) and add/commit the new data file:
```
cd scico/data/flax
git add new_data.npz
git commit -m "Add new data file"
```

Return to the scico repository root directory, add/commit the new data, and update submodule:

cd ../..  # pwd now `scico` repo root
git add data
git commit -m "Add data and update data module"

Push both repositories:

git submodule foreach --recursive 'git push' && git push

Type Checking#

All code is required to pass mypy type checking.

Install mypy:

conda install mypy

To run the type checker, execute the following from the scico repository root:

mypy --follow-imports=skip --ignore-missing-imports  --exclude "(numpy|test)" scico/

Tests#

All functions and classes should have corresponding pytest unit tests.

Running Tests#

To be able to run the tests, install pytest and, optionally, pytest-runner:

conda install pytest pytest-runner

The tests can be run by

pytest

or (if pytest-runner is installed)

python setup.py test

from the scico repository root directory. Tests can be run in an installed version of scico by

pytest --pyargs scico

When any significant changes are made to the test suite, the pytest-split test time database files in data/pytest should be updated using

pytest --store-durations --durations-path data/pytest/durations_ubuntu --level 2

(for Ubuntu CI), and

pytest --store-durations --durations-path data/pytest/durations_macos --level 1

(for MacOS CI). These updated files should be bzipped and committed into the scico-data repository, replacing the current versions.

Test Coverage#

Test coverage is a measure of the fraction of the package code that is exercised by the tests. While this should not be the primary criterion in designing tests, it is a useful tool for finding obvious areas of omission.

To be able to check test coverage, install coverage:

conda install coverage

A coverage report can be obtained by

coverage run
coverage report

Usage Examples#

New usage examples should adhere to the same general structure as the existing examples to ensure that the mechanism for automatically generating corresponding Jupyter notebooks functions correctly. In particular:

The initial lines of the script should consist of a comment block, followed by a blank line, followed by a multiline string with an RST heading on the first line, e.g.,

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# This file is part of the SCICO package. Details of the copyright
# and user license can be found in the 'LICENSE.txt' file distributed
# with the package.

"""
Script Title
============

Script description.
"""

The final line of the script is an input statement intended to avoid the script terminating immediately, thereby closing all figures:
```
input("\nWaiting for input to close figures and exit")
```
Citations are included using the standard Sphinx :cite:`cite-key` syntax, where cite-key is the key of an entry in docs/source/references.bib.
Cross-references to other components of the documentation are included using the syntax described in the nbsphinx documentation.
External links are included using Markdown syntax [link text](url).
When constructing a synthetic image/volume for use in the example, define a global variable N that controls the size of the problem, and where relevant, define a global variable maxiter that controls the number of iterations of optimization algorithms such as ADMM. Adhering to this convention allows the examples/scriptcheck.sh utility to automatically construct less computationally expensive versions of the example scripts for testing that they run without any errors.

Adding new examples#

The following steps show how to add a new example, new_example.py, to the packaged usage examples. We assume the scico repository has been cloned to scico/.

Note that the .py scripts are included in scico/examples/scripts, while the compiled Jupyter Notebooks are located in the scico-data submodule, which is symlinked to scico/data. When adding a new usage example, both the scico and scico-data repositories must be updated and kept in sync.

Warning

Ensure that all binary data (including raw data, images, .ipynb files) are added to scico-data, not the main scico repo.

Create new branches in the main scico repository as well as in the submodule corresponding to the scico-data repository (which can be achieved by following the usual branch creation procedure after changing the current directory to scico/data).
Add the new_example.py script to the scico/examples/scripts directory.
Add the basename of the script (i.e., without the pathname; in this case, new_example.py) to the appropriate section of examples/scripts/index.rst.
Convert your new example to a Jupyter notebook by changing directory to the scico/examples directory and following the instructions in scico/examples/README.rst.

Change directory to the data directory and add/commit the new Jupyter Notebook:

cd scico/data
git add notebooks/new_example.ipynb
git commit -m "Add new usage example"

Return to the main scico repository root directory, ensure the main branch is checked out, add/commit the new script and updated submodule:

cd ..  # pwd now `scico` repo root
git add data
git add examples/scripts/new_filename.py
git commit -m "Add usage example and update data module"

Push both repositories:

git submodule foreach --recursive 'git push' && git push