Skip to content

dev-python.md

Latest commit

 

History

History
335 lines (230 loc) · 10.4 KB

dev-python.md

File metadata and controls

335 lines (230 loc) · 10.4 KB
Raw

Developing Python components

Python style

Python code should be formatted to conform as far as possible to the PEP 8 -- Style Guide for Python Code.

The tools pylint and pycodestyle can help by highlighting deviations from the coding style. See Style checking Python code below.

See also the riboviz Style guide.

File names

Python file names must be in snake-case i.e., lower-case and delimited by underscores, not hyphens. For example, sample_sheets.py, get_cds_codons.py.

Comments

Python code should be commented using doc-strings. The doc-strings should include Sphinx-compliant reStructuredText markup. Examples are given below.

Modules should have a doc-string descring the module. For example:

"""
Workflow configuration parameter names.
"""

Constants and module-level variable should have doc-strings briefly describing what they are. For example:

INPUT_DIR = "dir_in"
""" Input directory. """

Functions should have doc-strings describing what they do and their parameters, return values (if applicable) and any exceptions (if applicable). Types of parameters and return values should also be documented. For example:

def load_sample_sheet(file_name, delimiter="\t", comment="#"):
    """
    Load a sample sheet from a file. The sample sheet is assumed to
    have a header with column names ``SampleID`` and ``TagRead``.

    :param file_name: File name
    :type file_name: str or unicode
    :param delimiter: Delimiter
    :type delimiter: str or unicode
    :param comment: Comment prefix
    :type comment: str or unicode
    :return: Sample sheet
    :rtype: pandas.core.frame.DataFrame
    :raise FileNotFoundError: If the file cannot be found or is \
    not a file
    :raise AssertionError: If there is no header with ``SampleID`` \
    and ``TagRead`` columns.
    """

To use fixed-width (teletype-style) text use `` (see the markup on SampleID and TagRead above).

For references to other Python modules, functions or constants use Sphinx's :py: roles. For example:

  • Cross-reference to a module (:py:mod:):
Count reads using :py:mod:`riboviz.tools.count_reads`.
  • Cross-reference to a constant (:py:constant:):
... also matching :py:const:`riboviz.workflow_files.UMI_EXTRACT_FQ` ...
  • Cross-reference to a function (:py:func:):
See :py:func:`riboviz.count_reads.count_reads` for information ...

Note that if the cross-reference is to an entity in the same module then only the local name of the entity needs to be specified. For example:

See :py:func:`count_reads` for information ...

For examples and further information, see:

See below for information on Creating Sphinx documentation from Python comments.

Style checking Python code

pylint and pycodestyle can be run on individual files, groups of files or every file in a directory and its subdirectories. For example:

$ pylint riboviz/check_fasta_gff.py
$ pycodestyle riboviz/check_fasta_gff.py
$ pylint riboviz/
$ pycodestyle riboviz/

Python command-line tools

Python command-line tools must be placed within riboviz/tools/ i.e. within the riboviz.tools module.

Command-line parsing must be implemented using argparse.

riboviz.tools scripts must not contain any functionality beyond defining and parsing command-line parameters then invoking the functionality of the script. The functionality of the script itself should be placed in module(s) under riboviz.

riboviz.tools scripts must contain an entry point defined as follows:

if __name__ == "__main__":
    main()

where the main() function is a function in the script responsible for parsingthe command-line parameters and invoking the functionality of the script itself, as defined in other riboviz module(s). See the existing riboviz.tools implementations for the pattern to use.

Defining command-line parameters

The dest parameter of ArgumentParser.add_argument can be use to explicitly define the Python variables into which a command-line parameter is to be placed. For example:

parser = argparse.ArgumentParser(description="Program")
parser.add_argument("-o", "--output-dir", dest="output_dir", nargs='?',
                    help="Output directory")
options = parser.parse_args()
output_dir = options.output_dir

Adding, renaming, and removing command-line tools

If adding, renaming or removing command-line tools:

  • Update 'Python command-line tools' table in docs/user/command-line-tools.md.

Comments and documentation

Creating Sphinx documentation from Python comments

Create Sphinx pages to reference source code:

$ sphinx-apidoc -f -o py-docs/ riboviz

Create HTML documentation

$ cd py-docs
$ make html

If there are errors in the comment formatting then these will be displayed.

Open py-docs/_build/html/index.html in a browser.

Manually check the rendered comments to see that they have no errors that cannot be caught during the build. These include whether or not you have:

  • Marked up ``teletype font`` text correctly.
  • Defined cross-references to modules, constants, and functions correctly.

Creating template Sphinx documentation files

The template Sphinx documentation files were originally created as follows:

$ sphinx-quickstart py-docs
> Separate source and build directories (y/n) [n]: y
> Project name: riboviz
> Author name(s): The University of Edinburgh; Rutgers University; University of California, Berkeley
> Project release []: 
> Project language [en]:

Edit py-docs/conf.py:

  • Uncomment:
# import os
# import sys
  • Replace:
# sys.path.insert(0, os.path.abspath('.'))
  • with:
sys.path.insert(0, os.path.abspath('..'))
  • Update:
extensions = [
]
  • to:
extensions = [
    'sphinx.ext.autodoc'
]

Edit py-docs/index.rst and replace content with:

riboviz code documentation
==========================

.. toctree::
   :maxdepth: 1
   :caption: Code documentation

   modules

Indices and tables:

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

Testing

Run Python tests and workflow tests

Run:

$ pytest --ignore-glob="*integration*"

Troubleshooting: PendingDeprecationWarning

PendingDeprecationWarning warnings can be ignored.

Useful pytest flags

  • -s: disable output capture so, for example, print messages are shown.
  • -v: verbose mode, displays names of test functions run.
  • -k: run a specific test function.
  • --cov-config=.coveragerc --cov-report term-missing --cov=riboviz: create a test coverage report which includes the line numbers of statements that were not executed.

Information on pytest fixtures and parameters

The riboviz integration and Python unit tests make extensive use of pytest fixtures and parameterised tests. For more information on pytest fixtures and parameters see:

Miscellaneous

Clone conda environments

If you want to install a new Python package to explore it without changing your current conda environment you can first clone your conda environment, then install the package into the clone. For example:

$ conda create --name riboviz-test-install --clone riboviz
$ conda activate riboviz-test-install

To remove an environment when you are finished with it you can run, for example:

$ conda deactivate riboviz-test-install
$ conda env remove --name riboviz-test-install

For more information, see conda's Managing environments.

Understanding YAML NULL and Python None

If a parameter in a YAML file has value null, NULL or no value at all then, after reading the file into Python (using the yaml library), it will have value None. For example, given a YAML file with:

a: null
b: NULL
c:

Loading this with

with open("config.yml") as f:
    config = yaml.load(f, Loader=yaml.SafeLoader)

would result in the following all having value None:

config['a']
config['b']
config['c']