RSE Skills

(in Python)

Jack Atkinson

ICCS RSE Team
University of Cambridge

Amy Pike

ICCS RSE Team
University of Cambridge

Marion Weinzierl

ICCS RSE Team
University of Cambridge

2024-07-11

Precursors

Slides and Materials

All materials are available at:

Licensing

Except where otherwise noted, these presentation materials are licensed under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) License.

image/svg+xml

Vectors and icons by SVG Repo used under CC0(1.0)

Precursors

  • Be nice (Python code of conduct)
  • Ask questions whenever they arise.
    • Someone else is probably wondering the same thing.
  • We will make mistakes.
    • Not all of them will be intentional.

What is Research Software?

Major Computational Programs

 

 

 

Data processing

 

 

Experiment support

 

 

 

Bathymetry by NOAA under public domain CTD Bottles by WHOI under public domain
Keeling Curve by Scripps under public domain
Dawn HPC by Joe Bishop with permission
Climate simulation by NSF under public domain

Why does this matter?

Why does this matter?

More widely than publishing papers, code is used in control and decision making:


  • Weather forecasting
  • Climate policy
  • Disease modelling (e.g. Covid)
  • Satellites and spacecraft1
  • Medical Equipment


Your code (or its derivatives) may well move from research to operational one day.

Margaret Hamilton and the Apollo XI by NASA under public domain

Why does this matter?1

def calc_p(n,t):
    return n*1.380649e-23*t
data = np.genfromtxt("mydata.csv")
p = calc_p(data[0,:],data[1,:]+273.15)
print(np.sum(p)/len(p))

What does this code do?

# Boltzmann Constant and 0 Kelvin
Kb = 1.380649e-23
T0 = 273.15

def calc_pres(n, t):
    """
    Calculate pressure using ideal gas law p = nkT

    Parameters:
        n : array of number densities of molecules [N m-3]
        t : array of temperatures in [K]
    Returns:
         array of pressures [Pa]
    """
    return n * Kb * t


# Read in data from file and convert T from [oC] to [K]
data = np.genfromtxt("mydata.csv")
n = data[0, :]
temp = data[1, :] + T0

# Calculate pressure, average, and print
pres = calc_pres(n, temp)
pres_av = np.sum(pres) / len(pres)
print(pres_av)

Virtual Environments

Virtual Environments

What?

  • A self-contained python environment
  • Packages installed in a local folder
  • Advised to use on a per-project basis

Why?

  • Avoid system pollution
  • Allow different versions
  • Reproducibility - set versions
$ python3 -m venv myvenv
$ source myvenv/bin/activate
(myvenv) $ pip install <packagename>
(myvenv) $ deactivate
$ 
PS> python -m venv myvenv
PS> source venv/bin/activate
(myvenv) PS> pip install <packagename>
(myvenv) PS> deactivate
PS>


For more information see the Real Python article.
For those using conda there are also environments, set up in a slightly different way.

Exercise 1

Scenario: you have just finished some simulations with a climate model that should improve precipitation modelling and have the output data as a netCDF file.

You know that your colleague has produced relevant figures and analysis before, so ask them for a copy of their code (yay, reuse :+1:).

Go to exercise 1 and:

  • Examine the code in precipitation_climatology.py
  • Set up a virtual environment
  • Install the necessary dependencies
  • Run the code - does it do what you thought?

Python PEPs

Python Enhancement Proposals

  • Technical documentation for the python community
  • Guidelines, standards, and best-practice

Relevant to us today are:

PEP8 and Formatting

PEP8 & Formatting

“Readability counts”
    - Tim Peters in the Zen of Python

By ensuring code aligns with PEP8 we:

  • standardise style,
  • conform to best-practices, and
  • improve code readability to
  • make code easier to share, and
  • reduce misinterpretation.

“But I don’t have time to read and memorise all of this…”

PEP8 & Formatting - Black

Black (Langa 2020) - black.readthedocs.io

  • a PEP 8 compliant formatter
    • Strict subset of PEP8
    • “Opinionated so you don’t have to be.”
  • For full details see Black style
  • Try online
(myvenv) $ pip install black
(myvenv) $ black myfile.py
(myvenv) $ black mydirectory/
(myvenv) PS> pip install black
(myvenv) PS> black myfile.py
(myvenv) PS> black mydirectory/

PEP8 & Formatting - Black - Example

def long_func(x, param_one, param_two=[], param_three=24, param_four=None,
        param_five="Empty Report", param_six=123456):


    val = 12*16 +(24) -10*param_one +  param_six

    if x > 5:
        
        print("x is greater than 5")


    else:
        print("x is less than or equal to 5")


    if param_four:
        print(param_five)



    print('You have called long_func.')
    print("This function has several params.")

    param_2.append(x*val)
    return param_2
def long_func(
    x,
    param_one,
    param_two=[],
    param_three=24,
    param_four=None,
    param_five="Empty Report",
    param_six=123456,
):
    val = 12 * 16 + (24) - 10 * param_one + param_six

    if x > 5:
        print("x is greater than 5")

    else:
        print("x is less than or equal to 5")

    if param_four:
        print(param_five)

    print("You have called long_func.")
    print("This function has several params.")

    param_2.append(x * val)
    return param_2

PEP8 & Formatting - Black

  • I suggest incorporating into your projects now
    • Well suited to incorporation into continuous integration of git hooks.
    • Widely-used standard1
  • A version for jupyter notebooks exists.

Exercise 2

Go to exercise 2 and:

  • install black
  • run black on precipitation_climatology.py
  • examine the output
    • Is it more readable?
    • Is there any aspect of the formatting style you find unintuitive?is it better?

Naming For Clarity

It may seem inconsequential, but carefully naming variables and methods can improve the readability of code massively and can help to make code self-documenting.

A few naming tips and conventions:

  • The name should show the intention, think about how someone else might read it (this could be future you)
  • Use pronounceable names e.g.
  •  ms    --> mass
 chclt --> chocolate
 stm   --> stem
  • avoid abbreviations and single letter variable names where possible
  • Use names that can be searched
  • One word per concept e.g. choose one of put, insert, add in the same code base

Naming For Clarity

  • Plurals to indicate groups, e.g. a list of dog objects would be dogs, not dog_list
  • Describe content rather than storage type e.g.
  • array       --> dogs
age_int     --> age
country_set --> countries
  • Naming booleans, use prefixes like is, has or can and avoid negations like not_green e.g.
  • purple    --> is_purple
not_plant --> is_plant
sidekick  --> has_sidekick
  • Keep it simple and use technical terms where appropriate

Explaining Variables

Without explaining variable:


def calculate_fare(age):
    if (age < 14):
        return 3
        ...

With explaining variable:


def calculate_fare(age):
    is_child = age < 14
    if (is_child):
        return 3
    ...

Explaining Variables

Without an explaining variable, it is hard to see what this code is doing:

import re

re.search("^\\+?[1-9][0-9]{7,14}$", "Sophie: CV56 9PQ, +12223334444")

With explaining variables:

It is easier to see the intention. The code is more self-documenting.

import re

phone_number_regex = "^\\+?[1-9][0-9]{7,14}$"
re.search(phone_number_regex, "Sophie: CV56 9PQ, +12223334444")

Exercise 3

Look through the code for any names of methods or variables that could be improved or clarified and update them. Note if you are using an IDE like Intellij or VSCode, you can use automatic renaming. Can you find an example from each of the suggestions listed below? Does this make the code easier to follow?

Consider the following:

  • The name should show the intention, think about how someone else might read it (this could be future you)
  • Use pronounceable names e.g. mass not ms, stem not stm
  • avoid abbreviations and single letter variable names where possible
  • One word per concept e.g. choose one of put, insert, add in the same code base
  • Use names that can be searched
  • Describe content rather than storage type
  • Naming booleans, use prefixes like is, has or can and avoid negations like not_green
  • Plurals to indicate groups, e.g. a list of dog objects would be dogs, not dog_list
  • Keep it simple and use technical terms where appropriate
  • Use explaining variables

PEP8 & Formatting - PyLint

Static Analysis

  • Check the code without running it
  • Catch issues before you run any code
  • Improve code quality1

There are various tools available:

  • pycodestyle
  • flake8
  • Pylint
  • ruff
(myvenv) $ pip install pylint
(myvenv) $ pylint myfile.py
(myvenv) $ pylint mydirectory/
(myvenv) PS> pip install pylint
(myvenv) PS> pylint myfile.py
(myvenv) PS> pylint mydirectory/

PEP8 & Formatting - PyLint - Example

def long_func(
    x,
    param_one,
    param_two=[],
    param_three=24,
    param_four=None,
    param_five="Empty Report",
    param_six=123456,
):
    val = 12 * 16 + (24) - 10 * param_one + param_six

    if x > 5:
        print("x is greater than 5")

    else:
        print("x is less than or equal to 5")

    if param_four:
        print(param_five)

    print("You have called long_func.")
    print("This function has several params.")

    param_2.append(x * val)
    return param_2
(myvenv) $ pylint long_func.py
************* Module long_func
long_func.py:1:0: C0116: Missing function or method docstring (missing-function-docstring)
long_func.py:1:0: W0102: Dangerous default value [] as argument (dangerous-default-value)
long_func.py:1:0: R0913: Too many arguments (7/5) (too-many-arguments)
long_func.py:24:4: E0602: Undefined variable 'param_2' (undefined-variable)
long_func.py:25:11: E0602: Undefined variable 'param_2' (undefined-variable)
long_func.py:4:4: W0613: Unused argument 'param_two' (unused-argument)
long_func.py:5:4: W0613: Unused argument 'param_three' (unused-argument)

------------------------------------------------------------------
Your code has been rated at 0.00/10

(myvenv) $

PEP8 & Formatting - PyLint - Example

def long_func(
    x,
    param_one,
    param_two=[],
    param_four=None,
    param_five="Empty Report",
    param_six=123456,
):
    val = 12 * 16 + (24) - 10 * param_one + param_six

    if x > 5:
        print("x is greater than 5")

    else:
        print("x is less than or equal to 5")

    if param_four:
        print(param_five)

    print("You have called long_func.")
    print("This function has several params.")

    param_two.append(x * val)
    return param_two
(myvenv) $ pylint long_func.py
************* Module long_func
long_func.py:1:0: C0114: Missing module docstring (missing-module-docstring)
long_func.py:1:0: C0116: Missing function or method docstring (missing-function-docstring)
long_func.py:1:0: W0102: Dangerous default value [] as argument (dangerous-default-value)
long_func.py:1:0: R0913: Too many arguments (6/5) (too-many-arguments)

------------------------------------------------------------------
Your code has been rated at 6.36/10 (previous run: 0.00/10, +6.36)

(myvenv) $


Search the error code to understand the issue:

IDE Integration

  • Catch issues before running PyLint
  • Gradually coerces you to become a better programmer
  • Available on all good text editors and emacs:

Exercise 4

Go to exercise 4 and:

  • install pylint
  • run pylint on precipitation_climatology.py
  • examine the report and try and address some of the issues.
    • Ignore missing docstrings and f-strings for now - we’ll come to them later.
    • Try and deal with: W0611 Unused imports, C0412 Ungrouped imports, W0102 Dangerous default
    • If you feel like it you could try and fix: W0621 Redefining name, W1514 Unexplicit open
    • Unless you are really keen don’t worry about: R0913 Too many arguments, C0103 Unconforming naming style.

Extensions:

  • try and add linting to your preferred text editor or IDE
  • explore the option to supress pylint warnings
  • explore the configuration options for pylint

Comments and Docstrings

Comments

Comments are tricky, and very much to taste.

Some thoughts:1

“Programs must be written for people to read and […] machines to execute.”
  - Hal Abelson

“A bad comment is worse than no comment at all.”

“A comment is a lie waiting to happen.”

=> Comments have to be maintained, just like the code, and there is no way to check them!

Cat code comment image by 35_equal_W

Comments to avoid

  • Dead code e.g.

    # plt.plot(time, velocity, "r0")
plt.plot(time, velocity, "kx")
# plt.plot(time, acceleration, "kx")
# plt.ylabel("acceleration")
plt.ylabel("velocity")

  • Variable definitions e.g.

    # Set Force
f = m * a

  • Redundant comments e.g. i += 1 # Increment i

Comments - some thoughts1

  • Comments should not duplicate the code.
  • Good comments do not excuse unclear code.
    • Comments should dispel confusion, not cause it.
    • If you can’t write a clear comment, there may be a problem with the code.
  • Explain unidiomatic code in comments.
  • Provide links to:
    • the original source of copied code.
    • external references where they will be most helpful.
  • Use comments to mark incomplete implementations.
  • Comments are not documentation.
    • Read by developers, documentation is for…

Docstrings

These are what make your code reusable (by you and others).

  • In python docstrings are designated at the start of ‘things’ using triple quotes: """...""".
  • PEP257 (Goodger and Rossum 2001) tells us what docstrings should say.
    Specific conventions tell us how they should say it.
  • Where comments describe how it works, docstrings describe how to use it.
    Unlike comments, docstrings follow a set format.

Various formatting options exist: numpy, Google, reST, etc.
We will use numpydoc as it is readable and widely used in scientific code.
Full guidance for numpydoc is available.

Docstrings

Key components:

  • A description of what the thing is.
  • A description of any inputs (Parameters).
  • A description of any outputs (Returns).

Consider also:

  • Extended summary
  • Errors raised
  • Usage examples
  • Key references
"""
Short one-line description.

Parameters
----------
name : type
    description of parameter

Returns
-------
name : type
    description of return value
"""

Docstrings

Key components:

  • A description of what the thing is.
  • A description of any inputs (Parameters).
  • A description of any outputs (Returns).
def calculate_gyroradius(mass, v_perp, charge, B, gamma=None):
    """
    Calculates the gyroradius of a charged particle in a magnetic field

    Parameters
    ----------
    mass : float
        The mass of the particle [kg]
    v_perp : float
        velocity perpendicular to magnetic field [m/s]
    charge : float
        particle charge [coulombs]
    B : float
        Magnetic field strength [teslas]
    gamma : float, optional
        Lorentz factor for relativistic case. default=None for non-relativistic case.

    Returns
    -------
    r_g : float
        Gyroradius of particle [m]

    Notes
    -----
    .. [1]  Walt, M, "Introduction to Geomagnetically Trapped Radiation,"
       Cambridge Atmospheric and Space Science Series, equation (2.4), 2005.
    """

    r_g = mass * v_perp / (abs(charge) * B)

    if gamma:
        r_g = r_g * gamma

    return r_g

Exercise 5

Go to exercise 5 and examine the comments:

  • Is there any dead code?
    • How is it best to handle it?
  • Are comments used sensibly?
    • Are any redundant and better off being removed?
    • Is there anywhere that would benefit from a comment?

Docstrings:

  • Work through the file adding docstrings where they are missing.
  • If you are unsure about variable types or meanings at any point you can sneak a look ahead to the code in exercise 6.

READMEs

  • First point of contact for a new user/contributor with the code repository
  • Should give essential information on
    • what this is
    • what it’s for
    • how to get started
  • In the best case it also tells you
    • who built/is building this
    • how to contribute
    • how to reuse
  • Usually written in Markdown
  • See also, e.g., https://www.makeareadme.com/ and https://readme.so/

Recommended README Content

  • Overview of software, including relevant papers
  • Project/release status
  • Requirements and environment
  • Installation
  • Building
  • Usage/Quick start guide (incl. simple example with expected output)
  • Documentation and Support: User Guide/Website/Help/Discussion forum
  • Contributing guide
  • Authors and Acknowledgement
  • Licence
  • Known Issues

Other potential files in your repository

  • LICENSE
  • CONTRIBUTING.md
  • CITATION.cff
  • CODE_OF_CONDUCT.md
  • CHANGES.md

Types of Licenses

https://www.techtarget.com/searchcio/definition/software-license

How to choose a license

  • https://choosealicense.com/licenses/
  • Permissive licenses:
    • Apache License 2.0
    • MIT License
  • Copyleft:
    • Means that copy/adaption has to use the same license
    • GNU General Public License v3.0

Example: MIT License

Copyright <YEAR> <COPYRIGHT HOLDER>

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Writing better (Python) code

Remove Magic Numbers

Numbers in code that are not immediately obvious.

  • Hard to read
  • Hard to maintain
  • Hard to adapt

Instead:

  • Name a variable conveying meaning
  • Set to a constant
  • Use a comment to explain

numberwang by Mitchell and Webb under fair use

Remove Magic Numbers

"""Module implementing pendulum equations."""
import numpy as np

def get_period(l):
    """..."""
    return 2.0 * np.pi * np.sqrt(l / 9.81)

def max_height(l, theta):
    """..."""
    return l * np.cos(theta)

def max_speed(l, theta):
    """..."""
    return np.sqrt(2.0 * 9.81 * max_height(l, theta))

def energy(m, l, theta):
    """..."""
    return m * 9.81 * max_height(l, theta)

def check_small_angle(theta):
    """..."""
    if theta <= np.pi / 1800.0:
        return True
    return False

def bpm(l):
    """..."""
    return 60.0 / get_period(l)



"""Module implementing pendulum equations."""
import numpy as np

GRAV = 9.81

def get_period(l):
    """..."""
    return 2.0 * np.pi * np.sqrt(l / GRAV)

def max_height(l, theta):
    """..."""
    return l * np.cos(theta)

def max_speed(l, theta):
    """..."""
    return np.sqrt(2.0 * GRAV * max_height(l, theta))

def energy(m, l, theta):
    """..."""
    return m * GRAV * max_height(l, theta)

def check_small_angle(theta, small_ang=np.pi/1800.0):
    """..."""
    if theta <= small_ang:
        return True
    return False

def bpm(l):
    """..."""
    # Divide 60 seconds by period [s] for beats per minute
    return 60.0 / get_period(l)

Exercise 6

Magic Numbers

  • Look through the code and identify any magic numbers.
  • Implement what you feel is the best approach in each case

Other things

Beyond the scope of today are a few other honourable mentions:

  • Functions and modules
  • Packaging
    • Breaking projects into modules and __init__.py
    • Distributing projects with pyproject.toml
  • Documentation
    • Auto-generation from docstrings with sphinx or mkdocs
  • Type hinting
    • Adding type hinting to python code - how and why?
    • Type checking with mypy

These lessons are beyond the scope of today.

Closing

Where can I get help?

The ICCS RSE team are always keen to support researchers with developing and applying the principles discussed today.

If you would like to discuss applying this to your own codebase consider signing up for an ICCS Climate Code Clinic:

  • 1hr slot
  • RSEs will review code in advance and provide feedback and guidance.
  • Online booking form

Where can I learn more?

  • References and links in these slides
  • Writing Clean Scientific Software Webinar (Murphy 2023)
  • RSE slack workspace


Get in touch:

 Jack Atkinson

 jackatkinson.net

 jwa34[AT]cam.ac.uk

 jatkinson1000

 @jatkinson1000@fosstodon.org

 Marion Weinzierl

 mw925[AT]cam.ac.uk

 MarionBWeinzierl

 @MarionBWeinzierl@mast.hpc.social


 Amy Pike

 ap766[AT]cam.ac.uk

 AmyOctoCat

References

The code in this workshop is based on a script from (Irving 2019).

Cannon, B, D Ingram, P Ganssle, P Gedam, S Eustace, T Kluyver, and T Chung. 2020. “PEP 621 – Storing project metadata in pyproject.toml.” https://peps.python.org/pep-0621/.
Goodger, D, and G van Rossum. 2001. “PEP 257 – Docstring Conventions.” https://peps.python.org/pep-0257/.
Irving, Damien. 2019. “Python for Atmosphere and Ocean Scientists.” Journal of Open Source Education 2 (16): 37. https://doi.org/10.21105/jose.00037.
Langa, Ł. 2020. “Black: The uncompromising Python code formatter.” https://github.com/psf/black. https://black.readthedocs.io/en/stable/.
Murphy, N. 2023. “Writing Clean Scientific Software.” In. Presented at the HPC Best Practices Webinar Series. https://www.youtube.com/watch?v=Q6Ksu_uX3bc.
Rossum, G van, B Warsaw, and A Coghlan. 2001, 2013. “PEP8 – Style Guide for Python Code.” https://peps.python.org/pep-0008/.
Spertus, E. 2021. “stackoverflow - Best practices for writing code comments.” https://stackoverflow.blog/2021/12/23/best-practices-for-writing-code-comments/.