Teaching Material Recap

Over the ML sessions at the summer school we have learnt about:

• Classification - categorising items based on information
• Regression - using information to predict another value

using:

• ANNs - using input features to make predictions
• CNNs - using image-like data as an input

Applications in geosciences:

See review of Kashinath et al. (2021)

Paramterisations

• Parameterisations are typically expensive
  • Microphysics and Radiation are top offenders
• Replace parameterisations with NNs
  • emulation of existing parameterisation
    e.g. Espinosa et al. (2022)
  • data-driven parameterisations
    • capture missing physics?
  • train with a high-resolution model
    access the benefits of subgrid model without the cost(?)

Machine Learning

We typically think of Deep Learning as an end-to-end process;
a black box with an input and an output.

Who’s that Pokémon?

\[\begin{bmatrix}\vdots\\a_{23}\\a_{24}\\a_{25}\\a_{26}\\a_{27}\\\vdots\\\end{bmatrix}=\begin{bmatrix}\vdots\\0\\0\\1\\0\\0\\\vdots\\\end{bmatrix}\] It’s Pikachu!

Machine Learning in Science

Replacing physics-based components

2 approaches:

• emulation, or
• data-driven.

Additional challenges:

• Physical compatibility
  • Physics-based models have conservation laws
    Required for accuracy and stability
• Language interoperation

Downscaling

• Can we get information for ‘free’?
• Train to predict ‘image’ from coarsened version.
  • Topography?

Forecasting

• Time-series
  • popular use
  • Recurrent Neural Nets
• Complete weather
  • FourCastNet, Pangu-Weather, GraphCast

Training data - considerations

How should we prepare our training data?

• Cyclic data?
  • e.g. diurnal, annual, other
  • use time as an input
  • use a [daily] average

Training data - implications

• A NN only knows as much as its training data.
• How do you predict the 1/100 event? 1/1000 event?
• How do you train for a changing climate?
  • And tipping points?

Structure/Physics-informed approach

There is a wide variety of ways to structure a Neural Net.

What is the most appropriate for our application.

What are potentiall pitfalls - don’t go in blind with an ML hammer!

Case study of Ukkonen (2022) for emulating radiative transfer:

• Recurrent Neural Network reflects physical propogation,
• and prevents spurious correlations.

Physical Compatibility

Many ML applications in climate science are more complex than other classical applications.

• our ML useage is often not end-to-end
• A stable/accurate offline model will not neccessarily be stable online (Furner et al. 2023).

Your NN is perfectly happy to have ‘negative rain’.

• Even with heavy penalties
• This is not a new problem in numerical parameterisations.
• How is it best to enforce physical constraints in NNs.

Redeployability

How easy is it to redeploy a ML model? - exactly what has it learned?

• Locked to a geographical location?
• Locked to numerical model?
  • Locked to a specific grid!?
• How do we handle inputs from different models?

Interfacing

Replacing physics-based components of larger models (emulation or data-driven) requires care.

• Language interoperation
• Physical compatibility

Interfacing - Possible solutions

Ideally need to:

• Not generate excess additional work for user
  • Not require excess knowledge of computing
  • Minimal learning curve
• Not add excess dependencies
• Be easy to maintain
• Maximise performance

Interfacing - Possible solutions

• Implement a NN in Fortran
• Forpy/CFFI
• SmartSim/Pipes
• Fortran-Keras Bridge

Interfacing - Our Solution

Interfacing - Our Solution

Ftorch and TF-lib

• Use Fortran’s intrinsic C-bindings to access the C/C++ APIs provided by ML frameworks
• Performance
• Ease of use
• Use frameworks’ implementations directly

Interfacing - Our Solution

Ftorch and TF-lib

• Use Fortran’s intrinsic C-bindings to access the C/C++ APIs provided by ML frameworks
• Performance
  • avoids python runtime
  • no-copy transfer (shared memory)
• Ease of use
• Use frameworks’ implementations directly

Interfacing - Our Solution

Ftorch and TF-lib

• Use Fortran’s intrinsic C-bindings to access the C/C++ APIs provided by ML frameworks
• Performance
• Ease of use
  • pleasant API (see next slides)
  • utilities for generating TorchScript/TF module provided
  • examples provided
• Use frameworks’ implementations directly

Interfacing - Our Solution

Ftorch and TF-lib

• Use Fortran’s intrinsic C-bindings to access the C/C++ APIs provided by ML frameworks
• Performance
• Ease of use
• Use frameworks’ implementations directly
  • feature support
  • future support
  • direct translation of python models ¹

Contact

For more information we can be reached at:

You can also contact the ICCS, make a resource allocation request, or visit us at the Summer School RSE Helpdesk.