Parallel performance of vector AD

Speaker: Troels Henriksen, University of Copenhagen

Abstract

Vector AD is the approach of passing in multiple seed vectors when performing automatic differentiation, essentially computing multiple rows or columns of the Jacobian. The potential advantage of this approach is to amortize the cost of the primal computation (or tape construction) over multiple derivatives, in contrast to the more common approach of redundantly recomputing the primal. However, now each primal variable is associated with multiple (co-)tangents, which may significantly which operations are done at the machine level, including challenges for memory management, locality, and parallelism.

In my talk I will report on our implementation of our parallelism-preserving vector AD implementation for Futhark, a high level data parallel language that targets GPU execution. I will discuss some of the low-level technical challenges that we have encountered, and investigate to which extent (and in which cases) vector AD provides potential performance advantages