High-throughput workflows with ASR and MyQueue

The Atomic Simulation Recipes (ASR) [1] is an open source Python framework for working with atomistic materials simulations in an efficient and sustainable way that is ideally suited for high-throughput computations. ASR contains a library of recipes, or high-level functions, that define specific atomistic simulations tasks using the Atomic Simulation Environment (ASE) [2]. The recipes can be combined into workflows that perform complex simulation tasks while keeping track of relevant metadata to ensure documentation and reproducibility of the data. The ASR also contains functionality for collecting the resulting data into databases and presenting them in a browser. 

So far, ASE has mainly handled atomistic quantities. New developments, initiated at the NOMAD workshop on High-throughput workflows [3], have addressed the ASE Calculator object, which is used to interface ASE with external electronic structure codes. An extension and refactoring of the Calculator object has made it possible to define and handle more advanced quantities, such as electronic quantities, within ASE and exchange them efficiently with the electronic structure codes. These developments will eventually make it possible to decouple ASR workflows from specific electronic structure codes, such that the same workflow can be executed with different computational engines. 

Recent work on ASR also involves expanding its capabilities towards high-throughput calculations in the exascale regime. This means using more scalable caching methods and expressing the flow of information within workflows in ways that facilitate the tracking of individual jobs by a queueing manager. In ASR, workflows are normally defined using MyQueue [4], a front-end, which works with Slurm, PBS, or LSF.



[1] Atomic Simulation Recipes: A Python framework and library for automated workflows, M. Gjerding et al., Computational Materials Science 199, 110731 (2021)

[2] The atomic simulation environment—a Python library for working with atoms, A. H. Larsen et al., J. Phys.: Condens. Matter 29 273002 (2017)


[4] MyQueue: Task and workflow scheduling system, J. J. Mortensen et al., Journal of Open Source Software, 5, 1844 (2020)