aiida-fireworks-scheduler is available at http://github.com/zhubonan/aiida-fireworks-scheduler
By default, each AiiDA calculation is mapped to a single job that is submitted directly to the cluster. Sometimes, it may be useful to allow multiple AiiDA calculations to be run inside a single cluster job, for example:
When there are limits on how many jobs a single user can have in the queue
When typical workflows will need to run many small calculations sequentially - the overall queuing time can become significant.
When calculations only require small amount of resources, but the user is forced to submit jobs of certain (larger) sizes.
This plugin allows AiiDA calculations to be performed with fireworks, which is a workflow engine that focus on high-throughput execution using flexible computing resourcess. Because the latter operates on a server-client mode, one must have a dedicated MongoDB server (LaunchPad) which must be accessible by both both the local and the remote computer.
How it works?¶
Below is a brief summary of the the life cycle of an AiiDA job:
The job gets submitted.
The daemon generate the input files and update them to the remote cluster.
The daemon submit the jobs according to the submission script generated.
The daemon monitors the jobs.
When the job is finished, the outputs and then parsed locally.
Thanks to the entrypoint based plugin system implemented in AiiDA, most of these steps can be extended by a plugin.
Here, we alter steps 3 and step 4 - rather than submitting the job to the actual scheduler of the remote computer, it submits to the LaunchPad server. The LaunchPad server stores all Firework 1 to be run, as well as those that have finished. The same happens in in step 4 - rather than querying the actual scheduler periodically, the plugin pulls job states from the LaunchPad instead.
Because now AiiDA merely registers jobs to the LaunchPad server, the user is now responsible for submitting actual jobs to the scheduler to run those Firework s.
A Firework is like a CalculationJob in AiiDA - it runs some code using the computer. Here, it simply executes the original AiiDA submission script as if it is run in a cluster allocation.
- User guide
- Developer guide
- API documentation
If you use this plugin or if you use AiiDA for your research, please cite the following work:
aiida-fireworks-scheduler is released under the MIT license.