consonance architecture

The various components of Consonance are illustrated below.

The queues in the system are provided by Rabbitmq. The main things to note are as follows:

• orders for new VMs and workflows/tools are queued through the web service
• the coordinator creates jobs from these orders and puts them into job queues, one per instance-type
• the container provisioner spins up VMs as necessary to match the number of expected pending and running VMs
• the worker runs on VMs, pulling necessary containers and input files while reporting back heartbeats to the results queue
• the coordinator saves stdout and stderr from completed jobs
• the container provisioner shuts down VMs from VMs that have no more work to do

The worker relies upon the the dockstore-descriptor-launcher in order to run CWL tools.

The container provisioner relies upon the Youxia library in order to abstract our interactions with AWS, Azure, and OpenStack (essentially creating VMs (on spot price markets if possible), maintaining a certain number of VMs, and then tearing them down).