Running Pipelines

How to run pipelines in the portal.

We currently have the ability to run pipelines in the portal!!!! These pipelines run in a cloud environment (Google Cloud Platform) and are intended to give an easy experience from running pipelines using raw files (currently fastq.gz) to offering a default interactive view of the study quickly. We are excited to make life easier!

Eventually, you will need a billing object to run pipelines but we offer free credits to get you going :-)

Step 0. Contact us to get alpha user access.

Email us at single_cell_portal@broadinstitute.org

Step 1. Get free credits!

Please go here to apply for free credits..

If set up correctly, free credits in FireCloud will work as free credits in the Single Cell Portal.

Step 2. Create a study

You will first need to create a study so there is a place to work with your file.

Please see instructions here.

Step 3. Upload data to a study

You will need to upload your raw data to your bucket associated with the study.

Here are instructions to upload to your bucket using gsutil.

Please note, it may be easier for you long term to collect all your fastqs.gz files in a "folder" in the bucket. This is simply done by adding the directory you want to copy to in the gsutil cp command (the directory will automatically be made for you).

# Initial example command
# If you were wanting to copy all fastq.gz files in your current directory and put them in the bucket
gsutil cp *.fastq.gz gs://xx-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

# Modified example command
# Same as above but collecting the fastq.gz files in a "fastqs" directory in the bucket.
gsutil cp *.fastq.gz gs://xx-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/fastqs/

Step 4. Run pipeline

Go to the "Workflows" tab in your study.

Select the workflow you would like to run and click next.

Select the samples of interest and click next.

The portal tried to guess file names and pairings.
Please review and if they are not right please download, edit, upload, and save.
When samples are paired and named correctly, select the samples you would like to run.
Click next.

Select parameters and submit.

Feel free to check in on the pipeline (click refresh as needed).

You can come back to the submission history to check on the status your workflows. Feel free to refresh as needed.

Buttons:
View Run Info: View submission information.
Sync: Copy the run outputs over to the portal. These will gain the sharing permissions of your study (eg. if the study is private or public, the synced files will be private or public respectively).
Show Errors: Show errors associated with errored workflow runs.
Delete Submission: Delete the submission and outputs generated by the submission but not the inputs.