CWL run of Alignment Workflow (CWLProv 0.6.0 Research Object)

<p>This dataset folder is a <strong>CWLProv Research Object</strong> that captures the <a href="http://commonwl.org/">Common Workflow Language</a> execution provenance, see <a href="https://w3id.org/cwl/prov/0.6.0">CWLProv 0.6.0</a> or use the <a href="https://pypi.org/project/cwlprov/">cwlprov Python tool</a> to explore.</p> <p>&nbsp;</p> <p>The CWL <a href="https://w3id.org/cwl/view/git/923ae325fa3a6d3ee3871539c65afa06ac86ac54/aligner/sbg-alignment-cwl/topmed-alignment.cwl">alignment workflow</a> included in this case study is designed by <a href="https://github.com/DataBiosphere">Data Biosphere</a>. It adapts the alignment pipeline originally developed at Abecasis Lab, The University of Michigan. This <a href="https://github.com/FarahZKhan/topmed-workflows/tree/cwlprov_testing">workflow</a> is part of NIH Data Commons initiative and comprises of four stages.</p> <p>First step, <strong><em>Pre-align</em></strong><em>,</em> accepts a <em>Compressed Alignment Map</em> (CRAM) file (a compressed format for BAM files developed <em>by European Bioinformatics Institute</em> (EBI)) and human genome reference sequence as input and using underlying software utilities of <strong>SAMtools</strong> such as <em>view</em>, <em>sort</em> and <em>fixmate</em> returns a list of fastq files which can be used as input for the next step.</p> <p>The next step <strong><em>Align</em></strong> also accepts the human reference genome as input along with the output files from <em>Pre-align</em> and uses <strong>BWA-mem</strong> to generate aligned reads as BAM files. <strong>SAMBLASTER</strong> is used to mark duplicate reads and <strong>SAMtools</strong> view to convert read files from SAM to BAM format.</p> <p>The BAM files generated after <em>lign</em> are sorted with <strong><em>SAMtool sort</em></strong>&#39;.</p> <p>Finally, these sorted alignment files are merged to produce single sorted BAM file using SAMtools merge in <strong><em>Post-align</em></strong> step.</p> <p>&nbsp;</p> <p><strong>Steps to reproduce</strong></p> <p>This analysis was run using a 16-core Linux cloud instance with 64GB RAM and pre-installed docker.</p> <ol> <li> <p>Install <strong>gsutils</strong><br> &nbsp;</p> <pre><code class="language-bash">export CLOUD_SDK_REPO="cloud-sdk-$(lsb_release -c -s)" echo "deb http://packages.cloud.google.com/apt $CLOUD_SDK_REPO main" | \ sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | \ sudo apt-key add - sudo apt-get update &amp;&amp; sudo apt-get install google-cloud-sdk</code></pre> <p>&nbsp;</p> </li> <li> <p>Get the data and make the analysis environment ready:<br> &nbsp;</p> <pre><code class="language-bash">git clone https://github.com/FarahZKhan/topmed-workflows.git cd topmed-workflows git checkout cwlprov_testing cd aligner/sbg-alignment-cwl # this is a custom script download google bucket files from json files and create a local json # it needs gsutil to be installed though git clone https://github.com/DailyDreaming/fetch_gs_frm_json.git # Wait... this should download ~18Gb. python2.7 fetch_gs_frm_json/dl_gsfiles_frm_json.py topmed-alignment.sample.json </code></pre> <p>&nbsp;</p> </li> <li> <p>Run the following commands to create the CWLProv Research Object:</p> <pre><code class="language-bash">time cwltool --no-match-user --provenance alignmnentwf0.6.0 --tmp-outdir-prefix=/CWLProv_workflow_testing/intermediate_temp/temp --tmpdir-prefix=/CWLProv_workflow_testing/intermediate_temp/temp topmed-alignment.cwl topmed-alignment.sample.json.new zip -r alignment_0.6.0_linux.zip alignment_0.6.0_linux sha256sum alignment_0.6.0_linux.zip &gt; alignment_0.6.0_linux.zip.sha25</code></pre> <p>&nbsp;</p> </li> </ol>