Difference between revisions of "Using Snakemake on the HPC Cluster"
From Bridges Lab Protocols
Davebridges (Talk | contribs) (Changed to code tags) |
Davebridges (Talk | contribs) (Switched back) |
||
| Line 2: | Line 2: | ||
* Install snakemake via mamba/conda (see https://snakemake.readthedocs.io/en/stable/getting_started/installation.html) | * Install snakemake via mamba/conda (see https://snakemake.readthedocs.io/en/stable/getting_started/installation.html) | ||
| − | * In the working folder create a snakemake environment with < | + | * In the working folder create a snakemake environment with <pre>mamba activate snakemake</pre> |
| − | * Install the cluster-generic tool to submit snakemake files to a cluster from pip with < | + | * Install the cluster-generic tool to submit snakemake files to a cluster from pip with <pre>pip install snakemake-executor-plugin-cluster-generic</pre> |
* Create a generic profile in ~/.config/snakemake/testprofile/config.yaml such as: | * Create a generic profile in ~/.config/snakemake/testprofile/config.yaml such as: | ||
| − | < | + | <pre> |
executor: slurm | executor: slurm | ||
jobs: 100 | jobs: 100 | ||
| Line 19: | Line 19: | ||
myrule: | myrule: | ||
mem_mb: attempt * 200 | mem_mb: attempt * 200 | ||
| − | </ | + | </pre> |
== Executing a Snakefile Workflow == | == Executing a Snakefile Workflow == | ||
* Create a Snakefile (see | * Create a Snakefile (see | ||
** You can over-ride resources with that flag otherwise it will use the values in testprofile | ** You can over-ride resources with that flag otherwise it will use the values in testprofile | ||
| − | * Run the Snakefile with this command < | + | * Run the Snakefile with this command <pre>snakemake --profile testprofile -j 1 --executor cluster-generic --cluster-generic-submit-cmd "batch" ... </pre> |
Latest revision as of 20:00, 16 May 2024
Setup
- Install snakemake via mamba/conda (see https://snakemake.readthedocs.io/en/stable/getting_started/installation.html)
- In the working folder create a snakemake environment with
mamba activate snakemake
- Install the cluster-generic tool to submit snakemake files to a cluster from pip with
pip install snakemake-executor-plugin-cluster-generic
- Create a generic profile in ~/.config/snakemake/testprofile/config.yaml such as:
executor: slurm
jobs: 100
default-resources:
mem_mb: max(1.5 * input.size_mb, 100)
account: <ACCOUNT_NAME>
partition: <PARTITION NAME>
set-threads:
myrule: max(input.size_mb / 5, 2)
set-resources:
myrule:
mem_mb: attempt * 200
Executing a Snakefile Workflow
- Create a Snakefile (see
- You can over-ride resources with that flag otherwise it will use the values in testprofile
- Run the Snakefile with this command
snakemake --profile testprofile -j 1 --executor cluster-generic --cluster-generic-submit-cmd "batch" ...
