OiO.lk Community platform!

Oio.lk is an excellent forum for developers, providing a wide range of resources, discussions, and support for those in the developer community. Join oio.lk today to connect with like-minded professionals, share insights, and stay updated on the latest trends and technologies in the development field.
  You need to log in or register to access the solved answers to this problem.
  • You have reached the maximum number of guest views allowed
  • Please register below to remove this limitation

Setting resources dynamically on snakemake

  • Thread starter Thread starter Agustin Carbajal
  • Start date Start date
A

Agustin Carbajal

Guest
Context
I am running a snakemake (v.7.32.4) pipeline using slurm task manager. I have set resources (time and memory for each rule) dynamically based on file size and # of tries, like this:

Code:
rule index:
    resources:
        mem_mb = lambda wildcards, input, attempt: (
            200 * attempt
        ),
        runtime = lambda wildcards, input, attempt: (
            "{minutes}min".format(
                minutes=max(
                    int((input.size_mb / 5000) * attempt),
                    1)
            )
        )

I have 2 related questions (should I split the post?):
1) Is it possible to set resources dinamically outside of snakefile? I tried to set that on the profile config file but didn't success (sometime ago, so cannot say exactly what I tried)

2) Having set resources dinamically inside Snakefile, how do I do a dry run or a rulegraph?
If I run a dry-run I get the following error:

Code:
WorkflowError:
Cannot parse runtime value into minutes for setting runtime resource: <TBD>

This seems logical to me since the file doesn't exist yet. Nevertheless, I would like to know the specifics of all steps (except resources of course) to be run before actually running them, is this possible?
Something similar happens if I try to do a rulegraph:

Code:
snakemake --profile Config/Profiles/slurm -np --rulegraph | dot -Tsvg > rulegraph.svg

In this case, I get an empty file (probably because of the error in the dry run?).
<p><strong>Context</strong><br />
I am running a snakemake (<strong>v.7.32.4</strong>) pipeline using slurm task manager. I have set resources (time and memory for each rule) dynamically based on file size and # of tries, like this:</p>
<pre><code>rule index:
resources:
mem_mb = lambda wildcards, input, attempt: (
200 * attempt
),
runtime = lambda wildcards, input, attempt: (
"{minutes}min".format(
minutes=max(
int((input.size_mb / 5000) * attempt),
1)
)
)
</code></pre>
<p>I have 2 related questions (should I split the post?):<br />
<strong>1) Is it possible to set resources dinamically outside of snakefile?</strong>
I tried to set that on the profile config file but didn't success (sometime ago, so cannot say exactly what I tried)</p>
<p><strong>2) Having set resources dinamically inside Snakefile, how do I do a dry run or a rulegraph</strong>?<br />
If I run a dry-run I get the following error:</p>
<pre><code>WorkflowError:
Cannot parse runtime value into minutes for setting runtime resource: <TBD>
</code></pre>
<p>This seems logical to me since the file doesn't exist yet. Nevertheless, I would like to know the specifics of all steps (except resources of course) to be run before actually running them, is this possible?<br />
Something similar happens if I try to do a rulegraph:</p>
<pre><code>snakemake --profile Config/Profiles/slurm -np --rulegraph | dot -Tsvg > rulegraph.svg
</code></pre>
<p>In this case, I get an empty file (probably because of the error in the dry run?).</p>
 
Top