Skip to content

Commit 297dfca

Browse files
cmeesterscmeesters
committed
Merge branch 'fix/Dresden' of github.com:snakemake/snakemake-hpc-teaching-material into fix/Dresden
2 parents 583df46 + 7fb6840 commit 297dfca

23 files changed

+785
-68
lines changed

‎config/config_dresden.yaml

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -34,13 +34,19 @@ course:
3434
# a condarc file on a cluster, copy and adjust
3535
# for your cluster.
3636
condarcfile: "common/condarc_barnard.tex"
37+
38+
39+
# We provide a common software stack to avoid long
40+
# installation sessions.
41+
softwarepath: "/projects/p_nhr_snakemake/software_stack"
3742
# these are the path names to contain sample data
3843
# see the README for explanations.
39-
pathtosetup: "/projects/p_nhr_snakemake"
40-
pathtosolutions: ""
44+
pathtosetup: "/projects/p_nhr_snakemake/tutorial"
45+
pathtosolutions: "/projects/p_nhr_snakemake/solutions"
4146
# after copying the setup path is:
42-
tutorialpath: "~/tasks/tutorial"
43-
# This is the selected slide to explain the 'Hello World' script
47+
tutorialpath: "~/tasks"
48+
deploy_url: "https://github.com/snakemake-workflows/rna-seq-kallisto-sleuth"
49+
# This is the selected slide to explain the 'Hello World' script
4450
# to introduce SLURM. It requires different account and partition
4551
# settings on every cluster.
4652
hello_world_script: "common/Hello_World_HPC_Barnard.tex"

‎images/misc/bricks.png

73.7 KB
Loading

‎setup_creators/copy_script_ZIH_Dresden.sh

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,15 +2,15 @@
22

33
# This script is used to teach on the cluster "Mogon II"
44

5-
CLUSTER_ALIAS="nhr410@login1.barnard.hpc.tu-dresden.de"
5+
CLUSTER_ALIAS="barnard"
66
BASEPATH="/projects/p_nhr_snakemake"
77

88
# creating remote directory:
99
ssh ${CLUSTER_ALIAS} "mkdir -p ${BASEPATH}"
1010

1111
scp condarc_barnard "${CLUSTER_ALIAS}:${BASEPATH}/condarc"
1212
scp get_tutorial.sh "${CLUSTER_ALIAS}:${BASEPATH}/get_tutorial.sh"
13-
scp install_mamba.sh "${CLUSTER_ALIAS}:${BASEPATH}/install_mamba.sh"
13+
scp install_conda.sh "${CLUSTER_ALIAS}:${BASEPATH}/install_conda.sh"
1414
scp environment.yaml "${CLUSTER_ALIAS}:${BASEPATH}/environment.yaml"
1515

1616
rsync -rtlv --chmod=D755 "tutorial" "${CLUSTER_ALIAS}:${BASEPATH}"

‎setup_creators/solutions/05_Snakefile_target

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# our samples are pre-configured
2-
SAMPLES = ["A", "B"]
2+
samples = ["A", "B"]
33

44
rule all:
55
input:
@@ -37,8 +37,8 @@ rule samtools_index:
3737
rule bcftools_call:
3838
input:
3939
fa="data/genome.fa",
40-
bam=expand("sorted_reads/{sample}.bam", sample=SAMPLES),
41-
bai=expand("sorted_reads/{sample}.bam.bai", sample=SAMPLES)
40+
bam=expand("sorted_reads/{sample}.bam", sample=samples),
41+
bai=expand("sorted_reads/{sample}.bam.bai", sample=samples)
4242
output:
4343
"calls/all.vcf"
4444
shell:

‎setup_creators/solutions/06_Snakefile_run

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# our samples are pre-configured
2-
SAMPLES = ["A", "B"]
2+
samples = ["A", "B"]
33

44
rule all:
55
input:
@@ -38,8 +38,8 @@ rule samtools_index:
3838
rule bcftools_call:
3939
input:
4040
fa="data/genome.fa",
41-
bam=expand("sorted_reads/{sample}.bam", sample=SAMPLES),
42-
bai=expand("sorted_reads/{sample}.bam.bai", sample=SAMPLES)
41+
bam=expand("sorted_reads/{sample}.bam", sample=samples),
42+
bai=expand("sorted_reads/{sample}.bam.bai", sample=samples)
4343
output:
4444
"calls/all.vcf"
4545
shell:

‎setup_creators/solutions/07_Snakefile_script

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# our samples are pre-configured
2-
SAMPLES = ["A", "B"]
2+
samples = ["A", "B"]
33

44

55
rule all:
@@ -40,8 +40,8 @@ rule samtools_index:
4040
rule bcftools_call:
4141
input:
4242
fa="data/genome.fa",
43-
bam=expand("sorted_reads/{sample}.bam", sample=SAMPLES),
44-
bai=expand("sorted_reads/{sample}.bam.bai", sample=SAMPLES)
43+
bam=expand("sorted_reads/{sample}.bam", sample=samples),
44+
bai=expand("sorted_reads/{sample}.bam.bai", sample=samples)
4545
output:
4646
"calls/all.vcf"
4747
shell:

‎setup_creators/solutions/08_Snakefile_script2

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# our samples are pre-configured
2-
SAMPLES = ["A", "B"]
2+
samples = ["A", "B"]
33

44
configfile: "config/config.yaml"
55

@@ -41,8 +41,8 @@ rule samtools_index:
4141
rule bcftools_call:
4242
input:
4343
fa="data/genome.fa",
44-
bam=expand("sorted_reads/{sample}.bam", sample=SAMPLES),
45-
bai=expand("sorted_reads/{sample}.bam.bai", sample=SAMPLES)
44+
bam=expand("sorted_reads/{sample}.bam", sample=samples),
45+
bai=expand("sorted_reads/{sample}.bam.bai", sample=samples)
4646
output:
4747
"calls/all.vcf"
4848
shell:

‎setup_creators/solutions/09_Snakefile_config

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# our samples are pre-configured
2-
SAMPLES = ["A", "B"]
2+
samples = ["A", "B"]
33

44
configfile: "config/config.yaml"
55

@@ -41,8 +41,8 @@ rule samtools_index:
4141
rule bcftools_call:
4242
input:
4343
fa="data/genome.fa",
44-
bam=expand("sorted_reads/{sample}.bam", sample=SAMPLES),
45-
bai=expand("sorted_reads/{sample}.bam.bai", sample=SAMPLES)
44+
bam=expand("sorted_reads/{sample}.bam", sample=samples),
45+
bai=expand("sorted_reads/{sample}.bam.bai", sample=samples)
4646
output:
4747
"calls/all.vcf"
4848
shell:

‎setup_creators/solutions/13_Snakemake_wrapper

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
configfile: "config/config.yaml"
2+
3+
4+
rule all:
5+
input:
6+
"calls/all.vcf",
7+
"calls/positions.png",
8+
"calls/quals.svg",
9+
10+
11+
def get_bwa_map_input_fastqs(wildcards):
12+
return config["samples"][wildcards.sample]
13+
14+
15+
rule bwa_mem:
16+
input:
17+
reads=get_bwa_map_input_fastqs,
18+
idx=multiext("data/genome.fa", ".amb", ".ann", ".pac", ".sa"),
19+
output:
20+
"sorted_reads/{sample}.bam",
21+
log:
22+
"logs/bwa_mem/{sample}.log",
23+
params:
24+
extra=r"-R '@RG\tID:{sample}\tSM:{sample}'",
25+
sorting="samtools", # Can be 'none', 'samtools' or 'picard'.
26+
sort_order="coordinate", # Can be 'queryname' or 'coordinate'.
27+
sort_extra="", # Extra args for samtools/picard.
28+
threads: 8
29+
wrapper:
30+
"v5.7.0/bio/bwa/mem"
31+
32+
33+
rule samtools_index:
34+
input:
35+
"sorted_reads/{sample}.bam",
36+
output:
37+
"sorted_reads/{sample}.bam.bai",
38+
log:
39+
"logs/samtools_index/{sample}.log",
40+
params:
41+
extra="", # optional params string
42+
threads: 4 # This value - 1 will be sent to -@
43+
wrapper:
44+
"v5.7.0/bio/samtools/index"
45+
46+
rule bcftools_call:
47+
input:
48+
fa="data/genome.fa",
49+
bam=expand("sorted_reads/{sample}.bam", sample=config["samples"]),
50+
bai=expand("sorted_reads/{sample}.bam.bai", sample=config["samples"]),
51+
output:
52+
"calls/all.vcf",
53+
log:
54+
"logs/bcftools_call/bcftools.log",
55+
shell:
56+
"(bcftools mpileup -f {input.fa} {input.bam} | "
57+
"bcftools call -mv - > {output}) 2> {log}"
58+
59+
60+
rule plot_positions:
61+
input:
62+
rules.bcftools_call.output,
63+
output:
64+
"calls/positions.png",
65+
script:
66+
"scripts/plot-positions.py"
67+
68+
69+
rule plot_quals:
70+
input:
71+
"calls/all.vcf",
72+
output:
73+
"calls/quals.svg",
74+
script:
75+
"scripts/plot-quals.py"

‎setup_creators/tutorial/05_Snakefile_target

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# our samples are pre-configured
2-
SAMPLES = ["A", "B"]
2+
samples = ["A", "B"]
33

44
#TASK: Figure out, which your final target is.
55
# This is the final target of our workflow, so far.
@@ -44,8 +44,8 @@ rule samtools_index:
4444
rule bcftools_call:
4545
input:
4646
fa="data/genome.fa",
47-
bam=expand("sorted_reads/{sample}.bam", sample=SAMPLES),
48-
bai=expand("sorted_reads/{sample}.bam.bai", sample=SAMPLES)
47+
bam=expand("sorted_reads/{sample}.bam", sample=samples),
48+
bai=expand("sorted_reads/{sample}.bam.bai", sample=samples)
4949
output:
5050
"calls/all.vcf"
5151
shell:

‎setup_creators/tutorial/06_Snakefile_run

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# our samples are pre-configured
2-
SAMPLES = ["A", "B"]
2+
samples = ["A", "B"]
33

44
#TASKS: This is an almost working Snakefile!
55
# But it is scrambled and contains a bug.
@@ -36,8 +36,8 @@ rule samtools_sort:
3636
rule bcftools_call:
3737
input:
3838
fa="data/genome.fa",
39-
bam=expand("sorted_reads/{sample}.bam", sample=SAMPLES),
40-
bai=expand("sorted_reads/{sample}.bam.bai", sample=SAMPLES)
39+
bam=expand("sorted_reads/{sample}.bam", sample=samples),
40+
bai=expand("sorted_reads/{sample}.bam.bai", sample=samples)
4141
output:
4242
"calls/all.vcf"
4343
shell:

‎slides/Snakemake_HPC_User_Creator_Combi.tex

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -68,7 +68,7 @@
6868
\include{common/HPC_101}
6969

7070
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
71-
\include{users/Selecting_Workflows}
71+
\include{creators/Selecting_Workflows}
7272

7373
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
7474
\include{creators/Getting_Started}
@@ -102,7 +102,10 @@
102102
\include{common/Reports}
103103

104104
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
105-
\include{common/using_wrappers}
105+
\include{creators/using_wrappers}
106+
107+
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
108+
\include{creators/modularization}
106109

107110
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
108111
\include{common/Contributing}

‎slides/common/HPC_101.tex

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ \section{How does Clustercomputing work?}
66
% source: https://en.m.wikipedia.org/wiki/File:Text-x-python.svg
77
}
88
\frame{
9-
\frametitle{Using Code-Studio}
9+
\frametitle{Running ordinary Batch Scripts}
1010
\begin{mdframed}[tikzsetting={draw=white,fill=white,fill opacity=0.8,
1111
line width=0pt},backgroundcolor=none,leftmargin=0,
1212
rightmargin=150,innertopmargin=4pt,roundcorner=10pt]

‎slides/common/InTouch.tex

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ \section{Staying Up to Date}
3131
\frametitle{Mastodon Announcement Bot II}
3232
\begin{columns}
3333
\begin{column}{0.5\textwidth}
34-
\includegraphics[width=0.8\textwidth]{contact/announcement_robot_example_screenshot.png}\\
34+
\includegraphics[width=0.8\textwidth]{contact/announcement_robot_example_sreenshot.png}\\
3535
Screenshot from Mastodon
3636
\end{column}
3737
\begin{column}{0.5\textwidth}

‎slides/common/software_environment.tex

Lines changed: 13 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -166,7 +166,7 @@ \subsection{Using Conda}
166166
}
167167
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
168168
\begin{frame}[fragile]
169-
\frametitle{Installing Mamba}
169+
\frametitle{Installing Conda}
170170
You \emph{could} run
171171
\begin{lstlisting}[language=Bash, style=Shell, basicstyle=\tiny,breaklines=true ]
172172
$ wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh
@@ -178,16 +178,16 @@ \subsection{Using Conda}
178178
\end{hint}
179179
Instead, please execute the installer script
180180
\begin{lstlisting}[language=Bash, style=Shell]
181-
$ bash install_mamba.sh
181+
$ bash install_conda.sh
182182
\end{lstlisting}
183183
\end{frame}
184184

185185
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
186186
\begin{frame}[fragile]
187-
\frametitle{Mamba - Installation - Part I}
187+
\frametitle{Conda - Installation - Part I}
188188
Start the installation script - if not done:
189189
\begin{lstlisting}[language=Bash, style=Shell]
190-
$ bash install_mamba.sh
190+
$ bash install_conda.sh
191191
\end{lstlisting}
192192
You need to confirm (with ``Enter'')
193193
\begin{itemize}[<+->]
@@ -200,7 +200,7 @@ \subsection{Using Conda}
200200

201201
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
202202
\begin{frame}[fragile]
203-
\frametitle{Installing Mamba - Part II}
203+
\frametitle{Installing Conda - Part II}
204204
\footnotesize
205205
\begin{columns}[t]
206206
\begin{column}{0.5\textwidth}
@@ -290,7 +290,7 @@ \subsection{Using Conda}
290290
\frametitle{Searching Software with Conda v. Mamba}
291291
First you might want to look for software. This is done with
292292
\begin{lstlisting}[language=Bash, style=Shell]
293-
$ mamba search <softwarename>
293+
$ conda search <softwarename>
294294
\end{lstlisting}
295295
\pause
296296
\begin{task}
@@ -299,7 +299,7 @@ \subsection{Using Conda}
299299
\pause
300300
This will list packages with channel and version information, e.\,g.
301301
\begin{lstlisting}[language=Bash, style=Shell, basicstyle=\tiny]
302-
$ mamba search minimap
302+
$ conda search minimap
303303
<snip>
304304
Loading channels: done
305305
# Name Version Build Channel
@@ -344,30 +344,30 @@ \subsection{Using Conda}
344344
\begin{onlyenv}<1|handout:0>
345345
We can create a new environment:
346346
\begin{lstlisting}[language=Bash, style=Shell]
347-
$ mamba @create@ ...
347+
$ conda @create@ ...
348348
\end{lstlisting}
349349
using the \altverb{create} keyword.
350350
\end{onlyenv}
351351
\begin{onlyenv}<2|handout:0>
352352
We can create a new environment:
353353
\begin{lstlisting}[language=Bash, style=Shell]
354-
$ mamba create \
354+
$ conda create \
355355
> ...
356356
\end{lstlisting}
357357
you may write everything in \emph{one} line. The \altverb{\\} breaks a line and \altverb{>} continues it - it is only to fit everything on a slide.
358358
\end{onlyenv}
359359
\begin{onlyenv}<3|handout:0>
360360
We can create a new environment:
361361
\begin{lstlisting}[language=Bash, style=Shell]
362-
$ mamba create \
362+
$ conda create \
363363
> @-n@ snakemake_base ...
364364
\end{lstlisting}
365365
\altverb{-n} denotes the name of the environment.
366366
\end{onlyenv}
367367
\begin{onlyenv}<4|handout:0>
368368
We can create a new environment - by cloning an existing one:
369369
\begin{lstlisting}[language=Bash, style=Shell]
370-
$ mamba create \
370+
$ conda create \
371371
> -n snakemake_base \
372372
> @--clone@ ...
373373
\end{lstlisting}
@@ -376,7 +376,7 @@ \subsection{Using Conda}
376376
\begin{onlyenv}<5|handout:0>
377377
We can create a new environment:
378378
\begin{lstlisting}[language=Bash, style=Shell]
379-
$ mamba create \
379+
$ conda create \
380380
> -c conda-forge -c bioconda \
381381
> --clone @<++course.softwarepath++>@
382382
\end{lstlisting}
@@ -393,7 +393,7 @@ \subsection{Using Conda}
393393
\frametitle{\HandsOn{Activating Environments}}
394394
It is time to activate a shared environment:
395395
\begin{lstlisting}[language=Bash, style=Shell]
396-
$ mamba activate <<++course.softwarepath++>>
396+
$ mamba activate <++course.softwarepath++>
397397
\end{lstlisting}
398398
Your prompt should now feature the name of the environment: \altverb{snakemake}.
399399
\end{frame}

0 commit comments

Comments
 (0)