uap – Robust, Consistent, and Reproducible Data Analysis

Authors:

Christoph Kämpf, Michael Specht, Sven-Holger Puppel, Alexander Scholz, Gero Doose, Kristin Reiche, Jana Hertel, Jörg Hackermüller

Description:

uap executes, controls and keeps track of the analysis of large data sets. It enables users to perform robust, consistent, and reproducible data analysis. uap encapsulates the usage of (bioinformatic) tools and handles data flow and processing during an analysis. Users can use predefined or self-made analysis steps to create custom analysis. Analysis steps encapsulate best practice usages for bioinformatic software tools. uap focuses on the analysis of high-throughput sequencing (HTS) data. But its plugin architecture allows users to add functionality, such that it can be used for any kind of large data analysis.

Usage:

uap is a command-line tool, implemented in Python. It requires a user-defined configuration file, which describes the analysis, as input.

Supported Platforms:

• Unix-like operating systems.
• High Performance Compute (HPC) cluster systems such as UGE, OGE/SGE and SLURM.
• see Tested Platforms for detailed information

Important Information

uap does NOT include all tools necessary for the data analysis. It expects that the required tools are already installed.

Table of contents

• Introducing uap
  • Core aspects
    • Robustness
    • Consistency
    • Reproducibility
    • Usability
• Software Design
  • Source and Processing Steps: Building Blocks of the Analysis
  • Runs: Atomic Units of the Analysis
  • Connections: Propagation of Data
  • Analysis as Directed Acyclic Graph
• Recommended uap Workflow
• Quick Start uap
  • Handle Genomic Data
  • Sequencing Data Analysis
    • General Steps
    • RNAseq Example – Reanalysing Data from Mercer et al., Nature Protoc. (2014)
    • ChIPseq Example – Reanalysing Data from Barski et al., Cell (2007)
• Installation of uap
  • Prerequisites
  • Downloading the Software
  • Setting Up Python Environment
  • Making uap Globally Available
• Analysis Configuration File
  • Sections of a Configuration File
    • destination_path Section
    • constants Section
    • steps Section
    • tools Section
    • cluster Section
  • Example Configurations
• Cluster Configuration File
  • Submit Script Template
• Command-Line Usage of uap
  • Subcommands
    • steps Subcommand
    • status Subcommand
    • run-info Subcommand
    • run-locally Subcommand
    • submit-to-cluster Subcommand
    • fix-problems Subcommand
    • volatilize Subcommand
    • render Subcommand
• Add New Functionality
  • Implement New Steps
    • Step 1: Import Statements and Logger
    • Step 2: Class Definition
    • Step 3: __init__ Method
    • Step 4: runs Method
    • Step 5: Add the new step to uap
  • Best Practices
    • Usage of dd and mkfifo
• Tested Platforms
• Annotation Files
  • known_paths
• Available steps
  • Source steps
    • bcl2fastq2_source
    • bcl2fastq_source
    • fastq_source
    • fetch_chrom_sizes_source
    • raw_file_source
    • raw_file_sources
    • raw_url_source
    • raw_url_sources
    • run_folder_source
  • Processing steps
    • bam_to_bedgraph_and_bigwig
    • bam_to_genome_browser
    • bowtie2
    • bowtie2_generate_index
    • bwa_backtrack
    • bwa_generate_index
    • bwa_mem
    • chromhmm_binarizebam
    • chromhmm_learnmodel
    • cuffcompare
    • cufflinks
    • cuffmerge
    • cutadapt
    • deepTools_bamCompare
    • deepTools_bamPEFragmentSize
    • deepTools_multiBamSummary
    • deepTools_plotFingerprint
    • discardLargeSplitsAndPairs
    • fastqc
    • fastx_quality_stats
    • fix_cutadapt
    • hisat2
    • htseq_count
    • macs2
    • merge_assembly
    • merge_fasta_files
    • merge_fastq_files
    • merge_numpy_zip_arrays
    • pear
    • pepr
    • pepr_postprocess
    • picard_add_replace_read_groups
    • picard_markduplicates
    • picard_merge_sam_bam_files
    • post_cufflinksSuite
    • preseq_complexity_curve
    • preseq_future_genome_coverage
    • preseq_future_yield
    • reformatCigar
    • remove_duplicate_reads_runs
    • rgt_thor
    • rseqc
    • s2c
    • sam_to_sorted_bam
    • samtools
    • samtools_faidx
    • samtools_index
    • samtools_merge
    • samtools_sort
    • samtools_stats
    • segemehl
    • segemehl_2017
    • segemehl_generate_index
    • segemehl_generate_index_bisulfite
    • sra_fastq_dump
    • stringtie
    • stringtie_merge
    • stringtie_prepDE
    • subsetMappedReads
    • tophat2
    • trim_galore
    • trimmomatic
• API documentation
  • Pipeline-specific modules
    • abstract_step
    • pipeline
    • run
    • task
  • Miscellaneous modules
    • process_pool
    • fscache
    • misc
• Information for uap Developers
  • Documentation
    • Create Documentation Locally
    • Create Documentation on RTD
  • Continuous Integration
    • .travis.yml

Remarks

This documentation has been created using sphinx and reStructuredText.

Indices and tables

• Index
• Module Index
• Search Page