Installing TADbit on GNU/Linux

TADbit requires python2 >= 2.6 or python3 >= 3.6 as well as several dependencies that are listed below.

Dependencies

Conda

Conda (http://conda.pydata.org/docs/index.html) is a package manager, mainly hosting python programs, that is very useful when no root access is available and the softwares have complicated dependencies.

To install it (in case you don’t already have it) just download the installer from http://conda.pydata.org/miniconda.html

%%bash

wget -nv https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh -O miniconda.sh
2018-09-28 00:39:07 URL:https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh [41787735/41787735] -> "miniconda.sh" [1]

And run it with all the default options. The installer will create a miniconda2 folder in your home directory where all the programs that you need will be stored (including python).

Alternatively you can also use this oneliner:

%%bash

bash miniconda.sh -b -p $HOME/miniconda2
PREFIX=/home/fransua/miniconda2
installing: python-2.7.15-h1571d57_0 ...
installing: ca-certificates-2018.03.07-0 ...
installing: conda-env-2.6.0-1 ...
installing: libgcc-ng-8.2.0-hdf63c60_1 ...
installing: libstdcxx-ng-8.2.0-hdf63c60_1 ...
installing: libffi-3.2.1-hd88cf55_4 ...
installing: ncurses-6.1-hf484d3e_0 ...
installing: openssl-1.0.2p-h14c3975_0 ...
installing: yaml-0.1.7-had09818_2 ...
installing: zlib-1.2.11-ha838bed_2 ...
installing: libedit-3.1.20170329-h6b74fdf_2 ...
installing: readline-7.0-h7b6447c_5 ...
installing: tk-8.6.8-hbc83047_0 ...
installing: sqlite-3.24.0-h84994c4_0 ...
installing: asn1crypto-0.24.0-py27_0 ...
installing: certifi-2018.8.24-py27_1 ...
installing: chardet-3.0.4-py27_1 ...
installing: enum34-1.1.6-py27_1 ...
installing: futures-3.2.0-py27_0 ...
installing: idna-2.7-py27_0 ...
installing: ipaddress-1.0.22-py27_0 ...
installing: pycosat-0.6.3-py27h14c3975_0 ...
installing: pycparser-2.18-py27_1 ...
installing: pysocks-1.6.8-py27_0 ...
installing: ruamel_yaml-0.15.46-py27h14c3975_0 ...
installing: six-1.11.0-py27_1 ...
installing: cffi-1.11.5-py27he75722e_1 ...
installing: setuptools-40.2.0-py27_0 ...
installing: cryptography-2.3.1-py27hc365091_0 ...
installing: wheel-0.31.1-py27_0 ...
installing: pip-10.0.1-py27_0 ...
installing: pyopenssl-18.0.0-py27_0 ...
installing: urllib3-1.23-py27_0 ...
installing: requests-2.19.1-py27_0 ...
installing: conda-4.5.11-py27_0 ...
installation finished.
Python 2.7.15 :: Anaconda, Inc.

Python libraries

Required:

apt-get install python-scipy
apt-get install python-numpy

Optional packages (but highly recommended):

apt-get install python-matplotlib

Note

Alternative install, you can install python-setuptools

and use easy_install to get these packages (e.g. “easy_install scipy”).

With conda you can install most of the needed dependencies:

%%bash

## required
conda config --add channels bioconda
conda config --add channels conda-forge
conda create -n tadbit -c conda-forge python=3.7
conda activate tadbit
conda install -y -q -c bioconda mcl
conda install -y -q future
conda install -y -q h5py
conda install -y -q samtools==1.12
conda install -y -q pysam
conda install -y -q matplotlib-base
conda install -y -q -c conda-forge scipy

## optional
conda install -y -q jupyter                                    # this notebook :)
conda install -y -q -c bioconda sra-tools                      # to download raw data from released experiment

IMP - 3D modeling

From the repositories

Since version 2.5 IMP is available in several repositories, like Ubuntu

sudo apt-get install imp

or in anaconda

conda install -c https://conda.anaconda.org/salilab imp (python 2 packages)
conda install -c conda-forge imp (python 3 packages)

These options may be easier than the source compilation.

From source

Check https://integrativemodeling.org/download-linux.html

MCL - clustering

MCL is the program used for clustering the 3D models generated by IMP. It can be downloaded from http://micans.org/mcl/; on Debian/Ubuntu machines it can be automatically installed with:

sudo apt-get install mcl

or in anaconda http://conda.pydata.org/docs/intro.html

conda install -y -q -c bioconda mcl

Note: if the MCL executable is not found by TADbit, an alternative clustering method will be used. Nevertheless we strongly recommend to use MCL.

GEM Mapper

The default mapper in TADbit is GEM, but bowtie2 and hisat2 are also supported.

GEM version 3, bowtie2 and hisat2 are available in bioconda.

%%bash

conda -y -q -c bioconda gem3-mapper
conda -y -q -c bioconda bowtie2
conda -y -q -c bioconda hisat2

If you prefer the old good GEM version 2, go to the download page: https://sourceforge.net/projects/gemlibrary/files/gem-library/Binary%20pre-release%202/ and download the i3 version (the other version is for older computers, and you usually won’t have to use it).

%%bash

wget -nv -O GEM.tbz2 https://sourceforge.net/projects/gemlibrary/files/gem-library/Binary%20pre-release%203/GEM-binaries-Linux-x86_64-core_i3-20130406-045632.tbz2/download
2019-01-15 14:12:09 URL:https://netcologne.dl.sourceforge.net/project/gemlibrary/gem-library/Binary%20pre-release%203/GEM-binaries-Linux-x86_64-core_i3-20130406-045632.tbz2 [33847526/33847526] -> "GEM.tbz2" [1]

Uncompress the archive:

%%bash

tar -xjvf GEM.tbz2
GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/
GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/gem-indexer_bwt-dna
GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/transcriptome-2-genome
GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/gem-mappability
GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/gem-indexer
GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/gem-indexer_generate
GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/gem-mappability-retriever
GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/External/
GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/External/gemtools
GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/External/LICENSE
GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/gemtools
GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/gem-rna-mapper
GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/gem-retriever
GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/gem-2-sam
GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/gem-indexer_fasta2meta+cont
GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/gtf-2-junctions
GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/gem-2-wig
GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/gem-2-gem
GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/gem-info
GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/compute-transcriptome
GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/LICENSE
GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/splits-2-junctions
GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/gem-mapper
GEM-binaries-Linux-x86_64-core_i3-20130406-045632/man/
GEM-binaries-Linux-x86_64-core_i3-20130406-045632/man/gem-indexer.man
GEM-binaries-Linux-x86_64-core_i3-20130406-045632/man/gem-mapper.man
GEM-binaries-Linux-x86_64-core_i3-20130406-045632/man/gem-2-sam.man
GEM-binaries-Linux-x86_64-core_i3-20130406-045632/man/gem-2-gem.man
GEM-binaries-Linux-x86_64-core_i3-20130406-045632/man/gem-mappability.man

And copy the needed binaries to somewhere in your PATH, like:

%%bash

rm -f GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/LICENCE
%%bash

cp GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/* ~/miniconda2/bin/

Cleanup

%%bash

rm -rf GEM-binaries-Linux-x86_64-core_i3-20121106-022124
rm -f GEM.tbz2

DryHiC for oneD normalization

Install dryhic from: https://github.com/qenvio/dryhic

From an R console type:

install.packages("devtools")

devtools::install_github("qenvio/dryhic")

Or execute this cell:

%%bash

R -e '
install.packages("devtools", repos="http://cran.us.r-project.org"); devtools::install_github("qenvio/dryhic")'
R version 3.4.2 (2017-09-28) -- "Short Summer"
Copyright (C) 2017 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> install.packages("devtools", repos="http://cran.us.r-project.org"); devtools::install_github("qenvio/dryhic", force=TRUE)
gfortran   -fpic  -g -O2 -fstack-protector-strong  -c fwdb.f -o fwdb.o
gfortran   -fpic  -g -O2 -fstack-protector-strong  -c viterbi.f -o viterbi.o
gcc -std=gnu99 -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -Wl,-z,relro -o dryhic.so fwdb.o viterbi.o -lgfortran -lm -lquadmath -L/usr/lib/R/lib -lR
>
>
Installing package into ‘/home/fransua/R/x86_64-pc-linux-gnu-library/3.4’
(as ‘lib’ is unspecified)
trying URL 'http://cran.us.r-project.org/src/contrib/devtools_1.13.6.tar.gz'
Content type 'application/x-gzip' length 486446 bytes (475 KB)
==================================================
downloaded 475 KB

* installing *source* package ‘devtools’ ...
** package ‘devtools’ successfully unpacked and MD5 sums checked
** R
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
* DONE (devtools)

The downloaded source packages are in
    ‘/tmp/Rtmp1j1AUG/downloaded_packages’
Downloading GitHub repo qenvio/dryhic@master
from URL https://api.github.com/repos/qenvio/dryhic/zipball/master
Installing dryhic
'/usr/lib/R/bin/R' --no-site-file --no-environ --no-save --no-restore --quiet  \
  CMD INSTALL '/tmp/Rtmp1j1AUG/devtools6acf4c30139e/qenvio-dryhic-764e0f0'  \
  --library='/home/fransua/R/x86_64-pc-linux-gnu-library/3.4' --install-tests

* installing *source* package ‘dryhic’ ...
** libs
installing to /home/fransua/R/x86_64-pc-linux-gnu-library/3.4/dryhic/libs
** R
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (dryhic)

DSRC FASTQ compressor

DSRC is a FASTQ compressor, it’s not needed, but we use it as the size of the files is significantly smaller than using gunzip (>30%), and, more importantly, the access to them can be parallelized, and is much faster than any other alternative.

It can be downloaded from https://github.com/lrog/dsrc

%%bash

wget -nv http://sun.aei.polsl.pl/dsrc/download/2.0rc/dsrc
2018-09-28 00:46:48 URL:http://sun.aei.polsl.pl/dsrc/download/2.0rc/dsrc [1761768/1761768] -> "dsrc" [1]
%%bash

chmod +x dsrc

And copy to somewhere in your PATH, like:

%%bash

mv dsrc ~/miniconda2/bin/

Chimera - visualization

Chimera is a program used for visualization and analysis of molecular structures. It is used in TADbit to visualize the generated 3D models. Chimera is available at: http://www.cgl.ucsf.edu/chimera/

This software is only needed for the visualization of 3D models from inside TADbit.

LiftOver

TADbit provides a wrapper for the LiftOver tool [Fujita2011]_ (download it from: http://hgdownload.cse.ucsc.edu/admin/exe/ ).

This can be used to ease the conversion of genomic TAD coordinates (e.g.: to align human TADs with mouse TADs).

Note

A ‘chain’ file may also be downloaded. For example, to convert coordinates to hg19, the chain file needed may be found at: http://hgdownload.cse.ucsc.edu/goldenPath/hg19/liftOver/

TADbit

Once all the needed library/software have been installed, TADbit can be downloaded, unpacked and installed as:

wget https://github.com/3DGenomes/tadbit/archive/master.zip -O tadbit.zip
unzip tadbit.zip
cd TADbit-master
python setup.py install

Note

IMP not found problem If you are under debian/Ubuntu machines, and you have followed the quick tutorial proposed here to install IMP, you may encounter some warning message stating that IMP is not installed. This is because root user is not using the PYTHONPATH declared in your bashrc file. You can perfectly ignore the warning, or just proceed like this:

sudo PYTHONPATH=$PYTHONPATH python setup.py install

Finally, run the test script to check that the installation completed successfully.

To do so, move to the test directory and run:

cd test
python test_all.py

Conda builds

Alternatively we regularly build a conda package in bioconda. The packages come without IMP and with gem v3 as the default mapper

%%bash
conda create -n tadbit -c conda-forge python=3.7
conda activate tadbit
conda install -c bioconda samtools==1.12 tadbit

If you want to install TADbit with IMP in conda this line will create a environment with them:

%%bash

conda create -n tadbit python=3.7 r-base r-essentials r-devtools imp tadbit -c conda-forge -c bioconda

If you wish then to have the last version of TADbit from github remove the TADbit conda package leaving the dependencies:

%%bash

conda remove tadbit --force

And install TADbit from source as explained before.