Installing TADbit on GNU/Linux¶
TADbit requires python2 >= 2.6 or python3 >= 3.6 as well as several dependencies that are listed below.
Dependencies¶
Conda¶
Conda (http://conda.pydata.org/docs/index.html) is a package manager, mainly hosting python programs, that is very useful when no root access is available and the softwares have complicated dependencies.
To install it (in case you don’t already have it) just download the installer from http://conda.pydata.org/miniconda.html
%%bash
wget -nv https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh -O miniconda.sh
2018-09-28 00:39:07 URL:https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh [41787735/41787735] -> "miniconda.sh" [1]
And run it with all the default options. The installer will create a
miniconda2 folder in your home directory where all the programs that
you need will be stored (including python).
Alternatively you can also use this oneliner:
%%bash
bash miniconda.sh -b -p $HOME/miniconda2
PREFIX=/home/fransua/miniconda2 installing: python-2.7.15-h1571d57_0 ... installing: ca-certificates-2018.03.07-0 ... installing: conda-env-2.6.0-1 ... installing: libgcc-ng-8.2.0-hdf63c60_1 ... installing: libstdcxx-ng-8.2.0-hdf63c60_1 ... installing: libffi-3.2.1-hd88cf55_4 ... installing: ncurses-6.1-hf484d3e_0 ... installing: openssl-1.0.2p-h14c3975_0 ... installing: yaml-0.1.7-had09818_2 ... installing: zlib-1.2.11-ha838bed_2 ... installing: libedit-3.1.20170329-h6b74fdf_2 ... installing: readline-7.0-h7b6447c_5 ... installing: tk-8.6.8-hbc83047_0 ... installing: sqlite-3.24.0-h84994c4_0 ... installing: asn1crypto-0.24.0-py27_0 ... installing: certifi-2018.8.24-py27_1 ... installing: chardet-3.0.4-py27_1 ... installing: enum34-1.1.6-py27_1 ... installing: futures-3.2.0-py27_0 ... installing: idna-2.7-py27_0 ... installing: ipaddress-1.0.22-py27_0 ... installing: pycosat-0.6.3-py27h14c3975_0 ... installing: pycparser-2.18-py27_1 ... installing: pysocks-1.6.8-py27_0 ... installing: ruamel_yaml-0.15.46-py27h14c3975_0 ... installing: six-1.11.0-py27_1 ... installing: cffi-1.11.5-py27he75722e_1 ... installing: setuptools-40.2.0-py27_0 ... installing: cryptography-2.3.1-py27hc365091_0 ... installing: wheel-0.31.1-py27_0 ... installing: pip-10.0.1-py27_0 ... installing: pyopenssl-18.0.0-py27_0 ... installing: urllib3-1.23-py27_0 ... installing: requests-2.19.1-py27_0 ... installing: conda-4.5.11-py27_0 ... installation finished.
Python 2.7.15 :: Anaconda, Inc.
Python libraries¶
Required:
apt-get install python-scipy
apt-get install python-numpy
Optional packages (but highly recommended):
apt-get install python-matplotlib
Note
Alternative install, you can install python-setuptools
and use easy_install to get these packages
(e.g. “easy_install scipy”).
With conda you can install most of the needed dependencies:
%%bash
## required
conda config --add channels bioconda
conda config --add channels conda-forge
conda create -n tadbit -c conda-forge python=3.7
conda activate tadbit
conda install -y -q -c bioconda mcl
conda install -y -q future
conda install -y -q h5py
conda install -y -q samtools==1.12
conda install -y -q pysam
conda install -y -q matplotlib-base
conda install -y -q -c conda-forge scipy
## optional
conda install -y -q jupyter # this notebook :)
conda install -y -q -c bioconda sra-tools # to download raw data from released experiment
IMP - 3D modeling¶
From the repositories¶
Since version 2.5 IMP is available in several repositories, like Ubuntu
sudo apt-get install imp
or in anaconda
conda install -c https://conda.anaconda.org/salilab imp (python 2 packages)
conda install -c conda-forge imp (python 3 packages)
These options may be easier than the source compilation.
From source¶
MCL - clustering¶
MCL is the program used for clustering the 3D models generated by IMP. It can be downloaded from http://micans.org/mcl/; on Debian/Ubuntu machines it can be automatically installed with:
sudo apt-get install mcl
or in anaconda http://conda.pydata.org/docs/intro.html
conda install -y -q -c bioconda mcl
Note: if the MCL executable is not found by TADbit, an alternative clustering method will be used. Nevertheless we strongly recommend to use MCL.
GEM Mapper¶
The default mapper in TADbit is GEM, but bowtie2 and hisat2 are also supported.
GEM version 3, bowtie2 and hisat2 are available in bioconda.
%%bash
conda -y -q -c bioconda gem3-mapper
conda -y -q -c bioconda bowtie2
conda -y -q -c bioconda hisat2
If you prefer the old good GEM version 2, go to the download page:
https://sourceforge.net/projects/gemlibrary/files/gem-library/Binary%20pre-release%202/
and download the i3 version (the other version is for older
computers, and you usually won’t have to use it).
%%bash
wget -nv -O GEM.tbz2 https://sourceforge.net/projects/gemlibrary/files/gem-library/Binary%20pre-release%203/GEM-binaries-Linux-x86_64-core_i3-20130406-045632.tbz2/download
2019-01-15 14:12:09 URL:https://netcologne.dl.sourceforge.net/project/gemlibrary/gem-library/Binary%20pre-release%203/GEM-binaries-Linux-x86_64-core_i3-20130406-045632.tbz2 [33847526/33847526] -> "GEM.tbz2" [1]
Uncompress the archive:
%%bash
tar -xjvf GEM.tbz2
GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/ GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/gem-indexer_bwt-dna GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/transcriptome-2-genome GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/gem-mappability GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/gem-indexer GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/gem-indexer_generate GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/gem-mappability-retriever GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/External/ GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/External/gemtools GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/External/LICENSE GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/gemtools GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/gem-rna-mapper GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/gem-retriever GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/gem-2-sam GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/gem-indexer_fasta2meta+cont GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/gtf-2-junctions GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/gem-2-wig GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/gem-2-gem GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/gem-info GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/compute-transcriptome GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/LICENSE GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/splits-2-junctions GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/gem-mapper GEM-binaries-Linux-x86_64-core_i3-20130406-045632/man/ GEM-binaries-Linux-x86_64-core_i3-20130406-045632/man/gem-indexer.man GEM-binaries-Linux-x86_64-core_i3-20130406-045632/man/gem-mapper.man GEM-binaries-Linux-x86_64-core_i3-20130406-045632/man/gem-2-sam.man GEM-binaries-Linux-x86_64-core_i3-20130406-045632/man/gem-2-gem.man GEM-binaries-Linux-x86_64-core_i3-20130406-045632/man/gem-mappability.man
And copy the needed binaries to somewhere in your PATH, like:
%%bash
rm -f GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/LICENCE
%%bash
cp GEM-binaries-Linux-x86_64-core_i3-20130406-045632/bin/* ~/miniconda2/bin/
Cleanup
%%bash
rm -rf GEM-binaries-Linux-x86_64-core_i3-20121106-022124
rm -f GEM.tbz2
DryHiC for oneD normalization¶
Install dryhic from: https://github.com/qenvio/dryhic
From an R console type:
install.packages("devtools")
devtools::install_github("qenvio/dryhic")
Or execute this cell:
%%bash
R -e '
install.packages("devtools", repos="http://cran.us.r-project.org"); devtools::install_github("qenvio/dryhic")'
R version 3.4.2 (2017-09-28) -- "Short Summer"
Copyright (C) 2017 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
Natural language support but running in an English locale
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
> install.packages("devtools", repos="http://cran.us.r-project.org"); devtools::install_github("qenvio/dryhic", force=TRUE)
gfortran -fpic -g -O2 -fstack-protector-strong -c fwdb.f -o fwdb.o
gfortran -fpic -g -O2 -fstack-protector-strong -c viterbi.f -o viterbi.o
gcc -std=gnu99 -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -Wl,-z,relro -o dryhic.so fwdb.o viterbi.o -lgfortran -lm -lquadmath -L/usr/lib/R/lib -lR
>
>
Installing package into ‘/home/fransua/R/x86_64-pc-linux-gnu-library/3.4’
(as ‘lib’ is unspecified)
trying URL 'http://cran.us.r-project.org/src/contrib/devtools_1.13.6.tar.gz'
Content type 'application/x-gzip' length 486446 bytes (475 KB)
==================================================
downloaded 475 KB
* installing *source* package ‘devtools’ ...
** package ‘devtools’ successfully unpacked and MD5 sums checked
** R
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
* DONE (devtools)
The downloaded source packages are in
‘/tmp/Rtmp1j1AUG/downloaded_packages’
Downloading GitHub repo qenvio/dryhic@master
from URL https://api.github.com/repos/qenvio/dryhic/zipball/master
Installing dryhic
'/usr/lib/R/bin/R' --no-site-file --no-environ --no-save --no-restore --quiet \
CMD INSTALL '/tmp/Rtmp1j1AUG/devtools6acf4c30139e/qenvio-dryhic-764e0f0' \
--library='/home/fransua/R/x86_64-pc-linux-gnu-library/3.4' --install-tests
* installing *source* package ‘dryhic’ ...
** libs
installing to /home/fransua/R/x86_64-pc-linux-gnu-library/3.4/dryhic/libs
** R
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (dryhic)
DSRC FASTQ compressor¶
DSRC is a FASTQ compressor, it’s not needed, but we use it as the size of the files is significantly smaller than using gunzip (>30%), and, more importantly, the access to them can be parallelized, and is much faster than any other alternative.
It can be downloaded from https://github.com/lrog/dsrc
%%bash
wget -nv http://sun.aei.polsl.pl/dsrc/download/2.0rc/dsrc
2018-09-28 00:46:48 URL:http://sun.aei.polsl.pl/dsrc/download/2.0rc/dsrc [1761768/1761768] -> "dsrc" [1]
%%bash
chmod +x dsrc
And copy to somewhere in your PATH, like:
%%bash
mv dsrc ~/miniconda2/bin/
Chimera - visualization¶
Chimera is a program used for visualization and analysis of molecular structures. It is used in TADbit to visualize the generated 3D models. Chimera is available at: http://www.cgl.ucsf.edu/chimera/
This software is only needed for the visualization of 3D models from inside TADbit.
LiftOver¶
TADbit provides a wrapper for the LiftOver tool [Fujita2011]_ (download it from: http://hgdownload.cse.ucsc.edu/admin/exe/ ).
This can be used to ease the conversion of genomic TAD coordinates (e.g.: to align human TADs with mouse TADs).
Note
A ‘chain’ file may also be downloaded. For example, to convert coordinates to hg19, the chain file needed may be found at: http://hgdownload.cse.ucsc.edu/goldenPath/hg19/liftOver/
TADbit¶
Once all the needed library/software have been installed, TADbit can be downloaded, unpacked and installed as:
wget https://github.com/3DGenomes/tadbit/archive/master.zip -O tadbit.zip
unzip tadbit.zip
cd TADbit-master
python setup.py install
Note
IMP not found problem If you are under debian/Ubuntu machines, and you have followed the quick tutorial proposed here to install IMP, you may encounter some warning message stating that IMP is not installed. This is because root user is not using the PYTHONPATH declared in your bashrc file. You can perfectly ignore the warning, or just proceed like this:
sudo PYTHONPATH=$PYTHONPATH python setup.py install
Finally, run the test script to check that the installation completed successfully.
To do so, move to the test directory and run:
cd test
python test_all.py
Conda builds¶
Alternatively we regularly build a conda package in bioconda. The packages come without IMP and with gem v3 as the default mapper
%%bash
conda create -n tadbit -c conda-forge python=3.7
conda activate tadbit
conda install -c bioconda samtools==1.12 tadbit
If you want to install TADbit with IMP in conda this line will create a environment with them:
%%bash
conda create -n tadbit python=3.7 r-base r-essentials r-devtools imp tadbit -c conda-forge -c bioconda
If you wish then to have the last version of TADbit from github remove the TADbit conda package leaving the dependencies:
%%bash
conda remove tadbit --force
And install TADbit from source as explained before.
