# ===========================================================================
#
#                            PUBLIC DOMAIN NOTICE
#               National Center for Biotechnology Information
#
#  This software/database is a "United States Government Work" under the
#  terms of the United States Copyright Act.  It was written as part of
#  the author's official duties as a United States Government employee and
#  thus cannot be copyrighted.  This software/database is freely available
#  to the public for use. The National Library of Medicine and the U.S.
#  Government have not placed any restriction on its use or reproduction.
#
#  Although all reasonable efforts have been taken to ensure the accuracy
#  and reliability of the software and data, the NLM and the U.S.
#  Government do not and cannot warrant the performance or results that
#  may be obtained by using this software or data. The NLM and the U.S.
#  Government disclaim all warranties, express or implied, including
#  warranties of performance, merchantability or fitness for any particular
#  purpose.
#
#  Please cite the author in any work or product based on this material.
#
# ===========================================================================


The NCBI SRA ( Sequence Read Archive )


Contact: sra-tools@ncbi.nlm.nih.gov
http://trace.ncbi.nlm.nih.gov/Traces/sra/std


Stand-alone BLAST searches against SRA runs in their native format.
-------------------------------------------------------------------

A stand-alone blastn application to perform BLAST searches directly against
native SRA files is included in this distribution. This application has been
tested in-house at the NCBI, but has not been heavily used, so this should be
considered a preliminary (alpha) release to a few experienced users. A 64-bit
LINUX application has been built for this testing.

The application is called "blastn_vdb".

The application can be invoked in much the same manner as the standard
blastn application:

1) blastn_vdb -help or blastn_vdb -h will produce usage messages.

2) The BLAST+ command-line manual at http://www.ncbi.nlm.nih.gov/books/NBK1763/
provides more details on the options, though not all blastn options are
available with blastn_vdb. Some options simply do not apply to sequences in SRA
(e.g., -gilist is missing as these sequences have not been assigned GI's). Some
options have not yet been implemented (e.g., -num_threads is currently disabled).


To search cached or on-demand SRA objects.
------------------------------------------
An example blastn_vdb command-line would be:

./blastn_vdb -db "ERR039542 ERR047215 ERR039539 ERR039540" -query nt.test -out test.out

The file nt.test contains the query in FASTA format, and it will be searched against
the reads in runs with accessions ERR039542 ERR047215 ERR039539 ERR039540.

If you have not already downloaded these objects using the vdb "prefetch" tool,
they will be retrieved on-demand from NCBI under standard configuration. For
alternative configuration information, please see the "README-vdb-config" file
in this distribution.

Searching with manually downloaded files.
-----------------------------------------
If you have manually downloaded files, e.g. via aspera or wget, etc., they may
be referred to as "local" files. You can pass one or more file paths to be used
collectively as the database. In this case the blastn_vdb command-line would be:

./blastn_vdb -db <SRR_file> -query <input_file> -out <output_file>

Where
<SRR_file> is the path (relative or absolute) and name of the SRRxxxxx file
<input_file> is a fasta file containing the sequence(s) to be BLASTed
<output_file> is the name specified for the output report of the blast search.

Example:

./blastn_vdb -db ./subdir/ERR039542.sra -query nt.test -out test.out

Querying multiple SRR files simultaneously:

./blastn_vdb -db "<SRR_file1> <SRR_file2> <SRR_file3>" -query <input.fa> -out <output_file>

Enclose the group of files to be included in the search set in "quotes", e.g.
"./SRR_file1.sra ./SRR_file2.sra ./SRR_file3.sra"

Example:

./blastn_vdb -db "./ERR039542.sra ./ERR047215.sra ./ERR039539.sra ./ERR039540.sra" -query nt.test -out test.out

Caveats
-------
There are some limitations on the currently available application:

1) Individual SRA data files containing more than 2 billion reads are not yet supported. For a
paired-end experiment this is actually a limitation of about 1 billion "spots".

2) Compressed SRA ("cSRA") is not yet fully supported. Currently, only the
unaligned fraction of reads are searched. Compressed SRA are runs containing
alignments (e.g., ERR230455). Runs can be checked with "vdb-dump" to report if
they contain alignment information:

    $ vdb-dump -E ERR230455
    enumerating the tables of database 'ERR230455'
    tbl #1: PRIMARY_ALIGNMENT
    tbl #2: REFERENCE
    tbl #3: SEQUENCE

3) You may need to prefix "./" to the run name for files in your current
directory.

4) The blast_formatter is not currently able to read native SRA files, so
reformatting of results saved as a blast archive is not yet supported.

Common errors and fixes.
------------------------

1) Failure to provide relative path to manually downloaded SRR file:

./blastn_vdb -db SRR770754.sra -query srr770754_test.fa -out test.out
Error: NCBI C++ Exception:
    "vdb2blast_util.cpp", line 253: Error: ncbi::CVDBBlastUtil::x_MakeSRASeqSrc()
    - VDB BlastSeqSrc construction failed: Failed to add any run to VDB runset: unsupported while allocating

Fix:
Include relative (e.g., "../" or "./") or absolute (e.g., "/home/user/SRA_BLAST_data/") file path with SRR file

# ===========================================================================
#
#                            PUBLIC DOMAIN NOTICE
#               National Center for Biotechnology Information
#
#  This software/database is a "United States Government Work" under the
#  terms of the United States Copyright Act.  It was written as part of
#  the author's official duties as a United States Government employee and
#  thus cannot be copyrighted.  This software/database is freely available
#  to the public for use. The National Library of Medicine and the U.S.
#  Government have not placed any restriction on its use or reproduction.
#
#  Although all reasonable efforts have been taken to ensure the accuracy
#  and reliability of the software and data, the NLM and the U.S.
#  Government do not and cannot warrant the performance or results that
#  may be obtained by using this software or data. The NLM and the U.S.
#  Government disclaim all warranties, express or implied, including
#  warranties of performance, merchantability or fitness for any particular
#  purpose.
#
#  Please cite the author in any work or product based on this material.
#
# ===========================================================================


The tool 'vdb-config' can be used to inspect or change the configuration
of the sra-toolkit.

When called without any parameters the tool reports the current configuration
in xml-format. No changes are made.

-----------------------------------------------------------------------------

vdb-config --restore-defaults

If called with this parameter the tool will bring the configuration into
default state.

-----------------------------------------------------------------------------

vdb-config -i

This will present the user with a colored configuration dialog.

The tab-key and the cursor-keys navigate the dialog. The item with the little
red square has the focus. A button or a checkbox can be 'pressed' with the
space or enter-key. To get out of the dialog without saving any changes
press the '6'-key or the 'q'-key or navigate to the 'exit'-button at the
bottom of the dialog and press the space or enter-key.


The "data source" part:

The "NCBI SRA" labeled checkbox enables/disables remote access to the SRA-
accession stored at NCBI. As long as the computer has internet-access and this
checkbox is enabled the user can access SRA-accessions directly without
downloading them.

A command like 'sra-pileup SRR341578' at the command-line will produce pileup
output of the given accession even if this accession has not been downloaded
before.
The tool will download the data on the fly from our servers.


There might be a checkbox labeled "site" below the "NCBI-SRA" one. If this
checkbox is not available you do not have a 'site'-installation of SRA-data.
If it is visible you do have such a site-installation and you can disable
access to this data.


The "local workspaces" part:

At the top are 2 buttons "import dbGaP-project" and "set default import path".

If you are not using dbGaP-projects (The database of Genotypes and Phenotypes)
you can ignore these 2 buttons.

The "import dbGaP-project" button presents you with another dialog to select
a ngc-file. You can navigate the directories of your computer to find and
select one of these files. By default the focus is in the files-list. It may
be empty.
Use the cursor-key: 'up' to focus the 'directories'-list. If you press enter
on any of the listed directory-names you change into this directory.
The '[ .. ]' entry brings you back into the parent directory. If you see
ngc-files in the lower 'files'-list press the tab-key to switch to the
'files'-list. Press enter on one of them to select this file for import. You
will see a success-message if the import was performed without errors.
On Windows you cannot switch from one drive-letter
to another when selecting.

The "set default import path" gives you the opportunity to specify a different
default location for dbGaP-projects - for instance if your home directory is
not big enough. You can always change the location for your dbGaP-project
after the import.


Below the 2 buttons is a list of local repositories. If there are no
dbGaP-projects this list has only one entry "Open Access Data". This is the
location where accessions get downloaded and cached. You can change these
locations if for instance your home directory where they are created by
default does not have enough space.
The change button brings up a directory-select dialog.

If you made any changes like enabling/disabling or changed a location, the
change is only written to the configuration if you exit the dialog via the
'save and exit' button.

-----------------------------------------------------------------------------


vdb-config -i --interactive-mode textual

This will present the user with a purely textual and sequential dialog. It is
intended to be used if the colored mode does not work, maybe because of
console issues.

# The NCBI SRA (Sequence Read Archive)

### Contact:
email: sra@ncbi.nlm.nih.gov

### Download
Visit our [download page](https://github.com/ncbi/sra-tools/wiki/01.-Downloading-SRA-Toolkit) for pre-built binaries.

### Change Log
Please check the CHANGES.md file for change history.

## The SRA Toolkit
The SRA Toolkit and SDK from NCBI is a collection of tools and libraries for
using data in the INSDC Sequence Read Archives.

### ANNOUNCEMENT:
NIH has released a request for information (RFI) to solicit community feedback on new proposed Sequence Read Archive (SRA) data formats. Learn more and share your thoughts at https://go.usa.gov/xvhdr. The response deadline is July 17th, 2020. We’d encourage you all to share with your colleagues and networks, and respond if you are an SRA submitter or data user.

SRA Toolkit 2.11.0 March 15, 2021

  **fasterq-dump**: does not exit with 0 any more if the given path is not found
  **fasterq-dump**: does not exit with 0 if accession is not found
  **fasterq-dump**: does not fail when requested to dump a run file with non-standard name
  **fasterq-dump**: available on windows
  **kfg, prefetch, vfs**: resolve WGS reference sequences into "Accession Directory"
  **kfg, sra-tools, vfs**: dropped support of protected repositories
  **kns, sra-tools**: fixed formatting of HTTP requests for proxy
  **ncbi-vdb, ngs, ngs-tools, sra-tools, vdb**: added support for 64-bit ARM (AArch64, Apple Silicon)
  **prefetch, vfs**: fixed download of protected non-run files
  **prefetch, vfs**: fixed segfault during download of JWT cart
  **prefetch, vfs**: respect requested version when downloading WGS files
  **sra-pileup**: now silent if requested slice has no alignments or reference-name does not exist
  **sratools**: added description and documentation of the sratools driver tool to GitHub wiki
  **sra-tools**: created a script to fix names of downloaded sra files
  **sra-tools**: created a script to move downloaded sra run files into proper directories
  **sratools**: disable-multithreading option removed from help text for tools that do not support it
  **sratools**: does not access remote repository when it is disabled
  **sra-tools, vfs**: recognize sra file names with version
  **vdb-dump**: exits with no-zero value if asked for non existing column

SRA Toolkit 2.10.8

kproc, fasterq-dump: fixed problem with seg-faults caused by too small stack used by threads
kdbmeta: allow to work with remote runs
kdb, vdb, vfs, sra-tools: fixed bug preventing use of path to directory created by prefetch if it ends with '/'
vfs, sra-tools, ngs-tools: report an error when file was encrypted for a different ngc file
prefetch: print error message when cannot resolve reference sequence
vfs, prefetch: download encrypted phenotype files with encrypted extension
vdb, sra-docker: config can auto-generate LIBS/GUID when in a docker container

SRA Toolkit 2.10.5
sratools: fixed a potential build problem in libutf8proc
ncbi-vdb, ngs, ngs-tools, sra-tools: all Linux builds now use g++ 7.3 (C++11 ABI)
prefetch: improvements were made to work in environments with bad network connections
prefetch, sratools: fixed the names of the --min-size and --max-size command line arguments when running prefetch

SRA Toolkit 2.10.4
kns, sra-tools:: fixed errors when using ngc file

SRA Toolkit 2.10.3
sraxf, fasterq-dump, fastq-dump, sam-dump: fixed a problem resulting in a segmentation fault

Release 2.10.2 of `sra-tools` provides access to all the **public and controlled-access dbGaP** of SRA in the AWS and GCP environments _(Linux only for this release)_. This vast archive's original submission format and SRA-formatted data can both be accessed and computed on these clouds, eliminating the need to download from NCBI FTP as well as improving performance.

The `prefetch` tool also retrieves **original submission files** in addition to ETL data for public and controlled-access dbGaP data.

With release 2.10.0 of `sra-tools` we have added cloud-native operation for AWS and GCP environments _(Linux only for this release)_, for use with the public SRA. `prefetch` is capable of retrieving original submission files in addition to ETL data.

With release 2.9.1 of `sra-tools` we have finally made available the tool `fasterq-dump`, a replacement for the much older `fastq-dump` tool. As its name implies, it runs faster, and is better suited for large-scale conversion of SRA objects into FASTQ files that are common on sites with enough disk space for temporary files. `fasterq-dump` is multi-threaded and performs bulk joins in a way that improves performance as compared to `fastq-dump`, which performs joins on a per-record basis _(and is single-threaded)_.

`fastq-dump` is still supported as it handles more corner cases than `fasterq-dump`, but it is likely to be deprecated in the future.

You can get more information about `fasterq-dump` in our Wiki at [https://github.com/ncbi/sra-tools/wiki/HowTo:-fasterq-dump](https://github.com/ncbi/sra-tools/wiki/HowTo:-fasterq-dump).

For additional information on using, configuring, and building the toolkit,
please visit our [wiki](https://github.com/ncbi/sra-tools/wiki)
or our web site at [NCBI](http://www.ncbi.nlm.nih.gov/Traces/sra/?view=toolkit_doc)


SRA Toolkit Development Team