Programs

Main CLI

These are the core programs of BPReveal. Each one takes a JSON configuration file.

Before you train

prepareBed: Given a set of regions and data tracks, reject regions that have too few (or too many) reads, or that have unmapped bases in the genome.
prepareTrainingData: Takes in bed and bigwig files and a genome, and generates an hdf5-format file containing the samples used for training.

Training your models

The first step is always trainSoloModel. If you’re not doing fancy bias correction, then it’s also the last step. trainTransformationModel and trainCombinedModel are just for if you’re doing ChromBPnet-style bias removal.

trainSoloModel: Takes in a training input configuration and trains up a model to predict the given data, with no bias correction. Saves the model to disk, along with information from the training phase.
trainTransformationModel: Takes in a bias (i.e., solo) model and the actual experimental (i.e., biology + bias) data. Derives a relation to best fit the bias profile onto the experimental data. Saves a new model to disk, adding a simple layer or two to do the regression.
trainCombinedModel: Takes a transformation model and experimental data and builds a model to explain the residuals. Saves both a combined model and the residual model alone to disk.

Working with models

These are the tools that you’ll use after you’ve trained your models.

makePredictions: Takes a trained model (solo, combined, residual, or even transformation models work) and predicts over the given regions or sequences.
interpretFlat: Generates shap scores of the same type as BPNet. Hypothetical contributions for each base are written to a modisco-compatible h5.
interpretPisa: Runs an all-to-all shap analysis on the given bed regions or fasta sequences.
makePisaFigure: Generates a handsome-looking PISA graph or plot. This is a thin wrapper around plotting.pisaPlot and plotting.pisaGraph.

Looking for motifs

After you run MoDISco, you can use BPReveal to scan for motif instances in the genome.

motifScan: Scan the genome for patterns of contribution scores that match motifs identified by modiscolite.
motifSeqletCutoffs: Loads the output from modiscolite and calculates cutoff values to use during motif scanning.

Utility CLI

These are little tools and utilities that help in dealing with models. These take arguments on the command line.

Before you train

checkJson: Take a json file and make sure that it’s valid input for one of the BPReveal programs. Can also be used to identify which BPReveal program a json belongs to.
lengthCalc: Given the parameters of a network, like input filter width, number of layers &c., determine the input width or output width.

Training your models

showTrainingProgress: Read in the log files generated by the training programs (when verbosity is INFO or DEBUG) and show you how well the model’s doing in real time.
showModel: (DEPRECATED, will be removed in 6.0.0) Make a pretty picture of your model.
makeLossPlots: Once you’ve trained a model, you can run this on the history file to get plots of all of the components of the loss.

Working with models

predictToBigwig: Takes the hdf5 file generated by the predict step and converts one track from it into a bigwig file.
shapToBigwig: Converts a shap hdf5 file (from interpretFlat) into a bigwig track for visualization.
shapToNumpy: Takes the interpretations from interpretFlat and converts them to numpy arrays that can be read in by modiscolite.
metrics: Calculates a suite of metrics about how good a model’s predictions are.

Looking for motifs

motifAddQuantiles: Takes the output from motifScan and adds quantile information for determining how good your motif matches were.

API

These are Python libraries that do most of the heavy lifting, and can be imported to do useful things in your code.

bedUtils: Useful functions for manipulating bed files, particularly for tiling the genome with regions and calculating metapeaks over very large data sets.
gaOptimize: Tools for evolving sequences that lead to desired profiles. It implements a genetic algorithm that supports insertions and deletions. You can also use the Organism class on its own to apply mutations to sequences. These mutations can include insertions and deletions.
interpretUtils: Functions for getting interpretation scores. Contains a streaming system for calculating PISA and flat importance scores. You should not normally need to interact with this module. Instead, use interpretFlat, interpretPisa, or easyInterpretFlat.
jaccard: Contains wrappers around C functions that calculate the sliding Jaccard similarity used to scan for motifs. You almost certainly don’t need to use this.
logUtils: Functions used to log information. It’s basically TensorFlow’s wrapper around the logging module in the standard library. You probably don’t need to use the logging functions yourself, but you may want to use the setVerbosity and setBooleanVerbosity functions.
motifUtils: Functions for dealing with motif scanning and modisco files. You probably don’t need to use this directly.
plotting: Utilities for making high-quality plots of your results. For PISA, you will probably want to use plotPisa or plotPisaGraph. For MoDISco results, there’s plotModiscoPattern.
schema: A set of JSON schemas that validate the inputs to the BPReveal programs. These are used to make sure that incorrect inputs trigger errors early, and that those errors are clearer to the user. You do not need to use this.
training: A very simple module that actually runs the training loop for trainSoloModel, trainTransformationModel, and trainCombinedModel. You should not need to use this directly.
ushuffle: A wrapper around the ushuffle library, used to perform shuffles of sequences that preserve k-mer distributions.
utils: Contains general-use utilities and a high-performance tool to generate predictions for many sequences.

Useful API features

Much of the BPReveal API is dedicated to supporting the CLI tools and a typical user won’t need to interact with it. But there are a few functions here and there that you might find helpful. Here are a few you should know about.

Data processing

To tile the genome with regions, you can use bedUtils.makeWhitelistSegments and bedUtils.tileSegments, or you can use bedUtils.createTilingRegions, which just wraps the two former functions.

For bed intervals, you can resize them with bedUtils.resize.

For working with bigwigs, you can use utils.loadChromSizes, utils.blankChromosomeArrays, and utils.writeBigwig to easily write data to a new bigwig file.

You can use bedUtils.metapeak to get the average profile over many regions, which is useful for plotting.

Making predictions

If you want to do this the easy way, use the Easy function, utils.easyPredict. This function will load up a model, make predictions, and then give you the profiles. It also cleans up after itself and releases the GPU.

For more intense predictions, or if you need the raw model outputs, use utils.ThreadedBatchPredictor. This spawns background threads that can run predictions at blinding speed, with multiple processes sharing the GPU for maximum throughput. This class supports streaming data, so you can make terabytes of predictions and process them as they come, letting your program run with a minimal memory footprint.

If you have model outputs (logits and logcounts) and want a predicted profile, use utils.logitsToProfile.

To efficiently convert DNA sequences to and from one-hot-encoded form, use utils.oneHotEncode and utils.oneHotDecode. These functions are optimized and can perform their calculations far faster than a naive implementation with dictionary lookups.

For applying mutations to sequences, I suggest using the Organism class in the gaOptimize module. While it is designed to be part of a genetic algorithm optimization, it can easily be used on its own to apply corruptors (called “corruptors” to avoid confusion with the genetic algorithm operation called “mutation”) to a single sequence.

Getting importance scores

If the interpretFlat CLI tool doesn’t do what you need, you can use utils.easyInterpretFlat to get importance scores. If you need something even more custom, you’ll have to wade through the arcane and complex interpretUtils module and I’m sorry for you.

Working with motifs

The motifUtils module contains helpers for working with Modisco pattern objects. Typically, you create a motifUtils.Pattern object and then call loadCwm and then loadSeqlets to load in the relevant data. Just about the only time you’d need to create a Pattern object is to plot it.

Showing off your results

There are a bunch of nifty tools for making high-quality plots in the plotting package. You can make PISA plots, PISA graph plots, and motif summary plots.

Tools

These are miscellaneous programs that are not part of BPReveal proper, but that I have found useful. They are not actively maintained, and tend to have subpar documentation.