A CRISPR off-target search is only as good as the sequences it searches
Rebuilding DivRef so CRISPR off-target searches can account for human variation
This post is part of Fulcrum’s series on practical informatics for CRISPR off-target analysis. In a companion post, Tim Dunn describes recent work on Sassy that helps enumerate plausible guide alignments at a locus instead of returning only one preferred answer. DivRef addresses the other side of the same problem, which is what sequence space should be searched in the first place.
Finding short-sequence matches in the human genome is a common problem in bioinformatics. CRISPR guide RNAs are a good example. The guide is designed to match the intended editing site, but close matches elsewhere in the genome can also matter, because the guide could also edit at those sites, causing unintentional effects. To evaluate those possible off-target sites, the search has to include the sequences where those matches could occur.
The standard human reference genome does not contain all of the common sequence variation present across populations. A single alternate allele, or a short haplotype of nearby alleles, can create a match that is not present in the reference assembly. For CRISPR off-target analysis, that means the reference alone is an incomplete search space.
DivRef was built to help with this problem. The resource contains FASTA records for common human haplotypes and variants, plus an index with population allele frequencies and variant metadata. Those FASTA files can be used directly with CRISPR off-target search tools.
I recently rebuilt the DivRef generation workflow because I wanted something more transparent, configurable, and easier to update.
Why rebuild it?
The existing DivRef generation workflow was a collection of standalone Python scripts and a Makefile. Some inputs were hard-coded; others were not fully recorded. That makes it difficult to answer basic questions later: exactly which variants were included, where did they come from, what thresholds were used, and how would the bundle change if those assumptions changed?
For some applications, that may be fine. For off-target workflows that need to account for population variation and make their assumptions visible, it is not enough.
I reimplemented the workflow in Snakemake, wrapping a Python toolkit, with a configuration schema and support for ingesting data from GCS or AWS Open Data. The workflow runs per chromosome, so a full rebuild with default parameters takes less than 20 hours on a laptop.
That makes it practical to rebuild DivRef when the underlying data change, or when a specific scientific application needs different allele frequency thresholds, population sets, or sequence windows. In general, that won’t occur very often, so a rebuild process that takes less than one day (or less on a VM with more resources) is reasonable.
What changed?
The new workflow makes several updates.
The rebuilt workflow now includes gnomAD 4.1 joint exome/genome variants in the single-variant records. The original DivRef documentation said these records were included, but the released resource did not contain them. The rebuild also updates the haplotype computation algorithm, adds chrX haplotypes, flags haplotypes with incompatible variant combinations, and exposes the main assumptions through configuration rather than hard-coded scripts.
For single-variant records, users can also specify additional populations from gnomAD 4.1. Several populations have enough samples in the larger joint dataset to be useful, even though they are underrepresented in the gnomAD 3.1.2 phased subset used to build multi-variant haplotypes.
The haplotype computation change is especially important. The original algorithm used staggered windows to aggregate nearby variants into haplotypes, then split and deduplicated the results. That approach can split true haplotypes across bin boundaries or choose one representative when the same haplotype appears with different allele counts.
The new algorithm walks each sample haplotype in genomic order, cuts parent blocks using a reference-aware gap rule, enumerates contiguous sub-fragments, and counts each parent block once. The result is deterministic and handles indel-driven gaps more cleanly.
The workflow also flags haplotypes with incompatibly phased variants instead of trying to repair them. Phasing is not perfect, especially around repeats, which can yield such cases. When the input data place mutually incompatible variants on the same haplotype, the workflow carries that information forward in the index. A downstream user can decide whether to keep or exclude those records for a particular application.
Key for off-target workflows
A CRISPR off-target search is only as good as the sequences it searches.
If a guide has a plausible off-target match created by a common haplotype, and that haplotype is not represented in the search space, the workflow will never report it. Better alignment logic cannot recover a candidate sequence that was never included.
That is why DivRef and Sassy are complementary. Sassy helps expose plausible alignments at a locus. DivRef helps make sure population-level variation and haplotype sequence are available to search in the first place.
For teams building or maintaining off-target workflows, this is the kind of detail that tends to matter late, after a method has already become part of a larger pipeline. The tool may run, the output may look reasonable, and the assumptions may still be too hidden for the next use case.
A rebuildable DivRef workflow makes those assumptions easier to inspect and change.
Read the technical write-up
This post is the short version. I wrote a more detailed technical walkthrough on GitHub covering the implementation, gnomAD 4.1 inclusion, haplotype computation, incompatible haplotype flags, chrX handling, and runtime details.
Read the full technical write-up on GitHub: https://github.com/fg-labs/divref-wf/blob/main/docs/blog.md
View the workflow: https://github.com/fg-labs/divref-wf/tree/main
Alison Meynert is a Principal Bioinformatics Scientist at Fulcrum Genomics, where she builds bioinformatics tools and pipelines for the genomics community. She previously ran the bioinformatics core facility at Edinburgh's Institute of Genetics and Cancer, where she built rare disease diagnostic pipelines for NHS Scotland. You can find her on LinkedIn and GitHub.
Fulcrum Genomics is a bioinformatics consulting firm built by scientists at the forefront of large-scale genomic research, with deep expertise in sequencing technology, pipeline engineering, and genomic data analysis for biotech, pharma, and academia. Engage us through project-based work, fractional R&D, or hourly consulting. Contact us to discuss your project.