Benchmarking is not validation
Clinical genomics runs on research-grade code. We need to validate it.
Clinical genomics runs on software that was usually built for research first.
That is not an insult. Research code has carried this field a very long way. Many of the tools we use every day were written by small groups, often underfunded, often in academic labs, usually to answer a scientific question before anyone knew the tool would become infrastructure. Some of those tools now sit underneath diagnostic pipelines, clinical trials, drug development programs, and patient reports.
The clinical assays built on top of that software are validated. The shared tools underneath them are more often benchmarked, trusted, and left alone until something breaks.
That gap is getting harder to defend.
Benchmarking asks whether a tool performed well on a defined dataset at a point in time. Verification asks whether a tool continues to do what it claims to do, across versions, inputs, and edge cases. Clinical validation asks whether the result is fit for the medical use in front of it.
Each activity is an attempt to answer a different question. Genomics has spent a lot of time benchmarking and then acting as though the other two activities come along for free, when they do not.
A mutation can depend on the software that reads it
Take FLT3 internal tandem duplications in acute myeloid leukemia. FLT3-ITDs are clinically important because they can affect risk stratification and treatment decisions. They are also awkward for short-read sequencing.
An ITD is an insertion created when a stretch of DNA is copied head-to-tail. Reads that span that kind of event do not always align cleanly to the reference genome. Depending on the aligner, the extra sequence may be represented as an insertion, or it may be soft-clipped and set aside as unmatched sequence.
That choice can decide whether the downstream mutation caller sees the alteration.
In one evaluation, BWA missed 85 bp and 95 bp duplications that another aligner detected because the signal was pushed into soft-clipped reads instead of aligned as the duplicated sequence. Same sample. Different aligner behavior. Different downstream visibility.
Getting that call right is not academic. FLT3-ITD status gates targeted therapy (quizartinib is approved specifically for FLT3-ITD AML) so a miscalled ITD can change the drugs a patient is offered. And the result is more than present-or-absent: the duplication’s length, position, polyclonality, and allelic burden are the annotation a lab relies on for interpretation and for tracking residual disease. That is the deeper version of the alignment problem above: when the duplicated sequence is soft-clipped instead of represented as an insertion, you don’t only risk missing the variant, you lose the metadata needed to characterize it.
We have seen versions of this problem in client work. We have built custom FLT3-ITD callers for multiple groups because there is no generic tool I trust to get this right everywhere. The right approach depends on the sequencing platform, read length, panel design, coverage profile, homopolymer behavior, aligner behavior, and caller assumptions.
That is the part people underestimate. A clinical result can depend on a default chosen years earlier by someone solving a different problem.
Nobody set out to make that fragile. It is what happens when tools are benchmarked once and then treated as fixed infrastructure.
What a benchmark can miss
A benchmark is useful. I like benchmarks. I have written plenty of them.
But a benchmark is a measurement. It does not give you a specification. It does not tell you what happens when two inputs tie, when clipping lands on a boundary, when quality encodings drift, when a dependency changes, when a tool is run twice on the same file, or when a new architecture exposes an assumption that was invisible on x86.
For clinical bioinformatics, that is not a minor distinction.
The assay gets validated end to end because the lab has to validate it. The tool underneath may never be re-verified unless the lab changes something obvious enough to trigger the work. Every lab repeats some version of that burden in isolation, while the shared tool itself often carries less validation infrastructure than the workflows built on top of it.
This is one reason maintenance in bioinformatics feels so broken. The field depends on shared software, but the cost of keeping that software reliable lands in odd places. A lab validates around it. A user files an issue. A maintainer triages it at night. A company forks it. A clinical team discovers the edge case after the tool has already shaped a result.
That is not a sustainable validation model.
What changed with AI-assisted development
AI did not make clinical validation unnecessary. Anyone saying otherwise is selling something dangerous.
What AI has changed is the cost of building verification infrastructure.
A coding model can help rewrite a tool. More importantly, it can help build the harness that checks the rewrite against a reference implementation. It can generate test cases, run comparisons, summarize differences, and help tighten conformance tests. That work used to take enough engineering time that most grants, labs, and product teams could not justify it. Now large parts of it can be done much faster.
The hard part did not go away. You still need someone who knows which differences matter. You need someone skeptical enough to distrust a green test suite. You need someone who understands the biology, the file formats, and the failure modes.
The machine can make experiments cheap. It cannot decide what counts as a clinically meaningful answer.
That combination is useful, but only if we use it honestly.
fgumi, and the bug I did not expect to find
fgumi is our Rust reimplementation of fgbio’s UMI tools.
Unique molecular identifiers are short barcodes that help distinguish true molecules from sequencing errors. The grouping step is not glamorous, but it sits in the middle of workflows where reproducibility is essential.
When we built fgumi, the important part was not just writing a faster or more portable implementation, it was building the verification harness.
The harness ran fgumi and fgbio on the same inputs and compared the outputs read by read. It did this continuously across versions and datasets. It caught the kinds of bugs ordinary unit tests miss: tie-breaking differences, sort-order drift, quality-score encoding problems, and fixes that later refactors accidentally undid.
Then it found something I did not expect.
The problem was not in the rewrite. It was in fgbio, the original tool I had recommended for years.
On the same patient sample, fgbio could group molecules differently across runs. When two barcodes had equal counts, the code broke the tie using hash-map order, which was not deterministic. That kind of bug can sit undetected for years because a one-time benchmark will not catch it. You have to run the same data repeatedly and compare behavior carefully.
The harness found other issues too, including numeric instability in the consensus caller, double-counted statistics, and edge cases around clipped bases. We fixed them upstream.
It turns out building the new tool, then refusing to let it disagree with the old one without explanation, made the old tool better.
bwa-mem3, and the work nobody wants to fund
Most foundational tools will not be rewritten from scratch. They will be inherited.
bwa-mem3 is our modernization of bwa-mem2, which descends from bwa, the aligner much of the field has depended on for more than a decade. We forked it for speed, but also for control.
Control is underrated. bwa-mem2 ran on x86, while a lot of current development happens on Apple Silicon and deployment increasingly includes Arm. Quality-of-life improvements and correctness work had slowed because maintainer attention is finite. If a tool is going to remain load-bearing, someone has to do the unglamorous work of portability, tests, continuous integration, documentation, benchmarking, and verification against the prior implementation.
So that is what we did.
The verification harness compares bwa-mem3 and bwa-mem2 read by read. Across more than 800 whole-genome comparisons and 5.5 billion aligned reads, the two agree on 99.9% of reads. The remaining differences are not hand-waved away. They are characterized. Some are deliberate improvements. Some are tolerated. Some get budgets that fail the build if they drift.
That is the kind of boring engineering clinical genomics needs more of.
It is also the kind of work the field has historically undervalued.
ferro-hgvs, and where I got it wrong
The same validation gap shows up further downstream, where variants get named.
A clinical report usually does not communicate a variant as a raw genomic coordinate. It uses HGVS nomenclature to describe DNA, RNA, and protein changes. That string is how the variant gets looked up in databases like ClinVar, dbSNP, and CIViC. If the string is wrong, the variant may fail to match the interpretation that already exists for it.
ferro-hgvs is our library for parsing and normalizing HGVS strings. We verified it against established parsers and tested it against ClinVar, dbSNP, CIViC, and tens of millions of variants from public datasets. By release 0.6.0, it handled every publicly reported clinically relevant variant I had put through it.
I thought it was basically done.
It was not.
I should be blunt here because this is the part people tend to skip in blog posts. Variant representation is not my home turf. My background is read alignment and sequencing data infrastructure. I have written aligners, helped write the SAM specification, and built fgbio. HGVS projection is a different specialty.
I leaned too hard on the harness.
That worked for fgumi because the test suite behind fgbio had been built over years, case by case, by people who knew the tool deeply. In that situation, the test suite was close to a specification. Verify against it and you have done a meaningful part of the work.
ferro-hgvs was different. I leaned on tests ported from other tools, plus large public variant corpora. That was useful, but it was not enough. A corpus is not a specification. It is only the set of cases someone collected.
A clinical-trial client found the edge for us. They needed projection from genome to transcript to protein on their own data. The harness was green. I was confident. My confidence came across as authority.
The tool did not behave the way I had implied it would.
No patient was harmed because human review was part of the validation process, but the miss was still ours. More specifically, it was mine.
We could not simply add their failing examples to our tests. The variants would have revealed the disease area and therapy the company was working on. So we had to anonymize the patterns while preserving the shape of the HGVS constructs. This amounted to keeping the grammar and removing the biology.
Since then, much of the work has been building the projection I had assumed was finished and hardening the harness so it cannot tell the same comfortable story twice.
That is the difference between verification and validation in one project. The harness verified ferro-hgvs against the references we gave it. Validation required real use, private data, a clinical context, and enough humility to admit the harness was not asking the whole question.
A green test suite is evidence. It is not absolution.
The same problem runs below our code
This gap does not stop with tools we build.
htsjdk is the Java library used by Picard, GATK, and many other genomics tools to read and write sequencing files. A bug in htsjdk can become a bug everywhere at once. But it does not need a flashy rewrite. It needs careful hardening for performance work, correctness fixes, thread-safety checks, regression tests, and review by people who understand how widely the library is used.
The same principle applies when general tools are the wrong answer.
SMN1 and SMN2, the gene pair involved in spinal muscular atrophy carrier screening, are so similar that standard short-read pipelines can struggle to assign reads confidently. In some populations, relying on a standard pipeline alone can miss a substantial fraction of carriers. The field’s answer was to build specialized, separately validated callers for that region.
Knowing where a general tool fails is part of validation.
What should change
Equivalence testing against a reference implementation should be standard for rewrites. If you replace a foundational tool, you should show where the new implementation agrees, where it differs, and why those differences are acceptable.
Public benchmarks should also include more clinical edge cases. Genome in a Bottle’s challenging medically relevant genes benchmark is a good model. The field needs more of that across ancestries, assay types, and regions routine pipelines still handle poorly: FLT3 duplications, SMN paralogs, CYP2D6, repeat expansions, difficult structural variants, awkward HGVS constructs, and all the cases that get dismissed as edge cases until they show up in a report.
Clinical labs will still need to validate their assays. That does not change. I have run clinical validations and helped lead bioinformatics in a CLIA lab, and I know enough to respect the difference between tool verification and clinical validation.
But assay-level validation has been asked to carry too much of the tool-level burden. Each lab validates its own configured pipeline against the samples it has. That does not certify every shared tool underneath across the range of inputs it will eventually see. It also means the same work gets repeated in isolation by groups that all depend on the same infrastructure.
We can do better than that.
Who pays for this?
Bioinformatics has never settled who pays for maintenance.
AI-assisted engineering makes that harder to ignore because one of the old excuses is weaker now. The harnesses are cheaper to build. The comparisons are cheaper to run. The scaffolding that used to be uneconomic is now within reach for many more teams.
That does not make validation free. Human judgment is still expensive, and it should be. Someone has to decide what the tool is allowed to do, what counts as a meaningful difference, which edge cases belong in the suite, and where the tool should not be used.
The labs, diagnostic developers, funders, software companies, and open-source users all benefit from this infrastructure. They should all expect to fund some part of it.
At Fulcrum, we are starting with the tools in front of us. The validation suites behind fgumi, bwa-mem3, and ferro-hgvs are worked examples of what continuous verification can look like for real bioinformatics software. They do not solve clinical validation. They do show that tool-level verification can be done routinely, and that it can find real bugs in software the field already depends on.
I would rather build that infrastructure now, while it is finally cheap enough to do properly, than explain later why we kept trusting research code without checking it.
Nils Homer is a Founding Partner at Fulcrum Genomics, where he builds bioinformatics tools and pipelines for the genomics community. He is the creator of fgbio and a co-author of the SAMtools paper. You can find him on LinkedIn or reach Fulcrum at contact@fulcrumgenomics.com
Fulcrum Genomics is a bioinformatics consulting firm built by scientists at the forefront of large-scale genomic research, with deep expertise in sequencing technology, pipeline engineering, and genomic data analysis for biotech, pharma, and academia. Engage us through project-based work, fractional R&D, or hourly consulting. Contact us to discuss your project.