Extracting HotSpotter From 15 Years of Wildbook IA

What This Project Is

Wildbook has one job that sounds simple and is not: given a photo of an animal, find the same individual in a database.

For spotted, striped, and textured animals, HotSpotter is the engine that makes this work. It extracts local visual features from a crop of the animal, compares those features against a gallery, weights the matches, aggregates evidence by individual, and returns a ranked list of likely identities.

For 15 years that engine lived inside WBIA. The production Docker image is roughly 20 GB. The codebase carries native extensions compiled against specific OpenCV versions, dynamic controller injection via decorator registration, species-specific behavior, dual-database backends, a ZMQ job engine, a GUI, and a REST API. HotSpotter works. You cannot use it without the rest of WBIA.

This project extracts the algorithmic core into hotspotter: a reusable, deterministic Python library. No rewrite. Same behavior. Clean boundary.

The Dependency Wall

WBIA is not a bad codebase. It is a successful research platform that kept absorbing jobs: experiments, live deployments, species-specific pipelines, visual review tools, caches, REST endpoints, database schemas, machine learning integrations. Over time those responsibilities accreted around the identification pipeline.

The relevant legacy area alone spans these files:

This table understates the problem. The runtime path also touches chip extraction, feature extraction through pyhesaff, spatial verification through vtool, controller methods injected into the central IBEISController, database-backed annotation metadata, dependency caches, REST API adapters, and configuration dictionaries that differ between deployed Wildbook branches.

HotSpotter also depends on a stack of native components:

Build order matters. wbia-utool first, then wbia-vtool, then wbia-tpl-pyhesaff, then the extracted package. Native extensions must compile against the container’s OpenCV environment. Version bumps that look harmless can alter numerical behavior or break compiled libraries.

We learned this the hard way. The PyPI wheel for wbia-pyhesaff crashed with a SIGSEGV on import. Its transitive dependency wbia-vtool bundled pre-compiled OpenCV 2.4.5 shared libraries that conflicted with system OpenCV 4.x. The wheel was unusable. We vendored all three dependencies as git submodules, compiled them from source in dependency order, and abandoned the PyPI path. Each submodule installs with pip install --no-deps. SETUPTOOLS_SCM_PRETEND_VERSION is set because Docker builds lack .git history.

This is why wbia-core stays Docker-first. The goal is not Python importing on one laptop. The goal is freezing the feature extraction and matching environment tightly enough that parity results mean something.

Why HotSpotter Is Difficult To Extract

HotSpotter is not a model file. It is a ten-step pipeline where every step has traps.

1. Crop the annotation into a WBIA-compatible chip. WBIA uses affine transforms and specific chip sizing. A small difference in crop geometry changes pixels, which changes keypoints, which changes nearest neighbors, which changes the top-ranked identity.

2. Extract Hessian-affine/SIFT features. The parity target uses pyhesaff with compiled native code. An OpenCV SIFT baseline works for a prototype but is not equivalent. Switching from OpenCV SIFT to pyhesaff was one of the major steps toward meaningful parity.

3. Build one global nearest-neighbor index. WBIA builds a single FLANN index over all database descriptors in batch-file order. Query excluded. The order in which descriptors are stacked into the KD-tree changes its internal structure. We discovered this the hard way: stacking queries-first instead of batch-order sent our neighbor ID match rate from 7.2% to 73%.

4. Compute LNBNN weights. WBIA divides raw squared distances by 2 * 512**2, conditionally square-roots them, selects normalizer columns, filters self and same-name matches only in the voting columns, and combines filter vectors multiplicatively.

5. Aggregate to name scores. WBIA scores at the individual-name level with fmech/nsum, max-csum, and sumamech, then aligns those scores back to canonical annotation representatives. A benchmark can look wrong because one side measures raw per-annotation scores and the other measures name-aligned scores.

6. Spatial verification. WBIA uses vtool with affine and homography checks, weighting, thresholds, shortlisting, and configuration defaults. A simplified OpenCV homography is useful as a transitional implementation but is not full parity.

This is less like “copy the algorithm” and more like archaeology with a test harness.

The Extraction Itself

The current wbia-core prototype pulls out the algorithmic pieces into a package under 2,000 lines:

The contrast is deliberate. Legacy WBIA builds full ChipMatch objects with feature-match lists, score-vector lists, filter-normalizer state, name grouping, and cache metadata. wbia-core keeps the data needed for the algorithm: feature sets, matches, scores, names, correspondences.

But the most important design decision is deciding what the package is not.

The Boundary

The target package is not a new WBIA. It is an algorithm library. It owns reusable HotSpotter pieces and nothing else.

This boundary prevents the extraction from inheriting every concern that made WBIA hard to operate.

The intended architecture splits across three layers:

wbia-core currently includes a Flask sidecar for benchmarking. That sidecar is transitional. If it becomes the production identity service, the boundary has already failed.

The Parity Trap

The most dangerous failure mode is a rewrite that looks cleaner but changes results.

Wildlife identification systems are judged by ranking behavior. If the old stack returns the correct individual first and the new stack returns it third, the new stack is easier to deploy but has regressed the conservation workflow.

To compare the two systems, we built a parquet trace infrastructure. WBIA runs with a monkeypatch that writes 10 pipeline stages to parquet files: chips, annotations, features, nearest neighbors, baseline filter, neighbor weights, chip matches before and after spatial verification, and final scores. Hotspotter writes the same 10 stages in the same format. A comparison script diffs them stage by stage.

Without this, parity work would be blind. With it, we could pinpoint exactly which stage diverged and why.

Getting to parity took months of detective work. Each fix uncovered the next hidden assumption.

The 30,000-keypoint mistake

The first prototype extracted features from full images. A 2000x3000 pixel photo produced 30,000 keypoints. WBIA crops each image to its annotation bbox first, producing about 200 keypoints. The difference: 150x more features, 50x score inflation, 30x slower queries. A single 10-annotation benchmark took 101 seconds. After adding chip extraction: 3 seconds.

The chip extraction saga

Chip extraction sounds trivial. Crop the bbox, resize to a target dimension. It is not trivial.

First attempt: crop to bbox, resize to 450px width. The WBIA source comment in Config.py line 782 said dim_size=450. Rank correlation improved from negative to 0.92 on the best query.

Wrong dimensions: The comment was outdated. The actual runtime default lived in a different class (core_annots.ChipConfig) and was dim_size=700, resize_dim='maxwh'. Fixing this improved the best query from 0.92 to 0.98.

Wrong method: crop + cv2.resize produces different pixels than cv2.warpAffine. On a real COCO image, 75% of pixels differed between the two approaches. Maximum difference: 26 out of 255. Root cause: warpAffine does single-step Lanczos interpolation from source to destination. Crop+resize does two steps, sampling slightly different source coordinates when width and height scale factors differ. Rewriting to use the exact affine matrix from WBIA’s vtool.chip.get_image_to_chip_transform pushed the best query to perfect rank correlation (rho=1.00).

The distance normalization detective story

WBIA normalizes raw FLANN squared Euclidean distances by dividing by 2 * 512^2 = 524288. The question: does it apply sqrt afterward?

First we removed sqrt, matching the raw squared-norm path. Score ratio improved but rankings stayed the same (sqrt is monotonic). Then we discovered sqrd_dist_on=False in WBIA’s Config means sqrt IS applied, contradicting our first reading. Re-added sqrt. Score ratio went from 0.57x to 0.69x closer to WBIA. Rankings unchanged because sqrt preserves order.

The FLANN descriptor ordering fix

The biggest single jump in parity came from an unexpected source: the order in which database descriptors are stacked into the FLANN index.

We stacked descriptors queries-first. WBIA stacks them in batch-file (annotation) order. This changed the internal KD-tree structure. After fixing the stacking order, neighbor ID match rate went from 7.2% to 73%. Column 0 (closest neighbor) matched at 90.4%.

The benchmark artifact

At one point, wbia-core appeared to trail WBIA by 17 percentage points in top-1 accuracy (75% vs 91.7%) on a 51-annotation benchmark. This was alarming.

Two bugs in the benchmark runner caused it. First, it read WBIA’s pre-name-scoring annot_score_list (raw per-annotation csum) instead of the post-name-scoring score_list (canonical name-aligned). WBIA was being evaluated on raw csum while wbia-core was evaluated on nsum/fmech. Different scoring methods.

Second, run_query() never passed annot_name_list to WBIA’s annotation API. WBIA auto-generated 50 unique names for 50 annotations. Name grouping had 50 groups of size 1. fmech degenerated to per-annotation csum with zero cross-annotation aggregation.

After both fixes, the systems agreed on 11 of 12 top-1 queries with scores within 1%. The extraction was not failing because the algorithm was wrong. The measurement harness was wrong.

The build environment ceiling

After all algorithmic fixes, features are now bit-identical: 36,423 of 36,423 descriptors match exactly between systems. Neighbor distances correlate at r=0.98. But 27% of FLANN neighbor assignments still diverge.

The cause: different Docker base images. Compiling wbia-tpl-pyhesaff on Debian Bookworm with GCC produces a different .so than compiling it on Ubuntu 22.04 with a different compiler version. Same C++ source, different machine code, different floating-point results. The KD-tree built from those descriptors has a different internal structure, returning different approximate neighbors for the same queries.

This is the noise floor. The scoring pipeline is confirmed correct (rho=1.00 on unambiguous queries). The remaining gap is environment alignment, not algorithm.

The Deployed Reality

This work does not happen in a vacuum. Wildbook is not one deployment with one config.

An audit of deployed Wildbook branches found 18 publicly reachable platforms, all still routing directly to WBIA endpoints. Branch-level IA configuration drifts across species and deployments. Some branches use default HotSpotter. Some use HotSpotter with sv_on=true. Trout has an n=20 setting. Other platforms use algorithm families that are not HotSpotter: CurvRank, OC_WDTW, Finfindr, KaggleSeven, Deepsense, MiewID.

hotspotter should not pretend to migrate every identification path. Its scope is HotSpotter. Other algorithm families need their own migration decisions.

Algorithmic Truth vs. Accidental Architecture

The engineering challenge is choosing which historical quirks are algorithmic truth and which are accidental architecture.

Quirks that matter for parity:

Legacy architecture that should not survive:

This distinction is the heart of the project.

A Migration Pattern for Legacy ML Systems

The extraction follows a pattern that generalizes:

1. Find the smallest behavior-preserving core.
2. Build the new package around pure data containers and pure functions.
3. Vendor or pin native dependencies until reproducibility is proven.
4. Keep a transitional sidecar for testing and integration only.
5. Compare against the legacy system using real data and recorded fixtures.
6. Treat benchmark bugs as first-class migration bugs.
7. Move service, persistence, and orchestration into a separate layer.
8. Rename and package only after the boundary is clear.

The package is still named wbia-core but the target is hotspotter. Remaining work: renaming, moving chip extraction out of the sidecar, quarantining the sidecar, deciding public scoring semantics, finishing config parity, and hardening tests around the library boundary.

What Success Looks Like

Success is not a passing import. Success is this:

from hotspotter import HotSpotterConfig, extract_features

features = extract_features(chip, HotSpotterConfig())

And below that, wildlife-id importing LNBNN weighting, name scoring, and spatial verification primitives without inheriting WBIA’s controller model or runtime assumptions.

A successful hotspotter package is:

• Reusable — imported as a library, not only run as a service
• Stateless — no hidden dependency on WBIA databases or controllers
• Deterministic — same image and config produce the same features and scores
• Parity-tested — compared against recorded WBIA behavior and benchmark data
• Boundary-conscious — does not own API, persistence, permissions, or orchestration

When the extraction is done, the most important result is not that the new package is smaller. It is that the boundary is explicit. HotSpotter becomes an algorithm library. wildlife-id becomes the stateful service. The pipeline owns batch orchestration. Wildbook calls a modern identification stack without dragging 15 years of unrelated runtime history into every request.

The current state: features are bit-identical (36,423 of 36,423 descriptors match). Neighbor distances correlate at r=0.98. The scoring pipeline produces perfect rank correlation on unambiguous queries. The remaining 27% neighbor divergence and 47% spatial verification disagreement are environment alignment problems: matching pyflann/numpy versions and compiler toolchains between Docker images. The algorithm work is done. The reproducibility work continues.

That is how a 20 GB legacy image starts turning back into a library.

This post is adapted from a working draft in the Wildbook-infra repository. The extraction is ongoing. The current prototype lives in wbia-core/src/wbia_core/.