The Problem
Wildbook powers wildlife identification for 18 conservation platforms. Shark fins, giraffe spots, zebra stripes, whale flukes. Thousands of researchers upload photos every week. The system matches each animal against a growing database and returns a ranked list of likely identities.
One of the core engines behind this workflow is HotSpotter. It extracts local visual features from an animal crop, compares those features against a gallery, weights the matches, aggregates evidence by individual, and returns a ranked list. In conservation terms, this is the difference between one more unstructured photo and a usable encounter in a population study.
The working implementation lives inside WBIA, a 15-year-old research-to-production codebase. It grew into a full application: web service, job system, database controller, image processing toolkit, experiment harness, GUI, cache layer, plugin ecosystem. HotSpotter is in there, but it is not packaged as a library. It is entangled with the world it grew up in.
The Numbers
The production Docker image is roughly 20 GB. The HotSpotter-related files alone span thousands of lines across a maze of controller methods, dependency caches, REST adapters, and species-specific configuration. The codebase carries years of compiled native extensions, dynamic controller injection, and operational assumptions from multiple generations of Wildbook deployments.
The Plan
We are extracting HotSpotter into a standalone Python package. Not rewriting it. Extracting it.
The current prototype is wbia-core. The target package is hotspotter — a reusable, deterministic HotSpotter library that can be imported, tested, benchmarked, and called by a modern identification service. The extraction follows these rules:
- Preserve behavior first. Clean the boundary second.
- Keep the native feature extraction stack reproducible via vendored submodules and a Docker-first build.
- Prove parity against WBIA with a COCO wildlife benchmark, replay fixtures, and multi-target comparison.
- Do not rebuild the WBIA application under a new name. Push service, persistence, and orchestration concerns into
wildlife-idandpipeline.
What Makes This Hard
HotSpotter is not a single model file. It is a pipeline with many small numerical decisions that cascade. A one-pixel difference in image decoding changes keypoints, which changes nearest neighbors, which changes the top-ranked identity.
What We Hit
The devlog reads like a debugging thriller. Here are the fights that shaped the architecture.
A SIGSEGV on import. The PyPI wheel for wbia-pyhesaff, the Hessian-affine SIFT extractor, crashed on import. Its transitive dependency wbia-vtool bundled pre-compiled OpenCV 2.4.5 shared libraries that conflicted with the system OpenCV 4.x. The wheel was unusable. We vendored all three dependencies as git submodules, compiled them from source in dependency order, and threw away the PyPI path entirely.
30,000 keypoints instead of 200. The first prototype extracted features from full images. WBIA crops each image to its annotation bbox first. The difference: 30,000 keypoints per image versus 200. 50x score inflation. 30x slower queries. The fix sounded simple (crop to bbox first) but opened the chip extraction saga.
75% of pixels differed. WBIA extracts chips with cv2.warpAffine. Our crop+resize approach produced different pixels on 75% of the chip surface. Max pixel difference: 26 out of 255. The root cause: warpAffine does single-step Lanczos interpolation from source to chip. Crop+resize does two steps, sampling slightly different source coordinates. We rewrote chip extraction to use the exact same affine matrix as WBIA.
Wrong chip dimensions. The WBIA source comments said dim_size=450. The actual runtime default was dim_size=700, resize_dim='maxwh'. The comment was outdated. The real default lived in a different class. Fixing this alone improved rank correlation from 0.92 to 0.98 on our best query.
Descriptor ordering broke FLANN. We stacked database descriptors queries-first. WBIA stacks them in batch-file order. This changed the internal KD-tree structure. Neighbor ID match rate jumped from 7.2% to 73% after the fix.
The benchmark was comparing different things. At one point wbia-core appeared to trail WBIA by 17 percentage points in top-1 accuracy. Two benchmark bugs caused this: the runner read WBIA’s pre-name-scoring scores instead of post-name-scoring scores, and it never passed annotation names to WBIA, making WBIA treat every annotation as a different individual. After fixing both, the systems agreed on 11 of 12 queries with scores within 1%.
Same source code, different results. Compiling wbia-tpl-pyhesaff on Debian Bookworm versus Ubuntu 22.04 produces different .so files. Different compilers, different optimizations. The same C++ source produces slightly different SIFT descriptors on identical chip images. This is the current noise floor.
Where We Are
The pipeline pieces are built. Feature extraction, FLANN indexing, LNBNN weighting, name scoring, spatial verification, and a COCO benchmark harness exist. Features are now bit-identical between systems: 36,423 of 36,423 descriptors match exactly. Neighbor distances correlate at r=0.98. The scoring pipeline produces perfect rank correlation on unambiguous queries.
Remaining work is closing the last gap: 27% of FLANN neighbor assignments still diverge due to pyflann/numpy version differences between Docker images, and spatial verification pruning agrees on only 47% of annotations. The algorithm is done. The remaining work is environment alignment.
The Full Story
We wrote a detailed case study that walks through the extraction — the architecture choices, parity traps, dependency problems, deployed config drift across 18 platforms, and the specific pattern we are following for other legacy ML migrations.