What is a hard negative in satellite image similarity search?

Aerial view of a solar panel farm laid out in rectangular blocks across a desert landscape.

A hard negative is a result that scores high on similarity but is wrong when you look closely. Not a random tile the model got confused about, the kind of wrong that makes sense once you see it: same roof pitch, same footprint size, same shadow geometry, different site entirely. If you've run a similarity search against a known facility and gotten back forty hits where thirty-one are genuinely interesting and nine are a quarry, a solar farm, or a half-built warehouse that happens to share your target's rectangular outline, you've already met the term. You just didn't have a name for it.

The opposite, an easy negative, is a tile so far off that no embedding model would ever rank it near your query: open water next to a built structure, forest next to a parking lot. Easy negatives don't cost you anything, you skip past them in half a second. Hard negatives cost you time because they require a second look, and across a search area with ten thousand candidate tiles, that second look adds up fast.

Image similarity search works by converting each tile into a vector, a string of numbers that captures visual structure rather than labeled categories. Two tiles end up close together in that vector space when they share geometry, texture, and spatial arrangement, not because a model has tagged them "industrial facility" or "storage yard." That's the whole point of using embeddings instead of a trained classifier: you don't need a labeled dataset of your target type, you need one example.

Hard negatives are built into the method this way, not a bug you can patch out. A query tile of a known processing site, say, a cluster of rectangular buildings with a loading apron and a perimeter road, will sit near other tiles with the same arrangement regardless of what's happening inside them. An agricultural equipment yard with a similar building count and road pattern can land at a similarity score close enough to your real matches that a threshold cutoff won't separate them cleanly. Cosine similarity on a learned embedding measures visual resemblance, not function, and a near-miss like that is exactly what you should expect the method to surface.

Working the result list instead of trusting the score

Treat the similarity score as a sort order, not a verdict. Pull the top candidates, scan them in rank order, and expect the hard negatives to cluster right alongside your true positives rather than trailing off at the bottom. That clustering is a useful signal: if a handful of near-miss tiles keep showing up right next to confirmed matches across different query runs, you're looking at a visual category the embedding genuinely can't separate from your target, not a one-off.

Hard negative mining, as a term, comes out of the machine learning side of this, retraining or fine-tuning a model specifically on the negatives it keeps getting wrong so the next version separates them better. That's a model-development exercise, not something you do at query time. What you can do at query time is narrow the field before you search: a tighter area of interest, a date range that excludes construction periods when footprints look different, or a second query tile from a slightly different angle of your known site, which often reshuffles which near-misses bubble up.

A ranked candidate list is a triage tool. The analyst's job is still the last ten seconds of looking at each tile, not the first second of trusting the number next to it. A similarity search that returns zero hard negatives across a wide area usually means the threshold is cutting the candidate pool too hard, not that the method has gotten more precise.

If you're running this kind of search yourself, Broad Area Search is built around that exact workflow: mark one known site, get back a ranked list of everything across your archive that resembles it, and work the list from the top down. Join the early access list if you want to try it against your own imagery.