Counting Fruit in Orchard Photos with Open-Vocabulary Detection

Counting fruit from photographs sounds like a solved problem — object detection is decades old. But a working orchard breaks the standard recipe: there is no labelled dataset for "durian hanging in dense canopy, backlit, photographed from below on a phone," and no budget to build one before the product exists. This is how CanopyTwin counts fruit today across 243 real survey photos, and what we learned building it.

Why not train a fruit detector first?

The classic approach — collect thousands of images, draw boxes, train a YOLO-style detector — front-loads exactly the work a new orchard can't do yet. Every crop, cultivar, and growth stage looks different, and a detector trained on someone else's mangoes is confidently wrong about your durians. So we inverted the order: start with a zero-shot, open-vocabulary detector, let real usage accumulate reviewed labels, and only train a specialised model once the data exists.

Open-vocabulary detection: boxes from a text prompt

The workhorse is Grounding DINO, a detector that takes a free-text prompt — "fruit", "durian", "flower cluster" — and returns bounding boxes with confidence scores for whatever the prompt names. No training, no fixed class list. Point it at a canopy photo with the prompt "fruit" and it draws boxes around fruit it has never been trained to find specifically.

The catch is that confidence thresholds matter enormously. One dense canopy photo yields well over a hundred raw boxes at a permissive threshold — most of them shadows, leaf clusters, and duplicate hits on the same fruit. Raise the threshold too far and real fruit disappears. Our UI keeps the threshold a live slider rather than a baked-in constant: the server returns everything above a low floor, and the reviewer sweeps to the level where boxes match reality for that particular photo. The right threshold is a property of the photo, not the model.

Zero-shot detection is a starting point, not an answer. Its job is to be cheap, immediate, and reviewable — not to be right unattended.

The ensemble: detector versus VLM, with a referee

Boxes are only one way to count. A vision-language model, asked directly "how many fruit are visible in this photo?", produces a number with reasoning — and it fails differently than a detector does. The detector over-counts textured backgrounds; the VLM under-counts occluded clusters but ignores shadows. So CanopyTwin runs both and compares. When the two counts agree within tolerance, the count is accepted. When they diverge, a referee pass re-examines the photo with both answers in hand and adjudicates. Disagreement between independent methods turns out to be the most useful quality signal we have — it flags exactly the photos a human should look at.

Counts are then deduplicated per tree, not summed per photo: photograph the same laden branch three times and a naive pipeline counts it three times. Each tree keeps its best single-photo count — which is how 158 raw fruit detections across the survey become 113 actual fruit on 19 fruiting trees.

Segments when boxes aren't enough

Boxes answer "how many"; they don't answer "which pixels". For measuring canopy extent, isolating a trunk from background, or letting a grower click one specific fruit, we use SAM 2 — either segment-everything over a whole frame or click-to-segment on a point. Masks are slower and pricier than boxes, so they're on-demand rather than part of the default counting path.

The bug that wasn't a model problem: EXIF orientation

The worst counting bug we shipped had nothing to do with models. Phones frequently store photos unrotated and record the intended orientation as EXIF metadata; browsers and photo apps apply the rotation invisibly, but a vision model reads raw pixels. The result: boxes that looked rotated 90° from the fruit they belonged to, only on some photos, only from some phones. The fix is boring and absolute — normalise EXIF orientation at every single point where a machine reads a farm photo, and strip the tag after baking the rotation in. If you're building anything that runs vision on user photos, do this first.

Serverless GPU economics

A bootstrapped product cannot keep a GPU warm around the clock, and orchard surveying is bursty by nature — hundreds of photos on survey day, nothing for a week. The detector runs on serverless GPU infrastructure that scales to zero: a T4 spins up on the first request (roughly 25 seconds cold, with model weights baked into the container image so nothing downloads at start), answers warm requests in one to two seconds, and shuts down after a few idle minutes. At per-second billing, a single detection costs a fraction of a cent, and an idle week costs nothing. Managed model catalogues fill the gaps — segmentation runs on a hosted SAM 2 endpoint because operating our own would buy us nothing.

The flywheel: zero-shot today, specialised tomorrow

Every detection a grower reviews — confirming boxes, fixing labels, excluding bad frames — is a training example with provenance. The pipeline tracks reviewed photos per crop, and once a crop crosses a few hundred reviewed images, the same boxes export as a standard training dataset for a small closed-set detector. That model runs in tens of milliseconds instead of seconds, cheap enough for on-device counting in the field with no connection. The open-vocabulary model doesn't get replaced — it stays as the auto-labeller for every new crop, cultivar, and season the specialised model hasn't seen yet.

1. 1Start zero-shot: open-vocabulary detection gives usable boxes on day one, for any crop, with no dataset.
2. 2Never trust one method: run a second, differently-wrong counter and treat disagreement as your review queue.
3. 3Dedupe at the tree level, not the photo level — the orchard total is what's on the trees.
4. 4Normalise EXIF orientation everywhere a model reads pixels.
5. 5Scale-to-zero GPUs fit bursty agricultural workloads; pay for seconds, not servers.
6. 6Design the review loop so daily use quietly builds the dataset for the model you'll want next year.