Annotation by Arithmetic: Exact Mark Placement on Construction Drawings by Separating Perception from Measurement

Abstract

Automated construction estimation requires more than correct quantities: an estimator must be able to see, on the drawing itself, exactly which drawn element each line item refers to. Vision-language models identify millwork items on architectural sheets reliably, but the bounding boxes they emit are typed from memory while attending to downscaled raster crops, and the resulting marks routinely land on adjacent bays, on appliances housed inside cabinets, or on unrelated views of the same sheet. We report on a production annotation system that reaches exact placement by refusing to let the model place anything. The model performs two tasks it does reliably: naming the items present and reading the printed dimension strings. A deterministic geometry engine performs everything else. It recovers the architect’s own rectangles from the vector layer of the CAD-exported PDF, harvests printed detail callout tags and per-view scale notes from the text layer, converts the item’s printed width into drawn points through the view’s own scale, and selects the unique drawn geometry whose measured span matches the printed claim. A closed inspection loop renders the resulting marks back onto the sheet, judges each one at reading distance with a vision model, and applies corrections only where they survive arithmetic constraints; a mark whose measured width already matches the printed claim is locked against revision. Ona production interior elevation sheet containing 38,545 vector segments across eight drawing views, the system places six of six casework marks on the correct drawn elements with a median width error of 0.1 inches at 3/8′′ = 1′-0′′ scale, versus zero of six exact placements for the same pipeline with model-typed coordinates. We contribute a taxonomy of placement failures observed across four instrumented production runs, including two failure classes located in the surrounding toolchain rather than the model, and argue that the division of labor, model as reader and machine as measurer, generalizes to other grounding tasks over machine-generated documents.