A pre-trained vision-language model evaluated without specialized domain fine-tuning achieved an average semantic similarity of 76 percent when compared against descriptions written by human engineering personnel, according to a benchmark investigation published in AI in Civil Engineering. The study examined how well ChatGPT-4v could interpret and describe visual components from construction sites, focusing specifically on structural elements, construction components, and common jobsite materials. Researchers evaluated the artificial intelligence model against annotations provided by human practitioners, including a civil engineer and engineering interns, to determine whether general-purpose multi-modal models are mature enough to interpret specialized civil engineering imagery without requiring preliminary model training.
Visual inspections remain a critical task across civil engineering and the broader construction sector, functioning as an essential foundation for digital twin asset representations and jobsite safety management. Modern construction environments generate an escalating volume of visual data from hand-held mobile phones and jobsite surveillance cameras, which project teams increasingly seek to integrate into automated inspection pipelines. While conventional computer vision tools have traditionally been deployed to identify objects or retrieve similar imagery through predefined algorithms, they lack the capacity to interpret and articulate contextual technical narratives. Pre-trained vision-language models combine visual processing networks with generative transformer architectures, offering the capability to convert unstructured visual scenes into structured descriptive language that can support digital twinning workflows and robotic inspection frameworks.
To evaluate descriptive capabilities in this domain, the research team designed an experimental pipeline utilizing two distinct image collections comprising both public and privately gathered field data. The first collection consisted of a condensed public dataset containing 70 images spanning 10 distinct material and object categories, specifically brick, concrete, electrical components, rebar, scaffolding, general site work, stairs, stone, structural elements, and wood. These public images, originally collected from external repositories such as Roboflow and WorkerSafety, ranged from 416 by 416 pixels to 640 by 640 pixels. Before deployment, the investigators preprocessed the images by removing exchangeable image file format orientation metadata, correcting pixel orientation, and resizing frames, while deliberately avoiding data augmentation techniques to preserve natural visual configurations.
Recognizing that public datasets often feature commonly recognizable compositions and might inadvertently overlap with pre-training data ingested by large foundation models, the researchers constructed a separate private dataset to provide a rigorous blind benchmark. This private collection comprised 82 original photographs captured directly across active construction jobsites in eastern Pennsylvania. The authors documented real-world conditions using two commercial smartphones: a OnePlus 12 equipped with a 48-megapixel primary camera, a 16-megapixel ultra-wide sensor, and an 8-megapixel telephoto camera, alongside a Samsung Galaxy S20 FE featuring dual 12-megapixel sensors with wide and ultra-wide optics, smart high dynamic range, optical image stabilization, and deep fusion processing. The resulting private photographs ranged in resolution from 373 by 497 pixels to 624 by 832 pixels, depicting drywall installations, structural masonry, concrete pours, and active jobsite assemblies.
The evaluation protocol compared descriptive texts generated by ChatGPT-4v, accessed via Bing Chat without fine-tuning, against independent descriptions drafted by three human annotators for every image across both datasets. For the public dataset, the human descriptors consisted of a practicing civil engineer and two engineering interns; for the private eastern Pennsylvania dataset, annotations were provided by an active civil engineering project manager alongside two engineering interns. The testing setup was executed on a desktop computing system equipped with an Intel Core i7-11700 processor running at 2.50 gigahertz, an NVIDIA GeForce RTX 3060 Ti graphics processing unit, Python 3.9, and the SentenceTransformers software library. Query response times for generating model descriptions averaged between 5 and 10 seconds per image, which the researchers noted was suitable for offline analytical evaluations.
To rigorously assess textual correspondence, the researchers evaluated both semantic text similarity and surface-level lexical similarity across multiple analytical frameworks. Surface-level lexical methods included traditional evaluation algorithms commonly used in computational linguistics, specifically BLEU, ROUGE, and METEOR, as well as a custom-developed Intersection over Union metric implemented directly in Python. In contrast to lexical approaches that depend entirely on exact word matches and n-gram overlap, semantic evaluation was conducted using 15 distinct pre-trained models from the SentenceTransformers framework. SentenceTransformers maps sentences into dense, high-dimensional vector spaces using Siamese and Triplet network configurations derived from fine-tuned BERT architectures, allowing cosine similarity calculations to measure contextual alignment regardless of sentence structure.
Initial benchmarking on controlled reference sentences demonstrated that semantic transformer models substantially outperformed lexical metrics in distinguishing conceptual equivalence. While lexical algorithms produced similarity scores between zero and 33 percent on test sentences conveying identical technical meanings, the top semantic SentenceTransformers models yielded scores exceeding 75 percent. Five semantic models demonstrated superior capability in discerning contextual nuance while maintaining low scores on irrelevant pairs: models 2, 3, 10, 12, and 4 achieved average similarity scores of 84 percent, 81 percent, 80 percent, 79 percent, and 77 percent, respectively, on semantically matching benchmarks. Based on an established 75 percent acceptance threshold, the authors identified models 3 and 4 as primary benchmarks, with model 3 exhibiting the greatest overall consistency and lower variability across comparative evaluations.
Prior to assessing artificial intelligence outputs, the authors measured inter-annotator agreement among the human participants to establish a baseline of natural human descriptive variability. In the public dataset, pairwise agreement between human annotators ranged from 0.80 to 0.81, with standard deviations between 5 percent and 7 percent, where the highest agreement occurred between the engineering interns and the civil engineer. In the private Pennsylvania dataset, inter-human agreement ranged from 0.77 to 0.81, with a consistent standard deviation of 6 percent. The authors emphasized that human disagreement reflected legitimate professional diversity, as inspectors and interns routinely emphasize different structural details or surface defects within the same complex visual scene.
When ChatGPT-4v was evaluated against human descriptions through whole-text analysis, the top-performing semantic model demonstrated descriptive alignment closely approaching human-to-human agreement ranges. For the online dataset, the primary SentenceTransformers model recorded similarity scores exceeding 80 percent, whereas model 2 achieved scores above 70 percent. On the private jobsite dataset, model 3 achieved whole-text similarity scores exceeding 79 percent with a standard deviation between 5 percent and 8 percent, while model 2 yielded scores above 68 percent with standard deviations between 9 percent and 13 percent. Across all aggregated evaluations, the best-performing configuration sustained an overall average similarity score of 76 percent with a standard deviation of 4 percent, indicating consistent semantic correspondence with human engineers.
A granular sentence pair-wise analysis, which evaluated individual sentences within each description by recording the maximum similarity score against corresponding human sentences, revealed systematically higher alignment scores than whole-text comparisons. Under sentence pair-wise measurement, model 3 scored above 78 percent on the public dataset and over 76 percent on the private dataset, maintaining tight standard deviations between 4 percent and 7 percent. The authors observed that ChatGPT-4v generated longer, more exhaustive descriptions than human engineers, who typically focused immediately on singular defects such as concrete cracks or structural anomalies. While the added context provided by the artificial intelligence model lowered whole-text alignment when human notes were terse, individual machine-generated sentences accurately mirrored the technical observations documented by human inspectors.
The study observed slightly higher descriptive performance on the public dataset compared to the private jobsite collection, indicating potential domain challenges when foundation models encounter uncurated real-world environments. The researchers noted that public datasets may contain more conventional subject matter or visual patterns to which pre-trained foundation models were previously exposed, whereas private construction photographs featured complex, less conventional compositions that slightly reduced model alignment. Furthermore, the authors identified limitations in their experimental framework, emphasizing that variations in camera optics, sensor settings, and native image resolutions were not systematically isolated, and that real-time jobsite deployment would require faster dedicated application programming interfaces rather than interactive web-based chat interfaces.
The authors concluded that while pre-trained vision-language models demonstrate notable capability in recognizing and articulating civil engineering components without prior fine-tuning, autonomous inspection systems will require further optimization. Future investigations must explore specialized fine-tuning on large-scale civil engineering datasets, targeting quantitative benchmark criteria of similarity scores exceeding 95 percent and standard deviations below 3 percent across both whole-text and granular sentence evaluations. The researchers also highlighted the need to benchmark multiple foundation model architectures and develop domain-specific evaluation metrics before generative vision-language models can be fully integrated into automated digital twin pipelines and robotic structural inspection systems.
Subject of Research: Evaluation of pre-trained vision-language AI models for automated image descriptions in civil engineering
Article Title: Semantic and lexical analysis of pre-trained vision language artificial intelligence models for automated image descriptions in civil engineering
Article References: Bazrafshan, P., Melag, K., & Ebrahimkhanlou, A. (2025). Semantic and lexical analysis of pre-trained vision language artificial intelligence models for automated image descriptions in civil engineering. AI in Civil Engineering, 4(1), Article 17. https://doi.org/10.1007/s43503-025-00063-9
Image Credits: AI Generated
DOI: 10.1007/s43503-025-00063-9
Keywords: artificial intelligence, vision-language models, civil engineering, ChatGPT-4v, construction inspection, SentenceTransformers, semantic similarity, digital twins, structural health monitoring, image description, Semantic, lexical
