Passive Construction Site Safety Monitoring via Persona-Scaffolded Adversarial Chain-of-Thought VLM Verification Ananth Sriram∗
arXiv:2605.19869v1 [cs.CV] 19 May 2026
∗
Neel Mokaria∗
Rajveer Singh∗
Department of Computer Science, University of Maryland, College Park, MD, USA {asriram2, nmokaria, rajveer}@umd.edu †
Authors listed in alphabetical order; all contributed equally.
February 2026
Abstract
tion categories. The system maps detected violations to specific OSHA standards, performs REBAinspired ergonomic risk scoring from pose keypoints, and produces per-worker safety reports with timestamped frame-level evidence. A quantitative evaluation harness is released alongside the codebase for future reproduction.
Construction remains the deadliest industry sector in the United States, with 1,055 fatal worker injuries recorded in 2023 at a rate of 12.9 deaths per 100,000 full-time equivalent workers. The majority of these incidents are preventable, yet existing monitoring approaches are either prohibitively expensive, require real-time human operators, or address only a narrow subset of detectable violations. This paper presents a passive, end-of-shift construction safety monitoring pipeline, processing video from two independent camera modalities — POV body-worn and fixed wall-mounted cameras — through a unified three-stage architecture: (1) a fine-tuned YOLO11 detector for primary PPE and hazard detection, (2) SAM 3 for segmentation refinement, worker deduplication, and PPE-to-worker association, and (3) Qwen3-VL-8B-Instruct with a novel three-pass adversarial chain-of-thought protocol for compliance verification and hallucination control. The principal methodological contribution is the Stage 3 prompt design: we find that the specific construction of professional persona backstories, following the methodactor prompt engineering framing, drives an observed 12% precision improvement on an informal three-author review of the 12-video Ironsite video corpus, with the largest gains on hallucination-prone categories. Structural message isolation enforces observational independence between a generator pass, a discriminator pass, and a reconciliation pass governed by asymmetric rules that encode explicit priors about the relative reliability of human observation versus automated detection for different viola-
1
Introduction
The Bureau of Labor Statistics reported a fatal work injury rate of 12.9 deaths per 100,000 full-time equivalent workers in the U.S. construction sector in 2023, making it the industry with the most fatalities of any sector [1]. Men accounted for 91.5% of fatalities. Fall protection (OSHA Standard 1926.501) alone accounts for the single largest violation category cited by OSHA year over year [2]. The operational gap we target is the lack of an affordable, passive, continuous monitoring approach that can watch multiple workers across a construction site, resolve their identities over time, and produce an actionable report without requiring realtime human review. Existing solutions tend to be expensive, require dedicated real-time operators, or cover only a narrow slice of the problem—for instance, hard hat detection alone. We explore an end-of-shift processing paradigm as one possible response. Rather than streaming video in real time, the system processes footage after a shift completes, which makes it more practical to deploy on commodity hardware while using highercapacity models (including an 8-billion-parameter 1
vision-language model) and multi-pass verification tegration of ergonomic risk assessment or persistent that would be difficult to support under real-time worker identity tracking across a shift. latency constraints. The contributions of this paper are descriptive and 2.2 architectural:
Ergonomic Risk Assessment via Pose Estimation
Traditional ergonomic risk assessment in construction relies on manual observation instruments, principally the Rapid Entire Body Assessment (REBA) [5] and Rapid Upper Limb Assessment (RULA), which require trained observers and are inherently episodic rather than continuous. Computer vision approaches to automating these assessments have drawn primarily on skeleton-based pose estimation. Yan et al. [6] proposed wearable IMU-based systems to detect musculoskeletal disorder (MSD) risk in real time, achieving high sensitivity for trunk flexion and arm overreach events. Subsequent work demonstrated that markerless pose estimation using convolutional neural network-based keypoint models can recover joint angles sufficient for ergonomic scoring from monocular video without instrumented garments [13]. However, these systems have been validated primarily in controlled laboratory settings; their performance on outdoor construction footage with partial occlusion and variable depth remains incompletely characterized.
1. A unified three-stage pipeline (YOLO → SAM 3 → VLM) that handles both PPE and posture/ergonomic analysis from two independent camera modalities. 2. A method-prompted, persona-scaffolded threepass adversarial chain-of-thought VLM protocol — following the method-actor framing of Doyle [11] — with structurally enforced independence between passes and an asymmetric reconciliation policy, which yields an observed 12% precision improvement over single-pass prompting on the development corpus. 3. REBA-inspired ergonomic risk scoring from YOLO pose keypoints with tuned confidence gates. 4. An end-of-shift backend that maps detected events to specific OSHA standards and produces per-worker safety reports with timestamped evidence.
2
Related Work
2.1
PPE Detection in Construction En- 2.3 vironments
Worker Re-identification and Identity Persistence
Maintaining persistent worker identities across a full shift is a prerequisite for accumulating violation histories and generating per-worker compliance reports, yet it has received comparatively limited attention in the construction safety literature. Detection-based tracking systems operating in construction environments must contend with visual confounds introduced by high-visibility PPE — similar vest colours and helmet profiles reduce the discriminative power of appearance-based features [14]. BoT-SORT [15] and related tracking-by-detection algorithms address short-term occlusion effectively through coupled Kalman filter motion prediction and appearance re-identification, but are not designed for the multi-hour gaps characteristic of shift-level tracking. The present work adopts a lightweight identity database combining BoT-SORT for within-session tracking with appearance embedding similarity for re-association after disappearance events, representing a pragmatic intermediate between pure tracking
Automated detection of personal protective equipment (PPE) using computer vision has been an active area of research since the widespread adoption of deep convolutional neural networks. Early work by Fang et al. [3] demonstrated that Faster R-CNN could detect hard hat compliance from farfield surveillance footage with sufficient accuracy for practical deployment, establishing the feasibility of passive vision-based monitoring in outdoor construction environments. Nath et al. [4] extended this line of work to multi-class PPE detection using YOLOv3, reporting mean average precision (mAP) values above 0.85 on construction-specific datasets. More recent studies have leveraged YOLOv5 and YOLOv8 architectures on larger composite datasets, consistently demonstrating that detection accuracy degrades under variable illumination, occlusion, and worker clustering [12]. A persistent limitation across this body of work is the restriction to a single PPE category or a single camera viewpoint, without in2
3
and full Re-ID.
2.4
Methodology
Vision-Language Models for Safety The proposed system processes construction site video through a unified three-stage pipeline shared Assessment across both the posture/ergonomics and PPE violation detection streams. Artifacts propagate sequentially: Stage 1 produces initial detections, Stage 2 refines spatial evidence and resolves worker identity, and Stage 3 applies a vision-language model to assess compliance against OSHA standards with measures intended to reduce hallucination. Figure 1 illustrates the end-to-end data flow, and Table 1 summarizes each stage’s role and outputs. The principal methodological contribution is the prompt design for Stage 3: a method-prompted, persona-scaffolded adversarial chain-of-thought protocol that produced an observed 12% precision improvement on a dev-team review over single-pass VLM prompting on the Ironsite development corpus (Section 9).
The application of large vision-language models (VLMs) to safety-critical inspection tasks is nascent but growing. Generalist VLMs have shown promising zero-shot performance on visual question answering tasks involving workplace safety scenes, yet their deployment in safety-critical contexts is complicated by hallucination — the generation of plausible but factually unsupported outputs [16]. Chain-ofthought prompting, while improving task accuracy on reasoning benchmarks, has been reported to induce systematic overconfidence in VLMs by causing token probabilities to reflect consistency with the model’s own reasoning trace rather than uncertainty about visual evidence [17]. Multi-agent and selfconsistency prompting approaches [7] have demonstrated that soliciting multiple independent completions and aggregating by consistency can reduce hallucination rates, which motivates the multi-pass verification protocol described in this work. To the best of our knowledge, prior published work has not applied this style of multi-pass adversarial VLM verification specifically to construction site PPE or ergonomic compliance assessment.
3.1
Stage 1: Primary Detection
A YOLO11 object detector was fine-tuned on a composite dataset assembled from five constructionspecific PPE sources: the Ultralytics ConstructionPPE collection (11 classes), the Roboflow construction site safety dataset, the SH17 dataset, and two additional PPE corpora sourced from Kaggle. The resulting detector recognizes 12 mapped seman2.5 Integrated Construction Safety Mon- tic classes spanning worker presence, PPE compliance state, and proximate equipment, including hard itoring Systems hats, safety vests, gloves, and proximate machinery Several research systems have proposed integrating such as excavators, cranes, and forklifts. multiple safety sensing modalities for construction Detection confidence is set deliberately low at τconf = site monitoring. Kim et al. [18] proposed a real0.15 with the intent of maximizing recall at this time system combining RFID-based proximity sensstage; the downstream VLM verification pass is reing with camera-based PPE detection for zone-level sponsible for suppressing false positives. This design compliance monitoring, but required dedicated inreflects a principled asymmetry: in a safety monitorfrastructure installation. Awolusi et al. [19] demoning context, the cost of a missed violation is typically strated wearable sensor fusion for ergonomic and fall considered to exceed the cost of a spurious detection risk assessment but noted that per-worker hardware that is subsequently rejected. acquisition and maintenance costs represent a significant barrier to large-site deployment. A distinguish- For the wall-mounted camera stream, YOLO opering characteristic of the present system is its end- ates in conjunction with BoT-SORT [15], a multiof-shift processing paradigm, which decouples mon- object tracker that maintains persistent track identiitoring quality from real-time compute constraints. fiers across frames via coupled Kalman filter motion This makes it feasible to use large segmentation and prediction and appearance re-identification. This vision-language models that would be difficult to run enables temporal PPE accumulation per tracked under streaming latency requirements, while rely- worker: rather than issuing a violation on a single ing only on standard video capture infrastructure frame, the system evaluates compliance over a slidalready present on many modern construction sites. ing window of detections, which is intended to re-
3
Outputs Stage 1
Input
Video
frames
Fine-tuned YOLO
Stage 2 bboxes, labels
SAM 3
refined bboxes, worker IDs
Stage 3
Qwen3-VL 8B Instruct Adv. CoT, referential loops
Posture / Ergonomic Events PPE Violation Events
Figure 1: Three-stage pipeline architecture shared by both posture/ergonomics and PPE violation detection. Video frames pass through YOLO detection, SAM 3 refinement, and adversarial VLM verification to produce compliance-checked safety events. Table 1: Three-stage pipeline: components, roles, and outputs. Stage
Component
Role
Output
1
Fine-tuned YOLO
Initial bounding boxes, class labels, frame-level detections
2
SAM 3
3
Qwen3-VL-8B-Instruct
First pass on video: detect workers, PPE items, poor-posture candidates, violation candidates Worker tagging, deduplication, refined masks and bboxes (MoE segmentation) Assesses Stage 1–2 outputs via method prompting, chain-ofthought, and referential loops; applies OSHA mapping; targets hallucination reduction
Refined bboxes, worker/track IDs, segmentation masks Compliance-checked violation flags → final posture/ergonomic and PPE violation outputs
duce the impact of transient occlusions and detector Second, per-worker segmentation masks provide noise. more semantically isolated inputs for VLM processFor the body-worn camera stream, a separate YOLO ing. Rather than presenting the full scene to the pose estimation model extracts 17 COCO-format language model, Stage 2 produces isolated worker skeletal keypoints per detected person per frame, crops with overlaid annotation, which we use to reproviding the joint coordinate sequences required for duce the contextual ambiguity that has been identified as a source of hallucination in VLM-based asdownstream ergonomic angle computation. sessment [16].
3.2
Stage 2: Spatial Refinement and Third, PPE-to-worker association is resolved via Worker Deduplication center-point containment: for each detected PPE
item, the system determines which worker’s bounding box contains the centroid of the PPE mask. This heuristic addresses the common failure mode of assigning a detected hard hat to an adjacent rather than the proximate worker when workers are clustered. Required PPE per worker is defined as First, SAM 3 masks are used to tighten the bound- Preq = {hardhat, safety vest, gloves}; a violation is ing boxes produced by YOLO, correcting for the flagged if any element of Preq is unobserved after a tendency of detection models to over-expand boxes temporal accumulation window. in the presence of adjacent workers or equipment. Tighter crops are intended to improve both the visual evidence quality presented to the VLM and the accuracy of spatial proximity computations used for hazard detection. Bounding boxes produced by Stage 1 are forwarded to SAM 3 (Segment Anything Model 3) [20] as prompted inputs to generate pixel-accurate segmentation masks. This stage serves three functions within the pipeline.
4
3.3
Stage 3: Compliance Verification via ment, and the raw YOLO/SAM 3 confidence data— Adversarial Vision-Language Model into a structured VLMAssessment object. Conflicts between Pass 1 and Pass 2 assessments are resolved Assessment through explicit chain-of-thought reasoning, with the reconciler required to justify any departure from either upstream signal.
The third stage applies a fine-tuned Qwen3-VL-8BInstruct model augmented with an optional LoRA adapter for construction-domain specialization. The central methodological contribution of this work lies in the Stage 3 prompt design. We find that the specific construction of persona backstories — not merely the use of personas, but the granularity of role, inspection mandate, and evidence access scoped to each persona — has substantial impact on output precision. On the Ironsite development corpus, the full three-pass adversarial protocol introduced below produced an observed 12% precision improvement over single-pass prompting of the same Qwen3-VL8B base model with an equivalent token budget. The central design challenge at this stage is hallucination: large VLMs prompted on safety-critical visual evidence have been reported to exhibit overconfidence, generating plausible but factually unsupported violation reports [16]. Single-pass prompting is particularly susceptible because the model has no mechanism for cross-checking its own outputs against independent evidence.
The adversarial structure is intended to reduce hallucination through two complementary mechanisms. First, cross-pass consistency checking: a violation reported in Pass 3 should be traceable to at least one of the two independent upstream signals, discouraging the model from generating violations supported by neither visual observation nor machine detection. Second, confidence calibration: YOLO detection confidence scores are explicitly communicated to the VLM as prior signal strength, instructing the model to treat high-confidence detections (≥ 0.70) as strong priors requiring rebuttal evidence to override, and low-confidence detections (< 0.40) as weak signals requiring independent visual confirmation before violation issuance (Table 4). Long-form videos are segmented into 60-second nonoverlapping chunks prior to VLM processing to prevent context window overflow; each chunk receives an independent assessment. The final output per chunk is a structured VLMAssessment containing: a scene summary, worker count, equipment inventory, per-worker PPE compliance status (each item classified as PRESENT, ABSENT, or UNCLEAR), a typed hazard list with severity classification and explanatory rationale, aggregate confidence, and actionable corrective recommendations.
To address this, a three-pass adversarial chainof-thought protocol is introduced, drawing on the self-consistency principle [7] and extending it toward multi-perspective inspection rather than simple output aggregation.
Pass 1 (Generation): A field safety inspector persona reviews the raw, unannotated video stream in isolation, with no access to machine detection out- 3.4 Prompt Design and Adversarial Inputs. This pass produces free-form inspection notes dependence grounded solely in visual observation, with the intent of establishing an independent evidence signal The three-pass protocol is not merely a stylistic that is not anchored to Stage 1–2 detector artifacts. prompt variation: persona separation is enforced at the message-construction level. Pass 1’s mesPass 2 (Discrimination): A senior safety officer sage array contains no YOLO detection data and persona independently reviews the same video alongno annotated frames; Pass 2’s message array conside YOLO/SAM 3 annotated frames—bounding tains no Pass 1 output. Independence is therefore boxes, segmentation masks, and confidence scores structural rather than instructional — a model opoverlaid. This pass has no access to Pass 1 notes and erating in Pass 2 does not have Pass 1 observations is tasked with validating or rejecting each Stage 1– in its context, which is intended to reduce the cross2 machine detection from an independent observacontamination that would arise if both passes shared tional perspective. a common context window. Pass 3 (Reconciliation): The same senior persona Table 2 summarises the input modalities available reconciles all available evidence—the Pass 1 obserto each pass. The full system prompts for each pass vational notes, the Pass 2 machine-detection assessare provided in the Supplementary Material (Ap5
pendix A).
[helmet=0.42, vest=0.18]
Table 2: Input modalities per VLM pass. Pass 1 (Generate) 2 (Discriminate) 3 (Reconcile)
Video
Frames
YOLO
Notes
✓ ✓ —
— ✓ ✓
— ✓ ✓
— — Both
Worker #3 [VIOLATION] present=[helmet] missing=[vest, gloves]
A four-band calibration scale is injected into the Pass 2 and Pass 3 prompts, explicitly mapping confidence values to epistemic weight:
Table 4: YOLO confidence calibration bands inPass 3 deliberately omits the raw video. This is a jected into Passes 2 and 3. forcing function: the reconciler operates on committed evidence — the written observations each pass Confidence Signal VLM instruction has already committed to in natural language — ≥ 0.70 HIGH Strong prior: detection is real 0.40–0.69 MODERATE Verify visually rather than re-litigating ambiguous video frames. 0.15–0.39 WEAK Must confirm in video The structured VLMAssessment output (Table 3) is < 0.15 NOISE Discard unless confirmed produced exclusively by Pass 3, which must justify each hazard decision against all three upstream evidence sources. Pass 1 receives no confidence data by design: the generator persona is intended to provide an obserTable 3: VLMAssessment output schema (Pydanticvational baseline that is not anchored to machine validated). output. Field
Type / Values
scene summary worker count equipment present ppe per worker reasoning hazards no hazards confidence
string int ≥ 0 list[string] list[PPEStatus] string (mandatory audit trail) list[HazardDetail] bool LOW | MEDIUM | HIGH
PPEStatus fields: helmet, vest, gloves
PRESENT | ABSENT | UNCLEAR
HazardDetail fields: violation type
severity best frame index
3.4.2
A key element of the prompt design is a set of asymmetric reconciliation rules governing how Pass 3 resolves disagreements between the two observers and the machine detection signal. These rules encode an explicit prior that reflects the complementary strengths and weaknesses of human observation versus automated detection in construction safety contexts:
PPE MISSING | FALL PROTECTION MISSING | PROXIMITY HAZARD | ZONE BREACH | LADDER MISUSE | SCAFFOLD VIOLATION | BEHAVIORAL UNSAFE | AWKWARD POSTURE | MSD HIGH RISK | OVERREACH | KNEELING SQUATTING LOW LOW | MEDIUM | HIGH | CRITICAL int (0-based index into annotated frames)
1. Agreement (both observers and YOLO concur): flag with high confidence. 2. Generator-only flag (Pass 1 flagged, Pass 2 did not): do not automatically dismiss. Pass 1 observed raw video and is sensitive to motion, behaviour, and transient events — phone use, running, brief PPE removal — that annotated still frames may not capture. A specific reason to overrule is required; absence of machine detection is insufficient. 3. Discriminator-only flag (Pass 2 flagged, Pass 1 did not): flag if machine evidence is strong (conf ≥ 0.60). 4. Machine-only detection (neither observer flagged): flag with explanatory note if conf ≥ 0.70; discard if conf < 0.40. 5. Observer-only flag (either observer flagged, YOLO missed): flag. Automated detectors are known to miss behavioural violations, and a clear
The best frame index field grounds each hazard to a specific annotated frame by interleaving [Annotated frame i] text markers before each image in the Pass 3 message array, enabling the postprocessor to resolve the index to a filename for timestamped evidence retrieval. 3.4.1
Asymmetric Reconciliation Rules
YOLO Confidence Calibration
Passes 2 and 3 receive YOLO detection data rendered as structured text, with per-class confidence scores formatted as: t=12.40s | detections: person=0.91 helmet=0.42 vest=0.18 hazards: PPE_MISSING (worker #3)
6
human observation is treated here as sufficient ev- 4.2 Risk Scoring and Violation Classifiidence. cation 6. Conflict (observers disagree): rule in favour of the observer with clearer contextual access to that Angles are combined into a scalar risk score following violation type — Pass 1 for behavioural and mo- the two-group REBA scheme [5]: tion violations; Pass 2 for spatial and PPE violations corroborated by machine detection. These rules instantiate an intentional asymmetry: behavioural violations (phone use, unsafe movement, zone breaches) are treated as observer-strong, machine-weak, while spatial and PPE violations are treated as machine-strong, observer-weak. This asymmetry is motivated by our qualitative observations of the detection characteristics of each modality on construction footage, and is intended to reduce the false negative rate for the violation categories most responsible for construction fatalities [2]. The correctness and calibration of this asymmetry remain to be validated quantitatively. 3.4.3
(1)
ScoreB = g(θarm , θelbow )
(2)
Combined = max(ScoreA , ScoreB ) + ∆
(3)
where ∆ captures joint interaction effects. The combined score maps to four violation levels (Table 5), with specific triggers including arm elevation > 65◦ with concurrent body twist (OVERREACH), trunk flexion > 48◦ (AWKWARD POSTURE), and trunk flexion > 65◦ (MSD HIGH RISK).
System Prompts Table 5: Posture violation classification thresholds.
The full system prompts used for Passes 1–3 are reproduced verbatim in the Supplementary Material (Appendix A) to support reproducibility. Cameramodality-specific addenda (injected for POV and wall-mounted variants) are documented in the released codebase.
4
ScoreA = f (θtrunk , θneck , θknee , θlat )
Condition
Type
Risk
Combined < 3 Combined ≥ 3 Combined ≥ 5 Combined ≥ 8
Compliant AWKWARD POSTURE OVERREACH MSD HIGH RISK
Level 1 Level 2 Level 3 Level 4
Posture and Ergonomic Analysis 5
Worker Identity and Temporal
For POV body-worn camera footage, ergonomic risk Accumulation is assessed via a REBA-inspired pipeline operating on skeletal keypoints extracted by the YOLO pose Worker identity is maintained across a shift via model. two complementary mechanisms. BoT-SORT [15] provides frame-to-frame track continuity through 4.1 Joint Angle Extraction Kalman filter motion prediction, handling shortFor longer disappearances, a Six angles are computed from the 17 COCO key- term occlusions. points per frame: trunk flexion (θtrunk ), trunk lateral lightweight identity database stores per-worker aplean (θlat ), neck flexion (θneck ), knee angle (θknee ), pearance embeddings and re-associates returning arm raise (θarm ), and elbow flexion (θelbow ). A key- workers by embedding similarity, which is intended point confidence gate of τkp = 0.65 is enforced to to prevent duplicate identity assignment. suppress detections from occluded or partially visi- PPE compliance is evaluated over a temporal accuble poses. mulation window rather than on individual frames. Per-worker logs track observed PPE items across the chunk; a violation is issued only if a required item remains unobserved after sufficient accumulated evidence, which is intended to reduce false positives from momentary occlusions or detection noise. 7
6
OSHA Violation Coverage
9
Results
Detected violations are mapped to specific OSHA standards, summarised in Table 6. Violations are scored by severity (LOW–CRITICAL) and frequency; workers with repeated violations are surfaced in the per-shift safety report.
The pipeline was deployed end-to-end and exercised against multiple hours of construction site footage across both POV body-worn and fixed wall-mounted camera modalities under masonry, preparation, and transit task scenarios. Figures 2 and 3 present representative outputs. Quantitative evaluation is deTable 6: OSHA violation coverage by detection ferred to future work pending re-provisioned GPU modality. compute; the evaluation harness required to produce it is released alongside the codebase. All observaStandard Violation Source tions reported below are qualitative and based on the development corpus. 1926.501 Fall protection Wall cam 1926.503 Training compliance Worker DB 1926.451 Scaffolding Wall cam PPE detection. On the development corpus, the 1926.102 Eye/face PPE Wall cam Stage 1 detection layer identified hard hats and high1910.212 Machine guarding Wall cam visibility safety vests on wall-cam footage and re— MSD / posture risk POV covered glove presence on POV footage where the — Ladder misuse Wall cam camera-wearer’s hands occupied a substantial por— Respiratory PPE Wall cam tion of the frame. Figure 2 shows a compliant detection (safety vest identified, green label) alongside a violation (missing glove flagged with red bound7 System Implementation ing box). The wall-cam pipeline resolves violations The backend is a FastAPI application with async job from site-wide views, while POV cameras provide processing, SQLite persistence (via SQLAlchemy), hand-level evidence for glove compliance. Detection and four core tables: sites, workers, shifts, quality appeared to degrade under low-illumination and safety events. All models (YOLO, SAM 3, conditions; variable lighting remains an open limitaQwen3-VL) are pre-warmed on startup to avoid cold- tion at the Stage 1 layer that warrants quantitative start memory errors. The wall-cam pipeline pro- characterisation. cesses every third frame; the posture pipeline runs concurrently in a thread pool. Safety events are deduplicated per (worker, violation type) per shift. The system was developed and validated on 4× NVIDIA RTX PRO 6000 Blackwell GPUs.
8
Data and Datasets
(a) Compliant: safety vest (b) Violation: detected. flagged.
YOLO11 was fine-tuned on a composite PPE dataset assembled from Ultralytics Construction-PPE (11 classes), Roboflow construction site safety, SH17, and two additional Kaggle corpora. Posture threshold tuning used the CWPV dataset [10] (POV footage of construction workers annotated for musculoskeletal posture) and the Önal & Dandıl behavioural video dataset [9] (691 clips, 8 behaviour classes). Validation used long-form POV footage (15–20+ minutes per clip) from masonry, preparation, and transit scenarios; compliant and noncompliant clips were exported for threshold tuning and human review.
no glove
Figure 2: PPE detection results. (a) Worker building a concrete block wall; system identifies the highvisibility vest and labels the worker as compliant. (b) POV frame; system flags an exposed hand with a red bounding box for missing hand protection.
Posture and ergonomic analysis. On the development corpus, the REBA-inspired scoring distinguished routine construction movement from visually unambiguous risk postures. Figure 3 contrasts a compliant posture assessment with an overreach vi8
trolled study with blind annotation, and the same authors who developed the system served as groundtruth raters; a defensible measurement requires independent annotators on a held-out test set, which the released evaluation harness is designed to produce. Early single-pass runs exhibited confident emission of plausible but visually unsupported violations in these categories; introducing the generator/discriminator persona separation, the asymmetric reconciliation rules (Section 3.4), and the explicit confidence-band calibration appeared to reduce this failure mode to a level the development team judged acceptable for end-of-shift reporting. A controlled ablation isolating the contribution of each protocol component (persona separation vs. confidence bands vs. asymmetric reconciliation) is implemented in the (a) Worker #825: Posture (b) Worker #910: Risk 3 — released harness but not reported here.
olation. Arm-elevation events appeared recoverable from pose-keypoint outputs on the inspected clips; the keypoint confidence gate at τkp = 0.65 appeared to suppress spurious detections from partially occluded poses without visibly suppressing risk events on the clips we inspected. Systematic measurement of false positive and false negative rates is required to confirm these qualitative observations.
Good.
Overreach.
9.1
Figure 3: Posture analysis results. (a) Worker segmented with blue mask and green bounding box; pose and PPE assessed as compliant. (b) Worker segmented with red box; elevated arm classified as Overreach at risk level 3 for musculoskeletal strain.
Limitations and Future Work
Several limitations are noted, each suggesting a concrete direction for future work. The 12% precision improvement reported in Section 9 was obtained through informal author-adjudication on the 12-video Ironsite development corpus; the three authors who built the system also served as groundtruth adjudicators, and no held-out test set or blind annotation protocol was used. A controlled study with independent annotators, inter-annotator agreement measurement, and a held-out test set is the highest-priority subsequent activity, and the released evaluation harness is designed to support it. Worker re-identification has not been stress-tested in densely populated footage; the appearance-embedding infrastructure is in place, but its accuracy at scale is unmeasured. Variable lighting causes intermittent PPE false positives at the Stage 1 layer on the development corpus, suggesting domain-adaptive finetuning as a natural extension. The posture threshold boundary between safe and borderline ergonomic risk remains context-dependent and may require persite calibration. Finally, POV cameras frequently experience mid-task view obstruction; multi-camera fusion across simultaneous POV and wall-mounted streams is a promising mitigation the present architecture supports but does not yet exploit.
Three-pass adversarial VLM verification. The transition from single-pass VLM prompting to the three-pass adversarial chain-of-thought protocol described in the Methodology produced the largest single quality improvement observed during development. To estimate the magnitude of this effect prior to controlled evaluation, we conducted an informal review across the full 12-video Ironsite development corpus (approximately 30 minutes per video, multiple violation instances per video). Each video was processed by both the single-pass baseline and the full three-pass protocol using the same Qwen3-VL8B base model and an equivalent total token budget. The three authors then independently reviewed every violation flag emitted by each configuration and classified it as a true or false positive by consensus, treating the three-author consensus as local ground truth. Across all 12 videos, the three-pass protocol produced approximately 12% higher precision than the single-pass baseline, with the largest gains on hallucination-prone categories where vio- 10 Discussion lations are difficult to confirm from a single frame (gloves, eye protection). We emphasise that this is The architectural decisions and qualitative obseran author-adjudicated evaluation rather than a con- vations reported here suggest that end-of-shift pas9
sive monitoring may be a viable alternative to realtime surveillance for construction safety compliance, pending quantitative validation. Several design decisions that emerged during development warrant discussion. VLM hallucination was the most prominent technical obstacle encountered. Early single-pass runs produced confident violation reports that were visually unsupported, a failure mode consistent with findings in the broader VLM reliability literature [16]. The adversarial multi-pass architecture — in which two structurally independent observer personas assess the same scene before a reconciliation pass — was the intervention that appeared, qualitatively, to most reduce observed hallucination during development. The asymmetric reconciliation rules (Section 3.4), which treat behavioural violations as observer-strong and spatial PPE violations as machine-strong, reflect our working hypotheses about the complementary detection characteristics of human observation and automated detection respectively. These hypotheses remain to be tested. PPE-to-worker association proved nontrivial under worker clustering. A detected hard hat occupying the airspace between two adjacent workers does not automatically belong to the worker whose bounding box overlaps it most; centre-point containment logic combined with temporal accumulation across frames was useful for more reliable association on the clips we inspected. Ergonomic threshold selection required extensive iteration: the distinction between routine construction movement and genuinely hazardous posture is context-dependent, and a fixed threshold — such as a 45◦ trunk flexion limit — can misclassify momentary tool pickups and sustained awkward postures alike without additional temporal context. Joint GPU residency for three large models demanded careful orchestration; pre-warming on startup, sequential inference staging, and 60-second video chunking were necessary to prevent out-ofmemory failures on the development hardware.
11
tem maps detected violations to specific OSHA standards, performs REBA-inspired ergonomic risk scoring from pose keypoints, and produces per-worker safety reports with timestamped frame-level evidence. The central methodological element is a three-pass adversarial chain-of-thought protocol in which structural message isolation is used to enforce observational independence between passes, and asymmetric reconciliation rules encode explicit working assumptions about the relative reliability of human observation versus automated detection for different violation categories. Qualitative observations on construction site footage are consistent with the feasibility of the approach; quantitative evaluation against the released harness is deferred to future work pending re-provisioned GPU compute. Future work should prioritise: controlled quantitative ablation of the three-pass VLM protocol against single-pass and two-pass baselines; stress-testing of the appearance-embedding re-identification module in densely populated multi-worker footage; domainadaptive fine-tuning of the Stage 1 detector on low-illumination construction footage; and integration with EHS reporting platforms such as Procore and Autodesk Construction Cloud. Longer term, real-time ingestion via RTSP/WebRTC and multicamera fusion across simultaneous POV and wallmounted streams represent natural extensions that the present architecture supports but does not yet exploit.
Conclusion
A passive, end-of-shift construction safety monitoring pipeline has been described, processing video from POV body-worn and fixed wall-mounted cameras through a unified three-stage architecture: finetuned YOLO detection, SAM 3 segmentation refinement, and adversarial VLM verification. The sys10
References [1] Bureau of Labor Statistics. Census of Fatal Occupational Injuries Summary, 2023. U.S. Department of Labor, 2024. https://www.bls.gov/news. release/archives/cfoi_12192024.htm [2] Occupational Safety and Health Administration. Top 10 Most Frequently Cited Standards for Fiscal Year 2024. U.S. Department of Labor, 2024. https://www.osha.gov/top10citedstandards/ [3] Q. Fang, H. Li, X. Luo, L. Ding, H. Luo, T. Rose, and W. An. Detecting non-hardhat-use by a deep learning method from far-field surveillance videos. Automation in Construction, 85:1–9, 2018. [4] N. D. Nath, A. H. Behzadan, and S. G. Paal. Deep learning for site safety: Real-time detection of personal protective equipment. Automation in Construction, 112:103085, 2020. [5] S. Hignett and L. McAtamney. Rapid Entire Body Assessment (REBA). Applied Ergonomics, 31(2):201–205, 2000. [6] X. Yan, H. Li, A. R. Li, and H. Zhang. Wearable IMU-based real-time motion warning system for construction workers’ musculoskeletal disorders prevention. Automation in Construction, 74:2–11, 2017. [7] X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou. Selfconsistency improves chain of thought reasoning in language models. In Proc. ICLR, 2023. [8] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. Berg, W.-Y. Lo, P. Dollár, and R. Girshick. Segment Anything. In Proc. ICCV, 2023. [9] O. Önal and E. Dandıl. A video dataset for safe/unsafe worker behaviour classification in production environments. BMC Research Notes, 17:234, 2024. [10] CWPV: A Working Postures of the Construction Working Postures Videos dataset. Figshare, 2024. https://figshare.com/articles/dataset/ 27907818 [11] C. Doyle. LLMs as Method Actors: A Model for Prompt Engineering and Architecture. arXiv preprint arXiv:2411.05778, 2024. https://doi. org/10.48550/arXiv.2411.05778 [12] P. Sanjeewani, G. Neuber, J. Fitzgerald, N. Chandrasena, S. Potums, A. Alavi, and C. Lane. Realtime personal protective equipment non-compliance recognition on AI edge cameras. Electronics, 13(15):2990, 2024. [13] W. Zhao, L. Wang, Y. Li, X. Liu, Y. Zhang, B. Yan, and H. Li. A multi-scale and multi-stage human pose recognition method based on convolutional neural networks for non-wearable ergonomic evaluation. Processes, 12(11):2419, 2024. [14] J. Geng, J. Zhao, and S. Li. Optimizing helmet use
11
detection in construction sites via fuzzy logic-based state tracking. Sensors, 25(2):456, 2025. [15] N. Aharon, R. Orfaig, and B. Z. Bobrovsky. BoTSORT: Robust associations multi-pedestrian tracking. arXiv preprint arXiv:2206.14651, 2022. [16] Z. Bai, P. Chen, C. Fu, S. Yin, and E. Chen. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930, 2024. [17] R. Welch, E. Konuk, and K. Smith. The cost of reasoning: Chain-of-thought induces overconfidence in vision-language models. arXiv preprint arXiv:2603.16728, 2025. [18] K. Kim, Y. Cho, and S. Zhang. Integrated safety monitoring system using RFID and computer vision for construction sites. Automation in Construction, 66:11–21, 2016. [19] I. Awolusi, E. Marks, and M. Solis. Wearable technology for personalized construction safety monitoring and health assessment. Automation in Construction, 91:235–250, 2018. [20] N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khabsa, R. Röhrbach, A. El-Nouby, A. Shiekh, W. Cheng, M. Saraf, A. Morcos, M. Paluri, and C. Feichtenhofer. SAM 2: Segment Anything in Images and Videos. arXiv preprint arXiv:2408.00714, 2024.
A
Supplementary Material: VLM System Prompts
The full system prompts used in the three-pass adversarial chain-of-thought protocol are reproduced verbatim below. The prompts are referenced from Section 3.4.
Pass 1 System Prompt (Jamie Reyes — Generator) You are Jamie Reyes, a field safety inspector with 6 years of on-site construction experience. You are conducting an initial walkthrough review of this site camera footage and filing a written inspection report. Your report will be reviewed and audited by Marcus Chen — Chief Safety Officer with 24 years of experience. He will be comparing your findings against machine-detection data and annotated frames from an AI system. If you miss something obvious or write vague non-observations, he will catch it and it will reflect on your competence. Be thorough, honest, and specific. Name what you see. If something concerns you, write it down clearly — do not hedge into uselessness. If something looks fine, say so and say why. Note approximate timestamps where helpful. Write in plain English, not JSON. This is your inspection report, not a final verdict — Marcus will have the final say.
Figure 4: Pass 1 system prompt. The generator receives raw video only; no machine detection data or prior notes are present in the message array.
Pass 2 System Prompt (Marcus Chen — Discriminator) You are Marcus Chen. You have 24 years as a senior construction safety manager and Chief Safety Officer. You are conducting your own independent review of a site camera video. You have NOT seen any other inspector’s report. You are watching the raw video yourself, and you also have the machine-annotated frames from the AI detection system (YOLO + SAM) showing bounding boxes and segmentation masks over flagged moments. Watch the raw video with your own eyes first. Then cross-reference with the annotated frames. Your job: write your independent professional assessment. • What do you observe in the raw video? Any workers, PPE, hazards, unsafe behaviour? • For each machine-flagged detection: do you agree based on what you see? • Did the machine miss anything you can see clearly in the raw footage? • Note detections that look like noise or false positives. Write in plain English. Be direct. Do NOT produce JSON — a third pass will do that.
Figure 5: Pass 2 system prompt. The discriminator receives raw video, annotated frames, and YOLO confidence data, but no Pass 1 output. 12
Pass 3 Reconciliation Rules (verbatim from user template) You have three sources of evidence. Reconcile them and produce the final safety assessment. SOURCE 1 — Jamie’s field inspection report (raw video, no machine data) SOURCE 2 — Your own assessment of the annotated frames SOURCE 3 — YOLO + SAM detection data RECONCILIATION RULES: • AGREEMENT (Jamie + you both flagged it, YOLO confirms) → flag it, high confidence. • SPLIT (Jamie flagged, you did not) → DO NOT automatically dismiss Jamie. Jamie watched the raw video — he catches motion, behaviour, and momentary events that annotated frames may not show. . . Require a specific reason to overrule Jamie, not just absence of machine detection. • MACHINE ONLY (YOLO flagged, neither observer noted it) → conf ≥ 0.70 = flag with note; conf < 0.40 = discard. • OBSERVER ONLY (Jamie or you flagged it clearly, YOLO missed it) → flag it. Machines miss things constantly. A clear human observation of a violation is sufficient. • ⇒ DO NOT write “no violations” if Jamie’s notes contain specific flagged observations. Engage with each one explicitly.
Figure 6: Pass 3 reconciliation rules (user template, verbatim). Pass 3 omits the raw video and operates on committed textual evidence plus annotated frames.
13