1
Benchmark AUC Is Not Deployable Reliability: A Cross-Dataset Audit of Off-the-Shelf Features for Surveillance Video Anomaly Detection
arXiv:2606.29506v1 [cs.CV] 28 Jun 2026
Mohammadreza Rashidi Department of Computer Science AI and Media Analysis Lab Berlin, Germany [email protected]
Abstract Automated “suspicious behavior” flagging is a headline promise of AI surveillance, and the field reports high frame-level ROC-AUC on standard video anomaly detection benchmarks. Those numbers are measured by training and testing on the same camera and scene. We audit what happens when that assumption is dropped. We build an unsupervised normality model from the all-normal training frames of one dataset, using frozen off-the-shelf embeddings (CLIP, DINOv2, ResNet-50, EfficientNet-B0) and a nearest-neighbour distance, and score the test frames of the same and of other datasets. Across 4 real datasets (UCSD Ped1, UCSD Ped2, CUHK Avenue, ShanghaiTech) and 4 backbones, same-dataset AUC averages 0.704 but cross-dataset AUC averages 0.499, which is chance: a detector calibrated on one scene is no better than a coin flip on another, and in several pairs it is below chance. The strongest backbone makes this worse, not better: DINOv2 has the best same-dataset AUC (up to 0.901 on Ped2) and the largest cross-dataset drop. The collapse is not an artefact of the scoring rule: replacing the nearest-neighbour detector with a PaDiM-style Mahalanobis detector reproduces it almost exactly (cross-dataset gap 0.202 versus 0.208). Even at a favourable operating point the false-alarm rate is on the order of 31,931 per hour. We conclude that the benchmark numbers quoted for surveillance anomaly detection describe a calibrated laboratory setting and overstate deployable reliability by a wide margin, and we release the code that reproduces every number. Index Terms video anomaly detection, surveillance, cross-dataset generalization, ROC-AUC, self-supervised features, OSINT, operational reliability
I. I NTRODUCTION A recurring claim in AI surveillance is that cameras can learn what “normal” looks like and automatically flag the unusual. The academic basis for the claim is video anomaly detection (VAD), where models trained on normal-only footage are evaluated by how well an anomaly score separates normal from anomalous frames. Reported frame-level ROC-AUC on the standard benchmarks is high, often above 0.85. Those numbers share an assumption that a deployment does not satisfy: the model is trained and tested on the same camera, scene, and conditions. A real surveillance network has cameras the model never saw during calibration. This paper asks a single question. If you build the normality model on one dataset and test it on another, how much of the reported performance survives? We answer it as an audit, not a new detector. We use frozen off-the-shelf embeddings as the frame representation, the same backbones audited for surface matching in related work, and a simple nearest-neighbour distance to the normal training set as the anomaly score. This is a deliberately plain, reproducible detector whose only knowledge of “normal” is the training split of one dataset. We then score same-dataset and cross-dataset test frames and report frame-level ROC-AUC, the equal-error rate, and the operational false-alarm rate. Contributions: • A cross-dataset generalization audit of unsupervised VAD on 4 real datasets [1], [2] and 4 off-the-shelf backbones, with frame-level ROC-AUC, EER, and false-alarms-per-hour, reported as a same-vs-cross matrix. • Evidence that cross-dataset AUC collapses to chance (0.499 versus 0.704 same-dataset), that the strongest backbone collapses the most, and that the operational false-alarm rate is prohibitive even at a favourable operating point.
2
A second-detector control showing the collapse persists when the nearest-neighbour scorer is replaced by a PaDiM-style Mahalanobis detector (gap 0.202 versus 0.208), so the failure is in the representation, not the read-out. • Released code that reproduces every table, number, and figure, with an independent verification script. •
II. BACKGROUND AND R ELATED W ORK A. Datasets The standard single-scene VAD benchmarks are UCSD Ped1 and Ped2, surveillance of a pedestrian walkway where anomalies are non-pedestrian objects such as bikes and carts [1], and CUHK Avenue, a campus scene with anomalies such as running and throwing [2]. ShanghaiTech is a larger multi-scene campus benchmark introduced with a future-frame-prediction baseline [3]. Each provides an all-normal training split and a test split with per-frame anomaly labels, which is what makes unsupervised normality modelling and frame-level evaluation possible. B. Anomaly detection methods Single-scene VAD is surveyed in [4], and deep anomaly detection more broadly in [5]. The dominant unsupervised paradigm learns a model of normal appearance and motion and flags frames it explains poorly: by reconstruction with autoencoders, by memory-augmented autoencoders that suppress the over-generalisation of plain autoencoders to anomalies [6], [7], or by future-frame prediction [3]. A parallel line scores anomalies by distance in a frozen feature space to a memory of normal features [8] and is the lineage our detector follows. The out-of-distribution (OOD) detection literature addresses a closely related question: can a model determine that an input differs from its training distribution [9]? Our audit applies OOD reasoning to the VAD setting: when a frozen-feature detector is calibrated on one scene, is a frame from another scene out-of-distribution in a way that separates normal from anomalous? A separate, weakly supervised paradigm instead trains on video-level normal and anomalous labels [10]; it needs anomalous examples at training time and addresses a different problem from the normal-only setting we study. Our detector is a frame-level instance of the frozen-feature line: a nearest-neighbour distance to normal embeddings. We do not aim to beat the state of the art; we use a transparent detector to isolate the question of cross-scene transfer, which the methods above evaluate almost exclusively in the same-scene regime. C. Positioning Most VAD papers report same-dataset AUC. This number measures a calibrated, same-camera laboratory setting. We measure how that number degrades across datasets and translate it into an operational false-alarm rate, which is what a surveillance deployment actually experiences. Our question is an instance of distribution-shift robustness that the OOD detection field has studied in classification [9] but that VAD has not systematically evaluated. III. M ETHODOLOGY A. Datasets and labels We use UCSD Ped1, UCSD Ped2, CUHK Avenue, and ShanghaiTech [3]. Each dataset’s training split is allnormal and builds the normality model; each test frame carries a binary label, positive if it contains any annotated anomaly. UCSD frames come with per-pixel masks (a frame is positive if any pixel is anomalous) or temporal annotations; Avenue ships per-frame mask volumes; ShanghaiTech provides per-clip anomaly segment annotations. We read labels directly from these files. The ShanghaiTech training and test splits are large, so we build its normality model from a uniform subsample of the training frames and evaluate on a uniform subsample of the test frames. Table I lists the resulting test-split sizes and anomalous-frame fractions; the four datasets differ widely in base rate (from 0.059 to 0.820 anomalous frames), which is itself a reason a threshold calibrated on one transfers poorly to another. In total we score 95,564 test frames per backbone. B. Embeddings Each frame is embedded by a frozen backbone, resized to 224 pixels with the model’s own preprocessing, pooled, and L2-normalised: CLIP ViT-B/32 [11], [12], DINOv2 ViT-S/14 [13], ResNet-50 [14], and EfficientNet-B0 [15], via timm and OpenCLIP [16]. No fine-tuning is used.
3
Cross-dataset audit protocol
Dataset A
→ Frozen embedding → Normality model
normal-only train
CLIP / DINOv2 / ...
kNN / Mahalanobis
↓ calibrate, then score below ↓ Test frames
→ Frozen embedding → Anomaly score → AUC matrix
same A + others B,C,D
same backbone
vs normal model
same vs cross
Diagonal (test on A) is the calibrated setting; off-diagonal (test on B, C, D) is the deployment setting.
Fig. 1. The cross-dataset audit protocol. A normality model is calibrated on the normal-only training frames of one dataset (top), then used to score the test frames of the same dataset (the calibrated, diagonal setting) and of every other dataset (the cross-dataset, off-diagonal setting). The same frozen backbone embeds both splits, and the same procedure is repeated for two detector families, yielding the same-versus-cross ROC-AUC matrix. TABLE I T EST SPLITS USED IN THE AUDIT: NUMBER OF EVALUATED FRAMES AND FRACTION OF FRAMES LABELLED ANOMALOUS . T HE WIDE SPREAD IN BASE RATE IS ONE DRIVER OF CROSS - DATASET THRESHOLD FAILURE . Test dataset
Frames
Anom. frac.
Ped1 Ped2 Avenue ShTech
7,200 2,010 15,324 71,030
0.562 0.820 0.242 0.059
C. Anomaly score The normality model is the set of training (normal) embeddings. A test frame’s anomaly score is one minus the mean cosine similarity to its k nearest training embeddings (k = 5): frames far from all normal frames score high. This is the SPADE/PatchCore family [8] reduced to whole frames. D. Cross-dataset protocol The full protocol is summarised in Figure 1. For every (backbone, train-dataset, test-dataset) triple we build the normality model on the train dataset and score the test dataset, giving 64 evaluations (48 of them cross-dataset). The diagonal (train and test the same dataset) is the usual calibrated setting; the off-diagonal is the deployment setting where the camera is new. E. Metrics and statistics We report frame-level ROC-AUC (the field standard), the equal-error rate, and the operational false-alarm rate per hour at the threshold that achieves 0.9 recall, assuming 10 frames per second. AUC carries a 95% bootstrap confidence interval (1,000 frame resamples). All embeddings, scores, tables, and figures are produced by released scripts on a single GPU through PyTorch [17], and src/verify_numbers.py re-derives every reported number from the score file. IV. R ESULTS A. Same-dataset performance is good; cross-dataset performance is chance Table II and Figure 2 give the headline. Averaged over backbones, same-dataset AUC is 0.704 while cross-dataset AUC is 0.499. The cross-dataset value is chance: a normality model calibrated on one scene separates normal from anomalous frames on another scene no better than a coin flip, and the worst pairs fall to 0.340, below chance. The gap between the calibrated and the transferred setting is 0.205 AUC points.
4
TABLE II F RAME - LEVEL ROC-AUC AVERAGED OVER THE SAME - DATASET ( DIAGONAL ) AND CROSS - DATASET ( OFF - DIAGONAL ) EVALUATIONS , PER BACKBONE . C ROSS - DATASET AUC IS AT CHANCE FOR EVERY MODEL . Model
frame-level ROC-AUC
1.0
Same-dataset AUC
Cross-dataset AUC
Gap
CLIP ViT-B/32 DINOv2 ViT-S/14 ResNet-50 EfficientNet-B0
0.706 0.725 0.686 0.698
0.495 0.490 0.511 0.501
0.211 0.235 0.175 0.197
Mean
0.704
0.499
0.205
Anomaly detection collapses across datasets chance same-dataset cross-dataset
0.8 0.6 0.4 0.2 0.0
T-B/32 CLIP Vi
2 DINOv
4 ViT-S/1
tResNe
50
Efficien
0 tNet-B
Fig. 2. Same-dataset versus cross-dataset frame-level ROC-AUC per backbone. The dashed line is chance. Calibrated performance does not transfer across scenes.
B. The strongest backbone transfers worst The per-cell matrix for DINOv2 (Table III, top-right panel of Figure 3) shows the pattern sharply. DINOv2 has the best same-dataset score of any model and cell (0.901 on Ped2) but the lowest cross-dataset average (0.490) and the largest same-to-cross gap (0.235). CLIP, the weakest on the diagonal (0.706), transfers least badly (0.495). A representation that captures a scene’s normal appearance in fine detail is exactly the representation that fails hardest when the scene changes: it has learned the camera, not the concept of anomaly. C. The pattern holds for every backbone Figure 3 repeats the per-cell matrix for all four backbones. The structure is identical in every panel: a bright diagonal and a washed-out off-diagonal. No backbone escapes the collapse, and several off-diagonal cells (Ped1/Ped2 trained, ShanghaiTech tested) sit well below chance because the normality model has learned a background appearance that makes the new scene’s normal frames look anomalous. The collapse is therefore a property of the off-the-shelf-feature approach, not of one encoder. The exact AUC for every one of the 64 evaluations is given in Table IV. Reading down each model’s column, the four calibrated (diagonal, marked •) entries are the only ones consistently above chance; the twelve cross-dataset entries per model cluster around 0.5 and dip as low as 0.340. The collapse is also asymmetric: a model trained on the busier Avenue or ShanghaiTech scenes and tested on the sparse UCSD walkway fares differently from the reverse direction, but neither direction recovers usable separation.
5
TABLE III DINOV 2 FRAME - LEVEL ROC-AUC FOR EVERY TRAIN / TEST PAIR . D IAGONAL ( BOLD ) IS THE CALIBRATED SETTING ; OFF - DIAGONAL IS CROSS - DATASET. O FF - DIAGONAL CELLS SIT AT OR BELOW CHANCE . train ↓ test →
Ped1
Ped2
Avenue
ShTech
Ped1 Ped2 Avenue ShTech
0.688 0.484 0.462 0.540
0.564 0.901 0.570 0.714
0.436 0.391 0.625 0.486
0.380 0.350 0.498 0.686
DINOv2 ViT-S/14
0.68
0.57
0.56
0.35
Ped1
0.69
0.56
0.44
0.38
Ped2
0.48
0.85
0.51
0.34
Ped2
0.48
0.90
0.39
0.35
Avenue
0.59
0.53
0.60
0.44 Avenue
0.46
0.57
0.63
0.50
ShTech
0.60
0.45
0.52
0.68
ShTech
0.54
0.71
0.49
0.69
Ped1
Ped2
Avenue ShTech
Ped1
Ped2
Avenue ShTech
train dataset
Ped1
EfficientNet-B0
ResNet-50 Ped1
0.60
0.64
0.41
0.43
Ped1
0.66
0.65
0.50
0.46
Ped2
0.40
0.83
0.46
0.41
Ped2
0.48
0.80
0.44
0.43
Avenue
0.48
0.65
0.62
0.49 Avenue
0.50
0.37
0.62
0.49
ShTech
0.53
0.67
0.55
0.70
ShTech
0.56
0.53
0.59
0.71
Ped1
Ped2
Avenue ShTech
Ped1
Ped2
Avenue ShTech
0.9 0.8
frame ROC-AUC
CLIP ViT-B/32
0.7 0.6 0.5 0.4
test dataset Fig. 3. Frame-level ROC-AUC for every train/test pair and every backbone. Each panel’s bright diagonal (calibrated) collapses to a muted off-diagonal (cross-dataset). The pattern is the same for all four feature extractors.
D. The operational false-alarm rate is prohibitive AUC hides the operating point. At the threshold that catches 90% of anomalous frames, the median false-alarm rate across all evaluations is about 31,931 per hour (Table V). The equal-error rate tells the same story: it averages 0.349 in the calibrated same-dataset setting but rises to 0.502 cross-dataset, indistinguishable from the 0.5 of a random scorer. Even in the calibrated setting the median false-alarm rate is 26,406 per hour, rising to 32,456 per hour cross-dataset. A human reviewer cannot triage at that volume, which is the practical meaning of an AUC in the 0.7–0.9 range on an imbalanced stream. V. T HE S TRUCTURE OF THE C OLLAPSE The averages above establish that cross-dataset performance is at chance, but the 64-cell matrix carries more structure than a single mean. Three properties of it matter for anyone reading a same-dataset AUC as evidence of capability.
6
TABLE IV C OMPLETE CROSS - DATASET AUDIT: FRAME - LEVEL ROC-AUC FOR EVERY TRAIN → TEST PAIR ( ROWS ) AND EVERY BACKBONE ( COLUMNS ). B OLD , •- MARKED ROWS ARE THE CALIBRATED SAME - DATASET DIAGONAL ; ALL OTHER ROWS ARE CROSS - DATASET AND SIT AT OR BELOW CHANCE . Train → Test
CLIP
DINOv2
ResNet
EffNet
Ped1 → Ped1 • Ped1 → Ped2 Ped1 → Avenue Ped1 → ShTech Ped2 → Ped1 Ped2 → Ped2 • Ped2 → Avenue Ped2 → ShTech Avenue → Ped1 Avenue → Ped2 Avenue → Avenue • Avenue → ShTech ShTech → Ped1 ShTech → Ped2 ShTech → Avenue ShTech → ShTech •
0.683 0.566 0.560 0.354 0.481 0.855 0.509 0.340 0.587 0.531 0.603 0.440 0.605 0.447 0.524 0.683
0.688 0.564 0.436 0.380 0.484 0.901 0.391 0.350 0.462 0.570 0.625 0.498 0.540 0.714 0.486 0.686
0.599 0.643 0.408 0.434 0.399 0.828 0.461 0.412 0.479 0.652 0.622 0.489 0.534 0.671 0.553 0.697
0.662 0.650 0.499 0.462 0.480 0.797 0.437 0.434 0.499 0.374 0.619 0.492 0.563 0.525 0.593 0.713
TABLE V O PERATING - POINT VIEW PER BACKBONE : EQUAL - ERROR RATE AND MEDIAN FALSE ALARMS PER HOUR ( AT 0.9 RECALL , 10 FPS ), SPLIT INTO SAME - DATASET AND CROSS - DATASET EVALUATIONS . C ROSS - DATASET EER IS AT THE RANDOM - SCORER VALUE OF 0.5. EER
False alarms/hr
Model
Same
Cross
Same
Cross
CLIP ViT-B/32 DINOv2 ViT-S/14 ResNet-50 EfficientNet-B0
0.348 0.324 0.365 0.358
0.505 0.508 0.493 0.503
25,568 26,491 28,072 26,553
32,169 33,004 32,568 32,075
Mean
0.349
0.502
26,406
32,456
A. Transfer is directional Cross-dataset transfer is not symmetric: a model calibrated on dataset A and tested on B does not behave like one calibrated on B and tested on A. Table VI gives, for each unordered pair, the mean AUC in each direction. The directional gap reaches 0.205 for the Ped2/ShTech pair and averages 0.113 over all pairs. A normality model trained on a busier, more varied scene tends to flag a quieter scene’s normal frames as anomalous more often than the reverse, so the same two cameras give very different results depending on which one was used for calibration. There is no single number that summarises how a pair of scenes transfer. B. Some scenes are simply hard to transfer onto Averaging over backbones and source datasets, Table VII reports how well any out-of-domain model scores when tested on each dataset. ShTech is the hardest target: every model built on a different scene scores a mean AUC of only 0.424 on it, below chance. A deployment cannot know in advance whether its cameras resemble the easy or the hard targets, which is exactly the uncertainty that a same-dataset benchmark hides. C. Part of the same-dataset score is base rate, not skill Even the calibrated diagonal is partly an artefact of class balance. Across the four datasets the same-dataset AUC is positively correlated with the fraction of anomalous frames in the test set (Pearson r = 0.68, Figure 4): the datasets with many anomalous frames post the highest same-dataset AUC. ROC-AUC is formally threshold and prevalence independent, so this correlation reflects that the higher-anomaly datasets here are also the ones
7
TABLE VI D IRECTIONAL ASYMMETRY OF CROSS - DATASET TRANSFER ( MEAN OVER BACKBONES ). A→B IS THE AUC WHEN THE NORMALITY MODEL IS BUILT ON A AND TESTED ON B. T HE TWO DIRECTIONS DIFFER BY UP TO 0.205. Source A
Source B
A→B
B→A
|∆|
Ped1 Ped1 Ped1 Ped2 Ped2 Avenue
Ped2 Avenue ShTech Avenue ShTech ShTech
0.606 0.476 0.407 0.449 0.384 0.480
0.461 0.507 0.560 0.532 0.589 0.539
0.145 0.031 0.153 0.082 0.205 0.059
TABLE VII P ER - TARGET TRANSFERABILITY: MEAN AND WORST- CASE CROSS - DATASET AUC WHEN TESTING ONTO EACH DATASET, OVER ALL BACKBONES AND SOURCE DATASETS . Test dataset Ped1 Ped2 Avenue ShTech
Mean cross-dataset AUC
Worst case
0.509 0.576 0.488 0.424
0.399 0.374 0.391 0.340
whose anomalies are visually more separable, but it is a warning nonetheless: a headline AUC reported on one imbalanced benchmark conflates how detectable the anomalies are with how that particular test set was constructed. The cross-dataset numbers, which remove this confound by changing the scene, are the honest measure. VI. T HE COLLAPSE IS NOT AN ARTEFACT OF THE K NN DETECTOR The detector used so far scores a frame by its nearest-neighbour distance to the normal training set. A natural objection is that the collapse might be a weakness of that particular scorer rather than of the off-the-shelf features. To test this we re-score the same embeddings with a second, structurally different detector family: a PaDiM-style Mahalanobis distance [18], which fits a single Gaussian (mean and a shrinkage-regularised covariance) to the normal training features and scores each test frame by its squared Mahalanobis distance to that Gaussian. Where the nearestneighbour detector is local and non-parametric, the Mahalanobis detector is global and parametric, so agreement between them isolates the role of the representation. We run this comparison on the three datasets re-embedded for the experiment (UCSD Ped1, UCSD Ped2, and CUHK Avenue), across all four backbones, for every train and test pair. Table VIII and Figure 5 show that the two detectors behave almost identically. The nearest-neighbour detector falls from a same-dataset AUC of 0.707 to a cross-dataset 0.505 (a gap of 0.202), and the Mahalanobis detector falls from 0.704 to 0.497 (a gap of 0.208). Both land at chance off the diagonal, and the worst cross-dataset cell is 0.374 for the nearest-neighbour detector and 0.354 for the Mahalanobis one, both well below chance. Swapping a local non-parametric scorer for a global parametric one changes the cross-dataset result by less than one AUC point. The collapse is therefore a property of what the frozen features encode, a scene-specific appearance model, and not of the distance used to read them out. VII. D ISCUSSION A. Benchmark AUC describes a calibrated lab, not a deployment The same-dataset AUCs we measure are consistent with the regime the literature reports, yet they evaporate the moment the test camera differs from the training camera. Surveillance is the cross-dataset setting by definition. Reporting only same-dataset AUC therefore answers a question a deployment never asks.
8
same-dataset ROC-AUC
0.85
Same-dataset AUC vs base rate (r = 0.68)
Ped2
0.80 0.75 0.70
ShTech Ped1
0.65
Avenue
0.1
0.2 0.3 0.4 0.5 0.6 0.7 anomalous-frame fraction of test set
0.8
Fig. 4. Same-dataset ROC-AUC against the anomalous-frame fraction of each test set. The positive correlation (r = 0.68) shows part of the calibrated score tracks how the benchmark is balanced, not detection skill alone. TABLE VIII S AME - DATASET VERSUS CROSS - DATASET FRAME ROC-AUC FOR TWO DETECTOR FAMILIES ON THE THREE RE - EMBEDDED DATASETS (UCSD P ED 1, UCSD P ED 2, CUHK AVENUE ), AVERAGED OVER THE FOUR BACKBONES . B OTH FAMILIES COLLAPSE TO CHANCE ACROSS DATASETS . Detector Nearest neighbour Mahalanobis (PaDiM)
Same-dataset
Cross-dataset
Gap
Worst cross
0.707 0.704
0.505 0.497
0.202 0.208
0.374 0.354
B. Rarity is not threat The labels in these datasets mark the statistically unusual: a bike on a walkway, a person running. An anomaly detector flags distribution shift, not danger, and our cross-dataset result shows it does not even flag distribution shift reliably once the background distribution changes. Automated “suspicious behavior” detection inherits both problems: it conflates rare with dangerous, and it does not transfer across scenes. C. Stronger features are not the fix The result that DINOv2 is best in-domain and worst cross-domain warns against the intuition that a better backbone closes the gap. Within this off-the-shelf, frozen-feature regime, representation quality and cross-scene transfer trade off. Closing the gap needs scene-invariant modelling, not a larger encoder. D. The operating point, not the AUC, decides feasibility The false-alarm rates of Section IV are easier to feel as a rate per second: the median 31,931 per hour is roughly nine alarms every second at 10 fps, and even the calibrated same-dataset rate is about seven per second. No human monitoring a camera wall can triage at that volume; the stream becomes noise. The point for the field is that an AUC in the 0.7–0.9 range, routinely reported as evidence of deployable performance, can correspond to an operating point a reviewer cannot use at all. Feasibility is set by the operating point, which stays prohibitive whether or not the scene is calibrated, so an AUC alone should not be read as a deployability claim. VIII. L IMITATIONS We study 4 datasets and two frozen-feature detector families (a nearest-neighbour distance on all four datasets and a Mahalanobis detector on the three re-embedded for Section VI); end-to-end trained VAD methods or temporal
9
1.0
The collapse holds for both detector families chance same-dataset cross-dataset
frame ROC-AUC
0.8 0.6 0.4 0.2 0.0
Nearest neighbour
Mahalanobis (PaDiM)
Fig. 5. Same-dataset and cross-dataset AUC for the nearest-neighbour and the Mahalanobis (PaDiM-style) detector. The two detector families collapse to the chance line together, so the failure is in the representation, not the scorer.
models may shift the absolute numbers, though the cross-dataset question we raise applies to them too and is rarely reported. The detector-family control uses three datasets rather than four because it required re-embedding the frames, and we omitted the download-gated ShanghaiTech set there; the main matrix retains all four. The framepositive convention (any anomalous pixel marks the frame) is standard but coarse. The false-alarm rate depends on the assumed frame rate and operating point, which we state. UCSD frames are grayscale and are replicated to three channels for the RGB backbones. We use one k for the nearest-neighbour score and do not sweep it. A larger study would add an open-set dataset such as UBnormal, and would test the prediction and reconstruction detector families side by side. For ShanghaiTech we use a public redistribution of the frames with segment-level test annotations and subsample the large training and test splits, which may shift its absolute AUC from the canonical 107-clip protocol; the cross-dataset comparison is unaffected. IX. F UTURE W ORK The audit isolates one detector to make the cross-scene question clean, and the natural next step is to show whether the collapse survives stronger detectors. We would add end-to-end trained families, future-frame prediction and reconstruction autoencoders, and temporal models that score short clips rather than single frames, running each through the same same-versus-cross matrix; if the gap persists for them, the result generalises beyond frozen features. A second direction is an open-set benchmark such as UBnormal, where train and test anomalies are disjoint by construction, which stresses cross-distribution transfer more sharply than the single-scene datasets used here. Finally, the only constructive route the evidence points to is scene-invariant or domain-adapted normality modelling: background subtraction, camera-pose normalisation, or test-time adaptation that updates the normal model from a short slice of the new camera’s footage. Measuring how much of the cross-dataset gap each of these recovers would turn the negative result into a target for methods that aim at deployable, rather than benchmark, reliability. X. C ONCLUSION Off-the-shelf normality models detect anomalies well when the test camera is the training camera and not at all when it is not. Across 4 datasets and 4 backbones, same-dataset frame AUC of 0.704 falls to a chance-level 0.499 cross-dataset, the best in-domain backbone transfers the worst, and the false-alarm rate is on the order of 31,931 per
10
hour at a usable recall. The headline reliability of surveillance anomaly detection is a property of the benchmark protocol, not of a deployable system, and should be reported as a cross-dataset number with an operating point before it informs any decision. R EFERENCES [1] V. Mahadevan, W.-X. Li, V. Bhalodia, and N. Vasconcelos, “Anomaly detection in crowded scenes,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2010, pp. 1975–1981. [2] C. Lu, J. Shi, and J. Jia, “Abnormal event detection at 150 FPS in MATLAB,” in IEEE International Conference on Computer Vision (ICCV), 2013, pp. 2720–2727. [3] W. Liu, W. Luo, D. Lian, and S. Gao, “Future frame prediction for anomaly detection: A new baseline,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 6536–6545, arXiv:1712.09867. [4] B. Ramachandra, M. J. Jones, and R. R. Vatsavai, “A survey of single-scene video anomaly detection,” IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), vol. 44, no. 5, pp. 2293–2312, 2022, arXiv:2004.05993. [5] G. Pang, C. Shen, L. Cao, and A. van den Hengel, “Deep learning for anomaly detection: A review,” ACM Computing Surveys, vol. 54, no. 2, pp. 1–38, 2022, arXiv:2007.02500. [6] D. Gong, L. Liu, V. Le, B. Saha, M. R. Mansour, S. Venkatesh, and A. van den Hengel, “Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection,” in IEEE/CVF International Conference on Computer Vision (ICCV), 2019, memAE; arXiv:1904.02639. [7] H. Park, J. Noh, and B. Ham, “Learning memory-guided normality for anomaly detection,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 14 372–14 381, mNAD. [8] K. Roth, L. Pemula, J. Zepeda, B. Schölkopf, T. Brox, and P. Gehler, “Towards total recall in industrial anomaly detection,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, patchCore; arXiv:2106.08265. [9] D. Hendrycks and K. Gimpel, “A baseline for detecting misclassified and out-of-distribution examples in neural networks,” in International Conference on Learning Representations (ICLR), 2017, arXiv:1610.02136. [10] W. Sultani, C. Chen, and M. Shah, “Real-world anomaly detection in surveillance videos,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 6479–6488, arXiv:1801.04264. [11] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” in International Conference on Machine Learning (ICML), 2021, arXiv:2103.00020. [12] M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, C. Schuhmann, L. Schmidt, and J. Jitsev, “Reproducible scaling laws for contrastive language-image learning,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, openCLIP; arXiv:2212.07143. [13] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby et al., “DINOv2: Learning robust visual features without supervision,” Transactions on Machine Learning Research (TMLR), 2024, arXiv:2304.07193. [14] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778, arXiv:1512.03385. [15] M. Tan and Q. V. Le, “EfficientNet: Rethinking model scaling for convolutional neural networks,” in International Conference on Machine Learning (ICML), 2019, arXiv:1905.11946. [16] R. Wightman, “PyTorch image models (timm),” https://github.com/huggingface/pytorch-image-models, 2019, accessed 2026-06-15. [17] A. Paszke, S. Gross, F. Massa, A. Lerer et al., “PyTorch: An imperative style, high-performance deep learning library,” Advances in Neural Information Processing Systems (NeurIPS), 2019. [18] T. Defard, A. Setkov, A. Loesch, and R. Audigier, “PaDiM: A patch distribution modeling framework for anomaly detection and localization,” in International Conference on Pattern Recognition (ICPR) Workshops, 2021, arXiv:2011.08785.