Evaluating Design Video Generation: Metrics for Compositional Fidelity
Adrienne Deganutti 1 Dingning Cao 1 Jaejung Seol 1 Elad Hirsch 1 Purvanshi Mehta 1
arXiv:2605.16223v1 [cs.GR] 15 May 2026
Abstract Generative video models are increasingly used in design animation tasks, yet no standardized evaluation framework exists for this domain. Unlike natural video generation, design animation imposes structured constraints: specific components shall animate with prescribed motion types, directions, speed and timing, while non-animated regions must remain stable and layout structure must be preserved. This paper provides a fully automated evaluation framework organized across four dimensions: layout fidelity, motion correctness, temporal quality, and content fidelity. This eliminates the reliance on subjective human evaluation and establishes a common basis for benchmarking progress in the field. We release the code and dataset here: https://github. com/purvanshi/lica-bench.
Figure 1. Overview of the proposed framework. We evaluate design videos across four dimensions: motion type, motion direction, duration, and text recoverability.
constraint-driven requirements — specific components must animate with prescribed motion types and timing while nonanimated regions remain stable and the spatial layout is preserved.
1. Introduction Design animations are a cornerstone of modern digital communication. Producing them requires specialized expertise: a designer must select which components to animate, choose appropriate motion types (e.g., fade, slide, scale, rotate), specify direction and timing, and ensure that the overall composition remains coherent throughout. Recent advances in generative video models have opened the door to automating this workflow, with models such as Sora (OpenAI, 2024) and Veo (Google DeepMind, 2024) now being applied to design animation tasks where a static layout and per-component motion specification are provided as input. Yet early benchmarking results reveal systematic shortcomings: models struggle to ground animation instructions to the correct visual regions, frequently applying a single dominant motion globally, animating the wrong elements, or hallucinating motion unrelated to the prompt. These failures underscore that design animation is not a subproblem of natural video synthesis but a distinct task with structured,
Despite this progress, evaluation of generated design animations remains ad hoc: most work reports Fréchet Video Distance or relies on small-scale human studies, neither of which captures the structured requirements unique to design animation, namely that specific components must move in prescribed ways while the rest of the composition remains intact. In this paper, we propose a fully automated evaluation framework organized along four complementary dimensions: layout fidelity, motion correctness, temporal quality, and content fidelity, that together provide a comprehensive and reproducible measure of design animation quality, eliminating the need for costly subjective assessment and establishing a common benchmark for the field. Our key contributions are as follows.
Lica World, San Francisco, United States of America. Correspondence to: Purvanshi Mehta <[email protected]>.
• The first dimension-decomposed evaluation framework for design animation, where each axis is independently grounded in the layout specification.
Preprint. May 18, 2026.
• Within motion correctness, a suite of four sub-metrics:
1
1
Evaluating Design Video Generation: Metrics for Compositional Fidelity
motion type, direction, animation duration, and component visible duration, with ground-truth signals automatically extracted from the layout metadata.
spans six primary visual domains with 30 subfields and varying video durations to stress-test temporal reasoning in MLLMs. These benchmarks target naturalistic or cinematic content and do not account for the compositional, typographic, or brand-driven properties of designed video.
• A two-track benchmark of 86 single-component clips (isolating one element on a uniform background) and 136 full-layout templates (894 component evaluations, ∼6.6 per template), spanning 15 animation types across image, text, and group component families.
Advertisement and Creative Video. A growing set of works recognizes the distinct challenges of commercial video content. VideoAds (Zhang et al., 2025b) benchmarks MLLMs on advertisement videos with complex temporal structures across visual finding, summarization, and reasoning tasks, finding that even the strongest models fall well short of human performance. AdsQA (Long et al., 2025) provides a QA benchmark derived from over 1,500 ad videos across five tasks centered on the persuasive structure of advertising. E-VAds (Liu et al., 2026) targets ecommerce short videos, identifying a gap in benchmarks that address commercial intent reasoning rather than general action recognition or commonsense QA. While these works probe semantic and narrative understanding of ads, none evaluate the design-specific dimensions: layout, typography and motion composition, that our benchmark is built around.
• Empirical validation showing that our framework recovers ground-truth motion attributes with 87.8% accuracy on single-component renders using a transparent, rule-based pipeline with no learned classifier, and a comparative study of Sora-2 and Veo-3.1 revealing that the two models exhibit sharply different failure modes along axes that aggregate perceptual scores cannot separate.
2. Related Works Graphic Design Datasets and Benchmarks. Static graphic design research has produced a range of layout datasets, from mobile UI collections such as Rico (Deka et al., 2017) to multi-layer template datasets like Crello (Yamaguchi, 2021), poster benchmarks such as PosterLayout (Hsu et al., 2023) and CGL (Zhou et al., 2022), and e-commerce banner datasets (Yu et al., 2024). On the evaluation side, DesignProbe (Lin et al., 2024) benchmarks MLLMs on color, typography, and layout understanding, while AesEval-Bench (An et al., 2026) targets aesthetic quality assessment and Graphic-Design-Bench (Deganutti et al., 2026) grounds tasks in full structural metadata to enable layer-aware evaluation. Recent visual-text benchmarks further show that text rendering remains a distinct failure mode for generative models: OCRGenBench (Zhang et al., 2025a) evaluates OCR-related generative capabilities across document, scene-text, artistic-text, and layout-rich settings, and PosterCraft (Chen et al., 2025) emphasizes that high-quality poster generation requires both accurate text rendering and aesthetic layout composition. Except for (Deganutti et al., 2026), these benchmarks operate on static images, leaving the temporal and motion dimensions of graphic design unaddressed.
3. Dataset Evaluating design video generation at the component level requires ground-truth data that existing video benchmarks do not provide: per-component bounding boxes, animation type and timing metadata, and deterministic reference renders against which metrics can be validated. Natural-video datasets lack layered structure entirely, and even GDB’s animation evaluation set (Deganutti et al., 2026) comprises only 10 samples per generation task with human-only evaluation, making it insufficient for automated metric development. We therefore construct a dedicated evaluation dataset from the LICA layered-composition corpus (Hirsch et al., 2026), which uniquely supplies the full component-level metadata our framework requires. Specifically, we draw from the public release lica-dataset and filter to entries whose sub-category is videos, yielding 136 source layouts spanning 66 unique templates. From each layout we extract the canvas specification (dimensions, background color, total duration), and for every component its type (text, image, or group), bounding box, and animation attributes including motion type (drawn from LICA’s 31-type taxonomy), direction, animation duration, trigger time, and visible duration. Each animated component carries exactly one motion type assigned at authoring time in LICA; we extract these assignments verbatim and do not synthesize or reassign animations. For text components we additionally extract the rendered string, font family, font size, weight, style, color, line height, letter spacing, and alignment. A component is treated as animated when it carries a non-empty animations list;
Video Generation and Understanding Benchmarks. General video benchmarks fall into two tracks. VBench (Huang et al., 2024) decomposes generation quality into 16 hierarchical dimensions, including subject consistency, motion smoothness, and temporal flickering. validated against human preference annotations. EvalCrafter (Liu et al., 2024) evaluates text-to-video models across visual, content, and motion quality dimensions using 700 prompts derived from real-world user data. For multimodal understanding, Video-MME (Fu et al., 2025) 2
Evaluating Design Video Generation: Metrics for Compositional Fidelity Table 1. Dataset statistics. The evaluation set is drawn from the LICA videos sub-category in lica-dataset. The full-layout track inherits LICA’s full 31-type motion vocabulary; the singlecomponent track restricts to a 15-type subset, of which 12 are observed after candidate filtering.
component trees are walked recursively so that animated elements nested inside groups are not missed. We construct evaluation data at two levels of complexity. Both tracks are rendered deterministically from the extracted layout JSON, so each sample’s ground-truth video and finalframe PNG are byte-exact references against which our metrics can be validated. The full-layout track preserves complete multi-component compositions, retaining both animated and static elements from each source layout and inheriting the full LICA motion vocabulary (up to 31 types per scene mix). All 136 video-subset layouts are kept as candidates and rendered to (layout JSON, rendered video MP4, final-frame PNG, prompt) tuples; animated component counts range from 0 to 30 per scene (mean 6.6 across all 136 layouts; 7.5 across the 120 layouts with at least one animated component). The resulting animated-component pool spans 520 image, 374 text, and 0 group elements across successful renders. The single-component track isolates one animated element per source layout onto a white canvas to provide an atomic motion benchmark. To keep this track focused on motion types that are both visually salient and frequent enough to score reliably, we restrict it to a 15-type subset of the LICA taxonomy (rise, pop, fade, wipe, pan, burst, ascend, bounce, tumble, rotate, drift, shift, skate, photorise, photoflow); for each source layout we traverse all components recursively and admit only those whose pre-existing animation type falls in this allowlist, enforcing a maximum of one accepted component per layout to avoid dense duplicates from the same source scene. The retained components keep their original LICA animation parameters verbatim; we do not modify or reassign motion types, directions, or timings. To keep extraction backgrounds clean while preserving visibility, the canvas defaults to white, switching to black when a text component’s color is near-white. Each accepted candidate is rendered, yielding 86 retained samples from 101 candidates, balanced across 37 image, 13 text, and 37 group components and covering 12 of the 15 allowlisted motion types (three types had no candidates clear all filters).
Source data (LICA, videos sub-category) Video layouts / unique templates Layouts with animated components Animated comp. per layout (mean, all / animated) Animated comp. per layout (min / max)
136 / 66 120 6.6 / 7.5 0 / 30
Full-layout track (136 samples) Animated components (image / text) Animated comp. per scene (min / max / mean) Motion types
520 / 374 0 / 30 / 6.6 21
Single-component track (86 samples) Component families (image / text / group) Motion types (before / after filter) Candidates before motion filter Motion-score threshold
37 / 13 / 36 15 / 12 101 0.03
4. Evaluation Framework The design video generation framework introduces a comprehensive evaluation framework spanning four competency dimensions: Motion Type, Direction, Duration, and Text Recoverability which reflects the natural pipeline of graphic design animation skills, enabling systematic assessment of model capabilities across a wide spectrum of practical applications. 4.1. Tracking We propose two tracking frameworks, each aligned with a benchmark setting: single-component and multi-component (full) layouts. The single-component framework uses temporal differencing across frames, while the multi-component framework relies on spatial matching against known layout geometry to separate and identify multiple components on each frame independently. Both frameworks output a standardized per-frame record that contains a bounding box and confidence score for every tracked component, making the evaluation metric layer entirely tracker-agnostic.
For each animated component, the prompt enumerates motion type, normalized direction, animation duration, speed, trigger phase, timing window (from, duration), and positional geometry, plus type-specific visual fields (text content and full typographic specification for text; alt descriptions for image and vector). Static elements are included in the layout JSON and final-frame render but excluded from the animated-component enumeration in the prompt. Every generated sample preserves provenance back to its source: the manifest records the originating template id, layout id, component ids, component-type list, animation-type list, and all artifact paths, so each evaluation sample is auditable to a specific LICA public dataset entry.
Contour OBB tracker (single-component layouts). Single-component renders sit on a white uniform background. Each frame is differenced against the rendered background color to isolate moving regions, which are then grouped into components and represented as oriented bounding boxes (OBBs). Because there is exactly one component of interest per video, no track-association step is required. This backend achieves 99.9% mean detection presence across the 86 single-component prompts for both evaluated video generation models, making it suitable as a near-ceiling reference for the metric pipeline. 3
Evaluating Design Video Generation: Metrics for Compositional Fidelity Table 2. Grouping of LICA motion verbs into observable motion classes.
YOLO-OBB tracker (full layouts). Videos in the full layout track contain multiple animated components (image and text) layered onto a background image or design. Therefore, unlike single-component layouts, the background cannot be used as a reference to track the components. To handle this, we finetune an oriented-bounding-box (OBB) variant of YOLOv11 (Khanam & Hussain, 2024) on ∼ 500k layouts from the LICA dataset (Hirsch et al., 2026) with a class set of {IMAGE, TEXT}. Rather than linking detections across frames with a multi-object tracker, we exploit the fact that the layout metadata provides ground-truth positions for every component. For each frame, we independently run the detector and use the Hungarian algorithm to match detected OBB polygons to known layout polygons based on polygon IoU. Because the layout metadata and the generated video may differ in resolution, we apply an isotropic scaling to the layout polygons before any IoU comparison.
Behavior
LICA verbs
Class
Translation rise, ascend, drift, pan, wipe, shift, scrapbook skate, photorise, photoflow, scrapbook, bounce, stomp Rotation
tumble, roll, rotate
rotate
Opacity
baseline, fade, flicker, blur, merge, fade clarify, succession, typewriter
Scale
burst, pop
pop
Oscillation
wiggle
wiggle
Oscillation
breathe, pulse
breathe
4.2. Motion-Type Observable Motion Groups. The layout metadata from the LICA dataset (Hirsch et al., 2026) contains a fine-grained vocabulary of thirty motion verbs (e.g., rise, ascend, drift, tumble, baseline, burst). Many of these verbs differ only in easing curve or timing and produce indistinguishable OBB trajectories (e.g., rise and ascend both translate the component upward). We therefore group the raw vocabulary into a smaller set of ten observable motion classes that a geometric tracker can reliably distinguish: ( ) static, fade, scrapbook, pop, wiggle, T = breathe, rotate, pan, sketch, neon
Figure 2. Visual examples of motion types.
credit): pop ↔ scrapbook (0.5), fade ↔ scrapbook (0.3), wiggle ↔ breathe (0.4), and the asymmetric entries scrapbook → pan (0.3), rotate → scrapbook (1.0), and scrapbook → rotate (0.5). The asymmetric rotate→scrapbook entry awards full credit because LICA’s tumble and roll entries combine a translational entry with rotation, so a scrapbook prediction is observationally equivalent. When the ground-truth label is unknown, the metric awards 1.0 for predictions of static or unknown and 0.0 otherwise.
A key design decision in this grouping concerns how the LICA dataset’s reference renderer executes motion. Most animations in the LICA dataset are on-enter effects: the component animates into its final position over a brief window (typically 0.56–1.12 s) and then remains static for the rest of the clip. From the tracker’s perspective, this appears as a short burst of motion at the start of the clip followed by a long static plateau. This signature matches the scrapbook class (a transient entry animation) rather than a sustained pan (continuous translation throughout the clip). The same applies to rotational entries such as tumble: the component rotates into place over ∼1 s and then stops, producing a brief rotational transient rather than sustained rotation. The grouping therefore reflects the observable trajectory behavior rather than the semantic intent of the animation grammar. Animations with similar temporal signatures are consequently assigned to the same observable class:
The baseline classifier is a rule-based decision tree described in Table 3, grounded in design-animation terminology. The tree structure mirrors the observable motions of animation families (e.g., continuous movement for pan, opacity changes for fade), while the decision thresholds are chosen empirically by analyzing ground-truth examples and setting boundaries that best separate the different animation types. Energy (E) is computed over the start, middle, and end thirds of each clip to capture temporal dynamics (high energy = lots of motion). The noise floor of the tracker is set to ε ≈ 3 × 10−3 such that any motion smaller than this is considered tracker jitter. All measurements are normalized by the diagonal of the canvas. An in-depth description of the motion classification rules is provided in Appendix B.
Motion-Type Classification. We classify the observed motion into t̂ ∈ T from tracked trajectory features and report accuracy with a partial-credit table for semantically related labels. We list pairs as (GT → predicted: 4
Evaluating Design Video Generation: Metrics for Compositional Fidelity Table 3. Overview of the priority-ordered motion classification rules. Additional details are given in Appendix B. #
Rule
1 2 3 4 5 6 7 8 9 10 11 12
low presence → static opacity transient → fade scale spike with low displacement → pop entry rotation transient → rotate entry motion transient → scrapbook sustained rotation → rotate high-frequency positional jitter → wiggle periodic scale oscillation → breathe sustained directional displacement → pan late opacity/presence change → fade residual motion energy → scrapbook otherwise → static
where ∆ denotes the per-frame difference, ct is the normalized centroid, st the scale, θt the rotation, and ot the opacity proxy. After smoothing, we define the predicted duration as the time between the first and last frames where Et exceeds 15% of its peak value. We report mean absolute error (MAE) in seconds. Component visible duration. Each component is specified to be visible from tstart to tend . We estimate the visible duration τ̂v as the longest contiguous run of detector presence, converted to seconds via the frame rate. We report MAE against the ground-truth visible interval. 4.5. Text Recoverability For text components, content fidelity requires that the strings specified by the layout remain readable in the generated video. We evaluate this at the component level by applying OCR to sampled video frames and matching recognized text lines to the text components in L. We sample frames at 2 fps for text recoverability evaluation, which provides a practical trade-off between score stability and evaluation cost (Appendix F).
4.3. Motion Direction Each animated component specifies a motion direction d belonging to one of eight compass directions, a rotation or none such that: ) ( right, left, up, down, up right, up left, down right d∈ down left, clockwise, anticlockwise, none
Let Itext be the set of text components in L, with groundtruth strings {gi }i∈Itext . For each sampled frame t ∈ T (V ), OCR returns recognized strings Pt = {pt,1 , . . . , pt,Kt }. We use character error rate as the matching cost,
We predict direction dˆ by measuring the displacement of the component centroid and bin the movement angle to the nearest compass direction used by the layout vocabulary. The segment of the trajectory used relies on the motion-type assigned in the previous step: (a) Transient motions (scrapbook, pop, fade): displacement from t = 0 to t = N/4, capturing the entry direction before the component settles; (b) Pan: net displacement over the full clip, since the component moves the entire time; (c) Rotate: the OBB angle series is unwrapped with period 90° (resolving the axis-swap ambiguity of cv2.minAreaRect) and the sign of the total angle change determines clockwise or anticlockwise when |∆θ| ≥ 45°; (d) Non-translational motions (static, wiggle, breathe): prediction is suppressed to none, as these motions have no meaningful linear direction. We report accuracy by comparing the predicted direction dˆ to the ground truth d.
dCER (g, p) =
Lev(g, p) . max(|g|, 1)
Here, Lev(g, p) denotes the Levenshtein distance between strings g and p, and |g| is the number of characters in the ground-truth string. For each frame, we build a cost matrix between layout strings and OCR lines and solve a minimum-cost bipartite assignment: X πt∗ = arg min dCER (gi , pt,πt (i) ). πt
i∈Itext
When fewer OCR lines than layout strings are detected, the OCR side is padded with dummy columns of cost 1. Extra OCR lines that are not assigned to any layout string are ignored. This frame-wise assignment handles multiple text components without relying on OCR reading order.
4.4. Animation & Component Duration We evaluate two related but distinct timing properties. Animation duration measures how long the motion itself lasts (e.g., a component might slide in over 0.5 seconds and then remain still for the rest of the video.) Component visible duration measures how long the component is on screen, regardless of whether it is moving.
After frame-wise assignment, each layout text component selects its best assigned OCR match over time: Atext (V, L) =
Animation duration. Each layout specifies a ground-truth animation duration τa (in seconds). We estimate τ̂a from the motion-energy signal that combines per-frame changes in centroid position, scale, rotation, and opacity:
1
X
max |Itext | t∈T (V ) i∈Itext 1 − dCER (gi , pt,πt∗ (i) ) + .
This aggregation allows different text components to become readable in different frames. We report this score
Et = ∥∆ct ∥ + 0.5 |∆st | + 0.3 |∆θt |/90 + 0.5 |∆ot | 5
Evaluating Design Video Generation: Metrics for Compositional Fidelity Table 4. Motion-type classification results across both benchmark tracks and video sources. GT (italicized) reports the evaluation ceiling on LICA renders and is excluded from best-value bolding. Single-component uses a contour OBB tracker. Full-layout uses a finetuned YOLO-OBB detector with per-frame spatial matching and is reported on the full set and tracker-reliable subset (presence ≥ 0.3). Motion accuracy uses partial credit. MAE is in seconds. Bold indicates the best value per column among generators within each track block. Track
Model
n / Rel.%
Pres. ↑ Motion ↑ Anim. MAE ↓ Comp. MAE ↓
GT Single Comp. Veo-3.1 Sora-2
86 86 86
0.930 0.999 1.000
0.878 0.649 0.602
3.01 4.34 2.41
0.65 3.99 3.85
Full (all)
GT Veo-3.1 Sora-2
894 894 894
0.759 0.499 0.513
0.594 0.455 0.423
2.92 2.15 2.87
2.87 5.22 5.27
Full (rel.)
GT 728 (81.4%) Veo-3.1 473 (52.9%) Sora-2 499 (55.8%)
0.921 0.892 0.885
0.690 0.655 0.615
3.13 2.69 3.52
1.53 3.75 3.95
5.2. Evaluation Pipeline Validation
as Best AR (best accuracy rate), since it averages the best assigned recognition score for each text component over time. We also report Hard AR, Htext , defined as the fraction of layout text components whose assigned candidate is an exact match in at least one sampled frame.
Before comparing generated videos, we validate the evaluation pipeline on LICA reference renders. For motion type, direction, and duration, the GT entries in Tables 4, 6, and 7 estimate the ceiling imposed by the tracker/classifier pipeline when the video exactly follows the layout specification. Text recoverability has an additional source of evaluator variation because candidate strings can be extracted either by OCR or by an LLM-based transcription model. We therefore validate both text evaluators on reference-rendered scenes, where the target text is known to be present (Table 5). In both cases, the evaluator only extracts candidate strings; the final score is computed by the same CER-based assignment metric defined in Section 4.5. All text scores use frames sampled at 2 fps, following Appendix F.
5. Experiments We evaluate two state-of-the-art video generation models, Sora-2 (OpenAI, 2024) and Veo-3.1 (Google DeepMind, 2024), on both benchmark tracks (single-component and full-layout). We additionally run the full pipeline on groundtruth videos from the LICA dataset (Hirsch et al., 2026) to establish an upper bound on what the tracker/classifier cascade can achieve, separating metric-pipeline limitations from generator limitations.
Best AR ↑
Hard AR ↑
OCR LLM
0.769 1.000
0.769 1.000
OCR LLM
0.862 0.960
0.659 0.940
5.1. Experimental Setup
Dataset
Evaluator
Both models receive identical layout specifications and prompt formulations; the resulting clips are scored without any per-model tuning of thresholds in the evaluation pipeline. The single-component track contains 86 videos per generator. The full-layout track contains 136 videos per generator, yielding 894 component-level evaluations per generator, averaging ∼7.3 animated components per layout. Single-component videos use the contour OBB tracker (Section 4), which is deterministic and model-free. Full-layout videos use the finetuned YOLO-OBB tracker as the primary backend. We define a component evaluation as trackerreliable when its detection presence fraction exceeds 0.3 (i.e., the component is detected in at least 30% of frames). We report metrics in Table 4 on both the all-components pool and the tracker-reliable subset and treat the gap between them as a first-class evaluation signal.
Single Full
Table 5. Text evaluator validation on reference-rendered scenes.
5.3. Motion-Type Classification. The results on motion-type classification are given in Table 4. We provide confusion matrices for all models in each track in Appendix C. We break down the sources of error below. Single-component. 83% of samples receive a perfect score. The remaining 12% of errors come from three confusable cases: (i) fade→scrapbook, short fade-ins induce transient centroid jitter from changing contours; (ii) pop↔scrapbook, objects simultaneously scale up and move into place, making it difficult to distinguish scaling motion from entry translation; and (iii) three low-amplitude cases collapsing to 6
Evaluating Design Video Generation: Metrics for Compositional Fidelity
5.4. Motion Direction
static below the tracker noise floor. These errors that persist on ground-truth renders reflect the limitation of the tracker signal (e.g., opacity is only indirectly observed via bbox changes).
We report direction accuracy in Table 6. Table 6. Motion direction evaluation on single-component and fulllayout tracks. Motion detected is the fraction of directional components for which the classifier correctly predicts any compass label other than none. Correctly none is the fraction of non-directional components where the classifier correctly predicts none. Correct half is the fraction of pan-family components whose predicted compass label falls in the correct horizontal half (left-family or right-family) relative to the LICA-canonical direction. Full-layout results use only tracker-reliable components, hence denominators vary by model.
Full-layout. The 19-point drop from single-component (0.878) to full-layout reliable (0.690) is dominated by the collapse of fade- and pop-family components onto the scrapbook class. Per-class GT accuracy is 0.957 for scrapbook, 0.946 for rotate, 0.489 for pop, 0.373 for fade, 0.154 for breathe, and 0.0 for static. Of the 340 non-perfect predictions on GT full-layout, three confusions account for 83% of them: (i) fade→scrapbook (156, 46% of nonperfect), (ii) pop→scrapbook (127, 37%), and (iii) rotate→scrapbook (67, full credit but visually equivalent). All three failures stem from missing motion cues: the YOLOOBB tracker has no opacity signal, so fades appear as small bbox jitter and are classified as scrapbook; multi-component pops weaken the scale-pulse signature; and tumble/roll entry motions look indistinguishable from translational scrapbook entries.
Metric
GT
Sora-2
Veo-3.1
54/58 (93.1%) 18/28 (64.3%) 12/40 (30.0%) 1/18 (5.6%) 26/86 (30.2%)
40/58 (69.0%) 10/28 (35.7%) 14/40 (35.0%) 0/18 (0.0%) 19/86 (22.1%)
42/58 (72.4%) 4/28 (14.3%) 14/40 (35.0%) 0/18 (0.0%) 8/86 (9.3%)
157/207 (75.8%) 120/292 (41.1%) 39/162 (24.1%) 0/45 (0.0%) 139/499 (27.9%)
177/203 (87.2%) 53/270 (19.6%) 60/157 (38.2%) 0/46 (0.0%) 85/473 (18.0%)
Single Component Motion detected Correctly ‘none’ Correct half (pan) Rotate CW/CCW Exact acc.
Full Layout (tracker-reliable) Motion detected Correctly ‘none’ Correct half (pan) Rotate CW/CCW Exact acc.
Generator accuracy. Sora-2 reaches 0.615 and Veo-3.1 reaches 0.655 on the tracker-reliable subset of full-layout, both close to the GT ceiling of 0.690. The remaining gap is explained by two additive effects: (i) the video generator under-animates the prescribed motion (producing weaker signals that more often fall below thresholds), and (ii) the generator adds spurious motion to supposedly-static components, introducing false positives that the GT renders do not exhibit.
303/332 (91.3%) 59/396 (14.9%) 92/258 (35.7%) 1/74 (1.4%) 105/728 (14.4%)
The direction classifier reliably detects the presence of directional motion: on GT single-components it identifies nonzero displacement in 93.1% of directional components, and on the tracker-reliable subset of GT full-layouts it still recovers detectable directional motion in 91.3% of cases. For panfamily components, the “correct half” metric is well above the 25% chance baseline of an 8-way bin on both tracks (30% single-component, 35.7% full-layout) but is limited by the short LICA entry transient: the ∼17 frames of motion are often dominated by vertical “rise-into-place” dynamics, which overwhelm the weaker horizontal signal controlled by the direction parameter. Rotate CW/CCW accuracy is very low, reflecting a fundamental limitation of OBB-based angle extraction that encodes axis orientation rather than rotation direction; the converse failure mode is also visible on the full-layout reliable pool (728 components), where the GT “correctly none” rate is only 14.9% because multicomponent scenes accumulate enough incidental drift to cross the 1%-diagonal threshold on non-directional components. Despite these limitations, the method succeeds at its primary design goal as it provides a per-component, automated indicator of whether the generator produced directional motion at all, a signal that frame-level perceptual metrics cannot surface.
Tracker backend effect. We compare the finetuned YOLO-OBB against a pretrained YOLOv11x-OBB (DOTAv1 weights, no domain-specific training) on the full-layout track. Because the matcher is purely spatial polygon IoU and does not gate on detection class, the pretrained model is mechanically able to hand detections to the layoutcomponent matcher; in practice, however, design-layout text and graphical components do not look like the aerial-image classes the pretrained model was trained on, so its detections are sparse and intermittent. Concretely, the finetuned tracker recovers 2.7–2.9× more reliable components than the pretrained on GT (728 vs. 266) and Sora-2 (499 vs. 169), and 1.7× more on Veo-3.1 (473 vs. 277); it also yields higher per-component presence on the reliable subset (0.921 vs. 0.782 on GT). Both effects feed directly into motion-type accuracy on the tracker-reliable subset, where the finetuned backend leads by 8.3 points on GT (0.690 vs. 0.607), 16.2 points on Sora-2 (0.615 vs. 0.453), and 10.0 points on Veo3.1 (0.655 vs. 0.555). The pretrained run is reported in this paragraph as a baseline for whether domain-specific tracker training was necessary at all.
Generator outputs approach the GT ceiling on the full-layout track (Sora-2 75.8%, Veo-3.1 87.2% vs. 91.3% GT), but remain noticeably below it on single-component motion (69–72% vs. 93% GT). Veo-3.1 slightly exceeds the GT ceiling on pan-direction fidelity (38.2% vs. 35.7% GT), while 7
Evaluating Design Video Generation: Metrics for Compositional Fidelity
Sora-2 achieves the highest exact accuracy (27.9%) mainly because its weaker entry transients are more often classified as none.
sults separately for single-component and full-layout scenes and under both OCR and LLM evaluators. Table 8. Text recoverability of frontier video generation models on design-centric text rendering tasks. We compare Sora-2 and Veo-3.1 across controlled single-component scenes and full-layout LICA video scenes, using both evaluator types described in Section 4.5.
5.5. Duration Metrics. We report two duration metrics: animation duration (how long the component exhibits active motion) and component visible duration (how long the component is on-screen). The energy-based animation-duration estimator (Section 4.4) measures the temporal extent of above-noise motion energy; the component-duration estimator uses the first-to-last detection window. Table 7 reports MAE, median absolute error, and signed bias for both metrics. Table 7. Duration estimation errors (seconds). Single-component results use all 86 samples; full-layout results are restricted to the tracker-reliable subset. Model
Animation Duration MAE ↓
Bias
MAE ↓
Med. AE ↓
Bias
0.53 2.94 5.36
+2.61 +2.11 +4.27
0.65 3.85 3.99
0.22 1.76 3.00
−0.65 −3.45 +0.49
2.20 2.96 2.31
+2.94 +3.35 +2.46
1.53 3.95 3.75
0.57 1.51 1.41
−0.55 −1.56 −2.09
Single Component GT Sora-2 Veo-3.1
3.01 2.41 4.34
Full Layout (reliable) GT Sora-2 Veo-3.1
3.13 3.52 2.69
Generator
Evaluator
Best AR ↑
Hard AR ↑
Single Single Single Single
Sora-2 Sora-2 Veo-3.1 Veo-3.1
OCR LLM OCR LLM
0.941 0.977 0.907 0.898
0.769 0.846 0.692 0.615
Full Full Full Full
Sora-2 Sora-2 Veo-3.1 Veo-3.1
OCR LLM OCR LLM
0.817 0.859 0.659 0.669
0.586 0.671 0.389 0.416
Text recoverability reveals a clear gap between reference renders and generated videos, especially in full-layout scenes. Although the LLM evaluator reaches 0.940 Hard AR on full-layout reference renders (Table 5), the best generated result reaches only 0.671 (Sora-2 with LLM), indicating that current generators often fail to preserve required text exactly in multi-component compositions. Sora-2 consistently outperforms Veo-3.1 on the full-layout track under both evaluators, while the smaller and backend-dependent gap on single-component prompts suggests that isolated text rendering is less diagnostic than text preservation in complete layouts.
Component Duration
Med. AE ↓
Dataset
On single-component GT renders, component-duration MAE is only 0.65 s thanks to a clean contour-tracker lock; animation duration is harder (median AE 0.53 s, mean inflated by a heavy tail) because post-animation contour jitter and settling drift continue to register above the energy threshold. On the full-layout track, component MAE rises to 1.53 s because the YOLO tracker occasionally loses and re-acquires components, fragmenting the detection window; animation MAE is 3.13 s (bias +2.94 s), consistent with the same over-estimation mechanism.
6. Conclusion We have presented a fully automated, dimensiondecomposed evaluation framework for generative design animation that measures motion type, direction, duration, and layout fidelity against ground-truth layout specifications without human judgement. Applied to Sora-2 and Veo-3.1, the framework exposes per-component failure modes such as under-animation, directional inaccuracy, and duration bias, that aggregate perceptual metrics conceal. Future work will extend the motion-correctness axis to character-level evaluation, and will incorporate learned detectors to complement the current rule-based classifier on motion families where geometric features alone are insufficient.
The single-component and full-layout tracks expose opposite ordering between Sora-2 and Veo-3.1 on animation duration. Sora-2 performs better on single-component clips (2.41,s vs. 4.34,s MAE) because its systematically shorter outputs truncate the motion envelope, while on full-layout Veo-3.1 achieves the lowest duration error (2.69,s vs. 3.52,s) as Sora-2’s weaker motions more often fall below the energy threshold. 5.6. Text Recoverability Unlike the motion and layout metrics above, text recoverability is computed directly from sampled video frames and does not require tracker-reliable component detections. We evaluate text content on paired Sora-2 and Veo-3.1 generations using the assignment metric in Section 4, reporting re8
Evaluating Design Video Generation: Metrics for Compositional Fidelity
References
large language models. arXiv preprint arXiv:2404.14801, 2024.
An, A., Sun, S., Huang, D., Cheng, M., Gao, Y., Li, J., Qiao, Y., and Bian, J. Can vision language models assess graphic design aesthetics? A benchmark, evaluation, and dataset perspective. arXiv preprint arXiv:2603.01083, 2026.
Liu, X., Hu, Y., Wu, L., Hu, P., Zou, Y., Xu, J., and Zheng, B. E-VAds: An E-commerce short videos understanding benchmark for MLLMs. arXiv preprint arXiv:2602.08355, 2026.
Chen, S., Lai, J., Gao, J., Ye, T., Chen, H., Shi, H., Shao, S., Lin, Y., Fei, S., Xing, Z., Jin, Y., Luo, J., Wei, X., and Zhu, L. Postercraft: Rethinking highquality aesthetic poster generation in a unified framework. arXiv preprint arXiv:2506.10741, 2025. URL https://arxiv.org/abs/2506.10741.
Liu, Y., Cun, X., Liu, X., Wang, X., Zhang, Y., Chen, H., Liu, Y., Zeng, T., Chan, R., and Shan, Y. Evalcrafter: Benchmarking and evaluating large video generation models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22139– 22149, 2024.
Deganutti, A., Hirsch, E., Zhu, H., Seol, J., and Mehta, P. Graphic-Design-Bench: A comprehensive benchmark for evaluating AI on graphic design tasks. arXiv preprint arXiv:2604.04192, 2026.
Long, X., Tian, K., Xu, P., Jia, G., Li, J., Yang, S., Shao, Y., Zhang, K., Jiang, C., Xu, H., et al. Adsqa: Towards advertisement video understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23396–23407, 2025.
Deka, B., Huang, Z., Franzen, C., Hibschman, J., Afergan, D., Li, Y., Nichols, J., and Kumar, R. Rico: A mobile app dataset for building data-driven design applications. In Proceedings of the 30th annual ACM symposium on user interface software and technology, pp. 845–854, 2017.
OpenAI. Sora: Creating video from text. https:// openai.com/sora, 2024. Yamaguchi, K. Canvasvae: Learning to generate vector graphic documents. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5481– 5489, 2021.
Fu, C., Dai, Y., Luo, Y., Li, L., Ren, S., Zhang, R., Wang, Z., Zhou, C., Shen, Y., Zhang, M., et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 24108–24118, 2025.
Yu, N., Chen, C.-C., Chen, Z., Meng, R., Wu, G., Josel, P., Niebles, J. C., Xiong, C., and Xu, R. Layoutdetr: detection transformer is a good multimodal layout designer. In European Conference on Computer Vision, pp. 169–187. Springer, 2024.
Google DeepMind. Veo: Text-to-video model. https: //deepmind.google/models/veo/, 2024.
Zhang, P., Xu, H., Zhang, J., Zheng, X., Xu, G., Zhang, Y., Liu, J., Yang, Z., Zhou, W., and Jin, L. OCRGenBench: A Comprehensive Benchmark for Evaluating OCR Generative Capabilities. arXiv preprint arXiv:2507.15085, 2025a.
Hirsch, E., Yadav, S., Garg, M., and Mehta, P. LICA: Layered Image Composition Annotations for graphic design research. arXiv preprint arXiv:2603.16098, 2026. Hsu, H. Y., He, X., Peng, Y., Kong, H., and Zhang, Q. Posterlayout: A new benchmark and approach for contentaware visual-textual presentation layout. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6018–6026, 2023.
Zhang, Z., Dou, W., Peng, L., Pan, H., Bagci, U., and Gong, B. VideoAds for fast-paced video understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 21812–21821, 2025b.
Huang, Z., He, Y., Yu, J., Zhang, F., Si, C., Jiang, Y., Zhang, Y., Wu, T., Jin, Q., Chanpaisit, N., et al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21807–21818, 2024.
Zhou, M., Xu, C., Ma, Y., Ge, T., Jiang, Y., and Xu, W. Composition-aware graphic layout GAN for visual-textual presentation designs. arXiv preprint arXiv:2205.00303, 2022.
Khanam, R. and Hussain, M. YOLOv11: An overview of the key architectural enhancements. arXiv preprint arXiv:2410.17725, 2024. Lin, J., Huang, D., Zhao, T., Zhan, D., and Lin, C.-Y. Designprobe: A graphic design benchmark for multimodal 9
Evaluating Design Video Generation: Metrics for Compositional Fidelity
A. Dataset Examples We provide representative samples from both evaluation tracks. Three from the full layout track (Section A.1) and eight from the single-component track (Section A.2). To keep the appendix compact, we provide the verbatim Track 1 prompt for one full-layout example (Example 1, Section A.1.1) and one single-component example (S3, Section A.2); all other prompts in our dataset follow the same schema and differ only in their per-component metadata. A.1. Full-Layout Examples Each full-layout example is presented as a frame-by-frame animation sampled at the entrance windows of its components, with a summary table reporting canvas size, total component count, animated component count, and the set of animation primitives invoked. The verbatim Track 1 prompt is given for Example 1 to illustrate the prompt format; Examples 2 and 3 use the same schema. Static elements (background, decorative vectors) are present in the rendered video and visible in the final frame but, following the convention introduced in Section 3, are excluded from the animated-component enumeration in the prompt. A.1.1. F ULL L AYOUT 01
Figure 3. Animation of Full Layout 01. The four entrance cohorts span a 37.1 s clip: opening cohort at t=0 s (thumbnails 0-5, 0-6 via rise down; text 0-9 via ascend up; text 0-10 via bounce), followed by group 0-11 (pan right) at t=13.13 s, group 0-12 (rise up) at t=24.38 s, and group 0-13 (pan left) at t=34.38 s.
Property
Value
Canvas Total components (recursive) Animated components Animation primitives Directions invoked Entrance regime
1080×1920, background rgb(252,246,243) 18 7 (2 IMAGE, 2 TEXT, 3 GROUP) rise, ascend, bounce, pan down, up, right, left Staggered (t=0, 13.13, 24.38, 34.38 s)
Table 9. Summary of full-layout Example 1.
Track 1 prompt (verbatim). Generate an animated video at 1080x1920 pixels with background color rgb(252, 246, 243). Components (14 total) with the following exact specifications: - Component 0-0-0 (IMAGE): - Position: left=-87.6px, top=0.0px (top-left corner of bounding box) - Size: width=1255.2px, height=1920.0px
10
Evaluating Design Video Generation: Metrics for Compositional Fidelity - Element attributes: description=This image features an abstract texture, dominated by shades of white and very ,→ light grey. It depicts a soft, crumpled surface, possibly paper or fabric, with numerous gentle folds, ,→ creases, and subtle shadows that create an undulating, organic pattern. The style is minimalist and textural, ,→ emphasizing the subtle variations in light and shadow across the uneven surface., has source asset - Animation: none (static component) - The component remains visible from t=0s to t=3.7361380000000004s - Component 0-1 (IMAGE): - Position: left=834.019px, top=111.918px (top-left corner of bounding box) - Size: width=210.939px, height=4px - Element attributes: description=This image appears to be entirely transparent, showing no visible content, ,→ subjects, or colors. It is essentially a blank canvas, with no discernible patterns, shapes, or text present ,→ within its frame., has source asset - Animation: none (static component) - The component remains visible from t=0s to t=3.7361380000000004s - Component 0-2 (IMAGE): - Position: left=834.019px, top=1784.13px (top-left corner of bounding box) - Size: width=210.939px, height=4px - Element attributes: description=The image is entirely transparent, revealing no discernible content or visual ,→ information. There are no main subjects, colors, patterns, text, or any other visible elements to describe ,→ within the frame. It presents as a completely empty and featureless space., has source asset - Animation: none (static component) - The component remains visible from t=0s to t=3.7361380000000004s - Component 0-3 (IMAGE): - Position: left=35.042px, top=111.918px (top-left corner of bounding box) - Size: width=210.939px, height=4px - Element attributes: description=The image is completely black, which indicates a transparent or empty space. ,→ There are no visible subjects, colors, or discernible elements within the frame. No text or specific style ,→ can be identified., has source asset - Animation: none (static component) - The component remains visible from t=0s to t=3.7361380000000004s - Component 0-4 (IMAGE): - Position: left=35.042px, top=1784.13px (top-left corner of bounding box) - Size: width=210.939px, height=4px - Element attributes: description=The image provided is entirely black, appearing transparent. There are no ,→ discernible subjects, colors, patterns, or text visible within it. It conveys no visual information., has ,→ source asset - Animation: none (static component) - The component remains visible from t=0s to t=3.7361380000000004s - Component 0-5 (IMAGE): - Position: left=497.359px, top=269.552px (top-left corner of bounding box) - Size: width=104.189px, height=104.189px - Element attributes: description=The image provided is completely transparent, resembling an empty space or a ,→ blank canvas. There are no visible subjects, colors, text, or any discernible elements whatsoever. It appears ,→ as an absence of visual information, offering no content to describe beyond its transparency., has source ,→ asset - Animation: - Motion type: rise - Direction: down - Animation duration: 0.56s - Speed: 1 - Trigger: onEnter (begins at t=0s) - The component remains visible until t=3.89s - Component 0-6 (IMAGE): - Position: left=649.59px, top=269.552px (top-left corner of bounding box) - Size: width=104.189px, height=104.189px - Element attributes: description=The image is completely black, presenting a solid and uniform absence of light. ,→ No discernible subjects, colors, patterns, or textual elements are visible within this dark frame, appearing ,→ as a blank space., has source asset - Animation: - Motion type: rise - Direction: down - Animation duration: 0.56s - Speed: 1 - Trigger: onEnter (begins at t=0s) - The component remains visible until t=3.89s - Component 0-7 (TEXT): - Content: "CONSECT CO." - Position: left=274.804px, top=76.2693px (top-left corner of bounding box) - Size: width=530.392px, height=95.1365px - Font: Open Sans--400, size 79px, weight 400, style normal - Color: rgb(1, 1, 1) - Line height: 111.0px, letter spacing: 0em, text alignment: center - Animation: none (static component) - The component remains visible from t=0s to t=3.7361380000000004s - Component 0-8 (TEXT):
11
Evaluating Design Video Generation: Metrics for Compositional Fidelity - Content: "@loremipsumsite" - Position: left=194.528px, top=1728.92px (top-left corner of bounding box) - Size: width=690.944px, height=83.0784px - Font: Lato--400, size 69px, weight 400, style normal - Color: rgb(1, 1, 1) - Line height: 97.0px, letter spacing: 0em, text alignment: center - Animation: none (static component) - The component remains visible from t=0s to t=3.7361380000000004s - Component 0-9 (TEXT): - Content: "NEW " - Position: left=85.1248px, top=252.865px (top-left corner of bounding box) - Size: width=464.329px, height=199.123px - Font: Montserrat--400, size 166px, weight 400, style normal - Color: rgb(1, 1, 1) - Line height: 232.4px, letter spacing: 0em, text alignment: left - Animation: - Motion type: ascend - Direction: up - Animation duration: 0.7s - Speed: 1 - Trigger: onEnter (begins at t=0s) - The component remains visible until t=3.89s - Component 0-10 (TEXT): - Content: "GATHERING" - Position: left=85.0673px, top=417.067px (top-left corner of bounding box) - Size: width=913.974px, height=167.742px - Font: Poppins--400, size 139px, weight 400, style normal - Color: rgb(1, 1, 1) - Line height: 195.7px, letter spacing: 0em, text alignment: left - Animation: - Motion type: bounce - Direction: unknown - Animation duration: 1.55s - Speed: unknown - Trigger: onEnter (begins at t=0s) - The component remains visible until t=3.89s - Component 0-11-0 (IMAGE): - Position: left=61.6317px, top=611.289px (top-left corner of bounding box) - Size: width=497.341px, height=493.2339999999999px - Element attributes: description=A young woman with long blonde hair partially covering her face strikes a ,→ dynamic pose, dressed in a stylish white high-neck crop top and matching bottom. The image has a warm, ,→ slightly desaturated filter, giving it a vintage or artistic aesthetic. She is set against a light-colored ,→ wall with prominent dark, parallel vertical lines, adding a modern architectural element to the background., ,→ has source asset - Animation: - Motion type: pan - Direction: right - Animation duration: 0.56s - Speed: 1 - Trigger: onEnter (begins at t=13.126650000000001s) - The component remains visible until t=16.57665s - Component 0-12-0 (IMAGE): - Position: left=61.6317px, top=1147.52px (top-left corner of bounding box) - Size: width=497.341px, height=477.308px - Element attributes: description=A stylish woman is seated against a plain, light gray background, striking a ,→ chic pose with one leg raised, highlighting her footwear. She wears a black and white animal print top paired ,→ with loose-fitting beige trousers and knee-high tan heeled boots. A fluffy, cream-colored jacket is draped ,→ over her arm, completing her sophisticated and fashionable ensemble., has source asset - Animation: - Motion type: rise - Direction: up - Animation duration: 0.56s - Speed: 1 - Trigger: onEnter (begins at t=24.37881s) - The component remains visible until t=27.44881s - Component 0-13-0 (IMAGE): - Position: left=597.686px, top=611.289px (top-left corner of bounding box) - Size: width=420.682px, height=1013.54px - Element attributes: description=A stylish woman stands against a muted orange wall, looking slightly upwards ,→ with a composed expression. She wears a chic coat with large geometric patterns in shades of beige, black, ,→ and dark brown, paired with a thick black scarf and trendy dark-rimmed sunglasses. She holds a structured, ,→ light tan shoulder bag in front of her, completing her elegant and autumnal ensemble., has source asset - Animation: - Motion type: pan - Direction: left - Animation duration: 0.56s - Speed: 1 - Trigger: onEnter (begins at t=34.38075s)
12
Evaluating Design Video Generation: Metrics for Compositional Fidelity - The component remains visible until t=37.12075s The rest of the canvas must remain a solid rgb(252, 246, 243) background with no other elements.
A.1.2. F ULL L AYOUT 02
Figure 4. Animation of Layout 02 sampled at t=0.0 s, t=0.4 s (mid-tumble of group 0-4, partial typewriter reveal), t=1.1 s (tumble settled, pop complete), and t=1.65 s (typewriter string complete). All three animated components share tfrom =0 s but differ in animation duration (1.12, 0.56, 1.65 s respectively).
Property
Value
Canvas Total duration Total components (recursive) Animated components Animation primitives Directions invoked Entrance regime
1080×1920, background rgb(225,229,234) 1.5 s 39 3 (1 GROUP, 1 IMAGE, 1 TEXT) tumble, pop, typewriter auto (rotation), unknown (scale, progressive reveal) Co-occurring at t=0, varying durations
Table 10. Summary of full-layout Example 2.
A.1.3. F ULL L AYOUT 03 This 1080×1920 layout exercises the pipeline at the upper end of animated-component density: every component in the tree is animated (12 of 12, in contrast to FL1’s 7 of 18 and FL3’s 11 of 13). It also uniquely combines three observable motion classes within a single composition: pop (2 images plus a pop-class text), rotate (6 tumbling images plus a tumbling banner pair), and the springy entrance family (burst, bounce). All 12 components share tfrom =0 but differ in animation duration (0.56 s for pop; 1.12 s for tumble; 1.68 s for burst; 2.0 s for bounce), so the entrance window resolves into four nested cohorts that complete in sequence within the first two seconds. The six-image tumble cluster makes this our most informative test for rotate-class detection under multi-component contention, since any within-class confusion or cross-class leakage to the adjacent pop images directly degrades motion-type accuracy on the tracker-reliable subset (cf. Section 5.3). The Track 1 prompt follows the schema given in Example 1. A.2. Single-Component Examples Each single-component sample isolates exactly one animated element on a canvas (white default; black when the component colour is near-white, per Section 3). The eight examples below are organized by the observable motion class assigned by our normalization map φ (Section 4.2). We provide the verbatim Track 1 prompt for sample S3 (the TEXT entry, which 13
Evaluating Design Video Generation: Metrics for Compositional Fidelity
Figure 5. Storyboard of Full Layout 03 sampled at four points across the entrance window: t=0.0 s (initial state), t=0.56 s (pop cohort complete: 0-0, 0-3, 0-11), t=1.12 s (tumble cohort complete: 0-1, 0-2, 0-4–0-8), and t=2.0 s (text burst 0-9 and bounce 0-10 complete, near final frame). The 12 animated components together exhaust the component tree. Property
Value
Canvas Total duration Total components (recursive) Animated components Animation primitives Directions invoked Entrance regime
1080×1920, background rgb(1,129,88) 5.0 s 12 12 (9 IMAGE, 3 TEXT) pop, tumble, burst, bounce auto (rotation), unknown (scale, springy) Parallel at t=0, four duration cohorts (0.56, 1.12, 1.68, 2.0 s)
Table 11. Summary of full-layout Example 4.
exercises the richest field set including font, colour, and line-height attributes); all other single-component prompts follow the same schema. #
Class
Type
LICA anim.
Direction
S1 S2 S3 S4 S5 S6 S7 S8
scrapbook scrapbook scrapbook scrapbook rotate rotate pop pop
IMAGE IMAGE TEXT GROUP GROUP IMAGE IMAGE IMAGE
rise rise rise rise tumble tumble pop pop
up up up up irrelevant irrelevant irrelevant irrelevant
Anim. dur.
Visible until
0.56 s 0.56 s 0.56 s 0.56 s 1.12 s 1.12 s 0.56 s 0.56 s
10.04 s 8.00 s 5.00 s 5.83 s 1.50 s 5.76 s 5.00 s 8.00 s
Table 12. Single-component samples with observable motion class, LICA animation primitive, direction, animation duration, and component visible duration. irrelevant denotes a non-directional motion (tumble and pop have no semantic compass direction).
Track 1 prompt for S3 (verbatim). S3 is a rise up TEXT entry rendering the headline “COLEGIO DE DANZA” on a white canvas. We show the prompt verbatim to illustrate the full text-component field schema (Content, Font, Color, line-height, letter-spacing, text alignment), which is a strict superset of the IMAGE and GROUP field sets used by the other seven samples. Generate a 4.75s animated video at 1080x1920 pixels with background color rgb(255, 255, 255).
14
Evaluating Design Video Generation: Metrics for Compositional Fidelity
Figure 6. Final-frame thumbnails of the eight single-component examples, grouped by observable motion class. Top row: scrapbook class (translational entries via LICA rise). Bottom-left pair: rotate class (tumble). Bottom-right pair: pop class. Component-type diversity (image, text, group) is balanced across classes.
Animated components: 1 (static components may also be present for context but are not listed below). Component 1: TEXT - Content: "COLEGIO DE DANZA" - Font: Lato--400, size 133px, weight 400, style normal - Color: rgb(239, 161, 168) - Line height: 159.0px, letter spacing: 0em, text alignment: center - Position: left=108px, top=671.777px - Size: width=864px, height=317.986px - Motion type: rise - Direction: up - Animation duration: 0.56s - Speed: 1 - Trigger: both (begins at t=0s) - Remains visible until t=5.0s Keep static components visible as context; only the listed animated components should be described for motion.
15
Evaluating Design Video Generation: Metrics for Compositional Fidelity
B. Motion-Type Classifier: Detailed Rules This appendix expands the high-level rule cascade from Table 3 (Section 4.2) into the exact thresholds and per-rule rationale used by the released implementation. The 12 rules in the main paper map onto 15 rules here because two collapse two cases each: the opacity-transient rule splits into a contour-tracker variant and a YOLO-tracker variant, and the late opacity/presence-change rule splits on residual energy. B.1. Feature definitions q For each tracked component we derive five 1-D signals: centroid trajectory (xt , yt ), scale curve st = (wt /w̃)(ht /h̃), OBB rotation θt , opacity proxy ot (detector confidence in YOLO mode; bbox area normalized by its running peak in contour mode), and presence pt ∈ {0, 1}. Scalar features used by the classifier: • dnet : net centroid displacement, normalized by the canvas diagonal. ∆max : maximum per-frame step. • scale range, pop height (largest scale spike above the settled baseline), start extent (∆max over the first quarter of the clip). • Estart/mid/end : composite P motion energy summed over the three thirds of the clip; Epeak = max(Estart , Eend ), ρ = Emid /Epeak , Etot = E. • θtot , θ90 (period-90◦ unwrap, resolving the axis-flip ambiguity of cv2.minAreaRect), θ̇max . • pos zx, sc zx (zero crossings of the detrended centroid and scale curves). • op range, opstart/mid/end , low mid frac (fraction of mid-third frames with opacity proxy below 0.5; YOLO-mode only). • presstart/mid/end (mean of pt over each third). All length features are normalized by the canvas diagonal. We use ε = 3 × 10−3 as the positional noise floor across both backends. B.2. Rule cascade Rationale. Rule 1 guards against degenerate chains. Rules 2–3 separate fade detection by tracker backend: the contour proxy responds linearly to alpha, so a fade produces both an opacity ramp and contour-shape jitter; YOLO confidence drops on any hard-to-localise frame, so the YOLO rule conditions on a sustained low-confidence middle plus the absence of a scale spike or centroid drift. Rules 4–6 implement the LICA on-enter signature – “endpoints hot, middle cold” – and dispatch to pop when the chain is dominated by a scale overshoot, to rotate when the period-90◦ unwrapped angle exceeds a quarter-turn (LICA’s tumble/roll), and to scrapbook otherwise. Rule 7 catches sustained rotations missed by the transient gate. Rule 8 admits two wiggle signatures: oscillating centroid (large-amplitude back-and-forth, gated to reject YOLO bbox jitter) or oscillating rotation. Rule 9 is the scale analogue. Rule 10 requires a pan to satisfy three conditions simultaneously (large net displacement, sustained mid-energy, high directional fraction dnet /Etot ); the directional fraction distinguishes a true pan from any oscillating motion that happens to travel. Rule 11 catches fades whose tracker signature was too weak to fire rules 2–3. Rules 12–13 are the presence-rescue branch for chains where the tracker lost the component during the entry/exit transient itself; we split on residual energy because in the low-energy bucket both true fades and true scrapbook entries are admissible, and fade is the higher-expected-value choice given the partial-credit table (5 fade GT vs. 4 scrapbook+pop GT in this bucket on the GT data). Rule 14 catches slow continuous motion below the pan threshold. B.3. Partial-credit scoring Rotate → scrapbook earns full credit because LICA’s tumble/roll entries combine a translational entry with rotation, so a scrapbook prediction is observationally equivalent to the OBB signal. Scrapbook → pan is asymmetric (only credited in one direction): a scrapbook that travels far enough to satisfy the pan gate is observationally close to a pan, but the reverse miss indicates the classifier inferred a transient on a sustained motion and is counted as a hard miss. On the single-component track LICA releases no motion label for some prompts (carried as unknown); the metric awards 1.0 for predictions of static or unknown on these and 0 otherwise, turning the score into a spurious-motion fraction. 16
Evaluating Design Video Generation: Metrics for Compositional Fidelity Table 13. Detailed motion-type rules with full thresholds. Rules are evaluated top-down; the first match wins.
#
Condition
Label
1
presfull < 0.05
static
2
contour mode; op range > 0.30, dnet < 0.06, and either opmid − min(opstart , opend ) > 0.15 or min(presstart , presend ) < 0.7 with presmid > 0.85 and the other endpoint > 0.85
fade
3
YOLO mode; low mid frac > 0.85, pop height < 0.20, scale range < 0.50, dnet < 0.05, Emid < 0.05
fade
−3
4
Epeak > 5 × 10
5
Epeak > 5 × 10−3 , ρ < 0.4, |θ90 | ≥ 90◦
6
Epeak > 5 × 10
−3
, ρ < 0.4, pop height > 0.10, start extent < 0.005, dnet < 0.03, Emid < 0.05
pop rotate
, ρ < 0.4, otherwise
scrapbook
◦
rotate
8
|θ90 | > 45 , scale range < 0.20, dnet < 0.05 pos zx ≥ 6, ∆max > 0.05 or |θtot | < 30◦ , θ̇max > 100◦ /s ; dnet < 0.03
9
sc zx ≥ 6, scale range > 0.06, ∆max < 0.005
breathe
10
dnet > 0.10, Emid > 0.02, dnet /Etot > 0.4
pan
11
op range > 0.30, scale range < 0.15, ∆max < 2ε, dnet < 2ε
fade
12
min(presstart , presend ) < 0.5, presmid > 0.85, Etot > 0.05
scrapbook
13
min(presstart , presend ) < 0.5, presmid > 0.85, Etot ≤ 0.05
fade
14
Etot > 0.05
scrapbook
15
none of the above
static
7
wiggle
Table 14. Partial-credit table for motion-type predictions. Rows are GT, columns are predicted; blank cells score 0. The table is intentionally asymmetric on rotate/scrapbook and pan/scrapbook. GT ↓ / Pred →
scrap.
pop
fade
rotate
pan
wig./bre.
pop scrapbook fade rotate wiggle breathe
0.5 1.0 0.3 1.0 – –
1.0 0.5 – – – –
– 0.3 1.0 – – –
– 0.5 – 1.0 – –
– 0.3 – – – –
– – – – 0.4 0.4
B.4. Threshold derivation All thresholds were chosen by inspecting GT-render feature distributions, not learned from generated videos. Four are most consequential. (i) ε = 3 × 10−3 : upper edge of contour-tracker centroid jitter on static-class GT renders, padded for YOLO mode. (ii) Epeak > 5 × 10−3 : chosen so LICA’s shortest authored animations (photoRise, ∼ 0.42 s) are admitted while contour jitter on static GT renders is rejected. (iii) ρ < 0.4: transient chains cluster at ρ < 0.2 on GT, sustained chains at ρ > 0.7, and 0.4 leaves a clean margin in both directions. (iv) Etot > 0.05: empirical valley between the static-GT mode (≈ 0.01) and the slow-scrapbook GT mode (≈ 0.2). The rest follow the same recipe: the empirical valley between the targeted class and its nearest confusable class.
17
Evaluating Design Video Generation: Metrics for Compositional Fidelity
C. Motion-Type Classification Results
Figure 7. Motion-type confusion matrices for GT renders, Veo-3.1, and Sora-2 across the single-component track (top), the full-layout track with all components (middle), and the tracker-reliable subset of the full-layout track (bottom). On single-component GT, scrapbook is strongly diagonal (89%), but the classifier collapses most rotate-family samples into scrapbook (14/18); fade and pop show moderate leakage to scrapbook as well. On the full-layout tracks the dominant pattern intensifies: fade, pop, and rotate rows collapse almost entirely to the scrapbook column, because the YOLO-OBB tracker cannot resolve the opacity signal that distinguishes fades nor the brief scale spike that distinguishes pops from a generic translational entry. Rotate-family components predicted as scrapbook receive full credit in the partial-credit scoring (Table 3) and are not penalized. Empty rows (e.g. wiggle in full-layout; breathe, wiggle, static, neon in single-component) indicate motion types absent from the ground-truth distribution at that track.
18
Evaluating Design Video Generation: Metrics for Compositional Fidelity
D. LLM Text Evaluator Prompt For LLM-based text recoverability evaluation, each sampled video frame is encoded as a JPEG image and passed to the model with a strict OCR instruction. The evaluator is asked to transcribe only visible text and return a JSON object containing one string per visual text line. We use the same prompt template across LLM evaluators unless otherwise noted. System prompt. You are a strict OCR transcription engine for one video frame. Transcribe exactly what is visible, without interpretation. Rules: 1) Do not translate, paraphrase, summarize, or grammar-correct. 2) Preserve exact characters: case, digits, punctuation, symbols, accents, apostrophes. 3) Preserve visual reading order: top-to-bottom, then left-to-right. 4) Keep each visual text line as a separate line item. 5) Never reorder words inside a line. 6) Never hallucinate missing words; if a character is unreadable, use ’?’. 7) If no readable text exists, return an empty lines list. Return only a JSON object with this shape: {"lines": ["line1", "line2"]}. No markdown fences. No extra keys. No explanations.
User payload. { "prompt version": "default", "task": "ocr from single video frame", "language hint": "en", "return schema": { "lines": ["string"] } }
The image is provided together with the user payload. The returned lines are parsed as text candidates and then evaluated using the text recoverability metric in Section 4.
E. Text Recoverability Backend Validation We compare OCR and LLM-based evaluators for the text recoverability metric. Prior work on aesthetic poster generation notes that traditional OCR can be unreliable for artistic typography and stylized design text, and uses VLM-based recognition to handle text regions in poster layouts (Chen et al., 2025). Motivated by this observation, we evaluate both conventional OCR backends and LLM-based transcription backends. In all cases, the backend is used only to extract text candidates from sampled frames; the final score is computed with the same CER-based assignment metric defined in Section 4. Full-layout runs use the video set with text-containing samples. All backends are evaluated on the same frames sampled at 2 fps, following the frame-sampling ablation in Appendix F. The single-component rows provide a controlled reference setting. For single-component LLM evaluation, we use Claude Opus 4.7 only, selected based on the full-layout backend comparison.
F. Frame Sampling Rate Ablation We evaluate the effect of frame sampling rate on text recoverability using the full-layout validation set. We vary the sampling rate from 0.5 to 8 fps while keeping the evaluator and assignment metric fixed. This ablation is used to justify the 2 fps default used in the main experiments.
Impact Statement The contribution of this paper is measurement rather than a new capability or generative artifact. By quantifying where frontier generative and vision-language models succeed and fail on design animation tasks, it is intended to help researchers, designers, and tool-builders make better-informed decisions about when these models are, and are not, ready to act as 19
Evaluating Design Video Generation: Metrics for Compositional Fidelity Best AR ↑
Hard AR ↑
EasyOCR PP-OCRv5 Tesseract Claude Opus 4.7
0.919 0.769 0.695 1.000
0.769 0.769 0.462 1.000
PP-OCRv5 EasyOCR Tesseract GPT-5.5 Gemini 3.1 Pro Claude Opus 4.7
0.862 0.903 0.565 0.951 0.950 0.960
0.659 0.584 0.314 0.920 0.922 0.940
Dataset
Backend
Single-component Single-component Single-component Single-component Full-layout Full-layout Full-layout Full-layout Full-layout Full-layout
Table 15. Backend validation for text recoverability. Best AR is the average per-component best-over-time recognition score, and Hard AR is the fraction of required text components exactly recovered. Bold indicates the best backend per dataset.
PP-OCRv5
Claude Opus 4.7
Score
Best AR
Hard AR
0.95
0.9
0.90
0.8
0.85
0.7
0.80
0.6 0.5
1
2
4
8
0.5
Sampling rate (fps)
1
2
4
8
Sampling rate (fps)
Figure 8. Effect of frame sampling rate on text recoverability in full-layout validation scenes. Both evaluators improve rapidly up to 2 fps and show diminishing returns beyond that point, supporting the 2 fps default used in the main experiments.
collaborators. Reliable design animation competence is also a prerequisite for accessibility-sensitive deployments, so exposing current gaps supports responsible use in professional workflows. We view this as supportive of human-AI co-creativity in its augmentation sense: surfacing current limitations so that design tools are deployed as aids to designers rather than as replacements. Beyond the failure modes reported in our experiments, we do not foresee specific additional societal consequences that warrant highlighting here.
20