arXiv:2604.12659v1 [cs.LG] 14 Apr 2026
Do VLMs Truly "Read" Candlesticks? A Multi-Scale Benchmark for Visual Stock Price Forecasting Kaiqi Hu
Linda Xiao
Shiyue Xu
University of Leeds Leeds, UK [email protected]
Sun Yat-sen University Zhuhai, China [email protected]
Sun Yat-sen University Zhuhai, China [email protected]
Ziyi Tang
Mingwen Liu
Sun Yat-sen University GuangZhou, China [email protected]
Likelihood Lab Guangzhou, China [email protected]
Abstract
Keywords
Vision–language models (VLMs) are increasingly applied to visual stock price forecasting, yet existing benchmarks inadequately evaluate their understanding of stock price in candlestick charts. First, prior studies fail to isolate VLMs’ comprehension of visual stock price inputs, and it remains unclear whether visual inputs genuinely improve predictive performance and whether VLMs truly comprehend candlestick patterns. Further, most existing datasets and evaluation setups are designed around single-period or tabular inputs. However, human analysts strongly rely on multi-scale candlestick charts, where longer-term horizons capture trend direction and shorter-term horizons provide cues for inflection points, making it difficult to systematically assess VLMs’ ability to integrate short-term and long-term visual market dynamics. To bridge this gap, we construct a multi-scale candlestick chart dataset and a standardized evaluation framework to assess VLMs’ ability to utilize multi-scale visual market signals. Evaluation combines confusionmatrix-based diagnostics with information coefficient (IC) time series metrics and includes XGBoost as a feature-based temporal baseline. Using this dataset, we benchmark representative VLMs and analyze their ability to leverage multi-scale stock price data. Experimental results show that most VLMs perform well only under persistent uptrend or downtrend conditions, while exhibiting weak predictive capability in more common market scenarios. We also identify significant prediction biases and limited sensitivity to explicitly specified forecast horizons in prompts, indicating inherent limitations in precise temporal reasoning.
Vision-Language Model; Visual Stock Price Forecasting; Multiperiod Candlestick Chart
CCS Concepts • Applied computing → Forecasting.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. Conference’17, Washington, DC, USA © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-x-xxxx-xxxx-x/YYYY/MM https://doi.org/10.1145/nnnnnnn.nnnnnnn
ACM Reference Format: Kaiqi Hu, Linda Xiao, Shiyue Xu, Ziyi Tang, and Mingwen Liu. 2026. Do VLMs Truly "Read" Candlesticks? A Multi-Scale Benchmark for Visual Stock Price Forecasting. In . ACM, New York, NY, USA, 10 pages. https: //doi.org/10.1145/nnnnnnn.nnnnnnn
1
Introduction
With the rapid advancement of artificial intelligence, Vision-Language Models (VLMs) have demonstrated significant potential in financial applications, particularly in stock price forecasting [10]. Candlestick charts, as a core analysis tool, visually encapsulate rich market dynamics and price patterns [2][12][13][16], making chartbased prediction a key VLMs’ application in finance. However, existing evaluation benchmarks exhibit critical limitations in assessing VLMs’ genuine comprehension of stock price information. Current research faces a fundamental methodological challenge: dataset complexity hinders capability attribution analysis. Mainstream studies employ highly heterogeneous multimodal datasets combining visual and textual content [8][23]. While mimicking real trading environments, this introduces critical flaws: unclear capability attribution when achieving strong performance, we cannot determine whether models truly “understand” chart patterns or primarily rely on textual signals [10]. Most studies lack rigorous ablation experiments to isolate modality contributions, preventing quantification of visual inputs’ marginal value. Furthermore, existing studies neglect the multi-time-scale analysis methodology central to technical analysis [10], thereby limiting the comprehensive assessment of VLMs’ capabilities. Professional traders systematically integrate multiple timeframes [11]: long-term frames reveal macro trends forming strategic backdrops [6][11], while short-term frames provide precise trading signals, breakouts, momentum shifts, and inflection points [18][25]. This hierarchical validation represents a critical cognitive advantage validated through decades of market practice. To address these limitations, this study proposes a systematic solution minimizing information interference while preserving essential analytical elements. Specifically, our contributions are:
Conference’17, July 2017, Washington, DC, USA
Hu et al.
Figure 1: Pipeline for constructing visual inputs and evaluation. • We construct dataset samples that incorporate both daily and weekly candlestick charts with forecast targets, forcing VLMs to extract information purely from visual patterns and eliminating multimodal confounding bias. • We employ a multidimensional evaluation framework that goes beyond traditional information coefficient (IC) metrics by introducing dual-dimensional confusion matrix analysis, which quantifies VLMs’ behavioral characteristics across different market states and stock attributes. • We conduct fine-grained analyses examining predictive biases under varying conditions, including false positive and false negative distributions as well as trend capture abilities. This framework provides methodological support for understanding multimodal prediction mechanisms and offers valuable insights for trading strategy optimization.
2 Related Work 2.1 Traditional Stock Forecasting Methods Stock price forecasting has long been central to financial research. Traditional statistical models, such as ARIMA, demonstrate potential for short-term predictions [9] but struggle with complex nonlinear relationships. Machine learning approaches enhance performance through feature engineering: Wu et al. showed XGBoost achieves high accuracy [22], while Sheng et al.’s multimodal LightGBM reached 75.85% balanced accuracy [19], though susceptible to overfitting. Deep learning opened new possibilities, with Yu et al. demonstrating ARIMA’s trend forecasting utility [24] and Nichani et al. validating hybrid ARIMA-LSTM superiority [15]. Image-based approaches treat candlestick charts as visual inputs: Sim et al. found CNNs enhance learning through convolution and pooling [21], Chootong et al. integrated neural networks with technical indicators and chart patterns for trading signals [2], and Kusuma et al. demonstrated CNNs uncover hidden patterns within candlestick images, achieving peak performance across multiple metrics [12].However, these methods require data or structured image processing and cannot extract effective information directly from candlestick charts for prediction like human traders do.
2.2
Image Applications and Challenges
In recent years, financial images have been widely applied to forecasting tasks. However, existing research predominantly employs highly complex multimodal datasets, hindering capability attribution analysis. Typical studies combine images with multiple information sources: Karadaş et al. incorporated tweet metrics (likes, retweets, comments, follower counts) into feature extraction [8]; Xu et al. provided models with pure time series, news sentiment, image trends, and fundamental tables [23]; Huang et al. fed FinLLaMA with financial domain knowledge (papers, reports) alongside images, achieving 55.73% cumulative return and 2.45 Sharpe ratio on TSLA [5]; Prado et al. visually encoded candlestick charts and identified 16 classic patterns through mathematical rules, finding statistical significance in some patterns [17]. While achieving high accuracy, these studies suffer from a fundamental flaw: the inability to determine whether performance stems from visual understanding or textual information. Critically, most lack ablation experiments isolating modality contributions, leaving visual comprehension capabilities a “black box”.
2.3
Multi-Time-Scale Analysis Methods
Multi-time-scale analysis constitutes technical analysis’s core methodology. Classical Dow Theory distinguishes primary trends, secondary trends, and short-term fluctuations [4]; Elliott Wave Theory depicts nested market structures [1]. Modern systems adopt multiperiod confirmation mechanisms like the “three-screen trading system” requiring validation across timeframes [7]. In deep learning, multi-scale modeling has emerged: Xu et al. provided minute, daily, and quarterly resolutions to capture short-term fluctuations, medium-term trends, and long-term fundamentals [23], though primarily targeting tabular data rather than visual candlestick charts. Existing research lacks a systematic evaluation of VLMs’ visual integration of multi-period charts. Shu et al. constructed timeseries datasets by stitching ten consecutive years of financial charts [20], yet these experiments failed to analyze whether models truly learned hierarchical reasoning, grasping long-term trends before
Do VLMs Truly "Read" Candlesticks? A Multi-Scale Benchmark for Visual Stock Price Forecasting
capturing short-term opportunities, or merely identified stronger statistical correlations in certain periods.
3
Dataset
Table 1: Stock Market Regime Distribution (Bull > 70%, Bear < 30%)
3.1
as the common numerical data source for subsequent analysis and evaluation.
3.2
This section describes our process for constructing a multi-scale candlestick chart dataset. The raw market data are obtained from two publicly accessible financial data sources. Historical data for the Chinese market are collected from TuShare, which provides structured daily OHLCV records for A-share stocks in China. Using this interface, we collect data for constituent stocks of HS300 from January 2015 to July 2025, together with trading codes and dates. In addition, historical daily OHLCV records for U.S. stocks are obtained from Yahoo Finance, covering stocks included in the S&P500 index over the same period. The multi-period candlestick chart dataset constructed from these historical OHLCV records encompasses diverse market conditions, including bull markets, bear markets, and periods of extreme price volatility, and is used as the visual modality in VLM-related experiments. In contrast, for numerical experiments such as XGBoost, the same historical stock data are processed through a numerical modality construction pipeline to form the numerical modality. All data access is conducted strictly for academic and noncommercial research purposes and in accordance with the terms of service of the respective data providers. Access to Chinese market data via TuShare is performed using an officially registered account and authenticated API token, while U.S. market data is retrieved from Yahoo Finance through its publicly accessible data interface without authenticated API credentials. The resulting dataset contains only publicly available, market-level financial information and does not involve any personal, private, or sensitive data.
Market Regime
Days
Percentage (%)
Bull Market Bear Market Sideways Market
558 310 1849
20.5 11.4 68.1
Total
2717
100.0
Conference’17, July 2017, Washington, DC, USA
Visual Modality Data Processing
To construct the visual modality of our dataset, we generate candlestick chart images at multiple time scales from the raw OHLCV time series using a unified and deterministic pipeline. Each image corresponds to a specific stock, temporal frequency, and cutoff date. The generation process for a single candlestick chart proceeds as follows. (1)The configuration parameters and the corresponding stock-level OHLCV data are reconstructed from the input arguments, forming a dedicated dataframe for each stock. (2)Multi-frequency candlestick data are computed. For the daily frequency, the OHLCV records are sorted by trade date, and moving averages (MA5: black; MA20: blue; MA90: purple) are computed directly from the daily closing prices. For lower-frequency views (weekly), the daily OHLCV series is resampled using standard aggregation rules to obtain open, high, low, close, and volume values for each period, followed by the computation of the same set of moving averages. (3)The candlestick data corresponding to the specified temporal frequency are selected for visualization. (4)The time series is truncated according to a predefined cutoff date. For each chart, only historical candlesticks strictly before the specified cutoff date are included, ensuring that no future information is introduced. (5)To ensure visual consistency and control input complexity, the number of displayed candlesticks is capped at a maximum of 50 bars. When the available history exceeds this limit, only the most recent 50 candlesticks are retained. (6) The candlestick chart is rendered and saved as a PNG image. Moving averages are overlaid using additional plot layers, and trading volume is displayed in a separate lower panel. Brief Summary: Each chart comprises a candlestick price chart with overlaid moving averages and a corresponding volume subplot. The header includes the stock symbol, time frame, and cutoff date, and each chart visualizes up to 50 candlesticks. Figure2 and 3 display the candlestick charts for the same stock at daily and weekly resolutions, respectively. The daily chart shows an upward trend, while the weekly chart indicates a downward trend. The actual 30-day future return for this stock is negative, illustrating how market patterns vary across time scales.
Raw Market Data (OHLCV)
At the raw market data stage, we apply minimal preprocessing to the original OHLCV(open, high, low, close, and volume) time series in order to preserve the underlying market dynamics and avoid introducing model- or modality-specific biases at an early stage. Modality-dependent processing steps for visual and numerical representations are described separately in subsequent subsections. Based on the closing price series, we define the future return as the target variable of interest, measuring the relative price change over a fixed forward horizon. Unless otherwise stated, we focus on a 30-day forward return (𝐻 = 30). The precise mathematical definition is provided in Section 4.1.3. All raw OHLCV records, together with the corresponding future return values, are aggregated into a unified CSV file, which serves
Figure 2: daily candlestick example
Conference’17, July 2017, Washington, DC, USA
Hu et al.
3.4
Dataset Statistic
The historical stock data spans from 2015 to 2025, during which candlestick charts are constructed over the same period. We select 300 constituent stocks from the HS300 index and 500 stocks from the S&P500 index, and generate daily and weekly candlestick charts for 32 cutoff dates. Table 2 summarizes the dataset statistics. Table 2: Total number of candlestick samples and raw OHLCV records Candlestick Samples
Figure 3: weekly candlestick example
HS300
S&P500
Total
HS300
S&P500
Total
Daily Weekly
34878 34866
61892 61888
96770 96754
714920 –
1290359 –
2005279 –
Total
69744
123780
193524
714920
1290359
2005279
3.5 After candlestick chart generation, all visual outputs are organized using a hierarchical directory structure to support systematic visual data processing. The top-level directory is indexed by the cutoff date associated with each observation, followed by the stock code at the second level. Within each stock-specific directory, candlestick charts at different temporal frequencies are stored separately. This organization enables the VLMs to access visual representations at multiple time scales for the same asset and cutoff date, facilitating the analysis of price dynamics across multiple time scales in a unified and consistent manner. To evaluate 30-day return forecasts, candlestick charts are sampled at 30-day intervals as a practical and consistent design choice. This sampling strategy simplifies the temporal relationship between visual inputs and target outcomes, improving the clarity and interpretability of the evaluation protocol. The 30-day interval is adopted for convenience rather than necessity; alternative sampling strategies are feasible, provided that the sampling interval is aligned with the definition of the future return. In contrast, shorter sampling intervals would substantially increase the number of visual inputs and associated API inference costs, while offering limited additional benefit for medium-term forecasting. Moreover, charts generated at short intervals tend to be highly similar, differing primarily in recent local price movements, which introduces redundancy and strong temporal correlation among samples. Overall, this visual data organization and sampling strategy establishes a structured, scalable, and interpretable visual input pipeline, balancing temporal coverage, dataset scale, and sample diversity, and providing a stable foundation for subsequent VLMbased evaluation.
3.3
Numerical Time-series Data processing
In addition to the visual modality, numerical time-series inputs are constructed to support baseline models and ensure a fair comparison across modalities. All numerical features are strictly aligned with the same cutoff dates and future return labels as the visual data, with implementation details deferred to Appendix A.
Raw OHLCV Records (Daily)
Frequency
Comparing to Existing Datasets
Table 3 compares our dataset with representative datasets used for time-series or finance forecasting. Existing datasets predominantly focus on textual and numerical signals. For example, FNSPID [3] integrates news text with high-frequency price series, while TimeMMD [14] provides a general-purpose time-series benchmark with image inputs that are not derived from chart data. Neither includes candlestick charts nor supports reasoning over technical chart structures. FinMultiTime [23] is the only prior dataset that explicitly incorporates candlestick chart images. However, it treats candlestick charts as auxiliary inputs by compressing long-horizon price movements into coarse trend labels, and therefore does not evaluate models’ native visual understanding of chart structures or cross-scale consistency. In contrast, our dataset treats candlestick charts as first-class visual inputs and explicitly supports multi-scale candlestick representations. Rather than optimizing for price prediction, our benchmark is designed to evaluate VLMs’ ability to reason over financial charts, including recognizing scale-dependent patterns and maintaining consistency across temporal resolutions. To the best of our knowledge, our dataset is the first to target multi-scale visual reasoning over candlestick charts as a primary evaluation objective.
4 Benchmark Tasks 4.1 Task Definition This study defines the stock trend prediction task as a return regression prediction problem based on visual analysis of candlestick charts. 4.1.1 Input. Each sample contains two candlestick charts for the same stock on the same date: the Daily Chart and the Weekly Chart. This design, with two time scales based on technical analyses across multiple time scales, effectively distinguishes between “pullbacks within an ongoing trend” and “trend reversals” while testing VLMs’ ability to integrate visual and textual information. 4.1.2 Output. Continuous value prediction 𝑟ˆ ∈ [−0.5, 1.0] directly corresponds to the expected future return. Reasons for choosing
Do VLMs Truly "Read" Candlesticks? A Multi-Scale Benchmark for Visual Stock Price Forecasting
Conference’17, July 2017, Washington, DC, USA
Table 3: Comparison with existing datasets. Dataset
Modalities
FNSPID (Nasdaq) Text + Price FinMultiTime Text + Table + Image + TS Time-MMD Text + Image + TS Ours Image
Candlestick
Multi-scale
Visual Reasoning
Time Span
No Yes No Yes
No No No Yes
No No No Yes
2009–2023 2009–2025 1989–2024 2015–2025
regression over classification include: preserving complete granular information for calculating financial metrics like IC; compatibility with classification evaluation (via positive or negative signs); and closer alignment with real investment decision scenarios. The output range [-0.5, 1.0] covers over 99% of actual cases and permits 𝑟ˆ = 0.000 to express “unclear trend”, avoiding forced predictions. 4.1.3
Prediction Goal. The 30-day forward yield is calculated as 𝑟 30 =
𝑃𝑡 +30 − 𝑃𝑡 , 𝑃𝑡
(1)
where 𝑃𝑡 denotes the closing price on the current date, and 𝑃𝑡 +30 represents the closing price 30 trading days later. The 30-day window resides within the “sweet spot” of technical analysis validity: it provides sufficient trend continuity while remaining within the applicable range of technical patterns.
4.2
Prompt Engineering
To translate the abstract regression prediction task defined in Section 4.1 into visual analysis instructions executable by VLMs. We designed a structured prompt for models so that they can complete the task in steps. The whole prompt is in the Appendix C. 4.2.1 Role Definition and Task Specification. The prompt begins by explicitly defining the model’s role as “Stock trend analyst with strong discriminative ability” to evaluate “price fluctuation magnitude” rather than simple binary judgments, and then emphasizes “STRICTLY NO bias”. We use two-period candlestick charts as inputs, where green and red candles indicate upward and downward price movements, respectively. The 30-day future return is defined according to Eq. (1). 4.2.2 Multidimensional Analysis Framework. This study employs a multidimensional technical analysis framework to predict 30day stock returns. First, candlestick patterns are decomposed to identify body and wick characteristics of red/green candle clusters and capture reversal signals. Second, the 5-day, 20-day, and 90day moving average system analyzes price positions, crossovers, and support/resistance effects. Third, volume analysis identifies key levels, abnormal spikes, and trend changes. Fourth, inflection points detect trend reversals and momentum shifts, with crossvalidation against prior indicators. Finally, linking daily and weekly signals allows us to examine the consistency between short-term fluctuations and long-term trends. By integrating price patterns, moving average signals, volume information, and multiple time scales, the framework predicts future returns. 4.2.3 Output Constraints and Few-Shot Learning. In this experiment, we enforce a strict output format to standardize model predictions. The model is required to output a single numerical score
Market
Primary Task
US Frequency forecasting US + China Multimodal forecasting Multi-domain Time-series analysis US + China VLMs testing
in the range [-0.5, 1.0], rounded to three decimal places and enclosed within a <score> tag, without any additional text. Positive values correspond to bullish predictions, negative values to bearish predictions, and zero indicates uncertainty in trend direction. To avoid systematic directional bias, the model is instructed to adopt no default preference and to base its predictions solely on the observed technical signals. Each stock date pair is assigned a unique score. The provided few-shot examples cover moderate gains, sharp increases, declines, uncertain trends, and extreme cases, with annotations illustrating the correspondence between score magnitude and future returns to guide scale calibration.
4.3
Evaluation Metrics
4.3.1 Confusion Matrices. In stock price prediction, TP indicates a predicted rise that actually occurs, TN indicates a predicted fall that actually occurs, FP indicates a predicted rise that does not occur, and FN indicates a predicted fall that does not occur. The TP Rate reflects the ability to capture upward opportunities, while the FP Rate reflects the proportion of false positives. Accuracy measures general prediction quality; Recall measures the ability to capture upward trends; Precision measures the reliability of predicting increases; Specificity measures the ability to avoid predicting declines; F1 combines prediction reliability and capture capability. Specific calculation formulas are detailed in Section 5.1.2. 4.3.2 IC (Information Coefficient). The Information Coefficient (IC) measures the linear correlation between predicted values and actual returns. The mean IC reflects the average level of predictive capability, while the median IC indicates predictive stability. The Information Coefficient Ratio (ICIR), calculated as the mean IC divided by the standard deviation of IC, measures risk-adjusted predictive capability. The proportion of statistically significant ICs indicates the frequency of predictive effectiveness. Rank IC employs Spearman’s rank correlation instead of linear correlation, offering greater robustness against outliers. Rank ICIR, mean Rank IC, and statistical significance of Rank IC are similar to those of IC. This metric is suitable for evaluating relative strength judgment capabilities.
4.4
Peeping Experiment
This experiment selected two models from the same product line with different release dates: GPT4o-2024-05-13 and GPT4o-2024-1120. Within the same prediction window (May 14, 2024, to November 19, 2024), both models forecast 30-day future returns based on daily and weekly candlestick charts up to each prediction date, with performance evaluated against actual outcomes. The experimental results show that the IC and several related metrics obtained by the two models are relatively close. No typical peeping pattern, where
Conference’17, July 2017, Washington, DC, USA
Hu et al.
Table 4: Confusion matrix and performance metrics for different models. Model
Market
TP
TN
FP
FN Total FP Rate (%) FN Rate (%) Accuracy Precision Recall Specificity
623 1834 580 1170
648 1670 471 1051
4775 3272
13.57 14.39
34.97 32.12
51.46% 53.48%
49.02% 27.17% 55.19% 35.56%
73.89% 0.3496 71.30% 0.4325
Gemini2.5-flash (nothinking) HS300 S&P 500 Gemini2.5-pro (thinking) HS300 S&P 500
1101 2754 1031 2575
1139 2324 1009 2169
1192 1794 1262 1925
4775 8468 4775 8408
23.85 27.44 21.13 25.80
24.96 21.19 26.43 22.89
51.18% 51.37% 52.44% 51.31%
49.15% 54.23% 50.54% 54.28%
54.11% 40.71% 59.35% 44.50%
GPT4o GPT5mini Qwen
HS300 HS300 HS300
978 1381 1101 1315 1206 1161 1321 1087 467 782 562 575
4775 4775 2386
23.06 27.66 23.55
27.54 22.76 24.10
49.40% 49.57% 52.35%
47.04% 42.65% 47.72% 52.59% 45.38% 44.82%
55.64% 0.4474 46.78% 0.5004 58.18% 0.4510
XGBoost
HS300
775 1846
5152
16.30
32.82
50.87%
47.99% 31.43%
68.73% 0.3798
1343 1596 1473 1739
840 1691
the later-released model significantly outperforms the earlier version of the model, was observed. Therefore, it is concluded that the selected VLMs did not explicitly utilize actual stock data after its release date to enhance its forecasting performance within this interval, providing preliminary empirical evidence for the interpretable reliability of the prediction results.
5
Experimental Results
0.4858 0.5722 0.4759 0.5571
Accuracy =
𝑇𝑃 +𝑇𝑁 𝑇𝑃 + 𝑇 𝑁 + 𝐹𝑃 + 𝐹𝑁
(2)
Precision =
𝑇𝑃 𝑇𝑃 + 𝐹𝑃
(3)
Recall =
𝑇𝑃 𝑇𝑃 + 𝐹𝑁
(4)
Specificity =
𝑇𝑁 𝑇 𝑁 + 𝐹𝑃
(5)
2 × Precision × Recall 2𝑇 𝑃 = Precision + Recall 2𝑇 𝑃 + 𝐹 𝑃 + 𝐹 𝑁
(6)
Classification Performance Analysis
5.1.1 Confusion Matrix. The left part of Table 4 presents the confusion matrix statistics for all models. The confusion matrix heatmap of the best-performing model, Claude-haiku-4-5, is presented as follows in Figure 4.
48.02% 60.55% 44.96% 57.22%
presents the performance of various models on these metrics.
This study employs a test window spanning from 2023.01.01 to 2025.01.01. Most models are evaluated using the HS300 as the stock universe. Models that have additionally been tested on the S&P 500 index are marked with parentheses to distinguish their expanded validation scope across different market environments.
5.1
F1
Claude-haiku-4-5 HS300 Claude-sonnet-4-5 (thinking) HS300
𝐹1 =
These findings reveal that different models naturally align with distinct investment philosophies. Claude-Haiku’s high precision suits conservative strategies, while GPT5mini’s high recall favors aggressive approaches, indicating that model selection should strategically match risk preferences rather than pursuing universal optimization.
5.2
Prediction Bias Analysis
Model calibration evaluates the degree of alignment between the predicted distribution and the true distribution. We employ distribution bias as the calibration metric, defined as:
Bias =
𝑁 pred_up 𝑁 pred_down 𝑁 true_up 𝑁 true_down − − − 𝑁 𝑁 𝑁 𝑁
Figure 4: Heatmap of Claude-Haiku-4-5
5.1.2 Metrics based on Confusion Matrix. Based on the confusion matrix, accuracy, precision, recall, specificity, and F1 score can be computed as defined in Section 4.3.1. The right part of Table 4
where N denotes the total sample size. A positive bias indicates the model systematically overestimates the probability of upward movements (optimistic), while a negative bias indicates underestimation (pessimistic). Figure 5 presents the bias values and calibration grade assessments for each model.
Do VLMs Truly "Read" Candlesticks? A Multi-Scale Benchmark for Visual Stock Price Forecasting
Conference’17, July 2017, Washington, DC, USA
Figure 5: Bias of each model. Figure 7: Rank IC Average and Rank IC Median
Well-calibrated ( | bias | ≤ 0.05), Slightly biased (0.05 < | bias | ≤ 0.15), Moderately biased (0.15 < | bias | ≤ 0.40), Strongly biased ( | bias | > 0.40).
Model bias evaluation reveals significant divergence across architectures, with traditional Machine Learning and VLMs exhibiting distinct directional preferences. Underlying causes warrant further investigation. This systematic heterogeneity enables constructing ensemble systems: integrating models with complementary biases may achieve bias cancellation and enhanced robustness, offering a promising methodological avenue for financial forecasting by leveraging diverse predictive strengths.
5.3
Information Coefficient Analysis
The Information Coefficient (IC) measures the correlation between predicted and actual returns, serving as a core metric for evaluating predictive capability. Unlike accuracy, which focuses on directional judgment, IC reflects a model’s ability to forecast the magnitude of returns, holding significant importance for portfolio optimization and risk management. Figure 6, Figure 7 and Table 5 present the IC and Rank IC series metrics for different models.
5.4
Performance Under Extreme Conditions
5.4.1 Bull and Bear Market Performance. In this article, market trend is defined as follows: a trading day is deemed as a bull market if the ratio of rising stocks > 70% on the day, and as a bear market if the ratio of falling stocks > 70%.Figure 8 shows each model’s performance on these days. In normal markets, model performance has no significant differences(50.9%-52.6%). In extreme markets, divergence emerges. All models demonstrated significantly higher accuracy in bear markets, indicating that models are better at identifying downside risks than capturing upside opportunities. Qwen achieves zero performance because no bull-market instances appear in its test period. This asymmetric predictive capability provides empirical evidence for investors to select models tailored to market conditions.
Figure 8: Bull or Bear Market Heatmap
Figure 6: IC Average and IC Median
5.4.2 Extreme Stock Cases Analysis. In this paper, extreme stocks are defined as follows: an individual stock is continuous rise: if the ratio of the certain stock’s rising days > 70% during the test period, and is continuous fall if the ratio of falling days > 70%. The predictive behavior of various models toward these stocks is examined, with results presented in the Figure 9. Different models exhibit varying accuracy rates for identifying continuously declining stocks
Conference’17, July 2017, Washington, DC, USA
Hu et al.
Table 5: IC Metrics and Rank IC Metrics R. denotes rank; Sig. denotes significant ratio; Med. denotes median. Unless otherwise stated, all reported results are obtained from experiments on the HS300 dataset.
IC (Complete Ranking Table) Model
Mean R.
Med. R.
ICIR R.
Rank IC (Complete Ranking Table) Sig. R.
Mean R.
Med. R.
ICIR R.
Sig. R.
Claude-haiku-4-5 −0.01030 9 0.03596 7 −0.04198 9 0.813 1 −0.00918 8 0.01887 6 −0.03474 8 0.750 1 Claude-sonnet-4-5(thinking) 0.04706 2 0.05450 3 0.23602 5 0.545 6 0.06198 2 0.08425 1 0.26291 4 0.636 5 Gemini2.5-flash(nothinking) 0.02576 7 0.05374 4 0.09820 7 0.750 3 0.02193 7 0.02210 5 0.07922 7 0.688 3 Gemini2.5-pro(thinking) 0.03047 5 0.07221 2 0.13092 6 0.625 5 0.03603 4 0.06528 2 0.13900 6 0.625 6 −0.01759 10 0.04426 5 −0.06865 10 0.750 3 −0.02024 9 0.00526 9 −0.07286 9 0.688 3 GPT4o GPT5mini −0.00864 8 0.00473 9 −0.03555 8 0.750 3 −0.02805 10 −0.05785 10 −0.10490 10 0.688 3 Qwen 0.04106 3 0.03646 6 0.35515 2 0.500 7 0.02844 6 0.00946 8 0.19820 5 0.375 8 Gemini2.5-flash(nothinking,S&P500) 0.03436 4 −0.01019 10 0.31250 3 0.412 9 0.03460 5 0.01302 7 0.33200 3 0.294 10 Gemini2.5-pro(thinking,S&P500) 0.08990 1 0.07628 1 0.79370 1 0.471 8 0.07676 1 0.06078 3 0.69660 1 0.471 7 XGBoost(30d) 0.03045 6 0.02951 8 0.31212 4 0.250 10 0.04023 3 0.02963 4 0.43373 2 0.313 9
and rising stocks, indicating their distinct capabilities in recognizing different stock behaviors. Claude-Haiku demonstrates optimal performance in identifying declining stocks, while GPT5mini excels at capturing upward trends. Notably, XGBoost exhibits the lowest prediction accuracy for rising stocks (43.8%), indicating its insensitivity to positive momentum signals. This heterogeneity provides a foundation for constructing model portfolio strategies.
Figure 9: Continuous Rise or Fall Heat-map
Figure 10
5.5
Time Sensitivity Analysis
Although models were instructed to predict 30-day returns, IC analysis comparing predictions against both 5-day and 30-day actual returns reveals significantly higher correlation with short-term outcomes. This indicates that current VLMs predominantly capture short-term market signals rather than long-horizon trends, exposing fundamental limitations in temporal understanding and task execution for extended prediction horizons. The results are as follows in Figure 10. IC metric comparison reveals that nearly all models exhibit significantly stronger correlation with 5-day returns than 30-day returns. For instance, Claude-Sonnet-4-5 achieves 5-day IC of 0.047 (ICIR: 0.236) versus 30-day IC of 0.042 (ICIR: 0.258). This discrepancy may stem from VLMs’ proficiency in capturing short-term technical patterns abundant in training data, but struggling with long-term predictions requiring deeper fundamental reasoning and managing cumulative uncertainty. These findings suggest VLMs function better as auxiliary tools for short-term trading signals rather than primary support for long-term investment decisions.
6
Discussion
Multi-scale candlestick analysis derives value from synchronizing structural fundamentals with dynamic price action across market cycles. However, its inherent latency proves problematic: Technical patterns confirm cycles retrospectively, missing early warning signals of policy shifts or economic inflections, rendering forecasts reactive rather than predictive. Despite this latency, candlestick charts consistently outperform equivalent tabular data, evidencing VLMs’ higher performance ceilings. Empirically, VLMs exhibit greater IC/Rank IC significance and systematically elevated median IC (despite comparable means) versus XGBoost, indicating superior predictive stability with reduced tail-risk failures and substantial optimization headroom. We attribute this to inherent structural privilege in visual representations. Tabular formats encode sequences as atomized numerical vectors, forcing pure statistical discovery of temporal rules. Conversely, candlestick charts externalize domain structure into visual syntax, spatial and morphological patterns encode centuriesrefined technical heuristics. This pre-structuring enables VLMs to transfer visual priors from natural images (shape recognition, hierarchical attention) to financial patterns, circumventing sparse-data
Do VLMs Truly "Read" Candlesticks? A Multi-Scale Benchmark for Visual Stock Price Forecasting
constraints that tabular models face. The asymmetry reflects differential architectural alignment rather than data superiority: VLMs exploit pre-encoded domain knowledge in visual grammar; tabular models must inductively reconstruct these abstractions from raw features.
7
Conclusion
This study aims to construct a multi-period candlestick chart dataset and a standardized evaluation framework to examine the ability of VLMs to utilize multi-scale visual market signals. This dataset addresses the issues of excessive complexity in existing datasets and insufficient emphasis on multi-period scales. It constructs a minimal dataset in which each sample includes daily and weekly candlestick charts along with the corresponding target value, while maintaining diversity in overall market trends, individual stock behavior patterns, and balanced sample distribution. The study tested seven widely used commercial VLMs, evaluating them based on model confusion matrices and IC series metrics. Experimental results indicate that while XGBoost, as the baseline, demonstrates superior stability and average ranking capability, certain VLMs show potential in specific areas: Claude-sonnet-45(thinking) exhibits strong predictive power in both price direction and return ranking, delivering the best overall performance; Qwen shows balanced direction prediction and strong ranking capability; Gemini2.5-flash achieves the most balanced direction prediction. We anticipate that future work will incorporate evaluations of additional financial concepts on this dataset, further refine model training, and conduct more detailed investigations into VLMs’ comprehension of candlestick charts.
References [1] Ross L. Beck. The gartley and elliott wave relationship. In The Gartley Trading Method, pages 27–40. Wiley, 2012. [2] C. Chootong and O. Sornil. Trading signal generation using a combination of chart patterns and indicators. International Journal of Computer Science Issues, 9:202–209, 2012. [3] Zihan Dong, Xinyu Fan, and Zhiyuan Peng. Fnspid: A comprehensive financial news dataset in time series. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’24, pages 1–10, New York, NY, USA, 2024. Association for Computing Machinery. [4] R. D. Edwards, J. Magee, and W. H. C. Bassetti. Replacing dow theory with john magee’s basing points procedure. In Technical Analysis of Stock Trends, pages 31–40. CRC Press, 2018. [5] J. Huang, M. Xiao, D. Li, Z. Jiang, Y. Yang, Y. Zhang, L. Qian, Y. Wang, X. Peng, Y. Ren, R. Xiang, Z. Chen, X. Zhang, Y. He, W. Han, S. Chen, L. Shen, D. Kim, Y. Yu, J. Tsujii, et al. Open-FinLLMs: Open multimodal large language models for financial applications. arXiv preprint, 2024. [6] V. A. Ivanyuk. A long-term forecasting technique based on a multi-trend forecast. Soft Measurements and Computing, 12(73):129–138, 2023. [7] Michael D. Archer John M. Bland, Jay M. Meisler. Trade #5: Don’t be a flatlander: Money management. In Forex Essentials in 15 Trades, pages 135–138. Wiley, 2012. [8] Furkan Karadaş, Bahaeddin Eravcı, and Ahmet Murat Özbayoğlu. Multimodal stock price prediction. arXiv preprint, January 2025. Also in Proceedings of the 17th International Conference on Agents and Artificial Intelligence, Vol. 3 (2025), pp. 687–694. [9] S. Khanderwal and D. Mohanty. Stock price prediction using ARIMA model. International Journal of Marketing & Human Resource Research, 2(2):98–107, 2021. [10] Tina Khezresmaeilzadeh, Parsa Razmara, Seyedarmin Azizi, Mohammad Erfan Sadeghi, and Erfan Baghaei Potraghloo. VISTA: Vision-language inference for training-free stock time-series analysis. arXiv preprint, 2025. arXiv:2505.18570v2. [11] S. S. Khurana, P. Singh, and N. K. Garg. Revolutionize AI trading bots with AutoML-based multi-timeframe bitcoin price prediction. SN Computer Science, 4(5), 2023. [12] R. M. I. Kusuma, T.-T. Ho, W.-C. Kao, Y.-Y. Ou, and K.-L. Hua. Using deep learning neural networks and candlestick chart representation to predict stock market. arXiv preprint, 2019.
Conference’17, July 2017, Washington, DC, USA
[13] M. Liang, S. Wu, X. Wang, and Q. Chen. A stock time series forecasting approach incorporating candlestick patterns and sequence similarity. Expert Systems with Applications, 205:117595, 2022. [14] Haoxin Liu, Shangqing Xu, Zhiyuan Zhao, Lingkai Kong, Harshavardhan Kamarthi, Aditya B. Sasanur, Megha Sharma, Jiaming Cui, Qingsong Wen, Chao Zhang, and B. Aditya Prakash. Time-mmd: Multi-domain multimodal dataset for time series analysis. In Advances in Neural Information Processing Systems, 2024. Datasets and Benchmarks Track. [15] R. Nichani, L. Gasmi, N. Laiche, and S. Kabou. Optimizing financial time series predictions with hybrid ARIMA, LSTM, and XGBoost models. Studies in Engineering and Exact Sciences, 5(2):e11188, 2024. [16] N. Nikam. Stock prediction using AIML based on candlestick chart analysis. International Journal for Research in Applied Science and Engineering Technology, 13(4):6867–6870, 2025. [17] H. A. do Prado, E. Ferneda, L. C. R. Morais, A. J. B. Luiz, and E. Matsura. On the effectiveness of candlestick chart analysis for the brazilian stock market. In Procedia Computer Science, volume 22, pages 1136–1145, 2013. [18] G. Roy, J. Fiaidhi, and S. Mohammed. Multi-timeframe algorithmic trading bots using thick data heuristics with deep reinforcement learning. Artificial Intelligence Evolution, pages 107–159, 2022. [19] Y. Sheng, Y. Qu, and D. Ma. Stock price crash prediction based on multimodal data machine learning models. Finance Research Letters, 62:105195, 2024. [20] D. Shu, H. Yuan, Y. Wang, Y. Liu, H. Zhang, H. Zhao, and M. Du. FinChart-Bench: Benchmarking financial chart comprehension in vision-language models. arXiv preprint, 2025. [21] H. S. Sim, H. I. Kim, and J. J. Ahn. Is deep learning for image recognition applicable to stock market prediction? Complexity, 2019(1), 2019. [22] Y. Wu. Stock price prediction based on simple decision tree random forest and xgboost. BCP Business & Management, 38:3383–3388, 2023. [23] W. Xu, D. Xiang, Y. Liu, X. Wang, Y. Ma, L. Zhang, S. Hu, C. Xu, and J. Zhang. FinMultiTime: A four-modal bilingual dataset for financial time-series analysis. arXiv preprint, September 2025. [24] Z. Yu. Stock price prediction using the ARIMA model. Highlights in Science, Engineering and Technology, 88:516–521, 2024. [25] W. Zhāng. Neural network-based algorithmic trading systems: Multi-timeframe analysis and high-frequency execution in cryptocurrency markets. arXiv preprint, 2025.
A
Construction of Numerical Inputs
Numerical time-series data are constructed to support baseline models and to ensure a fair and controlled comparison with the visual modality. Raw OHLCV records are first validated for completeness and integrity, chronologically ordered, and grouped by stock code to form continuous price series. To match the temporal resolutions used in candlestick chart generation, both daily and weekly OHLCV series are prepared, where the weekly series are obtained through standard aggregation of daily records. For each prediction task defined by a specific stock and cutoff date, a fixed-length historical window is extracted from the corresponding OHLCV series at the appropriate temporal resolution. These historical segments capture the same market context as the visual inputs and serve as the basis for numerical modeling. The extracted time-series windows are then transformed into numerical feature representations, which are used as inputs to XGBoost models. Importantly, all numerical inputs are strictly aligned with the same cutoff dates and future return labels as those used in the visual modality. This alignment ensures consistency across different data representations and enables a fair evaluation of numerical baselines against VLM-based models without introducing temporal leakage or information mismatch.
B
Image Preprocessing and System Optimization
To balance API cost, inference latency, and visual fidelity, we adopt a standardized image preprocessing and system optimization pipeline
Conference’17, July 2017, Washington, DC, USA
for all experiments. All input images are first converted to RGB format. For images with transparent channels, a white background is applied to ensure compatibility with subsequent JPEG encoding. Images are then proportionally resized and resampled using the LANCZOS filter to preserve fine-grained candlestick structures. To reduce bandwidth and API costs while maintaining visual readability, images are compressed using JPEG, yielding approximately a 60% reduction in API costs in our experiments. Finally, images are encoded in Base64 to conform to the standard HTTP request format. To improve robustness in parsing model outputs, we employ a three-tier regular-expression–based parsing strategy. The parser first attempts to match the standard output format, <score>numeric value</score>. If a confidence interval is present, the first numerical value is extracted. If neither pattern is matched, the parser falls back to scanning the full response for floating-point values within the valid range of [−0.5, 1.0]. Outputs that cannot be reliably parsed are preserved in their original form for subsequent error analysis. To support large-scale experiments, several system-level optimizations are implemented. Concurrent API requests are limited to five parallel threads, and locking mechanisms are used to prevent duplicate processing. Experimental results are written to CSV files in batches of ten entries, reducing I/O overhead by approximately 90%. In addition, a resume-from-breakpoint mechanism is enabled by maintaining a set of processed keys, allowing interrupted experiments to be safely resumed. Collectively, these design choices improve the stability, efficiency, and reproducibility of experiments involving thousands of stocks evaluated across multiple models.
C 1
2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
D
Our prompt Stock trend analyst with strong discriminative ability and STRICTLY NO bias: analyze 2 charts (daily/weekly) per stock/ date.GREEN = up, RED = down.Predicate:30-day Future Return Rate $=(Future Price - Current Price) / Current Price Analyze: 1. Decompose charts: - Red/green candle clusters: body/wick strength, reversal signals - Moving averages: price position, crossovers, support/resistance (5MA = black, 20MA = blue, 90MA = purple) - Volume: key levels, spikes, and trends - Inflection points: trend shifts, momentum changes 2. Cross-timeframe link: - Connect short-term (daily) and long-term (weekly) signals Rules: - Output ONLY <score>NUM</score>, where NUM is [-0.5, 1.0] with 3 decimal places. - +0.001 to +1.000: upward trend - -0.500 to -0.001: downward trend - 0.000: unclear trend - No default positive bias - One unique score per stock/date - No additional text Examples: <score>0.354</score> <score>-0.210</score> <score>0.000</score>
Limitation and further work
This study is based solely on evaluation metrics such as confusion matrices and information coefficients for time series, and does not yet include assessment tasks corresponding to financial concepts like resistance levels, support levels, or trend patterns. We
Hu et al.
anticipate that future work will supplement this dataset with such semantic annotations and evaluations to more accurately assess VLMs’ understanding of candlestick chart content. Additionally, the VLMs used in this study are prompt-based and untrained for taskspecific purposes, which may limit their predictive performance. Subsequent work will involve further training or fine-tuning of the models to explore VLMs’ understanding capabilities of candlestick charts more thoroughly. Nevertheless, the core significance of this study lies in constructing a candlestick chart dataset with two time scales. We hope our results will inspire further research.