1
DSCH-Loss: A Dynamic Semantic Channel Objective for Deep Semantic Hashing
arXiv:2607.24567v1 [cs.AI] 27 Jul 2026
Tobias J. Bauera Christian Riessb Daniel Loebenbergera,c a b Fraunhofer Institute for Applied and Friedrich-Alexander-Universität Integrated Security AISEC Erlangen-Nürnberg Weiden, Germany Erlangen, Germany {tobias.bauer, daniel.loebenberger} christian.riess @aisec.fraunhofer.de @fau.de
Abstract—Semantic hashing methods for generating short binary hash codes that allow efficient approximate nearest neighbor search in high-dimensional data spaces have gained extensive consideration in recent years. Deep learning-based methods offer better semantic capturing capabilities than traditional approaches relying on manual feature engineering. Moreover, they enable a data-driven approach to semantic hashing across diverse data modalities, yielding high-quality cross-modal hash codes within a shared Hamming space. Previous work investigated the properties of this Hamming space and introduced a loss function based on predefined so-called semantic channels with fixed width and Hamming distances derived from label similarities. However, this formulation also introduced discontinuities into the loss landscape, complicating optimization. Based on these observations, we propose a newly designed loss function, Dynamic Semantic Channel Hashing (DSCH), using dynamically sized and positioned semantic channels in order to avoid loss landscape discontinuities. Furthermore, we endorse the use of tie-aware Mean Average Precision (mAP) as evaluation metric as it addresses the ambiguity in sample retrieval ordering, which emerges from the discreteness of hash code distances. Finally, multiple experimental settings conducted on two popular datasets and incorporating two different model architectures provide strong evidence that training using the DSCH objective outperforms training using other state-ofthe-art loss functions. In a total of 35 out of 40 cross-modal and intra-modal retrieval tasks, models trained with DSCH achieve significantly higher tie-aware mAP scores across all four tested hash code lengths, showing compelling results across model architecture and used dataset. The mAP score uplifts are consistent and amount up to 1.75 percentage points compared to the respective second best. Index Terms—Deep semantic hashing, Cross-modal retrieval, Hamming space, Deep learning
I. I NTRODUCTION
T
HE task of finding approximate nearest neighbors in high-dimensional data spaces can be tackled using Deep Semantic Hashing (DSH) as the application of deep neural networks to the field of Semantic Hashing [1], [2]. Given a query sample, the goal is to retrieve semantically similar samples from a database or retrieval set. Mapping data samples to compact binary hash codes, i.e., hashing the samples, is a viable and storage-efficient [1] way to represent the semantics. In general, a hash function takes data of arbitrary length – often represented as a bit-string of any length – and outputs a binary hash code of a given fixed length, e.g., 32 bits. These short hash codes allow for computationally efficient nearest-neighbor
Christian Berglerc c Ostbayerische Technische Hochschule Amberg-Weiden Amberg, Germany c.bergler @oth-aw.de
search [1], [2]. Furthermore, DSH is also well-suited for crossmodal retrieval which extends the single-modal retrieval setting with multi-modal data samples and the task of finding similar samples in a different modality [1], [3], [4]. This is due to its data-driven training approach using multi-modal samples sharing a common semantic. The remainder of this paper is organized as follows. Section II presents fundamental concepts of DSH and crossmodal retrieval as well as specific model architectures and training procedures used as a basis for our experiments. In Section III, we propose a new and novel loss function for DSH. Next, Section IV describes the experimental setup, including the data material and model architectures as well as data preprocessing procedures, the training protocol, and evaluation metrics. Further, the experimental results are presented and analyzed in Section V. Section VI concludes this paper and provides perspectives on future research. II. R ELATED W ORK A. (Deep) Semantic Hashing Hashing is the process of mapping high-dimensional data to short binary hash codes by projecting the input into a lowdimensional Hamming space [1]. Such high-dimensional input data may comprise various modalities, including images, text, audio, and video, as well as other structured or unstructured data types [1], [2], [3]. In the context of Semantic Hashing, the goal is to find a hash function that maps semantically similar inputs to similar or identical hash codes to facilitate nearest neighbor search [1], [2]. The Hamming distance between any two hash codes can be easily computed as the number of differing binary digits (bits). Good semantic hash functions are characterized by their ability to output hash codes that are close, i.e., have a small distance, for similar inputs and simultaneously output distant hash codes for dissimilar inputs [1]. There are different algorithmic goals a concrete semantic hash function may pursue, e.g., it may be optimized for creating hash tables or it may be designed for hash code ranking [3]. Either way, these hash functions can either be crafted or learned and the paradigm of Deep Semantic Hashing emerges from end-to-end hash function learning using deep neural networks [1]. Luo et al. [1] present a comprehensive survey on this topic.
2
B. Cross-Modal Retrieval As stated in the previous section, Semantic Hashing allows for an efficient similarity search using high-quality hash codes as surrogates for the high-dimensional underlying data. In previous work, the hash function operates on single-modal input data, e.g., only images or only text, see [5], [6], [7], [8], [9], with one of the earliest work in the cross-modal DSH domain done by Jiang and Li [10]. More recent research, however, focuses on multi-modal hashing, which aims to project data from different modalities into a shared Hamming space [1], [3]. This allows for so-called cross-modal retrieval, such that for example images can be searched using text and vice versa. In their literature review, Wang et al. [3] present an elaborate taxonomy on cross-modal retrieval, differentiating between supervised and unsupervised as well as real-valued and hashing-based approaches. Although many combinations of media modalities are studied, the main focus of research currently lies upon the text-image retrieval [3] as seen in [11], [12], [13], [14], [15], [16], [17], [18], [19].
samples too close or too distant within the Hamming space and endures no penalties when the Hamming distance lies within the target channel. These channels have a fixed width of τ and their position within the Hamming space is determined by the label-similarity of the samples [18]. As their work provides the theoretical foundation for our loss formulation, the concept of semantic channels is elaborated in detail in Section III-B. The authors report state-of-the-art results using their method on two datasets [18]. Even though the used evaluation metric is widespread, the results are not directly comparable as the metric formula is not permutation-agnostic and, thus, produces different scores for semantically identical predictions. This issue is addressed in [27] and elaborated upon in Section IV-D. III. P ROPOSED M ETHOD A. Problem Statement and Notation
We define the datasets to be comprised of n samples, i.e., D = {xi }ni=0 , with each sample a pair of image and text, (I) (T) i.e., xi = ⟨xi , xi ⟩. Furthermore, each sample is annotated with a one-hot encoded label vector li ∈ {0, 1}c where c is C. Model Architectures to be learned crossIn the context of cross-modal DSH a variety of neural the number of labels in the dataset. The (M) modal hash function accepts inputs x of modality M, e.g., i network architectures have been proposed and applied to the (I) (T) any image x or text x , and produces a k-bit hash code i i problem [1], [3]. A fundamental approach is to use an off-the- (M) k b ∈ {−1, 1} after applying the sign function to the model shelf and often pre-trained image model, e.g., a Convolutional i (M) k Neural Network (CNN) [20], [21] such as ResNet [22], and output b̂i ∈ R . The notation used throughout this paper is replace its classification head with a hash-learning Multilayer summarized in Table I. Both datasets used in this paper consider two samples similar Perceptron (MLP) [1], [3]. For the text modality, it is common to use simple MLPs projecting Bag of Words (BoW) vectors if they have at least one label in common, mathematically to hash codes as seen in [10], [18], [23], [24]. However, recent xi ∼ xj ⇐⇒ li · lj > 0. Our goal, however, is to construct a publications incorporate the Transformer architecture [25] that loss function that incorporates real-valued semantic similarity operates on token embeddings and which underpins recent of samples. The more the label vectors of two samples align, the advances in Natural Language Processing [26]. DSH model higher their similarity score. Following [18], we use the cosine architectures can be categorized into two primary approaches: similarity between label vectors as inter-sample similarity score: single-stream and dual-stream architectures [3], [4]. The former li · l j Sij = cos (li , lj ) = ∈ [0, 1], 0 ≤ i, j ≤ n. (1) consists of a single module that operates on inputs of different |li | · |lj | modalities in a uniform manner, whereas the latter employs Furthermore, Hu et al. [18] formulate the Hamming distance d one module per modality [3], [4]. There also exist hybrid between two binary hash codes using the cosine similarity: approaches, which combine per-modality feature extraction k with a shared hash-learning module [4]. (m ) (m ) (m ) (m ) d bi i , bj j = 1 − cos bi i , bj j . (2) 2 D. Semantic Channel Hashing Hu et al. [18] report a compression of the Hamming space when using common DSH loss functions which constrain B. Semantic Channels Hu et al. [18] introduced the notion of semantic channels. dissimilar samples to be either orthogonal or as distant as possible in Hamming space. In the first case, the space of fully Their goal was to “allocate [...] an appropriate [target] Hamming or partially similar samples gets compressed and almost half distance to the hash codes bi and bj of samples xi and xj [...] of the Hamming space is not used according to the authors. based on their similarity Sij ,” [18] which is the cosine similarity In the latter case, semantically similar samples are able to of their labels as calculated in Equation (1). In essence, the utilize more of the Hamming space, however, now the space authors assigned each pair of samples a τ -wide “semantic for dissimilar samples gets compressed, which reduces the channel” within the Hamming space. The notion of a semantic channel is geometrically equivalent to a hyperspherical shell separability of these samples [18]. To overcome the limitations of current loss function con- or k-dimensional annulus. The position in terms of distance in straints, the authors propose a novel loss function named the Hamming space of said channels is based on their samples’ Semantic Channel Hashing (SCH) [18]. They address the label similarities. The loss function is designed so that the Hamming space compression issue by introducing target model is incentivized to project two samples into the shared channels for pairs of samples. The SCH loss ensures that Hamming space in a manner that satisfies the target Hamming during training the model is punished for putting pairs of distance constraint.
BAUER et al.: DSCH-LOSS: A DYNAMIC SEMANTIC CHANNEL OBJECTIVE FOR DEEP SEMANTIC HASHING
3
The loss landscape visualization of SCH depicted in Figure 1a clearly shows the discontinuity. Addressing this loss function discontinuity is an integral part of our proposed loss described Notation Description in Section III-C. D Particular dataset, e.g., Q query, R retrieval, T train In addition to the iterative loss formulation, Hu et al. present n = |D| Number of samples in selected dataset D a vectorized version [18, Equation (10)]. Albeit them stating it (I) (T) xi = ⟨xi , xi ⟩ i-th sample pair in D comprising of image and text being an equivalent formulation [18], there is a difference c Number of labels in D, e.g., c = 21 for NUS-WIDE (see Section IV-A) between the iterative and vectorized formulas in terms of li ∈ {0, 1}c One-hot encoded labels of sample xi Hamming distance calculation. In the iterative version they k Hash code length in bits (M) (M) use cosine similarity as seen in Equation (2), whereas in the bi ∈ {−1, 1}k Binary hash code of sample xi of modality M (M) (M) vectorized version they opted to approximate the Hamming b̂i ∈ Rk Real-valued hash code of xi prior quantization T (I) n×d I X ∈R Images of D with data dimensionality dI distance by B(mi ) × B(mj ) [18]. This is indeed equivalent (M) X(T) ∈ Rn×dT Texts of D with data dimensionality dT in the case of truly binary hash codes, i.e., Bij ∈ {−1, 1}, L ∈ {0, 1}n×c One-hot encoded labels of D √ (M) since for each hash code vector |bi | = k. However, for S ∈ [0, 1]n×n Inter-sample label cosine similarity (M) n×k B ∈ {−1, 1} Binary hash codes of samples of modality M in D backpropagation to be applicable, the loss function must be Real-valued hash codes prior quantization into B(M) B̂(M) ∈ Rn×k differentiable, which precludes the use of the sign function as it is non-differentiable at zero and its gradient is zero everywhere else. Therefore, the model outputs “almost-binary” In their paper, Hu et al. [18] split pairs of samples into three hash codes with “bits” that are still real-valued, i.e., B̂(M) ∈ R. distinct sets based on their label similarity: negative (Sij = 0), Consequently, the aforementioned Hamming distance ijapproxi√ fully positive (Sij = 1), and partially positive (0 < Sij < 1). (M) mation breaks since in general |b̂i | ̸= k. For this reason, Next, they defined a compound loss function on these three sets, our vectorized formulation employs the cosine similarity for using the label similarity and Equation (2) to calculate the upper Hamming distance calculation as well (see Section III-C). and lower target Hamming distances λ(u) and λ(l) , respectively. Additionally to these findings, Hu et al. introduce two FrobeThese upper and lower bounds define the aforementioned nius norms to their vectorized loss formulation. They claim channel. We re-formulate the equations given in [18] into better balancing between positive and negative samples [18]. a harmonized and simplified set of equations: However, we empirically found that incorporating the Frobenius ( k norm regularization induces model collapse during training, if S = 0, ij (l) (3) yielding trivial solutions in which the network outputs identical λij = k2 otherwise. 2 (1 − Sij ) − τ hash codes regardless of input. Consequently, in Section III-C ( k if S = 0, we defaulted to the L1,1 -norm, effectively summing all error ij (u) λij = k (4) terms, which is equivalent to the iterative formulation. otherwise. 2 (1 − Sij ) TABLE I M ATHEMATICAL NOTATION .
(u)
(l)
The channel width λij − λij is equal to τ for partially similar k or fully similar samples, whereas it is 2 for dissimilar samples, (u) (l) i.e., λij − λij ∈ τ, k2 . Analogously to Hu et al. [18], the loss function can be defined as a sum over all samples and all combinations of modalities: {I,T} n o n X X (l) (u) LSCH = ψij max 0, λij − d∗ , d∗ − λij , mi ,mj i,j=1
(m ) (m ) where d∗ = d b̂i i , b̂j j the observed Hamming distance, β if Sij = 0, and ψij = α if Sij = 1, the similarity set weight. (5) 1 otherwise Following [18], the weighting hyperparameters α and β are set to 1, effectively simplifying the weight in Equation (5) to ψij = 1 in all cases. However, even after this simplification, the similarity set distinctions in Equations (3) and (4) still remain, introducing discontinuities in the loss landscape at Sij = 0. This is because the lower channel bound jumps from (l) (l) λij = k2 for Sij = 0 to λij ≈ k2 − τ for Sij = ε with ε > 0 a small positive number. Similarly, the upper channel bound (u) (u) jumps from λij = k for Sij = 0 to λij ≈ k2 for Sij = ε.
C. Proposed Loss for Dynamic Semantic Channel Hashing In this section, we present our proposed loss function for Dynamic Semantic Channel Hashing (DSCH) which is based on the SCH loss by Hu et al. [18]. First, we specify the design goals that led to the formulation of the DSCH objective. Next, we give an iterative formulation as well as an optimized vectorized version. Then, we introduce an additional quantization loss to accommodate for different model output behaviors (see Section IV-B). In a final step, all loss terms are combined into a joint total loss function. All hyperparameters used in the formulas are summarized in Table II. 1) Design Goals: A major design goal for DSCH is to overcome the loss landscape discontinuity for slightly similar sample pairs, i.e., Sij ≈ 0 as described in Section III-B. Likewise, this discontinuity also involves the width of the target semantic channel, which also depends on whether the similarity is zero or not. We, therefore, propose a dynamic channel width w that is dependent on the label similarity and ranges from a configurable minimum channel width τ to k − λneg . For dissimilar sample pairs, we introduce a configurable minimum distance λneg = k2 , which is analogous to the behavior of SCH [18] (see Equation (3)). It is important to note that the channel width scaling may be non-linear in order to enforce τ -sized channels for almost similar sample pairs, i.e., Sij ≈ 1.
4
Additionally, our loss should incentivize the model to pull fully similar sample pairs, i.e., Sij = 1, close in Hamming space and, thus, should penalize any Hamming distance greater than zero for said pairs. Finally, as reasoned in Section III-B, a configurable loss curve hyperparameter is used instead of the Frobenius norm, with the default being a L1,1 -norm. 2) Iterative Formulation: Our proposed loss uses dynamic channel widths for sample pairs based on their label cosine similarity. The following Equations (6) to (8) define the computation of the semantic channel width wij , the leftmost semantic channel point pij , and the proposed loss LD : wij = (1 − Sij ) w · (k − λneg − τ ) + τ.
γ
(6)
pij = λneg · (1 − Sij ) − Sij · τ.
(7)
{I,T }
LD =
n X X
γ
(ψij max {0, pij − d∗ , d∗ − (pij + wij )}) ℓ ,
mi ,mj i,j=1
(m ) (m ) where d∗ = d b̂i i , b̂j j the observed Hamming distance, β if Sij = 0, and ψij = α if Sij = 1, the similarity set weight. (8) 1 otherwise Following Equation (6), the channel width gets interpolated between the minimum channel width τ and the reserved space for dissimilar samples, i.e., wij ∈ [τ, k − λneg ]. However, the interpolation is not linear, but controlled by the channel width curve modifier γw . This measure ensures that the width of the target semantic channel stays small for an extended period of similarity values while it approaches dissimilarity, and also retrains the continuity property. This is in stark contrast to the proposal of SCH by Hu et al. [18] that does not interpolate the channel width, but jumps between two values as described in Section III-B. Figure 1 depicts the loss landscapes of both algorithms as per Equations (5) and (8). The target semantic channel forms a trench in both loss landscapes. Apart from this, the loss landscape surface of DSCH is smooth without discontinuities as visualized in Figure 1b. We also provide an interactive 2D-sliced comparison visualization.1
(a) Semantic Channel Hashing with τ = 3
Observed Hamm ing distance d ∗
k
0.0
ity
si S mila ij r
y rit
0.5 k 2
be l
0
1.0
La
0.0
ij
k
be
Observed Hamm ing distance d ∗
ls i S mila
0.5 k 2
La
0
1.0
Loss value L
Loss value L
1 https://www.desmos.com/calculator/qog7tklzdz
(b) Dynamic Semantic Channel Hashing with γw = 2, τ = 1
Fig. 1. Visual comparison of the loss landscapes of Semantic Channel Hashing and Dynamic Semantic Channel Hashing. The loss landscape discontinuity at Sij ≈ 0 of SCH is clearly visible, whereas DSCH boasts a smooth surface.
Besides continuity, there is another reason in favor of channel width interpolation: a low similarity score of two samples means that they are only slightly similar, which in turn carries a large uncertainty about the semantic basis of their similarity. Thus, a wider semantic channel accommodates this uncertainty, whereas a fixed-sized channel width would impose an unjustified constraint on the Hamming space. Conversely, a high label similarity score implies a clear and unambiguous basis of similarity, allowing a more restrictive semantic channel width. In this regime the uncertainty is low, thus, a narrower channel is sufficient and even preferable. This property allows decreasing the minimum channel width to τ = 1 compared to the relatively wide channel of 3 in [18]. The semantic channel provides a distance-based ordering that guides the model at projecting the samples into the shared Hamming space. These insights provide compelling evidence in support of our proposal of non-linear channel width interpolation. The leftmost semantic channel point pij depends linearly on the label cosine similarity as stated in Equation (7). This (l) is analogous to the lower bound λij of SCH, defined in Equation (3). One design goal for DSCH was continuity, which is why we opted to interpolate between the two cases in Equation (3) based on the similarity. This resulted in the additional term Sij · τ in Equation (7). Finally, we combine the dynamic semantic channel width from Equation (6) with the updated computation of the leftmost semantic channel point from Equation (7) into our proposed loss presented in Equation (8). Analogously to SCH (see Equation (5)), the objective is defined as a sum over all samples and all combinations of modalities in order for the model to learn a Hamming space projection considering both intra-modal and cross-modal similarities. Furthermore, we also chose to introduce the option to enable custom weighting of the similarity sets using hyperparameters α and β, which is disabled by default, i.e., α = β = 1, resulting in ψij = 1 in all cases. Another hyperparameter is γℓ , which acts as a curve modifier to penalize large distance-errors, e.g., quadratically. It defaults to 1 for linear loss scaling. As stated previously in Section III-B, the label cosine similarity determines the Hamming distance of two hash code vectors following Equation (2). 3) Vectorized Formulation: The proposed loss can be vectorized in a straight-forward manner. Following the notation in [18], we construct matrices from elements, e.g., the matrix n×n Ψ = [ψij ] is constructed from elements ψij , where i is the row index and j is the column index. We denote an appropriately sized matrix with elements all ones as 1, the Haramad product as ⊙, and the element-wise power function as ◦. The ReLU activation function operates on elements, too, and the L1,1 -norm is denoted by ∥·∥1 . ! {I,T } X ∥(Ψ ⊙ ReLU (P − D∗ ))◦γℓ ∥1 + LD = , ◦γ ∥(Ψ ⊙ ReLU (D∗ − (P + W))) ℓ ∥1 m ,m i
j
h in×n the observed (m ) (m ) where D∗ = d B̂i i , B̂j j Hamming distances, and W = (k − λneg − τ ) · (1 − S) and P = λneg · (1 − S) − τ · S.
◦γw
+ τ · 1, (9)
BAUER et al.: DSCH-LOSS: A DYNAMIC SEMANTIC CHANNEL OBJECTIVE FOR DEEP SEMANTIC HASHING
TABLE II H YPERPARAMETERS AND THEIR INITIAL VALUE .
5
IV. E XPERIMENT S ETUP A. Data Material and Data Partitioning
We perform experiments on two well-known cross-modal multi-label benchmark datasets, namely NUS-WIDE [28] and Minimum distance of negative pairs MIR Flickr 25k [29]. Both provide samples comprising an τ =1 Minimum channel width γw = 8 Channel width curve modifier image, its user-provided hashtag list, and a set of manually γℓ = 1 Loss curve modifier annotated category labels. α=1 Fully positive sample pairs set weight NUS-WIDE [28] contains 269 648 samples, each consisting β =1 Negative sample pairs set weight κq = 0.01 Quantization error weight of one image and 0 to 632 hashtags. Additionally, each sample is annotated with 0 to 13 out of a set of 81 labels. In accordance to some previous research [1], [3], [10], [14], we select All operations in Equation (9) can be implemented efficiently the top-21 labels and discard any samples with no labels, using matrices. We provide a reference implementation that resulting in a subset of 195 834 samples. It is worth noting ships with both the iterative and vectorized versions of our that in the literature there is no consensus about the subset loss algorithm.2 selection process and some select the top-10 labels [4], [17], 4) Additional Cross-Modal Quantization Loss: A detailed [18] while others seem to use the full dataset [11]. We then description of the experimental models is provided in Secrandomly sample 2100 samples to form the query set Q with tion IV-B. In order to allow for different final activation the remaining samples forming the retrieval set R. Next, a functions, we introduce an additional quantization Lq as total of 10 500 samples are randomly selected from R to form regularization term to the DSCH objective. Inspired by Zhou the training set T . Mathematically, T ⊂ R and R ∩ Q = ∅. et al. [14], we define the quantization error as the distance of Although literature suggests sampling 100 instances per label the model output to the cross-modal quantized target hash code. as queries and 500 instances per label for training [1] or “an However, we employ the L1,1 -norm over the error instead of equal number of samples from all [labels],” [18] this is an the squared Frobenius norm. We present the quantization error, unsatisfactory solution to balance label occurrence probabilities. which is summed over each modality, in vectorized form: On the contrary, this naı̈ve approach leads to an even more {I,T } {I,T } unbalanced dataset due to multi-label interdependence. We, X X Lq = ∥B̂(m) − B∥1 , where B = sign B̂(m) . therefore, opted to use random sampling with fixed seed to m m closely match the dataset distribution instead of introducing (10) skew or using other methods, such as iterative stratification [30]. MIR Flickr 25k [29] contains 25 000 samples comprising This formulation of quantization error has the benefit of an one image and 0 to 75 hashtags as well as 0 to 14 labels out additional cross-modal error term introduced by the summation of a set of 24 labels. We follow a common subset selection of model outputs before quantization into B. This incentivizes process [3], [10], [14], [18] and select all samples that have the model further to align the Hamming space projection of at least one hashtag that is included in the top-20 hashtags. sample pairs, thus, enforcing a shared Hamming space. Furthermore, all samples that have no labels are discarded, 5) Overall Loss Function: In a final step, all previously resulting in a subset of 20 015 samples. Finally, we employ the defined loss terms are combined into an overall loss function. same subsampling strategy as with NUS-WIDE and select 2000 For improved stability during training, we opt to normalize the samples for Q and 10 000 samples for T in order to comply losses by batch size. Furthermore, the quantization error gets with other implementations [10], [14], [18]. weighted by a factor of κq = 0.01 as it would hurt performance For model selection, hyperparameter tuning, and to assess if weighted too strong. Our proposed overall loss function is model performance during development, we employ two defined as follows: additional partitions: the validation query set Vq and the 1 1 validation retrieval set Vr . To the best of our knowledge, L = 2 LD + κq · Lq . (11) n n there is no evidence in the literature for performing such As referenced earlier, all used hyperparameters and their train-validation-test split in the cross-modal retrieval domain. initial values are listed in Table II. Hyperparameter Description λneg = k2
2 https://github.com/Hirnmoder/dynamic-semantic-channel-hashing
Dataset Test Train Val
usable Q
TABLE III N UMBER OF SAMPLES PER DATASET AND DATA PARTITION .
discarded Q∩R=∅
R T
T ⊂R Vq
Vr
Vq ∩ Vr = ∅
Fig. 2. Schematic representation of the dataset partitioning process to obtain query, retrieval, train, validation query, and validation retrieval sets.
Partition Total Usable D (Test) Query Q (Test) Retrieval R Train T Validation Query Vq Validation Retrieval Vr
Numbers of samples NUS-WIDE
MIR Flickr 25k
269 648 195 834 2100 193 734 10 500 2100 181 134
25 000 20 015 2000 18 015 10 000 2000 6015
6
Our train-test-validation splitting approach is designed to accomplish three major goals: 1) perform validation on a query set with a similar size as the (test) query set, 2) make validation and train sets completely disjoint to assess model generalization capabilities, and 3) keep test and train set construction identical to existing literature for better comparison. To construct the validation partition, we first take the remainder of the retrieval set without training samples R \ T . Next, depending on the initial dataset being NUS-WIDE or MIR Flickr 25k, a total of 2100 or 2000 samples are selected at random to form the validation query set Vq , respectively. The remaining samples are then assigned to the validation retrieval set, i.e., Vr = R\T \Vq . The breakdown of the total dataset into all data partitions is depicted in Figure 2. Furthermore, the number of samples for each data partition is reported in Table III.
descriptions as text input [38], or cannot be used in a crossmodal retrieval setting [15]. Since OpenCLIP provides their own text preprocessing pipeline including tokenization and embedding, no custom preprocessing steps are implemented for CLIPHash. The maximum sequence length for the selected CLIP backbone is 77 tokens. In Figure 3, the two model architectures as well as an example forward flow are depicted. In both figures, purple shaded parts mark frozen, off-the-shelf pre-trained modules in each architecture. The dual-stream structure of TDSRDH is clearly visible in Figure 3a and the loss function acts as the only “connecting tissue” between modalities. This is different from the hybrid-stream structure of CLIPHash depicted in Figure 3b. Here, both modalities share some layers within the pre-trained CLIP model and – more importantly – they share a common hash-learning MLP. Furthermore, the loss function imposes additional cross-modal alignment constraints. B. Model Architecture and Data Preprocessing The configuration of the hash-learning MLPs, that is their To demonstrate the effectiveness of our proposed loss number of layers and the neurons per layer, is dependent function, we implement two model architectures, both of on the model and experiment. However, the defaults for which use the Transformer architecture [25]. In the following, TDSRDH are the configurations given in [14]. For example, both architectures are outlined and their respective data we denote a three-layer MLP with 512-dimensional inputs, preprocessing steps are described. a first hidden layer with 1024 neurons, a second hidden As a baseline, we employ a dual-stream model proposed layer with 8192 neurons, and an output of dimension k as by Zhou et al. [14] named TDSRDH. Its image stream tanh tanh −−→ 1024 −−−→ 8192 → − k. We also specify consists of a pre-trained ResNet-152 backbone and a three- follows: 512 − d=0.1 d=0.1 the activation functions and dropout probabilities used after layer hash-learning MLP. The text stream consists of a singleeach layer. In this example, the identity function, i.e., no layer Transformer with four heads followed by a three-layer activation function, is used for the last layer. Contrary to Zhou hash-learning MLP. The authors introduce a three-fold loss et al. [14] and Hu et al. [18], we employ the tanh activation comprising a cross-modal pairwise loss, a quantization loss, and a cross-modal triplet loss [14]. We implemented their function in the inner layers of the hash-learning MLP and use architecture and loss function based on the descriptions in no output activation function in order for the model to be able their paper as no code is publicly available. This is also to express semantic subtleties with outputs exceeding the ±1 true for data preprocessing. Furthermore, we were unable to bound that would be imposed by the tanh function. Dropout obtain a 512-dimensional Word2Vec [31] word embedding is never applied to the last layer. The image-stream hash-learning MLP of TDSRDH has the model mentioned by Zhou et al. [14]. Instead, we chose the Google News 300d3 embeddings and added a trainable following configuration based on [14], where 2048 is the outtanh tanh −−→ 4096 −−−→ 4096 → − k. linear layer to upscale the 300-dimensional word embeddings put size of ResNet-152: 2048 − d=0.1 d=0.1 to the 512 dimensions used in the Transformer. Next, we Similarly, following [14] the text-stream hash-learning MLP is tanh apply sinusoidal positional encoding as proposed by Vaswani configured as follows: 512 −tanh −−→ 1024 −−−→ 8192 → − k. d=0.1 d=0.1 et al. [25]. However, as described in Section IV-A, the datasets’ The CLIPHash model contains only a single hash-learning texts are essentially user-provided hashtags and not ordinary MLP, which retrieves a 4096-dimensional embedding vector descriptions. In order to cleanse and tokenize these texts as from the backbone and processes it in five layers as follows: tanh tanh tanh tanh well as correct misspellings, we use symspellpy [32]. The 4096 −−−→ 4096 −−−→ 4096 −−−→ 1024 −−−→ 256 → − k. d=0.1 d=0.1 d=0.1 d=0.1 maximum sequence length is configurable and set to 128 tokens. The second model architecture follows a hybrid-stream C. Model Training approach. As the multi-modal backbone we employ an offThe training procedure is implemented as a straight-forward the-shelf CLIP model [33], namely ViT-H-14-quickgelu, prebatched learning strategy to optimize the model parameters trained on the DFN-5B dataset [34] and implemented by using gradient descent. Although the DSH literature predomOpenCLIP [35]. We then append a MLP to map the CLIP inately reports the use of the SGD optimizer (see [14], [18], embeddings to the output Hamming space. We call this model [23], [24], [37]), we utilize the Adam optimizer [39] for faster CLIPHash. To the best of our knowledge, this architectural convergence as do some recent papers (see [12], [16], [40]). design has not previously been explored in the context of We configure Adam with an initial learning rate of 1 × 10−5 , image-text Semantic Hashing. A similar model architecture −8 was proposed for the video-text domain [36], and proposals an epsilon set to 1 × 10 , no weight decay, and momentum to the image-text domain either use BoW text inputs [24], hyperparameters set to β1 = 0.9 and β2 = 0.999. We train scheduler that a different text-stream model [17], [37], labels instead of for 200 epochs and use a custom learning rate−5 applies a cosine-annealing drop-off from 1 × 10 to 1 × 10−6 3 https://code.google.com/archive/p/word2vec/ between epochs 75 to 150. The batch size is set to 128.
BAUER et al.: DSCH-LOSS: A DYNAMIC SEMANTIC CHANNEL OBJECTIVE FOR DEEP SEMANTIC HASHING
Feature Hash-learning Extraction MLP
Input
butterfly, mariposa, vivid, tones, natural, . . .
Feature Hash-learning Extraction MLP
SingleLayer Transformer
Hash Code
−1 +1 . . . −1 +1
Hash Code
−1 −1 . . . +1 +1 HybridStream CLIP
Loss
Input
+1 −1 . . . −1 +1
Loss
ResNet 152
7
butterfly, mariposa, vivid, tones, natural, . . .
−1 +1 . . . +1 +1
(b) CLIPHash
(a) TDSRDH
Fig. 3. Schematic overview over the two model architectures employed for the experiments in this paper. Shaded in purple are off-the-shelf pre-trained modules, which are frozen during training. TABLE IV P ROGRESSIVE IMAGE AUGMENTATION MIXTURES .
None
Easy
Medium
Hard
0 to 19 20 to 39 40 to 59 60 to 79 80 to 99 100 to 149 150 to 199
100 % 25 % 25 % 25 % 25 % 25 % 25 %
0% 75 % 50 % 25 % 25 % 0% 0%
0% 0% 25 % 50 % 25 % 25 % 0%
0% 0% 0% 0% 25 % 50 % 75 %
Augmentations
None 0
—
—
1
Flip horizontally Rotate by fixed value Convert to grayscale Random crop
p=1 ±90◦ , ±45◦ — scale ∈ [0.5, 1.0], ratio = 1
1
Flip horizontally
p = 0.5
1 to 2
Configuration
Random rotation between −45◦ to 45◦ Convert to grayscale — Random crop scale ∈ [0.35, 1.1], ratio ∈ [0.9, 1.1] Random erasing scale ∈ [0.02, 0.05], ratio ∈ 13 , 3 ∗ Random color jitter b, c, s ∈ [0.8, 1.2], h ∈ [−0.1, 0.1] Random uniform noise blending ∈ [0.0, 0.5]
1
Flip horizontally
1 to 3†
Description
Easy
Level #
Medium
In order to demonstrate the efficacy of our proposed loss function, we also train the models with different losses. For fair comparison we try to match the training procedure and hyperparameter setup of the original papers. As mentioned in Section IV-B, the authors of the TDSRDH model proposed a three-fold loss function alongside [14]. When training using their TDSRDH-loss, we also follow their alternate-learning strategy [14], which means that two streams – image and text – get updated in an alternating manner. However, we use the Adam optimizer as stated above, and introduce a normalization to the loss terms, similar to the ones of our proposed loss described in Section III-C and, in particular, Equation (11). This also requires a re-evaluation of the TDSRDH-loss weighting hyperparameters, which we found matching the original paper’s performance claim when set to αTDSRDH = 0.05 and βTDSRDH = 1. The batch size, initial learning rate, and number of epochs match our settings described above.
Image Augmentation Mixture
Epoch Range
TABLE V I MAGE AUGMENTATION GROUPS BY HARDNESS LEVEL .
Hard
During training, we perform data augmentations to the image modality to enhance the diversity and variance of the training dataset as well as to incentivize the model to generalize on unseen data. We define four hardness levels of data augmentations and employ a progressive augmentation mixture strategy with fixed ratios at each step, which are listed in Table IV. The training starts with no augmentation at all and continues to add a growing portion of augmentations with increasing hardness, while also retaining 25 % unaltered images. The application of data augmentation in the DSH domain is uncommon since only Luo et al. [1] report a few training procedures with image augmentation, whereas the other surveys [2], [3], [4] contain no notice of it. However, augmenting training samples is standard practice in the Computer Vision domain [41], which is why we opted to implement model-free single-image as well as model-free non-instance-level multiimage augmentations. For each hardness level, we define one or more groups of augmentations as well as a count parameter # that indicates how many individual augmentations per group can be active. All augmentation groups and their count parameter as well as the individual augmentations and their configurations are listed in Table V clustered by hardness level.
Random rotation between −90◦ to 90◦ Convert to grayscale — Random crop scale ∈ [0.2, 1.1], ratio ∈ 34, 34 Random erasing scale ∈ [0.05, 0.15], ratio ∈ 14 , 4 Random color jitter∗ b, c, s ∈ [0.5, 1.5], h ∈ [−0.25, 0.25] Flip vertically p=1 Elastic transform α = 50, σ = 5 Random uniform noise blending ∈ [0.0, 0.9]
1†
Full-frame image blend Picture-in-picture
p = 0.5
blending ∈ [0.0, 0.45] scale ∈ [0.05, 0.25]
# is the count parameter, indicating how many augmentations of this group are active at any one time. ∗ b, c, s, h are brightness, contrast, saturation, and hue, respectively. † Only one of the two augmentation groups is active per sample. The upper group has a probability of 80 % to be selected, the lower group has 20 %.
8
When training with the SCH loss by Hu et al. [18], we use their hyperparameter setup and set αSCH = 1, βSCH = 1 and τSCH = 3. The training routine is identical to ours and updates both image and text stream simultaneously. In the accompanying code, the authors also employ normalization to the loss terms [18], which is similar to ours. However, we use the Adam optimizer with a different learning rate, a larger batch size of 128 instead of 32, and no weight decay in order to achieve better comparability with our proposed loss. It is important to note that we conduct each experiment at least three times with different initializations to obtain an averaged result for a decreased effect of model training randomness. This also allows us to assess model performance spread induced by random initializations. D. Evaluation Metrics For assessing the cross-modal retrieval performance of DSH models, the literature has agreed on several metrics with Mean Average Precision (mAP) and the precision-recall curve being one of the more popular ones [1], [2], [3]. Both metrics are also reported by the authors of the TDSRDH model [14] and the SCH loss [18]. Although these two metrics are used throughout the literature, there exist many different interpretations as to how to compute them and what specifications to report. This is especially true for mAP, where the cutoff (often reported as top-k or mAP@k [3]) differs hugely across papers – ranging from 50 to 5000 or no cutoff – see for example [10], [11], [16], [17], [18], [23], [24], [37], [40]. These inconsistencies across the literature effectively void model performance comparisons as different cutoffs result in different scores. He et al. [27] report an overestimation even when using a high cutoff value of 5000 compared to the true mAP. Furthermore, the authors point out a significant issue with the mAP formulation found throughout the literature: it is not permutation-agnostic [27], [42]. To clarify the issue, we take a look at the formula for calculating mAP, which is given in the literature [1], [2], [3]. To avoid confusion with the hash code length k, here the cutoff value is denoted as z with 1 ≤ z ≤ |R|: |Q|
1 X mAP@z = AP@z(qj ), |Q| j=1 where z > 0 the cutoff value
(12)
or z = |R| without cutoff, and qj ∈ Q elements of the query set. The mAP is the samples’ Average Precision (AP) value at the specified cutoff, averaged over all samples in the query set. The AP for one sample q ∈ Q can be calculated as follows: Pz P@i(q) · δ(q, ri ) Pz AP@z(q) = i=1 , i=1 δ(q, ri ) i
where P@i(q) = and δ(q, r) =
1X δ(q, rt ) the top-i retrieval precision, i t=1 ( 1 if samples q and r are label-similar, 0
otherwise,
and ri ∈ R the ith retrieved sample.
(13)
The δ function can be defined more formal for two samples xi and xj with labels li and lj as follows: ( ( 1 if Sij > 0, 1 if li · lj > 0, δ(xi , xj ) = = (14) 0 if Sij = 0, 0 if li · lj = 0. The issue with this definition of AP lies in the ordering of the retrieval samples ri . In the context of DSH, each sample gets assigned a discrete hash code within the Hamming space, thus, distances between samples are also discrete: (m ) (m ) d bi i , bj j ∈ {0, . . . , k}. Since the set of possible distances has exactly k + 1 elements, but in practice |R| > k + 1, there must exist at least two samples ri , rj with i ̸= j having the same Hamming distance from a given query sample. Then, there is no semantic ordering among ri and rj , thus, it is undefined which of the two samples appears first in the retrieval process. This ambiguity makes the common formulation of AP susceptible to randomness and poses the possibility of maliciously adjusting the retrieval sequence to achieve higher scores. To demonstrate this, let’s consider the following three semantically-equivalent retrieval sequences for a single query depicted in Figure 4: d=0
d=1
d=2
Seq. 1:
⊕ ⊕ ⊖
⊕ ⊖ ⊖
⊕ ⊖
AP = 0.830
Seq. 2:
⊖ ⊕ ⊕
⊖ ⊖ ⊕
⊖ ⊕
AP = 0.542
Seq. 3:
⊕ ⊖ ⊕
⊖ ⊕ ⊖
⊕ ⊖
AP = 0.710
Fig. 4. Demonstration of different retrieval sequences that are semantically equivalent, but result in significantly different Average Precision values. Retrieval samples that are similar to the query are represented as ⊕, and dissimilar retrieval samples as ⊖. The example is inspired by He et al. [27].
All three sequences are equivalent from a semantics standpoint, i.e., the sets defined by the Hamming distance to the query sample contain the same retrieval samples and are, therefore, identical. The only difference between the sequences is their internal ordering of samples within each Hamming distance set. Nevertheless, when calculating the AP using Equation (13), the results ranging from 0.542 to 0.830 are vastly different. To address this issue, He et al. [27] proposed a tie-aware calculation of the mAP metric for the field of DSH, which is the application of the formulations of McSherry and Najork [42]. Based on both [27] and [42], we define the tie-aware Average Precision as APT using our notation: 0 if nd = 0, + Nd k + if nd = 1, 1 X nd · N d APT (q) = + + + nd −1 N n −1 + d=0 nd X 1 + Nd−1 + t · ndd −1 o/w, n 1 + t + Nd−1 d t=0 P|R| where N + = i=1 δ(q, ri ) the number of positive samples, and nd = |{ri | d(q, ri ) = d}|, and n+ d = |{ri | d(q, ri ) = d ∧ δ(q, ri ) = 1}|, Pd and Nd = j=0 nj the cumulative sum over nd , Pd + and Nd+ = j=0 n+ j the cumulative sum over nd .
(15)
BAUER et al.: DSCH-LOSS: A DYNAMIC SEMANTIC CHANNEL OBJECTIVE FOR DEEP SEMANTIC HASHING
In Equation (15), nd is the number of samples r ∈ R that have a Hamming distance of d from the query q, and n+ d denotes the number of similar (positive) samples r ∈ R that have said Hamming distance of d from the query q. As expected, all sequences from the example in Figure 4 yield the same score of APT = 0.691, which falls between the limits set by the AP scores for Sequence 1 and Sequence 2. When using pre-shuffled datasets, the discrepancies between the tie-oblivious AP and the tie-aware APT scores is small, since a random permutation is likely to represent the average case. This is also why the reported tie-aware score is close to the tie-oblivious score of Sequence 3 in Figure 4 (0.691 vs. 0.710). The tie-aware formula for Mean Average Precision averages, analogously to Equation (12), the tie-aware AP scores over all samples in the query set:
9
fixed hash code length of k = 32 and the scores are calculated on the validation set. We conducted experiments on a total of seven possible values for γw . Since the tie-aware mAP scores are very similar for many hyperparameter values, no clear optimal γw could be found. Indeed, Table VII shows the top-3 performances per category highlighted and setting the hyperparameter to 6, 8, or 14 seems arbitrary. Since other precision-based metrics have shown similar inconclusiveness, we opted to report the ROC-AUC-score for these experiments as well in Table VIII. There, it is evident that γw = 8 is the optimal choice for this hyperparameter, which is the reason behind using this value in all other experiments and in Table II. So far, the experiments were conducted on the dual-stream TDSRDH model architecture. In a third experimental setting, models following the CLIPHash model architecture were trained with all three loss functions and a fixed hash code |Q| 1 X mAPT = APT (qj ). (16) length of k = 32. Again, both datasets were considered and |Q| j=1 when using DSCH, we set the hyperparameter γw = 8. The tie-aware mAP scores on the test set are reported in Table IX In our reference implementation, we provide the tie-aware for both model architectures to allow for easier comparison. mAP score calculation in two variants. The first implementation Similar to the first experimental setting, models trained on the is closely following Equations (15) and (16), whereas the proposed loss DSCH performed – with exception to the T→T second one is an optimized and vectorized version partly based task – consistently and considerably better than models trained on the code provided by He et al. [27]. on the reference losses SCH [18] and TDSRDH-loss [14], supporting our claim of improved hash code learning using V. E XPERIMENTS , R ESULTS AND D ISCUSSION the dynamic semantic channel width approach. Furthermore, In the first experimental setting, models based on the the results in Table IX clearly show that the larger CLIPHash TDSRDH model architecture are trained on the proposed DSCH models are significantly more capable of generating good hash objective as well as two reference loss functions: SCH [18] and codes than the smaller TDSRDH models regardless of training TDSRDH-loss [14]. To further demonstrate the effectiveness of loss function. This behavior indicates the possibility of better DSCH, experiments were conducted on both datasets and a total hash code generation by further optimizing and tailoring model of four different output hash code lengths k ∈ {16, 32, 64, 128}. architectures in a future work. Following common notation, we denote a retrieval task with query modality Mq and retrieval modality Mr as Mq → Mr . VI. C ONCLUSION AND F UTURE W ORK We report the tie-aware mAP scores on all four possible In this paper, we have proposed a new loss function in the retrieval tasks, i.e., the two cross-modal retrieval tasks I→T and T → I as well as the two single-modal retrieval tasks field of Deep Semantic Hashing for the tasks of cross-modal I→ I and T→T. The results of this first experimental setting as well as intra-modal retrieval. The proposed loss function, on the test query and retrieval sets are reported in Table VI. Dynamic Semantic Channel Hashing, extends prior work by They show a clear improvement in achieved scores of our providing a continuous loss landscape and a dynamic target proposed loss DSCH over the two reference losses in almost sematic channel width in the shared Hamming space dependent every scenario. Especially in the cross-modal retrieval tasks, on label similarity. The efficacy of DSCH was demonstrated DSCH outperforms SCH and TDSRDH-loss across all hash in various experimental settings comparing our method with code lengths with, in parts, a substantial margin. In a total prior work on both an existing dual-stream model architecture of three out of the 32 scenarios, DSCH performs subpar, and a hybrid-stream model architecture, as well as on two however, the achieved results are only marginally worse than datasets commonly found in literature. These experiments the respective bests. Moreover, the results of DSCH show only showed significantly better tie-aware Mean Average Precision small deviations in many scenarios, indicating improved model scores for all cross-modal tasks as well as the intra-modal stability regardless of initialization randomness compared to retrieval using images. For larger hash code lengths, DSCH the competing losses. In summary, these results demonstrate outperforms its competitors in all four tasks. The metric of the effectiveness of DSCH to guide model training towards tie-aware mAP scores was chosen to avoid evaluation bias discriminative and similarity-preserving hash codes of various emerging from the ambiguity of retrieval ordering, which is lengths and across datasets that can be utilized in both cross- inherent to the DSH domain due to a limited amount of discrete sample-to-sample distances in the shared Hamming space. modal and single-modal tasks. In order to find the optimal value for the channel width curve Nonetheless, there are still many aspects in the DSH domain modifier hyperparameter γw , a second set of experiments was subject to further research. For example, the contrastive learning conducted. Again, the TDSRDH model architecture and both approach employed in the pre-trained CLIPHash model used datasets were considered. The hyperparameter search used a in the experiments may be transferred to the DSH domain.
10
Dataset k
MIR Flickr 25k 64
NUS-WIDE 128
16
32
64
128
I→T
32
TDSRDH SCH DSCH
78.27 ± 0.34 80.31 ± 0.19 81.81 ± 0.04 82.30 ± 0.38 83.73 ± 0.83 85.55 ± 0.23 86.66 ± 0.04 88.06 ± 0.14 84.19 ± 0.37 86.49 ± 0.29 87.92 ± 0.14 88.69 ± 0.10
70.67 ± 0.33 72.60 ± 0.17 73.30 ± 0.20 73.92 ± 0.30 71.89 ± 0.68 74.12 ± 0.35 75.89 ± 0.10 76.89 ± 0.13 73.64 ± 0.16 75.60 ± 0.12 76.96 ± 0.08 78.03 ± 0.06
T→ I
16
TDSRDH SCH DSCH
73.05 ± 0.34 74.79 ± 0.26 76.01 ± 0.05 76.28 ± 0.44 74.28 ± 0.61 75.96 ± 0.43 77.33 ± 0.13 78.93 ± 0.02 74.67 ± 0.07 76.80 ± 0.18 78.62 ± 0.03 79.62 ± 0.23
65.23 ± 0.44 67.12 ± 0.14 67.76 ± 0.13 68.37 ± 0.44 64.32 ± 0.54 66.09 ± 0.47 68.05 ± 0.30 69.10 ± 0.24 65.93 ± 0.16 67.79 ± 0.33 69.08 ± 0.22 70.13 ± 0.22
I→ I
Loss
TDSRDH SCH DSCH
73.63 ± 0.35 75.31 ± 0.24 76.55 ± 0.29 76.73 ± 0.30 76.67 ± 0.68 78.31 ± 0.21 79.65 ± 0.09 81.17 ± 0.09 76.42 ± 0.17 78.52 ± 0.09 80.34 ± 0.14 81.31 ± 0.14
66.14 ± 0.56 68.04 ± 0.08 68.80 ± 0.09 69.39 ± 0.25 66.21 ± 0.71 68.09 ± 0.31 69.81 ± 0.25 70.96 ± 0.21 67.78 ± 0.39 69.24 ± 0.16 70.91 ± 0.27 71.92 ± 0.05
T→T
Task
TABLE VI C OMPARISON OF THREE LOSS FUNCTIONS BY TIE - AWARE MEAN AVERAGE PRECISION ( M AP T ) SCORES ( IN PERCENT ) FOR FOUR TASKS ON THE TEST SET. B EST PERFORMANCE PER TASK , DATASET, AND HASH CODE LENGTH IS HIGHLIGHTED IN BOLDFACE .
TDSRDH SCH DSCH
69.34 ± 0.36 70.73 ± 0.19 71.58 ± 0.14 70.67 ± 1.00 69.16 ± 0.59 70.80 ± 0.35 72.16 ± 0.19 73.52 ± 0.18 68.96 ± 0.02 70.80 ± 0.25 72.69 ± 0.15 73.79 ± 0.37
61.43 ± 0.57 63.19 ± 0.07 63.74 ± 0.15 64.13 ± 0.60 60.25 ± 0.50 62.03 ± 0.34 63.66 ± 0.33 64.60 ± 0.21 61.53 ± 0.28 63.03 ± 0.33 64.52 ± 0.26 65.36 ± 0.27
T→T I→ I T→ I I→T Task
TABLE VII C OMPARISON OF γw HYPERPARAMETER BY TIE - AWARE MEAN AVERAGE PRECISION ( M AP T ) SCORES ( IN PERCENT ) FOR FOUR TASKS ON THE VALIDATION SET. H ASH CODE LENGTH IS k = 32 FOR ALL EXPERIMENTS . T OP -3 PERFORMANCES PER TASK AND DATASET ARE HIGHLIGHTED IN BOLDFACE . γw Dataset
2
4
6
8
10
12
14
MIR Flickr 25k NUS-WIDE
83.67 ± 0.17 75.04 ± 0.08
84.25 ± 0.27 75.69 ± 0.17
84.41 ± 0.10 75.92 ± 0.09
84.53 ± 0.44 75.83 ± 0.17
84.16 ± 0.14 75.92 ± 0.20
84.57 ± 0.12 75.91 ± 0.14
84.43 ± 0.30 75.95 ± 0.13
MIR Flickr 25k NUS-WIDE
74.67 ± 0.07 66.90 ± 0.15
75.19 ± 0.21 67.67 ± 0.30
75.22 ± 0.33 67.92 ± 0.05
75.22 ± 0.23 67.85 ± 0.39
74.98 ± 0.15 67.84 ± 0.24
75.15 ± 0.08 67.93 ± 0.19
75.16 ± 0.37 67.81 ± 0.12
MIR Flickr 25k NUS-WIDE
75.86 ± 0.19 69.03 ± 0.18
76.53 ± 0.43 69.94 ± 0.35
76.64 ± 0.09 70.20 ± 0.28
76.86 ± 0.05 70.14 ± 0.21
76.21 ± 0.27 70.02 ± 0.17
76.60 ± 0.00 70.13 ± 0.57
76.76 ± 0.30 70.21 ± 0.16
MIR Flickr 25k NUS-WIDE
68.84 ± 0.06 62.45 ± 0.35
69.46 ± 0.31 63.43 ± 0.41
69.52 ± 0.23 63.60 ± 0.20
69.60 ± 0.08 63.65 ± 0.38
69.29 ± 0.10 63.51 ± 0.17
69.45 ± 0.05 63.59 ± 0.51
69.67 ± 0.33 63.59 ± 0.15
T→T I→ I T→ I I→T Task
TABLE VIII C OMPARISON OF γw HYPERPARAMETER BY ROC-AUC SCORES ( IN PERCENT ) FOR FOUR TASKS ON THE VALIDATION SET. H ASH CODE LENGTH IS k = 32 FOR ALL EXPERIMENTS . B EST PERFORMANCE PER TASK AND DATASET IS HIGHLIGHTED IN BOLDFACE . γw Dataset
2
4
6
8
10
12
14
MIR Flickr 25k NUS-WIDE
72.15 ± 0.04 79.58 ± 0.14
72.90 ± 0.44 80.94 ± 0.33
73.03 ± 0.16 81.43 ± 0.09
73.40 ± 0.24 81.55 ± 0.17
73.05 ± 0.49 81.28 ± 0.32
73.24 ± 0.19 81.45 ± 0.13
73.30 ± 0.30 81.40 ± 0.10
MIR Flickr 25k NUS-WIDE
71.50 ± 0.23 79.91 ± 0.14
72.25 ± 0.32 81.20 ± 0.30
72.31 ± 0.19 81.64 ± 0.24
72.57 ± 0.25 81.78 ± 0.39
72.08 ± 0.45 81.36 ± 0.16
72.45 ± 0.15 81.66 ± 0.30
72.57 ± 0.28 81.74 ± 0.11
MIR Flickr 25k NUS-WIDE
80.76 ± 0.33 85.21 ± 0.41
81.40 ± 0.37 86.36 ± 0.18
81.81 ± 0.07 86.89 ± 0.13
81.94 ± 0.57 87.12 ± 0.08
81.68 ± 0.13 86.96 ± 0.20
82.08 ± 0.30 86.93 ± 0.03
81.89 ± 0.24 86.93 ± 0.11
MIR Flickr 25k NUS-WIDE
64.71 ± 0.16 74.96 ± 0.35
65.60 ± 0.41 76.45 ± 0.38
65.52 ± 0.25 76.80 ± 0.23
65.94 ± 0.04 76.91 ± 0.33
65.57 ± 0.45 76.55 ± 0.34
65.67 ± 0.09 76.82 ± 0.42
65.84 ± 0.20 76.92 ± 0.11
Albeit the better performance of the CLIPHash model over the TDSRDH model architecture in the conducted experiments, the former is much larger in terms of parameters and number of layers and, thus, has vastly higher inference costs in both compute time and memory. Distillation learning in a teacher-student setting might provide remedy and allow smaller models to achieve similar scores at lower resource consumption. Additionally, the training procedure can be enriched with a wider set of data augmentation techniques, especially for the text domain, which was left untouched by the experiments in this paper. For example, the effect of token masking and random token replacement could be studied. The image data
augmentation pipeline can be extended as well, e.g., with more multi-image non-instance methods involving more than two images. This approach may also include the associated sample labels, which could be fused and/or mixed alongside the images or texts during data augmentation. Future work may look into the generation of adversarial examples during the training procedure for hard negative mining. Another aspect is the analysis of hash code generation robustness and trustworthiness, e.g., by applying explainable AI techniques, and the development of incremental hashing methods.
BAUER et al.: DSCH-LOSS: A DYNAMIC SEMANTIC CHANNEL OBJECTIVE FOR DEEP SEMANTIC HASHING
11
[1] [2]
[3]
[4] [5] [6]
[7]
[8]
[9] [10]
[11]
[12]
[13]
Task
Model Dataset Loss
CLIPHash MIR Flickr 25k NUS-WIDE
TDSRDH MIR Flickr 25k NUS-WIDE
I→T
TDSRDH SCH DSCH
86.36 ± 0.03 74.52 ± 0.22 88.82 ± 0.50 76.53 ± 0.32 89.51 ± 0.32 77.12 ± 0.15
80.31 ± 0.19 72.60 ± 0.17 85.55 ± 0.23 74.12 ± 0.35 86.49 ± 0.29 75.60 ± 0.12
T→ I
TDSRDH SCH DSCH
81.93 ± 0.09 71.17 ± 0.20 82.12 ± 0.48 71.30 ± 0.26 83.25 ± 0.24 72.17 ± 0.16
74.79 ± 0.26 67.12 ± 0.14 75.96 ± 0.43 66.09 ± 0.47 76.80 ± 0.18 67.79 ± 0.33
I→ I
TDSRDH SCH DSCH
81.92 ± 0.13 71.79 ± 0.17 82.63 ± 0.48 72.47 ± 0.40 83.27 ± 0.29 73.10 ± 0.15
75.31 ± 0.24 68.04 ± 0.08 78.31 ± 0.21 68.09 ± 0.31 78.52 ± 0.09 69.24 ± 0.16
T→T
TABLE IX C OMPARISON OF THREE LOSS FUNCTIONS BY TIE - AWARE MEAN AVERAGE PRECISION ( M AP T ) SCORES ( IN PERCENT ) FOR FOUR TASKS ON THE TEST SET. T HE HASH CODE LENGTH IS k = 32 FOR ALL EXPERIMENTS . B EST PERFORMANCE PER MODEL , DATASET, AND TASK IS HIGHLIGHTED IN BOLDFACE .
TDSRDH SCH DSCH
78.05 ± 0.08 68.68 ± 0.15 76.88 ± 0.45 67.84 ± 0.32 77.83 ± 0.18 68.66 ± 0.12
70.73 ± 0.19 63.19 ± 0.07 70.80 ± 0.35 62.03 ± 0.34 70.80 ± 0.25 63.03 ± 0.33
R EFERENCES
[14]
X. Luo et al., “A Survey on Deep Hashing Methods,” ACM Transactions on Knowledge Discovery from Data, vol. 17, no. 1, pp. 1–50, Feb. 28, 2023. DOI: 10.1145/3532624 A. Singh and S. Gupta, “Learning to hash: A comprehensive survey of deep learning-based hashing methods,” Knowledge and Information Systems, vol. 64, no. 10, pp. 2565–2597, Oct. 2022. DOI: 10.1007/ s10115-022-01734-0 T. Wang, F. Li, L. Zhu, J. Li, Z. Zhang, and H. T. Shen, “Cross-Modal Retrieval: A Systematic Review of Methods and Future Directions,” Proceedings of the IEEE, vol. 112, no. 11, pp. 1716–1754, Nov. 2024. DOI: 10.1109/JPROC.2024.3525147 K. Zhou, F. H. Hassan, and G. K. Hoon, “The State of the Art for Cross-Modal Retrieval: A Survey,” IEEE Access, vol. 11, pp. 138 568– 138 589, 2023. DOI: 10.1109/ACCESS.2023.3338548 J. Lu, V. E. Liong, and J. Zhou, “Deep Hashing for Scalable Image Search,” IEEE Transactions on Image Processing, vol. 26, no. 5, pp. 2352–2367, May 2017. DOI: 10.1109/TIP.2017.2678163 Y. Li, D. Wang, and L. Tang, “Robust and Secure Image Fingerprinting Learned by Neural Network,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 30, no. 2, pp. 362–375, Feb. 2020. DOI: 10.1109/TCSVT.2019.2890966 Y. Wang, X. Ou, J. Liang, and Z. Sun, “Deep Semantic Reconstruction Hashing for Similarity Retrieval,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 31, no. 1, pp. 387–400, Jan. 2021. DOI: 10.1109/TCSVT.2020.2974768 Y. Chen, S. Zhang, F. Liu, Z. Chang, M. Ye, and Z. Qi, “TransHash: Transformer-based Hamming Hashing for Efficient Image Retrieval,” in Proceedings of the 2022 International Conference on Multimedia Retrieval, Newark NJ USA: ACM, Jun. 27, 2022, pp. 127–136. DOI: 10.1145/3512527.3531405 R. Xuan, J. Shim, and S.-G. Lee, “Deep Semantic Hashing Using Pairwise Labels,” IEEE Access, vol. 9, pp. 91 934–91 949, 2021. DOI: 10.1109/ACCESS.2021.3092150 Q.-Y. Jiang and W.-J. Li, “Deep Cross-Modal Hashing,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI: IEEE, Jul. 2017, pp. 3270–3278. DOI: 10.1109/CVPR. 2017.348 J. Hong and H. Liu, “Deep Cross-modal Hashing Retrieval Based on Semantics Preserving and Vision Transformer,” in Proceedings of the 2022 6th International Conference on Electronic Information Technology and Computer Engineering, Xiamen China: ACM, Oct. 21, 2022, pp. 52–57. DOI: 10.1145/3573428.3573439 Q. Li, L. Ma, Z. Jiang, M. Li, and B. Jin, “TECMH: TransformerBased Cross-Modal Hashing For Fine-Grained Image-Text Retrieval,” Computers, Materials & Continua, vol. 75, no. 2, pp. 3713–3728, 2023. DOI: 10.32604/cmc.2023.037463 L. Jin, Z. Li, and J. Tang, “Deep Semantic Multimodal Hashing Network for Scalable Image-Text and Video-Text Retrievals,” IEEE Transactions on Neural Networks and Learning Systems, vol. 34, no. 4, pp. 1838–1851, Apr. 2023. DOI: 10.1109/TNNLS.2020.2997020
[15] [16]
[17]
[18]
[19] [20]
[21]
[22] [23]
[24]
[25] [26]
[27] [28]
S. Zhou et al., “Transformer-Based Discriminative and Strong Representation Deep Hashing for Cross-Modal Retrieval,” IEEE Access, vol. 11, pp. 140 041–140 055, 2023. DOI: 10.1109/ACCESS.2023. 3339581 X. Xia, G. Dong, F. Li, L. Zhu, and X. Ying, “When CLIP meets crossmodal hashing retrieval: A new strong baseline,” Information Fusion, vol. 100, p. 101 968, Dec. 2023. DOI: 10.1016/j.inffus.2023.101968 Q. Huang, S. Peng, X. Shen, Y.-H. Yuan, and S. Pan, “Similarity Preserving Transformer Cross-Modal Hashing for Video-Text Retrieval,” in Proceedings of the 32nd ACM International Conference on Multimedia, Melbourne VIC Australia: ACM, Oct. 28, 2024, pp. 5883–5891. DOI: 10.1145/3664647.3681606 J. Li, W. K. Wong, L. Jiang, X. Fang, S. Xie, and Y. Xu, “CKDH: CLIPBased Knowledge Distillation Hashing for Cross-Modal Retrieval,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 7, pp. 6530–6541, Jul. 2024. DOI: 10.1109/TCSVT.2024. 3350695 Z. Hu, Y.-M. Cheung, M. Li, and W. Lan, “Cross-Modal Hashing Method With Properties of Hamming Space: A New Perspective,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 12, pp. 7636–7650, Dec. 2024. DOI: 10.1109/TPAMI.2024.3392763 J. Zhu et al. “CLIP Multi-modal Hashing for Multimedia Retrieval.” arXiv: 2410.07783 [cs]. Y. LeCun, K. Kavukcuoglu, and C. Farabet, “Convolutional networks and applications in vision,” in Proceedings of 2010 IEEE International Symposium on Circuits and Systems, Paris, France: IEEE, May 2010, pp. 253–256. DOI: 10.1109/ISCAS.2010.5537907 A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” in Advances in Neural Information Processing Systems, vol. 25, Curran Associates, Inc., 2012. K. He, X. Zhang, S. Ren, and J. Sun. “Deep Residual Learning for Image Recognition.” arXiv: 1512.03385 [cs]. C. Deng, Z. Chen, X. Liu, X. Gao, and D. Tao, “Triplet-Based Deep Hashing Network for Cross-Modal Retrieval,” IEEE Transactions on Image Processing, vol. 27, no. 8, pp. 3893–3903, Aug. 2018. DOI: 10.1109/TIP.2018.2821921 L. Mingyong, L. Yewen, G. Mingyuan, and M. Longfei, “CLIP-based fusion-modal reconstructing hashing for large-scale unsupervised crossmodal retrieval,” International Journal of Multimedia Information Retrieval, vol. 12, no. 1, p. 2, Jun. 2023. DOI: 10.1007/s13735-02300268-7 A. Vaswani et al. “Attention Is All You Need.” arXiv: 1706.03762 [cs]. Supriyono, A. P. Wibawa, Suyono, and F. Kurniawan, “Advancements in natural language processing: Implications, challenges, and future directions,” Telematics and Informatics Reports, vol. 16, p. 100 173, Dec. 2024. DOI: 10.1016/j.teler.2024.100173 K. He, F. Cakir, S. A. Bargal, and S. Sclaroff. “Hashing as Tie-Aware Learning to Rank.” arXiv: 1705.08562 [stat]. T.-S. Chua, J. Tang, R. Hong, H. Li, Z. Luo, and Y. Zheng, “NUSWIDE: A real-world web image database from National University
12
[29]
[30]
[31] [32] [33] [34] [35] [36]
[37]
[38]
[39] [40] [41]
[42]
of Singapore,” in Proceedings of the ACM International Conference on Image and Video Retrieval, Santorini, Fira Greece: ACM, Jul. 8, 2009, pp. 1–9. DOI: 10.1145/1646396.1646452 M. J. Huiskes and M. S. Lew, “The MIR flickr retrieval evaluation,” in Proceedings of the 1st ACM International Conference on Multimedia Information Retrieval, Vancouver British Columbia Canada: ACM, Oct. 30, 2008, pp. 39–43. DOI: 10.1145/1460096.1460104 K. Sechidis, G. Tsoumakas, and I. Vlahavas, “On the Stratification of Multi-label Data,” in Machine Learning and Knowledge Discovery in Databases, D. Gunopulos, T. Hofmann, D. Malerba, and M. Vazirgiannis, Eds., vol. 6913, Berlin, Heidelberg: Springer Berlin Heidelberg, 2011, pp. 145–158. DOI: 10.1007/978-3-642-23808-6 10 T. Mikolov, K. Chen, G. Corrado, and J. Dean. “Efficient Estimation of Word Representations in Vector Space.” version 3. mammothb and W. Garbe, Symspellpy: Python SymSpell, version 6.9.0, Mar. 9, 2025. A. Radford et al., “Learning Transferable Visual Models From Natural Language Supervision,” Proceedings of the 38th International Conference on Machine Learning, vol. 139, pp. 8748–8763, Jul. 2021. A. Fang, A. M. Jose, A. Jain, L. Schmidt, A. Toshev, and V. Shankar. “Data Filtering Networks.” arXiv: 2309.17425 [cs]. G. Ilharco et al., OpenCLIP, version 3.1, Zenodo, Jul. 28, 2021. DOI: 10.5281/ZENODO.5143773 Y. Zhuo, Y. Li, J. Hsiao, C. Ho, and B. Li, “CLIP4Hashing: Unsupervised Deep Hashing for Cross-Modal Video-Text Retrieval,” in Proceedings of the 2022 International Conference on Multimedia Retrieval, ser. ICMR ’22, New York, NY, USA: Association for Computing Machinery, Jun. 27, 2022, pp. 158–166. DOI: 10.1145/ 3512527.3531381 M. Li, L. Ma, Y. Li, and M. Ge, “CCAH: A CLIP-Based Cycle Alignment Hashing Method for Unsupervised Vision-Text Retrieval,” International Journal of Intelligent Systems, vol. 2023, no. 1, p. 7 992 047, 2023. DOI: 10.1155/2023/7992047 L. Cao, H. Xiao, W. Song, and H. Li, “A Highly Efficient ZeroShot Cross-Modal Hashing Method Based on CLIP,” in 2024 5th International Seminar on Artificial Intelligence, Networking and Information Technology (AINIT), Mar. 2024, pp. 868–873. DOI: 10. 1109/AINIT61980.2024.10581810 D. P. Kingma and J. Ba. “Adam: A Method for Stochastic Optimization.” arXiv: 1412.6980 [cs]. Z. Qiu, Y. Pan, T. Yao, and T. Mei. “Deep Semantic Hashing with Generative Adversarial Networks.” arXiv: 1804.08275 [cs]. M. Xu, S. Yoon, A. Fuentes, and D. S. Park, “A Comprehensive Survey of Image Augmentation Techniques for Deep Learning,” Pattern Recognition, vol. 137, p. 109 347, May 2023. DOI: 10.1016/j.patcog. 2023.109347 F. McSherry and M. Najork, “Computing Information Retrieval Performance Measures Efficiently in the Presence of Tied Scores,” in Advances in Information Retrieval, C. Macdonald, I. Ounis, V. Plachouras, I. Ruthven, and R. W. White, Eds., vol. 4956, Berlin, Heidelberg: Springer Berlin Heidelberg, 2008, pp. 414–421. DOI: 10.1007/978-3-540-78646-7 38