ReLeVAnT: Relevance Lexical Vectors for Accurate Legal Text Classification Ishaan Gakhar Perssonify Levi & Korsinsky [email protected]
Harsh Nandwani Perssonify Levi & Korsinsky [email protected]
arXiv:2604.22292v1 [cs.CL] 24 Apr 2026
Abstract
2017) and training data curation for larger ML systems (Chalkidis et al., 2019) (Zhong et al., 2020). Most prevalent methods have limitations with respect to the type of documents, computational cost, or assumptions of access to metadata. Where works like Undavia et al. (2018) achieve 72.4% accuracy in the classification of SCOTUS cases into legal issues, their framework is topic-based and not fact-based, resulting in text being required in opinion formats. Similarly, Li et al. (2025) achieves impressive results on the mentioned dataset with remarkable speed; the proposed method requires indicative filenames and fails to handle out-of-scope documents, proving unsuitable for noisy naming conventions, often encountered when dealing with massive corpora. A substantial improvement of upto 20.6% in the accuracy of long legal document classification by Limsopatham (2021), however, the computational cost is very high and makes it infeasible for larger scales of data as applicable to large law firms. Although studies like de Queiroz Santos Filho et al. (2025), Wang et al. (2022), Watson et al. (2023) present interesting findings, all suffer from the requirement of metadata with the corpus of documents. In real-world scenarios, even major law firms do not possess verified, well-labelled metadata, which narrows the scope of applicability. To counter these limitations, the authors model classification as a discriminative task driven by phrase-level signals. This is centred on the idea that relevance in legal corpora can be determined by the presence (or absence) of highly indicative phrases (Ashley, 2017) (Fig. 1), rather than a broad similarity across the document. By explicitly modelling contrastive phrase-level signals, ReLeVAnT captures lexical markers that reliably and consistently distinguish court filings from non-filing documents, while avoiding reliance on metadata, document structure, or expensive representation learning, often facilitated by LLMs.
The classification of legal documents from an unstructured data corpus has several crucial applications in downstream tasks. Documents relevant to court filings are key in use cases such as drafting motions, memos, and outlines, as well as in tasks like docket summarisation, retrieval systems, and training data curation. Current methods classify based on provided metadata, LLM-extracted metadata, or multimodal methods. These methods depend on structured data, metadata, and extensive computational power. This task is approached from a perspective of leveraging discriminative features in the documents between classes. The authors propose ReLeVAnT, a framework for legal document binary classification. ReLeVAnT utilises n-gram processing, contrastive score matching, and a shallow neural network as the primary drivers for discriminative classification. It leverages one-time keyword extraction per corpus, followed by a shallow classifier to swiftly and reliably classify documents with 99.3% accuracy and 98.7% F1 score on the LexGLUE dataset.
1
Introduction
Binary classification of legal documents based on relevance is a multi-faceted task. Keeping downstream tasks in mind, the authors approach this problem statement from a robustness, coverage and computationally inexpensive perspective. Here, defining ‘relevance’ is key; within the scope of this work, it is considered to be linked to court filings and relatedness to proceedings. These relevant documents can be leveraged in the legal domain for a vast number of purposes, including legal workflow automation (Grossman and Cormack, 2010), docket summarization (Bhattacharya et al., 2021) (Saravanan et al., 2008), retrieval systems (Chalkidis et al., 2022) (Pipitone and Alami, 2024), litigation surveillance (Ashley, 2017) (Katz et al., 1
2
Related Works
2.1
Classical Text Classification & Retrieval Methods
Heuristics central to ReLeVAnT are grounded in proven foundations in the Language Modelling space. Term-Frequency signals for document analysis were popularised by (Sparck Jones, 1972). The key idea of combining term frequency and term rarity is central to ReLeVAnT as well; however, TF-IDF weighs terms globally, whereas the proposed method explicitly contrasts classes. Moreover, classical TF-IDF weighting increases with term frequency but does not explicitly model diminishing returns for repeated occurrences of the same term, which is especially important in longer documents, especially in the legal domain. Best Matching 25 (BM25, Robertson and Zaragoza (2009)) is an effective, keyword-based ranking algorithm used in search engines and information retrieval systems to rank documents by relevance to a query. As a descendant of TFIDF, it improves ranking by calculating term frequency (TF) and inverse document frequency (IDF) while normalising for document length. Like ReLeVAnT, BM25 analyses relevance based on lexical matching and leverages term-frequency importance for scoring. Here, relevance is determined by a query, whereas ReLeVAnT implicitly models classdependent relevance and does not require external queries. The work of Cortes and Vapnik (1995) in developing Support Vector Machines (SVMs) has allowed for consistent and reliable classification of high-dimensional features. Like SVMs, ReLeVAnT assumes linear separability in data, but instead of learning weights implicitly, it pre-selects discriminative features, referred to as keywords, explained in Section 3. This feature selection prior to classification reduces noise and allows for interpretability. In recent literature, contrastive learning is generally used to learn common, semantically meaningful features across modalities (Radford et al., 2021) or to learn via the distinction in data and noise, as leveraged in Noise Contrastive Estimation (NCE), Matsuda et al. (2021)). Although NCE assumes similar learning schemes to differentiate between meaningful data and noise, it trains Energy-Based Models (EBMs) probabilistically. In contrast, ReLeVAnT uses contrastive scoring as a lightweight heuristic to isolate discriminative phrases.
Figure 1: An example of the keywords found in excerpts of relevant and irrelevant documents. The ‘[CITE]’ placeholder is left behind during clause filtering. The relevant document has stronger signals of relevance appearing more often than the irrelevant document.
ReLeVAnT is a discriminative phrase-driven classification framework for legal documents that leverages contrastive scoring to extract phrases strongly associated with relevant filings. These signals are combined with normalised lexical features and entity filtering, then classified by a shallow neural network, yielding a cost-efficient, structureindependent alternative to computationally intensive, time-intensive embedding and LLM-based methods. The contributions of this work can be summarised as: • Introducing a novel, lightweight, structure and metadata independent legal document classifier. • Modelling relevance classification as a contrastive phrase-signal task, based on document-level and corpus-level frequency. • Demonstrating 98.7% F1 score on the LexGLUE dataset, showcasing considerably better performance at much lesser cost. The rest of the paper is structured as follows: Section 2 discusses related work pertaining to the given problem statement and their contributions and drawbacks. Section 3 describes the proposed methodology in detail, including the pipeline’s intricate design choices. Section 4 details the experimental setup and choice of experiments to validate the architecture. Section 5 demonstrates the impressive results and justifies the choices of hyperparameters and configuration. Section 6 discusses the potential extension of this work and further avenues to be taken to advance the domain of legal research. 2
To summarise, the central assumptions and concepts that underpin ReLeVAnT are inspired by established work in the Language Modelling space. However, at the time of writing this paper, no publicly available method implements explicit discriminative filtering to align lexical triggers with legal relevance, while effectively learning features that encode discrimination for classification. 2.2
makes them rigid and unsuitable for real-world scenarios with noisy firm data (e.g., irregular naming conventions and unstructured data). Notably, filename signals (Li et al., 2025) were used in conjunction with TF-IDF and lightweight models. This allowed for a lightweight, cheap solution, similar to ReLeVAnT, but encoding filenames as strong priors ignores document content and fails on noisy names. This limits generalizability across corpora and to unseen data, while reducing deployability in real-world settings. Similarly, the approach proposed by Sebastiani (2002) assumes that clean metadata on authors, keywords, publication venues, and tags are provided. This allows for quicker processing and faster classification, but can be inconsistent when metadata is missing or noisy, as is often the case in legal data (Ismaylovna, 2024). ReLeVAnT relies solely on document text, avoiding external dependencies on metadata that may be incomplete, inconsistent, noisy, or completely unavailable. Legal judgment classification in the animal protection domain was pioneered by (Watson et al., 2023), which uses domain-specific keywords as signals of relevance, as does the proposed method. However, their method relies on structured judgments and header cues, which are not always available or clean in court filings. Moreover, the narrow domain does not allow for generalisation across the legal corpora. Here, ReLeVAnT demonstrates generalizability beyond domain-specific corpora, is independent of structured headers, and does not rely on the implicit formatting of judgments. While methods like (Wang et al., 2022) model relationships between entities using graphs, they require accurate NER and more document structure than is available in real-world data. The sensitivity to extraction errors and to graph construction highlights the simplicity of ReLeVAnT and its NER filtering, which is merely a lightweight entity removal mechanism.
Legal Relevance and Document Filtering
The problem of identifying relevant documents within large legal corpora has been studied in the context of Technology-Assisted Review (TAR) and the TREC Legal Track (Cormack et al., 2010; Grossman et al., 2011). These works formalise relevance as a function of responsiveness to a given legal matter, typically requiring systems to rank documents based on their probability of relevance. The TREC Legal Track established standardised benchmarks for evaluating large-scale legal document filtering, demonstrating that classical text-based approaches, including retrieval models and shallow classifiers, can achieve strong performance when calibrated appropriately. However, these frameworks are inherently query-driven, requiring a predefined information need or topic description to guide relevance estimation, introducing dependency. Subsequent work by Cormack and Grossman (2015) introduced Continuous Active Learning (CAL), which iteratively refines relevance predictions through human-in-the-loop feedback. While CAL achieves high recall and efficiency in ediscovery settings, it relies on iterative labelling, seed queries, and interactive training workflows, which make it operationally complex. In contrast, ReLeVAnT formulates legal relevance as a fixed classification problem, independent of external queries or iterative feedback. By modelling relevance using discriminative phrase-level signals extracted directly from document text, ReLeVAnT provides a lightweight, fully automated alternative better suited to large, noisy, unstructured, heterogeneous corpora where metadata, queries, or human supervision may be unavailable. 2.3
2.4
Deep Learning-based methods
The advent of deep learning, especially autoregressive methods, has found application in legal text classification. However, a common problem with these methods is the computational and data costs, factors that ReLeVAnT is largely independent of. Recent works like (Limsopatham, 2021) leverage transformers to model contextual relationships and long-range dependencies, both of which are first, not directly necessary for determining relevance,
Metadata and Structure-based Methods
In the legal document classification literature, some of the top-performing methods leverage metadata, structured datasets, and curated features. Although these methods demonstrate impressive results, their reliance on metadata availability and structure 3
and incur high computational load and scalability issues. The classification of SCOTUS documents by Undavia et al. (2018) uses Word2Vec embeddings with autoregressive architectures and effectively captures patterns across tokens, like the n-gram extraction in ReLeVAnT. Learning these dense semantic embeddings requires a large, labelled dataset and optimising the global representation. This poses restrictions on topic-based modelling, as opposed to relevance-based, and is only explored for structured opinions by the court. Instead of extracting global semantic representations, ReLeVAnT directly targets the most discriminative signals across the corpora and models lexical signals which influence legal relevance. The work by Joulin et al. (2017) uses n-grams with a shallow model for text classification. Although the method has proven very effective, it uses character-level n-grams, which carry much less significance than word-level n-grams, especially from the perspective of legal relevance (complaint & compliance, dismiss & dismissal, dismiss & miss). Moreover, the idea of using discriminatory n-grams is novel to our method.
3
malised as: CSM(t) =
ft+ ft+ + (ft− )p + ϵ
(1)
where t is the candidate term (extracted n-gram), ft+ is the total frequency of t across all relevant documents, ft− is the total frequency of t across all irrelevant documents, ϵ is the smoothing constant and p is the penalty exponent applied to negative frequency to control the aggressiveness of punishing terms also in irrelevant docs. Finally, CSM(t) is the contrastive score for t in the range [0, 1], where 1 is highly specific to relevant documents, and 0 is common in irrelevant documents. Using these scores, the document-level frequency is computed. This is formalised as: DF(t) =
rt+ rt+ + rt− + ϵ
(2)
+ represents the fraction of where rt+ = d+ t / N relevant documents containing the term t, rt− = − is the fraction of irrelevant documents d− t / N − containing the term t, d+ t and dt are the number of positive and negative documents containing t, and N + and N − are the total number of positive and negative documents in the corpus. Moreover, a hard filter is implemented to reject terms that appear in any irrelevant document. DF(t) counts the fraction of relevant vs irrelevant documents containing the term t. This captures terms that might appear many times in a few irrelevant documents (inflating the negative frequency), but are actually spread across many relevant documents. Here, it should be noted that CSM(t) does not indicate absence, but rather a lack of lexical signal related to legal relevance in that case. The combined score is computed by averaging the term-level CSM scores and document-level DF scores.
Methodology
This section details the proposed methodology of ReLeVAnT. Building on the idea that discriminatory words are strong signals of relevance, the authors propose a 2-stage pipeline comprising the Keyword Extraction pipeline and the Neural Network-based classification pipeline (CLS). As shown in Fig. 2, the Keyword Extraction pipeline (KE) begins with NER-based filtering (Nadeau and Sekine, 2007). This removes all person names, but preserves tags of locations, clauses and organisations. This is particularly important in longer documents where entity names, such as names of plaintiffs, defendants, petitioners, respondents, appellants, appellees, occur very often, and can affect the keywords due to the extraction being frequency-based, as explained further. The remaining clauses and constitutional references are removed to filter out any residual entities left by NER. Then, n-grams are constructed from these filtered texts. These n-grams are key for representing relevance since the signal "Motion to Dismiss" is stronger than individual words like "Motion" or "Dismiss". With these extracted n-grams, contrastive scores are computed. This can be for-
S(t) = w · CSM (t) + (1 − w) · DF (t)
(3)
where w is the weighing hyperparameter discussed in the next section. This combined how ‘often’ and how ‘widely’ a term t appears in the corpus. This holistic score determines the absolute importance of t across the entire corpus. This combined score is weighted by the frequency of the specific keyword to determine the value of the respective index in the resulting vector. This is expressed as: 4
Figure 2: Illustration of the proposed method. The section in Blue highlights the KE stage, and the section in Pink highlights the CLS stage. The given neural network is only for visual purposes, and its exact architecture is detailed in Section 4.
x⃗j = S((tj ) · count(tj , doc)
Supreme Court opinions, respectively. EUR-LEX and UNFAIR-ToS were labelled as irrelevant due to their nature pertaining to European legislation and Terms of Service clauses, respectively. The CaseHOLD and LEDGAR subsets were excluded from this experiment because they are QA-taskcentric and contract-provision-centric, respectively. This dataset was split into train, val, and test sets, consisting of 29532 documents (201666 pages), 9675 documents (74538 pages), and 9007 documents (85441 pages), respectively. Of these, the train set has a relevant-irrelevant ratio of 47%-53%, the validation set has a ratio of 25%-75%, and the test set has a ratio of 27%-73%. These ratios represent realistic scenarios where law firms often have a lot more irrelevant data due to discovery documents, bills and news reports, which cause an imbalance. However, our method is invariant to class imbalance and achieves excellent performance in both situations, as shown in Table 1. To evaluate performance, accuracy and F1 score is utilised. All experiments are conducted using the LexGLUE dataset on HuggingFace and run on an Intel Core Ultra 9 H-class CPU. As explained in Section 3, eyecite (Cushman et al., 2021) is used to remove clauses and references. Moreover, the SpaCy core-small model is used for entity filtering. A smoothing constant of 0.01 is used in CSM.
(4)
where ⃗xj = count(tj , doc). These scores are used to construct a (sparse) vector per document, which is then used by a neural network to classify documents as relevant or irrelevant. For a given document d and a set of K keywords t1 , t2 , ..., tK , the feature vector is formally defined as: x1 (d) x2 (d) v(d) = . (5) . . xK (d) where each dimension is the raw frequency count xj (d) = count(tj , d). These feature vectors are constructed for each document, and the documents are classified by a simple feed-forward neural network with a sigmoid activation. The construction of these feature vectors using a well-weighted, relevance-focused discriminative frequency paradigm ensures accurate and reliable classification.
4
Experiments
To validate the proposed method, the LexGLUE (Chalkidis et al., 2022) benchmark was used. From the tasks in the dataset, ECtHR A, SCOTUS, EURLEX, and UNFAIR-ToS were used, with ECtHR A and SCOTUS labelled as relevant due to their direct relevance to European court cases and US
ReLeVAnT achieves state-of-the-art (SOTA) performance on the LexGLUE dataset, surpassing methods such as majority class selection, alwayspositive selection, and manually selected keywords, 5
Class Split Realistic Original
Accuracy↑ 99.3 98.9
F1↑ 98.7 98.1
N-gram 2-gram 3-gram 4-gram 5-gram 6-gram 7-gram 15-gram
Table 1: ReLeVAnT’s invariance to class imbalance seen in consistent results between less and more aggressive class imbalance scenarios. The ‘Realistic’ scenario has a relevant-irrelevant class ratio of 47%-53%, whereas the ‘Original’ ratio in the dataset is 19%-81%.
Method Majority selection Always positive selection Manually chosen keywords ReLeVAnT
Accuracy↑ 73.4 26.6 83.7 99.3
F1↑ 0 42.1 73.7 98.7
MTF 10 20 30 50 100 250 400 1000 5000
as shown in Table 2. Each intricacy of ReLeVAnT is supported by thorough experiments and solid intuition. The decision to exclude lemmatisation and stemming as preprocessing methods is explained in Table 3. Similarly, the n-gram range is thoroughly investigated in Table 4, where the best configuration of 4-grams is decided. The choice of Minimum Term Frequency (MTF) is made from extensive evaluation, as evident in Table 5. Here, the optimal choice is at least 30 keywords. Similar experiments are conducted for the penalty exponent (explained in (1)) in Table 6. This choice of 10 is central to the scoring and affects the pipeline’s final performance. The document frequency weight hyperparameter is also investigated to balance the weight between document-level and cross-document presence, as shown in Table 7. Finally, variants of the neural network architecture are selected in Table 8 to determine the optimal configuration. Accuracy↑ 95.8 98.1
Accuracy↑ 97.9 98 98.1 98.1 98.1 98.1 98.1
F1↑ 96 96.3 96.4 96.4 96.4 96.3 96.4
Table 4: Comparison of results of ReLeVAnT across several n-gram ranges. This is a key hyperparameter in the KE pipeline.
Table 2: Comparison of results between ReLeVAnT with and without lemmatisation and Stemming. The −LS suffix indicates ReLeVAnT with lemmatisation and stemming at KE and CLS.
Method ReLeVAnT-LS ReLeVAnT
#keywords 3322 2958 2965 2966 2966 2966 2966
#keywords 3533 3533 3533 2966 1458 449 220 40 3
Accuracy↑ 98.4 98.4 98.4 98.1 97.4 97 96.3 93.4 87.9
F1↑ 97.1 97 97 96.4 95 94.2 92.7 86.5 70.8
Table 5: Comparison of results of ReLeVAnT across MTF thresholds. This is a key hyperparameter in the KE pipeline.
PenEx 1 2 10 50 Infinite
#keywords 3533 3533 3533 3533 3533
Accuracy↑ 98.4 98.4 98.6 98.5 98.1
F1↑ 97 97 97.3 97.1 96.4
Table 6: Comparison of results of ReLeVAnT across various penalty exponent thresholds. This is a key hyperparameter in the KE pipeline.
DocFreq 0.25 0.33 0.5 0.66 0.75 0.9
F1↑ 92.1 96.4
Table 3: Comparison of results between ReLeVAnT with and without lemmatisation and Stemming. The −LS suffix indicates ReLeVAnT with lemmatisation and stemming at KE and CLS.
#keywords 3533 6228 9140 9145 9146 9154
Accuracy↑ 98.1 98.9 99.2 99.3 99.3 98.9
F1↑ 96.4 97.9 98.5 98.6 98.7 98
Table 7: Comparison of results of ReLeVAnT across various document frequency weights. This is a key hyperparameter in the KE pipeline.
6
NN Architecture A1 A2 A3
Accuracy↑ 99.3 98.6 26.6
F1↑ 98.7 97.5 42.1
set of experiments dealt with lemmatisation and stemming. As evident in Table 3, ReLeVAnT without this processing step performs considerably better, by upto 2.6% in accuracy and 4.3% in F1 score. This can be attributed to the narrowing of vocabulary, leading to a loss of semantic meaning in a term-sensitive domain like Law. For example, ‘pleading’ and ‘plead’ is processed as the same word ‘plead’; however, ‘pleading’ may indicate the current state of a case, whereas ‘plead’ could refer to a clause, a reference, or an event in a news report. Both cases differ in relevance due to the precision of the language used, which is affected by lemmatisation and stemming. Table 4 justifies the choice of 4-gram as the optimal choice. As seen here, the performance of ReLeVAnT improves upto the 4-grams, then stagnates near the same value, while the #keywords also remain the same. This choice allows for optimal performance and scalable results. An experiment with 15-grams is also conducted to test for edge cases of very long repeated sentences, which evidently does not affect performance significantly. Table 5 demonstrates optimal performance at MTF 30. Similarly, the performance peaks at 30, with a remarkable accuracy of 98.4% and an F1 score of 97%, then diminishes as the MTF threshold rises. This is believed to be due to crucial keywords being missed that may be more ‘rare’ and convey important relevance signals, but may not pass the strict threshold. Hence, an MTF of 30 is chosen. Table 6 points to the choice of the penalty exponent in the CSM modelling, as explained in Eq. 1. Here, a higher penalty exponent points to stricter ‘punishment’ for keywords that occur in negative documents as well. Tuning this to 10 produces the best results, as lower values assign higher scores to keywords that are more prevalent in irrelevant documents, while higher values are ‘too strict’. With this configuration, 98.6% accuracy and 97.3% F1 score are obtained. Table 7 showcases the best performance of ReLeVAnT at 0.75 weight. This indicates that preference for in-document frequency results in better performance over frequency across documents. This reveals an interesting finding: local frequency is more influential than global frequency across the corpus. Table 8 details the architectures explored and their corresponding results. The structure of 512256-128-64-1 performs considerably better, show-
Table 8: Comparison of results of ReLeVAnT across various NN models. This is a key factor in the CLS pipeline. The architectures in #nodes for each of A1, A2, and A3 are [512-256-128-64-1], [1024-512-256128-64-32-1], and [2048-256-64-1].
Vector Config Top 3 keywords Random vectors ReLeVAnT-K ReLeVAnT
Accuracy↑ 87.9 73.4 98.4 99.3
F1↑ 70.8 0 97 98.7
Table 9: Comparison of results of ReLeVAnT across specific vector configurations. These experiments are crucial to cross-validate our findings. ReLeVAnT-K refers to the entire keyword list except the top 3 keywords.
All networks used ReLU for the hidden layers and sigmoid for the output layer (threshold 0.4), with a maximum of 500 iterations, an adaptive learning rate, and early stopping. To cross-check the validity of the remarkable results, additional experiments were conducted to isolate the highest importance keywords (based on frequency). To assess performance with only the top 3 keywords (‘remanded’, ‘testimony’, ‘congressional’) and without exactly those 3, and to crosscheck whether the neural network is only learning the underlying bias, an experiment with random feature vectors was conducted. These results are evident in Table 9.
5
Results
Experimental results reveal the impressive performance of ReLeVAnT relative to baselines such as majority-class prediction, always-positive selection, and manual keyword selection, as shown in Table 2. ReLeVAnT also demonstrates robustness against the relevant class in cases of class imbalance, as shown in Table 1. Here, a difference of 0.4% in accuracy and 0.6% in F1 score was observed. This is because the discriminative nature of the keywords selected ensures robustness against imbalances due to near independence of negative frequency at CSM, as explained in Eq. 1. To explore the scope of the vocabulary, the first 7
casing 99.3% accuracy and 98.7% F1. A2 and A3 are deeper and shallower networks, respectively, which can lead to overfitting or the curse of dimensionality, significantly reducing performance. To investigate dependency on top keywords, the authors isolated the top 3 most influential keywords. The exact same setup with just 3 demonstrates passable performance of 87.9% accuracy and 70.8% F1 score. This supports our initial assumption of discriminatory words (in this case, ‘remanded’, ‘testimony’, and ‘congressional’) being a strong, separable signal of legal relevance. ‘Remanded’ is present in higher court opinions, a ‘testimony’ is evidence given by a witness under oath, and ‘congressional’ is related to the Congress of the US. All three are intuitively strong markers that would be present only in documents relevant to law. Moreover, removing these 3 to assess the generalizability of the proposed model reveals that the neural network can compensate for the lack of very strong signals and performs very similarly to when the entire keyword list is presented, with only a 0.9% drop in accuracy and a 1.7% drop in F1 score. To assess the neural network’s randomness, random feature vectors were generated for each document, yielding an exact class split accuracy of 0 and an F1 score of 0. These results further reinforce our hypothesis about the legal relevance and its degree, as measured by the frequency of discriminatory signals.
6
Levi for his valuable inputs and guidance.
References Kevin D Ashley. 2017. Artificial intelligence and legal analytics: new tools for law practice in the digital age. Cambridge University Press. Paheli Bhattacharya, Soham Poddar, Koustav Rudra, Kripabandhu Ghosh, and Saptarshi Ghosh. 2021. Incorporating domain knowledge for extractive summarization of legal case documents. In Proceedings of the eighteenth international conference on artificial intelligence and law, pages 22–31. Ilias Chalkidis, Emmanouil Fergadiotis, Prodromos Malakasiotis, and Ion Androutsopoulos. 2019. Largescale multi-label text classification on eu legislation. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 6314– 6322. Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Katz, and Nikolaos Aletras. 2022. Lexglue: A benchmark dataset for legal language understanding in english. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4310–4330. Gordon V. Cormack and Maura R. Grossman. 2015. Autonomy and reliability of continuous active learning for technology-assisted review. arXiv preprint arXiv:1504.06868. Gordon V. Cormack, Maura R. Grossman, Bruce Hedin, and Douglas W. Oard. 2010. Overview of the trec 2010 legal track. In Proceedings of the Text REtrieval Conference (TREC). National Institute of Standards and Technology (NIST).
Conclusion and Future Works
This work introduced ReLeVAnT, a lightweight, contrastive, phrase-driven framework that leverages a shallow classifier to achieve state-of-the-art results on binary classification tasks for court filings: relevant vs irrelevant documents. It is completely independent of metadata, filenames, document and directory structure, allowing for massive coverage and robustness at an exponentially lower cost than other methods in the legal text field. Future work includes multiclass and multilabel classification, as well as downstream tasks. This would enable broader use while tuning the granularity of tasks to find applications across more verticals, possibly outside of law.
7
Corinna Cortes and Vladimir Vapnik. 1995. Supportvector networks. Machine learning, 20(3):273–297. Jack Cushman, Matthew Dahl, and Michael Lissner. 2021. eyecite: A tool for parsing legal citations. Journal of Open Source Software, 6(66):3617. José Jorge de Queiroz Santos Filho, Filipe Araújo Dantas, Melquezedeque da Silva Lima, Shirley Barbosa dos Santos, Galileu Genesis, Maria Gabriely Lima da Salva, Álvaro Farias Pinheiro, and Eraylson Galdino da Silva. 2025. Comparing machine learning and an expert system for legal document classification. In Conference on Digital Government Research, volume 26. Maura R Grossman and Gordon V Cormack. 2010. Technology-assisted review in e-discovery can be more effective and more efficient than exhaustive manual review. Rich. JL & Tech., 17:1.
Acknowledgment
We extend our gratitude to Perssonify for allocating resources that made this work possible. Moreover, we thank Levi & Korsinsky, LLP for their continued support of our work, and particularly Joseph E.
Maura R. Grossman, Gordon V. Cormack, Bruce Hedin, and Douglas W. Oard. 2011. Overview of the trec 2011 legal track. In Proceedings of the Text REtrieval
8
Conference (TREC). National Institute of Standards and Technology (NIST).
Karen Sparck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 28(1):11–21.
BJ Ismaylovna. 2024. Problems of admissibility and reliability of metadata as evidence. international journal of. Law and Policy, 2(8):1.
Samir Undavia, Adam Meyers, and John E Ortega. 2018. A comparative study of classifying legal documents with neural networks. In 2018 Federated conference on computer science and information systems (FedCSIS), pages 515–522. IEEE.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomáš Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th conference of the European chapter of the association for computational linguistics: volume 2, short papers, pages 427–431.
Qiqi Wang, Kaiqi Zhao, Robert Amor, Benjamin Liu, and Ruofan Wang. 2022. D2gclf: Document-tograph classifier for legal document classification. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2208–2221.
Daniel Martin Katz, Michael J Bommarito II, and Josh Blackman. 2017. A general approach for predicting the behavior of the supreme court of the united states. PloS one, 12(4):e0174698.
Joe Watson, Guy Aglionby, and Samuel March. 2023. Using machine learning to create a repository of judgments concerning a new practice area: a case study in animal protection law. Artificial Intelligence and Law, 31(2):293–324.
Zhijian Li, Stefan Larson, and Kevin Leach. 2025. Document classification using file names. In Proceedings of the 2025 ACM Symposium on Document Engineering, pages 1–10.
Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020. How does nlp benefit legal system: A summary of legal artificial intelligence. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 5218–5230.
Nut Limsopatham. 2021. Effectively leveraging bert for legal document classification. In Proceedings of the natural legal language processing workshop 2021, pages 210–216. Takeru Matsuda, Masatoshi Uehara, and Aapo Hyvarinen. 2021. Information criteria for non-normalized models. Journal of Machine Learning Research, 22(158):1–33. David Nadeau and Satoshi Sekine. 2007. A survey of named entity recognition and classification. Lingvisticae Investigationes, 30(1):3–26. Nicholas Pipitone and Ghita Houir Alami. 2024. Legalbench-rag: A benchmark for retrievalaugmented generation in the legal domain. arXiv preprint arXiv:2408.10343. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PmLR. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond, volume 4. Now Publishers Inc. Murali Saravanan, Balaraman Ravindran, and S Raman. 2008. Automatic identification of rhetorical roles using conditional random fields for legal document summarization. In Proceedings of the Third International Joint Conference on Natural Language Processing: Volume-I. Fabrizio Sebastiani. 2002. Machine learning in automated text categorization. In ACM Computing Surveys, volume 34, pages 1–47. ACM.
9