A Fine-Tuned BERT Classifier for Personal-Letter Titles in Late-Ming and Early-Qing Collected Works∗ Queenie Luo [email protected] Harvard University
arXiv:2605.23103v1 [cs.CL] 21 May 2026
Abstract
Section 2 places Lepton in three literatures: Classical Chinese pretrained language models, computational text classification, and the CBDB ecosystem. Section 3 frames the central philological question, such as “what counts as a letter title?”, and reports the annotation decisions. Section 4 describes the corpus and training pipeline and evaluates Lepton against regex and TF–IDF baselines on a heldout test set. Section 5 discusses what the model attends to, what it does not do, and where it is likely to fail beyond its training distribution.
I present Lepton (Letter Prediction), a fine-tuned BERT classifier that predicts whether a title in a Classical Chinese wenji 文集 table of contents is a personal letter (書) or a closely confusable preface (序, particularly the farewell-preface 送序). Lepton finetunes bert-base-chinese [7] on 5,438 hand-labeled wenji titles from thirty-three late-Ming and early-Qing literati. I’ve deployed the model1 on Hugging Face and has been used at the China Biographical Database (CBDB) to identify approximately fifty-five thousand letters across mid-Ming through early-Qing wenji, populating the Ming Letter Platform.
2 Background and related work Lepton sits at the intersection of two literatures: pretrained language models for Classical Chinese, and the digital humanities tradition of comparative evaluation on historical text. By Lepton’s deployment (November 2022), three domain-adapted Classical-Chinese backbones were available — AnchiBERT [19], GuwenBERT [14], and SikuBERT / SikuRoBERTa [13]; SikuGPT [4] arrived later. Their published benchmarks target sentence-level tasks (word segmentation, punctuation, NER, POS) rather than title-level genre classification, so none offers a direct point of comparison. I fine-tune bert-base-chinese [7] directly; §4 gives the rationale. The closest digital humanities precedent on Classical Chinese is Broadwell, Chen and Shepard [3], who apply topic modeling and Jensen–Shannon divergence to the Quan Tang shi. Beyond Chinese, work on historical-text classification — Croatian census records [10], Finnish OCR [9], ancient inscriptions [18], eighteenthcentury Austrian newspapers [16] — establishes the comparativeevaluation and annotation-as-construction expectations that Sections 4.3 and 3 are written to meet.
Keywords Classical Chinese, BERT, text classification, digital humanities, China Biographical Database, NLP
1
Introduction
The wenji is the canonical form in which a Chinese literatus’s lifetime output reaches the modern reader. Its table of contents is a curated, genre-mixed list assembled by an editor (often the author’s student, son, or grandson) who decided which texts merited preservation and under what generic heading. For ten thousand wenji, that curated structure becomes the problem: a corpus no individual can read but that contains, in its title-level metadata, exactly the genre signal that would make it tractable. The class of false positives that matters most is not poetry or memorials but prefaces (xu 序), and the genre signal a regex would lean on turns out to be unreliable in both directions. Only 9 % of letter titles in my labeled corpus end with the character 書; the rest end with 啓, with a recipient title, or with no terminal genre marker at all. Letters are identifiable not by their final character but by their opening verb — 與, 答 (and the orthographic variants 荅 and 畣), 復, 報, 寄, 上, 奉 — and these verbs also appear at the head of preface titles. A regex on titles ending in 書 misses 91 % of letters; a regex on the opening verb sweeps up prefaces and requires hand-disambiguation. Lepton (Letter Prediction) is a fine-tuned BERT classifier that predits whether a title denotes a personal letter (書) or a closely confusable preface (序, particularly the farewell-preface 送序). It is finetuned on 3,206 letter titles and 2,232 preface titles from thirtythree late-Ming through early-Qing literati. The model is opensource, deployed on Hugging Face, and has been used at the China Biographical Database (CBDB) to identify approximately fifty-five thousand letters across a corpus of mid-Ming through early-Qing wenji, populating what the team calls the Ming Letter Platform.
3 What counts as a “letter title”? In Classical Chinese collected works (wenji 文集), letters circulate under several near-synonymous genre labels: shu 書 is the central case, with qi 啓, jian 牋 / 箋, tie 帖, and die 牒 as adjacent boundary cases. In the labeled corpus of 3,206 letter titles, the operative signal is at the start of the title rather than the end: only 9.2 % end in 書, while a verb of transmission in the opening position characterises the class (Table 1). The first eight verbs cover 82 % of letter titles, and three of them — 答, 荅, 畣 — are orthographic variants of “reply” that a characterlevel model handles automatically where a regex would have to enumerate each form. The same character-variation pattern runs through the preface side (序 89 %, 敘 3.4 %, 叙 1.8 %, 引 2.6 %). A typical letter title reads [verb] + [recipient], with a terminal genre marker optional. Distinguishing letters from poetry, memorials, or inscriptions is straightforward; the hard case is the preface (xu 序), particularly the occasion-prefaces — songxu 送序, shouxu 壽序, zengxu 贈序
∗
Technical report documenting the Lepton classifier deployed at the China Biographical Database (CBDB), Harvard University, on 4 November 2022. 1 https://huggingface.co/cbdb/ClassicalChineseLetterClassification 1
Luo
Table 1: Opening-verb distribution in the 3,206 letter titles. Opening verb
Force
Count
%
與 答 (荅, 畣) 柬 復 報 又 上 寄 奉 再 Other
“with” (peer-to-peer) reply short note returning a letter reporting back “again” (continuation) submitting upward, to a superior sending at a distance respectfully “again” (re-submission)
1,087 992 163 131 109 86 62 42 35 28 471
33.9 30.9 5.1 4.1 3.4 2.7 1.9 1.3 1.1 0.9 14.7
4.2
4.3
Test set, baselines, and metrics
I stratified the 5,438-row labeled corpus into an 80 / 10 / 10 train / dev / test split (seed=42, class balance preserved). I evaluated:
— which like letters name a recipient and inhabit the same socialnetwork slot in CBDB’s relational schema, but encode a public tribute rather than a one-to-one address. The opening syntax can be near-identical; the discriminating signal is the verb (答 vs. nothing) and the terminal marker (書 vs. 序). The labeled corpus contains 3,206 letter titles and 2,232 preface titles from thirty-three mid-Ming through early-Qing literati, each row carrying CBDB metadata (writer ID, recipient ID where known, relationship code, wenji, juan). Lepton learns the binary shu = 1, xu = 0; §4 details corpus construction.
3.1
Architecture and training
Lepton fine-tunes bert-base-chinese [7] (∼102M parameters) with a BertForSequenceClassification head over the [CLS] token; all parameters are updated. I chose a modern-Chinese rather than a domain-adapted ClassicalChinese backbone for two reasons. Empirically, the discriminating characters — the opening verbs and terminal markers of Table 1 — are well-represented in modern-Chinese pretraining data, so I expected the checkpoint to transfer to a title-level signal carried by shared characters rather than pre-modern syntax. At the time Lepton was trained (2020), bert-base-chinese was the mature widely-available Chinese BERT; domain-adapted alternatives were not yet a low-friction option.
(1) Majority class: predict “letter” for every title. (2) Regex (ends with 書): the naive historian’s baseline. (3) Regex (starts with a verb of transmission): titles whose first character is one of 與, 答, 荅, 畣, 柬, 復, 報, 又, 上, 寄, 奉, 再, 致. (4) Regex disjunction: starts-with-verb OR ends-with-書. (5) TF–IDF character n-grams + logistic regression: with several n-gram ranges, min_df=2, class_weight='balanced'. (6) Lepton (deployed): the Hugging Face checkpoint (finetuned bert-base-chinese).
What Lepton is and is not classifying
I report accuracy, letter-class precision / recall / F1, and macroF1. I also report a confusion matrix because the class imbalance (59 % letter) makes accuracy a misleading single number.
Lepton does not identify recipients, extract dates, distinguish private letters (sishu 私書) from official correspondence (gongshu 公 書), or classify from bodies. Three further limits:
4.4
• Dynastic. Trained on roughly 1550–1700; behaviour on Song or Yuan material is uncharacterised. • Generic. The negative class is the preface, not arbitrary non-letter text; poetry titles and inscriptions are out-ofdistribution. • Architectural. The binary choice is by design: building around the one false-positive class that matters produces a sharper interpretable contrast than a flatter scheme [16].
Results
Three observations: First, the two regex baselines that rely on the terminal 書 marker fail decisively (F1 = 0.166 and 0.150). This confirms Section 3’s observation: a * 書 regex retrieves under one in ten letters. Second, the opening-verb regex reaches F1 = 0.915 with near-perfect precision (0.989). This is the strong historian-withoutML baseline; any neural contribution has to clear this number, not majority class. Adding verb-start OR 書-end raises F1 to 0.920. Third, the TF–IDF character n-gram baselines (ranges (2, 2) through (2, 4)) cluster tightly at F1 = 0.935–0.937 with high recall (≈ 0.99) but lower precision (≈ 0.89), reflecting the same opening-verb signal the regex picks up plus some additional discrimination from short character spans. The deployed Lepton checkpoint attains F1 = 0.984 on the matched test subset — the strongest of any system reported here, with perfect precision on the letter class and a small recall gap that § 5.3 unpacks.
4 Methods and evaluation 4.1 The labeled corpus For each of the thirty-three authors, an annotator located the collection in Lidai bieji ku 歷代別集庫, Siku xilie shujuku 四庫系列數 據庫, or Zhongguo jiben guji ku 中國基本古籍庫 V7.0, opened the table of contents (目錄), identified the juan containing letters and prefaces, and scraped or hand-entered the titles. Each row carries CBDB metadata: writer ID, recipient ID where named, relationship code (e.g. 致書 Y, 答 Y 書), wenji text ID, and juan number. The distribution across authors is uneven — Feng Mengzhen 馮夢禎 (729), Yuan Hongdao 袁宏道 (528), Qian Qianyi 錢謙益 (487), Zou Yuanbiao 鄒元標 (470), and Tu Long 屠隆 (415) account for 47 % of the data.
4.5
Error analysis
Lepton’s six errors on the matched test subset are all of one kind: very short, elliptical letter titles where the opening-verb signal is absent because the title is the recipient’s name or a continuation marker: 2
A Fine-Tuned BERT Classifier for Letter Titles
Table 2: System comparison on the held-out test set. The deployed Lepton row is evaluated on the matched subset for which deployment-time predictions are available. System Majority class Regex: ends with 書 Regex: verb-start AND 書-end Regex: starts with verb of transm. Regex: verb-start OR 書-end TF–IDF char 2-2 + LogReg TF–IDF char 2-3 + LogReg TF–IDF char 2-4 + LogReg Lepton (deployed, fine-tuned bert-base-chinese)
Accuracy
P (letter)
R (letter)
F1 (letter)
Macro-F1
TP / FP / FN / TN
0.590 0.463 0.458 0.906 0.912 0.919 0.921 0.921 0.977
0.590 1.000 1.000 0.989 0.989 0.884 0.888 0.888 1.000
1.000 0.090 0.081 0.851 0.860 0.994 0.991 0.991 0.968
0.742 0.166 0.150 0.915 0.920 0.935 0.937 0.937 0.984
0.371 0.385 0.376 0.905 0.911 0.914 0.916 0.916 0.971
321 / 223 / 0 / 0 29 / 0 / 292 / 223 26 / 0 / 295 / 223 273 / 3 / 48 / 220 276 / 3 / 45 / 220 319 / 42 / 2 / 181 318 / 40 / 3 / 183 318 / 40 / 3 / 183 182 / 0 / 6 / 70
• Mei yan 寐言 (2 chars; Gu Xiancheng 顧憲成); Bo Xiu 伯 修 (2 chars, recipient’s zi; Yuan Hongdao 袁宏道); Yu Xi lu 諭西虜 (3 chars; Sun Chengzong 孫承宗); Gu Sheng-bo xiuzhuan 顧升伯修撰 (Yuan Hongdao); Yu tie 諭帖 (Sun Chengzong). Lepton, having learned what a canonical letter title “looks like” from the labeled distribution, defaults to preface on these out-oftemplate inputs. The error mode is coherent that it falls along the axis the model was trained to discriminate, but it shows the limit of title-surface classification: the six recipients above are letters because a Ming or Qing editor said so, not because their titles share the genre signal Section 3 names.
5
[2] Peter K. Bol. 2007. Creating a GIS for the History of China. In Placing History: How Maps, Spatial Data, and GIS Are Changing Historical Scholarship, ESRI Press, 27–59. [3] George Aaron Broadwell, Jack W. Chen, and Jeffrey R. Tharsen. 2019. Reading the Quan Tang shi: Literary History, Topic Modeling, Divergence Measures. Digital Humanities Quarterly 13, 4 (2019). [4] Liu Chang, Wang Dongbo, Zhao Zhixiao, Hu Haotian, Liu Jiangfeng, Lu Si, Si Shen, and Liu Liu. 2023. SikuGPT: A Generative Pre-trained Model for Intelligent Information Processing of Ancient Texts. arXiv:2304.07778. [5] Hilde De Weerdt, Brent Ho, and Wing Kong Hou. 2016. MARKUS: Text Analysis and Reading Platform. https://dh.chinese-empires.eu/markus/. [6] Hilde De Weerdt and Csaba Oláh Horváth (eds.). 2023. Special issue on East Asian digital humanities. International Journal of Digital Humanities. [7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT, 4171–4186. [8] James E. Dobson. 2021. Vector hermeneutics: On the interpretation of vector space models of text. Digital Scholarship in the Humanities 37, 1 (2021), 81–93. [9] Kimmo Kettunen, Eetu Mäkelä, Teemu Ruokolainen, Juha Kuokkala, and Laura Löfberg. 2017. Old Content and Modern Tools — Searching Named Entities in a Finnish OCRed Historical Newspaper Collection 1771–1910. Digital Humanities Quarterly 11, 3 (2017). [10] Davor Lauc and Darko Vitek. 2021. Inferring Standard Name Form, Gender and Nobility from Historical Texts Using Stable Model Semantics. Digital Humanities Quarterly 15, 1 (2021). [11] Queenie Luo and Yung-Sung Chuang. 2024. Cleansing Jewel: A Neural Spelling Correction Model Built on Google OCR-ed Tibetan Manuscripts. ACM Transactions on Asian and Low-Resource Language Information Processing 23, 5, Article 73 (May 2024), 11 pages. https://doi.org/10.1145/3654811. [12] Donald Sturgeon. 2021. Chinese Text Project: A Dynamic Digital Library of Premodern Chinese. Digital Scholarship in the Humanities 36, Suppl. 1 (2021), i101–i112. [13] Dongbo Wang, Chang Liu, Zhixiao Zhao, Si Shen, Liu Liu, Bin Li, Haotian Hu, Mengcheng Wu, Litao Lin, Xue Zhao, and Xiyu Wang. 2022. Construction and Application of Pre-trained Models of Siku Quanshu in Orientation to Digital Humanities (SikuBERT and SikuRoBERTa). Library Tribune 42, 6 (2022), 31–43. [14] Ethan Yan. 2021. GuwenBERT: A Pre-trained Language Model for Classical Chinese. https://github.com/Ethan-yt/guwenbert. [15] J. Berenike Herrmann, Simone Rebora, Massimo Salgaro, and others. 2023. Tool criticism in computational literary studies. Digital Humanities Quarterly. [16] Claudia Resch, Daniela Fasching Rastinger, and Maria Kirchmair. 2023. Building an Iterative Annotation Schema for Eighteenth- Century Austrian Newspapers. Digital Humanities Quarterly. [17] Carolyn Strange, Daniel McNamara, Josh Wodak, and Ian Wood. 2014. Mining for the Meanings of a Murder: The Impact of OCR Quality on the Use of Digitized Historical Newspapers. Digital Humanities Quarterly 8, 1 (2014). [18] Shinya Tagami and Michael L. Satlow. 2023. Dating Ancient Inscriptions with Machine Learning. Digital Humanities Quarterly. [19] Huishuang Tian, Kexin Yang, Dayiheng Liu, and Jiancheng Lv. 2020. AnchiBERT: A Pre-Trained Model for Ancient Chinese Language Understanding and Generation. arXiv:2009.11473.
Discussion
The TF–IDF baselines’ top-weighted features — 與, 答, 荅, 柬, 復, 報 on the letter side and 序, 敘, 引, 叙 on the preface side — are exactly the characters §3 names, and Lepton’s perfect letter-class precision on the matched subset suggests the deployed model has internalized the same signal robustly enough to admit no false positives. The convergence between the linear baselines and the philological account meets Dobson’s [8] call for ML to expose features rather than scores. In deployment, Lepton has identified roughly fifty-five thousand letters across mid-Ming through early-Qing wenji, populating the Ming Letter Platform that now anchors social-network research on the period. Beyond this use case, the work contributes to the digital humanities community an open-source classifier and labeled corpus for Classical Chinese epistolary-genre identification, a worked example of modern-Chinese pretraining transferring to a title-level pre-modern task without continued pretraining, and evidence that interpretable n-gram baselines sit within striking distance of neural models on signal-rich short-text classification.
Acknowledgments I thank Katherine Enright, Hongsu Wang, Peter Bol for the support of this project. This project is conducted under the China Biographical Database (CBDB) Group at Harvard University, whose data, tooling, and collective expertise made the work possible.
References [1] David M. Berry. 2023. The Explainability Turn. Digital Humanities Quarterly 17, 2 (2023). 3