Conceptio › Archive › PhilArchive
PhilArchive open access

Multi-Model Cross-Evaluation of Frontier Large Language Models for Scientific Writing: A 4 × 4 × 7 Tensor Benchmark and a Visual-Analytics Framework

Gramatikoff, Kosi
PhilArchive · Papers
Open Source ↗
philosophy
Background. Scientific writing now routinely involves several large language models (LLMs), yet comparisons of their suitability are mostly single-evaluator, single-score leaderboards. We asked whether four frontier model families—Gemini, Claude, GPT and Grok—agree when they evaluate themselves and one another, and how such multidimensional judgements can be communicated visually. Methods. Each system was given an identical prompt and produced a self- and cross-evaluation on seven parameters (functional capability, context and memory, scholarly register, mathematical reasoning, multimodal integration, error resiliency, and empirical/reference precision), yielding a 4 evaluators × 4 targets × 7 parameters tensor (112 scores). We computed all-evaluator and peer-only consensus, within-evaluator z-scores to control leniency, self-preference bias, inter-evaluator agreement (Kendall’s W, intraclass correlation), and rank robustness by leave-one-out and Monte-Carlo weighting. Nine figures were designed to expose different tensor slices. Results. Evaluator leniency differed by 1.13 points (Claude 8.04; GPT 9.16). Grok ranked last under every evaluator and every analytic scenario. The top three were statistically near-tied on all-evaluator means (Gemini 8.92, Claude 8.89, GPT 8.89); the leader depended on whether self-ratings were included (Gemini or Claude). Evaluators were unanimous on three parameter leaders—Gemini for context and multimodality, Claude for scholarly register—and disagreed most on error resiliency (largest SD 1.03) and mathematical reasoning (W = 0.28). Self-ratings exceeded peer ratings for Gemini (+0.66), GPT (+0.64) and Grok (+0.39) but fell below them for Claude (−1.10). Absolute agreement was modest (ICC(2,1) = 0.365) but rose after leniency adjustment (0.675). Recommended workflows converged on Gemini for corpus ingestion and figure reading, Claude for drafting and review, GPT for quantitative work, and mandatory external citation verification. Conclusions. Under uncertainty of this size, single-number rankings are not defensible; profile-based, leniency-adjusted, bias-aware presentation is. The framework and scripts are provided so the tensor can be updated as models change. All scores are model-generated opinions, not measurements.
Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

This document is indexed with metadata only: full text is not available in the archive for this record. Open the official source ↗

Related documents

Record · ID 1505742
Retrieved via Conceptio: every document is proof-bundled with source, license, and retrieval metadata.
Bring this data into your agent: Connect via MCP or API · 20 free credits to start · no card required
Read the Docs → · View CLI source on GitHub · Conceptio Open Knowledge Archive