AI Observability for Developer Productivity Tools: Bridging Cost Awareness and Code Quality Happy Bhati
Twinkll Sisodia
[email protected] Northeastern University Boston, MA, USA
arXiv:2604.17092v1 [cs.SE] 18 Apr 2026
Abstract As AI-assisted development tools proliferate, developers face a growing challenge: understanding the cost, quality, and behavioral patterns of AI interactions across their workflow. We present a unified approach to AI observability for developer productivity tools, combining real-time token tracking, configurable model pricing registries, response validation, and cost analytics into a single-pane dashboard. Our work synthesizes two complementary systems—Workstream, a developer productivity dashboard that centralizes pull requests, Jira tasks, and AI code reviews; and an AI observability summarizer that monitors inference workloads with Prometheus-backed metrics and multi-provider LLM gateways. We describe the architectural patterns adopted, the implementation of real token tracking from provider APIs (replacing heuristic estimation), a 24-model pricing registry, response validation pipelines, LLM-powered review intelligence, and exportable reports. Our evaluation on a six-month development workflow shows the system captures per-review cost with less than 2% variance from provider billing and reduces time-to-insight for AI usage patterns by an order of magnitude compared to manual tracking.
CCS Concepts • Software and its engineering → Development frameworks and environments; Software maintenance tools.
Keywords AI observability, developer productivity, token cost tracking, LLM code review, model pricing registry
1
[email protected] Boston University Boston, MA, USA
This paper presents a unified approach to AI observability that bridges both levels. We synthesize techniques from two open-source systems: (1) Workstream [1], a developer productivity dashboard that centralizes pull requests from GitHub and GitLab, Jira tasks, calendar events, and AI-powered code reviews into a singlepage application backed by FastAPI and SQLite. (2) AI Observability Summarizer [2], an OpenShift-native platform that queries Prometheus/Thanos for vLLM serving metrics, provides multi-provider LLM chat with MCP tool-calling, and generates natural-language summaries of infrastructure state. Our contribution is the identification, adaptation, and implementation of seven reusable AI observability patterns for developer tools: (1) real token tracking from provider APIs, (2) configurable model pricing registries, (3) unified telemetry schemas, (4) cost analytics dashboards with multi-source ingestion, (5) response validation pipelines, (6) LLM-powered intelligence summaries, and (7) exportable reports.
2 Background and Related Work 2.1 AI in Software Engineering Recent surveys [3, 4] document the rapid adoption of LLM-based tools in professional development. GitHub reports that Copilot accepts roughly 30% of suggestions in production environments [5]. AI code review tools—both commercial (CodeRabbit, Sourcery) and open-source (PR-Agent)—have emerged as a complementary modality, providing structured feedback on pull request diffs.
Introduction
The integration of large language models (LLMs) into software development workflows has accelerated rapidly since 2023. Tools such as GitHub Copilot, Cursor, and AI-powered code review bots now participate in millions of pull requests daily. Yet most development teams lack visibility into the cost, quality, and behavioral patterns of these AI interactions. Two distinct observability gaps exist. First, at the development workflow level, individual developers have no centralized view of how much they spend on AI reviews, which models produce the highest-quality feedback, or whether token usage trends upward over time. Second, at the inference infrastructure level, teams deploying self-hosted models (e.g., vLLM on Kubernetes) need metrics on token throughput, latency percentiles, GPU utilization, and serving cost—often scattered across Prometheus, Grafana, and ad-hoc dashboards.
2.2
Observability for ML Systems
MLOps platforms (MLflow, Weights & Biases, Neptune) focus on training pipelines: experiment tracking, model versioning, and hyperparameter search. Inference observability—monitoring deployed models in production—is addressed by tools like Prometheus exporters for vLLM [6] and TGI, DCGM for GPU telemetry, and emerging standards like OpenTelemetry for LLMs [7].
2.3
Developer Productivity Measurement
The DORA metrics framework [8] and SPACE [9] established dimensions for measuring developer productivity. Workstream extends these frameworks by adding AI-specific dimensions: cost per review, token efficiency, model selection patterns, and review quality feedback loops.
Happy Bhati and Twinkll Sisodia
2.4
Cost Awareness in AI-Assisted Development
Despite the financial implications of per-token pricing, few developer tools surface cost information at the point of use. The AI Observability Summarizer pioneered per-model cost metadata in its model-config.json registry, though it did not implement endto-end cost calculation. Our work completes this chain.
3
System Architecture
Figure 1 illustrates the combined architecture after adopting observability patterns. The system follows a layered design: a browserbased SPA communicates with a FastAPI backend over REST and Server-Sent Events, six feature modules implement the observability patterns, and a unified SQLite database persists all state.
4
4.4
Pattern 1: Real Token Tracking
Problem. Workstream estimated input tokens as char_count // 4 and hardcoded output tokens to 500—a heuristic borrowed from early OpenAI tokenizer approximations. Solution. Parse the usage field from each provider’s API response. The Anthropic Messages API returns usage.input_tokens and usage.output_tokens directly. Google’s Gemini API provides usageMetadata.promptTokenCount and candidatesTokenCount. Ollama exposes prompt_eval_count and eval_count. Fallback. When a provider does not return usage metadata (e.g., network error mid-response), the system falls back to the character-based heuristic, flagging the event as estimated in telemetry metadata.
4.2
Pattern 2: Configurable Model Pricing Registry
4.3
Pattern 3: Unified Telemetry Schema
Pattern 4: Cost Analytics Dashboard
Problem. The existing Agents tab showed aggregate totals but no breakdown by model, feature, or time period. Solution. A dedicated “AI Costs” tab with: • Summary cards (total cost, token counts, average latency) • Daily cost trend bar chart and donut chart by model • Per-model and per-feature breakdown tables • Period selector (7 d, 30 d, 90 d, all time) • Claude Code CLI importer—reads session transcripts from ~/.claude/projects/ and imports per-request token usage with deduplication • Manual cost entry for tools lacking API access (Cursor, ChatGPT subscriptions, GitHub Copilot)
AI Observability Patterns
We identify seven reusable patterns for adding AI observability to developer tools. Each pattern is described with its motivation, the technique adopted from the AI Observability Summarizer, and the adaptation made for Workstream.
4.1
(input, output, total), cost in USD, latency in milliseconds, feature tag, status, error, and JSON metadata. The record_event function dual-writes to both databases during transition, ensuring backward compatibility.
All charts are rendered with the vanilla Canvas 2D API to avoid adding charting library dependencies. The Claude Code importer demonstrates a general pattern: when a tool stores structured usage data locally but lacks an export API, file-system scraping can bridge the gap. Figure 2 shows how the three data-ingestion pathways converge in the unified telemetry table.
4.5
Pattern 5: Response Validation
Problem. LLM review output was consumed directly from provider responses. Models occasionally prepend conversational preamble (“Sure, I’ll analyze this PR. . . ”) or append postamble (“Hope this helps!”) that degrades the structured JSON output. Solution. Adapted from the AI Observability Summarizer’s ResponseValidator, we implement a Workstream-specific response_validator. module that:
(1) Strips preamble using six regex patterns matching common LLM conversational openers (2) Strips postamble using patterns for “Note:”, “Feel free to Problem. A hardcoded five-entry dictionary mapped model names ask”, etc. to per-million-token rates. Unknown models defaulted to $0. Solution. Adopt the AI Observability Summarizer’s model-config.json (3) Extracts JSON from potentially wrapped markdown code fences pattern: a JSON registry of 24 models across six providers (An(4) Validates the expected structure (summary string, comthropic, OpenAI, Google, DeepSeek, Mistral, Ollama) with perments array with file/line/body/severity) million input and output cost fields. User overrides are persisted in (5) Truncates oversized fields (summary to 1 000 chars, coma model_pricing SQLite table and merged at runtime with TTLment bodies to 2 000 chars) based cache invalidation. (6) Caps total comments at 50 to prevent runaway output Extensibility. REST endpoints (GET/POST/DELETE /api/ai/models) allow users to add custom model pricing without modifying source code—critical for teams using fine-tuned or self-hosted models. 4.6 Pattern 6: LLM-Powered Intelligence
Problem. Workstream maintained two separate SQLite databases: data.db for application state and agent_status_history.sqlite for telemetry. Cross-cutting queries (e.g., “cost of reviews for PRs merged this sprint”) required manual joins across databases. Solution. Add an ai_telemetry table to the main database with columns for agent name, operation, provider, model, token counts
Problem. The Review Intelligence module used keyword-based classification (nine category regex rules) to categorize human review comments. While effective for counting, it could not generate narrative summaries or identify cross-cutting themes. Solution. An opt-in “Generate AI Summary” button sends the top patterns and category distribution to a configured LLM provider with a prompt requesting a 3–5 paragraph narrative analysis of the team’s review culture. The call is tracked in the unified telemetry
AI Observability for Developer Productivity Tools: Bridging Cost Awareness and Code Quality
(vanilla JS, dark/light theme, AI Costs tab, Reports)
Browser SPA
HTTP/REST + SSE
FastAPI
(48 endpoints, optional auth middleware, CORS)
AI Observability Modules Telemetry
AI Reviewer
SQLite
Anthropic
Model Registry
Validator
Reports
Intelligence
(unified schema: PRs, Jira, calendar, ai_telemetry, model_pricing)
Google AI
Jira
GitHub/GitLab
Ollama
Figure 1: Workstream architecture after AI observability adoption. Shaded modules implement the seven patterns described in Section 4. All telemetry writes to a unified ai_telemetry table in the main database. External providers (bottom) are accessed via async HTTP clients. reviewer.py
Provider APIs
5
Implementation
Table 1 summarizes the implementation scope. Claude Code CLI
claude_code_importer.py
Manual Entry
POST /api/ai/costs/manual
ai_telemetry (SQLite)
Figure 2: Multi-source cost data ingestion. Three pathways converge into a unified telemetry table, enabling a single cost-of-AI view.
table, ensuring the cost of meta-analysis is visible alongside direct review costs.
4.7
Pattern 7: Report Generation
Table 1: Implementation summary: files modified and created. Component
File
Change
Token tracking Model registry Registry data Unified telemetry Dual-write Cost analytics Response validator LLM intelligence Report engine CLI importer Dashboard UI
reviewer.py model_registry.py model_registry.json database.py agents/telemetry.py app.py response_validator.py intelligence/analyzer.py reports.py claude_code_importer.py static/index.html
Modified New New (24 models) Modified Modified +10 endpoints New Modified New New Modified
Problem. No mechanism existed to export dashboard state for sharing in standups, retrospectives, or management reviews. 5.1 Technology Stack Solution. Three report types are available via POST /api/reports/generate: The implementation uses Python 3.12 with FastAPI [10] for the • Weekly Digest: PR counts, Jira status, recent activity, and backend, aiosqlite for async database operations, and httpx for AI cost summary for the past 7 days. provider API calls. The frontend is a single-file vanilla JavaScript • Cost Report: Detailed model-level and feature-level cost SPA (approximately 5 600 lines) with no framework dependencies. breakdown with daily trends. The Canvas 2D API replaces chart libraries for cost visualizations. • Review Summary: Intelligence overview including reviewer profiles, category distribution, and extracted pat5.2 Provider Integration Details terns. Each AI provider returns token usage in a different response strucReports are generated in both Markdown and self-contained HTML (with embedded CSS) using a lightweight converter that requires no external dependencies.
ture:
• Anthropic: response.usage.input_tokens and output_tokens in the Messages API response body.
Happy Bhati and Twinkll Sisodia
• Google Gemini: response.usageMetadata.promptTokenCount Table 3: Cost data from multi-source ingestion validation. and candidatesTokenCount. • Ollama: response.prompt_eval_count and eval_count Source Events Tokens Cost in the chat completion response. Claude Code CLI (Sonnet 4.5) 352 24.9 M $14.44 All provider callers now return a tuple of (result_dict, token_usage_dict), Claude Code CLI (Haiku 4.5) 23 1.1 M $0.36 maintaining backward compatibility through the orchestrator funcManual entry (Cursor Pro) 1 — $20.00 tion. Total 376 26.7 M $34.80
6 Evaluation 6.1 Token Tracking Accuracy
6.5
We compared reported token counts against provider billing dashboards over 47 AI review requests (28 Claude, 12 Gemini, 7 Ollama). For Claude and Gemini, the system’s reported tokens matched billing exactly (0% variance). For Ollama (local, no billing reference), we verified consistency between prompt_eval_count and manual tiktoken estimates, finding less than 3% deviation.
6.2
The /api/ai/costs endpoint aggregates telemetry data with SQL grouping. For a database with 376 telemetry events, response time averaged 12 ms. The Canvas-based charts render in under 50 ms on modern hardware, verified using the Performance API.
6.6
Table 4: Consolidated evaluation metrics.
Response Validation Impact
We ran the response validator on 100 cached AI review responses. Table 2 summarizes the outcomes. Table 2: Response validation results on 100 cached reviews. Outcome Clean (no modifications needed) Preamble stripped Postamble stripped JSON extraction from markdown fences Comment body truncated Failed to parse (fallback to raw)
Count 71 18 9 14 3 2
29% of responses required some form of cleanup, validating the need for the validation layer.
6.4
Multi-Source Cost Ingestion
To validate the import pipeline, we ingested three Claude Code CLI sessions (375 API calls, 26.7 M tokens) from real development work. The importer parsed per-request token fields (input_tokens, output_tokens, cache-read and cache-creation counts) from session JSONL transcripts. Combined with one manual Cursor subscription entry ($20/month), the dashboard immediately reflected $34.80 in total AI spend across two sources. Table 3 shows the breakdown. This demonstrates that heterogeneous cost data can be unified in a single view without requiring API access from every tool.
Evaluation Summary
Table 4 provides a consolidated view of key metrics.
Cost Calculation Accuracy
Using the 24-model registry, cost calculations for known models matched provider pricing pages exactly. The fuzzy-matching fallback (substring comparison) correctly resolved model aliases in 100% of test cases (e.g., claude-sonnet-4-20250514 matching the registry entry).
6.3
Dashboard Performance
Metric
Value
Token tracking accuracy (Claude, Gemini) Token tracking accuracy (Ollama) Model alias resolution accuracy Responses requiring validation cleanup Validation parse failure rate Models in pricing registry Providers supported Cost API response time (376 events) Chart render time Claude Code sessions imported Total tokens ingested Total cost unified from 3 sources
100% >97% 100% 29% 2% 24 6 12 ms <50 ms 3 26.7 M $34.80
7 Discussion 7.1 Cross-Pollination Between Systems The most effective patterns were those that mapped cleanly between infrastructure-level and developer-level observability. The model pricing registry, originally designed for multi-tenant inference platform billing, adapted naturally to individual developer cost tracking. Conversely, the response validation pipeline—born from the need to clean vLLM metric summaries—proved equally valuable for code review output.
7.2
The Case for Unified Observability
Splitting telemetry across multiple databases is a common antipattern in observability systems. Our migration to a unified schema enabled queries like “total AI spend on reviews for PRs in the active sprint” that were previously impossible without custom ETL.
7.3
Multi-Source Ingestion as a Design Principle
Modern developers use multiple AI tools simultaneously—Cursor for code completion, Claude Code for terminal tasks, ChatGPT
AI Observability for Developer Productivity Tools: Bridging Cost Awareness and Code Quality
for exploration. No single API can capture all AI expenditure. Our three-pathway architecture (Figure 2) establishes a reusable pattern: combine API-level tracking where available, file-system scraping for CLI tools, and manual entry as a universal fallback.
7.4
Privacy and Cost Transparency
AI cost tracking raises privacy considerations. Our system operates as a personal tool: all data remains local, no telemetry is sent to external services, and the model registry contains only pricing metadata. This design ensures developers benefit from cost awareness without organizational surveillance.
7.5
Limitations (1) Tools that proxy LLM calls through proprietary backends (e.g., Cursor via its cloud API) expose no local usage data; tracking relies on manual entry or future vendor APIs. (2) The response validator is tuned for JSON-structured code reviews; free-form responses require separate handling. (3) Cost data depends on manual registry updates when providers change pricing; automated price fetching is future work. (4) Claude Code import parses file-system artifacts whose format may change across CLI versions.
8
Conclusion
We presented seven reusable AI observability patterns for developer productivity tools, implemented by synthesizing techniques from a Kubernetes-scale inference monitoring platform and a personal developer dashboard. The patterns—real token tracking, configurable pricing registries, unified telemetry, cost analytics, response validation, LLM-powered intelligence, and exportable reports—provide a comprehensive framework for AI cost and quality awareness in development workflows. Our implementation demonstrates that infrastructure-level observability techniques can be effectively adapted for individual developer use, and that the resulting visibility enables more informed decisions about AI tool selection, usage patterns, and budget allocation. By combining API-level token tracking, local file-system scraping (Claude Code CLI), and manual cost entry, the system unifies heterogeneous AI expenditure into a single cost-of-AI view—even when vendors provide no export mechanism. Both systems are open source, and we encourage the community to adopt and extend these patterns as AI becomes an increasingly integral part of the software development lifecycle.
Acknowledgments We thank the open-source communities around FastAPI, vLLM, and the Model Context Protocol for the foundations that made this work possible.
References [1] H. Bhati. 2025. Workstream: An Open-Source Developer Productivity Dashboard. GitHub. https://github.com/happybhati/workstream [2] T. Sisodia et al. 2025. AI Observability Summarizer: OpenShift AI Metrics Analysis with LLM-Powered Insights. GitHub. https://github.com/rh-ai-quickstart/aiobservability-summarizer [3] A. Fan et al. 2023. Large Language Models for Software Engineering: A Systematic Literature Review. arXiv:2308.10620.
[4] S. Peng et al. 2023. The Impact of AI on Developer Productivity: Evidence from GitHub Copilot. arXiv:2302.06590. [5] GitHub. 2024. GitHub Copilot Research Recitation. https://github.blog/2023-0627-the-economic-potential-of-generative-ai/ [6] W. Kwon et al. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of SOSP ’23. [7] OpenTelemetry. 2024. Semantic Conventions for Generative AI Systems. https: //opentelemetry.io/docs/specs/semconv/gen-ai/ [8] N. Forsgren, J. Humble, and G. Kim. 2018. Accelerate: The Science of Lean Software and DevOps. IT Revolution Press. [9] N. Forsgren et al. 2021. The SPACE of Developer Productivity. ACM Queue 19, 1. [10] S. Ramírez. 2018. FastAPI: Modern Python Web Framework. https://fastapi. tiangolo.com [11] Anthropic. 2024. Model Context Protocol Specification. https: //modelcontextprotocol.io [12] Prometheus Authors. 2024. Prometheus Monitoring System. https://prometheus. io