How a cost question became a calculator, a paper, and a book

Kumar, who is lead on the AKD project, asked me a simple question a few months back: what does it cost to run our AI agent in production for the AKD team? How would we convey or calculate this to other teams who would want to use AKD?

I didn't really know how to answer it — I didn't even know what all is involved. I have been helping manage backend, frontend, and infrastructure for running agentic workflows, but I hadn't been involved in actual agent development myself.

I thought this could be a good opportunity to learn what is involved, and that it would also help me learn more about agents and LLMs themselves. Kill multiple birds with one stone.

I didn't know much about token rates, cache hit rates, tool overhead, or traffic profiles, or which ones mattered most. So I took the longer route — 15 nights of evenings and weekends instead of a 2-day spreadsheet — to get the complete map before I trusted any number.

The first POC

I built an AI-enabled calculator POC. There were components for calculating cost, and I was trying to get AI to parse the user's intent and auto-select which components to activate. It was kind of working.

But after I had this prototype, there was an urgent need to present some numbers to stakeholders. Along with Kumar, Buddy, who is our deputy manager for DSE, had also asked several times for a cost estimate, because EIE was planning to have a public chat interface and the costing mattered a lot.

Paridhi, who is leading backend efforts for EIE and AKD, ran the real experiments — built a spreadsheet, did some back-of-envelope math, made a few best-guess assumptions. She got a number we could defend if anyone asked. She worked with Gjore, Lily, and other members of the EIE team to present them to stakeholders.

In the meantime I tried to build simulated numbers to match her math. I pivoted from the chat-driven idea and built a single-page calculator just for EIE.

It was a single-page interactive cost estimator for a NASA-style public-facing LLM agent. User adjusts ~20 controls (traffic, model, caching, rate-limiting, FactReasoner, hosting) and a sticky headline block recomputes monthly cost in real time, with full breakdowns and an API-vs-self-host comparison.

Paridhi's spreadsheet already had the EIE team covered. What I was building was more complicated than they needed, and harder to verify or communicate end-to-end. The team rightly used Paridhi's numbers to talk to stakeholders.

I kept going on personal time because the question itself stayed with me — and I wanted a tool that would help the next team to ask, not just EIE: something reusable they could pick up, not a one-off spreadsheet that lives in someone's inbox. So I built a generalized version, one that could serve AKD and any other NASA team facing the same procurement question.

Building such a calculator had been on my mind even before the EIE use case — Kumar had asked for it many times to present to Rahul, Senior Data Science Strategist at NASA, leading AI R&D and Foundation Models for Science, and the lead on the AI for Science Project at MSFC ODSI, which includes the AKD sub-project.

Although I wanted a chat-based calculator — where the team would describe their use case, AI would construct the architecture, choose components matching the query, and build out the cost, all in natural language — it was slow going. My time was split across several projects — Research Repository, Science Analytics Platform, AKD, Data Systems Evolution — and home was already full.

I resorted to vibe coding to build this — but boy, it takes a lot of time and energy to work with AI. You have to verify its output and its math, and build multiple verification gates and cross-checks before you can trust what it produces.

The token simulator (turned out richer than expected)

The first thing I built was a token simulator — a browser-based engine that mocks LLM calls with adjustable knobs, so teams could predict cost without hitting real APIs.

The simulator itself we didn't end up needing — but its per-agent token math turned out to be the keeper. I took that math, modified it substantially, and built what is now live at calc.ajinkya.ai around it.

The paper and the blog posts

While I was deep in this, I wanted to write it up. So I wrote a blog post about how to build such a calculator and a paper to give it formal shape. I also built benchmarking code that anyone can run to cross-verify each claim in the paper against the code, and trace the numbers.

I did a lot of revisions on the paper. My central argument and contributions shifted as I worked on it. AI had missed things and made a few mistakes, so I had to keep going in different directions, using different LLMs to cross-verify things.

Along the way I learned about prompt caching, templated tool responses, FactReasoner, MCP-style tool registries, cascading verification, and a lot more — and that learning is consolidated in a draft book that is now also live.

Before you draw conclusions

I built this tool for myself, to understand what all needs to be there when an honest cost-calculation is needed. I'm not sure whether others will really find it useful or not — but I'm leaving it public, and I hope they do.

I started this knowing almost nothing, and I wanted to map what's actually involved before trusting any of it. AI helped me build the tool, the posts, and the paper — the groundwork for the real learning, which starts now. Because I wanted it to be a learning activity, I added the tooltips and "why this matters" boxes, so the calculator works as an educational tool too.

Kumar asked what one agent costs to run. The honest answer turned out to be a calculator, a paper, and a book — and now anyone can trace the math for themselves.