We Processed 52,406 API Calls for $2.41. Here's the Stack.
Think back to the first time you picked a phone plan. Most of us had no idea how much data we used each month, so we went for the generous plan. Running out felt worse than overpaying. A few months later, the usage report showed we were using less than half of what we paid for. The plan was not really the problem. We had simply never measured our own usage. The same thing happens when teams add AI features. The best-known model becomes the default, and at a few dozen calls a day the bill barely registers. Once an agent takes over repetitive work, though, calls climb into the tens of thousands fast. From that point on, model choice stops being a matter of taste and becomes a question of cost structure. So we decided to measure it ourselves. The AIEEV marketing team built a research agent for real day-to-day work, ran more than 50,000 model calls, and checked the bill. Did this job actually need an expensive model? And what would the same work have cost on other APIs? 🧾
🏃 Handing 40,000 posts a week to an agent
Every week, the AIEEV marketing team looks at what AI infrastructure users are talking about to find topics for our blog and newsletter. Communities carry the kind of honest voice that surveys and interviews rarely capture. The problem is volume. Hundreds of posts and comments go up every day, and no one can read them all. This agent took over that job. It collected 43,498 posts and comments from the last 90 days on Hacker News that mention one of 10 keywords such as GPU, inference and LLM cost. It then checks each one for relevance to AI infrastructure. For relevant posts, it pulls out the user's concern and a short quote in the original wording. Finally, it rolls everything up into a weekly brief.
🧩 The stack: small models for easy work, bigger models for hard work
We did not give every step to one model. Each step needs a different skill. Classification, which decides whether a post is relevant, makes the most calls, but each decision is fairly simple. Extraction makes far fewer calls, but it has to understand the English source, summarize the concern in Korean for our team and quote the original wording exactly. That takes stronger language understanding. So we split the work like this.
Step | Model (Air API) | Calls | Why this model | Cost |
1. Classify | Qwen3.5-9B | 43,498 | Fastest and cheapest. 118.7 TPS throughput. $0.10 input / $0.15 output per 1M tokens | $1.20 |
2. Extract | Qwen3.6-35B-A3B | 8,861 | Understands English sources and writes solid Korean summaries. $0.14 input / $1.00 output per 1M tokens | $1.06 |
3. Summarize | Qwen3.6-35B-A3B | 47 | Rolls extraction results into weekly briefs, 200 at a time | $0.15 |
*Throughput as shown in the Air API console on September 29, 2026.
The point is not to find the single best model. It is to match each model to the difficulty of the task. Air API is OpenAI-compatible, so if you already have code written with the OpenAI SDK, you only need to change two things: the API key and the base URL. This agent used the OpenAI Python SDK as is. We only changed the API key, the base URL and the model endpoint for each step.
from openai import OpenAI
client = OpenAI(
api_key="YOUR_AIEEV_API_KEY",
base_url="https://ap-1.aieev.cloud/endpoints/qwen3-5-9b/v1",
)
response = client.chat.completions.create(
model="qwen/qwen3.5-9b",
messages=[{"role": "system", "content": CLASSIFY_PROMPT},
{"role": "user", "content": post_text}],
response_format={"type": "json_object"}, # always return valid JSON
extra_body={"chat_template_kwargs": {"enable_thinking": False}}, # turn off thinking mode
)Before running the full dataset, we tested 200 posts first and found a problem that had a big effect on cost. {"relevant": false} is all a classification answer needs, yet the model was spending up to 2,840 output tokens to produce it. Qwen models have thinking mode on by default, which means they write out a reasoning process before answering, and those tokens are billed as output. So we turned thinking mode off for classification (enable_thinking: false). Output dropped to an average of 21 tokens per call. For simple classification that needs no deep reasoning, one setting can cut wasted tokens and output cost by an order of magnitude or more. We also turned on JSON mode, and across more than 50,000 calls there was not a single formatting error.
💰 The result: 52,406 calls for $2.41
After fixing what the 200-post test revealed, we ran the full job. Here is what came back.
Item | Result |
Total calls | 52,406 |
Run time | 41.6 minutes |
Errors | 0 |
Input / output tokens | 13,256,056 / 1,742,702 |
Air API bill | $2.41 |
What would the same work have cost on other APIs? We took the input and output tokens this run actually used and priced them at each provider's standard rates from their official pricing pages. The answer depended heavily on the model.
Against price-competitive options such as the official DeepSeek API, gpt-5.6-luna and gpt-5-mini, the gap was 1.3x to 2.8x. Running the same job on top-tier models like gpt-5 or Claude Sonnet 5 would have cost 14x to 18x more. Top-tier models are worth it for some tasks. But most of this job was answering one question over and over: is this post about AI infrastructure? If a step does not need a top-tier model, the price difference is pure cost. In the other direction, running every step on Qwen3.5-9B alone would have brought the total down to $1.59. We still used the 35B model for extraction, because there we cared more about understanding the English source and writing a good Korean summary than about saving a little more. The goal was never to use the cheapest model everywhere. It was to tell apart the steps worth paying more for from the ones that are not.
(Note) How we compared
Each provider splits text into tokens differently, so real bills can differ from these estimates. Caching and batch discounts are not included, and prices are as of September 29, 2026. In the quality test below, we called the OpenAI models directly: input token counts were nearly identical (Air API 35,551 / OpenAI 35,034), while OpenAI output was 27% to 44% higher because it includes reasoning tokens.
🔍 Quality: do you get what you pay for?
So far, the $2.41 figure stands out. But cheap only matters if the answers are good. If a low-cost model makes lots of mistakes, someone has to check its work, and that time costs money too. So we compared quality as well as price. We randomly picked 100 results that Air API had processed (50 judged relevant, 50 judged not relevant) and gave the same prompts to gpt-5-mini and gpt-5. We treated the 76 cases where all three models agreed as correct, then read the source for each of the 24 disagreements and scored them by hand.
Model | Classification errors (of 100) | Measured cost (100 posts) | Cost vs. Air API |
Air API (9B + 35B) | 6 | $0.0086 | 1x |
gpt-5-mini | 13 | $0.0252 | About 3x |
gpt-5 | 3 | $0.1363 | About 16x |
As expected, gpt-5 was the most accurate, with 3 classification errors out of 100. Air API followed with 6, and gpt-5-mini had 13. But the extra accuracy came at a price. Processing the same 100 posts cost about 16x more on gpt-5 than on Air API. Air API, meanwhile, made less than half as many errors as gpt-5-mini at about a third of the cost. The errors were also different in kind. Five of Air API's six errors came from reading "AI infrastructure" broadly, tagging posts about general AI topics such as jobs or productivity debates as relevant. For research, that kind of error is easy to filter out later. gpt-5-mini, on the other hand, sometimes missed posts about GPUs or local inference, and sometimes flagged unrelated posts, such as database upgrades, as relevant. On extraction, the three models were close. None of them failed to extract from any of the 50 relevant posts. When asked to quote the user's words exactly, Air API and gpt-5-mini did so in 46 of 50 cases and gpt-5 in 34. gpt-5 tended to polish the wording rather than copy it. There was one weak spot. In the summarization step, a Chinese word slipped into a Korean sentence once. At the scale of tens of thousands of calls, rare slips like this can happen. In production, it is worth adding a simple post-processing check for characters outside the expected languages.
*Quality results are based on a 100-post sample and may differ for other kinds of tasks.
📊 What the agent's summaries showed: users talk about predictability more than price
In the final step, the agent read the extracted concerns 200 at a time and wrote a weekly brief with the top five recurring concerns, representative customer quotes and suggested content topics. The 8,861 relevant posts produced 46 briefs. Below is the list of top concerns from one of them, exactly as the agent wrote it.
Brief from the third week of September (top 5 concerns from one batch of 200)
The cost vs. performance vs. security balance is breaking down
Uncertain GPU ROI and financial risk
Security weaknesses and misuse risk in open-source models
Limits of local inference and hardware bottlenecks
Moving away from cloud and API dependence, and the push for data sovereignty
On its own, a single brief reads like the list you would expect. But when we stacked the 45 well-formed briefs together and compared them with the classification results, three patterns appeared that raw counts alone would not show.
1. The concern people raise most is not the one they feel most strongly about
Concern | Posts (share) | Negative posts | Ranked #1 in briefs (on its own) |
Performance and reliability | 4,181 (47%) | 34% | 1 / 45 |
Cost | 3,086 (35%) | 58% | 35 / 45 |
By raw count, performance and reliability (47%) came up more often than cost (35%). In the summaries, the order flipped. 42 of 45 briefs put cost in their top concern (7 of them paired it with performance, as in "the balance between cost and performance"), and all 45 listed cost somewhere in their top five. Only one brief ranked performance and reliability first on its own. The difference came down to emotional temperature. Only 34% of performance and reliability posts were negative, and many were neutral exchanges such as benchmark comparisons or configuration tips. Among cost posts, 58% were negative. In short, performance came up more often, but cost came up with more urgency. The problems people mention most and the ones that frustrate them most are not always the same. A simple frequency count would have missed that.
2. The real cost complaint is less "it's expensive" and more "I can't tell what I'll pay"
A closer look at cost posts revealed another pattern. People were not just saying prices were too high. 13 of 45 briefs (29%) listed unpredictable spending or opaque pricing among the top concerns. The same themes kept coming up: subscription usage limits that are hard to gauge, token-based billing being easier to reason about than GPU-hour quotas, and agents burning far more tokens than expected. For users, the question was not only how expensive something is, but whether they can predict what this month's bill will be. We have seen the same interest in our own content. Our earlier post on how to plan AI infrastructure costs up front and keep usage and spend visible drew strong attention. This analysis suggests that interest is not just about finding a lower price. It is also about wanting a cost structure that is predictable and easy to understand.
"I have no clue what 1x is on a given day"
"token-based pricing is much easier to reason about than GPU-time quotas"
"watching CI in an agent loop is that it burns tons of tokens"
3. Users are weighing the cost burden against the operational burden
9.4% of cost posts also mentioned running models locally or self-hosting. In the briefs, 37 of 45 (82%) listed hardware or local execution among the top concerns, and 27 (60%) listed dependence on a single provider. At the same time, people noted that going local means hitting a different wall: not enough VRAM, slower speeds and more to operate. Users were caught between API bills that feel too high and hardware that cannot keep up if they run things themselves.
"straight up can't afford frontier models"
What the marketing team took away
"AI infrastructure users want lower prices" is something anyone could guess. What these summaries gave us was a more specific priority. What users actually said was closer to "I want to know what I'll pay before I pay it" and "I don't want to be locked into one provider." Showing clearly how costs are calculated and what the bill will be matters as much as lowering the price. That is also why this post shares every number: calls, tokens and the final bill.
*Concern types and themes are approximate counts, grouped by keyword from the agent's classifications and the concerns listed in its briefs. Hacker News is an English-language developer community, so these concerns may not map exactly to every market.
🛠️ Three things we learned
Putting cost and quality side by side made one thing clear: how you use a model changes the outcome as much as which model you pick. If your team runs agents or is thinking about adopting them, here are three takeaways.
Cost is decided at the step with the most calls.
In this experiment, classification made up 83% of all calls and half of the total cost. If you want to cut costs, start with the model behind the step you call most often, not the most expensive model in the stack.
One setting can matter more than switching models.
Turning off thinking mode in the classification step cut output from 780 to 2,840 tokens per call down to an average of 21. Leaving thinking mode on for tasks that only need a short judgment can multiply costs many times over, without changing models at all. Before comparing price sheets, check how many tokens your model is actually using and whether that is the right amount for the task.
100 samples are enough to decide whether to switch models.
The 100-post quality test cost less than $0.20 across all three models combined. There is no need to guess whether a cheaper model is good enough. Pull about 100 real examples and compare. Accuracy alone is not enough, though. Look at what kind of errors each model makes. An error that is easy to filter out later and an error that misses data you must catch are both "one error," but they mean very different things.
The prompts we actually used
The instructions behind all 52,406 calls were surprisingly short. Below are the prompts the marketing team used for classification, extraction and summarization, translated from the Korean originals. Each prompt went in as the system prompt, and the post to analyze was sent separately as the user message. Thinking mode was off for all three steps, and JSON mode was on for classification and extraction.
1. Classify (Qwen3.5-9B)
The following is a Hacker News post or comment. Reply in JSON only.
{"relevant": true|false, // is it about AI infrastructure, GPUs, inference cost or model serving?
"category": "cost burden"|"performance/reliability"|"adoption question"|"product/launch news"|"other",
"sentiment": "negative"|"neutral"|"positive"}
2. Extract (Qwen3.6-35B-A3B)
From the post below, extract the AI infrastructure user's real concern and reply in JSON only.
{"pain_point": "one-sentence summary in Korean",
"customer_phrase": "a short phrase that shows the customer's own language (verbatim from the English original)",
"content_idea": "one-line blog topic that addresses this concern (in Korean)"}
3. Summarize (Qwen3.6-35B-A3B)
Below is a list of pain points collected from the AI infrastructure community. Write a weekly brief in Korean for marketers.
Include the top 5 recurring concerns, representative customer language (quote each item's original English phrase verbatim) and 3 suggested content topics. No exaggeration, and do not invent numbers that are not in the list.
As you can see, the instructions are only a few lines long. Handling 50,000 posts did not take long prompts. It took breaking the work into small steps and choosing the right model and settings for each one.
Wrapping up
There are real moments when a top-tier model is the right call. In this experiment, gpt-5 was still the most accurate classifier. But for work repeated tens of thousands of times a day, it is worth asking whether that accuracy gap is worth 16x the cost. Just as you would check your usage history before changing phone plans, it pays to measure before you choose an AI model. The 200-post test and the 100-post comparison together cost less than a dollar. If you already have AI models wired into your workflows, try running the same work on Air API and compare the results and the bill. Air API offers a range of open-source models through an OpenAI-compatible interface, including the Qwen3.5-9B and Qwen3.6-35B-A3B models used in this experiment. Just swap the API key and base URL in your existing code.
🚀 Qwen3.5-9B : fast and cost-efficient for classification and other repetitive work
👀 Qwen3.6-35B-A3B: for extraction and summarization that need stronger understanding
🔥 Air API docs: how to connect and get started
