The pipeline is the test

I have been working on a side project built on the Riot Games API. The API gives you a lot of data: match history, champion stats, and performance metrics across dozens of dimensions. What it does not give you is a direct answer to the question I actually cared about: how strong is a champion right now compared to the rest of the roster?

Raw win rates are skewed by sample size. Damage totals do not mean much until you account for role and game length. Popularity skews almost everything. A champion played ten thousand times in a patch and one played forty times should not be compared on raw numbers.

This is the same problem as sorting products by star rating. An item with two five-star reviews will beat one with two thousand reviews averaging 4.8 if you sort by the raw average. The usual fix is to pull small samples toward the overall average until they have enough data to earn their distance from it. A/B tests have the same trap: a variant that looks like a clear winner after a day is often just noise that has not settled yet.

The pipeline starts from a seed list of players pulled from the top three ranked ladders: Challenger, Grandmaster, and Master. That list came to 7,283 players. For each player, the collector fetches their last twenty ranked solo games, then the full details and timeline for each match. Raw payloads are stored separately from the cleaned, metric-ready data, so I can rerun calculations when I change a definition without downloading anything again. A computation layer groups matches by patch and rank and aggregates stats per champion and role, and the leaderboard and tier list both read from that derived layer.

Rate limits shaped most of that design. Every match takes two requests, one for the details and one for the timeline, and the collector waits two seconds between requests and backs off when the API returns a 429. A fixed delay is cruder than reading the rate limit headers Riot sends back, but it was predictable and easy to reason about at this volume. It also makes the math simple: about forty requests per player, which means collecting the full seed list would take roughly a week of nonstop requests. I collected from the first handful of players instead, which gave me 284 matches across three patches.

At that cost, refetching was never an option, so keeping the raw payloads was what let me change my mind about how a metric was calculated. Most web apps that sync from a third-party API end up in the same place, whether the source is a CMS, a payment provider, or GitHub. The remote API is slow, rate limited, and not under your control, so you keep your own copy of what it returned and build everything else on top of it. A table of untouched responses and a script that can rebuild everything downstream covers most of the value.

My mistake was spending too long on ingestion before connecting it to anything else. The incoming data looked clean and the normalization logic seemed right, but I had no way of knowing whether the output meant anything until the whole thing ran end to end, from ingestion through computation to a UI I could actually read.

As soon as I did that, problems I could not have found in isolation became obvious. Some champions the pipeline ranked highly were ones any regular player would tell you were weak that week. Some fields meant something different from what I had assumed. And the sample sizes were much thinner than the totals suggested.

Roles were the clearest field problem. The API labels the support position UTILITY, while my stats structure expected SUPPORT. Nothing failed. Support games quietly collected in their own bucket, and the support column stayed empty. Types would not have caught it either, since a typed client only confirms the value is a string, not what it means. The fix was to stop translating. The data now keeps the API's own value through every layer, and the UI is the only place UTILITY becomes "Support". That leaves one vocabulary in the data and one place where display labels are decided.

Rank was a tradeoff rather than a bug. Each match is tagged with the rank of the seed player who led me to it, not the rank of everyone in the lobby. Looking up all ten players would have added nine requests to every match, taking it from two requests to eleven. At this scale that was not worth it, as long as "Master tier" is read as games a Master player was in. In practice the question did not come up, because every match in the dataset came from a Challenger player's history. Analytics has the same issue with attribution: a conversion gets credited to whichever touchpoint you happened to record, and the label starts to look like a fact once it is on a dashboard.

The sample size problem was the one that mattered most. 284 matches sounds like a reasonable start, but it is 2,840 champion picks spread across a roster of about 170 champions, five roles, and three patches. The earliest patch had five matches. Most champion and role combinations had a handful of games, and a champion that won three of its four games could land at the top of the tier list.

I have not added sample size adjustment yet, and I think that is the right order. At this size, pulling small samples toward the average would move almost every champion to the middle, and the result would look precise without being any more true. The next step is a minimum game threshold that shows "not enough games" instead of a score, with proper smoothing once there is enough data for it to change anything.

None of these problems lived in a single layer. Each layer could have passed its own tests and the system would still have been wrong, because the bugs were in the assumptions between layers. That is the case for building a thin walking skeleton of every layer before making any one of them good, and it is why integration tests and preview deployments catch problems that unit tests do not.

The most useful test I had was a person who knows the game looking at the output. In web work, that is the editor who notices a dashboard number is off, or the support person who knows which feature customers actually complain about. That knowledge is worth turning into automated checks. One that would have caught several of these issues early: every game has five winners and five losers, so the win rate across all picks in a patch has to come out to exactly fifty percent. If it does not, something upstream is dropping or double counting games. Most systems have an invariant like that, such as totals that should match or counts that should never go negative, and they are cheap to check on every run.

I would connect every layer in the first week, before ingestion felt finished, and add the fifty percent check at the same time. I would also decide up front which questions the dataset is big enough to answer. 284 matches was plenty to find bugs in the pipeline, and nowhere near enough to say which champions are strong. If I extend the project, the order is clear: a minimum game threshold first, then a seed list sampled across every rank instead of only the top ladders, and only then the smoothing that the tier list needs to be trustworthy.