What the AI Agents Caught, and What Only a Real Device Could

A scorekeeper, two stores, and $2.99

Rack Up went live on the App Store on 2026-08-17, eight days after the Android version went live on Google Play. It is a scorekeeper for tile-based word games played with physical tiles. The tiles are on the table in front of you and the app does the arithmetic: letter and word multipliers, the 50-point bonus when seven tiles go down at once, what is still unseen in the bag, and a score sheet where any turn can be corrected rather than only the last one. It is not a game and it has no board.

It costs $2.99 once. No ads, no subscription, no account, no tracking. Out of the box it makes no network requests at all, and optional word definitions are the only feature that touches the network, off by default. You can buy it on the App Store or on Google Play.

The app says a word is "not in the word list" rather than "not a word" because of which list it carries. Rather than license a tournament word list, it ships the public-domain ENABLE list, 168,501 playable words, checked entirely offline. A list that is not the official one has no business declaring anything a non-word, so it reports what it holds and the check never blocks a play. It tells you, and the table decides.

I built this on my own, with AI agents doing a large share of the engineering. What follows is not an argument for that arrangement. It is the specific record of what the agents caught, what they had no way of catching, and what the two stores did that no document warned me about.

Review works better as an argument than as a chorus

An agent asked "does this look right?" agrees with the code in front of it far more often than the code deserves. The change that made agent review worth running was making the agents refute each other instead of assess the work.

The shape is plain. Several review lanes read the same committed diff independently, each raising findings in its own lane, and every finding is then handed to three more agents whose only instruction is to argue that it is wrong. Whatever survives three attempts at refutation is what I spend my afternoon on. Running that across 37 agents before the tablet layout went up to Google produced 11 findings across four lenses, of which five survived and four were real defects introduced by that release. The six that evaporated are why the process is worth running: a review that hands you 11 things to look at, six of them noise, is one you learn to skim.

The day after that, with the first iOS build already sitting in Apple's queue, a second round put 21 agents on the whole app and confirmed 10 defects with none refuted. That round is why the submission was pulled back before anyone at Apple looked at it. One of the ten changed who won.

The defect that changed who won

At the end of a game every player loses the value of the tiles left on their rack, and whoever plays out collects the total of everyone else's leftovers. The player who went out has no rack, by definition. The app let you type one into their row anyway, and the engine then did three things with it: subtracted it from their score, pooled it with everyone else's, and paid them that whole pot plus their own leftover rack a second time on top.

Take two players. The one who went out has 15 points of tiles sitting in their row by mistake, a Q and a K. The other is holding two A tiles, worth 2. The correct adjustments are plus 2 for the player who went out and minus 2 for the other. The engine returned plus 17 and minus 2.

That is not a display glitch. It is the wrong final score, and in a close game the wrong winner, written into the saved history where nobody would think to question it. It survived my own reading of that function more than once, because the code is reasonable line by line. It did not survive execution: the finding arrived with both arrays printed next to each other, what the rules require beside what the function returned. That gap between reading code and running it is most of what I get out of agents.

The scoring lives in one block of pure functions inside the web source, and both the phone app's copy of it and the copy the tests import are generated from that block, so the fix was made once and both builds got it. The test that guards it now asserts more than the example. It asserts the property behind it: when somebody goes out, the adjustments have to sum to zero.

The contrast failure the checker could not see

The second one has a different shape, and it is the more useful lesson.

The app gives each player a colour, six in all. They were chosen against the dark theme, where the worst of them still measures 5.2 to 1 on the surfaces it is drawn on, comfortably past the accessibility floor. On the light theme, which is what the app shows unless the phone is set to dark, those same six colours rendered as text measured between 1.66 and 2.72 to 1. The floor for normal text is 4.5 to 1. Hard to read for anyone, and for some people not readable at all.

There was already an automated contrast checker in the repo, and it was passing. It passed because those pairs were not in the list of pairs it checked. A tool that measures only what you remembered to hand it will return a clean report on exactly the day you needed it not to.

The colour fix took an afternoon. The durable half was rewriting the checker to enumerate all 48 player-colour pairs, the ink on every premium fill, and the faded tile treatments whose opacity a flat pair check cannot see, and to compare the web and phone colour maps against each other so the two cannot drift apart. The defect belonged to one release. The checker is permanent, and it is why the four accessibility features I am willing to claim to Apple are ones the code can be measured against rather than ones I hoped were true. Two more the form offers are left unclaimed, because they are unfinished work rather than unfilled boxes.

What the agents could not have caught

Here is the counterweight, and without it this would be an advertisement.

Typecheck and tests are blind to entire classes of failure. React requires that hooks are called in the same order on every render, and one of the fixes from that 37-agent audit added component state in the obvious place, immediately beside the code that reads it, which happened to sit below two early returns. The typechecker had nothing to say, because the code is perfectly typed. Ninety-five tests stayed green, because the logic under test is pure and runs in plain Node, while this was React component structure. Starting a new game threw "Rendered more hooks than during the previous render".

It was found by installing that build on a device and pressing New Game. Nothing else in the pipeline could have found it, and it was one button away from being a buyer's first experience of the app.

The repo now carries a lint configuration whose entire job is that single rule, with no style rules in it at all, because I do not want to have that argument with myself. And I did not trust it until I recreated the bug on purpose and watched the linter fail on it. A guard you have never seen fail is a guard you are assuming.

The pattern under all three stories is the same. Agents are strong on failures that live inside the code: the wrong formula, the unmeasured pair, the swallowed error. They are weak on failures that only exist once the code meets a runtime, a device, or a store. That second category is still mine to find.

The store mechanics nobody writes about

Three of these appear in no checklist I found, and two of them cost a day each.

The first is the release type. App Store Connect lets you choose whether an approved version goes on sale by itself or waits for you, and I set mine to manual. I verified it. The submission tool re-asserted it seconds before filing. It reverted to automatic on its own anyway, more than once, including after the submission was in. Apple's approval would have put a paid app on sale unattended.

The only reason that did not happen is the second one, which is the setting that actually gates a launch. App Availability is a separate record, and it does not exist until you create it. For hours the version read READY_FOR_SALE while the app was live in exactly zero countries: the availability endpoint returned a 404 because no such record had ever been created, the public lookup API returned zero results, and the storefront page returned a 404 of its own. Choosing the territories, 175 of them, is what launched the app. Treat the release type as advisory and treat availability as the switch.

The third is Guideline 2.1, Information Needed, which is the questionnaire a first app from a new account draws rather than a finding against the app. Apple asked for six written answers and a screen recording made on a physical device. That recording had no way to exist, because an app with no TestFlight group cannot be installed on any device, so there was nothing to record. Creating an internal group over the API is three calls. The answers now live permanently in the review notes field, all seven of Apple's items in Apple's own order, rather than in a reply nobody can diff. Asking for exactly that is the last line of Apple's own message.

What it was actually for

I do not want to close by pretending the market is bigger than it is.

Category research settled the commercial case before a line of code was written. The leading scorekeeper of this kind is a 10,000-install app with 44 reviews. The next one down has 13 reviews. The one entry a reviewer recommended over the leader, in the leader's own reviews, has since been pulled from the store entirely. Nobody is making money here, the $2.99 price already conceded that, and no amount of craft changes the shape of a market that small.

I built it anyway, for two reasons I am willing to defend in public. The first is that it is good at the exact four things that repeat across the complaints in that category's reviews, and it gets used at my own table. The second is that a two-store pipeline is a capability, and the capability transfers: signing keys, review, pricing, privacy and accessibility declarations, staged rollout, and the dozen small settings that only announce themselves by breaking. I have now done all of it once, on a product where being wrong was cheap. The next one will be a larger bet, and it will not be the first time.

If you want to see the scoring without paying for it, the free browser version plays a full game on exactly the same code, and the scoring guide writes out the five scoring rules that start most table arguments. The browser version will hold the game you are in the middle of, and it keeps nothing behind it: no past games, no statistics that outlive a game, no head to head record. That is the difference between the free one and the paid one, and it is a line I would rather draw plainly than let somebody discover after they have paid.