Does a search engine help a deck-building agent?

I recently got into Magic: The Gathering, and without knowing it signed up for a learning curve of more than 34,000 unique cards. Yes, over 34,000. Once I got the taste for it, I started looking for cards that might fit my decks, and I quickly turned to an LLM and a handful of websites full of card data to get me up to speed.

The first wall I hit was the rules. The LLM could not reliably check that the cards it wanted to add were allowed in the deck: wrong colours, cards that are not legal, cards already in the list. The second wall was taste. It kept reaching for the same popular, mainstream cards everyone already plays. So I started building structure around it: a database of every card's official text, tools the agent can call to check its picks, and a search over the text. Then the ball started rolling, and this post is where it rolled to.

01 · The dataEvery Magic card's official text: 3.1M tokens, too much to read for every task

Everything here runs on Scryfall's Oracle cards data. Scryfall is the community's card database, and it publishes a fresh copy every day. The Oracle file has one entry per distinct card, whatever the printing, with the card's current official rules text: the Oracle text that Wizards of the Coast keeps up to date. That text can differ from the printed card: older printings of Lightning Bolt say “target creature or player”, while the Oracle text says “any target”.

Each entry also carries the mana cost, type line, colour identity and legality in every format. Scryfall's Tagger project adds community labels for what cards do; the tools show them as roles.

Cards

34,646one per distinct card; tokens, emblems and art cards dropped

Rules text

3.1M tokensabout 7.5M characters across all cards

Function tags

4,577Tagger labels such as mana-rock and sweeper

Rulings

78,973official clarifications, shown by get_card

1 2 3 4 5

What search_cards returns for it

1. Lightning Bolt · {R} (MV 1) · Red · Instant · uncommon Lightning Bolt deals 3 damage to any target. roles: burn-any legal: commander ✓

• 1name
• 2mana cost, and the colour and mana value it gives
• 3type line
• 4rarity, from the set symbol's colour
• 5rules text, as the current Oracle wording
• art and flavour text: never sent
• not on the card: search rank, Tagger roles, format legality

The simplest design would skip search entirely and give the agent every card to read. I tried that.

On a 12-task sample, reading everything gave slightly better answers than search (not statistically significant). The full 113 tasks would have read about 302M tokens, so I stopped there.

02 · How the test sets are builtThree test sets, and two graders

Each grader is a separate Claude model that grades cards one batch at a time, 15 cards per call, in random order. It never sees which agent proposed a card, and it gets no popularity or price data. Hidden random cards are mixed into the batches, to check that the grader does not accept everything.

Agents and graders are Claude Sonnet 5.5 in Claude Code, with no web, file or shell access: only the tools under test. No list of correct cards and no popularity or price data ever reaches an agent. The deck-fit criteria were settled on the tuning decks before any test deck was graded.

03 · Search vs no search infrastructureWith the same instructions, search finds more of the right cards, with fewer tokens

The same agent ran the 113 agent search tasks from Section 2 three ways.

• Search: the mtg-search search tool, with minimal instructions: the request, a budget of 5 tool calls and the answer format, and nothing on how to search.
• Grep, same prompt: a regular expression over the plain card text, with no ranking and no filters, and the same minimal instructions as search.
• Grep, tuned prompt: the identical grep tool, with a prompt written after a first test run. It shows how a card is laid out in the file, explains how matching works, and adds general search advice and an example pattern. The only difference between the two grep runs is the prompt.

*Read everything: the 12-task sample from Section 1. **Two slow tasks (159 s and 322 s); the median is 11.3 s. Rescored on 15 October 2026 after a fix to card-name matching: a few answers, such as "Lightning Bolt", had been matched to a double-faced card that shares the name. Every change is under a point.

With the same instructions, grep falls behind on card attributes, such as a 1-mana red burn instant. Only 48% of its top 10 is relevant, against 100% for search. A card's mana cost, type line and rules text sit on separate lines of the file, and without a description of the layout grep guessed it wrong: 29% of its calls matched nothing. On deck needs, whose rules-text phrases sit on one line, the two stay close (95% vs 98%).

Search needed no prompt tuning. Its first prompt, never tuned, gives the same results as the minimal one: 98.6% vs 98.3% relevant cards in the top 10.

04 · Choosing the enginemLateOn reads rules text best, and the wording of the query matters most

I compared search engines on the search benchmark: 363 queries, each with a list of the cards that answer it. The score is nDCG@10 on a 0 to 100 scale; 100 means the correct cards fill the top 10, best first. mLateOn is a multi-vector text model: it matches each word of the query to the closest word in each card (the late interaction primer draws it).

So mtg-search runs mLateOn, and its tool description tells the agent to write queries the way card text is worded. The Docker image returns exactly the same top 10 as the tested search on every benchmark query, in 130 to 340 ms per search. It has three read-only tools, search_cards, get_card and find_similar, with five filters.

The agent searches only when told when to. Offered the tools without a cue, Sonnet made 0 tool calls in 42 deck tasks. Each tool description therefore opens with “Before recommending cards for a deck, call this to find candidates instead of listing them from memory.” With that sentence the agent searched in 25 of 30 tasks on the tuning decks; without it, in 10 of 30.

05 · Why one way of scoring is not enoughPlayer add rates favour popular cards, and miss a real gain in deck fit

Every automatic list of correct cards has a blind spot. The tag and pattern lists miss cards that do the job in other words: in the agent search tasks, 22 to 23% of each agent's top-10 cards were graded relevant but were missing from the lists (636 such cards among all the agents' picks). The patterns also reward exactly the wording grep can rebuild.

Player add rates have a different blind spot. They reward what many players add, and that is mostly the popular cards. A deck can also be improved in many different ways.

Commander · test deck

Auntie Ool, Cursewretch

Blight Curse, Lorwyn Eclipsed Commander 2026 · black, red, green

From memory fit 0.33

No tool calls. All 20 adds are legal.

1. 1Bloodtithe Harvester
2. 1Vraska, Golgari Queen
3. 1Toxic Deluge
4. 0Damnation
5. 0Blood Artist
6. 0Zulaport Cutthroat
7. 0Pitiless Plunderer
8. 2Overgrown Tomb
9. 2Blood Crypt
10. 1Stomping Ground
11. 1Cultivate
12. 1Kodama's Reach
13. 0Pawn of Ulamog
14. 1Deadly Rollick
15. 0Bloodghast
16. 0Jade Mage
17. 1Sheoldred, the Apocalypse
18. 0Doubling Season
19. 1Vampire Nighthawk
20. 0Rhythm of the Wild

With mtg-search fit 0.95

One search_cards call, colour identity BRG · legal in Commander:

• Put a -1/-1 counter on target creature.
• Whenever you put one or more -1/-1 counters on a creature
• Proliferate.
• Put a -1/-1 counter on each creature your opponents control

1. 2Obelisk Spider
2. 2Spitting Dilophosaurus
3. 2Shadow Urchin
4. 2Grim Affliction
5. 1High Perfect Morcant
6. 2Persistent Constrictor
7. 2Nightmare Sower
8. 2Decimator Beetle
9. 2Exemplar of Strength
10. 2Blight Rot
11. 2Scar
12. 2Contagion Engine
13. 2Soulstinger
14. 2Bile-Vial Boggart
15. 2Festering Mummy
16. 2Cultbrand Cinder
17. 2Instill Infection
18. 2Defiant Greatmaw
19. 2Blur of Blades
20. 1Grief Tyrant

From memorymedian rank 224

With mtg-searchmedian rank 13,816

101001,00010,000

most playedEDHREC rank, log scalerarely played

2clear upgrade1reasonable0no reason to be in this deck

From memory: the agent was offered the tools without any cue to use them, and never called them. With mtg-search: the server with a cue to search; in the other 60% of tasks the agent still answered from memory. Breaking a deck rule means outside the commander's colours, not legal in Commander, or already in the deck.

On the same adds, the two ways of scoring disagree. The deck-fit score shows a clear gain: better on 13 of 14 decks, and it holds on rule-abiding adds alone (0.52 vs 0.67). The match with player add rates is not significant.

The reason is popularity. With mtg-search, the agent's typical add is around the 1,000th most-played card on EDHREC, against the 300th from memory, and player add rates favour the most-played cards. The deck-fit grader shows no such pull: if anything, its top grades went to less popular cards.

In the swap task (10 one-for-one swaps), the deck-fit gain was smaller and not significant. The deck-fit grader is the same model family as the agent.

06 · Reading the resultThis is a small corpus: at scale the gap between grep and search will grow

34,646 cards and 3.1M tokens of text is a small corpus. A grep over it takes a fraction of a second, and a strong agent can page through the matches. Grep also had a second advantage: Sonnet knows Magic's rules-text templating by heart, so it writes patterns that hit.

Even so, grep matched search's relevance only with a prompt that describes the card file, and then it used twice the tokens, because its matches come unranked, in file order. That cost scales with the corpus. More documents mean more matches per pattern to page through, while ranked search still returns the best ten. In a larger collection, or one whose wording the model has not memorised, expect the token and turn gap to widen, and the quality gap with it. This is an expectation, not something measured here.

Still openWhat I would check next

• A grader from another model family, to check that Sonnet does not favour its own picks. It needs API budget.
• Fresh test decks, once players' data on them matures, around late November.
• Repeats: the share of tasks in which the agent searches swings by 20 points between identical runs.

The server, the benchmarks and the run records are private for now: Scryfall's terms forbid republishing its card data, and the deck data comes from EDHREC with permission for private evaluation only. Sources: Scryfall bulk data, Scryfall Tagger, mLateOn, MTGJSON, EDHREC. Card text © Wizards of the Coast, used under the Fan Content Policy.