AI Visibility Tracking: How to Measure Without Fooling Yourself
You track AI visibility by asking the same questions to the same selected AI engines on a repeating schedule, and recording, per question and per engine, how often your institution is recommended, mentioned, or listed on a map. A single check is one observation, not a trend. Honest tracking reports the engine, date, run count, and method behind every figure.
The short answer
You track AI visibility over time by fixing three things and repeating: the questions, the engines, and the schedule. Ask the same questions to the same selected engines on dated runs, and record each outcome in one of three categories: recommended in the written answer, mentioned without a recommendation, or shown as a map listing. The measurement unit is runs that named your institution out of runs made, per question and per engine. This method works whether you run it by hand in a spreadsheet or with a tool.
The traps are all in the shortcuts: changing the questions midstream, pooling different engines into one number, or treating one good answer as a trend.
Why a single check fools you
An answer engine does not hand back a fixed list. The same question, asked twice, can name different institutions, in a different order, with different reasoning. One run is one observation. It tells you what happened that time, on that engine, on that date.
This cuts both ways. A single run that names you is not proof you are visible. A single run that omits you is not proof you are invisible. What you can trust is a rate built from repeated runs: out of the runs made for this question on this engine, how many named you. Small samples still wobble, so read early numbers as directional and let the run count grow before you act on them.
The tracking method, step by step
1. Fix the question set. Write down the questions a real customer would ask an assistant. Examples: best checking account for a student near a city you serve, which credit union has the lowest auto rates in your county, where to get a small business loan with bad credit. Ten to twenty questions is a workable start. Store the exact wording. If the wording changes, the trend breaks.
2. Fix the engine list. Pick the assistants your customers actually use and write the list down. Each engine reads different sources and behaves differently, so results are only comparable within an engine. An engine you did not run gets reported as not run and stays out of every denominator. Never let a skipped engine silently count as a zero.
3. Run on a schedule and date everything. Weekly or monthly is enough for most institutions. Each run gets a date, the engine, the question, and the outcome. Dated runs are what turn anecdotes into a series you can plot.
4. Classify each outcome into one of three categories. These are distinct and should never be merged:
Recommended: the written answer names your institution as a suggestion, the way a person would recommend you.
Mentioned: your name appears, but as context, comparison, or a list entry, not a recommendation.
Map listing: you appear in a map or local pack result attached to the answer.
A mention is not a recommendation. The results page named ten is a different event from the assistant said open your account here. Track them separately or your numbers will flatter you.
5. Report with the method attached. Every figure you share should identify the engine, the date or date range, the run count, and the method. "Named in 6 of 10 runs of question 4 on Engine B, March runs" is a reproducible figure. "Our AI visibility is up" is not.
What a tracking sheet looks like
A spreadsheet is enough. One row per run, with columns for date, engine, question ID, outcome category, and whether a competitor was named instead. From that you can compute, per question and per engine, the share of runs that named you, and watch it move across months.
Two extra columns pay for themselves. First, which competitor was named when you were not, because that tells you whose content the engine prefers. Second, the sources the engine cited, when it shows them, because that tells you what to fix: a missing rates page, facts trapped in a PDF, or a third-party review site that knows more about you than your own site does.
How to read movement without fooling yourself
Compare like with like. Only compare the same question on the same engine across dates. A blended score across engines hides which engine changed and hides the method.
Demand run counts. A shift from 2 of 5 runs to 3 of 5 is noise-sized. A shift sustained across a growing run count is a signal.
Tie changes to causes. When your rate moves after you published a clear rates page or fixed your branch data, note it next to the run. When it moves with no change on your side, suspect the engine changed its behavior, which happens.
Watch the category, not just the name. Moving from mentioned to recommended is a bigger win than moving from absent to mentioned. Track category transitions, not just presence.
Where hand tracking runs out
Manual runs work for a small question set and two or three engines. The cost grows with every question and engine you add, and skipped runs quietly corrupt the series. When checking by hand runs out, Scanley measures this across the AI engines on a repeating schedule, using the same categories and the same per-question, per-engine reporting described in how Scanley measures AI search visibility.
Whether you automate or not, the discipline is identical: fixed questions, fixed engines, dated runs, three outcome categories, and honest denominators. Any institution can start this week with a spreadsheet and an hour. If you market for a credit union or a community bank, the question set is where your local advantage lives, since assistants answer at the city and county level where you actually compete.
Questions people ask
How often should I run AI visibility checks?
Weekly or monthly is enough for most institutions. What matters is that the schedule is fixed and the runs are dated, so you build a comparable series instead of a pile of anecdotes.
Can I combine results from different AI engines into one score?
No. Each engine reads different sources and behaves differently, so pooling them hides which engine changed and hides the method. Report per question and per engine, and exclude any engine you did not run from the denominator.
What is the difference between being mentioned and being recommended?
A recommendation is the written answer suggesting your institution, the way a person would. A mention is your name appearing as context or a list entry. They are distinct outcomes and should be tracked in separate categories.
Is one check enough to know if AI assistants name my bank?
No. The same question can produce different answers on repeated runs, so one run is a single observation. You need repeated runs of the same question on the same engine before the rate means anything.
Do I need a paid tool to start tracking?
No. A spreadsheet with one row per run, recording date, engine, question, and outcome category, is a working system. Tools become useful when the question set and engine list grow past what you will run by hand.