The number everyone reports
Ask a sponsor how an AI or data literacy program went and you will usually hear three numbers: a completion rate, a satisfaction score and a headcount. They are easy to collect, because the learning system produces them on its own. They are also the wrong answer to the question the program was funded to answer.
The business did not pay for people to finish a course. It paid for better decisions on data, for AI tools that get used, and for fewer preventable mistakes. A completion rate says nothing about any of those. Everyone can finish the course and nobody change how they work.
Our position is simple. A literacy program is measured by what people do differently afterwards. Attendance and completion are recorded, because someone may need to see what was done, but they are not the result.
Why attendance wins by default
It is rarely laziness. Attendance wins because it is the only number that exists without planning. Three habits push programs towards it.
- No baseline. If nobody measured where each group stood before the program, there is nothing to compare against afterwards. The only defensible number left is participation.
- One course for everyone. When a director and an analyst take the same course, there is no role-specific behavior to look for, so there is nothing specific to measure. Both tune out.
- Someone else's examples. Exercises on generic data do not map to the tools people open on Monday, so the change you would want to observe never has a place to happen.
Training evaluation has had a vocabulary for this gap for decades. The Kirkpatrick Model separates four levels: reaction, learning, behavior and results. Its behavior level asks how far people perform the critical behaviors in their own environment, and its results level asks whether the initiative produced the business impact it was designed for. A satisfaction survey sits on the first level. A completion count is not on the ladder at all. The questions a sponsor actually cares about live on the third and fourth.
What to measure instead
Measure use and judgment, in the work, for each role. In practice that comes down to four measures, each taken before and after the program.
- Use. Are people using the approved tools and the governed data for the tasks the program covered? Usage records from the tools themselves are the most honest source, where your policies allow you to read them.
- Quality of work. Is the work better? For an analyst that might mean documented, reproducible analysis built on governed data. For a manager, a decision that cites the definition behind the number it relies on.
- Appropriate use. Do people know what not to do? What stays out of a prompt, when a person must decide, and when to escalate. This is judgment, and it can be checked with scenarios and by reviewing real cases.
- Confidence. Do people feel able to use the tools and to question what comes back? Self-reported confidence is weak evidence on its own. Next to the other three it is useful, because people who lack it tend to leave the tools unused.
Notice what is missing: hours delivered, seats filled, modules completed. Keep those records. Do not report them as the outcome.
Use is not the same as good use. A team that pastes customer records into an unapproved chatbot has high AI adoption and a serious problem. That is why appropriate use sits next to adoption in the measures, not underneath it.
Set the baseline before you teach
You cannot show change without a starting point. Take the baseline for each group, and score data literacy and AI literacy separately. They are different skills. A finance team can read a variance report critically and still accept whatever an assistant tells them. A product team can be fluent with AI tools and vague about the definitions behind its own metrics. The gap between the two scores, role by role, is usually the most useful finding of the exercise, and it decides what to teach first. We set out the difference in Data literacy and AI literacy are not the same skill.
The baseline also defines the after-measures. If a track is for managers who need to decide where AI belongs in a workflow, the after-measure is whether they made those decisions, with a person kept where one must stay. Agree the measures with the sponsor before any content is built. If the measures cannot be agreed, the program is not ready to design.
Our free AI & Data Literacy Assessment scores the two tracks separately and is a reasonable first look.
A measurement plan by role
Each role needs its own behaviors to look for, and its own source of evidence. The table below is a starting point, not a template to copy.
| Role | Behavior to look for | Where the evidence comes from |
|---|---|---|
| Executives | Ask for the definition behind a number; judge an AI proposal on value, readiness, risk and who is accountable | Investment papers and board materials reviewed before and after; proposals approved with a named owner |
| Managers | Use team data to decide, and know when it is not good enough; decide where AI belongs in a workflow | Decision records; workflow changes that keep a named human checkpoint |
| Analysts | Build on governed data, document what they did, and verify what AI tools return | Peer review of real work; analysis built on governed sources; documentation present |
| Frontline | Enter and read data correctly at the point of work; use approved tools safely | Data-quality checks at entry; tool usage; scenario checks on what never goes into a prompt |
The behaviors follow the role competencies. Which evidence you can collect depends on your policies and on what your systems record.
When to measure, and what to do with it
Measure at three points: the baseline, shortly after each track ends, and again once enough time has passed for the habit to stick or fade. The last measurement is the honest one. Behavior that lasts only while the trainer is in the building was not learned.
Then act on what you find. Where use lags, reinforce it: a short follow-up on the specific task, a champion inside the team, or a fix to the tool access that was quietly blocking people. Where appropriate use lags, treat it as a risk to manage, not as a training statistic.
Keep the evidence throughout: baselines, role competencies, records and results. That record lets a leader show what was done and what changed. It does not, by itself, make an organization compliant with any rule. Whether it meets a particular obligation is a legal question, and we do not give legal advice. If you operate in the EU or in Illinois, our guides to the EU AI Act and to Illinois AI law set out what each covers.
A checklist for sponsors
Before you approve a literacy program, or the report at the end of one, ask:
- Is there a baseline for each group, with data and AI scored separately?
- Were the after-measures agreed in writing before content was built?
- Does each role have its own behaviors to look for?
- Are exercises run on your own tools, data and policies?
- Is appropriate use measured alongside adoption?
- Is there a follow-up measurement after the habit has had time to stick or fade?
- Is there a named owner for reinforcement where use lags?
- Is the evidence kept and handed over to you, not left inside a vendor's dashboard?
If most of the answers are no, the program will report attendance, because attendance will be the only number it has.
