PART 3: Your LLM Can Read Everything. That Doesn't Mean It Understands Your Business.
SAGE STRATEGY. POWERED BY NEXERA.
LLMs are excellent at finding patterns. Revenue decisions require another step.
A model reviews thousands of customer interactions and identifies a recurring theme. Customers who use certain language appear more likely to churn. Accounts with a particular behavior seem healthier. A support issue appears repeatedly before a renewal is lost.
The finding may be important. It is not yet reliable enough to guide a decision.
Pattern recognition tells us what to investigate. Prediction requires evidence that the pattern is consistently connected to an outcome and appears early enough for the organization to respond.
Intuition can be wrong
One Customer Success team operated for years on an assumption that felt obvious: silence from support was good news, while higher case volume was a warning sign.
When the team tested that assumption against actual renewal outcomes, the relationship was reversed. Accounts with the fewest support cases churned at a noticeably higher rate than accounts with several open cases.
The original assumption was intuitive. It was not predictive.
In this environment, a lack of support cases could indicate disengagement rather than customer health. Customers with active cases were still interacting with the company and attempting to solve problems.
This does not mean more support cases are always good. It means the signal must be interpreted in context and tested against what customers actually did next.
NIST recommends that organizations “evaluate claims of model capabilities using empirically validated methods.”
An interesting pattern is a starting point. It still needs to be tested against representative data and real outcomes.
Source: NIST, Generative AI Profile, 2024
What turns a pattern into a useful signal?
A potential signal should survive several questions:
Is it consistently connected to the outcome we care about?
Does the relationship hold across products, customer segments, and time periods?
Does the signal appear early enough for the organization to respond?
Could another factor, such as account size or lifecycle stage, explain both the pattern and the outcome?
Is the signal reliable enough to justify the cost and risk of acting on it?
Can the organization measure whether its response improved the result?
How to validate a pattern without creating a black box
Validation requires two separate tests.
The first asks:
Did the model interpret the customer information correctly?
Domain experts can review a representative group of records to determine whether the model correctly identified and classified the relevant customer language.
That review should include clear examples as well as difficult ones, such as ambiguous language, resolved issues, conflicting information, and missing context.
The organization should measure both types of error:
False alarms, where the model identifies a signal that is not present
Missed signals, where the model fails to identify something important
The second test asks:
Does the pattern actually help predict the outcome?
When historical customer and outcome data are available, this testing can begin immediately. The organization should use only information that would have been available before the renewal, churn, expansion, or other outcome occurred.
The pattern should then be tested across:
Different products
Customer segments
Lifecycle stages
Time periods
Account sizes and commercial structures
The analysis should also consider simpler explanations. A signal may appear related to churn when the real driver is customer size, product maturity, lifecycle stage, or another factor.
One useful test is to develop the signal using older data and then evaluate it against a more recent period that was not used to create it. If sufficient historical data are not available, the organization may need to monitor the signal prospectively through a renewal cycle.
Rare outcomes require extra care
Churn is usually uncommon relative to the number of customers who renew. That matters when evaluating a signal.
If only one account in 20 churns, a signal can identify most of those churn cases and still produce many false alarms if it also flags too many healthy accounts.
A signal is not useful simply because it catches risk. It must also distinguish meaningful risk from normal customer behavior well enough to support an appropriate response.
This is why organizations must evaluate both:
How many real risks the signal identifies
How many unnecessary alerts it creates
Define the response before automating it
A signal can be tested to determine whether it is connected to an outcome. That does not prove that the organization’s response will change the outcome.
To test that, the business must define what should happen when the signal appears.
Before any action is automated, the organization should establish:
The action the signal should trigger
The intended customer or business outcome
The person or team accountable for the response
When human review is required
Which errors are acceptable and which are not
How the action can be paused if it causes unintended results
Validation does not end when the signal is launched. Customer behavior changes, products evolve, and models are updated.
The organization must continue monitoring whether the signal remains reliable and whether the response improves customer and revenue outcomes.
The measure of success is not how many alerts the system creates. It is whether the intervention produces a better result.
Use qualitative and quantitative evidence together
Customer language is often the best place to discover what structured data has missed.
A repeated phrase in support notes or renewal calls may reveal a product gap, competitive threat, or value problem long before it becomes a field in the CRM or is raised directly with a CSM.
But qualitative evidence should guide the investigation, not end it.
The strongest analysis connects what customers say with what they do and what ultimately happens commercially.
One comment may be anecdotal. Hundreds of similar comments may reveal a pattern. Renewal, adoption, and revenue data help determine whether that pattern matters.
Each form of evidence answers a different question:
Qualitative data explains what customers are experiencing and why.
Quantitative data shows how common the pattern is and whether it is connected to customer and revenue outcomes.
Used together, they provide a stronger basis for action than either one can provide alone.
The goal is not more alerts
An unvalidated pattern can create noise at scale.
If every interesting theme becomes a risk alert, CSMs learn to ignore the system. If weak signals automatically trigger customer outreach, the organization can create confusion and damage customer trust.
The goal is not to produce the greatest number of insights.
It is to identify which signals are reliable enough to support a decision, define the right response, and determine where human judgment or automation should be used.
Our approach is straightforward:
Use customer language to identify what needs to be investigated. Test the pattern against customer and revenue outcomes. Act when the evidence supports it.
