Measuring Information Gain & Content Novelty Scores for AI Engines

Ask why a competitor’s page keeps getting cited by ChatGPT Search and AI Overviews while yours — better written, better structured — gets synthesized and skipped, and the answer is very often a single unmeasured variable: your page tells the engine nothing it did not already have.

Search systems have a name for this variable. Information gain is the amount of new information a document contributes relative to documents already seen, and it has quietly become one of the most decisive ranking inputs in generative retrieval. The good news: unlike authority or link equity, novelty is measurable at your desk, before you publish, with about thirty lines of Python. This article covers the mechanism, the measurement workflow, and the score thresholds we use in MarketLens audits.

Why Answer Engines Rank Novelty, Not Just Relevance

Classic ranking asks “is this document relevant and trustworthy?” Ten near-identical pages can all answer yes. Generative retrieval asks a harder question: “given the four sources I have already selected, does adding this fifth one improve the answer?” Redundant documents fail that test no matter how well-written they are.

Google’s patent literature describes computing an information gain score for documents based on what new information they provide beyond documents the user has already interacted with — and the retrieval-augmented pipelines behind AI Overviews, ChatGPT Search, and Perplexity apply the same economics to citation slots. An answer paragraph typically cites two to six sources; every slot spent on a me-too page is a wasted slot. Rerankers are tuned accordingly, favoring source sets with low mutual redundancy.

The GEO-bench study out of Princeton tested nine content-side tactics against generative engines and found that adding statistics and adding direct quotations were among the most effective. Both are, at bottom, the same move: injecting information the rest of the candidate pool lacks. How that plays out on one specific surface is taken up in AI Overviews versus featured snippets.

The Embedding Method: Scoring Novelty in Practice

You cannot read Google’s internal score, but you can compute a faithful proxy: semantic distance between your draft and the existing coverage an engine would retrieve alongside it.

The workflow:

1. Assemble the reference corpus. Pull the main text of the top 5–8 pages for your target query (ranking pages for classic search; cited pages if you are targeting a specific answer engine).
2. Chunk everything into ~200-word passages — the granularity retrieval systems actually operate at.
3. Embed and compare. For each passage of your draft, record its maximum cosine similarity against any corpus passage. Maximum, not mean: a section is redundant if anyone already says it, not if the average page does.

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("all-MiniLM-L6-v2")
corpus_emb = model.encode(corpus_passages, normalize_embeddings=True)
draft_emb  = model.encode(draft_passages,  normalize_embeddings=True)

sims = util.cos_sim(draft_emb, corpus_emb)          # [draft x corpus]
max_sim = sims.max(dim=1).values

for passage, s in zip(draft_passages, max_sim):
    band = "RESTATED" if s > 0.90 else "PARTIAL" if s > 0.75 else "NOVEL"
    print(f"{s:.2f}  {band}  {passage[:60]}")

Interpretation bands we apply in audits:

Two caveats keep the method honest. Very low similarity is only valuable if the passage is still on-query — novelty without relevance is noise, which is why the fourth band demands a relevance check. And small embedding models compress nuance; if two passages disagree numerically while discussing the same metric, they may still score 0.92. Treat the score as a triage tool, then read the flagged sections yourself.

Where Novelty Actually Comes From

Teams often hear “add unique information” and produce unique phrasing. Embedding models see straight through paraphrase — that is precisely what makes them good novelty detectors. Durable sources of measurable gain:

• First-party data. Your analytics, your case results, your survey. Our dental clinic case study is a worked example: Search Console clicks went from 8,122 to 18,747 across the same four months year over year, and direct contact actions from 503 to 1,135 — on a site that had been declining for two years before the work started. No other document on the web contains those numbers, which is exactly what makes the page hard to synthesize away.
• Named specifics where the consensus is vague. Exact tool versions, price points, latency figures, config flags. Each converts a generic sentence into an extractable fact.
• Disagreement with the consensus, argued. If nine sources repeat a best practice you have measured to be wrong, that section will score deep in the novel band — and engines seeking answer diversity will surface it.
• Cross-domain synthesis. Connecting two literatures that do not usually cite each other (say, RAG evaluation research and content strategy) produces passages with no close neighbor in either corpus.

Placement then decides whether the novelty gets retrieved at all. Retrieval skews heavily toward the opening portion of a document, so burying your original data in section seven forfeits much of its value — the mechanics are covered in our guide to structural extractability and positional retrieval bias.

Tracking Gain After Publication

Pre-publish scoring predicts; post-publish data confirms. Fold novelty into your reporting stack by tagging high-novelty pages in your analytics and comparing their citation performance against consensus-band pages — if you track citation share across engines using the commercial GEO metrics like share of voice and citation frequency, segment that metric by pre-publish novelty band.

That segmentation is the experiment worth running on your own corpus, because it is the only version of this claim you will be able to defend. Score a batch of pages before launch, record the band, and compare citation performance three months later. If the novel-band pages do not outperform, the threshold was wrong for your topic — which is useful to know before you rebuild an editorial process around it.

The next step is mechanical: before your next article ships, run the thirty-line scorer against its top five competitors. Cut what scores above 0.90, add one dataset nobody else has, and move it into the first third of the page. That single loop — score, differentiate, elevate — is the highest-leverage editorial habit in generative search. A MarketLens Premium Audit runs this novelty analysis across your entire existing corpus and returns a ranked differentiation roadmap.