Evaluating News Personalisation
Personalisation is no longer a recommendation widget. It has become a strategy for helping readers discover more journalism, return more often, and ultimately become loyal subscribers.
For many years, personalisation in news was almost synonymous with a single box on the homepage or below an article: "Recommended for you."
That also made evaluation simple. If readers clicked those recommendations more often than a manually curated selection, related articles, or a list of popular stories, the recommender system was considered successful.
Today, that view is too narrow.
Personalisation now appears throughout the reader journey: on homepages, article pages, section pages, mobile apps, newsletters, push notifications, personalised briefings, and increasingly also in conversational and AI-powered interfaces.
That changes how it should be evaluated.
If personalisation is no longer a standalone widget, it should no longer be measured as one. Its success should be evaluated in the same way publishers evaluate any other strategy for engagement, retention, and subscription growth.
The real question is no longer:
How many clicks did the recommendation widget receive?
It is:
Did personalisation help readers discover more valuable journalism?
The Reader Journey Is the Product
A recommendation system is not a product on its own. It is one of many ways in which readers discover journalism.
Readers may find articles through editorial curation, breaking news, search, newsletters, app feeds, push notifications, regional sections, popular lists, personalised recommendations, or AI assistants. These touchpoints look different, but they serve the same purpose: helping readers find journalism that is relevant, valuable, and worth their time.
That is why personalisation should be evaluated as part of the broader reader journey, not as the performance of a single module.
The goal is not to maximise recommendation clicks in isolation. The goal is to make the publisher’s relationship with the reader more successful.
If readers consume more journalism, return more frequently, build stronger habits, and are ultimately more likely to subscribe, then personalisation has succeeded. That remains true regardless of whether the next article was discovered through a homepage module, an app feed, a newsletter recommendation, a push notification, an editorial selection, or an AI-powered assistant.
Discovery Surfaces Should Not Compete
Publishers often evaluate each discovery surface separately.
Recommendation widgets have their own CTR. Newsletters are measured by open rates and click rates. Push notifications are evaluated by opens. App feeds have their own engagement metrics. AI assistants may be measured by interactions, referrals, follow-up questions, or saved articles.
These metrics are useful, but they can also create the wrong incentive. Each surface starts competing for credit, while the real objective is shared.
A reader does not care which interface receives the credit. They simply decide whether the publisher helped them discover something worth reading.
That is why the most important question is not which module, channel, or interface generated the click. The most important question is whether the reader continued their journey with the publisher’s journalism.
From CTR to Continuation
CTR is often treated as one metric, but it can mean different things.
Item CTR measures the attractiveness of an individual recommendation.
Item CTR = clicks on an item / impressions of that item
It helps evaluate which articles readers find relevant in a specific context.
List CTR measures the performance of a recommendation list, module, feed, or newsletter block.
List CTR = recommendation list clicks / recommendation list impressions
It helps compare layouts, algorithms, and individual discovery surfaces.
Both metrics are useful, but they still evaluate personalisation in isolation.
At Froomle, we believe the more meaningful question is whether a reader touchpoint leads to valuable continued engagement.
On a website, this can be measured as Page CTR.
Page CTR = pages with at least one continuation / page views
Page CTR does not ask which widget generated the click. It simply asks whether the page encouraged the reader to continue to another article.
Whether the reader clicked a personalised recommendation, a breaking news headline, a regional story, an editorial teaser, a popular article, or any other link to another article, the page has done its job.
The same principle applies beyond the website.
In an app, the question may be whether a screen, feed, or notification leads to an article read. In a newsletter, it may be whether the email leads to a meaningful visit. In an AI assistant, it may be whether the interaction leads to a trusted article, a saved story, a follow-up question, or another meaningful action.
The exact metric depends on the interface. The principle remains the same:
Did this touchpoint help the reader continue with journalism they value?
Avoiding Cannibalisation
The advantage of continuation-oriented metrics becomes clear when different surfaces compete for attention.
Imagine moving a personalised recommendation widget higher on an article page. Its CTR will probably increase. At first sight, that looks like success.
But if those additional clicks simply replace clicks that would otherwise have gone to editorial teasers, regional stories, or the "Most Popular" module, the publisher has not gained anything. The recommendation widget looks better, but the reader journey has not improved.
This is cannibalisation.
The same problem can occur across channels. A push notification may increase app opens while reducing newsletter clicks. A personalised feed may attract attention away from editorial selections. An AI assistant may answer a reader’s question directly without leading the reader back to the publisher’s journalism.
If every surface is evaluated separately, these shifts can easily be mistaken for progress.
That is why evaluation should focus on the total reader journey. The important question is not where the credit goes, but whether the reader consumed more valuable journalism, returned more often, and developed a stronger relationship with the publisher.
Page CTR and Session Length
On websites, Page CTR has another useful property: it is closely related to session length.
Every time a page encourages a reader to continue, another page view is created. That new page again creates another opportunity to continue reading. If the probability that a page generates another page is p, then the expected session length follows a simple geometric process:
1 + p + p² + p³ + ... = 1 / (1 - p)
For the Page CTR values typically observed by publishers, the higher-order terms are relatively small:
1 / (1 - p) = 1 + p + p² + ... ≈ 1 + p
In practice, this means that Page CTR and session length measure almost the same underlying behaviour: whether readers keep finding one more article worth reading.
The practical difference is that Page CTR is easier to measure. Session length can only be calculated after a visit has ended and requires reconstructing sessions. Page CTR creates a new observation for every page view, making experiments faster, more efficient, and easier to interpret.
From Engagement to Loyalty
Publishers are not trying to maximise clicks, opens, or interactions for their own sake. They are trying to build loyal readers.
Readers who consistently discover valuable journalism stay longer, return more often, and are more likely to become subscribers. Of course, retention and subscription conversion depend on many factors: editorial quality, trust, pricing, brand recognition, habit formation, and the overall product experience all matter.
But content discovery plays an important role. If readers repeatedly find articles that feel relevant, useful, surprising, or personally important, the publisher becomes more valuable to them.
The relationship is simple:
Better content discovery
↓
More meaningful continuations
↓
Longer and more valuable engagement
↓
Higher reader retention
↓
More subscription conversions
Continuation is not the final business objective. It is an early signal that readers are finding journalism worth their time.
Our Philosophy
At Froomle, we monitor Item CTR and List CTR because they remain useful for understanding the performance of individual recommendations, modules, feeds, newsletters, and algorithms.
But we believe the most meaningful evaluation happens one level higher.
Personalisation should not compete with editorial curation, breaking news, regional journalism, popular stories, newsletters, push notifications, app feeds, or AI-powered discovery experiences. It should help the entire reader journey succeed.
That is why we believe Page CTR is one of the most useful metrics for evaluating news personalisation on websites, and why the same continuation-oriented thinking should be applied to every interface where readers discover journalism.
In the end, personalisation should not be judged by the number of clicks it generates for itself.
It should be judged by whether it helps publishers build more engaged, more loyal readers.