Evaluating News Personalisation

Personalisation is no longer a recommendation widget. It has become a strategy for helping readers discover more journalism, return more often, and ultimately become loyal subscribers.

For many years, personalisation in news was almost synonymous with a single widget on the homepage or at the bottom of an article: "Recommended for you."

Success was easy to define. If readers clicked those recommendations more often than a manually curated selection or a selection of the most popular articles, the recommender system was considered successful.

Today, personalisation has evolved far beyond a single recommendation box.

Leading publishers increasingly use personalisation throughout the reader journey. Recommendations appear on the homepage, article pages, section pages, newsletters, mobile apps, and push notifications. Personalisation has become one of the key strategies for helping readers discover relevant journalism, increasing engagement, strengthening reader loyalty, and supporting subscription growth.

That evolution fundamentally changes how personalisation should be evaluated. If personalisation is no longer a standalone feature, but an integral part of the digital news experience, it should no longer be measured as an isolated recommendation widget either.

Instead, we believe its success should be evaluated in exactly the same way publishers evaluate any other strategy designed to increase reader engagement, reader retention, and subscriber conversion.

After all, readers should not need to consciously distinguish between editorial curation, breaking news, search, newsletters, regional sections, or personalised recommendations. They simply decide whether the website helps them discover another article worth reading.

For that reason, the most meaningful question is no longer:

How many clicks did the recommendation widget receive?

Instead, it is:

Did personalisation make the news website more successful?

That shift in perspective has important consequences for how recommendation systems should be evaluated.

The Page Is the Product

A recommendation system is not a product on its own.

It is simply one of many ways readers discover journalism. Editorial curation, breaking news, "Most Popular" lists, related articles, search, newsletters, and personalised recommendations all exist for exactly the same reason:

To help readers discover another article worth reading.

Because they pursue the same objective, we believe they should ultimately be evaluated using the same business metrics.

The goal of personalisation is therefore not to maximise recommendation clicks.

The goal is to make the news website itself more successful.

If readers consume more journalism, stay engaged for longer, return more frequently, and are ultimately more likely to subscribe, then personalisation has succeeded, regardless of which navigation element or recommendation module generated a click.

Recommendations Should Not Compete With the Website

If a personalization engine could perfectly predict reader behavior, a news webpage would only need to display a single article: the exact one you are about to read next. But humans aren't algorithms. We are beautifully unpredictable. On any given day, a reader might visit a motorsport site to check F1 race results, get distracted by an editorial on vintage engines, and leave after reading a breaking news piece about a driver transfer.

Because of this unpredictability, a modern news page contains many different opportunities to continue reading. Some are editorially curated. Others are generated automatically. Readers may encounter:

• Personalised recommendations
• Continue reading
• Most popular
• Breaking news
• Regional news
• Editor's picks
• Related articles

These elements are often evaluated independently. The personalised recommendation widget is measured by its own CTR. The "Most Popular" widget is measured separately. Editorial modules have their own dashboards. Yet from the reader's perspective, these modules are not competitors. They are simply different paths towards the next interesting article.

The reader does not care how an article was discovered. The publisher should not care either. The only question that really matters is:

Did this page encourage the reader to continue reading?

The Different Meanings of CTR

This is where click-through rate becomes surprisingly subtle.

The term CTR is often used as if it refers to a single metric.

In reality, there are several different CTRs, each answering a different question.

Item CTR

Item CTR measures how attractive an individual recommendation is.

Item CTR = clicks on an item / impressions of that item

It answers a straightforward question:

How attractive is this recommendation?

This metric is extremely useful for training recommendation algorithms, comparing ranking models, and understanding which articles readers find most relevant.

List CTR

Readers never see recommendations individually.

They see recommendation modules.

A recommendation list therefore has its own CTR.

List CTR = recommendation list clicks / recommendation list impressions

It answers:

Did this recommendation module persuade the reader to continue reading?

List CTR is useful for evaluating individual recommendation widgets and comparing different layouts or algorithms.

However, it still evaluates each recommendation module in isolation.

Page CTR

At Froomle, we believe one of the most meaningful engagement metrics is Page CTR.

Instead of asking which widget generated the click, Page CTR simply measures whether a page successfully encouraged the reader to continue to another article.

Page CTR = pages with at least one continuation / page views

Every continuation counts equally.

Whether the reader clicked:

• a personalised recommendation,
• a breaking news headline,
• a regional story,
• an editorial teaser,
• a "Most Popular" article,
• or any other link leading to another article,

the page has achieved its purpose.

Page CTR therefore answers a much more fundamental question:

Did this page successfully keep the reader engaged?

Solving the Cannibalisation Problem

The advantage of Page CTR becomes clear when recommendation modules compete for attention.

Imagine moving a personalised recommendation widget higher on the page.

Its CTR immediately increases.

Success?

Perhaps.

But suppose those additional recommendation clicks simply replaced clicks that would previously have gone to editorial recommendations or the "Most Popular" widget.

The personalised widget now looks more successful.

The website, however, has not gained a single additional page view.

Nothing has changed except where the clicks occurred.

This phenomenon is known as cannibalisation.

Widget-level metrics cannot distinguish between genuine improvements and internal competition.

Page CTR naturally avoids this problem.

It measures whether the page generated another article view, regardless of which module received the credit.

A Common Objective for Every Strategy

Another advantage of Page CTR is that it allows completely different content strategies to be compared fairly.

One experiment may prioritise personalisation.

Another may emphasise recency and breaking news.

A third may give more visibility to editorially curated journalism.

Yet another may focus on regional content or popular stories.

These strategies serve different editorial purposes, but they all attempt to answer the same question:

What should the reader discover next?

Because Page CTR measures the success of the page rather than individual modules, all of these approaches can be evaluated using a common objective.

This allows publishers to find the right balance between personalisation, popularity, recency, and editorial judgement, instead of optimising each recommendation module independently.

Page CTR Is Closely Related to Session Length

Page CTR has another attractive property.

Every time a page encourages a reader to continue, another page is created.

That new page again provides another opportunity to continue reading.

Mathematically, this forms a simple geometric process.

If the probability that a page generates another page is p, then the expected session length becomes:

1 + p + p^2 + p^3 + ... = 1 / (1 - p)

For the Page CTR values typically observed by publishers, the higher-order terms become very small.

Using the Taylor expansion:

1 / (1 - p) = 1 + p + p^2 + ... ≈ 1 + p

In practice, this means that increasing Page CTR by one percentage point produces approximately the same relative increase in expected session length.

The two metrics therefore measure almost exactly the same underlying behaviour.

The important practical difference is that Page CTR is much easier to measure.

Session length can only be calculated after a reader leaves the website and requires reconstructing browsing sessions.

Page CTR, on the other hand, generates a new observation for every page view.

This makes experiments faster, statistically more efficient, and easier to interpret.

From Engagement to Loyalty

Ultimately, publishers are not trying to maximise clicks.

They are trying to build loyal readers.

Readers who consistently discover valuable journalism stay longer, return more often, and are more likely to become subscribers.

Retention and subscription conversion depend on many factors beyond personalisation, including editorial quality, trust, pricing, and brand recognition.

Nevertheless, we believe they are natural downstream consequences of consistently helping readers discover content they genuinely value.

Viewed this way, the relationship between the metrics becomes remarkably simple.

Better content discovery
            ↓
Higher Page CTR
            ↓
Longer reading sessions
            ↓
Higher reader retention
            ↓
More subscription conversions

Page CTR is therefore not the ultimate business objective.

It is an early indicator of whether readers are consistently finding one more article worth reading.

Our Philosophy

At Froomle, we certainly monitor Item CTR and List CTR.

Both remain valuable for understanding the performance of individual recommendation algorithms and recommendation modules.

But we believe the most meaningful evaluation happens one level higher.

Recommendations should not compete with editorial content.

They should not compete with "Most Popular".

They should not compete with regional journalism.

They should help the entire website succeed.

That is why we believe Page CTR is one of the most meaningful metrics for evaluating news personalisation.

It naturally avoids cannibalisation, allows different recommendation strategies to be compared fairly, closely reflects session length, and we believe it is strongly connected to the long-term outcomes publishers ultimately care about: reader retention and subscription growth.

In the end, personalisation should not be judged by the number of clicks it generates.

It should be judged by whether it helps publishers build more engaged, more loyal readers.