Where Are the Values?

A Systematic Literature Review on

News Recommender Systems
CHRISTINE BAUER, Paris Lodron University Salzburg, Salzburg, Austria
CHANDNI BAGCHI, Utrecht University, Utrecht, Netherlands
OLUSANMI A. HUNDOGAN, Utrecht University, Utrecht, Netherlands
KARIN VAN ES, Utrecht University, Utrecht, Netherlands

In the recommender systems field, it is increasingly recognized that focusing on accuracy measures is limiting
and misguided. Unsurprisingly, in recent years, the field has witnessed more interest in the research of values
“beyond accuracy.” This trend is particularly pronounced in the news domain where recommender systems
perform parts of the editorial function, required to uphold journalistic values of news organizations. In the
literature, various values and approaches have been proposed and evaluated. This article reviews the current
state of the proposed news recommender systems (NRS). We perform a systematic literature review, analyzing
183 papers. The primary aim is to study the development, scope, and focus of value-aware NRS over time.
In contrast to previous surveys, we are particularly interested in identifying the range of values discussed
and evaluated in the context of NRS and embrace an interdisciplinary view. We identified a total of 40 values,
categorized into five value groups. Most research on value-aware NRS has taken an algorithmic approach,
whereas conceptual discussions are comparably scarce. Often, algorithms are evaluated by accuracy-based
metrics, but the values are not evaluated with respective measures. Overall, our work identifies research gaps
concerning values that have not received much attention. Values need to be targeted on a more fine-grained
and specific level.
CCS Concepts: • General and reference → Surveys and overviews; • Information systems → Recom-
mender systems; • Human-centered computing;
Additional Key Words and Phrases: Recommender systems, human values, systematic literature review, news
recommendation
ACM Reference Format:
Christine Bauer, Chandni Bagchi, Olusanmi A. Hundogan, and Karin van Es. 2024. Where Are the Values?
A Systematic Literature Review on News Recommender Systems. ACM Trans. Recomm. Syst. 2, 3, Article 23
(June 2024), 40 pages. https://doi.org/10.1145/3654805

This publication has been supported in part by AI Labs Utrecht University in collaboration with DPG Media. Additionally,
it has been supported by the EXDIGIT (Excellence in Digital Sciences and Interdisciplinary Technology) project, funded
by Land Salzburg under grant number 20204-WISS/263/6-6022.
Authors’ addresses: C. Bauer (Corresponding author), Department of Artificial Intelligence and Human Interfaces, Paris
Lodron University Salzburg, Salzburg, Austria; e-mail: [email protected]; C. Bagchi and O. A. Hundogan, Utrecht
University, Utrecht, Netherlands; e-mails: [email protected], [email protected]; K. van Es, Department of
Media and Culture, Utrecht University, Utrecht, Netherlands; e-mail: [email protected].

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs International 4.0

License.
© 2024 Copyright held by the owner/author(s).
ACM 2770-6699/2024/06-ART23
https://doi.org/10.1145/3654805

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
23:2 C. Bauer et al.

1 INTRODUCTION
Recommender systems (RS) pervade our everyday life: Many online platforms integrate such sys-
tems to help users discover relevant items such as movies [46], fashion [45], jobs [111], or social
matching [212]. Essentially, RS are a means to help users deal with information and choice over-
load [9] by recommending items that might be interesting to the user; often, such recommenda-
tions are personalized to the user [196, 198].
While the optimization of accuracy in RS has been a long-standing focus—thus, increasing a
recommender algorithm’s performance in accurately predicting a user’s rating—there is growing
awareness that relying solely on accuracy metrics is restrictive and misguided [75, 146]. In recent
years, the RS field has witnessed more interest in research that goes “beyond accuracy” [3, 75, 103].
In fact, Kaminskas and Bridge [103] identified a shift in RS research towards including beyond-
accuracy objectives. Their survey demonstrates that the most extensively studied and integrated
beyond-accuracy objectives include diversity, serendipity, novelty, and coverage. These, we sug-
gest, can thus be considered the “standard” beyond accuracy values in RS research. Then, later,
the focus extended beyond just those standard beyond-accuracy metrics towards a broader range
of values. For instance, using the term “value-aware RS” [32], research has paid attention to the
business value of RS [98]. Often, such works focus on optimizing the economic value of recom-
mendations by balancing the interests of multiple stakeholders [1]. For a recent literature review
on value-aware RS, see De Biasio et al. [52]. However, this strong business orientation does not
necessarily embrace the wide spectrum of values beyond economic and utility aspects. Interest-
ingly, works considering and investigating a wide range of values in RS rarely use the term “value-
aware.” Instead, these works typically (only) specify those values they are concentrating on (e.g.,
privacy [99], fairness [61], trust [34]) or subsume some values under other overlapping umbrella
terms and concepts (e.g., ethics [150, 225]).
Paying attention to values when designing systems is not restricted to RS and is not a recent
idea at all, as it traces back to the 1980s and earlier. Algorithms are often perceived as objective
procedures for solving problems [125]. However, adopting this technical perspective overlooks
the fact that algorithmic systems are socio-technical in nature [206]. Culture and cultural nuances
play an important role in how and why these systems function as they do [206]. In other words,
technologies reflect the values of the cultures in which they are made [36, 143, 249]. This recogni-
tion in the 1980s and 1990s laid the foundation for the development of approaches such as Value
Sensitive Design (VSD) [66–68]—a concept that was popularized in the information systems and
human-computer interaction fields. VSD centers on the engagement and balancing of human val-
ues in the design process of technologies. In this context, the term “value” has been broadly defined
as “what a person or group of people consider important in life” [68].
Within RS research, news is a specific domain in which values have received considerable atten-
tion. In part, this attention is because news recommender systems (NRS) are a part of the editorial
function of news organizations and need to uphold journalistic values [218]. With the increasing
spread of false and misleading information (“fake news”) [8, 262], the demand for considering and
acknowledging journalistic values has become louder and more evident. This example illustrates
that the importance of certain values is also domain-dependent. For instance, journalistic values
are a crucial cornerstone in the news sector but are less relevant in other areas (e.g., in games).
Moreover, (relevant) values are not only domain-specific but can also be specific to an organiza-
tion or product. As Bastian et al. [14] point out,
“[...] value interpretations and prioritization can vary between news organizations and
even individual practitioners. An important implication of this finding is that respon-
sible, value-aware use of and implementation of [news recommender systems] require

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
Where are the Values? A Systematic Literature Review on News Recommender Systems 23:3

news organizations to engage internally in an organization-wide process of identify-

ing their core values with regard to news recommender [systems’] use, which can
subsequently inform their strategies to achieve value-sensitive design.” [14, p. 855]
Given that values can vary by domain and may be specific to an organization or product, it would
be expected that academic discourse and practice would encompass a broad range of values and
variations of approaches in NRS.
As previous works (e.g., References [14, 25, 88, 90, 155, 214]) stress, values are essential in the
journalistic process. Indeed, as we demonstrate in this article, a range of values and methodologies
has been proposed and assessed within the academic discourse on NRS. However, while knowledge
about a wide set of values expands, these are scattered across papers and research communities.
We tackle this research gap through a systematic literature review, analyzing and synthesizing
183 papers on NRS. We seek to trace and reflect on the scale, research fields, and range of values in
papers on recommender systems within the news domain.
The systematic review offers three key contributions: First, from our analyzed corpus, we identi-
fied a total of 40 values and developed a categorization scheme to group these values into five value
groups. Second, our review synthesizes the body of research on value-aware NRS, tracing its devel-
opment back to 1995. This includes an overview of the research approaches and metrics employed
in this research, not only on a general level but also in relation to specific value groups and indi-
vidual values. Third, we highlight the prolific authors and author teams on value-aware NRS. Our
work’s novelty lies in its focus on value-aware NRS and embracing an interdisciplinary perspective.
The remainder of this article is organized as follows: In Section 2, we examine related work. In
Section 3, we discuss how we selected and categorized existing research for consideration in our
review. Section 4 provides insights into the development, scope, and focus of value-aware NRS
over time. Section 5 offers a discussion of the identified trends. The conclusion section explores
potential avenues for future research (Section 6).

2 RELATED WORK
In this section, we start by discussing the specifics of the news domain concerning the integration
of values (Section 2.1). Subsequently, we provide a brief overview of research on NRS (Section 2.2)
and discuss the motivation to target objectives “beyond accuracy” (Section 2.3).

2.1 Specifics of the News Domain

In the news domain, the role of values in RS has attracted considerable attention. This specific focus
is largely due to the recognition that news plays a crucial role in supporting democratic functions.
As such, algorithmic personalization has sparked much concern within this domain about so-called
“filter bubbles” [174] and “echo chambers” [223]. Some raise concerns that news recommenders
might exacerbate political divisions among individuals and potentially harm the development of an
informed public. Evidence supporting the filter bubble hypothesis is, however, limited [31, 149]. For
example, research by Nechushtai and Lewis [164] found that users from various states and political
leanings were recommended similar news items, undermining the idea that algorithms necessarily
create echo chambers. Nonetheless, they observed a high degree of homogeneity and concentration
in the news recommendations, indicating that popular news providers are reinforced in popularity.
This observation raises an important question about the desired role of NRS.
Bastian et al. [14] interviewed media practitioners (e.g., journalists, data scientists, and product
managers) from quality newspapers in the Netherlands and Switzerland to gain insights into how
they perceive algorithmic NRS and to understand what values they consider important in the de-
sign of these systems. The study revealed that media practitioners believe that NRS should not

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
23:4 C. Bauer et al.

be exempt from upholding journalistic values. For them, it is important that values such as trans-
parency, diversity, editorial autonomy, a broad information offer, personal relevance, usability, and
surprise are taken into consideration in how these systems are designed and implemented [14]. The
news organization that Lu et al. [136] worked for participated in the research of Bastian et al. [14].
They conducted further research seeking to identify values that were both desirable and technically
feasible to implement in these systems, identifying two: (i) timely and fresh content and (ii) sur-
prising readers. The former was modeled as dynamism and the latter as serendipity. Importantly,
Lu et al. [136] demonstrated that introducing dynamism into NRS can be achieved without sac-
rificing accuracy. This finding challenges the common assumption that incorporating values into
such systems necessarily involves a tradeoff. Instead, their study suggests that pursuing multiple
objectives simultaneously in NRS is feasible.

2.2 News Recommender Systems

NRS have been the subject of several review papers. Our work stands apart from these previous
studies in several key aspects: (i) Focus on value-aware NRS: To date, no other literature review
focuses on value-aware NRS specifically. (ii) Little overlap of references: The overlap of cited ref-
erences with other systematic reviews of news recommenders is low: 16% of our references also
appear in Karimi et al. [106], 14% in Raza and Ding [195], 12% in Mitova et al. [151], and 7% in Feng
et al. [64]. The overlap of references with other survey papers is marginal (e.g., De Biasio et al. [52]
3%, Özgöbek et al. [171] 3%, Li and Wang [131] 2%, Qin and Lu [190] 2%, Borges and Lorena [26]
< 1%, and Dwivedi and Arya [60] < 1%). Our review article features 158 references (i.e., 59% of
our references) that have not been covered by any of the aforementioned survey papers. (iii) Inter-
disciplinary view: Our literature review covers references investigating NRS from different angles,
including papers from computer science and journalism alike.
Most literature reviews on NRS specifically review the underlying algorithmic approaches [26,
128, 171]. Karimi et al. [106] additionally focus on empirical evaluation and the users’ perception
of the systems. As Raza and Ding [195] point out, these reviews generally take the perspective
of computer scientists (e.g., References [26, 60, 64, 106, 109, 131, 171]). A notable exception is a
more recent review by Mitova et al. [151] that takes a political communication perspective on
NRS, synthesizing findings concerning journalistic distribution and audience acquisition of politi-
cal information for democracy and identifying research gaps. In contrast to these previous reviews,
our review takes an interdisciplinary approach. Our primary goal is to examine the development,
scope, and focus of value-aware NRS over time. We specifically aim to identify and scrutinize the
spectrum of values discussed and incorporated in NRS research. Additionally, we investigate the
metrics employed to optimize and assess these values within the systems.
Our study, which concentrates on elements beyond accuracy, aligns closely with surveys by
Karimi et al. [106] and Raza and Ding [195]. Karimi et al. [106], in their survey of 140 papers
published between 2005 and 2016, examine the general challenges, algorithmic approaches, and
methodological issues related to the evaluation of NRS. They observe a steady increase in the
number of papers on NRS throughout this period, suggesting that it has become an important
subtopic within RS research. While their findings indicate that the primary optimization goal in
NRS research is to accurately predict relevance for news readers, Karimi et al. [106] note that this
approach is often not optimal and explain this by providing vivid examples:
“If, for example, a user is interested in politics and has shown interest in articles about
an ongoing presidential election in the past, recommending more articles about this
topic is probably a good choice. However, recommending solely articles about the elec-
tion, or solely about politics, might be too monotonous for users and would probably

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
Where are the Values? A Systematic Literature Review on News Recommender Systems 23:5

not lead to high user engagement in the future. In case of news aggregation site, it is
furthermore important that the recommended news are not too similar to each other.
Presenting three articles from three different sources about, e.g., the same plane acci-
dent might be of little value for users.” [106, p. 1209]
Given the limitations of an accuracy-centric perspective, it has become increasingly important
to consider other quality aspects in NRS research [75, 106, 195]. To balance accuracy, it is crucial to
consider quality aspects such as diversity, novelty, and serendipity alongside traditional accuracy
metrics. These qualities are often discussed as beyond-accuracy aspects in the broader RS litera-
ture [75, 103]. Karimi et al. [106] note that from around 2011 onward, a growing number of NRS
papers consider beyond-accuracy aspects. However, their work also emphasizes that much work
still needs to be done [106]. Significantly, they raise an important critique of the work being done,
“while some papers take aspects like diversity or novelty into consideration in the de-
sign process of their algorithm, they do not explicitly quantify any improvements w.r.t.
these aspects with standard metrics in their experimental evaluation.” [106, p. 1214]
This highlights the importance of ensuring that the measures used in RS align with the intended
goals [234].
Raza and Ding [195] seek to broaden this perspective by conducting a survey that not only
examines the technical aspects of NRS but also investigates the effects of these systems on user
behavior. Additionally, they explore the development and application of deep learning in the news
domain. They also highlight the importance of using beyond-accuracy aspects in evaluating the
quality of news recommenders,
“typical accuracy-centric approaches may fail to consider other aspects of user experi-
ences (such as choice satisfaction, perceived system effectiveness, better recommenda-
tions, and exposure to different points of view) when evaluating the recommendation
quality.” [195, p. 3]
Their survey finds that accuracy remains a standard evaluation measure for the quality of NRS.
Furthermore, they also conclude that although some research has been done on diversity [195,
p. 16], a very limited number of works investigate novelty, coverage, and user experience.
In the following section, we discuss beyond accuracy more broadly in RS literature.

2.3 Beyond Accuracy

As mentioned, optimizing accuracy has commonly been the primary goal in RS research [98]. Typ-
ically, RS research has relied on a standard set of accuracy-based metrics, including Precision and
Recall [21, 84], to evaluate a recommender systems’ success. Already in 2004, Herlocker et al. [92]
wrote about matching the evaluation of RS to user needs. They postulated that recommendations
should not just be accurate but also useful (e.g., recommending bananas to people in grocery stores
is too obvious to be useful) and claim, “[we] need comprehensive quality measures that combine
accuracy with other [aspects such as] serendipity and coverage, so algorithm designers can make
sensible trade-offs to serve users better” [92]. Importantly, there are many aspects of user satisfac-
tion that accuracy-based metrics are unable to measure [267].
McNee et al. [146] later went as far as to claim that the narrow focus on improving accuracy
in RS has actually hurt the field. They argue that having a high level of accuracy in an RS does
not necessarily mean it is effectively aiding users in discovering items that genuinely interest
them. They provide the need for a user-centric perspective that is pleasurable rather than helpful
or simply accurate. Adamopoulos [3] underscores their plea, stating that many existing RS have
focused “on providing more accurate rather than more useful recommendations.”

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
23:6 C. Bauer et al.

As outlined above, it is increasingly acknowledged that values other than accuracy play a signif-
icant role in improving the overall quality of an RS. Kaminskas and Bridge [103] survey the most
widely discussed beyond-accuracy objectives: diversity, serendipity, novelty, and coverage. Diver-
sity in recommendations ensures that the recommendations are not too similar. Kaminskas and
Bridge [103] argue, with reference to the field of information retrieval, that diversifying retrieval
results could potentially lead to increased user satisfaction. This is because an exclusive focus
on maximizing retrieval accuracy may result in too-similar recommendations. They argue that
sometimes accuracy needs to be sacrificed for increased user satisfaction. Serendipity in recom-
mendations allows users to encounter in unplanned ways what they find interesting [22], whereas
novelty denotes items previously unknown or new to the user. Finally, coverage concerns the ex-
tent to which recommendations cover the full range of available items in the catalog.

3 METHODS
With this study, we seek to understand which values have been addressed in RS research in the
news domain and when and how they have been discussed. To do so, we rely on a systematic
literature review [114].
The main motivations for conducting a systematic literature review on value-aware NRS are as
follows: Papers on value-aware NRS are scattered across a wide variety of outlets with different
aims, scopes, and target audiences across various research communities. A systematic literature
review is a promising method for rigorously synthesizing the existing body of knowledge on a
well-defined topic [116, 122, 230]. Furthermore, a systematic literature review’s explicit, rigorous,
and reproducible procedure allows to reduce biases [19, 116, 230]. As such, a systematic literature
review is a natural choice to target our research goal.
For the literature review, we systematically searched for papers on NRS, narrowed the scope
to papers concerned with values, and analyzed the research landscape. Figure 1 illustrates the
procedure, which we describe in the following subsections. First, we detail the literature search
(Section 3.1) and the corresponding criteria for selection (Section 3.2), followed by an explanation
of the coding process (Section 3.3).

3.1 Literature Search and Criteria

For the literature search, we followed the systematic literature review procedure according to the
guidelines by Kitchenham et al. [114]. The search strategy to identify papers to be included in our
sample consisted of several consecutive stages, illustrated in Figure 1.
First, we performed a scoping review of relevant published literature to develop an effective
search strategy. From a comparison of search results using the databases Scopus, ACM Digital
Library, Wiley Online Library, EBSCO, Web of Science, IEEE Xplore, and WorldCat, all of which
contain papers relevant to technology and computer science, we concluded that the search results
of Scopus also contain the papers from the other databases. Springer Link turned out to be inef-
ficient for our research, because it mostly produced results that were outside our project’s scope,
which was detrimental to the search; for this reason, it was omitted from the search. Beyond the
technology and computer science angle (e.g., papers appearing in conference proceedings of Rec-
Sys or SIGIR), the search in Scopus also resulted in relevant papers taking a news and journalism
perspective (e.g., papers in the journal Digital Journalism) and embracing a broader scope of dig-
ital sciences (e.g., papers appearing in the conference proceedings of CHI, CHIIR, or Hypertext).
Accordingly, we sampled papers found in Scopus, where we searched in an unspecified time frame.
As our literature review explicitly focuses on NRS, we searched for papers indexed with the
keywords news recommendation or news personalization considering spelling variations. Thus, we
searched for the search terms news recommend*, news personal*, or personali* news in the title or

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
Where are the Values? A Systematic Literature Review on News Recommender Systems 23:7

Fig. 1. PRISMA diagram detailing the paper selection process.

the keywords. The search string was determined in a process of trial-and-error in which we tried
various combinations. We compared the number of results and the relevancy of the results per
search string. The search string that provided papers relevant to our research was finally selected.
The query syntax looks as follows1 :

( TITLE ( "news recommend*" OR "news personali*" OR "personali* news" )

OR
KEY ( "news recommend*" OR "news personali*" OR "personali* news" ) )
AND
( LIMIT-TO ( SRCTYPE , "p" ) OR LIMIT-TO ( SRCTYPE , "j" ) )
AND
( LIMIT-TO ( DOCTYPE , "cp" ) OR LIMIT-TO ( DOCTYPE , "ar" ) )
AND
( LIMIT-TO ( LANGUAGE , "English" ) )
We chose to search for English-language conference papers and articles in journals and confer-
ence proceedings. As a result of our query on 19 April 2022, Scopus rendered 609 papers.

3.2 Data Cleansing and Selection of Papers for the Sample

As the aim of this review was to identify values beyond accuracy within the news domain, the
four authors investigated the 609 retrieved papers and reviewed them against the inclusion and
exclusion criteria described below.

1 DOCTYPE indicates the document type, which is conference paper (cp) and article (ar). SRCTYPE indicates the source,

which are journal (j) and conference proceedings (p).

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
23:8 C. Bauer et al.

The criteria for inclusion were that the papers must report on values other than accuracy. A
paper was considered out of scope (and, thus, excluded) if any of the following criteria was met
(exclusion criteria):
— The paper’s core contribution was about the production of news articles.
— The paper’s core contribution was about evaluation methods of NRS.
— The document resembled a project proposal.
— The document was a dissertation.
— The document was a collection of conference papers (e.g., workshop proceedings).2
Reasons for these exclusion criteria reflect that our review focuses on news recommendations
to news consumers rather than providing support for journalists and editors in the news article
creation process. Further, papers describing evaluation methods in general do not contribute to our
work’s focus, namely, values in NRS. Moreover, we consider only peer-reviewed research papers
as a quality criterion; this excludes research proposals and editorials (e.g., editorials to conference
proceedings). This also refers to dissertations, which undergo a peer-review process similar to re-
search papers in conferences and journals; the dissertations’ contents are frequently also published
as research papers, which are part of the sample.
The four authors screened the retrieved 609 papers against these criteria by examining titles,
abstracts, and of the main text, mainly the “Results” sections. For this task, the papers were divided
among the authors, sorted in chronological order, and split into two halves; one half consisted of
papers published before 2016 and the other half published after 2016. Two authors screened one
half of the papers independently from each other, while the other two authors screened the other
half independently. Disagreement about excluding certain papers was first resolved among the
two authors assigned to the respective bulk of papers and subsequently discussed with all four
authors to reach a unanimous consensus. This procedure led to the exclusion of 365 papers. The
exclusion of many sources was based on the criterion that they did not pertain to values but instead
concentrated solely on accuracy or click-through rate, which did not align with the focus of our
research. After this process, 244 papers were left.

3.3 Review of the Selected Papers in Full Text (Coding)

The four authors reviewed the 244 papers in full text.
The following coding scheme was developed inductively from raw data:
— Type of paper (algorithmic work, conceptual, user experiment, interview(s), review)
— Additional details about the previous category (type of evaluation)
— Domain (financial, sports, etc.), if applicable
— Platform (social media, Twitter, video, mobile app, etc.)
— Problem statement
— Datasets
— Values
— Metrics
The four authors categorized the papers according to the coding system. Similar to the screening
process mentioned above, we divided the papers among the authors. This time, the papers were
sorted alphabetically by the surname of the paper’s first author before being divided into two
halves. We altered the pairings of authors; two authors coded one half of the papers, and the other
two authors the other half, again independently from each other. Subsequently, the coding was

2 Note that proceedings and editorials to proceedings were excluded. Papers in such proceedings that were retrieved in our

search—and met the inclusion and exclusion criteria—were included.

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
Where are the Values? A Systematic Literature Review on News Recommender Systems 23:9

discussed, and conflicts were resolved to reach a unanimous decision. In this process, a duplicate
was found, 3 papers were not accessible,3 and 55 were found irrelevant. (For example, Usher [232]
reports on empirical research concerning how start-ups in the news domain differ from traditional
journalism; thus, not focusing on recommender systems. Said et al. [203] describes a production
recommender system infrastructure that allows research systems to be evaluated in situ, as an
effort to move evaluation methodology forward.) Finally, 183 papers remained, making up our
final corpus for analysis.
When coding the papers, we identified a total of 40 values. The coding process revealed that
some values are closer to each other than others. For instance, some value codings required dis-
cussions among the raters to reach unanimous decisions. Based on this observation and given the
large number of different values, it appeared adequate to aggregate values into categories. Sub-
sequently, three authors aggregated the identified values into five categories in a joint iterative
process. Each of the 40 identified values was written down on a sticky note. The authors sat to-
gether in this process so all had a fresh reminder and an overview of the identified values. The
sticky notes were put next to each other on a big table so all were visible; we refer to this as the
pool of values. Then, taking turns without a specific order, the team members relocated sticky
notes to the table’s lower end to group them if they were considered similar.
While doing so, the team member explained why those values were considered similar to each
other (or similar to the already grouped ones). The sticky notes were left grouped if the other
team members (temporarily) agreed. If they disagreed, then an explanation for disagreement was
provided, and the moved sticky notes went back to where they had been before or back to the pool
of values if the group did not get support from at least two team members anymore. This process
was iterative, with some sticky notes being moved back-and-forth between groups and the pool
multiple times. Based on the grouping explanations, the team created a label for the respective
group. The respective label was written down on a separate sticky note in a different color and put
next to the sticky note group. This, too, was an iterative process, and several groups were relabeled
several times. All sticky notes remained visible throughout the session. With some values and
value groups, this led to many discussions, which—ultimately—also resulted in the emergence of
subgroups. For example, there was heavy discussion about whether the values in the now-named
value group “responsible agency” should form a separate value group or be listed among the values
of the “responsibility” group. The unanimous decision was that the values “agency,” “autonomy,”
and “future impact” should form a subgroup of the value group “responsibility,” as these have the
underlying theme of giving a person agency. Furthermore, this process also led to the merging of
values. For instance, “censorship” and “instrumentalization” and “propaganda” were merged into
“objectivity”; “shifting user interests” and “interests over time” were merged into “temporality of
interests.” In addition, the label for value group “editorial values” emerged after intense discussion.
First, we tended to label this value group “journalistic values.” However, as “journalistic values”
forms a distinct value within the very same value group and other authors (e.g., References [136,
218]) subsume several values under “editorial values,” we chose for the “editorial values” for the
group label. We detail the identified values in Section 4.3, where we also present the categorization
scheme (Figure 4).
The coding scheme allowed for coding a paper for multiple attributes within a category. For
instance, a paper may discuss five different values; hence, all were coded. Also, regarding the type
of paper, multiple codings were possible. However, almost all papers were only one type (e.g.,
conceptual work). There were only 7 papers that have been double-assigned in terms of their
paper types: Epure et al. [62] and Li et al. [129] are categorized as algorithmic work and analysis;

3 We contacted the authors but have not received a response.

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
23:10 C. Bauer et al.

Fig. 2. Number of general NRS papers and value-aware NRS papers over time with fitted trend.

Viana and Soares [236] and Jain et al. [96] are characterized as both algorithmic work and user
study; Bastian et al. [15] is categorized as both conceptual work and a review paper; Krebs et al.
[117] is categorized as both conceptual and analytical work; Wang et al. [242] is algorithmic and
conceptual work.

4 RESULTS
Before considering what values are discussed in the literature and how value-aware NRS are evalu-
ated, we briefly review the number of articles on NRS and values specifically. Here, we also identify
the types of papers produced and explore the prolific authors publishing on values in NRS.

4.1 General Overview

Figure 2 provides an overview of the number of papers published over time—on NRS in general
and value-aware NRS in particular. The rhombus-shaped symbols (in red) represent the papers
focused on NRS overall, and the circle-shaped symbols (in blue) refer to the subset of papers in
which we identified values beyond accuracy. The rhombus-shaped (red) symbols in the graph re-
veal an increase in the total number of papers published on NRS from 1995 to 2022. There was
an initial surge in the number of papers observed after 2008, followed by another sharp increase
from 2016 onward. Raza and Ding [195] conjecture that the increase in the later years (from 2016
onward) might be credited to both the CLEF NEWSREEL Challenge,4 [30] providing resources for
the evaluation and optimization of news recommenders, as well as the emergence and develop-
ment of RS based on deep learning, which happened around that same time. Further, Raza and
Ding [195] claim that the higher number of publications on NRS in 2021 is linked to the release of
the benchmark dataset MIND (by Microsoft) [252].
While the absolute number of papers on value-aware NRS appears at first sight to remain rela-
tively stable (Figure 2), having a closer look at the proportion of value-aware NRS papers compared
to the overall number of published papers on NRS paints a different picture. Over the years, the
percentage of value-aware NRS papers (compared to overall NRS papers) varies greatly. In short,
4 https://www.newsreelchallenge.org/

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
Where are the Values? A Systematic Literature Review on News Recommender Systems 23:11

while the number of NRS papers has grown rapidly from 2008 onward, the number of value-aware
NRS papers did not grow proportionally. As Figure 2 shows, the absolute number of value-aware
NRS papers has increased in the past two decades. The first value-aware NRS paper in our cor-
pus was published in 1995, after which there was a six-year gap until the next value-aware NRS
paper. Value-aware NRS papers started picking up in 2010, with the number gradually increasing
over time. A noticeable decline in the number of value-aware NRS papers occurred in 2018, which
corresponds to a general stagnation in the publication of NRS papers during the same year.
Looking into the research approaches taken to investigate values in NRS, we found that most
research takes an algorithmic approach (108 papers, 59%), focusing on the development of algo-
rithms. Compared to this, only a few works take a conceptual (30 papers, 16.4%) or analytical
approach (23 papers, 12.6%).
To clarify, in contrast to the algorithmic works, analytical works do not necessarily introduce
new algorithms. Instead, they prioritize the evaluation and comparison of existing recommender
approaches concerning specific values. Conceptual works, in contrast, neither implement the crite-
ria algorithmically nor analyze existing approaches; instead, they reflect on values—their relevance,
need, and conceptualization.
Compared to algorithmic, conceptual, and analytical works, user studies do not appear often
(15 papers, 8.2%). Review papers (9 papers, 4.9%) and interview-based research (5 papers, 2.7%)
are even rarer. While the user studies centered on news consumers, only one of the 5 interview-
based works focused on news readers (i.e., Reference [85]). Instead, 4 of the 5 interview-based
works were conducted through interviews with people employed or active in the news domain (e.g.,
References [14, 24, 59]) and with parliament members or party officials (e.g., Reference [80]). For
values, we argue, it is crucial to listen and “check” with news readers and practitioners about their
concepts of and experiences of these values rather than impose our own assumptions about them
in the implementation of recommender systems. However, as indicated, this type of work is lagging
behind.
Figure 3 depicts the temporal evolution of these approaches in the publications.5 Despite annual
fluctuations, reflective in part of the dynamics inherent in academic publishing, it underscores the
finding that algorithmic work predominantly characterizes the literature on NRS dealing with val-
ues. It furthermore demonstrates that reviews, analyses, and conceptual work gradually constitute
a growing proportion of the overall output. The increased prominence of conceptual work is par-
ticularly notable here compared to the other two categories.
The distribution of these different research approaches raises questions about possible gaps in
NRS research. Considering that the number of algorithmic work significantly outweighs concep-
tual work, the perhaps most fundamental question is the following: Do journalistic teams have a
different understanding of these values than the tech teams responsible for the development of
NRS? Additionally, the limited number of user studies and interviews raises questions such as: Do
we know how users experience these values? Are they even aware of the values embedded into
these systems? What are the specific expectations and needs of practitioners in the news domain?
Are these expectations and needs being met?

4.2 Prolific Authors on Value-aware News Recommendation

To yield insights regarding who does research on values in NRS and the status of these publications
within RS research, we ranked the top authors concerning their number of publications in our
corpus (Table 1). We used five publications as the cutoff point.

5 Seven papers (i.e., References [15, 62, 96, 117, 129, 236, 242]) have double-assignments concerning type of paper; thus,

n = 190.

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
23:12 C. Bauer et al.

Fig. 3. Temporal evolution of the types of papers.

Table 1. Authors with the Highest Number of Publications in the Sample

Author # Papers Affiliation

Natali Helberger 7 University of Amsterdam, Amsterdam, The Netherlands
Mykola Makhortykh 7 Institute of Communication and Media Studies, University
of Bern, Bern, Switzerland
Balaji Padmanabhan 7 Muma College of Business, University of South Florida,
Tampa, FL, USA
Shankar Prawesh 6 Industrial and Management Engineering, IT Kanpur, Kan-
pur, UP, India
Jon Atle Gulla 6 Norwegian University of Science and Technology, Trond-
heim, Norway
Özlem Özgöbek 6 Norwegian University of Science and Technology, Trond-
heim, Norway
Mariella Bastian 5 University of Amsterdam, Amsterdam, The Netherlands
Jon Espen Ingvaldsen 5 Norwegian University of Science and Technology, Trond-
heim, Norway

Regarding the question of who, we identified three clusters of collaborators in this ranking. The
first cluster concerns the scholars Balaji Padmanabhan and Shankar Prawesh, who published on
value-aware NRS in our corpus together—between 2011 and 2015—when they both worked at
the Department of Information Systems and Decision Sciences at the University of South Florida,
Muma College of Business, USA. These authors primarily consider manipulation-resistant NRS,
discussing the problem of the self-reinforcing nature of “most popular” type lists.
The second cluster centers around Natali Helberger from the Institute of Information Law at
the University of Amsterdam, The Netherlands. In our corpus, Helberger has several (co-)authored

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
Where are the Values? A Systematic Literature Review on News Recommender Systems 23:13

publications from 2018 to the present. These works tackle the democratic role of NRS and reflect on
values—mainly diversity. These publications are linked to Helberger’s PersoNews project6 (2015–
2021) on the impact of personalized news for democracy, funded by the European Research Council.
Mykola Makhortykh worked as a postdoctoral researcher in Data Science at the Amsterdam School
of Communication Science. Makhortykh was connected to this project, too, studying algorithmic
(un)fairness in news personalization systems. Finally, Mariella Bastian worked as a postdoctoral
researcher at the Institute for Information Law on the PersoNews project.
Third, we observe a cluster with Jon Espen Ingvaldsen, Özlem Özgöbek, and Jon Atle Gulla. They
all work at the Department of Computer and Information Science at the Norwegian University of
Science and Technology, Norway. Together, they have published on context-aware, user-driven
news recommendation, the intricacies of time in news recommenders, and user-controlled news
recommenders. Ingvaldsen and Gulla published about NRS in relation to location awareness and
geographical proximity, mostly around 2015. Gulla and Özgöbek published together on topics such
as exploratory news recommendations and interactive mobile news recommenders. They also pub-
lished on news recommenders with other co-authors.
By setting a cut-off point of five papers, we aim to spotlight academics who have consistently
engaged with the subject matter, showing a continuing a line of inquiry, and thereby distinguishing
them from those who have contributed more occasionally. Earlier, we described that, despite an
uptick in publications related to NRS, the proportion of studies focused on values has not kept up
with this growth. We have also identified that only three groups of authors account for 20% (49
out of 244) of the papers on value-aware NRS within our dataset. Furthermore, these publications
are connected to large research funding, indicating that research in this domain is still limited in
the studied time frame. Collectively, these findings underscore the niche and ad hoc nature of the
field. In a more mature field, the threshold for identifying the top researchers based on the number
of publications would have been considerably higher.

4.3 Identified Values and Value Groups in News Recommendation

A core interest of this review is to bring to light the range of values discussed and considered
in NRS research. In our study, we adopted a broad definition of values in accordance with Value
Sensitive Design (VSD). This definition encompasses all factors deemed important. In our sample
of 183 papers, we identified 40 values. In an iterative process (see Section 3.3), this multitude of
values could be aggregated into five categories (value groups). Figure 4 presents an overview of
the identified values and value groups, which we present and discuss in the following:
The first value group is termed standard values (90 occurrences). It embraces the values diversity,
popularity, novelty, and coverage. These values are considered “standard,” because—as discussed
earlier—they are the most discussed beyond-accuracy values. Rather unsurprisingly, these values
are also widely discussed within the field of NRS.
The responsibility values (88 occurrences) embrace privacy, explainability, accountability, trans-
parency, trust, fairness, and manipulation prevention. These values point to news providers’ re-
sponsibility towards their users and society. In addition, this value group includes responsibility
values that specifically concern providing users the opportunity to act and, thus, form a subgroup
of responsibility values. This subgroup, termed responsible agency, encompasses autonomy, agency,
and future impact.
The user experience (UX) values (81 occurrences) refer to values concerned with aspects that pri-
marily target how users experience the NRS. Our sample features the following UX values: tempo-
rality of interests, engagement, user satisfaction, curiosity, emotion, serendipity, fatigue, surprise,

6 https://doi.org/10.3030/638514

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
23:14 C. Bauer et al.

Fig. 4. Overview of value groups.

and convenience. While these UX values are essentially not news-domain-specific, they are also
relevant in the news domain. We note that serendipity is also one of the standard beyond-accuracy
values widely discussed in the RS field in general [103]. Still, we hold with Smets et al. [215] that
“serendipity should be understood as a user experience rather than a mere offline evaluation metric
such as diversity or novelty.” Hence, our categorization scheme includes serendipity in UX values.
The editorial values (91 occurrences) reflect an organization-centered perspective and encom-
passes freedom, objectivity, authority, credibility, democracy, journalistic values, and editorial in-
fluence. These values are inherent in the news domain and strongly associated with journalistic and
editorial values (for details, see Bastian et al. [14] and Lu et al. [136]). Lu et al. [136] emphasize that
these values must be considered when implementing RS in the news domain. Similar to the respon-
sibility values, a subgroup emerged for the editorial values. This subgroup termed recommender-
specific encompasses context, location, recency, relevancy, and saliency. While these values em-
brace editorial values—and are, thus, integrated into this group—the recommender-specific ones
form a subgroup, as these are specifically instrumental within the context of RS.
The technical values (15 occurrences) embrace values associated with the technical operation of
an RS: scalability, real-time capabilities, personalization, utility, and tradeoff. We note that many
values may conflict with technical values; thus, improving all of them is challenging or infeasi-
ble. When researchers recognize and acknowledge that several values must be considered despite
potentially creating tradeoffs, optimizing such tradeoffs can be considered a value on its own.
Table 2 provides an overview of the total number of occurrences of values summed up per value
group (2nd column) and the number of papers that address at least one value in the respective value
group (3rd column). Overall, Table 2 indicates that editorial values are featured most often (91 oc-
currences), followed by standard and responsibility values (90 and 88 occurrences, respectively).
UX values (81 occurrences) are featured only marginally less. In comparison, technical values are
mentioned to a far lower extent (15 occurrences). As explained, it is unsurprising that the standard
values are featured often.
The high number of occurrences for editorial values pinpoints that domain-specifics are im-
portant. As we show in Section 4.4 (particularly Figure 5), responsibility values gained particular
attention from 2019 onward. By comparison, technical values receive very little attention in value-
aware NRS research. There are several possible explanations for this. One possibility is that, in
some cases, other values may take precedence over technical values, leading to their relatively

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
Where are the Values? A Systematic Literature Review on News Recommender Systems 23:15

Table 2. Total Number of Occurrences of Values

(Grouped by Value Groups) and Total Number of
Unique Papers per Value Group

Value Group # Occurrences # Papers

Standard 90 75
Responsibility 88 50
UX 81 66
Editorial 91 75
Technical 15 13

lower emphasis. Additionally, technical values may not be questioned, as they are already part of
a functioning system. Last, it is plausible that technical values are simply of greater importance in
domains other than news.
While the second column in Table 2 presented the total number of occurrences of values summed
up per value group, the third column presents the number of papers that address at least one value
in the respective value group. Standard values and editorial values (75 papers, respectively) are also
the most covered value groups in this regard, whereas technical values (13 papers) are addressed
the least. For standard, UX, and editorial values, the number of papers is only slightly lower than
the number of occurrences in the respective value group (standard values: 90 occurrences in 75 pa-
pers; UX values: 81 occurrences in 66 papers; editorial values: 91 occurrences in 75 papers). Tech-
nical values are mostly addressed individually—thus, one at a time (15 occurrences in 13 papers);
rarely together in one paper. However, regarding the responsibility values, we see stark differences,
because values in this group appear 88 times, yet across only 50 papers. This observation points
out that the values in this group are often addressed together within one paper.

4.4 Value Groups in the Discourse on News Recommender Systems

Diving into the identified value groups (as described in Section 4.3), we see interesting publishing
patterns over time (Figure 5). The value groups discussed from early publications onward are the
editorial and the standard values. This observation is expected, because these values represent the
basic tenets of the news domain and the RS field, respectively. From 2004 onward, papers about
responsibility and UX values start surfacing. Increasingly, almost all value groups receive more
attention in the early 2010s. The only exception is the technical values group, which remains sta-
ble throughout; we note that we identified a total of only 13 papers addressing those values, and
due to this limited number, this observation is inconclusive. Notable is the rise in editorial values
starting in 2013 and declining in 2018. Around the same time, a surge can be observed for UX val-
ues. Another interesting development is the increased discussion of responsibility values in 2019.
More broadly, from 2018 onward, we observe a shift towards responsibility, standard, and UX val-
ues rather than editorial values. This trend corresponds to overall developments in research—also
outside the news domain; for instance, with the establishment of the ACM Conference on Fair-
ness, Accountability, and Transparency (ACM FAccT),7 launched that year, we witness a general
acknowledgment of responsibility values in the research and development of algorithmic systems.
To get an overview of which values have been implemented in algorithmic approaches and
which ones have been discussed on a conceptual level, Figure 6 shows the value groups per paper
type. While all value groups feature in algorithmic work, responsibility values occur less often

7 https://facctconference.org.
Note that, in 2018, the conference’s name was FAT*. The conference was affiliated with ACM
in 2019. After the 2020 conference, the conference changed its name to ACM FAccT.

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
23:16 C. Bauer et al.

Fig. 5. Occurrence of value groups in value-aware NRS papers over time.

Fig. 6. Occurrence of value groups among paper types.

in algorithmic work (16 occurrences) compared to UX values (43 occurrences), editorial values
(42 occurrences), and standard values (39 occurrences). Instead, in conceptual work, responsibility
values are the most represented value group (19 occurrences). Standard values (14 occurrences)
and editorial values (12 occurrences) range in the middle in conceptual works; other value groups
appear in conceptual works to a far lower extent (6 occurrences and lower). From this observation,
we infer that while responsibility values have been integrated, discussion on a conceptual level is
still needed.
Unsurprisingly, the standard value group is featured in algorithmic work a lot (39 occurrences)
because these are considered “standard” already. In addition, these values are discussed in
conceptual works (14 occurrences), indicating ongoing research to tease out those values on a
conceptual level.

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
Where are the Values? A Systematic Literature Review on News Recommender Systems 23:17

Similar to the standard values (39 algorithmic papers and 14 conceptual works), editorial values
are strongly featured in algorithmic work (42 occurrences), while there is an ongoing discussion
in conceptual work (12 occurrences).
Further, we note that there are, in total, only 15 papers with user studies. Interestingly, though,
these cover all value groups.

4.5 Values within Value Groups

Having discussed the values in the publications on NRS (Section 4.4), this section examines the
values within the value groups in more detail. Figure 7 illustrates the number of occurrences of
individual values per value group. This figure shows that diversity far outnumbers other values in
terms of occurrences (62 occurrences). The second most popular value is recency (39 occurrences),
followed by temporality of interests (26 occurrences).
Besides recency, location (19 occurrences) is featured substantially among the editorial values,
followed by democracy, context, and objectivity (10, 7, and 6 occurrences, respectively). Still, the
latter three values (i.e., democracy, context, and objectivity) are far less often considered than
the overall most popular ones. We note that these are more abstract concepts than recency and
location, which could explain why these are, in comparison, considered less.
Among the responsibility values, transparency and trust are considered most often (18 occur-
rences each). Interestingly, these two values concern features that are often discussed in the context
of RS in general: A lack of trust and transparency are the aspects that make people less receptive
to RS [169, 244]. Especially when it comes to NRS, these values are vital [102, 200] because—from
a democratic perspective—trust in news is essential for the ideal of the informed citizen [50]; and
the increasing spread of “fake news” [217] has an impact on people’s trust in media [108, 245].
Further, from Figure 7, we see that the high popularity of standard values is primarily due to
diversity (62 occurrences), which is the most featured value overall. Aside from diversity, novelty
(15 occurrences) and coverage (12 occurrences) are featured often, compared to diversity to a far
lower extent. Coverage is a value that is relevant for the news provider, and novelty is what keeps
news readers interested [136].
Within the UX values group, temporality of interests (26 occurrences), serendipity (20 occur-
rences), and emotion (15 occurrences) are tackled most often. In comparison, other UX values
appear far less frequently. It is interesting to observe that serendipity is frequently considered in
the context of NRS. This observation indicates that it is considered important that NRS do not only
provide news readers with the news they want to read but also serve users in a way they do not nec-
essarily expect. Shifting user interests is a topic that RS have to account for in general [58, 243, 257].
As the news domain is highly concerned with recency and (unexpected) real-life events, user in-
terest shifts may follow different patterns in the news domain compared to other domains.
The technical values rarely occur in value-aware NRS papers (15 occurrences in 13 papers).
Real-time capabilities and utility are addressed several times (5 occurrences, respectively), whereas
tradeoff is only once.
As diversity is the most addressed value (62 occurrences), we detail what diversity embraces. In
general, RS literature frequently calls for more diverse recommendations and suggests diversifi-
cation approaches to ensure a certain level of diversity in the recommendations (e.g., References
[4, 78])—however, without necessarily specifying how diversity should manifest.8 In other respects,
some works address a particular type of diversity (e.g., Ziegler et al. [267] specifically addressed
topic diversification in a book recommendation setting). Notably, various types of diversity are
addressed in the news domain. Topic Diversity occurs the most (34 occurrences), which indicates

8 For surveys on diversity in recommender systems, see Kaminskas and Bridge [103] and Kunaver and Požrl [121].

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
23:18 C. Bauer et al.

Fig. 7. Occurrence of values in value-aware NRS papers.

that NRS researchers are interested in showing news readers a variety of topics. The second diver-
sity type is viewpoint (19 occurrences), which is very topical in the news domain. A valid concern
around NRS is that they might show news readers only articles from one (political) viewpoint.
Offering users viewpoint diversity may be a productive counter to that and address fears of, for
instance, increased societal polarization. Compared to topic and viewpoint diversity, the other
diversity types (i.e., diversity concerning sources, people, events, semantics, sentiments, authors,

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
Where are the Values? A Systematic Literature Review on News Recommender Systems 23:19

genders, and temporal aspects) are considered only a few times each (5 occurrences or less). No-
tably, 3 works consider 3 diversity types within their work. By delving into specifics, the papers
in our sample distinguish from papers generally claiming for diversification. In addition, 6 papers
discuss various angles of diversity without singling out any specific diversity type.

4.6 Evaluating Value-aware News Recommender Systems and Measuring Values

The most popular evaluation setup used in the value-aware NRS papers is offline evaluation (73),
followed—with a large gap—by laboratory study (27), online evaluation (18), and simulation (8).
It is important to note that not every kind of work includes an evaluation (e.g., most conceptual
works and reviews). A large majority of papers on RS (in general) employ offline evaluation (see,
e.g., References [20, 54]), which is similarly reflected in our sample. This indicates that the evalua-
tion of value-aware NRS most often does not involve interaction with real users; instead, it relies
on predicting preference or behavior based on historical data. While it has its eligibility and bene-
fits as a controlled environment and for establishing baselines, it is limited to reflecting past rather
than current and future behaviors and preferences. This is particularly disputable in the news do-
main, where—by definition—recency plays a critical role. Moreover, it uses simplified user models
that do not capture the complexity of responses and the various factors influencing how these
recommendations are experienced. Particularly when it comes to integrating values into RS, it is
crucial to involve real users in the evaluation process. This is to avoid drawing and amplifying ex-
isting assumptions based on correlations. For instance, with regard to diversity: Do users perceive
the implemented diversity?
With regard to the relation between evaluation types and value groups, there are some interest-
ing patterns that warrant further research, beyond the scope of this article. For instance, offline
evaluation is used to a far lower degree when responsibility values were involved (6 occurrences)
compared to standard, UX, and editorial values (29, 29, and 28 occurrences, respectively). More-
over, only two papers used online evaluation when responsibility values were involved. Note that
standard, UX, and editorial values were evaluated with online evaluation only slightly more (5,
7, and 9 occurrences, respectively). Interestingly, laboratory studies are on similar levels for stan-
dard, responsibility, UX, and editorial values (10, 12, 12, and 8 occurrences, respectively). Given
the limited number of papers addressing technical values in our sample, it is not possible to draw
conclusions in that regard. Additionally, the small number of instances (only 10) where simulation
was used as an evaluation approach makes it difficult to derive meaningful observations regarding
this method.
Accuracy-based metrics clearly dominate in our sample on value-aware NRS. Throughout the
timeline, the accuracy family, which includes accuracy, precision, recall, and F1, is consistently
used the most (in total, 115 occurrences). This observation that accuracy is the most-used measure
in NRS has also been found in other reviews (e.g., Reference [195]). Also, click-through rate (CTR)
and normalized discounted cumulative gain (nDCG) are frequently used (20 and 25, respectively). It
is striking that diversity is heavily addressed in algorithmic work (note: 34 of the 109 algorithmic
works address diversity). However, intra-list similarity (ILS) [267], which is the widely used mea-
sure for diversity [100], is relatively rarely used in our sample (11 papers). Beyond these, further
metrics occur in our sample, yet scarcely—often only once—(e.g., hit rate (HR), mean reciprocal
rank (MRR), distortion, Jaccard similarity, Gini coefficient, root mean square error (RMSE)).
To sum up, accuracy-based measures remain dominant in the field, even with regard to the news
domain. CTR and nDCG gain attention from 2013 onward. Although used in the past few years,
other measure types have not gained momentum. In this light, we also propose that, although a
wide range of values is considered in papers on NRS, these values are not being evaluated (which
is also in line with the observation by Karimi et al. [106, p. 1212]). Reflecting on this matter, van Es

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
23:20 C. Bauer et al.

et al. [234] point to the importance of aligning concept, design, and evaluation. For instance. Hel-
berger et al. [91] explore how diversity can be conceptualized in very different ways. Each concep-
tualization implies a different operationalization and benchmarks and metrics. These evaluation
metrics are, however, merely proxies in that they stand in for the concept it tries to capture [159,
p. 4]. This means that there can be disagreement on what the right benchmarks and metrics should
be. Ideally, relevant stakeholders (e.g., computer scientists, journalists, advertisers) find common
ground in how alignment should be achieved.

5 DISCUSSION
In Section 4, we presented an overview of the literature on value-aware NRS. This section raises
discussions prompted by these findings.
First, Raza and Ding [195, p. 16] found that there is some effort to introduce diversity in news
recommendation but very limited work on novelty, coverage, and user experience. Our findings
underscore that diversity is the most published beyond-accuracy value of all. As explained by
Helberger and team [25, 89–91, 154], this value is of high concern within the news domain, as it is
linked to policy objectives and normative ideals. However, this means other values are relatively
understudied. Is this disproportional attention for diversity really warranted? Or is it a fairly “easy”
value to implement into recommender systems? To make matters even more complex, as touched
on earlier, does the incorporation of these values necessarily involve tradeoffs?
Second, there is tension between the abstractness (complexity) of values and their operational-
ization (simplification). There are indications that many values that are discussed in the litera-
ture are oversimplified and detached from their original meaning. Values are complex phenomena
that need to be targeted on a more fine-grained and specific level to understand to which extent
they are embedded in RS or what implications RS have on specific values. We have found that it
is now being done in the case of diversity, where researchers look into specific sub-dimensions.
However, here, most work is on topic and viewpoint diversity, glossing over many other forms of
diversity.
Breaking down these values into more specific and actionable components is a task that needs
to be undertaken for other values as well. Currently, some values are too broad or coarse, and a
more detailed examination is required to make them practical and applicable. As a consequence,
it is also difficult to measure those. For instance, explainability: What is explained (procedure or
outcome)? To whom is it explained or explainable (a user or the developers)? In what level of detail
is it explained? While there is work in this direction (see Zhang and Chen [263]), this is seemingly
not happening in the news domain.
A third observation relates to the “stability” of the values that are being implemented. For in-
stance, our review suggests that some values (e.g., editorial values and standard values) have fre-
quently been integrated into algorithmic work, while it is also tackled on a conceptual level. In-
deed, it is a well-studied phenomenon that values undergo changes over time [248], whereby these
changes are not necessarily due to time effects, but rather emanate from time-invariant contextual
influences [229]. In the context of our review, this suggests that early algorithmic works might
not be capturing and integrating values in the same way as later work, because these values were
conceptualized (on a deeper level) only later. This raises questions concerning the comparability
of works, as conceptualizations of values may vary over time.
Fourth, as pointed out by Stray [218], lots of works focus on principles—thus, “written de-
scriptions of the values that technical systems should uphold” [218]—for news recommenders
rather than metrics, evaluation, datasets, and feedback. Our findings suggest that many of these
value-ware NRS are evaluated by accuracy metrics. However, we need to use metrics that are

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
Where are the Values? A Systematic Literature Review on News Recommender Systems 23:21

aligned with the goals. As Jannach and Bauer [97] suggest, the RS research community has
fallen prey to what they call the McNamara fallacy: a focus on quantitative and easy-to-take
measures in offline experiments. As such, the effectiveness of the algorithms in practice remains
unknown.
Then, there is the lack of understanding whether users actually perceive the values in question.
Does making recommendations more diverse indeed increase user satisfaction? Recent work out-
side the NRS field suggests a discrepancy between measures and human perception [100]. In the
news domain, there is—to date—only one small study on this issue, suggesting that body text sim-
ilarity is most representative of human perception (compared to, e.g., the similarity of authors or
images) [216]. More research is required to understand whether users perceive specific values and
whether that correlates to greater satisfaction.
Finally, designing (N)RS designed in a value-aware fashion raises ethical questions. Should we
nudge users towards “healthier” news consumption? As Helberger et al. [91] explain,
“influencing people’s choices, even for good and legitimate reasons, can sit at odds
with users’ conceptions of personal autonomy, freedom from manipulation and pri-
vacy. This is even more so if diversity-sensitive design is used to realize more norma-
tive, societal objectives, such as serving democratic discourse rather than the interests
of individual users.” [91, p. 201]
Answering this requires reflective discussions about different stakeholder values and how these are
balanced in designing these systems. The inability of users to detect certain biases in recommended
news invites conversations about transparency, responsibility, accountability, and explainability.
With traditional news outlets, the public is aware of their ideological slant, and their editors are
typically willing and able to publicly discuss why certain editorial decisions were made. Studying
users’ perceptions and needs concerning the role of NRS in this context requires more user surveys
and interviews.

6 CONCLUSIONS
With this systematic literature review, we have traced and reflected on the scale, research fields,
and range of values discussed and engaged with in the scientific discourse on recommender sys-
tems in the news domain. Our review suggests that value-aware NRS is still an under-researched
area of interest, particularly within computer science. We observed that although values gain more
attention in NRS research, it still constitutes a relatively small and ad hoc “field” and has not grown
proportionally with the RS field as a whole. This concluding section summarizes the main findings
regarding values and news recommender systems.
In our review, we found that most value-aware NRS research has taken an algorithmic approach.
Conceptual papers, analytical works, review papers, user studies, and interview-based research
are far rarer. This suggests a possible research gap concerning users’ experiences of values and
alignment between editorial and tech staff on what these values mean.
Moreover, the driving force bringing attention to value-aware NRS seemingly comes from fields
outside computer science (e.g., information systems and media studies) and is linked to collabora-
tions on specific topics or within funded projects.
Further, our work identified and categorized values into value groups. In our corpus, we iden-
tified five different value groups: editorial values, responsibility values, standard values, techni-
cal values, and UX values. Within these value groups, diversity (standard values) far outnumbers
other values in terms of occurrences. Most of the publications on diversity tend to deal with topic
diversity, followed by viewpoint diversity. Second, in terms of occurrences in the value groups, is
recency (editorial values). Both diversity and recency are very relevant to the news domain. The

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
23:22 C. Bauer et al.

former concerns the relationship between news and democracy, and the latter concerns the con-
nection between news and the capture of unfolding developments. However, it does leave room
and invite research on other values that are less investigated so far, but may improve recommender
systems in this domain.
Furthermore, we observe that values and value group aspects are system-centered, whereas
others are more user-centered. For instance, coverage, popularity, and explainability affect the
system as a whole. In contrast, privacy and trust are values expressed by users. Future research
could investigate whether certain values and value groups might be optimized or designed (or not),
depending on whether these values (or value groups) concern the system or user.
Finally, we found that recommendations are often evaluated by accuracy-based metrics. Thus,
although many principles for news recommenders have been developed, there is still much work
to be done in aligning these principles with relevant metrics that evaluate success and exploring
the matter of potential tradeoffs. As indicated, this task is not easy and requires more inter- and
transdisciplinary exchanges that help translate abstract values into design principles. It also neces-
sitates ongoing collaboration between industry and academia. This also relates to the observation
that offline methods are currently the by far the most used evaluation methods. The benefit of
an academia-industry collaboration would be that these algorithms are made and tested in prac-
tice, providing the context and constraints under which these systems and their makers need to
function.
This survey is subject to several limitations. First, the corpus of publications was determined by
a specific query and was limited to publications in English, introducing potential biases in the selec-
tion of literature. Research conducted in other languages or with different terminology may have
been excluded. Second, the field of NRS has been rapidly evolving, and the survey’s cutoff date may
have led to the omission of recent developments and papers. The dynamic nature of the field re-
quires continuous updates to capture the latest research. Third, we systematically approached the
selection and coding of papers by employing several coders, randomizing the assignment of papers
to coders, and including reflexivity and dialogue within the research team. While this approach
helps reduce systematic biases, a qualitative approach like ours may still retain some subjectivity.
A similar limitation concerns categorizing values into value groups, where other research teams
may result in different groupings and labels of value groups. Last, this survey represents a dis-
tant reading of NRS publications, where publications are treated as comparable units. However,
the significance and aspirations of these publications may vary significantly. Individual papers
may have different goals, methodologies, and impacts that are not captured by this bird’s-eye
view.
These caveats notwithstanding, our systematic literature review contributes to the body of
knowledge in several ways. We have identified a comprehensive set of values (total of 40 values)
in our analyzed corpus, and we have developed a categorization scheme to group these values into
five value groups. This provides a solid basis for future research to build upon. Future research may
expand and refine the set of values and the categorization scheme. Moreover, our review synthe-
sizes the body of research on value-aware NRS across disciplines and communities, tracing back
to 1995. Among others, this synthesis clearly indicates the under-researched values that could be
relevant to explore further. Furthermore, the analysis gives direction where values need to be tar-
geted on a more fine-grained and specific level. Finally, our analysis suggests that the driving force
bringing attention to value-aware NRS seemingly comes from fields outside computer science (e.g.,
information systems and media studies). To move forward, interdisciplinary and transdisciplinary
research collaborations are strongly encouraged. While this is often advocated, the fundamental
challenge is to put these collaborations into practice.

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
Where are the Values? A Systematic Literature Review on News Recommender Systems 23:23

APPENDIX

Table 3. Overview of the Sample

Reference Journal/Conference Name Journal or Conference Year

Abdollahpouri et al. [2] WWW 2021 Conference 2021
Agarwal and Singhal [5] ICROIT 2014 Conference 2014
Ahn et al. [6] WWW 2007 Conference 2007
Alanazi et al. [7] HT 2016 Conference 2016
Ashraf et al. [10] ICSCEE 2018 Conference 2018
Atoum and Yakti [11] ICTCS 2017 Conference 2017
Babanejad et al. [12] INRA 2019 Conference 2020
Bader [13] SICN 2019 Conference 2019
Blanco et al. [23] CIKM 2012 Conference 2012
Boutet et al. [27] IPDPS 2013 Conference 2013
Bozdag and van de Poel [28] PICMET 2013 Conference 2013
Caldarelli et al. [35] UMAP-ExtProc 2016 Conference 2016
Carbone and Vlassov [37] ICCAC 2015 Conference 2015
Chakraborty and Ganguly [38] ASONAM 2018 Conference 2018
Chen et al. [42] WI-IAT 2008 Conference 2008
Chesnais et al. [43] International Workshop on Conference 1995
Community Networking
Ciobanu and Lommatzsch [44] CLEF 2016 Conference 2016
Cotter et al. [47] CHI EA 2017 Conference 2017
Cui et al. [48] SPML 2021 Conference 2021
Dacon and Liu [49] WWW 2021 Conference 2021
Daneshi et al. [51] ICMEW 2013 Conference 2013
Desarkar and Shinde [55] DSAA 2014 Conference 2014
Epure et al. [62] RecSys 2017 Conference 2017
Gabrilovich et al. [69] WWW 2004 Conference 2004
Gao et al. [70] WI-IAT 2011 Conference 2011
Gao et al. [71] SmartBlock 2020 Conference 2020
Garcin et al. [72] RecSys 2013 Conference 2013
Garcin et al. [73] RecSys 2014 Conference 2014
Garrido et al. [74] SISY 2015 Conference 2015
Gebremeskel and de Vries [76] CLEF 2015 Conference 2015
Gharahighehi and Vens [77] OHARS 2020 Conference 2020
Gulla et al. [83] UMAP-ExtProc 2016 Conference 2016
Harambam et al. [85] RecSys 2019 Conference 2019
Hassan and McCrickard [87] WWW 2019 Conference 2019
Hu et al. [93] HICSS 2012 Conference 2012
Ingvaldsen et al. [94] IntRS@RecSys 2015 Conference 2015
Islambouli et al. [95] HUMAN 2021 Conference 2021
Jain et al. [96] HotMobile 2017 Conference 2017
Kang et al. [104] ICACT 2014 Conference 2014
Karimi et al. [105] INRA 2019 Conference 2020
Kazai et al. [110] SIGIR 2016 Conference 2016
Khattar et al. [112] ICDMW 2017 Conference 2017
Kille and Albayrak [113] RecTemp 2017 Conference 2017
(Continued)

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
23:24 C. Bauer et al.

Table 3. Continued
Reference Journal/Conference Name Journal or Conference Year
Krebs et al. [117] CHI EA 2019 Conference 2019
Kulkarni et al. [118] ICCUBEA 2019 Conference 2019
Kumar et al. [119] ICDMW 2017 Conference 2017
Kumar et al. [120] ICDMW 2017 Conference 2017
Lenhart and Herzog [124] CBRecSys 2016 Conference 2016
Li et al. [127] SIGIR 2011 Conference 2011
Li et al. [129] RecSys 2011 Conference 2011
Liu et al. [133] WWW 2021 Conference 2021
Loecherbach et al. [134] WWW 2021 Conference 2021
Lommatzsch et al. [135] WI 2017 Conference 2017
Lu et al. [136] UMAP 2020 Conference 2020
Lu et al. [137] SIGIR 2019 Conference 2019
Lu and Liu [138] CCIS 2016 Conference 2016
Lv et al. [141] WWW 2011 Conference 2011
Ma et al. [142] WWW 2016 Conference 2016
Maksai et al. [145] RecSys 2015 Conference 2015
Meguebli et al. [147] KDIR 2014 Conference 2014
Mohallick and Özgöbek [153] WI 2017 Conference 2017
Mulder et al. [158] FAccT 2021 Conference 2021
Muralidhar et al. [160] ICTAI 2015 Conference 2016
Nagaki et al. [161] MOBIQUITOUS 2016 Conference 2016
Natarajan and Moh [163] CTS 2016 Conference 2016
Niu et al. [165] CHI 2018 Conference 2018
Niu and Al-Doulat [166] CHIIR 2021 Conference 2021
Noh et al. [167] BIGCOMP 2014 Conference 2014
O’Banion et al. [168] RSWeb 2012 Conference 2012
Oh et al. [170] ICACT 2014 Conference 2014
Özgöbek et al. [171] WEBIST 2014 Conference 2014
Özgöbek et al. [172] WEBIST 2015 Conference 2015
Panteli et al. [173] INRA 2019 Conference 2020
Patankar et al. [176] ICSC 2019 Conference 2019
Pfahler and Morik [177] FATE/MM 2020 Conference 2020
Phelan et al. [178] WWW 2011 Conference 2011
Pon et al. [179] KDD 2007 Conference 2007
Pon et al. [180] WIDM 2008 Conference 2008
Prawesh and Padmanabhan [182] RecSys 2011 Conference 2011
Prawesh and Padmanabhan [183] AMCIS 2012 Conference 2012
Prawesh and Padmanabhan [184] ICIS 2012 Conference 2012
Prawesh and Padmanabhan [186] WITS 2015 Conference 2015
Qi et al. [188] ACL-IJCNLP 2021 Conference 2021
Qi et al. [189] EMNLP 2020 Conference 2020
Qin and Zhang [191] CONF-CDS 2021 Conference 2021
Raza and Ding [192] Big Data 2019 Conference 2019
Raza and Ding [193] Big Data 2020 Conference 2020
Raza and Ding [194] Big Data 2021 Conference 2021
Reuver and Mattis [197] EACL 2021 Conference 2021
(Continued)

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
Where are the Values? A Systematic Literature Review on News Recommender Systems 23:25

Table 3. Continued
Reference Journal/Conference Name Journal or Conference Year
Robindro et al. [199] ICCCA 2017 Conference 2017
Bathla et al. [17] ICRITO 2015 Conference 2015
Sadhasivam et al. [201] ICECCS 2014 Conference 2015
Saravia et al. [205] TAAI 2017 Conference 2018
Sertkan et al. [208] CBI 2019 Conference 2019
Shan et al. [209] ICCC 2016 Conference 2017
Streibel and Alnemr [219] CIKM 2011 Conference 2011
Su et al. [220] SMAP 2016 Conference 2016
Sullivan et al. [221] ACM UMAP 2019 Adjunct Conference 2019
Sun et al. [222] SCC 2021 Conference 2021
Suppasert et al. [224] ICT-ISPC 2017 Conference 2017
Tasci and Cicekli [226] KDIR 2014 Conference 2014
Tavakolifard et al. [227] WWW 2013 Companion Conference 2013
Tintarev et al. [228] UMAP 2018 Conference 2018
Verheij et al. [235] WI 2012 Conference 2012
Vrijenhoek et al. [237] CHIIR 2021 Conference 2021
Wanaka and Tsubouchi [238] Urb-IoT 2016 Conference 2016
Wang et al. [239] CIKM 2021 Conference 2021
Wang et al. [240] SIGIR 2010 Conference 2010
Wang et al. [241] ICDMW 2021 Conference 2021
Wang et al. [242] ICDE 2015 Conference 2015
Chen et al. [41] CMC 2009 Conference 2009
Werner and Lommatzsch [246] CLEF 2014 Conference 2014
Wongchokprasitti and Brusilovsky [250] ICAS 2007 Conference 2007
Wu et al. [251] IJCAI 2020 Conference 2020
Wu et al. [253] BigComp 2016 Conference 2016
Xie et al. [255] CIKM 2013 Conference 2013
Xue et al. [256] ETT and GRS 2008 Conference 2008
Yeung and Yang [258] DeSE 2010 Conference 2010
Yeung et al. [259] CICSyN 2010 Conference 2010
Zeleník and Bieliková [261] WEBIST 2011 Conference 2011
Zhao et al. [264] WI-IAT 2020 Conference 2020
Zhu et al. [265] ICDM 2014 Conference 2014
Bastian et al. [14] Digital Journalism Journal 2021
Bastian et al. [15] International Journal of Journal 2019
Conflict Management
Bastian et al. [16] Internet Policy Review Journal 2020
Beam [18] Communication Research Journal 2014
Bodó [24] Digital Journalism Journal 2019
Bodó et al. [25] Digital Journalism Journal 2019
Briguez et al. [29] International Journal on Artificial Journal 2013
Intelligence Tools
Burr et al. [33] Minds and Machines Journal 2018
Chakraborty et al. [39] Information Retrieval Journal Journal 2019
Chen et al. [40] IEEE Access Journal 2017
De Pessemier et al. [53] Multimedia Tools and Applications Journal 2016
Descampe et al. [56] AI & Society Journal 2022
Díaz et al. [57] Online Information Review Journal 2001
Dovbysh et al. [59] Digital Journalism Journal 2022
Eskens [63] International Data Privacy Law Journal 2019
Feng et al. [64] IEEE Access Journal 2020
Feng et al. [65] Journal of Web Engineering Journal 2021
Gharahighehi and Vens [78] Personal and Ubiquitous Computing Journal 2021
(Continued)

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
23:26 C. Bauer et al.

Table 3. Continued
Reference Journal/Conference Name Journal or Conference Year
Gharahighehi et al. [79] Information Processing & Management Journal 2021
Grön and Nelimarkka [80] Proceedings of the ACM on Journal 2020
Human-Computer Interaction
Gu et al. [81] Neural Computing and Applications Journal 2016
Gu et al. [82] The Scientific World Journal Journal 2014
Harambam et al. [86] Philosophical Transactions of the Journal 2018
Royal Society A: Mathematical,
Physical and Engineering Sciences
Heitz et al. [88] Digital Journalism Journal 2022
Helberger [90] Digital Journalism Journal 2019
Joris et al. [101] Digital Journalism Journal 2021
Karimi et al. [106] Information Processing & Management Journal 2018
Karimi et al. [107] Journal of Information Science Journal 2021
Koo et al. [115] Knowledge and Information Systems Journal 2021
Lee and Park [123] Expert Systems with Applications Journal 2007
Li et al. [126] Journalism & Mass Communication Journal 2020
Quarterly
Li et al. [130] Expert Systems with Applications Journal 2014
Li and Wang [131] IEEE Access Journal 2019
Li et al. [132] Information Sciences Journal 2010
Lu et al. [139] Journal of Systems and Software Journal 2014
Lunardi et al. [140] Applied Soft Computing Journal 2020
Makhortykh and Bastian [144] Media, War & Conflict Journal 2022
Meguebli et al. [148] World Wide Web Journal 2017
Mizgajski and Morzy [152] User Modeling and User-Adapted Journal 2019
Interaction
Møller [155] Digital Journalism Journal 2022
Montes-García et al. [156] Expert Systems with Applications Journal 2013
Monzer et al. [157] Digital Journalism Journal 2020
Nanas et al. [162] Information Processing & Management Journal 2010
Parizi et al. [175] Journal of Digital Information Journal 2016
Management
Portilla [181] El Profesional de la Información Journal 2018
Prawesh and Padmanabhan [185] Information Systems Research Journal 2014
Prawesh and Padmanabhan [187] PLOS ONE Journal 2021
Raza and Ding [195] Artificial Intelligence Review Journal 2022
Sagui et al. [202] Inteligencia Artificial Journal 2008
Saranya and Sudha Sadasivam [204] Mobile Networks and Applications Journal 2017
Semenov et al. [207] Expert Systems with Applications Journal 2022
Shin [210] Computers in Human Behavior Journal 2020
Shin [211] Journalism Studies Journal 2021
Sivetc and Wijermars [213] Media and Communication Journal 2021
Smets et al. [214] Digital Journalism Journal 2022
Turcotte et al. [231] Journal of Computer-Mediated Journal 2015
Communication
van Drunen et al. [233] International Data Privacy Law Journal 2019
Viana and Soares [236] International Journal on Artificial Journal 2017
Intelligence Tools
Wieland et al. [247] Media and Communication Journal 2021
Xiao et al. [254] China Communications Journal 2015
Yoon et al. [260] Applied Mathematics and Information Journal 2015
Sciences
Zhu et al. [266] IEEE Access Journal 2018

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
Where are the Values? A Systematic Literature Review on News Recommender Systems 23:27

REFERENCES
[1] Himan Abdollahpouri, Gediminas Adomavicius, Robin Burke, Ido Guy, Dietmar Jannach, Toshihiro Kamishima, Jan
Krasnodebski, and Luiz Pizzato. 2020. Multistakeholder recommendation: Survey and research directions. User Model.
User-adapt. Interact. 30 (2020), 127–158. DOI:https://doi.org/10.1007/s11257-019-09256-1
[2] Himan Abdollahpouri, Edward C. Malthouse, Joseph A. Konstan, Bamshad Mobasher, and Jeremy Gilbert. 2021. To-
ward the next generation of news recommender systems. In Proceedings of the Web Conference (WWW’21). ACM,
New York, NY, 402–406. DOI:https://doi.org/10.1145/3442442.3452327
[3] Panagiotis Adamopoulos. 2013. Beyond rating prediction accuracy: On new perspectives in recommender systems. In
Proceedings of the 7th ACM Conference on Recommender Systems (RecSys’13). ACM, New York, NY, 459–462. DOI:https:
//doi.org/10.1145/2507157.2508073
[4] Gediminas Adomavicius and YoungOk Kwon. 2012. Improving aggregate recommendation diversity using ranking-
based techniques. IEEE Trans. Knowl. Data Eng. 24, 5 (2012), 896–911. DOI:https://doi.org/10.1109/TKDE.2011.15
[5] Shikha Agarwal and Archana Singhal. 2014. Handling skewed results in news recommendations by focused analysis
of semantic user profiles. In Proceedings of the International Conference on Reliability Optimization and Information
Technology (ICROIT’14). IEEE, 74–79. DOI:https://doi.org/10.1109/icroit.2014.6798295
[6] Jae-wook Ahn, Peter Brusilovsky, Jonathan Grady, Daqing He, and Sue Yeon Syn. 2007. Open user profiles for adap-
tive news systems: Help or harm? In Proceedings of the 16th International Conference on World Wide Web (WWW’07).
ACM, New York, NY, 11–20. DOI:https://doi.org/10.1145/1242572.1242575
[7] Sultan Alanazi, James Goulding, and Derek McAuley. 2016. Cross-system recommendation. In Proceedings of the 27th
ACM Conference on Hypertext and Social Media (HT’16). ACM, New York, NY, 183–188. DOI:https://doi.org/10.1145/
2914586.2914640
[8] Jonathan Albright. 2017. Welcome to the era of fake news. Media Commun. 5, 2 (2017), 87–89. DOI:https://doi.org/
10.17645/mac.v5i2.977
[9] Muhammad Aljukhadar, Sylvain Senecal, and Charles-Etienne Daoust. 2012. Using recommendation agents to
cope with information overload. Int. J. Electron. Commerce 17, 2 (2012), 41–70. DOI:https://doi.org/10.2753/jec1086-
4415170202
[10] Murtaza Ashraf, Ghalib Ahmed Tahir, Sundus Abrar, Mustafa Abdulaali, Saqib Mushtaq, and Hamid Mukthar. 2018.
Personalized news recommendation based on multi-agent framework using social media preferences. In Proceedings
of the International Conference on Smart Computing and Electronic Enterprise (ICSCEE’18). 7. DOI:https://doi.org/10.
1109/icscee.2018.8538403
[11] Jalal Omer Atoum and Ibrahim Mohamed Yakti. 2017. A framework for real time news recommendations. In Proceed-
ings of the International Conference on New Trends in Computing Sciences (ICTCS’17). 89–93. DOI:https://doi.org/10.
1109/ictcs.2017.17
[12] Nastaran Babanejad, Ameeta Agrawal, Heidar Davoudi, Aijun An, and Manos Papagelis. 2019. Leveraging emotion
features in news recommendations. In Proceedings of the 7th International Workshop on News Recommendation and
Analytics (INRA’19), in Conjunction with the 13th ACM Conference on Recommender Systems (RecSys’19), Vol. 2554.
70–78. Retrieved from https://ceur-ws.org/Vol-2554/paper_10.pdf
[13] Rusul S. Bader. 2019. EANRS: An emotional Arabic news recommender system. In Proceedings of the 4th Scientific
International Conference Najaf (SICN’19). 139–144. DOI:https://doi.org/10.1109/sicn47020.2019.9019374
[14] Mariella Bastian, Natali Helberger, and Mykola Makhortykh. 2021. Safeguarding the journalistic DNA: Attitudes
towards the role of professional values in algorithmic news recommender designs. Digit. Journal. 9, 6 (2021), 835–
863. DOI:https://doi.org/10.1080/21670811.2021.1912622
[15] Mariella Bastian, Mykola Makhortykh, and Tom Dobber. 2019. News personalization for peace: How algorithmic
recommendations can impact conflict coverage. Int. J. Conf. Manag. 30, 3 (2019), 309–328. DOI:https://doi.org/10.
1108/ijcma-02-2019-0032
[16] Mariella Bastian, Mykola Makhortykh, Jaron Harambam, and Max van Drunen. 2020. Explanations of news personal-
isation across countries and media types. Internet Polic. Rev. 9, 4 (2020), 34. DOI:https://doi.org/10.14763/2020.4.1504
[17] Ruchika Bathla, Ajay Vikram Singh, and Dolly Sharma. 2015. Evaluation criteria for measuring the performance
of recommender systems. In Proceedings of the 4th International Conference on Reliability, Infocom Technologies and
Optimization: Trends and Future Directions (ICRITO’15). 6. DOI:https://doi.org/10.1109/icrito.2015.7359280
[18] Michael A. Beam. 2014. Automating the news: How personalized news recommender system design choices impact
news reception. Commun. Res. 41, 8 (2014), 1019–1041. DOI:https://doi.org/10.1177/0093650213497979
[19] Nizar Becheikh, Réjean Landry, and Nabil Amara. 2006. Lessons from innovation empirical studies in the manufac-
turing sector: A systematic review of the literature from 1993–2003. Technovation 26, 5 (2006), 644–664. DOI:https:
//doi.org/10.1016/j.technovation.2005.06.016
[20] Joeran Beel, Bela Gipp, Stefan Langer, and Corinna Breitinger. 2015. Research-paper recommender systems: A liter-
ature survey. Int. J. Digit. Libr. 17, 4 (2015), 305–338. DOI:https://doi.org/10.1007/s00799-015-0156-0

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
23:28 C. Bauer et al.

[21] Alejandro Bellogín and Alan Said. 2018. Offline and online evaluation of recommendations. In Collaborative Rec-
ommendations, Shlomo Berkovsky, Iván Cantador, and Domonkos Tikk (Eds.). 295–328. DOI:https://doi.org/10.1142/
9789813275355_0009
[22] Lennart Björneborn. 2017. Three key affordances for serendipity: Toward a framework connecting environmental
and personal factors in serendipitous encounters. J. Docum. 73, 5 (2017), 1053–1081. DOI:https://doi.org/10.1108/JD-
07-2016-0097
[23] Roi Blanco, Diego Ceccarelli, Claudio Lucchese, Raffaele Perego, and Fabrizio Silvestri. 2012. You should read this!
Let me explain you why. In Proceedings of the 21st ACM International Conference on Information and Knowledge
management (CIKM’12). ACM, New York, NY, 1995–1999. DOI:https://doi.org/10.1145/2396761.2398559
[24] Balázs Bodó. 2019. Selling news to audiences: A qualitative inquiry into the emerging logics of algorithmic news per-
sonalization in european quality news media. Digit. J. 7, 8 (2019), 1054–1075. DOI:https://doi.org/10.1080/21670811.
2019.1624185
[25] Balázs Bodó, Natali Helberger, Sarah Eskens, and Judith Möller. 2019. Interested in diversity: The role of user attitudes,
algorithmic feedback loops, and policy in news personalization. Digit. J. 7, 2 (2019), 206–229. DOI:https://doi.org/10.
1080/21670811.2018.1521292
[26] Hugo L. Borges and Ana C. Lorena. 2010. A survey on recommender systems for news data. In Smart Information
and Knowledge Management: Advances, Challenges, and Critical Issues, Edward Szczerbicki and Ngoc Thanh Nguyen
(Eds.). Springer Berlin, 129–151. DOI:https://doi.org/10.1007/978-3-642-04584-4_6
[27] Antoine Boutet, Davide Frey, Rachid Guerraoui, Arnaud Jegou, and Anne-Marie Kermarrec. 2013. WHATSUP: A
decentralized instant news recommender. In Proceedings of the IEEE 27th International Symposium on Parallel and
Distributed Processing (IPDPS’13). IEEE, 741–752. DOI:https://doi.org/10.1109/ipdps.2013.47
[28] Engin Bozdag and Ibo van de Poel. 2013. Designing for diversity in online news recommenders. In Proceedings of the
Conference on Technology Management in the IT-Driven Services (PICMET’13). 1101–1106.
[29] Cristian E. Briguez, Marcela Capobianco, and Ana G. Maguitman. 2013. A theoretical framework for trust-based
news recommender systems and its implementation using defeasible argumentation. Int. J. Artif. Intell. Tools 22, 04
(2013). DOI:https://doi.org/10.1142/s0218213013500218
[30] Torben Brodt and Frank Hopfgartner. 2014. Shedding light on a living lab: The CLEF NEWSREEL open recommen-
dation platform. In Proceedings of the 5th Information Interaction in Context Symposium (IIiX’14). ACM, New York,
NY, 223–226. DOI:https://doi.org/10.1145/2637002.2637028
[31] Axel Bruns. 2019. Are Filter Bubbles Real? Polity Press, Medford, MA.
[32] Robin Burke, Gediminas Adomavicius, Ido Guy, Jan Krasnodebski, Luiz Pizzato, Yi Zhang, and Himan Abdollahpouri.
2017. VAMS 2017: Workshop on value-aware and multistakeholder recommendation. In Proceedings of the 11th ACM
Conference on Recommender Systems (RecSys’17). ACM, New York, NY, 378–379. DOI:https://doi.org/10.1145/3109859.
3109957
[33] Christopher Burr, Nello Cristianini, and James Ladyman. 2018. An analysis of the interaction between intelligent
software agents and human users. Minds Mach. 28, 4 (2018), 735–774. DOI:https://doi.org/10.1007/s11023-018-9479-0
[34] Wanling Cai, Yucheng Jin, and Li Chen. 2022. Impacts of personal characteristics on user trust in conversational
recommender systems. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI’22). ACM,
New York, NY, Article 489, 14 pages. DOI:https://doi.org/10.1145/3491102.3517471
[35] Sirian Caldarelli, D. F. Gurini, A. Micarelli, and G. Sansonetti. 2016. A signal-based approach to news recommendation.
In UMAP 2016 Extended Proceedings (UMAP-ExtProc 2016), Vol. 1618. 4. Retrieved from https://ceur-ws.org/Vol-1618/
INRA_paper3.pdf
[36] Michel Callon and John Law. 1991. A Sociology of Monsters: Essays on Power, Technology and Domination. Routledge,
London, UK.
[37] Paris Carbone and Vladimir Vlassov. 2015. Auto-scoring of personalised news in the real-time web: Challenges,
overview and evaluation of the state-of-the-art solutions. In Proceedings of the International Conference on Cloud and
Autonomic Computing (ICCAC’15). 169–180. DOI:https://doi.org/10.1109/iccac.2015.9
[38] Abhijnan Chakraborty and Niloy Ganguly. 2018. Analyzing the news coverage of personalized newspapers. In Pro-
ceedings of the IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM’18).
IEEE, 540–543. DOI:https://doi.org/10.1109/asonam.2018.8508812
[39] Abhijnan Chakraborty, Saptarshi Ghosh, Niloy Ganguly, and Krishna P. Gummadi. 2019. Optimizing the recency-
relevance-diversity trade-offs in non-personalized news recommendations. Inf. Retriev. J. 22, 5 (2019), 447–475.
DOI:https://doi.org/10.1007/s10791-019-09351-2
[40] Cheng Chen, Xiangwu Meng, Zhenghua Xu, and Thomas Lukasiewicz. 2017. Location-aware personalized news
recommendation with deep semantic analysis. IEEE Access 5 (2017), 1624–1638. DOI:https://doi.org/10.1109/access.
2017.2655150

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
Where are the Values? A Systematic Literature Review on News Recommender Systems 23:29

[41] Wei Chen, Li-jun Zhang, Chun Chen, and Jia-jun Bu. 2009. A hybrid phonic web news recommender system for per-
vasive access. In Proceedings of the WRI International Conference on Communications and Mobile Computing (CMC’09),
Vol. 3. 122–126. DOI:https://doi.org/10.1109/cmc.2009.20
[42] Wei Chen, Li-jun Zhang, Can Wang, Chun Chen, and Jia-jun Bu. 2008. Pervasive web news recommendation for visu-
ally impaired people. In Proceedings of the IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent
Agent Technology (WI–IAT’08). 119–122. DOI:https://doi.org/10.1109/wiiat.2008.43
[43] Pascal R. Chesnais, Matthew J. Mucklo, and Jonathan A. Sheena. 1995. Fishwrap personalized news system. In Pro-
ceedings of the International Workshop on Community Networking. 275–282.
[44] Alexandru Ciobanu and Andreas Lommatzsch. 2016. Development of a news recommender system based on Apache
Flink. In Proceedings of the Conference and Labs of the Evaluation Forum (CLEF’16), Vol. 1609. 606–617. Retrieved from
https://ceur-ws.org/Vol-1609/16090606.pdf
[45] Humberto Jesús Corona Pampín and Reza Shirvany (Eds.). 2023. In Proceedings of the 4th Workshop at the Rec-
ommender Systems Conference: Recommender Systems in Fashion and Retail. Springer, Cham, Germany. DOI:https:
//doi.org/10.1007/978-3-031-22192-7
[46] Dan Cosley, Shyong K. Lam, Istvan Albert, Joseph A. Konstan, and John Riedl. 2003. Is seeing believing? How rec-
ommender system interfaces affect users’ opinions. In Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems (CHI’03). ACM, New York, NY, 585–592. DOI:https://doi.org/10.1145/642611.642713
[47] Kelley Cotter, Janghee Cho, and Emilee Rader. 2017. Explaining the news feed algorithm. In Proceedings of the CHI
Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA’17), Vol. Part F127655. 1553–1560.
DOI:https://doi.org/10.1145/3027063.3053114
[48] Chengkun Cui, Chen Wang, Cong Wen, Xiaoyin Fang, and Yuanyuan Deng. 2021. Intelligent news recommendation
system for automobile based on bio-sensing and physiological computing. In Proceedings of the 4th International Con-
ference on Signal Processing and Machine Learning (SPML’21). 107–111. DOI:https://doi.org/10.1145/3483207.3483225
[49] Jamell Dacon and Haochen Liu. 2021. Does gender matter in the news? Detecting and examining gender bias in
news articles. In Proceedings of the Web Conference (WWW’21). ACM, New York, NY, 385–392. DOI:https://doi.org/
10.1145/3442442.3452325
[50] Robert A. Dahl. 2020. On Democracy. Yale University Press, New Haven, CT. DOI:https://doi.org/10.12987/
9780300257991
[51] M. Daneshi, P. Vajda, D. M. Chen, S. S. Tsai, M. C. Yu, A. F. Araujo, H. Chen, and B. Girod. 2013. Eigennews: Generating
and delivering personalized news video. In Proceedings of the IEEE International Conference on Multimedia and Expo
Workshops (ICMEW’13). 6. DOI:https://doi.org/10.1109/icmew.2013.6618439
[52] Alvise De Biasio, Andrea Montagna, Fabio Aiolli, and Nicolò Navarin. 2023. A systematic review of value-aware
recommender systems. Expert Syst. Applic. 226, Article 120131 (2023), 16 pages. DOI:https://doi.org/10.1016/j.eswa.
2023.120131
[53] Toon De Pessemier, Cédric Courtois, Kris Vanhecke, Kristin Van Damme, Luc Martens, and Lieven De Marez. 2016.
A user-centric evaluation of context-aware recommendations for a mobile news service. Multim. Tools Applic. 75, 6
(2016), 3323–3351. DOI:https://doi.org/10.1007/s11042-014-2437-9
[54] Zohreh Dehghani Champiri, Adeleh Asemi, and Salim Siti Salwah Binti. 2019. Meta-analysis of evaluation meth-
ods and metrics used in context-aware scholarly recommender systems. Knowl. Inf. Syst. 61, 2 (2019), 1147–1178.
DOI:https://doi.org/10.1007/s10115-018-1324-5
[55] Maunendra Sankar Desarkar and Neha Shinde. 2014. Diversification in news recommendation for privacy concerned
users. In Proceedings of the International Conference on Data Science and Advanced Analytics (DSAA’14). 135–141.
DOI:https://doi.org/10.1109/dsaa.2014.7058064
[56] Antonin Descampe, Clément Massart, Simon Poelman, François-Xavier Standaert, and Olivier Standaert. 2022. Auto-
mated news recommendation in front of adversarial examples and the technical limits of transparency in algorithmic
accountability. AI Societ. 37, 1 (2022), 67–80. DOI:https://doi.org/10.1007/s00146-021-01159-3
[57] Alberto Díaz, Pablo Gervás, Antonio García, and Inmaculada Chacón. 2001. Sections, categories and keywords as
interest specification tools for personalised news services. Online Inf. Rev. 25, 3 (2001), 149–160. DOI:https://doi.org/
10.1108/14684520110395272
[58] Hao Ding, Qing Liu, and Guangwei Hu. 2022. TDTMF: A recommendation model based on user temporal interest
drift and latent review topic evolution with regularization factor. Inf. Process. Manag. 59, 5, Article 103037 (2022).
DOI:https://doi.org/10.1016/j.ipm.2022.103037
[59] Olga Dovbysh, Mariëlle Wijermars, and Mykola Makhortykh. 2022. How to reach nirvana: Yandex, news personal-
isation, and the future of Russian journalistic media. Digit. J. 10, 10 (2022), 1855–1874. DOI:https://doi.org/10.1080/
21670811.2021.2024080
[60] Sanjay K. Dwivedi and Chandrakala Arya. 2016. A survey of news recommendation approaches. In Proceedings of
the International Conference on ICT in Business Industry & Government (ICTBIG’16). IEEE, 6. DOI:https://doi.org/10.
1109/ICTBIG.2016.7892681

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
23:30 C. Bauer et al.

[61] Michael D. Ekstrand, Anubrata Das, Robin Burke, and Fernando Diaz. 2022. Fairness in information access systems.
Found. Trends Inf. Retr. 16, 1–2 (2022), 177. DOI:https://doi.org/10.1561/1500000079
[62] Elena Viorica Epure, Benjamin Kille, Jon Espen Ingvaldsen, Rebecca Deneckere, Camille Salinesi, and Sahin Albayrak.
2017. Recommending personalized news in short user sessions. In Proceedings of the 11th ACM Conference on Recom-
mender Systems (RecSys’17). ACM, New York, NY, 121–129. DOI:https://doi.org/10.1145/3109859.3109894
[63] Sarah Eskens. 2019. A right to reset your user profile and more: GDPR-rights for personalized news consumers. Int.
Data Privac. Law 9, 3 (2019), 153–172. DOI:https://doi.org/10.1093/idpl/ipz007
[64] Chong Feng, Muzammil Khan, Arif Ur Rahman, and Arshad Ahmad. 2020. News recommendation systems: Accom-
plishments, challenges & future directions. IEEE Access 8 (2020), 16702–16725. DOI:https://doi.org/10.1109/access.
2020.2967792
[65] Shuaishuai Feng, Junyan Meng, and Jiaxing Zhang. 2021. News recommendation systems in the era of information
overload. J. Web Eng. 20, 2 (2021), 459–470. DOI:https://doi.org/10.13052/jwe1540-9589.20210
[66] Batya Friedman. 1996. Value-sensitive design. Interactions 3, 6 (1996), 16–23. DOI:https://doi.org/10.1145/242485.
242493
[67] Batya Friedman and David G. Hendry. 2019. Value Sensitive Design: Shaping Technology with Moral Imagination. MIT
Press.
[68] Batya Friedman, Peter H. Kahn, Alan Borning, and Alina Huldtgren. 2013. Value sensitive design and information
systems. Early Engag. New Technol.: Open. Lab. 16 (2023), 55–95. DOI:https://doi.org/10.1007/978-94-007-7844-3_4
[69] Evgeniy Gabrilovich, Susan Dumais, and Eric Horvitz. 2004. Newsjunkie: Providing personalized newsfeeds via anal-
ysis of information novelty. In Proceedings of the 13th International Conference on World Wide Web (WWW’04). ACM,
New York, NY, 482–490. DOI:https://doi.org/10.1145/988672.988738
[70] Qi Gao, Fabian Abel, Geert-Jan Houben, and Ke Tao. 2011. Interweaving trend and user modeling for personalized
news recommendation. In Proceedings of the IEEE/WIC/ACM International Conferences on Web Intelligence and Intel-
ligent Agent Technology (WI-IAT’11), Vol. 1. 100–103. DOI:https://doi.org/10.1109/wi-iat.2011.74
[71] Yihang Gao, Hui Zhao, Qian Zhou, Meikang Qiu, and Meiqin Liu. 2020. An improved news recommendation algo-
rithm based on text similarity. In Proceedings of the 3rd International Conference on Smart BlockChain (SmartBlock’20).
132–136. DOI:https://doi.org/10.1109/SmartBlock52591.2020.00031
[72] Florent Garcin, Christos Dimitrakakis, and Boi Faltings. 2013. Personalized news recommendation with context
trees. In Proceedings of the 7th ACM Conference on Recommender Systems (RecSys’13). ACM, New York, NY, 105–112.
DOI:https://doi.org/10.1145/2507157.2507166
[73] Florent Garcin, Boi Faltings, Olivier Donatsch, Ayar Alazzawi, Christophe Bruttin, and Amr Huber. 2014. Offline
and online evaluation of news recommender systems at swissinfo.ch. In Proceedings of the 8th ACM Conference on
Recommender systems (RecSys’14). ACM, New York, NY, 169–176. DOI:https://doi.org/10.1145/2645710.2645745
[74] Angel Luis Garrido, María G. Buey, Sergio Ilarri, Igor Fürstner, and Livia Szedmina. 2015. KGNR: A knowledge-based
geographical news recommender. In Proceedings of the IEEE 13th International Symposium on Intelligent Systems and
Informatics (SISY’15). 195–198. DOI:https://doi.org/10.1109/sisy.2015.7325378
[75] Mouzhi Ge, Carla Delgado-Battenfeld, and Dietmar Jannach. 2010. Beyond accuracy: Evaluating recommender sys-
tems by coverage and serendipity. In Proceedings of the 4th ACM Conference on Recommender Systems (RecSys’10).
ACM, New York, NY, 257–260. DOI:https://doi.org/10.1145/1864708.1864761
[76] Gebrekirstos G. Gebremeskel and Arjen P. de Vries. 2015. The degree of randomness in a live recommender systems
evaluation. In CLEF2015 Working Notes (CLEF’15), Vol. 1391. Retrieved from https://ceur-ws.org/Vol-1391/155-CR.
pdf
[77] Alireza Gharahighehi and Celine Vens. 2020. Making session-based news recommenders diversity-aware. In Pro-
ceedings of the Workshop on Online Misinformation- and Harm-aware Recommender Systems, co-located with the 14th
ACM Conference on Recommender Systems (RecSys’20) (OHARS’20), Vol. 2758. 60–66. Retrieved from https://ceur-
ws.org/Vol-2758/OHARS-paper5.pdf
[78] Alireza Gharahighehi and Celine Vens. 2021. Diversification in session-based news recommender systems. Person.
Ubiq. Comput. 27, 1 (2021), 5–15. DOI:https://doi.org/10.1007/s00779-021-01606-4
[79] Alireza Gharahighehi, Celine Vens, and Konstantinos Pliakos. 2021. Fair multi-stakeholder news recommender sys-
tem with hypergraph ranking. Inf. Process. Manag. 58, 5 (2021). DOI:https://doi.org/10.1016/j.ipm.2021.102663
[80] Kirsikka Grön and Matti Nelimarkka. 2020. Party politics, values and the design of social media services. Proc. ACM
Hum.-comput. Interact. 4, CSCW2 (2020), 29. DOI:https://doi.org/10.1145/3415175
[81] Wanrong Gu, Shoubin Dong, and Mingquan Chen. 2016. Personalized news recommendation based on articles chain
building. Neural Comput. Applic. 27, 5 (2016), 1263–1272. DOI:https://doi.org/10.1007/s00521-015-1932-x
[82] Wanrong Gu, Shoubin Dong, Zhizhao Zeng, and Jinchao He. 2014. An effective news recommendation method for
microblog user. Scient. World J. 2014 (2014), 14. DOI:https://doi.org/10.1155/2014/907515

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
Where are the Values? A Systematic Literature Review on News Recommender Systems 23:31

[83] Jon Atle Gulla, Arne Dag Fidjestøl, Jon Espen Ingvaldsen, Cristina Marco, Xiaomeng Su, and Özlem Özgöbek. 2016.
The intricacies of time in news recommendation. In UMAP 2016 Extended Proceedings (UMAP-ExtProc’16), Vol. 1618.
Retrieved from https://ceur-ws.org/Vol-1618/INRA_paper4.pdf
[84] Asela Gunawardana, Guy Shani, and Sivan Yogev. 2022. Evaluating recommender systems. In Recommender Systems
Handbook, Francesco Ricci, Lior Rokach, and Bracha Shapira (Eds.). Springer US, New York, NY, 547–601. DOI:https:
//doi.org/10.1007/978-1-0716-2197-4_15
[85] Jaron Harambam, Dimitrios Bountouridis, Mykola Makhortykh, and Joris van Hoboken. 2019. Designing for the
better by taking users into account: A qualitative evaluation of user control mechanisms in (news) recommender
systems. In Proceedings of the 13th ACM Conference on Recommender Systems (RecSys’19). ACM, New York, NY, 69–
77. DOI:https://doi.org/10.1145/3298689.3347014
[86] Jaron Harambam, Natali Helberger, and Joris van Hoboken. 2018. Democratizing algorithmic news recommenders:
How to materialize voice in a technologically saturated media ecosystem. Philos. Trans. R. Societ. A: Math., Phys. Eng.
Sci. 376, 2133 (2018), 21. DOI:https://doi.org/10.1098/rsta.2018.0088
[87] Taka Hassan and D. Scott McCrickard. 2019. Trust and trustworthiness in social recommender systems. In Proceedings
of the World Wide Web Conference (WWW’19). ACM, New York, NY, 529–532. DOI:https://doi.org/10.1145/3308560.
3317596
[88] Lucien Heitz, Juliane A. Lischka, Alena Birrer, Bibek Paudel, Suzanne Tolmeijer, Laura Laugwitz, and Abraham
Bernstein. 2022. Benefits of diverse news recommendations for democracy: A user study. Digit. J. 10, 10 (2022), 1710–
1730. DOI:https://doi.org/10.1080/21670811.2021.2021804
[89] Natali Helberger. 2011. Diversity by design. J. Inf. Polic. 1 (2011), 441–469. DOI:https://doi.org/10.5325/jinfopoli.1.
2011.0441
[90] Natali Helberger. 2019. On the democratic role of news recommenders. Digit. J. 7, 8 (2019), 993–1012. DOI:https:
//doi.org/10.1080/21670811.2019.1623700
[91] Natali Helberger, Kari Karppinen, and Lucia D’Acunto. 2018. Exposure diversity as a design principle for recom-
mender systems. Inf., Commun. Societ. 21, 2 (2018), 191–207. DOI:https://doi.org/10.1080/1369118X.2016.1271900
[92] Jonathan L. Herlocker, Joseph A. Konstan, Loren G. Terveen, and John T. Riedl. 2004. Evaluating collaborative filtering
recommender systems. ACM Trans. Inf. Syst. 22, 1 (Jan. 2004), 5–53. DOI:https://doi.org/10.1145/963770.963772
[93] Chuan Hu, Chen Zhang, Tiejun Wang, and Qing Li. 2012. An adaptive recommendation system in social media. In
Proceedings of the 45th Hawaii International Conference on System Sciences (HICSS’12). 1759–1767. DOI:https://doi.
org/10.1109/hicss.2012.94
[94] Jon Espen Ingvaldsen, Jon Atle Gulla, and Özlem Özgöbek. 2015. User controlled news recommendations. In Proceed-
ings of the Joint Workshop on Interfaces and Human Decision Making for Recommender Systems, co-located with ACM
Conference on Recommender Systems (IntRS@RecSys’15), Vol. 1438. 45–48. Retrieved from https://ceur-ws.org/Vol-
1438/paper8.pdf
[95] Rania Islambouli, Sandy Ingram, and Denis Gillet. 2021. A user centered news recommendation system. In Proceedings
of the 4th Workshop on Human Factors in Hypertext (HUMAN’21). 15–16. DOI:https://doi.org/10.1145/3468143.3483931
[96] Shashank Jain, Vivek Tiwari, Aruna Balasubramanian, Niranjan Balasubramanian, and Supriyo Chakraborty. 2017.
PrIA: A private intelligent assistant. In Proceedings of the 18th International Workshop on Mobile Computing Systems
and Applications (HotMobile’17). 91–96. DOI:https://doi.org/10.1145/3032970.3032988
[97] Dietmar Jannach and Christine Bauer. 2020. Escaping the McNamara fallacy: Toward more impactful recommender
systems research. AI Mag. 41, 4 (2020), 79–95. DOI:https://doi.org/10.1609/aimag.v41i4.5312
[98] Dietmar Jannach, Paul Resnick, Alexander Tuzhilin, and Markus Zanker. 2016. Recommender systems—Beyond ma-
trix completion. Commun. ACM 59, 11 (2016), 94–102. DOI:https://doi.org/10.1145/2891406
[99] Arjan J. P. Jeckmans, Michael Beye, Zekeriya Erkin, Pieter Hartel, Reginald L. Lagendijk, and Qiang Tang. 2013.
Privacy in recommender systems. In Social Media Retrieval, Naeem Ramzan, Roelof van Zwol, Jong-Seok Lee, Kai
Clüver, and Xian-Sheng Hua (Eds.). Springer, London, UK, 263–281. DOI:https://doi.org/10.1007/978-1-4471-4555-
4_12
[100] Mathias Jesse, Christine Bauer, and Dietmar Jannach. 2022. Intra-list similarity and human diversity perceptions
of recommendations: the details matter. User Model. User-adapt. Interact. 33 (2022), 769–802. DOI:https://doi.org/10.
1007/s11257-022-09351-w
[101] Glen Joris, Frederik De Grove, Kristin Van Damme, and Lieven De Marez. 2021. Appreciating news algorithms: Ex-
amining audiences’ perceptions to different news selection mechanisms. Digit. J. 9, 5 (2021), 589–618. DOI:https:
//doi.org/10.1080/21670811.2021.1912626
[102] Antonis Kalogeropoulos, Jane Suiter, Linards Udris, and Mark Eisenegger. 2019. News media trust and news con-
sumption: Factors related to trust in news in 35 countries. Int. J. Commun. 13 (2019). Retrieved from https://ijoc.org/
index.php/ijoc/article/view/10141

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
23:32 C. Bauer et al.

[103] Marius Kaminskas and Derek Bridge. 2017. Diversity, serendipity, novelty, and coverage: A survey and empirical
analysis of beyond-accuracy objectives in recommender systems. ACM Trans. Interact. Intell. Syst. 7, 1, Article 2
(2017), 42 pages. DOI:https://doi.org/10.1145/2926720
[104] Dong-Yup Kang, Dong-Kyun Han, Gyumin Sim, Jong Hyuk Jung, Hyun Ki-Jeon, Soobin Lee, Joonyoung Park,
and Seunghyeon Moon. 2014. PADAC2 : Real-time news recommendation system with heterogeneous social foot-
prints. In Proceedings of the 16th International Conference on Advanced Communication Technology (ICACT’14). 79–82.
DOI:https://doi.org/10.1109/icact.2014.6778925
[105] Mozhgan Karimi, Boris Cule, and Bart Goethals. 2019. On-the-fly news recommendation using sequential patterns. In
Proceedings of the 7th International Workshop on News Recommendation and Analytics (INRA’19), in conjunction with
13th ACM Conference on Recommender Systems (RecSys’19), Vol. 2554. 29–34. Retrieved from https://ceur-ws.org/Vol-
2554/paper_05.pdf
[106] Mozhgan Karimi, Dietmar Jannach, and Michael Jugovac. 2018. News recommender systems: Survey and roads ahead.
Inf. Process. Manag. 54, 6 (2018), 1203–1227. DOI:https://doi.org/10.1016/j.ipm.2018.04.008
[107] Samaneh Karimi, Azadeh Shakery, and Rakesh Verma. 2021. Online news media website ranking using user-
generated content. J. Inf. Sci. 47, 3 (2021), 340–358. DOI:https://doi.org/10.1177/0165551519894928
[108] Rune Karlsen and Toril Aalberg. 2023. Social media and trust in news: An experimental study of the effect of Facebook
on news story credibility. Digit. J. 11, 1 (2023), 144–160. DOI:https://doi.org/10.1080/21670811.2021.1945938
[109] Bhumika D. Karwa and Madhushree B. 2015. A survey on various techniques of personalized news recommendation
system. Int. J. Sci. Advanc. Res. Technol. 1, 12 (2015), 44–50.
[110] Gabriella Kazai, Iskander Yusof, and Daoud Clarke. 2016. Personalised news and blog recommendations based on
user location, Facebook and Twitter user profiling. In Proceedings of the 39th International ACM SIGIR Conference on
Research and Development in Information Retrieval (SIGIR’16). ACM, New York, NY, 1129–1132. DOI:https://doi.org/
10.1145/2911451.2911464
[111] Krishnaram Kenthapadi, Benjamin Le, and Ganesh Venkataraman. 2017. Personalized job recommendation system
at LinkedIn: Practical challenges and lessons learned. In Proceedings of the 11th ACM Conference on Recommender
Systems (RecSys’17). ACM, New York, NY, 346–347. DOI:https://doi.org/10.1145/3109859.3109921
[112] Dhruv Khattar, Vaibhav Kumar, and Vasudeva Varma. 2017. Leveraging moderate user data for news recommen-
dation. In Proceedings of the IEEE International Conference on Data Mining Workshops (ICDMW’17). IEEE, 757–760.
DOI:https://doi.org/10.1109/icdmw.2017.104
[113] Benjamin Kille and Sahin Albayrak. 2017. On the decaying utility of news recommendation models. In Proceedings
of the 1st Workshop on Temporal Reasoning in Recommender Systems, co-located with 11th International Conference on
Recommender Systems (RecSys’17) (RecTemp’17), Vol. 1922. 4–8. Retrieved from https://ceur-ws.org/Vol-1922/paper2.
pdf
[114] Barbara Kitchenham, Stuart Charters, David Budgen, Pearl Brereton, Mark Turner, Steve Linkman, Magne Jørgensen,
Emilia Mendes, and Giuseppe Visaggio. 2007. Guidelines for Performing Systematic Literature Reviews in Software
Engineering. EBSE Technical Report EBSE-2007-01, version 2.3. Keele University and University of Durham.
[115] Bonhun Koo, Hyunsik Jeon, and U. Kang. 2021. PGT: News recommendation coalescing personal and global temporal
preferences. Knowl. Inf. Syst. 63, 12 (2021), 3139–3158. DOI:https://doi.org/10.1007/s10115-021-01618-9
[116] Sascha Kraus, Matthias Breier, and Sonia Dasí-Rodríguez. 2020. The art of crafting a systematic literature review in
entrepreneurship research. Int. Entrepren. Manag. J. 16, 3 (2020), 1023–1042. DOI:https://doi.org/10.1007/s11365-020-
00635-4
[117] Luciana Monteiro Krebs, Oscar Luis Alvarado Rodriguez, Pierre Dewitte, Jef Ausloos, David Geerts, Laurens Naudts,
and Katrien Verbert. 2019. Tell me what you know: GDPR implications on designing transparency and accountability
for news recommender systems. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI
EA’19). 6. DOI:https://doi.org/10.1145/3290607.3312808
[118] Hrishikesh Kulkarni, Tejas Joshi, Nikhil Sanap, Rohan Kalyanpur, and Manisha Marathe. 2019. Personalized newspa-
per based on emotional traits using machine learning. In Proceedings of the 5th International Conference on Computing,
Communication, Control and Automation (ICCUBEA’19). 5. DOI:https://doi.org/10.1109/iccubea47591.2019.9128691
[119] Vaibhav Kumar, Dhruv Khattar, Shashank Gupta, Manish Gupta, and Vasudeva Varma. 2017. User profiling based
deep neural network for temporal news recommendation. In Proceedings of the IEEE International Conference on Data
Mining Workshops (ICDMW’17). IEEE, 765–772. DOI:https://doi.org/10.1109/icdmw.2017.106
[120] Vaibhav Kumar, Dhruv Khattar, Shashank Gupta, and Vasudeva Varma. 2017. Word semantics based 3-D convolu-
tional neural networks for news recommendation. In Proceedings of the IEEE International Conference on Data Mining
Workshops (ICDMW’17). IEEE, 761–764. DOI:https://doi.org/10.1109/icdmw.2017.105
[121] Matevž Kunaver and Tomaž Požrl. 2017. Diversity in recommender systems—A survey. Knowl.-based Syst. 123 (2017),
154–162. DOI:https://doi.org/10.1016/j.knosys.2017.02.009

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
Where are the Values? A Systematic Literature Review on News Recommender Systems 23:33

[122] Shiksha Kushwah, Amandeep Dhir, Mahim Sagar, and Bhumika Gupta. 2019. Determinants of organic food con-
sumption: A systematic literature review on motives and barriers. Appetite 143, Article 104402 (2019). DOI:https:
//doi.org/10.1016/j.appet.2019.104402
[123] H. J. Lee and Sung Joo Park. 2007. MONERS: A news recommender for the mobile web. Expert Syst. Applic. 32, 1
(2007), 143–150. DOI:https://doi.org/10.1016/j.eswa.2005.11.010
[124] Philip Lenhart and Daniel Herzog. 2016. Combining content-based and collaborative filtering for personalized sports
news recommendations. In Proceedings of the 3rd Workshop on New Trends in Content-based Recommender Systems,
co-located with ACM Conference on Recommender Systems (RecSys’16) (CBRecSys’16), Vol. 1673. 3–10. Retrieved from
https://ceur-ws.org/Vol-1673/paper1.pdf
[125] Bruno Lepri, Nuria Oliver, Emmanuel Letouzé, Alex Pentland, and Patrick Vinck. 2018. Fair, transparent, and account-
able algorithmic decision-making processes: The premise, the proposed solutions, and the open challenges. Philos.
Technol. 31 (2018), 611–627. DOI:https://doi.org/10.1007/s13347-017-0279-x
[126] Cong Li, Cheng Hong, and Zifei Fay Chen. 2020. Effects of uniqueness, news valence, and liking on personalization of
company news. Journal. Mass Commun. Quart. 97, 4 (2020), 890–912. DOI:https://doi.org/10.1177/1077699020923604
[127] Lei Li, Dingding Wang, Tao Li, Daniel Knox, and Balaji Padmanabhan. 2011. SCENE: A scalable two-stage personal-
ized news recommendation system. In Proceedings of the 34th International ACM SIGIR Conference on Research and
Development in Information Retrieval (SIGIR’11). ACM, New York, NY, 125–134. DOI:https://doi.org/10.1145/2009916.
2009937
[128] Lei Li, Ding-Ding Wang, Shun-Zhi Zhu, and Tao Li. 2011. Personalized news recommendation: A review and an
experimental investigation. J. Comput. Sci. Technol. 26, 5 (2011), 754–766. DOI:https://doi.org/10.1007/s11390-011-
0175-2
[129] Lei Li, Li Zheng, and Tao Li. 2011. LOGO: A long-short user interest integration in personalized news recommenda-
tion. In Proceedings of the 5th ACM Conference on Recommender Systems (RecSys’11). ACM, New York, NY, 317–320.
DOI:https://doi.org/10.1145/2043932.2043992
[130] Lei Li, Li Zheng, Fan Yang, and Tao Li. 2014. Modeling and broadening temporal user interest in personalized news
recommendation. Expert Syst. Applic. 41, 7 (2014), 3168–3177. DOI:https://doi.org/10.1016/j.eswa.2013.11.020
[131] Miaomiao Li and Licheng Wang. 2019. A survey on personalized news recommendation technology. IEEE Access 7
(2019), 145861–145879. DOI:https://doi.org/10.1109/access.2019.2944927
[132] Qing Li, Jia Wang, Yuanzhu Peter Chen, and Zhangxi Lin. 2010. User comments for news recommendation in forum-
based social media. Inf. Sci. 180, 24 (2010), 4929–4939. DOI:https://doi.org/10.1016/j.ins.2010.08.044
[133] Ping Liu, Karthik Shivaram, Aron Culotta, Matthew A. Shapiro, and Mustafa Bilgic. 2021. The interaction between
political typology and filter bubbles in news recommendation algorithms. In Proceedings of the Web Conference
(WWW’21). ACM, New York, NY, 3791–3801. DOI:https://doi.org/10.1145/3442381.3450113
[134] Felicia Loecherbach, Kasper Welbers, Judith Möller, Damian Trilling, and Wouter van Atteveldt. 2021. Is this a click
towards diversity? Explaining when and why news users make diverse choices. In Proceedings of the 13th ACM Web
Science Conference (WWW’21). ACM, New York, NY, 282–290. DOI:https://doi.org/10.1145/3447535.3462506
[135] Andreas Lommatzsch, Benjamin Kille, and Sahin Albayrak. 2017. Incorporating context and trends in news recom-
mender systems. In Proceedings of the International Conference on Web Intelligence (WI’17). ACM, New York, NY,
1062–1068. DOI:https://doi.org/10.1145/3106426.3109433
[136] Feng Lu, Anca Dumitrache, and David Graus. 2020. Beyond optimizing for clicks: Incorporating editorial values in
news recommendation. In Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization
(UMAP’20). ACM, New York, NY, 145–153. DOI:https://doi.org/10.1145/3340631.3394864
[137] Hongyu Lu, Min Zhang, Weizhi Ma, Ce Wang, Feng Xia, Yiqun Liu, Leyu Lin, and Shaoping Ma. 2019. Effects of
user negative experience in mobile news streaming. In Proceedings of the 42nd International ACM SIGIR Conference
on Research and Development in Information Retrieval (SIGIR’19). ACM, New York, NY, 705–714. DOI:https://doi.org/
10.1145/3331184.3331247
[138] Meilian Lu and Jinliang Liu. 2016. Hier-UIM: A hierarchy user interest model for personalized news recommender. In
Proceedings of the 4th International Conference on Cloud Computing and Intelligence Systems (CCIS’16). IEEE, 249–254.
DOI:https://doi.org/10.1109/ccis.2016.7790263
[139] Meilian Lu, Zhen Qin, Yiming Cao, Zhichao Liu, and Mengxing Wang. 2014. Scalable news recommendation using
multi-dimensional similarity and Jaccard-Kmeans clustering. J. Syst. Softw. 95 (2014), 242–251. DOI:https://doi.org/
10.1016/j.jss.2014.04.046
[140] Gabriel Machado Lunardi, Guilherme Medeiros Machado, Vinicius Maran, and José Palazzo M. de Oliveira. 2020. A
metric for filter bubble measurement in recommender algorithms considering the news domain. Appl. Soft Comput.
97, Article 106771 (2020), 12 pages. DOI:https://doi.org/10.1016/j.asoc.2020.106771
[141] Yuanhua Lv, Taesup Moon, Pranam Kolari, Zhaohui Zheng, Xuanhui Wang, and Yi Chang. 2011. Learning to model
relatedness for news recommendation. In Proceedings of the 20th International Conference on World Wide Web
(WWW’11). ACM, New York, NY, 57–66. DOI:https://doi.org/10.1145/1963405.1963417

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
23:34 C. Bauer et al.

[142] Hao Ma, Xueqing Liu, and Zhihong Shen. 2016. User fatigue in online news recommendation. In Proceedings of the
25th International Conference on World Wide Web (WWW’16). ACM, New York, NY, 1363–1372. DOI:https://doi.org/
10.1145/2872427.2874813
[143] Donald MacKenzie and Judy Wajcman. 1999. The Social Shaping of Technology. Open University Press, Buckingham,
UK.
[144] Mykola Makhortykh and Mariella Bastian. 2022. Personalizing the war: Perspectives for the adoption of news rec-
ommendation algorithms in the media coverage of the conflict in Eastern Ukraine. Media, War Confl. 15, 1 (2022),
25–45. DOI:https://doi.org/10.1177/1750635220906254
[145] Andrii Maksai, Florent Garcin, and Boi Faltings. 2015. Predicting online performance of news recommender systems
through richer evaluation metrics. In Proceedings of the 9th ACM Conference on Recommender Systems (RecSys’15).
ACM, New York, NY, 179–186. DOI:https://doi.org/10.1145/2792838.2800184
[146] Sean M. McNee, John Riedl, and Joseph A. Konstan. 2006. Being accurate is not enough: How accuracy metrics have
hurt recommender systems. In CHI’06 Extended Abstracts on Human Factors in Computing Systems (CHI EA’06). ACM,
New York, NY, 1097–1101. DOI:https://doi.org/10.1145/1125451.1125659
[147] Youssef Meguebli, Mouna Kacimi, Bich-Liên Doan, and Fabrice Popineau. 2014. Stories around you: A two-stage
personalized news recommendation. In Proceedings of the International Conference on Knowledge Discovery and In-
formation Retrieval (KDIR’14). 473–479. DOI:https://doi.org/10.5220/0005159804730479
[148] Youssef Meguebli, Mouna Kacimi, Bich-Liên Doan, and Fabrice Popineau. 2017. Towards better news article recom-
mendation. World Wide Web 20, 6 (2017), 1293–1312. DOI:https://doi.org/10.1007/s11280-017-0436-2
[149] Lien Michiels, Jens Leysen, Annelien Smets, and Bart Goethals. 2022. What are filter bubbles really? A review of the
conceptual and empirical work. In Adjunct Proceedings of the 30th ACM Conference on User Modeling, Adaptation and
Personalization (UMAP’22 Adjunct). ACM, New York, NY, 274–279. DOI:https://doi.org/10.1145/3511047.3538028
[150] Silvia Milano, Mariarosaria Taddeo, and Luciano Floridi. 2020. Recommender systems and their ethical challenges.
AI Societ. 35, 4 (2020), 957–967. DOI:https://doi.org/10.1007/s00146-020-00950-y
[151] Eliza Mitova, Sina Blassnig, Edina Strikovic, Aleksandra Urman, Aniko Hannak, Claes H. de Vreese, and Frank Esser.
2022. News recommender systems: A programmatic research review. Ann. Int. Commun. Assoc. 47, 1 (2022), 84–113.
DOI:https://doi.org/10.1080/23808985.2022.2142149
[152] Jan Mizgajski and Mikołaj Morzy. 2019. Affective recommender systems in online news industry: How emotions
influence reading choices. User Model. User-adapt. Interact. 29, 2 (2019), 345–379. DOI:https://doi.org/10.1007/s11257-
018-9213-x
[153] Itishree Mohallick and Özlem Özgöbek. 2017. Exploring privacy concerns in news recommender systems. In Pro-
ceedings of the IEEE/WIC/ACM International Conference on Web Intelligence (WI’17). 1054–1061. DOI:https://doi.org/
10.1145/3106426.3109435
[154] Judith Möller, Damian Trilling, Natali Helberger, and Bram van Es. 2018. Do not blame it on the algorithm: An
empirical assessment of multiple recommender systems and their impact on content diversity. Inf., Commun. Societ.
21, 7 (2018), 959–977. DOI:https://doi.org/10.1080/1369118X.2018.1444076
[155] Lynge Asbjørn Møller. 2022. Between personal and public interest: How algorithmic news recommendation recon-
ciles with journalism as an ideology. Digit. J. 10, 10 (2022), 1794–1812. DOI:https://doi.org/10.1080/21670811.2022.
2032782
[156] Alejandro Montes-García, Jose María Álvarez-Rodríguez, Jose Emilio Labra-Gayo, and Marcos Martínez-Merino.
2013. Towards a journalist-based news recommendation system: The Wesomender approach. Expert Syst. Applic.
40, 17 (2013), 6735–6741. DOI:https://doi.org/10.1016/j.eswa.2013.06.032
[157] Cristina Monzer, Judith Möller, Natali Helberger, and Sarah Eskens. 2020. User perspectives on the news personal-
isation process: Agency, trust and utility as building blocks. Digit. J. 8, 9 (2020), 1142–1162. DOI:https://doi.org/10.
1080/21670811.2020.1773291
[158] Mats Mulder, Oana Inel, Jasper Oosterman, and Nava Tintarev. 2021. Operationalizing framing to support multiper-
spective recommendations of opinion pieces. In Proceedings of the ACM Conference on Fairness, Accountability, and
Transparency (FAccT’21). ACM, New York, NY, 478–488. DOI:https://doi.org/10.1145/3442188.3445911
[159] Dylan Mulvin. 2021. Proxies: The Cultural Work of Standing in. MIT Press, Cambridge, MA. DOI:https://doi.org/10.
7551/mitpress/11765.001.0001
[160] Nikhil Muralidhar, Huzefa Rangwala, and Eui-Hong Sam Han. 2015. Recommending temporally relevant news con-
tent from implicit feedback data. In Proceedings of the IEEE 27th International Conference on Tools with Artificial
Intelligence (ICTAI’15). IEEE, 689–696. DOI:https://doi.org/10.1109/ictai.2015.104
[161] Saki Nagaki, Yuto Yamaguchi, Toshiyuki Amagasa, and Hiroyuki Kitagawa. 2016. Local attention analysis and pre-
diction of online news articles in Twitter. In Adjunct Proceedings of the 13th International Conference on Mobile and
Ubiquitous Systems: Computing Networking and Services (MOBIQUITOUS’16). 136–141. DOI:https://doi.org/10.1145/
3004010.3004042

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
Where are the Values? A Systematic Literature Review on News Recommender Systems 23:35

[162] Nikolaos Nanas, Manolis Vavalis, and Elias Houstis. 2010. Personalised news and scientific literature aggregation.
Inf. Process. Manag. 46, 3 (2010), 268–283. DOI:https://doi.org/10.1016/j.ipm.2009.07.005
[163] Suraj Natarajan and Melody Moh. 2016. Recommending news based on hybrid user profile, popularity, trends, and
location. In Proceedings of the International Conference on Collaboration Technologies and Systems (CTS’16). 204–211.
DOI:https://doi.org/10.1109/CTS.2016.48
[164] Efrat Nechushtai and Seth C. Lewis. 2019. What kind of news gatekeepers do we want machines to be? Filter bubbles,
fragmentation, and the normative dimensions of algorithmic recommendations. Comput. Hum. Behav. 90 (2019), 298–
307. DOI:https://doi.org/10.1016/j.chb.2018.07.043
[165] Xi Niu, Fakhri Abbas, Mary Lou Maher, and Kazjon Grace. 2018. Surprise me if you can: Serendipity in health in-
formation. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI’18). 12. DOI:https:
//doi.org/10.1145/3173574.3173597
[166] Xi Niu and Ahmad Al-Doulat. 2021. LuckyFind: Leveraging surprise to improve user satisfaction and inspire curiosity
in a recommender system. In Proceedings of the Conference on Human Information Interaction and Retrieval (CHIIR’21).
163–172. DOI:https://doi.org/10.1145/3406522.3446017
[167] Yunseok Noh, Yong-Hwan Oh, and Seong-Bae Park. 2014. A location-based personalized news recommendation.
In Proceedings of the International Conference on Big Data and Smart Computing (BIGCOMP’14). 99–104. DOI:https:
//doi.org/10.1109/BIGCOMP.2014.6741416
[168] Shawn O’Banion, Larry Birnbaum, and Kristian Hammond. 2012. Social media-driven news personalization. In
Proceedings of the 4th ACM RecSys Workshop on Recommender Systems and the Social Web (RSWeb’12). 45–52.
DOI:https://doi.org/10.1145/2365934.2365943
[169] John O’Donovan and Barry Smyth. 2005. Trust in recommender systems. In Proceedings of the 10th International
Conference on Intelligent User Interfaces (IUI’05). ACM, New York, NY, 167–174. DOI:https://doi.org/10.1145/1040830.
1040870
[170] Kyo-Joong Oh, Won-Jo Lee, Chae-Gyun Lim, and Ho-Jin Choi. 2014. Personalized news recommendation using clas-
sified keywords to capture user preference. In Proceedings of the 16th International Conference on Advanced Commu-
nication Technology (ICACT’14). 1283–1287. DOI:https://doi.org/10.1109/ICACT.2014.6779166
[171] Özlem Özgöbek, Jon Atle Gulla, and R. Cenk Erdur. 2014. A survey on challenges and methods in news recommen-
dation. In Proceedings of the 10th International Conference on Web Information Systems and Technologies (WEBIST’14),
Vol. 2. 278–285. DOI:https://doi.org/10.5220/0004844202780285
[172] Özlem Özgöbek, Jon Atle Gulla, and R. Cenk Erdur. 2015. Recommending sources in news recommender systems. In
Proceedings of the 11th International Conference on Web Information Systems and Technologies (WEBIST’15). 526–532.
DOI:https://doi.org/10.5220/0005489205260532
[173] Maria Panteli, Alessandro Piscopo, Adam Harland, Jonathan Tutcher, and Felix Mercer Moss. 2019. Recommendation
systems for news articles at the BBC. In Proceedings of the 7th International Workshop on News Recommendation and
Analytics (INRA’19), in conjunction with the 13th ACM Conference on Recommender Systems (RecSys’19) (INRA’19),
Vol. 2554. 44–52. Retrieved from https://ceur-ws.org/Vol-2554/paper_07.pdf
[174] Eli Pariser. 2011. The Filter Bubble: What the Internet Is Hiding from You. Penguin UK.
[175] Ali Hakimi Parizi, Mohammad Kazemifard, and Mohsen Asghari. 2016. EmoNews: An emotional news recommender
system. J. Digit. Inf. Manag. 14, 6 (2016), 392–402.
[176] Anish Patankar, Joy Bose, and Harshit Khanna. 2019. A bias aware news recommendation system. In Proceedings of
the IEEE 13th International Conference on Semantic Computing (ICSC’19). IEEE, 232–238. DOI:https://doi.org/10.1109/
icosc.2019.8665610
[177] Lukas Pfahler and Katharina Morik. 2020. Fighting filter bubbles with adversarial training. In Proceedings of the
2nd International Workshop on Fairness, Accountability, Transparency and Ethics in Multimedia (FATE/MM’20). 20–22.
DOI:https://doi.org/10.1145/3422841.3423535
[178] Owen Phelan, Kevin McCarthy, Mike Bennett, and Barry Smyth. 2011. On using the real-time web for news recom-
mendation & discovery. In Proceedings of the 20th International Conference Companion on World Wide Web (WWW’11).
ACM, New York, NY, 103–104. DOI:https://doi.org/10.1145/1963192.1963245
[179] Raymond K. Pon, Alfonso F. Cárdenas, David Buttler, and Terence Critchlow. 2007. Tracking multiple topics for
finding interesting articles. In Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery
and Data Mining (KDD’07). ACM, New York, NY, 560–569. DOI:https://doi.org/10.1145/1281192.1281253
[180] Raymond K. Pon, Alfonso F. Cárdenas, and David J. Buttler. 2008. Online selection of parameters in the Rocchio
algorithm for identifying interesting news articles. In Proceedings of the 10th ACM Workshop on Web Information and
Data Management (WIDM’08). 141–148. DOI:https://doi.org/10.1145/1458502.1458525
[181] Idoia Portilla. 2018. Privacy concerns about information sharing as trade-off for personalized news. El Profesional de
la Información 27, 1 (2018), 19–26. DOI:https://doi.org/10.3145/epi.2018.ene.02

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
23:36 C. Bauer et al.

[182] Shankar Prawesh and Balaji Padmanabhan. 2011. The “top N”’ news recommender: Count distortion and manipula-
tion resistance. In Proceedings of the 5th ACM Conference on Recommender Systems (RecSys’11). ACM, New York, NY,
237–244. DOI:https://doi.org/10.1145/2043932.2043974
[183] Shankar Prawesh and Balaji Padmanabhan. 2012. Analysis of probabilistic news recommender systems. In Proceed-
ings of the 18th Americas Conference on Information Systems 2012 (AMCIS’12), Vol. 1. 183–189.
[184] Shankar Prawesh and Balaji Padmanabhan. 2012. News recommender systems with feedback. In Proceedings of the
International Conference on Information Systems (ICIS’12), Vol. 4. 3397–3412.
[185] Shankar Prawesh and Balaji Padmanabhan. 2014. The “Most Popular News” recommender: Count amplification and
manipulation resistance. Inf. Syst. Res. 25, 3 (2014), 569–589. DOI:https://doi.org/10.1287/isre.2014.0529
[186] Shankar Prawesh and Balaji Padmanabhan. 2015. Multi-objective news recommender systems. In Proceedings of the
25th Annual Workshop on Information Technologies and Systems (WITS’15).
[187] Shankar Prawesh and Balaji Padmanabhan. 2021. A complex systems perspective of news recommender systems:
Guiding emergent outcomes with feedback models. PLoS One 16, 1, Article e0245096 (2021). DOI:https://doi.org/10.
1371/journal.pone.0245096
[188] Tao Qi, Fangzhao Wu, Chuhan Wu, and Yongfeng Huang. 2021. PP-Rec: News recommendation with personalized
user interest and time-aware news popularity. In Proceedings of the 59th Annual Meeting of the Association for Com-
putational Linguistics and the 11th International Joint Conference on Natural Language (ACL-IJCNLP’21). 5457–5467.
DOI:https://doi.org/10.18653/v1/2021.acl-long.424
[189] Tao Qi, Fangzhao Wu, Chuhan Wu, Yongfeng Huang, and Xing Xie. 2020. Privacy-preserving news recommendation
model learning. In Proceedings of the Findings of the Association for Computational Linguistics: EMNLP (EMNLP’20).
Association for Computational Linguistics, 1423–1432. DOI:https://doi.org/10.18653/v1/2020.findings-emnlp.128
[190] Jing Qin and Peng Lu. 2020. Application of news features in news recommendation methods: A survey. In Data
Science. ICPCSEE 2020, Pinle Qin, Hongzhi Wang, Guanglu Sun, and Zeguang Lu (Eds.). Springer Singapore, 113–125.
DOI:https://doi.org/10.1007/978-981-15-7984-4_9
[191] Zhongtai Qin and Mingjun Zhang. 2021. Research on news recommendation algorithm based on user interest and
timeliness modeling. In Proceedings of the 2nd International Conference on Computing and Data Science (CONF-
CDS’21), Vol. PartF168982. 6. DOI:https://doi.org/10.1145/3448734.3450933
[192] Shaina Raza and Chen Ding. 2019. News recommender system considering temporal dynamics and news taxonomy.
In Proceedings of the IEEE International Conference on Big Data (Big Data’19). IEEE, 920–929. DOI:https://doi.org/10.
1109/BigData47090.2019.9005459
[193] Shaina Raza and Chen Ding. 2020. A regularized model to trade-off between accuracy and diversity in a news rec-
ommender system. In Proceedings of the IEEE International Conference on Big Data (Big Data’20). IEEE, 551–560.
DOI:https://doi.org/10.1109/BigData50022.2020.9378340
[194] Shaina Raza and Chen Ding. 2021. Deep neural network to tradeoff between accuracy and diversity in a news rec-
ommender system. In Proceedings of the IEEE International Conference on Big Data (Big Data’21). IEEE, 5246–5256.
DOI:https://doi.org/10.1109/BigData52589.2021.9671467
[195] Shaina Raza and Chen Ding. 2022. News recommender system: A review of recent progress, challenges, and oppor-
tunities. Artif. Intell. Rev. 55, 1 (2022), 749–800. DOI:https://doi.org/10.1007/s10462-021-10043-x
[196] Paul Resnick and Hal R. Varian. 1997. Recommender systems. Commun. ACM 40, 3 (1997), 56–58. DOI:https://doi.
org/10.1145/245108.245121
[197] Myrthe Reuver and Nicolas Mattis. 2021. Implementing evaluation metrics based on theories of democracy in news
comment recommendation (Hackathon Report). In Proceedings of the EACL Hackashop on News Media Content Anal-
ysis and Automated Report Generation (EACL’21). 134–139.
[198] Francesco Ricci, Lior Rokach, and Bracha Shapira. 2022. Recommender systems: Techniques, applications, and chal-
lenges. In Recommender Systems Handbook (3rd ed.), Francesco Ricci, Lior Rokach, and Bracha Shapira (Eds.). Springer
US, New York, NY, 1–35. DOI:https://doi.org/10.1007/978-1-0716-2197-4_1
[199] Khumukcham Robindro, Kshetrimayum Nilakanta, Deepen Naorem, and Ningthoujam Gourakishwar Singh. 2017.
An unsupervised content based news personalization using geolocation information. In Proceedings of the Interna-
tional Conference on Computing, Communication and Automation (ICCCA’17). 128–132. DOI:https://doi.org/10.1109/
ccaa.2017.8229785
[200] Uta Russmann and Andreas Hess. 2020. News consumption and trust in online and social media: An in-depth qual-
itative study of young adults in Austria. Int. J. Commun. 14 (2020). Retrieved from https://ijoc.org/index.php/ijoc/
article/view/13774
[201] G. Sudha Sadhasivam, K. G. Saranya, and E. M. Praveen. 2015. Personalisation of news recommendation using genetic
algorithm. In Proceedings of the 3rd International Conference on Eco-friendly Computing and Communication Systems
(ICECCS’14). 23–28. DOI:https://doi.org/10.1109/Eco-friendly.2014.80

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
Where are the Values? A Systematic Literature Review on News Recommender Systems 23:37

[202] Fernando Martin Sagui, Ana Gabriela Maguitman, Carlos Iván Chesñevar, and Guillermo Ricardo Simari. 2008.
Modeling news trust: A defeasible logic programming approach. Inteligenc. Artif. 12, 40 (2008), 63–72. DOI:https:
//doi.org/10.4114/ia.v12i40.977
[203] Alan Said, Jimmy Lin, Alejandro Bellogín, and Arjen de Vries. 2013. A month in the life of a production news rec-
ommender system. In Proceedings of the Workshop on Living Labs for Information Retrieval Evaluation (LivingLab’13).
ACM, New York, NY, 7–10. DOI:https://doi.org/10.1145/2513150.2513159
[204] K. G. Saranya and G. Sudha Sadasivam. 2017. Personalized news article recommendation with novelty using collabo-
rative filtering based rough set theory. Mob. Netw. Applic. 22, 4 (2017), 719–729. DOI:https://doi.org/10.1007/s11036-
017-0842-9
[205] Elvis Saravia, Adam Liu, and Yi-Shin Chen. 2017. Clustering social news based on user affection. In Proceedings of
the Conference on Technologies and Applications of Artificial Intelligence (TAAI’17). 4. DOI:https://doi.org/10.1109/taai.
2017.37
[206] Nick Seaver. 2022. Computing Taste: Algorithms and the Makers of Music Recommendation. University of Chicago
Press. DOI:https://doi.org/10.7208/chicago/9780226822969
[207] Alexander Semenov, Maciej Rysz, Gaurav Pandey, and Guanglin Xu. 2022. Diversity in news recommendations using
contextual bandits. Expert Syst. Applica. 195 (2022). DOI:https://doi.org/10.1016/j.eswa.2021.116478
[208] Mete Sertkan, Julia Neidhardt, and Hannes Werthner. 2019. Documents, topics, and authors: Text mining of online
news. In Proceedings of the IEEE 21st Conference on Business Informatics (CBI’19), Vol. 1. IEEE, 405–413. DOI:https:
//doi.org/10.1109/cbi.2019.00053
[209] Liu Shan, Dong Yao, and Chai Jianping. 2016. Research of personalized news recommendation system based on
hybrid collaborative filtering algorithm. In Proceedings of the 2nd IEEE International Conference on Computer and
Communications (ICCC’16). IEEE, 865–869. DOI:https://doi.org/10.1109/CompComm.2016.7924826
[210] Donghee Shin. 2020. How do users interact with algorithm recommender systems? The interaction of users, algo-
rithms, and performance. Comput. Hum. Behav. 109 (2020). DOI:https://doi.org/10.1016/j.chb.2020.106344
[211] Donghee Shin. 2021. Why does explainability matter in news analytic systems? Proposing explainable analytic jour-
nalism. Journal. Stud. 22, 8 (2021), 1047–1065. DOI:https://doi.org/10.1080/1461670x.2021.1916984
[212] Jyoti Shokeen and Chhavi Rana. 2020. Social recommender systems: Techniques, domains, metrics, datasets and
future scope. J. Intell. Inf. Syst. 54, 3 (2020), 633–667. DOI:https://doi.org/10.1007/s10844-019-00578-5
[213] Liudmila Sivetc and Mariëlle Wijermars. 2021. The vulnerabilities of trusted notifier-models in Russia: The case of
netoscope. Media Commun. 9, 4 (2021), 27–38. DOI:https://doi.org/10.17645/mac.v9i4.4237
[214] Annelien Smets, Jonathan Hendrickx, and Pieter Ballon. 2022. We’re in this together: A multi-stakeholder approach
for news recommenders. Digit. J. 10, 10 (2022), 1813–1831. DOI:https://doi.org/10.1080/21670811.2021.2024079
[215] Annelien Smets, Lien Michiels, Toine Bogers, and Lennart Björneborn. 2022. Serendipity in recommender systems
beyond the algorithm: A feature repository and experimental design. In Proceedings of the 9th Joint Workshop on
Interfaces and Human Decision Making for Recommender Systems, co-located with the 16th ACM Conference on Recom-
mender Systems (RecSys’22). 44–66. Retrieved from https://ceur-ws.org/Vol-3222/paper4.pdf
[216] Alain D. Starke, Sebastian Øverhaug, and Christoph Trattner. 2021. Predicting feature-based similarity in the news
domain using human judgments. In Proceedings of the 9th International Workshop on News Recommendation and
Analytics (INRA’21).
[217] Michael A. Stefanone, Matthew Vollmer, and Jessica M. Covert. 2019. In news we trust? Examining credibility and
sharing behaviors of fake news. In Proceedings of the 10th International Conference on Social Media and Society (SM-
Society’19). ACM, New York, NY, 136–147. DOI:https://doi.org/10.1145/3328529.3328554
[218] Jonathan Stray. 2023. Editorial values for news recommenders: Translating principles to engineering. In News Quality
in the Digital Age, Regina G. Lawrence and Philip M. Napoli (Eds.). Routledge, New York, N, 151–165. DOI:https:
//doi.org/10.4324/9781003257998-13
[219] Olga Streibel and Rehab Alnemr. 2011. Trend-based and reputation-versed personalized news network. In Proceedings
of the 3rd International Workshop on Search and Mining User-generated Contents (CIKM’11). 3–10. DOI:https://doi.org/
10.1145/2065023.2065027
[220] Xiaomeng Su, Özlem Özgöbek, Jon Atle Gulla, Jon Espen Ingvaldsen, and Arne Dag Fidjestøl. 2016. Interactive mobile
news recommender system: A preliminary study of usability factors. In Proceedings of the 11th International Workshop
on Semantic and Social Media Adaptation and Personalization (SMAP’16). 71–76. DOI:https://doi.org/10.1109/smap.
2016.7753387
[221] Emily Sullivan, Dimitrios Bountouridis, Jaron Harambam, Shabnam Najafian, Felicia Loecherbach, Mykola
Makhortykh, Domokos Kelen, Daricia Wilkinson, David Graus, and Nava Tintarev. 2019. Reading news with a pur-
pose: Explaining user profiles for self-actualization. In Adjunct Publication Proceedings of the 27th Conference on
User Modeling, Adaptation and Personalization (ACM UMAP’19 Adjunct). ACM, New York, NY, 241–245. DOI:https:
//doi.org/10.1145/3314183.3323456

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
23:38 C. Bauer et al.

[222] Yumin Sun, Fangzhou Yi, Cheng Zeng, Bing Li, Peng He, Jinxia Qiao, and Yinghui Zhou. 2021. A hybrid approach to
news recommendation based on knowledge graph and long short-term user preferences. In Proceedings of the IEEE
International Conference on Services Computing (SCC’21). IEEE, 165–173. DOI:https://doi.org/10.1109/scc53864.2021.
00029
[223] Cass R. Sunstein. 2001. Echo Chambers: Bush v. Gore, Impeachment, and Beyond. Princeton University Press, Princeton,
NJ.
[224] Paniddaporn Suppasert, Ravikarn Pungprasert, Kamonchanok Putkhaw, and Suppawong Tuarob. 2017. Newsaday: A
personalized thai news recommendation system. In Proceedings of the 6th ICT International Student Project Conference
(ICT-ISPC’17). 4. DOI:https://doi.org/10.1109/ict-ispc.2017.8075321
[225] Tiffany Ya Tang and Pinata Winoto. 2016. I should not recommend it to you even if you will like it: The ethics of
recommender systems. New Rev. Hypermed. Multim. 22, 1–2 (2016), 111–138. DOI:https://doi.org/10.1080/13614568.
2015.1052099
[226] Servet Tasci and Ilyas Cicekli. 2014. A media tracking and news recommendation system. In Proceedings of the Inter-
national Conference on Knowledge Discovery and Information Retrieval (KDIR’14). 53–60. DOI:https://doi.org/10.5220/
0005072000530060
[227] Mozhgan Tavakolifard, Jon Atle Gulla, Kevin C. Almeroth, Jon Espen Ingvaldesn, Gaute Nygreen, and Erik Berg.
2013. Tailored news in the palm of your hand: A multi-perspective transparent approach to news recommendation.
In Proceedings of the 22nd International Conference on World Wide Web (WWW’13). ACM, New York, NY, 305–308.
DOI:https://doi.org/10.1145/2487788.2487930
[228] Nava Tintarev, Emily Sullivan, Dror Guldin, Sihang Qiu, and Daan Odjik. 2018. Same, same, but different: Algorith-
mic diversification of viewpoints in news. In Proceedings of the 26th Conference on User Modeling, Adaptation and
Personalization (UMAP’18). 7–13. DOI:https://doi.org/10.1145/3213586.3226203
[229] Raül Tormos, Christin-Melanie Vauclair, and Henrik Dobewall. 2017. Does contextual change affect basic human
values? A dynamic comparative multilevel analysis across 32 European countries. J. Cross-Cult. Psychol. 48, 4 (2017),
490–510. DOI:https://doi.org/10.1177/0022022117692675
[230] David Tranfield, David Denyer, and Palminder Smart. 2003. Towards a methodology for developing evidence-
informed management knowledge by means of systematic review. Brit. J. Manag. 14, 3 (2003), 207–222. DOI:https:
//doi.org/10.1111/1467-8551.00375
[231] Jason Turcotte, Chance York, Jacob Irving, Rosanne M. Scholl, and Raymond J. Pingree. 2015. News recommendations
from social media opinion leaders: Effects on media trust and information seeking. J. Comput.-Mediat. Commun. 20,
5 (2015), 520–535. DOI:https://doi.org/10.1111/jcc4.12127
[232] Nikki Usher. 2017. Venture-backed news startups and the field of journalism. Digit. J. 5, 9 (2017), 1116–1133.
DOI:https://doi.org/10.1080/21670811.2016.1272064
[233] Max Z. van Drunen, Natali Helberger, and Mariella Bastian. 2019. Know your algorithm: What media organizations
need to explain to their users about news personalization. Int. Data Privac. Law 9, 4 (2019), 220–235. DOI:https:
//doi.org/10.1093/idpl/ipz011
[234] Karin F. van Es. 2017. An impending crisis of imagination: Data-driven personalization in public service broadcasters.
Media@ LSE Work. Paper Series 43 (2017), 18.
[235] Arnout Verheij, Allard Kleijn, Flavius Frasincar, and Frederik Hogenboom. 2012. A comparison study for novelty
control mechanisms applied to web news stories. In Proceedings of the IEEE/WIC/ACM International Conferences on
Web Intelligence and Intelligent Agent Technology (WI’12). 431–436. DOI:https://doi.org/10.1109/wi-iat.2012.128
[236] Paula Viana and Márcio Soares. 2017. A hybrid approach for personalized news recommendation in a mobil-
ity scenario using long-short user interest. Int. J. Artif. Intell. Tools 26, 02 (2017). DOI:https://doi.org/10.1142/
s0218213017600120
[237] Sanne Vrijenhoek, Mesut Kaya, Nadia Metoui, Judith Möller, Daan Odijk, and Natali Helberger. 2021. Recommenders
with a mission: Assessing diversity in news recommendations. In Proceedings of the Conference on Human Information
Interaction and Retrieval (CHIIR’21). 173–183. DOI:https://doi.org/10.1145/3406522.3446019
[238] Shinnosuke Wanaka and Kota Tsubouchi. 2016. Location history knows what you like: Estimation of user preference
from daily location movement. In Proceedings of the 2nd EAI International Conference on IoT in Urban Space (Urb-
IoT’16). ACM, New York, NY, 8–13. DOI:https://doi.org/10.1145/2962735.2962753
[239] Jingkun Wang, Yipu Chen, Zichun Wang, and Wen Zhao. 2021. Popularity-enhanced news recommendation with
multi-view interest representation. In Proceedings of the 30th ACM International Conference on Information & Knowl-
edge Management (CIKM’21). ACM, New York, NY, 1949–1958. DOI:https://doi.org/10.1145/3459637.3482462
[240] Jia Wang, Qing Li, and Yuanzhu Peter Chen. 2010. User comments for news recommendation in social media. In
Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval
(SIGIR’10). ACM, New York, NY, 881–882. DOI:https://doi.org/10.1145/1835449.1835663

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
Where are the Values? A Systematic Literature Review on News Recommender Systems 23:39

[241] Jun Wang, Jinghua Tan, Jiujiu Chen, and Hanlei Jin. 2021. A knowledge-aware and time-sensitive financial news
recommendation system based on firm relation derivation. In Proceedings of the International Conference on Data
Mining Workshops (ICDMW’21). 1104–1111. DOI:https://doi.org/10.1109/icdmw53433.2021.00146
[242] Shaoqing Wang, Benyou Zou, Cuiping Li, Kankan Zhao, Qiang Liu, and Hong Chen. 2015. CROWN: A Context-aware
RecOmmender for Web News. In Proceedings of the IEEE 31st International Conference on Data Engineering (ICDE’15).
IEEE, 1420–1423. DOI:https://doi.org/10.1109/icde.2015.7113391
[243] Charinya Wangwatcharakul and Sartra Wongthanavasu. 2018. Improving dynamic recommender system based on
item clustering for preference drifts. In Proceedings of the 15th International Joint Conference on Computer Science and
Software Engineering (JCSSE’18). IEEE, 6. DOI:https://doi.org/10.1109/JCSSE.2018.8457395
[244] Jonas Wanner, Lukas-Valentin Herm, Kai Heinrich, and Christian Janiesch. 2022. The effect of transparency and
trust on intelligent system acceptance: Evidence from a user-based study. Electron. Mark. 32, 4 (2022), 2079–2102.
DOI:https://doi.org/10.1007/s12525-022-00593-5
[245] Herman Wasserman and Dani Madrid-Morales. 2019. An exploratory study of “fake news” and media trust in
Kenya, Nigeria and South Africa. Afric. Journal. Stud. 40, 1 (2019), 107–123. DOI:https://doi.org/10.1080/23743670.
2019.1627230
[246] Sebastian Werner and Andreas Lommatzsch. 2014. Optimizing and evaluating stream-based news recommendation
algorithms. In CLEF2014 Working Notes (CLEF 2014), Vol. 1180. 813–824.
[247] Mareike Wieland, Gerret von Nordheim, and Katharina Kleinen-von Königslöw. 2021. One recommender fits all? An
exploration of user satisfaction with text-based news recommender systems. Media Commun. 9, 4 (2021), 208–221.
DOI:https://doi.org/10.17645/mac.v9i4.4241
[248] Robin M. Williams Jr. 1979. Change and stability in values and value systems: A sociological perspective. In Under-
standing Human Values: Individual and Societal, Milton Rokeach (Ed.). Free Press, New York, NY, 15–46.
[249] Langdon Winner. 2017. Do artifacts have politics? In Computer Ethics. Routledge, 177–192. DOI:https://doi.org/10.
4324/9781315259697-21
[250] Chirayu Wongchokprasitti and Peter Brusilovsky. 2007. NewsMe: A case study for adaptive news systems with open
user model. In Proceedings of the 3rd International Conference on Autonomic and Autonomous Systems (ICAS’07). 69–74.
DOI:https://doi.org/10.1109/conielecomp.2007.88
[251] Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. 2020. User modeling with click preference and reading sat-
isfaction for news recommendation. In Proceedings of the 29th International Joint Conference on Artificial Intelligence
(IJCAI’20). 3023–3029. DOI:https://doi.org/10.5555/3491440.3491858
[252] Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng
Gao, Winnie Wu, and Ming Zhou. 2020. MIND: A large-scale dataset for news recommendation. In Proceedings of
the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics,
3597–3606. DOI:https://doi.org/10.18653/v1/2020.acl-main.331
[253] Yang Wu, Tang Rui, and Lu Ling. 2016. A fused method for news recommendation. In Proceedings of the Interna-
tional Conference on Big Data and Smart Computing (BigComp’16). 341–344. DOI:https://doi.org/10.1109/bigcomp.
2016.7425943
[254] Yingyuan Xiao, Pengqiang Ai, Ching-hsien Hsu, Hongya Wang, and Xu Jiao. 2015. Time-ordered collaborative filter-
ing for news recommendation. China Commun. 12, 12 (2015), 53–62. DOI:https://doi.org/10.1109/cc.2015.7385528
[255] Yanan Xie, Liang Chen, Kunyang Jia, Lichuan Ji, and Jian Wu. 2013. iNewsBox: Modeling and exploiting implicit
feedback for building personalized news radio. In Proceedings of the 22nd ACM International Conference on Conference
on Information & Knowledge Management (CIKM’13). ACM, New York, NY, 2485–2488. DOI:https://doi.org/10.1145/
2505515.2508199
[256] Yuan Xue, Chen Zhang, Changzheng Zhou, Xun Lin, and Qing Li. 2008. An effective news recommendation in
social media based on users’ preference. In Proceedings of the International Workshop on Education Technology and
Training and International Workshop on Geoscience and Remote Sensing (ETT and GRS’08), Vol. 1. 627–631. DOI:https:
//doi.org/10.1109/ETTandGRS.2008.298
[257] Wenwen Ye, Shuaiqiang Wang, Xu Chen, Xuepeng Wang, Zheng Qin, and Dawei Yin. 2020. Time matters: Sequential
recommendation with complex temporal information. In Proceedings of the 43rd International ACM SIGIR Conference
on Research and Development in Information Retrieval (SIGIR’20). ACM, New York, NY, 1459–1468. DOI:https://doi.
org/10.1145/3397271.3401154
[258] Kam Fung Yeung and Yanyan Yang. 2010. A proactive personalized mobile news recommendation system. In Pro-
ceedings of the 3rd International Conference on Developments in eSystems Engineering (DeSE’10). 207–212. DOI:https:
//doi.org/10.1109/DeSE.2010.40
[259] Kam Fung Yeung, Yanyan Yang, and David Ndzi. 2010. Context-aware news recommender in mobile hybrid P2P
network. In Proceedings of the 2nd International Conference on Computational Intelligence, Communication Systems
and Networks (CICSyN’10). 54–59. DOI:https://doi.org/10.1109/CICSyN.2010.48

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.
23:40 C. Bauer et al.

[260] Hee Geun Yoon, Hyun Je Song, Seong-Bae Park, and Kweon Yang Kim. 2015. A personalized news recommendation
using user location and news contents. Appl. Math. Inf. Sci. 9, 2 (2015), 439–449. DOI:https://doi.org/10.12785/amis/
092L19
[261] Dušan Zeleník and Mária Bieliková. 2011. News recommending based on text similarity and user behaviour. In
Proceedings of the 7th International Conference on Web Information Systems and Technologies (WEBIST’11). 302–307.
DOI:https://doi.org/10.5220/0003339403020307
[262] Xichen Zhang and Ali A. Ghorbani. 2020. An overview of online fake news: Characterization, detection, and discus-
sion. Inf. Process. Manag. 57, 2, Article 102025 (2020), 26 pages. DOI:https://doi.org/10.1016/j.ipm.2019.03.004
[263] Yongfeng Zhang and Xu Chen. 2020. Explainable recommendation: A survey and new perspectives. Found. Trends
Inf. Retr. 14, 1 (2020), 101. DOI:https://doi.org/10.1561/1500000066
[264] Yunwei Zhao, Can Wang, Han Han, Min Shu, and Wenlei Wang. 2020. An impact evaluation framework of per-
sonalized news aggregation and recommendation systems. In Proceedings of the IEEE/WIC/ACM International Joint
Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT’20). 893–900. DOI:https://doi.org/10.1109/
wiiat50758.2020.00136
[265] Chen Zhu, Hengshu Zhu, Yong Ge, Enhong Chen, and Qi Liu. 2014. Tracking the evolution of social emotions: A
time-aware topic modeling perspective. In Proceedings of the IEEE International Conference on Data Mining (ICDM’14).
IEEE, 697–706. DOI:https://doi.org/10.1109/icdm.2014.121
[266] Zhiliang Zhu, Deyang Li, Jie Liang, Guoqi Liu, and Hai Yu. 2018. A dynamic personalized news recommendation
system based on BAP user profiling method. IEEE Access 6 (2018), 41068–41078. DOI:https://doi.org/10.1109/access.
2018.2858564
[267] Cai-Nicolas Ziegler, Sean M. McNee, Joseph A. Konstan, and Georg Lausen. 2005. Improving recommendation lists
through topic diversification. In Proceedings of the 14th International Conference on World Wide Web (WWW’05).
22–32. DOI:https://doi.org/10.1145/1060745.1060754

Received 22 August 2023; revised 1 February 2024; accepted 6 March 2024

ACM Trans. Recomm. Syst., Vol. 2, No. 3, Article 23. Publication date: June 2024.