Notifications/Feature requirements: Difference between revisions

Content deleted Content added

Inline

Revision as of 23:55, 11 February 2013

This document is a work in progress. Comments are appreciated but this is not a final draft.

This page describes feature requirements for a new notifications system for MediaWiki, code-named Echo. Features below are for the first release of Echo in early 2013.

To learn more about Echo, check out this project hub, these testing tips, the user experience page and other related documents.

Job Opening: We're looking for a new software engineer to join our team to develop Echo and other editor engagement projects. Check out the job description. If you think this would be a good fit for you, contact us. It's a great team, a wonderful mission and a really fun job. Even if this is not right for you, help us spread the word!

Overview

Echo is designed to replace and augment existing notification systems on MediaWiki sites, as well as provide significantly more control to both users and developers as to how their notifications are handled, read, and deleted. This new notifications system seeks to unify the delivery of interaction messages in MediaWiki core, through a common API that can provide a uniform interface for users, as well as a scaleable, high-performance platform for developers. For a quick visual overview of this project, check the Echo slides.

Problems & Solutions

We aim to solve these core problems:

There is no central notification system on Media Wiki sites
The current ad-hoc approach is inefficient
Users are not notified of key events
Users are confused by current notices

Echo will be developed to provide these solutions:

Provide a unified user experience
Help developers add it to their code
Promote editor engagement

Guidelines

Here are some general guidelines for providing the best experience for our users.

Focus on quality

Every notification should be useful to our users. Quality is key to prevent users from turning off notifications.

Only important events

Notification should only feature the most relevant events, from the user's perspective -- highlighting just the key items from the watchlist, talkpage and other feeds.

Newsworthy stories

Notifications should be informative, like a good news story, and report objectively about who did what, when and where.

Keep them short and sweet

Notifications should be concise, with the basic message under 10 words (up to 20 words if there are multiple user names, long article names or text snippets - which should be truncated after x characters)

Actionable

Make notifications actionable, but don't overload the user with too many actions. Less is more.

Daily limits?

Don't send more than X notifications per day, if possible, so people don't view this as spam. (under discussion)

Active users only?

Only send notifications to relatively active users. If a user stops visiting the site or clicking on the notifications, we should reduce the flow right away, so it is not perceived as spam. (under discussion)

Measure the impact

Track how often users click on each type of notifications. If we find that clickthrough falls below a certain threshold (e.g. 15%) for certain types of notifications, we may want to consider removing them from the feed -- or not make them default options in preferences.

These guidelines are still preliminary and subject to revision. Once we refine them some more, they may be useful when communicating with developers to use our Echo API to add notifications for their own extensions.

User groups

Workflow: Notifications for a new user's first steps on Wikipedia

For the first release, we will support three types of registered users:

new editor (our primary target)
active editor
very active editor/admin

However, we will focus on new editors (over experienced editors) for this first release, because new users need notifications more than power users. Specifically, we will concentrate on some of the first notifications which a new user will receive after creating an account on Wikipedia, as shown in the workflow to the right.

This doesn't mean we will not support advanced editors, but we will initially emphasize notifications that help engage new users, who need this service most urgently.

Anonymous, unregistered users will not be targeted for this release.

Key features

Key features for Echo's first release include:

user menu
flyout
all-notifications page
email notifications
preferences page

For an overview of how these features work together, check the user touchpoints map (see thumbnail to the right).

In phase 1 of this project (Oct. 2012-Jan. 2013), we will create a simple version of these features, focusing on a first desktop version for both web and email delivery, as described below.

Other features under consideration for later development may be added in separate sections below, including a mobile version, which will not be developed until the second release in spring 2013 (after WMF's mobile platform starts supporting log-in features).

User Menu

The first visible touchpoint for users is a 'Notifications' badge in the top menu that appears after the user's name in the upper right corner of any web page (referred to below as the 'user menu' or 'growler').

Notifications badge
For the first release, we now plan to simply show a 'badge' (also called a 'growler') without a label next to the user name.

Notifications badge If the user has notifications they have not already seen, the badge will be red and will show the number of such 'unseen' notifications. This is likely be a red rectangle with a number ranging from 1 to 999, to match best practices. This badge will auto-update as outlined below, so that if a new notification comes in, the number will be changed as soon as possible. To be clear, the number in the red badge refers to new, unseen notifications that occurred since the last time you looked at the flyout.

Notification flyout Clicking on the "Notifications badge' will display the Notifications Flyout and zero out the badge, turning it gray. Clicking on it again (or anywhere outside of the notifications flyout), will close that flyout, and mark all notifications as read. The badge will then be grayed out, not red and will no longer have a number, until new unseen notifications are added. However, you may still click on that gray badge, which will display the flyout again, with its most recent notifications.

No notifications
The gray badge with the number zero would be shown next to the user name, even if there are no new notifications (or no notifications at all), so that the user can still click on it to easily access the flyout, the all-notifications archive or preferences. If the user has not received any notifications at all, the flyout and all-notifications pages would display this line of text where the notifications would have appeared: "You have no notifications at this time."

The flyout will feature a short list of new notifications, when the user clicks on the badge in the user menu. See mockup to the right.

Purpose
The purpose of the flyout is to:

make users aware of new activity that relates to them
give them a short list of the most recent notifications
enable them to take action on any event in that list
view more notifications on the all-notifications page

Number
We currently plan to list up to 10 notifications in the flyout, but would only show about 6 notifications at a time, and the user would need to scroll to see the rest. If the user wants to see more, they can click on 'See all' to go to the 'all-notifications' page. It is likely that these numbers will vary between devices, based on the height of your display -- and will be reduced for smaller screens, since scrolling in a flyout is not the best user experience. We will tweak these numbers once we have a first working version of the flyout we can test on various devices.

Order
Notifications will be shown in reverse chronological order, with the latest notifications shown first, based on their timestamp. The most recent notifications will remain on the flyout until replaced by newer notifications, which will push them down the stack incrementally, until they fall off from the flyout list. (The same ordering principle will be used in the all-notifications page, except that the list will be longer and separated by days: today, yesterday, all the way back to the past 7 days).

Scrollbar
We will provide a scrollbar in the flyout for devices that can support them. This will be needed in order to see up to 10 items in the flyout, which will not all fit at once in the first view of the flyout, particularly if they all have payloads.

Highlights
When you first open the flyout, its 'new' notifications will be highlighted, so you can tell at a glance which are new. 'New' means a notification that you haven't seen yet, and that occurred recently, since the last time you opened flyout. To be clear, there should be as many highlighted notifications as the number in the red badge. If you click on a highlighted notification, it becomes 'read' and loses its highlight.

If needed, we may also want to provide a different, grayed-out highlight to indicate items that you have already clicked on previously. Its opacity could be reduced, making it light gray, so you can tell it apart from the ones you have not yet clicked on. However, this is a lower priority than the primary highlight for 'new' items described above.

Types
The flyout would include different thypes of notifications, including these sample types now in development:

Edit reversions (e.g.: 'Vibha undid your edit to Breakfast')
Reviews ('Ryan reviewed a page you started: Breakfast.')
Mentions ('Fabrice mentioned you on this page: Dog')

Sample notifications we are working on for the first release are listed below.

Grammar
The proposed grammar for each flyout notification follows this general format:

Fabrice Florin posted on your <talk page>.
"Thanks for your great work!"  1 hour ago

This would be based on a common 'subject-verb-object' structure, with a few important attributes, as outlined in this pseudo code;

'<actor-name> <action> <object>: ["<payload-details>"] <timestamp>'

Ideally, notifications would help answer these key questions about each event: who? what? where? when? why? how? -- if available in a concise format -- to give the user a better sense of what this event is about. Note that many events will not cover all these answers.

Depending on the type of notification, we may use passive, rather than active voice. For example, we would use the active voice with the subject first for talk page messages or mentions that are related to person-to-person interactions. But for notifications where the object is more important than the actor, we may use passive voice, where the object is named first, then the actor, as shown in this example:

"<Breakfast>" was reviewed by Fabrice Florin. 
Tags: Copy edit  1 hour ago

Examples
Here are a few more examples of notifications:

Reviewed and marked for deletion:

[icon] "<Breakfast>" was reviewed by Kaldari and marked for deletion.
Tags: Unambiguous copyright infringement   1 day ago

Edit undone:

[icon] Your edit to "Breakfast" was <reverted> by Kaldari. <View Changes> 
"This is already covered in the 'Nutrition' section."   1 minute ago'

See more examples in the #Sample_Notifications section below.

Links
Each notification will typically contain a single link to the most important action the user can take to respond to the event they are being notified about.

This primary action could link to any of these possible pages:

your talk page
an article page
a difference page
a discussion page

The primary action for each notification type is now described in this prioritized notifications list and will be posted in the #Sample_Notifications section below.

For information clarity, we will bold the subject, action and object (as shown above), even if they do not have a link attached to them.

Note that the 'all-preferences' page will display more links than the flyout, to provide users with a wider range of actions.

All links from the flyout should open in the same window, not a new window.

Other links
The flyout would include a 'See all' link to the 'All Notifications' page.

If space allows, we may also provide one more link to the 'Notifications preferences' page.

We will not display any 'Clear New' button at this time.

Payload
When important messages or event details are available in a concise format, we will aim to include short text snippets, to give the user a sense of what they might get if they click on it. But these snippets would be short, and truncated after 140? characters, with three punctuation marks ('...') to indicate there's more.

Timestamp
The timestamp will be expressed as a 'time elapsed' indicator (e.g.: 1 min. ago, 2 hours ago, 3 days ago). It may only appear when the user hovers over the notification, to reduce clutter.

No notifications message
If the user has not received any notifications at all, the flyout would display this line of text where the notifications would have appeared: "You have no notifications at this time."

Privacy
For the record, all personal Echo notifications will be completely private, both in the flyout and all-notifications pages, like all other websites. To clarify, even if a public notification is sent to the user in future releases (e.g. 'a new edition of the Signpost has been published'), the content of that notification will obviously be public (but the fact that you received that notification will remain private).

All notifications

The 'All notifications' page will feature a longer list of notifications, when the user clicks on 'See all' in the flyout. See mockup to the right. (Note: this page is sometimes called 'Archive' in other documents about this project; however, that label is for internal purposes only, and will not be shown to end-users.)

Purpose
The purpose of the 'All notifications' page is to give users a full list of all notifications received in the past week, in order to:

make them aware of recent events that relate to them
find previous events they might have missed or ignored
enable them to take a wider range of actions on any event in that list
show them how they can change their notification preferences

Sample All-Notifications Page Contents
Here is an example of what the All-Notifications page could include (rough draft):

All notifications                                 <Preferences>
 
TODAY
 
[icon] "<Breakfast>" was reviewed by <Kaldari>. 
TagS: Copy edits, No references, Stub.      1 min. ago

[icon] <VBamba> posted on <your talk page>
"Hey Fabrice, nice work on the Breakfast article!"   1 hour ago


MARCH 28
 
[icon] <Breakfast> was reviewed and marked for deletion by <Okeyes (WMF)>. 
Tags: Unambiguous copyright infringement.   1 day ago

[icon] Your edit to <Breakfast> was reverted by <Kaldari>. <View Changes>  
"This is already covered in the 'Nutrition' section."    1 day ago


MARCH 27
 
[icon] <Bsitu)> posted on your <talk page>.
"Hi Fabrice, I think you are making progress with the Breakfast article, but I think that you should
add a section on Nutrition." ... <More>  2 days ago

etc.

Numbers
We currently plan to list up notifications for the past 7 days in the 'All notifications' page.

That number will vary from one user to another. For a new user, the total number of notifications may end up being only 10 notifications for that week, but for a more experienced user, it could include over 100 for that same week.

There is no need to include the total number of notifications on this page at this time.

By default, this page would only show about 25 notifications at a time, and the user would need to scroll to the bottom to see the rest (using infinite scrolling or 'More' button). (We will tweak this number once we have a first working version of the the 'All notifications' page that we can test on various devices.)

Order
Notifications will be shown in reverse chronological order, with the latest notifications shown first, based on their timestamp. The newest notifications will push back the older ones further down the list.

Sections
The 'All notifications' page will be separated into different sections, by days: Today, Yesterday, then actual dates ('Nov. 12'), all the way back to the past 7 days.

Length
Notifications will be relatively short in this 'All notifications' page, given that there could be more to list at once. So we will aim to keep them to 1 to 3 lines if possible, so it's easier to scroll down the list and see many of them at a glance.

Highlights
It doesn't seem necessary to show highlights to separate 'new' or 'unread' -- unlike on the flyout, where it seems more useful.

Icons
The 'All notifications' page would show small icons to identify the different types of notifications (e.g.: edit reversions, page reviews, talk page messages or mentions).

Grammar
The proposed grammar for each notification would be consistent with what we are using in the flyout and email copy, though slight variations in language may be needed for each format.

Links
While we are planning on offering a single link on the flyout, the 'All notifications' page will provide separate links for the subject and object (and sometimes action), to give more flexibility to the end user on that page (as Facebook and Google do already on their the 'All notifications' pages.)

Here are some examples of how these links might appear on the all-notifications page:

[icon] <Kaldari> posted on your <talk page>: "Thanks for your great work!" 1 hour ago

[icon] <Breakfast> was reviewed by <Kaldari>. 
Tags: Copy edit  1 hour ago

[icon] <Breakfast> was reviewed and marked for deletion by <Okeyes>.
Tags: Unambiguous copyright infringement.  1 day ago

[icon] Your edit to <Breakfast> was <reverted> by <Kaldari>. <View Changes>  
"This is already covered in the 'Nutrition' section."  1 minute ago'

See more examples in the #Sample_Notifications section below.

Here are examples of where these separate links might go to:

The link for an actor-name <Ryan> would go to their user page.
The link for an article-name <Breakfast> would go to that article.
The link for <View Changes> would go to the diff page showing how your revision was changed.
The link for <More> would go to the talk page (or other page) showing the full message.

See more examples in the #Sample_Notifications section below.

All links from the all-notifications page should open in the same window, not a new window.

Payload
There are two primary types of payload for our first release:

text snippets (for talk page messages or reverted edits)
tags (for page reviews)

Text snippets: When important messages or event details are available in a concise format, we will aim to display short text snippets in the notifications on the 'All notifications' page, to give the user a sense of what they might get if they click on it. But these snippets will be short, and truncated after 255 characters, with a 'More' link to the page that the snippet is being excerpted from. These text snippets will typically be pulled from the 'edit summaries' (which can be handwritten, automatically generated, or a hybrid of the two), though in some cases it may be possible to pull the first line of the actual message (e.g. when a user adds a new section on a talk page). See more examples in the #Sample_Notifications section below. These snippets should be displayed in between quotes, using a special font style (e.g. gray), to distinguish them from the notification narrative. They should typically be smaller than the main notification message, but not so small that they are hard to read by older users.

Tags: In the case of notifications for page reviews, we will display the specific tags that were added to the page during the review. These tags will be comma-separated and kept short, so that we would only show the first 16? characters of a tag, followed by 3 dots (...) if they exceed that limit. In a later stage, we may provide a link to a flyout that would explain what these notifications mean, as proposed in the second mockup to the right. See specific examples in the #Sample_Notifications section below.

Tag Flyouts
In a later stage, we are considering showing a flyout on this page to explain what tags mean, when displayed in notifications, as illustrated in the mockup to the right. We plan to postpone this feature until we have a complete notifications system in place.

Timestamp
The timestamp would be expressed as a 'time elapsed' indicator (e.g.: 1 min. ago, 2 hours ago, 3 days ago).

Link to Preferences
We will provide on the All-notifications page a link to the Notification preferences, so that people can adjust their settings after seeing the list of their current notifications. This link will simply say 'Preferences'. At a later stage, we may want to consider providing a short summary of which preferences they have currently selected in the upper right corner of the page, so it's easier to understand why they are getting some of these notifications.

All-Notifications URL and Visibility
The URL for this All-Notifications page would be very short (e.g.: 'http://en.wikipedia.org/wiki/Special:Notifications'). This page will only visible to logged-in users, and they will only be able to see their own page, not anyone else's. (For debugging purposes, though, we may want to have a temporary way to inspect people's pages, to troubleshoot during development).

No notifications message
If the user has not received any notifications at all, the all-notifications pages would display this line of text where the notifications would have appeared: "You have no notifications at this time."

Best Practices
These feature requirements are inspired in part by best practices used by other top sites or platforms (e.g.: Facebook, Twitter, Google, LinkedIn, Quora), as shown in these slides, and summarized below:

many top sites go back up to a year in their all-notifications page (Facebook only shows 7 days)
most sites support up to hundreds of notifications on this page
notifications range from only 1 line (Facebook) to up to 9 lines (Google)
most sites include text snippets and time stamps for all notifications
most sites include either an icon or a photo for all notifications

Email notifications

Subject line of an individual Notification

Email notifications are alerts that are sent to you when important events take place that involve you or your activity on the site.

Purpose
The purpose of email notifications is to inform you when something happens that relates to you, even when you are not on the site. These emails will enable you to do the following:

learn what just happened (who, what, where, when, why and/or how?).
go to the page where the event took place, so you can check it out for yourself.
take quick action, if you want to.

Message Types
Depending on your preferences, you can receive one of three types of email messages:

individual notifications
daily digest
weekly digest

Note that no messages will be sent if the user checks the 'no email notifications' option in their preferences.

Email Formats
Email notifications will eventually be available in two different formats:

plain text (first release on MediaWiki)
HTML (next release on En-Wiki)

For the MediaWiki release, we will only support plain text emails, and add HTML format options in the En-Wiki release.

We will send both plain text and HTML to all users by default, and if their email client doesn't support HTML, it will show plain text instead. There will not be any email format preferences unless we hear a strong user demand for this, to keep preferences to a minimum.

Individual Email Notifications

Plain text email notifications

Sample Message - Individual Notification
Here is an example of an individual email notification, sent in plain text:

From: Wikipedia <notifications@wikipedia.org>

Reply to: Wikipedia <no-reply-notifications@wikipedia.org>

To: Fabrice Florin (WMF) <fflorin@wikimedia.org>

Subject: You have a new talkpage message

Wikipedia user Shiftchange posted on your talk page:

"I am trying to expand on hydrocarbons and the environment topic so
that users may have more accurate information. Hydrocarbons is a
pretty critical educational topic too ..."

View more:

https://en.wikipedia.org/wiki/Breakfast

________________________________________________

To control which emails we send you, visit:
http://en.wikipedia.org/wiki/Special:Preferences#Notifications
Wikimedia Foundation, 149 New Montgomery St., 3rd Fl., San Francisco, CA 94105.

Message Contents
Most email messages will typically include these contents, as illustrated in the sample above:

Header
- From
- Reply to
- To
- Subject

Body
- Event summary
- Payload (e.g. comment, if any)
- Call to action
- Action link

Footer
- Preferences notice (*)
- Preferences link (*)
- Company address (*)

(*) The items asterisked above are likely to remain constant in most email messages.

Each of these content types are specified one at a time after the sample message below.

Note that in the case of email digests, multiple notifications may be bundled by category in a single message, which may require dividers in between categories.

Header Contents
The email headers will include these fields:

From

The 'From' field will display the name of the site that is sending the notification, with an email address that includes the word 'notifications', as in this example:

From: Wikipedia <notifications@wikipedia.org>

Reply to

The 'Reply to' field will also display the name of the site that is sending the notification, with a different email address for tracking purposes, as in this example:

Reply to: Wikipedia <no-reply-notifications@wikipedia.org>

To

The 'To' field will display the userID of the user that is receiving the notification, with the email address they specified in preferences, as in this example:

To: Fabrice Florin (WMF) <fflorin@wikimedia.org>

Subject

The 'Subject' field will feature a short phrase describing the event, as in this example:

You have a new talk page message

The subject will aim to identify the action and the object of this event, while staying short and sweet.

Body Contents
The body of the email notification will include these fields:

Event summary

The 'Event summary' field will consist of a sentence or two describing what happened, who did it, where and when, as in this example:

Wikipedia user Shiftchange posted on your talk page

or:

Wikipedia user Kaldari reviewed this page: "Breakfast"

Event summaries may vary based on the type of notification. For example, the name of the page will be included when the object is an article, but not if it is a talk page. Some summaries will have more words than others -- and could even include a second phrase, if needed.

Each major group of notifications will share the same phrasing and phrase order, if possible (e.g. all instances of the 'page review' notifications would inherit the same sentence construction).

Payload

The 'Payload' field will display relevant information to help understand how or why the event took place, such as tag names for a page review, as in this example:

Tags: Copy edit, No references, Stub

... or, in the case of a talk page post:

Wikipedia user Kaldari posted this on your talk page: 

"Hey Fabrice, nice work on the Breakfast article! 
I just linked it to the Nutrition article, and added more info on calories ..."

The payload will not display more than a maximum number of characters (e.g.: 255 characters, as in the all-notifications page). If the payload exceeds that limit, the email notification will only display the first few characters up to that maximum, with three dots at the end, to indicate that it was truncated and that there is more to it. Words should not be truncated, if possible. See the 'All-notifications' section above for more recommendations on payload contents.

See more examples in the #Sample_Notifications section below.

Call to action and link

The 'Call to action' and 'action link' fields will will include a short call to action, as well as a separate link shown on a line by itself, as in this example:

View more:

https://en.wikipedia.org/wiki/Breakfast

Calls to action and links may vary based on the type of notification. Ideally, the selected link will help users find out what happened and take quick action, if they can and want to.

There should always be a call to action and link, if possible, to invite users to return to the site and participate more actively. We may want to avoid links that exceed 72 characters, if at all possible, so that line wrapping doesn't break the URL on some email clients.

Footer Contents
The email footer will include these fields, which will generally stay constant across all notifications.

For the first release, the plain text version of this email footer could be separated from the body of the message by a divider. In future releases, when we implement an HTML version of the email notifications, this email footer could be shown in smaller gray text font, as most other top sites do.

Preferences notice and link

The 'Preferences notice' field will display in small text the email address used for this subscription, along with instructions on how to change their preferences. The 'link' field will enable the user to change their email notifications preferences, and be listed on a line by itself, as in this example:

To control the emails we send you, visit:
http://en.wikipedia.org/wiki/Special:Preferences#Notifications

This link will take users directly to the preferences for email notifications and will require them to log in.

In future releases, we may consider updating this link to enable the user to immediately unsubscribe with a single click that doesn't require them to log in to change preferences manually.

There should always be at least a preferences link, which should ideally not exceed 72 characters, so that line wrapping doesn't break the URL on some email clients.

Company address

This short legal notice with Wikimedia's corporate address will be added at the end of the notifications.

Wikimedia Foundation, 149 New Montgomery St., 3rd Fl., San Francisco, CA 94105.

The corporate address is required by law, and implemented by other top sites such as Facebook, Google or LinkedIn.

HTML email notifications

File:HTML EmailUX-26.png

HTML Email UX for Single Notification

This section is in progress and should be completed shortly.

A preliminary mockup is shown to the right, for discussion purposes.

Bundled Email Digests

Email digests are bundled notifications that are sent daily or weekly.

Digest Goals
The goal of email digests is to group notifications into daily or weekly updates, so that:

users can get fewer emails sent to them, by getting regular updates instead.
users can get an activity summary of what is happening on the site, even if they don't have time to go there.
Wikipedia can stay in touch with users and engage them remotely, bringing them back to the site more often.
users can select which events to check and take immediate action on the most important ones (HTML only).

Email Frequency
The frequency of bundled email digests will be controlled by the user, in their preferences. This will enable them to select either 'daily' or 'weekly'. But the exact time and date will be determined by the Echo software.

Ideally, these digests would be sent at about the same time of day (or the same day of the week), if technically feasible, so that the user can expect to get them somewhat regularly. The actual time will be determined after consultation with engineering and operations teams. If at all possible, it would make sense to send the daily emails in the early mornings for the majority of our target users (e.g.: 3am PT for English Wikipedia). And it would also seem helpful to send the weekly digests on a Monday, at the start of the week. More research will be needed to finalize these proposed times and dates, so we can verify that these proposed assumptions are correct.

Contents
The contents of the email digest messages will be roughly the same as the all-notifications archive page. We will not attempt to exclude the notifications that have already been 'read' by the user, at least not for the first release. So it is possible that a user might see some notifications they have seen before, but we think this will be OK, since we are presenting these emails as 'summary of activity'.
Categories
We propose to group bundled notifications in categories, by notification type, as outlined below.

The rationale for this approach is that it will allow us to show the most important types of notification first, for editor engagement purposes. For example, we would list talk page messages first, to encourage users to interact with others. We would also list positive notifications before negative ones (e.g. edit reverts would be last).

Here are the main categories, listed in order of priority:

Talk page message
- post on your talk page

Talk page message (other)
- reply to your post on another talk page
- post on the talk page of an article you started

WikiLove
- special message and barnstar on your talk page

Mention
- mention on another talkpage

Cross-reference
- link to a page you started
- your page was added to a Wikiproject

Recognition
- your page was rated as good
- your page was featured

Page review
- review of a page you started
- review with maintenance tags
- review with deletion tags

Edit revert
- edit undone
- edit rolled back

(only show this if user opts in for this type of negative notification)

Threshold
We will only send up to 10 notifications at a time in a bundled email digest, to prevent the user from being overwhelmed. If the number of notifications exceeds that threshold, the most important notifications would be included in that short list, and the older notifications excluded, if possible. For example, in the case of a daily bundled notification, we would list all the notifications that occurred in the past 24 hours, then sort them by priority (see above order), then show the first 10 notifications from that sorted list, grouped by the categories above.

Plain text email digests

Sample Message - Daily Digest (Bundled Notification)
Here is an example of a daily email digest, which is a bundled email notification (preliminary draft):

From: Wikipedia <notifications@wikipedia.org>

Reply to: Wikipedia <no-reply-notifications@wikipedia.org>

To: Fabrice Florin (WMF) <fflorin@wikimedia.org>

Subject: You have 6 notifications today

Fabrice,

You have 6 notifications on Wikipedia today. View them here: 
http://en.wikipedia.org/wiki/Special:Notifications

3 Talk page messages

 • Shiftchange posted on your talk page
 • Accedie posted on your talk page
 • Kaldari posted on your talk page

________________________________________________

2 Page reviews

• "Breakfast" was reviewed by Kaldari
• "Dehli Metro" was reviewed by Accedie

________________________________________________

1 Edit revert

• Your edit to Nazi Germany was reverted by Kaldari

 ________________________________________________

To control which emails we send you, visit:
http://en.wikipedia.org/wiki/Special:Preferences#Notifications
Wikimedia Foundation, 149 New Montgomery St., 3rd Fl., San Francisco, CA 94105.

HTML email digests

This section is in progress and should be completed shortly.

Other email specifications

Email Cadence
Email frequency can be adjusted to some extent in the user preferences, but is in fact more complex than that, if we want to avoid spamming users. For example, the initial notification about an event should be sent as close to real time as possible, but we may want to batch subsequent notifications if they exceed more than 3? emails in a given day. Sending all notifications immediately could lead the user being overwhelmed and unsubscribing. Sending all notifications batched into x hour bundles may be a reasonable low-effort compromise, but could mean missing an opportunity to drag users back to the site to do more work while things are still fresh. We will need to get the cadence right for a large percentage of users, or risk losing the privilege of communicating with them that way. For the first release, we will implement a simple frequency solution, but will want to tweak its cadence in the second release.

Best Practices
These feature requirements are inspired in part by best practices used by other top sites or platforms (e.g.: Facebook, Twitter, Google, LinkedIn, Quora), as shown in these slides, and summarized below:

most email notifications are HTML, requiring us to support this format as soon as possible

(for practical reasons, we will only implement plain text emails for the first release)

the subject line always includes a brief event description and the name of the site
average of 3 links per email (min. 2, max. 7 in example slides)
3 out of 5 sites provide a daily and/or weekly digest
most sites provide a quick way to unsubscribe

Preferences

The preferences page will feature a short list of options for email and web notifications, as part of the main user preferences.

Purpose
The purpose of the preferences is to give users some control over which notifications to receive by email, how often to receive them and whether or not to display the web notifications flyout in the user menu.

Tab
A special 'Notifications' tab will be displayed within the main user preferences page. Ideally, it would be listed in second or third position after the first tab you see when you open preferences. We may want to have a one-line explanation of what notifications are below the tab: "We send you notifications when users take actions on Wikipedia that involve you. You can change which notifications to receive below. Learn more >'. ('Learn more' will link to the help page or FAQ where we explain how notifications work.)

Links
This notifications preferences tab will be linked from the 'All Notifications' page (and possibly from the 'Notifications' flyout, if space allows).

Sections
The first version of preferences would consist of these sections:

Email Frequency
Email Notifications
Web Settings

Email Frequency

This section will enable users to choose how often they want to receive notifications by email, using radio buttons or similar selection method.

Here are the proposed contents for that section:

Email Frequency

I would like to receive:
• no emails about these events
• individual notifications as they come in
• a daily summary of most important notifications
• a weekly summary of most important notifications

Notifications are being sent to myemail@mysite.com .
<Change your email address>

Email Notifications

This first section will enable users to choose which notifications to receive by email, using checkboxes.

Here are the proposed contents for that section:

Notification Types

Email me when someone:

• posts on my talk page
• mentions me
• send me wiki love
• links to a page I started  
• features a page I started  
• reviews a page I started
• reverts my edit

For now, we are listing the first notification types now in development, but may add more options in the future. From a technical standpoint:

'reverts and edit' includes edits undone and rolled back
'reviews a page' includes all versions of reviewed, including reviewed + tagged, reviewed + marked for deletion
'features a page' includes all versions of rated, featured or today's featured article

Web Settings

This third section will enable users to choose whether or not they want to see the notifications web flyout and badge, using a check box.

Here are the proposed contents for that section:

Web Settings

Show badge in the top right menu

By default, this option would be checked. But users who really don't want to see notifications on their browsers would have the option to turn off the flyout and the badge. They would, however, continue to have an 'All-notifications' page, even if it's not visible in their UI.

Web Notifications

After testing the beta version of Echo, many users have requested the option to control which notifications are included in their flyout and archive, in much the same way as they control email notifications. This is something which Wikipedians are likely to be very concerned about, so we may need to provide them with greater control than would be offered on a typical consumer site.

This 'Web Notifications' feature could be offered in much the same way as we do now, by providing multiple check boxes for each option: one for email and one for onsite. Note that this is the method which Google now provides for setting email and phone notifications (see Google notifications settings screenshot).

To that end, we may want to consider these contents for a revised preferences section, which might look a bit like this:

Notify me on the web or by email when someone ...	Web	Email
• posts on my talk page	*	√
• reviews a page I started	√	-
• links to a page I started	-	-
• links to my user page	√	√
• rates a page I started	√	√
• reverts my edit	√	-

Note: For talk page notifications, users could only disable email notifications, not web notifications, because it is important that users be notified if someone leaves a message for them. We may want to display a tooltip over that disabled checkbox to explain this exception ('You cannot disable talk page notifications on the web.'). For similar reasons, users would not have the option to disable general system messages (e.g.: 'your user rights have changed'), because it is critical that users be notified of such important events; therefore, there are no preferences for system messages, for which both web and email notifications would be provided.

Defaults by User Group

To better serve the unique needs of different user groups, we propose separate default settings for new and experienced users. As a rule of thumb, a new user could benefit from more notifications than an experienced user, due to their lower level of activity and need for more guidance.

This could be accomplished by looking up the user's registration date and user group and specifying different settings based on these criteria.

Here are proposed defaults for each user group, for discussion purposes.

New users

Defined as users registered in 2013, who do not have special user rights (except for auto-confirmed users, who should be included in this group)
Enable individual email notifications
Show badge (for onsite notifications)
Enable onsite and email notifications for:
- talkpage posts
- user mentions
- wiki love
- page links
- page reviews
Disable onsite and email notifications for:
- reverts my edit

Experienced users

Defined as users who registered before 2013 and/or have special user rights (e.g.: reviewer, rollbacker, stewart, administrator, etc.)
No email notifications (but send one email notification to invite them to opt-in after first month)
Show badge for onsite notifications (not shown for first month, while we debug en-wiki)
Enable onsite and email notifications for:
- talkpage posts
- user mentions
- wiki love
- page reviews
- reverts my edit
Disable onsite and email notifications for:
- page links

The above default settings are proposed here for discussion purposes, and will be revised and expanded based on user feedback.

Other Email Preferences

We may also want to have a section for selecting HTML vs. email later on, as well as reminding people of which email address their notifications are going to, with a link to the preferences that let you set your email address

See All-Notifications
It may be helpful to include at the end of this preferences section a link to the all-notifications page: 'See All Notifications.' This would make it easier for users to go back and forth between Preferences and All-notifications to see if they have the right settings.

Best Practices
These feature requirements are inspired in part by best practices used by other top sites or platforms (e.g.: Facebook, Twitter, Google, LinkedIn, Quora), as shown in these slides, and summarized below:

most settings are about email notifications
average of 35 settings, 20 turned on by default
typically grouped by object type (e.g.: pages, messages)
3 out of 5 sites we studied provide a daily and/or weekly digest

Dismiss

Dismiss flyout detail

The 'Dismiss' tool will enable users to turn off notifications they don't want to see in the flyout or the all-notifications page, so they don't have to go to the preferences page if they don't want to.

Purpose
The purpose of this feature is to give users more control over the notifications they receive, so they can:

quickly turn off notification types they don't want to receive
do this in the flyout or the all-notifications page, so they don't have to go to preferences
change their preferences so they no longer dismissed notification types, both on the web and by email

Functions

Here is how the dismiss feature would work, as shown in the mockups to the right:

When the user hovers over a notifications, a gray "(x)" button appears on the right side.
When the user clicks on that "x" button, the notification is replaced with "Turn off all 'Edit Reverted' notifications?", with a "Turn off" button and "Cancel" link
- If the user clicks "Turn off", disable both web and email user preference for that notification type, and show this message: "You have turned off 'Edit Reverted' notifications. <Undo>"
- If the user clicks "Turn off", show the notification again and do nothing else.
- If the user clicks "Undo", show the notification again and do nothing else.
Once the user turns off a particular notification type, all notifications of this type will be dynamically removed from the flyout and all-notifications page and will no longer appear in the emails either
If later on the user wants to re-enable a dismissed notification type, they can do this in preferences

Exceptions
The dismiss button would not be available for certain types of notifications which are considered critical and time-sensitive, such as:

talk page messages
system notifications (e.g. your user rights have changed)
onboarding (e.g. welcome, check out your watchlist)

These notifications cannot be dismissed and no 'X' button will appear when you mouse over them in the flyout or all-notifications page.

Best Practices
Here are some examples of how this dismiss feature is implemented on other sites like Facebook:

Bundling

The 'Bundling' feature will enable users to only receive a single notification instead of many notifications about related events, so they are not overloaded with too much information. Bundling aims to address one of the key challenges for this product: how to make sure that we inform people without spamming them.

Purpose
The purpose of this feature is to:

reduce the number of notifications users receive for related events
show a single notification instead, bundling all related notifications into one
provide a single link where users can take meaningful action on all bundled notifications

Bundled Notifications
Only certain types of notifications can be bundled to meet the above objectives, in order to provide a single link where users can take action on all the bundled notifications. In the current list of notifications under development, these two types seem to be good candidates for bundling:

talk page messages
page links

Here are examples of how these bundled notifications would be displayed in the flyout:

Tom Morris and 3 others posted on <your talk page>:
"Thanks for your good work on the 'Breakfast' article. ..."

(instead of showing several different notifications about posts on your talk page)

or:

'Portland Public Library' was linked by Kaldari and 2 others. <See all links to this page>.

(instead of showing several different notifications about links to your started page -- the bundled notification would point to the special page 'What links here')

A key requirement is that all notifications to be bundled share the same link, so that a user can easily take action on all these notifications from that same page. Developers of new notification types will have access to a special setting where they can enable or disable this bundling method. To prevent spam, this bundling setting will be turned on by default for new notification types, but clear guidelines will be provided to developers about the single link requirement. We may also want to include an automated check that a single link is shared by all bundled notifications, to prevent developers from inadvertently making it hard for users to take action.

We don't expect to bundle other notification types currently in development for the first release, because they are either unlikely to trigger multiple notifications, or can not provide a single link where the user can take meaningful action. That said, we are considering other types of notifications, which might benefit from bundling:

'5 editors contributed to <Porcelain money> since your last edit.' (See 'Activity for your recent edit')
'<Your watchlist> has 15 new events which may interest you.' (See 'Check out your watchlist')
'Your page on <Higgs Boson> is becoming popular (1k views, 20 comments and linked by 2 other pages today)'
'You have 6 new notifications on <Commons>' (the first cross-wiki notification we expect to implement)

Functional requirements
Different functional requirements will be needed for bundling web notifications in the flyout and all-notifications page, as well as for email notifications.

For web notifications in the flyout, functional requirements include:

For each notification type that has web bundling enabled, look for all the recent events that occurred in the last 7 days
Sort all notification events by link, then group notifications that share the same link, and sort each group by date, only going back to the last 7 days
For each group, construct a bundled message that identifies the last user who triggered a notification, then gives the number of other contributors
The message will include only one action link for all bundled events on the flyout (e.g. a link to your talk page)
If a payload is required for that notification type, only show the most recent text snippet from the last bundled event
Show the bundled notification as soon as all messages are ready

For web notifications in the all-notifications page, functional requirements include:

For each notification type that has web bundling enabled, follow the same basic requirements in the all-notifications page as for the flyout, but only for the last 7 days
For events that occurred earlier than in the past 7 days, we can stop bundling notifications and display individual notifications instead, as outlined below
For each event that occurred earlier than in the past 7 days, simply show a notification for each event that identifies the user who triggered it
Bundled messages will be constructed in the same format as for the flyout, but links to users and other pages will be added on all-notifications page

For email notifications, functional requirements include:

For each notification type that has email bundling enabled, follow the same basic requirements as for web notifications for constructing messages, as outlined above
Send the first notification for each notification type and for each link group right away (if individual notifications are set in preferences)
If more events occur for that notification type and link group, delay sending a bundled notification for 4 hours (for individual notifications)
If even more events occur for that type and group after the first four hour period, send any bundled notifications every four hours (and reset the counter, as needed)
For the email digests, list the bundled notifications that took place that day or week for each type and link group, and send them at their regularly scheduled time

For example, here is what a user might receive over the course of a day (with preferences set to individual email notifications):

1pm: 'Tam Valley' was linked by Kaldari. [single event notification]
5pm: 'Tam Valley' was linked by Tom Morris and 2 others [bundled notification]
9pm: 'Tam Valley' was linked by BSitu [single notification]
1am: [no events took place, send nothing, start cycle again]
4am: 'Tam Valley' was linked by Fabrice Florin [single notification]
5am: 'Tam Valley' was linked Okeyes and 1 other [bundled notification]

Note that for maximum flexibility, we propose different settings for web and email bundling (for example, we could enable web bundling for talk page notifications, but disable email bundling to match current user expectations about this important notification type). Also, we propose that these bundled notifications do NOT show links under usernames in the all-notifications page: this greatly reduces the workload for developers, without affecting the user experience significantly.

Dependencies
Bundling features require a robust job queue infrastructure, which now does not exist for the current version of Echo.

Best Practices
Here are some examples of how this bundling feature is implemented on other sites like Facebook or Google:

JobQueue

Various asynchronous backend tasks performed by MediaWiki are added to one or more job queues and popped off those queues and processed. The current JobQueue class stores its data in MySQL tables and uses Memcached for performance and some features. The number of waiting jobs ebbs and flows through the day, but it is not uncommon to have hundreds of thousands waiting and for there to be a significant delay between queueing a job and it getting performed. Jobs will range from relatively cheap tasks like sending a password rest email to fairly expensive ones like transcoding media files.

Adding Echo notifications to the job queue massively increases the number of cheap to process jobs in the queue. It requires, at least for en-wiki, a new type of JobQueue class that can efficiently route notifications to users without load on the database, without waiting in line between longer running, less urgent tasks, and without delaying important tasks that impact many users

The new queue type is likely to be Redis based although this is not a requirement. It should provide a subset of the interface of the current JobQueueDB implementation that is sufficient for Echo's queue needs. Specifically it should provide:

* doBatchPush() that adds one or more jobs to the queue
* doPop() that takes a job off for processing.

Management methods like doAck(), doIsEmpty() and doGetSize() are desirable but not required.

Page update specific functionality like doDeduplicateRootJob() are not needed as that functionality will stay on the existing queue for the foreseeable future.

Echo Notification Storage

Job queues are not generally persistent, but the system needs to both deliver notifications to users in a timely manner, and also present previously sent notifications for later viewing. This means that as well as putting new notifications into a queue, the system also needs to add them to a persistent mailbox store.

Old notifications are of very limited value, so it is desirable to flush notifications that are more than 30 days old.

Notification Types

We expect that a wide range of notifications, will be developed over time for this project. Many of the ideas we've discussed are described in more detail below, as well as on this prioritized notifications list.

As a rule of thumb, we aim to develop notifications that are:

interactive: inviting new users to interact with other contributors
positive: making new users feel good about participating on Wikipedia
frequent: likely to happen regularly, to keep them coming back

Notification ideas are listed below in different sections corresponding to these categories (positive, interactive, neutral, negative, infrequent, under investigation).

Note that only a few of these notifications would be developed for the first release.

First Notifications

For the first release of Echo, we are currently working on a first set of notifications, as outlined below and described in more detail in their respective sections.

We have now developed these notifications:

Talkpage Message ('you have a message')
Page Review ('your page was reviewed')
Page Link ('your page was linked')
Edit reverted ('your edit was reverted')

We have started development for these notifications:

User Mention ('someone mentioned you')
Welcome ('welcome')

Here is a short list of other notifications we are now considering for the first release:

Your watchlist ('check out your watchlist')
Thank you ('someone thanked you')
Reply to Message ('someone replied to you')
Page Rated ('your page was rated/featured')
User rights change ('your user rights have changed')
Blocked user post ('a user you blocked left a comment')

Not all these notifications may be included in this first release, and we are now investigating their feasibility and desirability to determine which ones to develop first.

Interactive

This group includes notifications that invite new users to interact with other users on a regular basis (but that are not consistently positive or negative).

Talkpage Message

This notification occurs when someone posts on your talkpage. This includes events like starting a new section ('new message' or 'template'), or editing an existing section ('respond' or 'cleanup').

<Smallbones> posted on <your talk page>:
"Thanks for all your fine work on Wiki Loves Monuments!".

Trigger: User edit (click 'Edit' -- or 'New section')
Main Link: Talk page (to edited section, if possible)
Payload: Show 'Edit summary' (or 'New section first line')
Frequency: High
Priority: High
Category: Talk page message

Flyout Summary: [icon] Fabrice Florin posted on <your talk page>
Archive Summary: [icon] <VBamba> posted on <your talk page>
Email Subject 1: You have a message on your talk page [generic, from an anonymous user]
Email Subject 2: Shiftchange posted on your talk page [specific, from a registered user]
Email Summary: Wikipedia user Shiftchange posted on your talk page.

For interactive notifications such as this one, it seems helpful to include the user name in the email subject line 2 ('Shiftchange posted on your talk page'), instead of a generic subject line 1 ('You have a message on your talk page'). This may help new users recognize other members and form stronger relationships with them as a result. But we would only show a user name for registered users, not for anonymous users (IP addresses): in cases like this, we would use a generic message instead. If this is too complex for the first release, we may start with just the generic message.

For the first release, we're not required to specify different messages for each event type, and can simply use 'post on your talk page' as a general-purpose summary, as proposed above. For the new user we are now targeting, it does not seem necessary to have special messages such as 'started a new section on your talk page', which seem more complicated than newbies need -- and could be inaccurate.

Special requirements
For talk page messages supported by Echo, we plan to disable all the yellow talk page notices which now appear when your talk page has been edited. Disabling this current default new messages bar is basically a binary choice, not one dependent on notification type. Also, the yellow bars now tell you whether the talk page edit was one or multiple editors, which is not yet part of the Echo interface. If practical, we would not provide notifications for minor edits, unless they exceed 100 bytes per edit. Other than these caveats, we now expect that the proposed implementation for the first release would cover most of the needs of those who get the current yellow bar.

User Mention

This notification occurs when your username is mentioned by someone else, with a link on any talk page (other than yours):

<Sun Creator> mentioned you on <this page>:
"I am impressed by the work that <Kaldari> has done on this project."

Trigger: User edit including a link to your user page from any talk page other than yours, made by someone other than you
Main Link: Talk page where your name was added (go to the edited section that includes your name, if possible)
Payload: Show phrase that includes your name, if easy to do (or Edit summary as a backup)
Frequency: Medium
Priority: High
Category: Mention

Flyout Summary: [icon] Fabrice Florin mentioned you on <this page>
Archive Summary: [icon] <VBamba> mentioned you on <this page>
Email Subject 1: Someone mentioned you on Wikipedia [generic, from an anonymous user]
Email Subject 2: Sun Creator mentioned you on Wikipedia [specific, from a registered user]
Email Summary: Wikipedia user Shiftchange mentioned you on <this page>.

Reply to Talkpage Message

This notification occurs when someone replies to a message you left on their talkpage:

<Jimbo> replied to your post on <their talk page>:
"You are welcome. I completely agree with you".

Trigger: User edit on other talkpage section you started
Main Link: Talk page (to edited section, if possible)
Payload: Show 'Edit summary'
Frequency: High
Priority: High
Category: Talk page message

Started Page - Talkpage Message

This notification occurs when someone leaves a message on the talk page of a page you started:

<Knowledgekid87> posted on <the talk page> of <Transcarpathian Art Institute>.
"Good first stab. Have you considered expanding the history section?"

Trigger: User edit on talkpage of article you started
Main Link: Talk page (to edited section, if possible)
Payload: Show 'Edit summary' (or 'New section first line')
Frequency: Medium
Priority: Medium
Category: Talk page message

Positive

This group includes notifications that can provide positive reinforcement to new users on a regular basis. (Infrequent ones are listed in a separate group).

WikiLove

This notification occurs when someone posts WikiLove on your talkpage:

<Utar> sent you <WikiLove>:
"Good call! Thanks for making that revision."

Trigger: User WikiLove post
Main Link: Talk page (to edited section)
Payload: Show first line of message
Frequency: Medium
Priority: Very High
Category: WikiLove

Flyout Summary: [icon] Fabrice Florin sent you <WikiLove>
Archive Summary: [icon] <VBamba>sent you <WikiLove>
Email Subject 1: Someone sent you WikiLove [generic, from an anonymous user]
Email Subject 2: Utar sent you WikiLove [specific, from a registered user]
Email Summary: Wikipedia user Shiftchange mentioned you on <this page>.

If needed, we could use the generic email subject 1 above for the first release, to reduce complexity and deploy sooner.

Started Page - Linked

This notification occurs when a page you started is linked from another page:

<Portland Public Library> was linked by <Peteforsyth>. <See all links to this page>.

or, if multiple links were made to this page since the first one, bundle additional notifications as so:

<Portland Public Library> was linked by <Peteforsyth> and 2 others. <See all links to this page>.

Trigger: User edit (linking to page you started)
Main Link: 'What links here' page (see example)
Payload: None
Frequency: Medium
Priority: High (positive)
Category: Cross-reference

Flyout Summary: [icon] Portland Public Library was linked by bsitu. <See all links to this page>.
All-notifications Summary: [icon] <Tamalpais Valley> was linked by <Fabrice Florin>. <See all links to this page>.
Email Subject 1: A page you started was linked on Wikipedia [generic]
Email Subject 2: Life of Pi (film) was linked by Kaldari [specific]
Email Summary: <Portland Public Library> was linked by Wikipedia user Vbamba. <See all links to this page>.

Web preference default: Enabled for new users, disabled for others
Email preference default: Enabled for new users, disabled for others
Web bundling: Enabled
Email bundling: Enabled
Dismiss feature: Enabled (web-only)

If needed, we could use the generic email subject 1 above for the first release, to reduce complexity and deploy sooner. For notifications that are about a page, such as this one, it would seem helpful to include the page name and the user name in the email subject line 2. However, this could be an issue for long article titles, which may need to be truncated after 20? characters, so the subject line can be understood. But if this is too complex for the first release, we could start with just the generic subject line 1.

Special requirements: This notification will only be shown/sent for article namespace links, and only for internal links in double brackets. We will not show or send it for talkpage links, transcluded links, redirects or page moves -- or any other page links that could create spam. This notification would be opt-in for existing users, which means that it would only be shown/sent to current users if the check it in their preferences. However, it would be checked by default for new users (see defaults section of the preferences page).

Started Page - WikiProject

This notification occurs when a page you started is added to a Wiki Project:

<Muir Woods> has been added to the <California WikiProject>.

Trigger: User edit (to WikiProject page?)
Main Link: WikiProject page (to edited section, if possible)
Payload: Show 'Edit summary' ?
Frequency: Low
Priority: High (positive)
Category: Cross-reference

Featured Feedback

This notification occurs when feedback you posted is featured by an editor:

Your feedback on <Tamalpais Valley> has been featured by <Kaldari>.

Trigger: Your feedback was 'featured' by another user (found useful by an editor, or marked as helpful by a reader)
Main Link: Feedback page for this article (showing new post on top of 'featured' list, as we do now for displaying your last post on AFT)
Payload: Show your comment from the featured post (or moderator note, if any)
Frequency: Medium
Priority: High (positive)
Category: Feedback

Flyout Summary: [icon] Your feedback on <Tamalpais Valley> has been featured by Kaldari.
Archive Summary: [icon] Your feedback on <Tamalpais Valley> has been featured by <Kaldari>.
Email Subject 1: Your feedback has been featured by Kaldari [generic]
Email Subject 2: Your feedback on <Tamalpais Valley> has been featured by Kaldari [specific]
Email Summary: Your feedback on <Tamalpais Valley> has been featured by Wikipedia editor Kaldari.

Neutral

This group includes notifications that are not particularly interactive, and not consistently positive or negative.

Started Page - Marked as reviewed

This notification occurs when a page you started is marked as reviewed (and possibly tagged in the process):

<Porcelain money> was reviewed by <Sriharsh1234>.

Trigger: Editor review (or patrol) of page you started
Main Link: Your page
Payload: Tags (if any)
Frequency: Low
Priority: Medium
Category: Page review

Flyout Summary: [icon] <Porcelain money> was reviewed by <bsitu>.
Archive Summary: [icon] <Tamalpais Valley> was reviewed by <HFung>
Email Subject 1: A page you started was reviewed on Wikipedia [generic]
Email Subject 2: Life of Pi (film) was reviewed by TheHelpfulOne [specific]
Email Summary: <Tamalpais Valley> was reviewed by Wikipedia user Jorm

For notifications that are about a page, such as this one, it would seem helpful to include the page name and the user name in the email subject line 2 ('Life of Pi (film) was reviewed by TheHelpfulOne'). However, this could be an issue for long article titles, which may need to be truncated after 20? characters, so the subject line can be understood. If this is too complex for the first release, we could start with just the generic subject line 1 ('A page you started was reviewed on Wikipedia').

If needed, we could use the same summaries above for all notifications related to 'started page reviews' below. For the first release, it doesn't seem essential to change the wording for each subgroup, if we think the tags tell clearly what the issue is. See sections below for more details.

Note that there are 5 or more possible variations of this type of notification, which are described in more detail on this prioritized notifications list.

Started Page - Maintenance tags

This notification occurs when a page you started is marked as reviewed and also tagged in the process:

<Transcarpathian Art Institute> was reviewed by <QatarStarsLeague>.
Tags: Orphan, No categories, Copy edit

Trigger: Editor review (or patrol) of page you started -- with tags
Main Link: Your page (with templates)
Payload: Tags
Frequency: Low
Priority: Medium
Category: Page review

Flyout Summary: [icon] <Transcarpathian Art Institute> was tagged by QatarStarsLeague. Tags: ...
Archive Summary: [icon] <Transcarpathian Art Institute> was reviewed and tagged by <QatarStarsLeague>. Tags: ...
Email Subject 1: A page you started was tagged on Wikipedia [generic]
Email Subject 2: Life of Pi (film) was tagged by TheHelpfulOne [specific]
Email Summary: <Transcarpathian Art Institute> was reviewed and tagged by QatarStarsLeague. Tags: ...

If needed, we could use the generic email subject 1 above for the first release, to reduce complexity and deploy sooner.

Note that there are 72 possible tags that may need to be supported for this notification type.

Started Page - Feedback

This notification occurs when a page you started features new feedback:

<Tamalpais Valley> has new feedback.

Trigger: New feedback 'featured' on page you started ('featured' = marked as helpful or useful)
Main Link: Feedback page for your page (showing new post on top of 'featured' list)
Payload: Show comment from featured post
Frequency: Medium
Priority: Medium (neutral)
Category: Feedback

Flyout Summary: [icon] <Tamalpais Valley> has new feedback.
Archive Summary: [icon] <Tamalpais Valley> has new feedback by anonymous user <76.181.147.57>.
Email Subject 1: A page you started has new feedback [generic]
Email Subject 2: <Tamalpais Valley> has new feedback [specific]
Email Summary: <Tamalpais Valley> has new feedback by anonymous Wikipedia user 76.181.147.57.

If needed, we could use the generic email subject 1 above for the first release, to reduce complexity and deploy sooner. For notifications that are about a page, such as this one, it would seem helpful to include the page name and the user name in the email subject line 2. However, this could be an issue for long article titles, which may need to be truncated after 20? characters, so the subject line can be understood. But if this is too complex for the first release, we could start with just the generic subject line 1.

Negative

This group includes notifications that can make new users feel bad about participating on Wikipedia. In general, we recommend that these notifications only be shown if the user 'opts-in' in their preferences.

Started Page - Marked for deletion

This notification occurs when a page you started is marked as reviewed and also nominated for deletion, including one or more deletion tags (AFD / CSD / PROD):

<Government of Kazakhstan Airline> was reviewed and marked for deletion by <Jetstreamer>, 
who started <this discussion>. 
"This article may not be suitable for inclusion in Wikipedia, according 
to Wikipedia's policies and guidelines."

Trigger: Editor review (or patrol) of page you started -- with deletion tags
Main Link: Your page (with templates)
Payload: Tags
Frequency: Low
Priority: Medium (negative, but important)
Category: Page review

Flyout Summary: [icon] <Government of Kazakhstan Airline> was marked for deletion by Tchay. Tags: ...
Archive Summary: [icon] <Transcarpathian Art Institute> was reviewed and marked for deletion by <Jorm>. Tags: ...
Email Subject 1: A page you started was marked for deletion [generic]
Email Subject 2: Angry Birds was marked for deletion by TheHelpfulOne [specific]
Email Summary: <Angry Birds> was marked for deletion by Wikipedia user QatarStarsLeague. Tags: ...

If needed, we could use the generic email subject 1 above for the first release, to reduce complexity and deploy sooner.

Note that there are about a dozen possible tags that may need to be supported for this notification type.

Edit reverted

Sample notifications: edit reverts and reviews

This notification occurs when your edit was just reverted (undone or rolled back):

Your edit on <Hurricane Sandy> was <reverted> by <Knowledgekid87>.

Your edit on <Hurricane Sandy> has been reverted by <NewsAndEventsGuy>. 
"Undid revision 521530151: With what other sentence in the lead is it redundant?"

Trigger: Editor reversion (click 'Undo', 'Rollback', manual or automated)
Main Link: Diff page (shows difference between your edit and theirs)
Payload: Show 'Edit summary'
Frequency: High
Priority: Low (negative, opt-in)
Category: Edit revert

Flyout Summary: [icon] Your edit on <Government of Kazakhstan Airline> was reverted by Knowledgekid87.
Archive Summary: [icon] <Transcarpathian Art Institute> was reverted by <Eloquence>.
Email Subject 1: Your edit was reverted [generic]
Email Subject 2: Your edit on Angry Birds was reverted by NewsAndEventsGuy [specific]
Email Summary: Your edit on <Angry Birds> was reverted by Wikipedia user NewsAndEventsGuy.

If needed, we could use the generic email subject 1 above for the first release, to reduce complexity and deploy sooner.

Infrequent Notifications

This group includes notifications that are not likely to happen often for new users -- and therefore could be held back for future releases, even though they can be very positive.

Started Page - Rated

Sample notifications: recognitions and cross-references

This notification occurs when a page you started receives a Good article rating:

<Angry Birds> was rated as <"Good">.

Trigger: Editor adds category to page you started
Main Link: Your page (or category page?)
Payload: None
Frequency: Low
Priority: Very High (positive)
Category: Recognition

Started Page - Featured

This notification occurs when a page you started is linked from another page:

<Thomas Percy> was just <featured>.

Trigger: Editor adds category to page you started
Main Link: Your page (or category page?)
Payload: None
Frequency: Low
Priority: Very High (positive)
Category: Recognition

Started Page - Today's Featured Article

This notification occurs when a page you started is Today's Featured Article:

<Nancy Drew> is <Today's Featured Article>.

Trigger: Not sure
Main Link: Your page (or Main Page?)
Payload: None
Frequency: Low
Priority: Very High (positive)
Category: Recognition

Welcome

This notification occurs when a user creates a new account:

Welcome to Wikipedia

Trigger: Account creation
Main Link: A welcome page?
Payload: None
Frequency: Low
Priority: Very High (positive)
Category: Milestone

Note: This notification would be developed by the E3 team.

Under investigation

This group includes notifications that show potential for engaging users, but that require more research to determine their feasibility.

User rights change

This power user notification occurs when your your user rights have changed (e.g.: 'you now have rollbacker privileges').

Your user rights have changed - you now have <rollbacker> privileges.

Trigger: Administrator action
Main Link: User rights page
Payload: None
Frequency: One-time

Blocked user post

This power user notification occurs when someone you blocked (as an administrator) has left a message on their talk page. (This currently requires administrators to watchlist the talk pages of people they've blocked).

Someone you blocked has left a message on their <talk page>.

Trigger: Blocked user edit
Main Link: Blocked user talk page
Payload: Edit summary or first sentence
Frequency: Low

Answer to your question

This notification occurs when your question is answered (in Help / Teahouse/ Moodbar).

<Kaldari> answered a question you asked in <Teahouse>.

Trigger: User edit (in section you started on helpdesk page)
Main Link: Helpdesk page (to edited section, if possible)
Payload: Show 'Edit summary' (or 'answer first line'?)
Frequency: Low

Activity for your recent edit

This notification occurs when other editors contribute to a page you recently edited (and your edit survived).

5 editors contributed to <Porcelain money> since you last edit.

Trigger: User edits (with no revert)
Main Link: Article page
Payload: None
Frequency: High

Features under consideration

These features are being considered for Echo, though it is not clear yet if they will be included in the first release or future releases. They generally aim to add value for new and experienced editors through a variety of ways (e.g.: by making the watchlist more visible to new users, or by making the notification tools more helpful to experienced users). We aim to investigate these features in coming weeks, and determine which of these can be realistically implemented for the first release, then flesh out their requirements further.

Positive notifications

We would like to provide more positive notifications for the users who have completed an unreverted edit, leading them towards a "happy path" towards further contributions.

Rationale: Oftentimes, new users receive no positive feedback from the community about their first edits, even if they are productive; they only get negative feedback if their edits are reverted. Many studies have shown that positive reinforcement plays an important role in increasing a user's productivity, and some have pointed out that it takes up to 5 positive notifications to make up for one negative one -- hence the important of such notifications. The assumption here is that some of these users are well-intentioned and could become constructive editors.

This problem can be illustrated by this visualization of new user's first edits. Our research suggests that about 74% of a new editor's first actions are article edits, and yet we currently offer no positive notifications to support these first article edits -- only a negative notification ('your edit was reverted').

Here are some of our first ideas for giving positive feedback to new users at the time they need it the most. Please add any other ideas for solving this important issue in the talk page.

Thank you notification

This feature would enable editors to send a 'Thank you' notification to users who make constructive edits, to give them positive feedback. To make this possible, we would add a 'Thank' link next to 'Undo' for each edit by a logged in user (see mockup). When you hover over that link, a tooltip would say 'Send a thank you notification to this user.'. When an editor clicks on that link, the link would change to 'thanked' and this notification would be sent to the new editor, including a text snippet from the edit summary:

'<Kaldari> thanked you for your edit on Golden-crowned Sparrow:
 'Added reference about the bird's song' 
 <View your edit>'

This notification would link to your diff page for that edit. The user link in the notification would enable the new user to interact with the editor who thanked them. If multiple editors thank you for the same edit, we would bundle the notification to say: 'Kaldari and 2 others thanked you for your edit on Golden-crowned Sparrow. '[latest edit summary.]' <View your edit>', in the same way as we plan to bundle page links or talk page messages.

To keep things simple for this feature, we would not display the number of 'thanked' tags to anyone but the recipient of that notification. (Though over time, if the community finds this feature productive, we could expand it with a counter showing the number of positive responses, in future releases.)

This feature would be restricted to logged-in users only, because notifications do not work for anonymous users at this time. So the link would not appear next to anonymous edits (or edits from bots). If necessary, we could restrict this feature even more, and only show the link for new editors that have fewer than 100 edits (though other more experienced editors might find it valuable as well).

A simple preference under 'Miscellaneous' would allow users who don't want to see this link to turn it off, in the same way as WikiLove can be turned off (e.g.: 'Enable the Thank link to send positive notifications to users').

Alternative names we considered for this feature: 'Like', '+1', 'Mark as helpful' or 'Mark as useful'. We could also use an icon, such as a heart, thumbs-up, or checkmark.

From a technical standpoint, this feature would be pretty easy to implement in our time frame for the first release, according to our developers. The main question is whether or not our community would agree to this feature. But the outcome for new users is likely to be very constructive, from an editor engagement standpoint.

Huggle Notifications

Once we have an external API, we have the opportunity to hook into third-party anti-vandalism tools like Huggle (or ClueBot). Some of these tools could be adapted to provide a "Mark as useful" button (Huggle already has such a button), which could send a notification to the user who made the 'good' edit saying something like:

'Kaldari marked your edit as useful for Golden-crowned Sparrow. <View your edit>'

This would require us to provide a hook (or API) to the third-party developers, but it's actually a lot less work than having to find some other trigger in MediaWiki for it to take its cues off, and has a better signal:noise ratio than any trigger we've yet discussed.

While this seems like a reasonable solution to investigate, it may not be a viable candidate for Echo's first release, because the hooks / API requirements and coordination with third-party developers are likely to take several months to set up.

Contributions since your last edit

One possible solution we have discussed is giving new users a bundled notification that '20 people have contributed to this page since your last edit'. While this type of activity is currently handled by the watchlist, this type of 'contributions' notification could have a positive impact in getting the new user to go back to a page they edited earlier (particularly if they don't yet feel motivated to use the watchlist). This is almost certainly better than only sending them negative notifications when their edits are reverted (which is like a slap in the face) -- or sending them no notifications whatsoever after their first edits (which is what we are doing now).

However, this particular notification idea may not always have a positive impact if the user's previous edit was manually reverted or overwritten, which we cannot easily detect, sadly. Also, this would only work for new users with low levels of activity, and could quickly become overwhelming for more active users. In order to address this issue, it may be possible to define a threshold that would only send a notification that 'x people have contributed to this page' if x exceeds a certain percentage y of the total number of editors for this page (e.g.: y = 20%? ). (Note that this notification could be based on the number of edits to that page, instead of editors, if that is easier for development purposes.)

From a technical standpoint, this is the least feasible idea, as it would require generating Echo events for every edit on Wikipedia for every editor of every article (even if they are bundled or filtered to a certain group of users), which could quickly explode the database -- and also likely crash the servers with massive numbers of expensive queries.

Active user notifications

We have already identified a number of positive notification ideas for active new users in the [Echo_(Notifications)/Feature_requirements#Positive 'Positive' section below]. They include:

WikiLove
Started Page -Linked
Started Page - WikiProject

While these positive notifications are likely to motivate active new users, a challenge we face is how to give positive reinforcement to new editors immediately after their first few edits, before they become active enough to start new pages or be noticed for a WikiLove message. Also note that we are already sending notifications for WikiLove messages, since they are being posted on your talk page; so this reduces the urgency to create a special WikiLove notification for the first release.

Other ideas will be added to this section as we brainstorm them with our team and community.

Watchlist notifications

This section is about interactions between Echo and the watchlist, which are intended to work together as a system for surfacing things that happen on MediaWiki and Wikipedia sites.

Our team deliberately identified certain items as out of scope for Echo, since they would be covered by the watchlist. For example, we didn't see a need to send an Echo notification for every item that appears in the watchlist, to avoid redundancy. In general, we view the watchlist as providing ambient awareness about lots of minor events, while Echo is intended to notify the user about important events that impact the user directly.

That said, we would like to make the watchlist (and the "notifications" inherent in the watchlist) more visible to new users, who may not be aware that this tool even exists -- or may not be motivated to check it, since it is empty when they first register. So we need to at a minimum let new users know that the watchlist exists and potentially make some preference changes to make watchlists easier to use.

Here are some of the possible ways we can ensure that new users are aware that 1) they have a watchlist and/or 2) there are new items in that watchlist. The first couple features below seem like likely candidates for the first release, while the others are more likely to be considered for future releases.

Watchlist Guided Tour

This proposed feature would show a guided tour to new users, the first time they visit their watchlist.

This would offer the following benefits:

explain what the watchlist is for
invite users to click on the star button to add items to their watchlist
show them how to switch between watched articles and recent changes

The Editor Experimentation team (E3) has expressed interest in developing this feature as part of their onboarding project.

This guided tour could be promoted with a popup call to action that would appear for 10 seconds on any page, once a user has items in their watchlist. This call to action might look like this:

'Check out new changes in <your watchlist>'

or:

'Did you know you have a watchlist? <Learn how it works>.'

In either case, this would link to the watchlist page, where a guided tour would greet them.

Special requirements:

This guided tour would only be shown once (though a small 'What is this?' link could be included on the watchlist page, in case they want to see it again).
It would only be shown if the user has already made a couple edits (which would automatically add the pages they edited to the watchlist, thanks to the related 'Add pages to new user watchlists' feature).
By default, we would show the 'All watched' tab first when the guided tour starts (see mockup).
Note: We considered pre-filling your watchlist with a few articles so we could show the guided tour earlier in the new user's lifecycle, but ruled this out for now, because it would be inconsistent with the current behavior.

Check out your watchlist

This feature would send a bundled notification informing new users that there is new activity on their watchlist page. The notification might look like this:

'Check out <your watchlist>: it has 15 new changes this week.'

This notification could be sent once a week for the first three months after a new user's registration (or it could be sent only if they have not visited their watchlist in more than x days, if feasible).

The rationale for this notification is that most new users forget that they have a watchlist and do not get in the habit of checking it regularly until they become active. This feature would keep nudging them to check it out, and encourage this important habit over time.

This feature is a bit more complex than [#How_to_use_your_watchlist|'How to use your watchlist']] (see above), but would be a good candidate if we have the bandwith to develop more notifications after the initial release.

If keeping track of 'new' changes proves difficult to implement, we could simply remove the word 'new' and just let the users know how many changes are now listed on their watchlist, which should be easier to extract.

Here are more parameters about this notification:

Trigger: weekly (up to 3 months?)
Main Link: watchlist page
Payload: None
Frequency: Medium
Priority: High (useful to new users)
Category: System Message

Flyout Summary: [icon] 'Check out <your watchlist>: it has 15 new changes this week.'
All-notifications Summary: [icon] 'Check out <your watchlist>: it has 15 new changes this week.'
Email Subject 1:'Check out your watchlist'
Email Summary: 'Check out <your watchlist>: it has 15 new changes this week.' [could add a bit more copy if needed]

Web preference default: Enabled for new editors, disabled for experienced editors
Email preference default: Enabled for new editors, disabled for experienced editors
Web bundling: Disabled
Email bundling: Disabled
Dismiss feature: Enabled

Special requirements: This notification would only be sent weekly (for up to 3 months?), only to new editors.

Add pages to new user watchlists

If you are a new user, this feature will automatically add pages that you edit or start to your watchlist.

The rationale for doing this is to fill in new users' watchlists based on their recent edits, so they can find this tool more valuable and start using it more often.

This could be simply implemented by changing the default for new users to automatically check the current Watchlist preference called 'Add pages and files I edit to my watchlist.'

If a user wants to stop this from happening at any time, they can turn off that preference.

Note: Someone has already a proposed revision for this feature, which is awaiting review. So we could simply approve or tweak that revision, then merge it ourselves. Ask Steven Walling for more details.

Other watchlist ideas

Here are some other ideas which we considered, but postponed for future releases.

Watchlist page filters

The watchlist page itself could be improved to make it simpler and less overwhelming for new users. Maryana and Vibha have prepared this mockup to showing how the watchlist filters could be improved with just a few simple changes.

While this feature seems like an easy and obvious improvement, some experienced users might prefer the current layout, which they are already comfortable with. So this would require that we discuss these changes with community members first (and/or that we only make the improvements available to new users (with an opt-in preference for current users).

We agreed that this is not a must-have for the first release, but that we would try to fit it in shortly after, if we can get development resources for this simple improvement.

How to use your watchlist

This proposed feature would send a one-time notification x days after registration (e.g.: 'How to use your watchlist').

This would offer the following benefits:

make the user aware that they have a watchlist
provide a link to the watchlist, so they can check it out
explain that they can use the star button to add items to their watchlist

The Editor Experimentation team (E3) has expressed interest in developing this notification as part of their onboarding project.

This notification might link to a guided tour on the watchlist page itself -- or a special page explaining how to use the watchlist.

In the flyout, this notification might look like this:

'Did you know you have a watchlist? <Learn how it works>.'

Here are more parameters about this notification:

Trigger: x days after registration
Main Link: watchlist page (with a a guided tour?) -- or special page showing
Payload: None
Frequency: One-time
Priority: High (useful to new users)
Category: System Message

Flyout Summary: [icon] Did you know you have a watchlist? <Learn how it works>.
All-notifications Summary: [icon] Did you know you have a watchlist? <Learn how it works>.
Email Subject 1: Wikipedia tip: How to use your watchlist
Email Summary: Did you know you have a watchlist? <Learn how it works>. [could add a bit more copy if needed]

Web preference default: None for system messages
Email preference default: None for system messages
Web bundling: Disabled
Email bundling: Disabled
Dismiss feature: Disabled

Special requirements: This notification will only be sent once.

Watchlist Counter
A third way is to put a little counter next to the watchlist link in the user menu, so they can see there are new items from any page they visit on Wikipedia.

Watchlist popups
Another idea that has been suggested would pop a notification saying there are new events in the watchlist until the new user has clicked n times on the watchlist (default off for experienced editors), until the user builds a habit.

Power user notifications

We would like to provide one or more notifications that would be valuable to experienced editors in the first release.

Rationale: the existing feature set for Echo's first release is now targeted mainly at the new user. This is intentional, since experienced users already know where to get information and don't need notifications as much as new users. But it would be nice if we had at least one feature that filled a need for the experienced editor.

Criteria: We plan to select one or two low-risk power-user notifications for the first release, as long as they can match the following criteria:

are viewed as really useful by the editor community
are feasible, with a low effort in engineering that will not delay the schedule
can be used by as many power users as possible over a period of time, so we get more bang for the buck
are simple enough that we can manage subsequent feature requests that will inevitably come from experienced editors

In coming weeks, we will to recommend notifications that generally address these criteria and help make this new product useful to as many experienced users as possible.

For example, Oliver has proposed these two notifications, based on IRC conversations with about a dozen power users:

user rights notifications ('your user rights have changed - you now have rollbacker privileges')
blocked user post ('someone you blocked has left a message on their talk page' -- currently requires administrators to watchlist the talk pages of people they've blocked)

In other conversations, we identified a few more useful notifications which would benefit many active editors:

'your page was rated or featured'
'your page was added to a wikiproject').

You can see some of the notification ideas we discussed at the end of this Google spreadsheet (in the yellow section). The next step is to boil down these various ideas into a short list, and run them by more power users, so we can identify the best prospects for development later this quarter.

Talk Counter

We have discussed the idea of offering a separate counter for the 'Talk' link that appears in the user menu, so that users know the number of unread messages they have on their talk page. The rationale for this proposed feature is to make it easier for people to find out about these personal messages, which could otherwise get lost in a stream of lower-priority notifications. We also discussed adding a flyout for this Talk links, where notifications could be listed separately from the main notifications flyout, but concluded that this would raise too many issues, making it impractical for the first release. For example, people are now used to clicking on the 'Talk' link to go to the talk page, and would resent having to click twice to go there. Also, there will inevitably be some dependencies with 'Flow', our upcoming user-to-user messaging product, and it seems better to hold off on developing a new talk notification features until that new product is further along.

Mark all as read

Provide a function that would let users mark notifications in the flyout as "read". This is particularly useful for power users who get a lot of notifications and want a quick way to get rid of them without paging through their archive, which requires many clicks. We want to investigate how this feature would fit with the current system, and whether the effect would be to remove the highlights or the entire notifications from the flyout (the latter case might be better labeled as a "Clear all" function).

API

Once Echo is further along, we plan to provide an API allowing 3rd parties to send notifications (rather than just extensions using hooks). The rationale is that other developers could expand on some of the basic notification framework we are now working on, without requiring the WMF to develop all new notifications. But this would also require us to develop a set of policies governing their use, to prevent folks from spamming our users. It's going to be a challenge to find the right policies to support both goals, and these policies will require extensive deliberations with the Wikipedia community. This is inevitably going to be a major project, both on the technical and community fronts, and is therefore out of scope for our first release. For now, we will continue to research our options, as well as review best practices from other popular platforms that provide a Notifications API to developers, such as Facebook (see their API overview and design guidelines) and Android. (Note that Facebook tracks whether notifications sent by third parties with their API are meeting certain thresholds in terms of clickthrough and uses that data to weed out low-performing or spammy notifications.)

Metrics

Research Goals

Here are our goals for collecting and analyzing data for the Echo notifications project.

We want to measure how effective Echo is in helping users:

learn about activity related to them
take action on notifications
participate on Wikipedia

We plan to answer these questions through a variety of measurements, some of which would be integrated in the Echo tools. Here are some examples of measurements we could take on, for discussion purposes. After an initial feasibility study and prioritization session, we will identify a few key metrics that we think are required and practical for our first en-wiki deployment.

Target Users

For Echo's first release, our primary target user will be new editors, but the tool will be available to other user groups as well (albeit with different default preferences for each group). Here are the options we are considering for testing on the English Wikipedia in early 2013:

New users

Defined as users registered in 2013, who do not have special user rights (except for auto-confirmed users, who should be included in this group)
Enable individual email notifications
Disable onsite and email notifications for edit reversions

Experienced users

Defined as users who registered before 2013 and/or have special user rights (e.g.: reviewer, rollbacker, stewart, administrator, etc.)
No email notifications (but send one email notification to invite them to opt-in)
Disable onsite and email notifications for page links

For more details on our first release plans, read this section on defaults by user group.

Volume Metrics

Note that some of the breakdowns proposed below (e.g. new vs. active user, or source breakdown) are useful across other types of metrics (e.g. posts vs. views vs. clicks). For example, it would be helpful to have a matrix table showing how many posts, views and clicks were generated by each user group across a range of sources.

Posts

How many notifications are being posted to users in a given week? (but not necessarily viewed)

for new users
for active users
for very active users

Note:

Rationale for this request: determine whether specific classes of users receive too many notifications

(This can be done by analyzing echo data, each notification is associated with a user along with triggering timestamp, with user edit count as a way to group new/active/very active users, we will know how many notifications are posted to each user group in a given week. However, a user may increase their edit counts in a very short amount of time, a new user two weeks ago may become an active user today, in this case, we would want to use EvengLogging to track the edit count at the time of triggering the notification)

Views

How many notifications are actually viewed in a given week?

on the flyout
on the archive page(s)
via email (plain text vs. HTML)

Notes:

We expect to use EventLogger to track these views
We would need to discuss building view tracking into our html email template and a server end to record views
Potential privacy concerns for HTML tracking, but OTRS has support for email impressions
For users with many pages of archive, how many click past the first page?

(We can track view of flyout by tracking the click on the badge, we can also track the view on archive page on page load, I am not sure if we can track view in emails unless users click on something in the email)

Clicks

How many notifications are being clicked on in a given week?

interactive notifications (e.g.: talk page messages, user mentions)
positive notifications (e.g.: wiki love, page link, promoted feedback)
neutral notifications (e.g.: page reviews, user mentions)
negative notifications (e.g.: edit reverts, page deletions)

Notes:

Break out clickthrough stats by source (flyout/archive/text-email/html-email)

(All these clicks can be tracked by EventLogging)

Preference Metrics

We want to measure changes in user preferences for both web and email notifications.

The primary rationale is to track how many users have web and/or email preferences enabled, and how many are disabling preferences.

All preferences

How many web and email notification types are enabled per user, on average?
How many users disabled both web and email notifications altogether this week?

Web preferences

How many web notification types are enabled per user, on average?
How many users disabled web notifications altogether this week?

Email Preferences

How many email notification types are enabled per user, on average?
How many users disabled email notifications altogether this week?
How many users switched to daily digest? weekly digest?
How many emails are being sent in a week to new/active/very active users?
For users who disabled or reduced their email settings, how many emails had been sent to them in the past day/week?

Notes:

Control for users with authenticated email addresses

Engagement Metrics

Users

How many unique users are clicking on their notifications:

every day
once a week
once a month
rarely or never

(We can analyze this by using the data from clicks)

Actions

How many users go on to take these follow-up actions?

make an article edit (ns0) -- this is the most important action
start a page
post a message
post feedback
upload a file
other contributions
do nothing

Notes:

We need to define what is the scope of a "follow up" action.
Since it is hard to determine which actions take place as a direct result of a notification, it might make sense to compare a group of users who are getting Echo notifications with a control group without Echo, then track the overall number of edits, new pages and other contributions from both groups over a period of time.
This would let us determine whether or not Echo is helping engage users who receive notifications more than other users.

(It's quite difficult to track user follow-up actions especially when it involves multiple sub-actions. For example, if a user posts on my talk page about an article edit, I get the notification, click on the link in the flyout and get redirected to the talk page, click on the article link in the talk page, then start editing by clicking the edit link, it involves many actions in the middle, any of these actions can be initiated from other places)

Quality Metrics

Usage

Which notification types are used the most? the least?

Welcome
Talk page message
Started page message
Wikilove
etc.

Notes:

The purpose of this request is to determine whether any of these notifications should be eliminated due to insufficient usage
To normalize the data, we may want to focus on the ratio of clickthrough versus frequency (e.g.: measure the number of clicks divided by number of notifications)

Productivity

Which notification types appear to be most productive in terms of follow-up actions?

Welcome
Talk page message
Started page message
Wikilove
etc.

Usefulness

How useful are notifications to users overall? to new vs. experienced users?

Very useful
Useful
Neutral
Not very useful
Not useful at all

Notes:

This measurement of customer satisfaction could be based on a simple survey shown to users for a short period of time.

Dashboards

General dashboards

We would like some live dashboards showing aggregated data above, dynamically-generated.

Specific dashboards

We would like some specific dashboards for each notification type, so that we (or developers using our API) can track their effectiveness.

User dashboards

We would like some individual user dashboards, so we can observe how typical users use this tool. It would also be useful to look at an average by user type as well (e.g. new user vs. active user average).

@@ Line 1,478: / Line 1,478: @@
 * Payload: None
 * Frequency: High
-<br>
-====Check out your watchlist====
-This notification occurs when your watchlist has new activity that relates to you.
- <Your watchlist> has 15 new events which may interest you.
-* Trigger: Watchlist events (if you haven't visited watchlist yet)
-* Main Link: Watchlist page
-* Payload: None
-* Frequency: Low (if only shown the first time)
 <br>