This page contains updates on the Growth team's work to increase new editor retention in Czech and Korean Wikipedias. The planning for this work began in June 2018. The team works on multiple projects at a time, but they will all be discussed on this page. During July, August, September, and October 2018, the team was also simultaneously working on a project to improve the Articles for Creation process and New Pages Feed in English Wikipedia, which is discussed on its own page.
How to get involved
It is important that our work is grounded in the reality of the communities we hope to help. If you have thoughts or ideas around this project or our team's work, please comment on this project's talk page.
Project updates
Update 2020-08-30: new mockups of "add a link"
Screen from a design concept for the "add a link" structured taskNew mockups of "add a link": back in May, we had a good community discussion about the "add a link" structured task (summary here). We then took community thoughts into account as we made new design mockups. We hope community members can check them out and react to them.
Russian Wikipedia: we have deployed the Growth features to Russian Wikipedia, which gets about 14,000 newcomers per month. Next, we're planning a deployment to Polish Wikipedia.
Variants C and D engineering: we are mainly engineering on this project, meant to increase how many newcomers attempt suggested edits.
Update 2020-08-24: planning for "add a link"
Meetings on "add a link": this week, the team will have a series of meetings planning for the "add a link" structured task. We'll post in-progress mockups of our ideas during this week so that community members can see and give us advice. We're also simultaneously planning to user test the designs to get thoughts from people new to editing.
"Add an image": although we are designing our first structured task using a link recommendation algorithm, we are also considering a future structured task that would prompt users to add an image from Commons to an article that doesn't have any images. A member of the WMF Research team is working on an algorithm to make intelligent suggestions, and we are spending some time looking through its results to see if it could become good enough to give to newcomers in a structured task. We are working in this Phabricator task.
Variants C and D engineering: this continues to be our main focus in terms of what we're building now, even as we plan for our next projects.
Update 2020-08-17: highest newcomers task numbers
Measurement: this past week had the highest volume of newcomer task edits (1,103 edits) and editors (193 editors) since the feature's release. We believe this is caused by both the "guidance" elements of newcomer tasks, and the addition of Persian and Hebrew Wikipedias in recent weeks. The graph of numbers of editors by week is shown here. More details and graphs on our measurements can be found on the project page.Number of editors using newcomer tasks as of 2020-08-17
Hebrew Wikipedia: we have deployed the Growth features to Hebrew Wikipedia, which gets about 2,000 newcomers per month. Next, we're planning deployments to Russian and Polish Wikipedias.
Upcoming meetings on "add a link": although the team's engineering is currently focused on Variants C and D, we are planning for our work next quarter on our first structured task workflow of "add a link". Next week, the team will have a series of meetings about three main areas:
Data analysis: to help us predict how newcomers might engage with "add a link".
User experience design: to develop some design ideas that we'll share with community members.
Engineering: to make an architectural plan of how to store and serve link recommendations.
Update 2020-08-10: deployed to Persian Wikipedia
Persian Wikipedia: we have deployed the Growth features to Persian Wikipedia, which gets about 6,000 newcomers per month, making it the third largest wiki for Growth features. Next, we're planning deployments to Hebrew and Russian Wikipedia.
Language question: we have added a "language question" to the welcome survey in all Wikipedias that have Growth features. This question will help us find out how many newcomers may be capable of editing with the Content Translation tool. See the question in the accompanying screenshot.Welcome survey, showing the new language question.
Variants C and D: the team continues to work on these homepage variants, and will be engineering on it for the next several weeks.
Update 2020-07-17: working on Variants C and D
Variant testing
The team has planned out how we will build Variants C and D, which are new version of the newcomer homepage that are meant to encourage more newcomers to try suggested edits.
We originally showed the designs in this update, but the images are included again here.
This will be the team's main project going forward in the coming weeks.
Other forthcoming work
In this task, we will soon be deploying a question to the welcome survey asking the languages that a newcomer knows.
In this task, we will soon be altering the help panel in Czech Wikipedia so that questions go to mentors, not the help desk. This will be a pilot to see if newcomers have better experiences through their mentors than the help desk.
Ever since newcomers tasks was deployed in November, we've known that the feature has "performance" issues. By "performance", we are referring to the time it takes for the feature to load on the page.
When newcomers visit the homepage, all the rest of the page appears immediately, and the newcomer tasks can take a couple seconds to appear. This is because queries and searches are running in the background to generate the lists of tasks.
This past week and next week, we are doing some work to measure exactly how long this step is taking for users around the world. We will use the data to decide if the experience is slow enough that the team should work on speeding it up.
Variant testing
We have been planning for the engineering work on Variant C and D, and we are starting the work this week.
Update 2020-07-05: asking language question
Conferences
Last week, we presented at the online-only Polish Wikiconference, showing the community how the Growth features work, and discussing whether they would be a fit for Polish Wikipedia. Slides are available here.
The work on guidance cleanup is now almost finished, and we're going to be moving on to future projects this week.
We continued to see elevated rates of newcomers clicking edit and saving edits on tasks, which we hope and think is attributable to guidance.
Language question
In the coming year, we are considering adding a new type of suggested edit to the newcomer tasks feed: an integration with the Content Translation tool. The idea is that, for newcomers who know more than one language, we would recommend articles to translate, or sections in articles to translate.
To prepare for this, we are adding a question to the welcome survey that asks newcomers which languages they know. This will help us have a sense of how many newcomers might be candidates for using the Content Translation tool, and it may also provide some interesting research results.
Update 2020-06-29: guidance cleanup and next projects
Last week was our second week of cleanup around the guidance release. We made many small interface improvements, as well as a few larger ones related to making it easier to navigate.
We may have a couple guidance things still to finish up this week, and then we'll be moving on to next projects.
Next projects
The main next project that the team will be moving onto is our next variant test: Variant C vs. D. This test will make suggested edits a much more prominent part of the newcomer homepage.
Before starting on that, we have a handful of interim tasks to do, including:
Piloting mentorship in the help panel, to see if newcomers and experienced users have a better experience by asking mentor questions in the help panel, instead of help desk questions.
We released guidance on June 15 in all wikis that have suggested edits.
We think we are seeing some early positive results from guidance.Number of distinct users saving a suggested edit, broken out by week and wiki. The week after the release of guidance had the most users since the suggested edits was first deployed.
Last week had the highest number of suggested edits completed since the feature was first deployed in 2019: 917 edits.
Last week also had the highest number of distinct users completing suggested edits: 139 users.
The team is now in the midst of spending two weeks cleaning up the feature, making small improvements, and fixing bugs. One change we're making will make it easier for users to navigate and minimize the help panel.
We spent last week testing, filing, and fixing bugs for guidance.
The biggest challenges have been around the logic and animations for how the panel displays the right information at the right time.
We'll be releasing to newcomers this week, and we'll continue to fix and adjust in the coming week or two. One example is this issue, in which we're going to make it easier to toggle between the guidance and the article that the user is editing.
Variant testing
After we finish working on guidance, the team will turn to our next variant test: Variant C vs. D.
This test will show suggested edits prominently on the newcomer homepage, and test different paths of onboarding the user to the feature.
The team is focused on finalizing the code for releasing guidance next week.
We have been testing, fixing bugs, and sorting out the edge cases around when and how the guidance should appear for newcomers.
We encourage everyone to start translating the many messages for the feature. To help, we've put together this translation guide, explaining where the messages fit into the feature.
Variant testing
In this update, we talked about results from Variant A vs. B. We found that Variant B, which has the user skip onboarding, leads to higher engagement with the suggested edits module.
Even though we had those results, we had not switched newcomers over to all receiving Variant B, because we also saw evidence that the onboarding may important for helping newcomers complete edits. Variant A seems to perform about 40% better than Variant B on mobile.
We now have enough data to look at whether one of the variants is more associated with completing edits. The result is that Variant A leads to more newcomers completing edits on moble, while the two variants do not have differences on desktop.
Therefore, we will be converting all newcomers over to Variant A while we do the work in the coming weeks to build Variant C and Variant D, our next test.
Overall, the test showed us that while Variant B's stronger affordance of the suggested edits module increases interaction, onboarding to the module is also important. We are using both these points in the designs for Variants C and D.Wireframe from our structured task explorations, showing how a user might leave feedback on an algorithm's suggestions.
Update 2020-06-01: new structured task wireframes
Wireframe from our structured task explorations, showing how a user might leave feedback on an algorithm's suggestions.
We have continued to have good discussion about structured tasks. So far, about 30 users have participated across four languages.
This week, we talked a lot about how we might automatically suggest spelling fixes in arbitrary languages. You can see our notes here from a conversation with User:Beland, who created a system to search for spelling errors in English Wikipedia.
Our team's designer has created new wireframes exploring some of the concepts that we think would be important for structured tasks. You can see them all here, with an example posted here.
We are planning on releasing guidance during the week June 15.
This week and next week are about finalizing animations, instrumentation, and testing for bugs.
The written content for the feature is now available for translation, and we're encouraging all wikis using Growth features to translate it in the next week if they can.
Update 2020-05-26: deployed to French Wikipedia
Scaling
We deployed the Growth features to French Wikipedia on 2020-05-19. So far, there have been 95 suggested edits from 36 different users. There have also been many mentorship and help desk questions.
Our discussion about structured tasks is continuing. We hope community members can visit the project page for structured tasks and give their thoughts.
So far, we've heard from about 20 community members across many different communities, and some of the main points of discussion are:
Generally positive feedback -- it sounds like we are thinking on the right track.
It is important that newcomers have the opportunity and are encouraged to learn to edit using the traditional editors, as opposed to only learning new structured editing tools.
Discussion of whether copyediting or reverting vandalism can be good structured tasks for newcomers.
Sketch of an idea for a structured workflow for adding links to an article, geared toward helping the newcomer edit at high volume.We hope community members can visit the project page for structured tasks and give their thoughts on the questions posted on the talk page.
This is a project we're in the middle of thinking about, and we want to hear advice and reactions from community members in as many projects as we can.
We plan to conduct this conversation over the next five weeks (through June 18). We'll be sending the information out in our newsletter and pinging users who we think would want to take part.
If you can translate the project page to your language, that will help us gather more community opinions.
We are continuing to finish up engineering work on guidance.
Regarding our previous update about having many translatable messages, we have decided to keep all the messages, and monitor whether communities struggle to translate them all.
Scaling
We are planning to deploy Growth features to French Wikipedia, which will be our first large Wikipedia. We'll start working on that deployment this week.
Update 2020-05-03: thinking about "structured tasks"
We are nearing the end of the engineering work for "guidance", currently working on the instrumentation.
One of our challenges is managing the many messages that will need to be translated. Since this feature guides users through completing suggested edits, it contains a substantial amount of text and examples that we wrote carefully to be helpful for newcomers. But since there are five different task types, with six or seven different steps, along with some examples sentences, and some differences between desktop and mobile -- there are about 50 messages to translate for this feature. This may be too many for communities to do easily and thoroughly. We are currently thinking about how to make this better, and welcome any ideas.
The idea is that we think we could break down editing workflows into a series of steps that newcomers can accomplish easily, especially with the assistance of machine learning.
This is an exciting idea, but there are also many potential pitfalls. This new project page is in progress, and after getting some more content on there, we'll start a community discussion so that we figure out the potential for this idea.
In our previous update, we mentioned that one idea for how we could more easily scale to bigger wikis is by converting the help panel over to ask mentor questions instead of help desk questions.
We're going to pilot this idea in Czech Wikipedia to see how it goes, and then decide if it is an improvement that can be brought to our other wikis.
Technical requirements are now written for our next test, Variant C vs. D. They can be found under this Phabricator tasks.
Scaling
As we prepare to deploy our features to larger Wikipedias, one important question is how we will scale the "help desk" and "mentorship features". Both these features require the involvement of experienced community members to answer questions. Large Wikipedias are expected to produce a large number of questions, and we want to make sure there are enough community members available to answer them.
One idea we have is to remove the "help desk" element from the help panel and to replace it with questions to mentors instead -- essentially creating one route for questions from newcomers: to their mentors. It will be easier for wikis to scale their mentorship capacity up and down than their help desk capacity. We think such a change could also enhance the newcomers' experience with suggested edits.
We want to try this on Czech Wikipedia in the coming weeks to see what the effects are on the volume of work given to mentors. The details are on this Phabricator task.
Update 2020-04-13: deployments to more wikis
Scaling
Last week, we deployed the newcomer tasks workflow to four additional Wikipedias that already had all the Growth features apart from newcomer tasks: Ukrainian, Hungarian, Armenian, and Basque.Mockup of Variant C on desktop
We also deployed all Growth features to Serbian Wikipedia.
We deployed all Growth features except newcomer tasks to French Wiktionary.
Together, newcomers on these new wikis have made 33 suggested edits since last week.
We finalized designs for our next variant test: Variant C vs Variant D. Variant C feature an initiated suggested edits module with optional onboarding. Variant D also features the newcomer tasks flow prominently, but immediately gives users the option to choose topics as part of mandatory onboarding. See mockups at right.
The analysis for our first variant test is now complete. It started in December, and split newcomers into these two groups:
Variant A: receive the original suggested edits workflow, in which they clicked a call-to-action to pass through two onboarding overlays before seeing the suggested edits module.
Variant B: arrive on the homepage with the suggested edits module being displayed, experiencing no onboarding overlays.
On desktop, Variant B yields double the interaction (clicking on anything in the suggested edits module), 60% more navigation (clicking on arrows to navigate to different suggested articles), and 30% more clicking on suggested articles.
On mobile, the variants perform the same for interaction and navigation, but Variant B leads to 15% less clicking on suggested articles.
Takeaways include:
Making the suggested edits module more prominent on the homepage gets more users to interact with it.
Though the overlays were not present on Variant B, they may still play an important role in giving the user context for what they're supposed to be doing with the module.
We will incorporate these takeaways as we design the next set of variants to test.
Update 2020-03-30: "top-of-funnel" work
Guidance
In terms of engineering, the team continues to work on "guidance", and it is possible to follow our progress in this Phabricator task.
Top-of-funnel work
We are now designing and planning around the project we'll be working on after releasing guidance: "top-of-funnel" work. We use this term because it is work that is meant to increase the number of newcomers who begin the "newcomer tasks conversion funnel". Right now, only about 20% of newcomers who visit their homepage interact with newcomer tasks, and we think that design changes can increase that.
We'll be doing this top of funnel work as a series of variant tests, which we started writing about here. Our very first variant test is almost complete, and the results will be posted next week.
You can look at some of the variant ideas we want to test in these mockups. They include ideas like:
Making the suggested edits module dominate the homepage.
Letting users peruse a list of task recommendations, instead of just looking at one option at a time.
Suggesting to newcomers who have previous edits that they edit articles they are reading.
Update 2020-03-22: slower work on "guidance"
The Growth team, and the rest of the WMF, are adjusting our pace of work as the world deals with the COVID-19 pandemic. We'll be working more slowly and more carefully in the coming weeks, allowing time for team members to take care of their families and themselves. See this blog post for a message from the WMF's CEO about how our staff is planning to adjust.
That said, our priority continues to be working on "guidance". The engineering work continues on that project.
We are also continuing to plan for how we will scale Growth features to many more wikis. We will begin reaching out to potential communities in the coming weeks.
The Growth team is currently planning our work for the coming fiscal year, which begins in July. We plan to build on the newcomer tasks project, which has been showing promising results.
In a previous update, we talked about evaluating a "link recommendation" algorithm. The idea is that a good algorithm could help us build a type of newcomer task that points newcomers to specific links that may need to be added -- giving them small tasks that fit their skill level, without requiring a huge amount of context.
We have since been developing this idea, and we're now sketching out an initiative we're calling "structured tasks". In short, we think that we can break difficult wiki editing workflows into steps that are much easier for newcomers to do, and much easier to do from mobile devices. We would put these structured tasks into the newcomer tasks workflow. They could include things like adding a link, adding an image, adding a reference, etc.
In the coming weeks, we'll post more information on wiki explaining this idea and asking for communities to react. While we think this could be a really exciting idea, it will be a lot of work, and there are many potential pitfalls. We'll need community thoughts to make sure we get it right.
We deployed ORES topic models to Arabic, Vietnamese, and Czech Wikipedias on 2020-03-05. Korean Wikipedian's models had some issues that are causing its deployment to be delayed.
In the coming weeks, we'll be publishing some information about how other Wikimedians and developers can access these models for their own work. As an example, one way to use them is with the "articletopic" search keyword. This works by typing something like "articletopic:sports" in a Wikipedia search bar to retrieve articles that are likely to be about sports. This page and links from it show the 64 possible topics to search.
This past week, we detailed exactly what user actions we'll log to understand how the feature is being used after we deploy it. The details are on this Phabricator task.
Update 2020-03-01: deploying ORES topics this week
We are planning to deploy our upgrade of the topic matching algorithm this week.
In our evaluation of the new ORES models, we found that the new models perform substantially better than the "morelike" algorithm we have been using for topic matching since Jan 13. We evaluated the models by looking at ten articles classified by the models into each topic, and then counting how many of the articles seemed to fit well in the topic.
The "morelike" algorithm averaged between 5.8 and 7.5 across the topics, depending on the language.
The ORES models averaged between 8.0 and 9.3 across the topics, depending on the language. The also have the added benefit of more topic granularity.
Engineering is underway to guide users through suggested edits via the help panel.
This past week, we drafted the language that will explain to users how to complete the different task types (copyedit, add links, add references, update article, and expand article). The challenge was how to give the right level of detail, while not overwhelming or confusing the newcomer. We want newcomers to complete a simple edit as quickly as possible.
We have also been deciding which data to record to understand the way that guidance will and will not be used.
We will soon be analyzing the results of our first homepage variant test, in which half of newcomers receive the suggested edits module already initiated on their homepage when they arrive.
As we await those results, we have designed additional homepage variants that attempt to combine the best of the variants from the first test. Those are being set up to be tested here.
We will also be designing and prioritizing a test that makes suggested edits the clear focus on the homepage, attempting to encourage more users to enter its workflow.
Update 2020-02-23: upcoming deployment of ORES topics
We are planning to deploy our upgrade of the topic matching algorithm either this coming week or the week after.
This upgrade will increase the number of topics available to newcomers from 27 to 64.
We will be altering the user interface to help newcomers navigate the larger number of topics. You can try out the upcoming interface in this prototype.
Planning to scale to more wikis
Over the past 1.5 years of the Growth team, we have deployed our features to nine wikis. As we learn that the features we're building are valuable to those wikis, we are now planning to deploy to many more wikis in the coming year.
The reason it takes work from staff and communities to deploy Growth features is that they have required community input to configure. For instance, community members supply our team with the most important links to help pages, and also supply us with a list of mentors for the mentorship module.
Our team is starting to make plans about how to streamline deployments so that we can work with more wikis more quickly. We'll be posting about those plans in coming weeks.
Engineering continues on loading ORES topic scores into the Search API, which will enable newcomer tasks to use them, and also enable other features to draw on topics.
Ambassadors are now evaluating the new scores to identify any topics that perform poorly. That evaluation is happening in this Phabricator task.
Link recommendation
All newcomer task suggestions are currently based on maintenance templates, a decision we made among many choices of where to source tasks to recommend.
Maintenance templates do not give newcomers much direction on how or where to edit in an article, and we believe they would benefit from having specific places in the article to edit.
We also see that many newcomers use newcomer tasks to add internal blue links to many articles.
Therefore, we are partnering with a researcher to attempt an algorithm that can recommend specific words or phrases in an article that should be made into blue links.
Ambassadors are evaluating that algorithm in this task, and we will decide within the coming weeks whether it might be good enough to incorporate into newcomer tasks.
Increased affordance of newcomer tasks
Coming up, we'll be working on some ideas to encourage more newcomers to try out the newcomer tasks workflow.
Homepage layout: we will test a variant of the homepage in which newcomer tasks is featured prominently as the main call-to-action.
Entry point while reading: for newcomers who have already done a newcomer task, we will create a design that suggests they edit articles they are reading that have maintenance templates on them.
Update 2020-02-10: beginning engineering on guidance
In watching user behavior after the deployment of topic matching on 2020-01-21, we see that about 75% of users who have the option to choose topics actually choose some. Though we'll need to take more time to see how topics have impacted the rate that users do suggested edits, this initial number is a good sign that users are interested in using topics.
The team is now primarily concentrating on upgrading the topics to use the ORES models instead of our initial "morelike" algorithm. This week, our team's ambassadors will be testing them to make sure they yield strong results.
We are also beginning engineering work on this part of the newcomer tasks workflow.
After completing user tests on the prototypes for this feature, we discovered a couple important findings:
Our mobile design, in which the panel "peeks" up from the bottom of the page, works.
Users are really interested in seeing demo videos of how to complete edits. Videos are difficult to produce, keep up-to-date, and scale, and so the team will need to talk in the future about how to address this need.
With that information from the user tests, we were able to create these versions of the prototypes, which are what we intend to build now:
We deployed topic matching to our four target wikis on 2020-01-21, meaning that all users with the homepage can access newcomer tasks and choose topics to narrow them.
We're now working on the next upgrade for topics: using the ORES models instead of our initial "morelike" algorithm. These models are expected to give better results, and we expect to begin using them in February.
We'll also be beginning the engineering work to add guidance to the newcomer tasks experience, so that newcomers understand how to edit the suggested articles once they arrive.
Deploying to additional wikis
On 2020-01-22, we deployed all the Growth features besides newcomer tasks to Ukrainian, Hungarian, and Armenian Wikipedias. These are our next three "non-target" wikis, meaning they receive all but our latest feature.
Our updated date for deploying topic matching is 2020-01-21.
This past week, we have been testing and fixing bugs.
Deploying to additional wikis
We are preparing to deploy Growth team features to Ukrainian, Hungarian, and Armenian Wikipedias.
These will be our next set of "non-target" wikis, meaning that they receiving features one step after our target wikis of Czech, Korean, Arabic, and Vietnamese Wikipedias.
Prototype of how the help panel will "peek" from the bottom of the page to offer guidance on mobileV1.1 (topic matching)
We are almost finished building the topic matching addition to newcomer tasks, using the "morelike" algorithm. This is planned to be deployed 2020-01-14.
After about a month, we will replace the morelike algorithm with the higher-performing ORES model. This model will be made available through the Search API so that other Mediawiki features will be able to draw on it.
After we deploy topic matching, the team will move on to engineering this final part of the newcomer tasks workflow.
We ran 12 user tests on desktop and mobile in December and we are currently analyzing the results to determine the final designs that we will build. These are the prototypes we tested:
After posting initial numbers last week, our team's ambassadors went through 98 edits done through newcomer tasks, and found quite encouraging results:
90% of the edits are good faith
80% of the edits are productive
Many newcomers are correcting typos, adding links, and adding content.
We see that some users of newcomer tasks are doing their first or second edits. There are also many who created their accounts a few months ago, and started doing newcomer tasks after discovering the new feature on their homepage.
An improvement made in the past week increases the loading speed of the module.
After planning out the engineering work, we realized that the earliest that ORES models can be used for this feature is mid-February. Therefore, we have started building with an alternative algorithm: "morelike". We will deploy the first version with morelike and switch to ORES when ready.
The team has deployed several bug fixes and improvements to the initial version of newcomer tasks. One example is that all suggested edit cards now show the number of page views that the article has had in the previous two months. This is intended to convey to newcomers that their edit will have impact (see accompanying image).Screenshot of suggested edits module showing number of views
We have published initial numbers on how far users go into the newcomer tasks workflow, including the number of users who have made edits made so far (68 users).
We have started our first "variant test" for newcomer tasks, in which we test two versions of the feature against each other to see which performs better. In this first test, half of users will have the original workflow in which they go through overlays explaining the suggested edits module. The other half will arrive on their homepage with the suggested edits module already present. We will see which leads to more clicks and successful edits.
It is now possible to filter Recent Changes feeds to edits coming from newcomer tasks using the "#Newcomer tasks" tag. We hope that experienced users will use this to thank newcomers and encourage them in their work!
Technical work for making topic designations available for this feature work can be found here in Phabricator. The team is now moving most of our attention to this work.
Update 2019-12-08: newcomer tasks cleanup and first results
V1.0 (basic workflow)Mockup of concept for using the help panel to guide suggested edits
The team is continuing work on small bugs and UI improvements to this initial version of newcomer tasks. Though we're eager to move on to V1.1 (topic matching), we've decided that it's important to fix these issues so that they don't cause complications in the future.
The data we see for newcomer tasks so far is exciting. Healthy numbers of users are initiating the module and interacting with it, and we're seeing higher numbers of users than we expected who are clicking on tasks and editing them. In looking at some of the edits that newcomers make via the module, most look like they are in good faith and are productive. This is really encouraging news, because we expected few users to click on tasks without V1.1 (topic matching) and few users to complete edits without V1.2 (guidance).
We will publish some initial numbers within the next week. We'll also be asking our team's ambassadors to look at edits made through the module to get a sense of whether they are productive edits or not. This will inform our design for V1.2 (guidance).
The engineering work for this is beginning, with collaborations with the Search and Scoring teams to make topic model scores available for the Growth team's software.
We intend to release this version in January 2020.
V1.2 (guidance)
We are in the process of designing exactly how the help panel will be adapted to guide newcomers while they are doing suggested edits.
See the accompanying image for an example mockup of one of the concepts (the actual content is just a placeholder; i.e. we will not be including videos in this version).
On 2019-11-20, we released newcomer tasks in Arabic, Korean, Vietnamese, and Czech Wikipedias. All users who have the newcomer homepage will now see newcomer tasks, which means that half of newcomers will have a chance to see it, and the other half will remain in our control group, not receiving the homepage at all.
We were able to fix most of the important bugs before the release, but there is still clean-up work to do in the next week.
The deployment went well, and we see from the early data that newcomers are able to interact with all the parts of the workflow, with some going all the way to saving edits in suggested articles. We'll continue to watch this data to watch for issues and draw any possible early conclusions.
Over the next weeks, we'll be working simultaneously on the backend algorithm for ascertaining the topic of an article, and on the frontend interfaces in which users choose topics.
Update 2019-11-17: upcoming release of newcomer tasks
Mockup of how newcomers will select their topics of interestMentor training: our team's Czech ambassador held our first "mentor training" with four members of the Czech Wikipedia community in-person in Prague. Attendees said this training went well, and our next steps are to adapt it to be given online with communities that have members located far away from each other.
Newcomer tasksMockup of how newcomers will select their topics of interest
V1.0: we have spent the last week testing newcomer tasks and fixing bugs. The workflow is nearly ready to be released to our target wikis either this coming week or the following week.
V1.1 (topic matching): in addition to working with the Scoring team to improve topic matching methods, we'll also be working with the Search team to find a good way to store topic predictions of articles so that the newcomer tasks workflow can easily access them. We're talking with that team this coming week. We've also finalized our designs for this version, shown in the accompanying image.
V2.0 (additional task types): we chose to use maintenance templates as the source of tasks for the first version of newcomer tasks because all wikis have them, and there are many different kinds. They were the easiest way for us to get started with this feature. But they may not be the best sorts of tasks for newcomers to do. We're starting to work with the Research team on an algorithm that will recommend specific internal links for newcomers to add to articles -- as opposed to the suggestion to add links to an article "in general". We think this will be an even simpler task with which newcomers can get started.
Update 2019-11-08: testing newcomer tasks
Screenshot of newcomer tasks feature in Arabic Wikipedia
Mentor training: our team's Czech ambassador will be trying out a new mentor training in person in the Czech Republic. For those in the area, the signup page is here.
V1.0: we are now testing newcomer tasks v1.0 in both the beta and production environments to make sure that the designs are implemented correctly and that translations are strong. Please contact us if you would like to help test!
V1.1 (topic matching): we have started to work the Scoring team to improve the ORES model they have built to predict the topic of an article (Art, Music, Sports, etc.) The Scoring team will be working on clarifying the topic values used in the model and on making the model have full coverage in our target wikis.
The team is getting close to completing the engineering for V1.0 of newcomer tasks, which will move us into the testing phase.
Users who have their homepage turned on in Beta wiki can try out newcomer tasks. Please contact us if you would like to try it and need help. We're interested in any opinions or questions!
Planning meetings
We have been posting notes from our team's planning meetings last week.
Try out some suggested edits: we did a hands-on exercise in which we attempted to edit articles with maintenance templates. This helped us understand what challenges we can expect newcomers to face, and gave us ideas for addressing them.
Variant testing: after deploying the first version of newcomer tasks, we want to start testing different variants of the feature, so that we can improve it iteratively. In this session, we brainstormed different variants to test.
Update 2019-10-27: takeaways from planning meetings
The team continues engineering on the first version of the feature, to be released during November.
We are preparing for v1.1, an upcoming version that will add topic matching to newcomer tasks, so that newcomers can receive suggested edits based on their topics of interest. We have prototyped and evaluated three different methods for topic matching (see this Phabricator task for details). Early results look like a new ORES model for predicting the topic of an article may be a promising choice.
We are also preparing for v1.2, an upcoming version that will add guidance to the newcomer tasks workflow, so that newcomers who select articles to edit will have some explanation of what they're supposed to do. We'll use the help panel for this, and have some initial designs that can be seen in this mockup.
Planning meetings
Last week, the team held a series of remote planning meetings in which we spent concentrated time thinking about our long-term and short-term goals. Below are summaries of some of our sessions:A wireframe from the design exercise on creating a "dream homepage"
Integrated newcomer experience: the central question for the week was, "What does an integrated newcomer experience look like?" Our team has been building several different features for newcomers, but we believe they need to fit together into a cohesive experience. For instance, how might the newcomer tasks experience fit together with upcoming ideas for helping newcomers to build a user profile?
Try out some suggested edits: we wanted to make sure we understand what newcomers will experience when they use the upcoming suggested edits module (the module in the newcomer tasks flow). To do this, we each attempted to edit two articles that have maintenance templates, and took notes on what worked well and poorly.
Pre-mortem of newcomer tasks: we made a list of potential negative outcomes from newcomer tasks (e.g. "Very few newcomers click on articles to edit") and made reasons why those things might occur. We'll use this list to anticipate these outcomes and try to avoid them.
Newcomer tasks v1.1 (topic matching): after deploying the first version of newcomer tasks, we'll work on adding topic matching, so that newcomers can receive tasks that fit their interests. In this session, we reviewed the results of our different potential algorithms to determine which might work best for v1.1.
Newcomer tasks v1.2 (guidance): after deploying v1.1, we'll work on adding guidance for newcomer tasks through the help panel. In this session, we talked about initial designs and potential specifications.
Variant testing: after deploying the first version of newcomer tasks, we want to start testing different variants of the feature, so that we can improve it iteratively. In this session, we brainstormed different variants to test. For instance, we may test a version in which users proactively initiate the feature, against a version where the feature is already pre-initiated.
Sequencing our work: there is a long list of things the team wants to build and improve, and in this session we talked about the right order to work on those things.
Designing a dream homepage: in this design exercise, each team member developed an opinion on which homepage features could be removed, could be merged, moved to other places in the platform, or should be added. We each discussed how our "dream" homepage created an integrated newcomer experience.
The team is in the midst of engineering the first version of this feature, to be released during November.
We've completed a measurement plan that defines what questions we want to answer about the feature and what data we'll collect to do it. That will be posted in the coming weeks. We are currently working with other departments to make sure we are adhering to security and privacy best practices.
One innovation we plan is to be able to test multiple variants of newcomer tasks at the same time -- instead of just testing one variant against a control group. This will help us learn more quickly what works best for newcomers.
Planning meetings
Next week, the team will have a series of planning meetings, in which we'll work to more clearly define the future of our team and the newcomer experience we're building in the coming year.
We're looking forward to posting some of the things we accomplish during the week.
Update 2019-10-14: next set of designs for newcomer tasks
Newcomer tasksMockup of new "overlay" that explains newcomer tasks to mobile users
We completed user tests of the mobile version of newcomer tasks, and the notes are published here. Here are some highlights:
In general, users understood how to access and use the module, and what it was for.
They are surprised that “adding an image” is considered a difficult task.
The most important thing for us to improve is the guidance that users will get in the help panel while they are doing a task. It is challenging to show guidance on mobile while also showing the article to be worked on.
After the results from the mobile user tests, we've altered the designs a little bit and finished writing the wording for the feature. We have new versions of both the desktop and mobile designs, which the team's engineers are building now:
The team is finalizing our "measurement plan" for newcomer tasks, which lays out what data we'll record and how we will experiment with the feature to learn how to improve it. We'll be working on getting approvals from different WMF departments, and then publish a draft on wiki so communities can follow along.
Mentor trainings
Growth team features, such as the help panel and mentorship module, rely on experienced editors to answer questions from newcomers as "mentors".
In order to help experienced editors become better mentors, we have written a training and are planning to test it with the Czech community in the coming months. If it goes well, we will try to do the training with other interested communities.
The help panel was first deployed to newcomers in January 2019, and we have now finished analyzing data to determine its impact. A brief summary is below, and a more in-depth analysis is forthcoming.
In summary, the help panel has not shown an increase in activation (whether a user makes their first edit) or retention (whether a user returns to edit again).
The help panel does cause slight changes in edit volume (the number of edits made), but the results are conflicting. It increases edits during a newcomer's first day, but decreases them over the following two weeks.
Although increases in activation and retention are our team's ultimate goal, we also observe that large numbers of newcomers open the help panel (20%) and interact with it once open (50%). This tells us there is demand for the sort of help that the help panel offers.
Taken together, these results are disappointing -- we were certainly hoping that the help panel had a positive impact on the metrics. While we have many ideas for how to improve the help panel, we have decided to keep our attention on the newcomer homepage and newcomer tasks projects for the coming months. Those projects have promising early results and reach many more newcomers than the help panel does. The help panel will also be used as part of the newcomer tasks project, and will remain as part of the newcomer experience that the Growth team is building. We may revisit it in more depth in the future.
We have completed six user tests of the desktop version of newcomer tasks. All testers felt positively about the feature, and we have some notes on design changes to make. We'll post those next week.
Next, we'll be designing and testing the mobile version of newcomer tasks, as the team's engineers write the code for the desktop version.
We are also working on our measurement plan for the feature, deciding which interactions we'll record data for, and how we will A/B test it.
At the same time, we are prototyping approaches for "topic matching" -- how to show tasks to newcomers related to the topics they're interested in. This is a big challenge, and is tracked in this task.
Newcomer homepageShare of newcomers who discover their homepage increased when features were deployed at the two dotted lines.
In previous updates, we've mentioned our work on "features to aid discovery". Those are features that encourage newcomers to visit their homepage, hopefully ensuring that all newcomers at least know that their homepage is available. These features were deployed in July, and our goal was to double the number of newcomers who visit their homepage. We have now analyzed the results, and we see that we have exceeded this goal! See the graph at the right for details.
In Czech Wikipedia, the percentage of newcomers visiting their homepage has gone from 34% to 72%.
In Korean Wikipedia, the percentage of newcomers visiting their homepage has gone from 19% to 49%.
Since deploying the newcomer homepage in May 2019, we've looked at how often different components of the page are used. The page contains two ways for a newcomer to ask a question: the help module and the mentorship module. Users use the mentorship module to ask questions about 15 times more often than the help module. Therefore, to make the homepage simpler, are removing the ability to use the help module to ask a question.
The page includes these links to the video and slides from the Growth team's presentation about our work and learnings in the past year. Those links are a great way to learn more about our work, and we welcome any thoughts, reactions, or follow-up questions.
These are some of our topline notes:
Alignment on newcomer retention: It seems like Wikimania attendees generally believe that newcomer retention is an important problem.
Connecting offline to online: Many Wikimania attendees are very active in off-wiki events, such as edit-a-thons. Because of this perspective, we heard a lot of enthusiasm for ideas that connect our features better to offline events, such as making homepage mentors correspond to offline mentors.
Mentor dashboard: When talking about the mentorship module in the newcomer homepage, we heard experienced users request some sort of dashboard with which they could monitor newcomers who may need help, and have a way keep track of and thank them for their work.
The team is now engineering on the newcomer tasks project. You can follow along in the Phabricator tasks listed in this high level task.
Our evolving designs can always be found in these mockups (use arrow keys to navigate). Those mockups contain explorations of all the difference parts of the user journey, even though we'll only be building a subset of the features for the first version. See this list for what will be in the first version.
Next, we'll be running user tests on the mockups to further refine our design even as we engineer the code.
Update 2019-09-07: specifications almost ready for newcomer tasks
Mockup of guidance while doing a task using the help panel
This week has been all about finalizing plans for the newcomer homepage.
Engineering work will start next week.
We've settled on plans for a first version that has these capabilities:
Modifications to the welcome survey to more clearly personalize the newcomer tasks
Adding a module on the newcomer homepage that provides suggested edit tasks.
Allowing users to filter tasks based on topics of interest and difficulty of task.
Giving users guidance through the help panel once they are attempting a task.
The two most difficult parts of this will be (a) filtering to topics of interest and (b) guidance through the help panel. One or both of these capabilities may not be part of the very first version, but will come soon thereafter.
Update 2019-08-30: working on newcomer tasks
Wikimania
Several members of the Growth team were at Wikimania in Stockholm.
We led sessions about our team's work and other growth-related topics.
We also had many interesting conversations with community members.
Full notes are forthcoming in a future update.
Mockup of possible design for newcomer tasks moduleNewcomer tasks
The team is focused on this new, ambitious project to recommend easy tasks for newcomers to get started with editing.
The project page now lays out our approach to sourcing tasks and to design. We plan to use maintenance templates to find tasks, and then to match them to newcomers' topics of interest.
We are in the midst of finalizing draft designs and investigating the technical approach for matching tasks to newcomers.
We will be moving on to engineering the backend and user testing the front end designs over the next week.
We hope community members can look over our plans and react on the talk page. If our plans don't look like they will work, we want to know right away!
The test of the welcome survey has completed in Arabic Wikipedia, and shows that the welcome survey does not negatively impact users and cause them to leave the wiki (the same result as on Czech and Korean Wikipedias). This means that we can safely use the welcome survey on that wiki to personalize newcomer tasks.
Update 2019-08-11: preparing for Wikimania
Wikimania
Wikimania starts this coming week, and several members of the Growth team will be present.
We are helping to organize the Community Growth space, which is a conference track focusing on how to bring newcomers into our movement and our projects.
This week, we have been bringing more technical definition to the project and we have been working on initial designs.
We are likely to use maintenance templates to find tasks to recommend to newcomers. All our target wikis use these, and they are the quickest approach to include in the initial version of the feature.
We are looking into algorithms that will allow us to match articles to newcomers based on the interests they describe in the welcome survey.
Since much of the team will be at Wikimania, we'll not make much progress over the next two weeks, but our priorities will be (a) technical evaluations of different approaches, and (b) designs for the "feature discovery" component.
Our team spent time planning and designing around the newcomer tasks project.
We've made some important decisions this week:
The newcomer persona we'll be thinking about most when designing is the Knowledge Sharer. This is a user who arrives on Wikipedia with some substantial edits in mind they want to make. We think we can increase their retention by giving them easy tasks that grow their skills, so that they are successful making the edits they arrived to make.
We're now thinking about the newcomer tasks project as being just one of the ways to help newcomers get started. Newcomers prefer to learn in different ways. Some may want to read a tutorial. Some may want to practice in the sandbox. Some may want to learn by doing. We want to give them choices, with newcomer tasks being the one we build first.
We decided that the top priority for the first version of task recommendations is the interface on the newcomer homepage where newcomers receive tasks, as opposed to prioritizing the quality of the tasks, or the guidance newcomers receive when doing the tasks. Those are important parts, but the top priority will be seeing how much demand there is for the tasks.
We've narrowed to a set of potential types of tasks to recommend, and currently thinking through them in terms of how easily they can be implemented and whether they will work well on our four target wikis of Czech, Korean, Arabic, and Vietnamese Wikipedias. This narrowed list includes things like using article improvement templates, the content translation tool, or adding links for orphan articles.
Other notes
Arabic newcomers have been using the help panel and newcomer homepage at levels comparable to the other three target wikis, with particularly high levels opening the help panel, and dozens of mentorship questions being asked from the homepage.
We are currently planning to develop a "mentor training" to help new communities learn how to mentor newcomers. We'll be taking notes in this Phabricator task, and welcome ideas and recommendations from everyone! The first training will be piloted in the Czech community.
This week, our team began a project to add task recommendations to the newcomer homepage. This will be a major project during the next six months.
We believe that recommending relevant and interesting tasks to newcomers will help them learn to make the edits they want to make, and give them a path toward contributing regularly.
Right now, the team is in a planning and design phase. We'll be adding more notes to the project page, but the big question right now is where to find the tasks to recommend to the newcomers. Please check out the page, and post any ideas or comments so far on its talk page.
Other notes
The newcomer homepage and help panel were deployed to Arabic Wikipedia this week. Arabic Wikipedia now has all the Growth team features.
The "discovery features" mentioned in previous updates were deployed to all our target Wikipedias this week, which is aimed at increasing the number of newcomers who find their homepage.
Feature to increase discovery of homepage from Special:ContributionsThis week, we'll attempt to deploy three features to increase the discovery rate of the homepage (they are listed in the update from 2019-07-12 below). See the accompanying image showing one of the features: a call to action on a newcomer's Special:Contributions page to encourage them to visit their homepage if they don't have any contributions yet.
We will also attempt to deploy all Growth features (welcome survey, help panel, and newcomer homepage) to Arabic Wikipedia, which is our fourth target wiki. The challenge with this deployment is that, unlike our other target wikis, Arabic Wikipedia uses Structured Discussions for its help desk page, and we needed to do additional work to make sure the help panel and homepage can post questions to that page correctly.
Update 2019-07-15: current goals and priorities
Example mentor introduction
The team has set goals for the coming three months. These are part of longer-term objectives for the coming year that we will be discussing with communities in the weeks ahead. We'll be posting more information on all these goals and initiatives.
Newcomer homepage: increase activity through a task recommendations module. Now that we have seen several weeks of positive activity on the newcomer homepage, we think that the most important thing to add is a way for newcomers to find tasks to work on. The challenge will be recommending the right kind of tasks at the right point of their journey.
Newcomer homepage: increase feature discovery rate by 100%. Right now, only 20% - 30% of newcomers ever visit their homepage. We want to double that number by making sure all newcomers know how to find it via features linked from this task.
Deploy new experiments to four target wikis simultaneously (Korean, Czech, Vietnamese, Arabic). The team is now working with four target wikis, and we want all wikis to be receiving all our work at the same time, which increases our team's efficiency.
Deploy stable Growth features to five non-target wikis. Some of the Growth features, such as the welcome survey and help panel, are strong enough that wikis who want to try them are welcome to by working on this checklist.
Research and design for next project, either (a) structured user profile or (b) topic/neighborhood features. After we work on task recommendations, there are future developments to the homepage we want to plan. See our team's early notes on structured user profile here and on topic/neighborhood features here.
Help panel: increase usefulness through improvements to affordance, search, and UX flow. We have looked closely at data and anecdotes from the usage of the help panel, and we plan to pursue specific improvements to increase its effectiveness.
Mentor introductions: we've seen in homepage data so far that the mentorship module is much more popular for asking questions than the help module. We think that may be because of the "personal touch" that comes from having a specific mentor. To accent that personal touch, we have released the ability for mentors to customize their own introductory messages. See the screenshot at right and this set of guidelines for writing good introductions.
Renamed mobile menu item: to avoid user confusion between their homepage and the existing Wikipedia main page on mobile, we have altered the mobile side menu to refer to the main page as "Main page", instead of as "Home". See this task for images.
Priorities
We are currently prioritizing "features to aid discovery", in pursuit of our goal to increase the number of newcomers who discover their homepage.
GuidedTour after account creation: after creating their accounts, all newcomers will be informed of how to find their homepage through a GuidedTour.
Button on empty contributions page: many newcomers visit their Contributions page seeking to make their first contribution, but instead they find an empty page with no results. When their page is empty, a button will encourage them to visit their homepage.
Redirect from email confirmation: many newcomers confirm their email addresses, but the wiki page they land on after confirming doesn't explain what to do next. This feature will bring newcomers to their homepage when they confirm their email address.
Update 2019-07-05: mobile homepage release and homepage leading indicators
The mobile version of the homepage was released on June 27. Newcomers in our target wikis can now access their homepage both from the mobile devices and from desktop. We expect this to increase the number of newcomers who visit their homepage by about 50%.
As of July 1, the homepage is also activated in Vietnamese Wikipedia (both mobile and desktop).
Leading indicators
Before deploying the homepage, the team decided on a set of metrics that we would evaluate to see whether the feature is behaving in a healthy way. We've published our first evaluation of those numbers here. In summary, we are happy about the homepage's performance so far. We see about half of visitors clicking on something on the page, and many of them visiting multiple times. We also see that they are taking actions that the page recommends, such as confirming their email address, starting their user page, or asking a question to their mentor. One of the most interesting observations so far is that while 25 users have asked questions to their mentors, only one has asked a question to the help desk.
Update 2019-06-22: mobile homepage work
Screenshot of Mediawiki's newcomer mobile homepage feature in beta
This week has been about fixing bugs and doing validation on the mobile newcomer homepage. We currently plan to release the mobile version during the week of June 24.
A couple of the most important issues have been:
Making sure that text in some of the longer languages (such as Czech) fit comfortable on a mobile screen. We developed "scrolling tabs" to help deal with this issue.
Allowing homepage modules to load as overlays instead of as new pages, which will make navigation faster and smoother.
Making sure that all our logging data is recorded correctly so we can evaluate the impact of the homepage.
The team is largely focused on the release of the mobile version of the newcomer homepage. We are continuously making small fixes as the software is being tested on the Beta Wiki.
About 30% of Czech and Korean Wikipedia accounts are created from mobile devices, so we expect this work to open up the homepage to a substantial number of newcomers.
Anyone can try out the mobile homepage by going to this link on a mobile device.
Growth features becoming available to more wikis
Several features built by the Growth team are now stable enough to be turned on for additional communities who want to experiment with them.
We have assembled this checklist that lays out the translations and configurations needed before the features can be turned on.
Please contact us if you are interested in moving toward turning on the features in your wiki.
Update 2019-06-07: adding Arabic Wikipedia
The Growth team will begin to work with the Arabic Wikipedia community as a new target wiki. This is in addition to Korean, Czech, and Vietnamese Wikipedias.
Wikimania 2019 is coming up in August. The conference will include a "Community Growth" space, for sessions about how our communities expand through software and programs. If you have ideas for sessions, please submit them before the deadline on June 9.
After about a month of usage, we see a few interesting trends that make us believe usage is going well:
About half of users who visit the homepage click on a link or button.
About half of users visit the homepage more than once, with about a fifth of users visiting on multiple days.
Users are interacting with all the different modules on the page -- there is no clear favorite.
Users have been asking questions to their mentors -- but not on the help desk.
The primary work right now is about making the homepage work on mobile, which is expected to increase the number of newcomers seeing it. See the accompanying screenshot for the beta version of the software currently being tested on Beta Wiki. You can also see our notes from live user tests that we did of the mobile homepage here. We expect to deploy this to users in two weeks.
The team is also working on several features to help newcomers discover their homepage. Our goal is for all newcomers to learn that they have a homepage and how to find it.
Update 2019-05-28: takeaways from Hackathon
Session about new editors at Wikimedia Hackathon 2019Members of the Growth team were at Wikimedia Hackathon in Prague from May 17 - 19. One of the three Hackathon focus areas was "new editor experiences", and this was an opportunity to help the participants understand the challenges of new editors so they could work on projects that helped.
Hackathon was also an opportunity to spend time with the members of the Czech community who have been answering incoming questions from newcomers via the help panel and newcomer homepage.
We held two sessions with participants, each of which was attended by about 20 people. We also held a special session for members of the Czech community.
We spoke to members of many different communities and started conversations about the potential for Growth team features to be deployed in their Wikipedias: Hungarian, Basque, Slovak, Ukrainian, Armenian, Tagalog.
Several Hackathon projects directly related to Growth team features or to newcomers in general:
We heard many interesting ideas from attendees as they learned about Growth team work:
We had multiple discussions about the benefits and challenges with live chat for the help panel. We talked about potential hybrid solutions that would solve the problems of volunteer coverage to answer chats, and challenges around moderation. We'll continue to have this conversation on our team.
Maybe after newcomers have their homepage for a while, their mentorship module could ask them if they want to become a mentor.
After creating an account, a user's experience barely changes at all. Some newcomers don't realize their account creation succeeded, and they create a second account.
Perhaps we could use the Welcome Survey to re-survey newcomers after a month or so to learn about their experience so far.
We could make it easy for newcomers to "thank" mentors or help desk responders right from the page itself (instead of going to View History).
The Homepage Impact module can also show crosswiki edits or user contributions to different wiki projects - Translate, Education program etc
Update 2019-05-10: newcomers receiving homepage
Hackathon: many members of the Growth team will be at Wikimedia Hackathon next week, where we will have in-person time with the Czech community and will help participants pursue projects related to the Hackathon focus area of "new editor experiences". Please visit with us if you will be at Hackathon!
Newcomer homepageEmail confirmation screen -- a good moment to bring a newcomer to their homepage (shown in French Wikipedia)
After fixing a couple of small issues, we began the homepage experiment on May 6. Half of all new accounts in Czech and Korean Wikipedias now have access to their homepages on desktop by clicking their username in personal tools.
Because only 15 - 30% of newcomers will discover their homepage this way, we're working on two things to increase discovery.
First, we are working on the mobile version. This past week, we ran five user tests that showed we have a strong design, but that we need to be clearer in explaining the impact and mentorship modules. Full notes from these tests are forthcoming.
Update 2019-05-03: homepage release and help panel experiment results
Members of WMF's Growth team working on designs for a potential "user profile" feature for MediawikiNewcomer homepage
The homepage was deployed in Czech and Korean Wikipedias on May 2, but is not yet being exposed to any newcomers. This weekend, community ambassadors in those Wikipedias will test things out to make sure there are not any major bugs.
The experiment will start on May 6, in which half of new accounts on those Wikipedias will have access to their homepages.
We've published our plan for how we will be tracking and measuring usage of the homepage here.
Vietnamese Wikipedia is still in the process of translating the feature and assembling a list of mentors. We will deploy in that wiki once those elements are ready.
The team is now starting to build the mobile version of the homepage, which is tracked here. The current mockups of the mobile experience can be viewed here.
We published the results of the first activation experiment, meant to detect whether the help panel causes more newcomers to make their first edit. This experiment finds no effect on activation from the help panel. This is disappointing, but it is only one of several ways we will judge the help panel. Other sources of data make us feel like the help panel has a good amount of potential that has not yet been unlocked. We will publish more information in the coming weeks.
The team is currently planning our next set of work for the help panel, which is meant to help more users find answers to their questions.
Week of planning activities
We published a set of notes from a team activity in which we thought about a "user profile" project to accompany the newcomer homepage.
Update 2019-04-26: homepage release next week
Mockups of potential designs for mobile version of homepageNewcomer homepage
The homepage is planned for release on May 2 in Czech, Korean, and Vietnamese Wikipedias.
Once deployed, it will be available on desktop for half of all new accounts in those wikis as part of an A/B test.
After deployment, the next priority will be building a mobile version. See the accompanying mockups for how this design is starting to look, and please comment here with any reactions. We'll be user testing this design in the coming weeks as we begin building it.
Last week, the Growth team had several all-day planning sessions to talk about the future of the help panel, the newcomer homepage, and of the team's work in general.
We will be publishing notes and decisions from those sessions in the coming week.
Our prospective release date for the newcomer homepage has moved to May 2 because of technical deployment schedules.Mockup of a potential layout for the "Your recent questions" capability in the mentorship module.
The final touches of the homepage's UI are now complete and being tested.
We will be attempting to include one additional feature before release: an easy way for newcomers who ask questions to their mentors or to the help desk to navigate back to their responses. See the accompanying mockup, and please post any thoughts or ideas you have to the project's talk page.
Week of planning activities
Next week, the Growth team will be having a series of discussions in which we'll start to lay out the future of our existing features, and to plan for our next year. We'll discuss:
Evaluation of the help panel and options for continued iteration
Prioritizing the next set of features for the newcomer homepage
How our knowledge of newcomers and their needs has changed, and what that means for our longer-term strategy
We will report back on what we learned and discussed in two weeks.
Update 2019-04-05: planning for newcomer homepage release
We have set a prospective release date for the newcomer homepage of April 23. During that week, we plan to deploy the homepage to half of newcomers on Czech, Korean, and Vietnamese Wikipedias.
The user interface of the homepage is now falling into place, with final touches being worked on now. See a screenshot from Beta Wiki on the right (which looks a lot like last week's mockup!)
We are in the process of getting privacy and engineering approval for our instrumentation approach, and testing that approach.
We've finished designing the start module, and have begun to implement it. It's possible to click through its various states in these interactive mockups.Mockup of potential layout for newcomer homepage
We have also decided on a design and layout for the homepage as a whole. See the accompanying image, which shows that the start module has the highest affordance, and also highlights the help module on the righthand side.
We are now working on instrumentation of the homepage so that we'll be able to learn exactly how newcomers use it, and therefore be able to improve. That is under development in T219250.
We currently plan to deploy this feature during the month of April.
Before deploying the help panel, the team decided on a set of quantitative leading indicators that we would evaluate to see whether the feature is behaving in a healthy way. We've published our first evaluation of those numbers here. In summary, the help panel seems to be behaving in a healthy way, with good numbers of newcomers opening and using it. The leading indicators have exposed a couple areas for improvement, which we have worked on by introducing search and by displaying the feature in additional contexts.
Mockup of potential design for start moduleThe biggest takeaway from our live users tests of newcomer homepage mockups is that users are looking for a clear place to get started.
Learning from that takeaway, we reframed the "account completion" module to be a "start here" module that gives clear and concrete ways for newcomers to start learning or editing. It also gives newcomers feedback as they complete tasks, showing that they are done.
The latest mockup of this module is in the accompanying image. The detailed specifications are in this Phabricator task. We welcome any thoughts and reactions from community members on the project talk page.
Next week, we'll begin work on the instrumentation of the homepage so that we'll be able to understand how newcomers do and do not use it.
The experiment comparing versions of the welcome survey in Vietnamese Wikipedia has exposed that the abandonment rate for desktop accounts is substantially higher when those accounts receive the survey.
Curiously, mobile accounts do not have a higher abandonment rate when they receive the survey. The detailed statistics are in this Phabricator task.
We are not sure what is causing this difference in abandonment rate, but we turned off the survey in Vietnamese Wikipedia while we investigate. We want to make sure it is not causing good-faith newcomers to leave the site. It is possible that bot accounts are causing this.
Update 2019-03-15: plans for the coming months
Mockups of potential design for newcomer homepagePlans for the coming months
Our team has laid out our plans for the months of April, May, and June.
We have decided to focus our effort on iterating the help panel and newcomer homepage projects. We have decided not to continue work on the engagement emails project. We think that by focusing on two projects, we will be able to make more progress than by spreading across three.
We have decided what the initial version of the homepage will contain. See here for the details.
We completed five live user tests of newcomer homepage mockups and we are analyzing the results to decide whether our approach needs any changes. The accompanying image is our latest mockup.
We are also in the process of deciding exactly how we will instrument and measure usage of the homepage.
The latest code improves the visuals of the impact module and also adds the beginnings of the help module, which is like a help panel embedded right into the page. See the corresponding image f its current state on the beta wiki.
We are planning out how we will instrument and measure the homepage. It is more complex than our previous work, because a homepage with four modules is like a feature with four mini-features, each of which have different elements we will want to measure.
We have completed the experiment that compares Variation A to Variation C, which were two versions of the survey interface.
Variation A (the minimal interface) outperforms or ties with Variation C (the modern interface) on all measures, including response rate, abandonment rate, and activation rate.
Therefore, we are now showing Variation A to all new users in Czech and Korean Wikipedias, with no further experiments being run in those wikis. The experiment to compare surveys is continuing in Vietnamese Wikipedia.
Update 2019-03-01: first visuals for evolving newcomer homepage
Code has been merged that creates the homepage and adds the first module, the impact module. See the corresponding image of its current state on the beta wiki -- and note that we have not yet spent time on making the user interface look right.
Code is being written to increase "emailability" -- the number of newcomers who it is possible to be email. Changes will be tested to the "Special:CreateAccount" page and to the content of the verification email in our target wikis.
Update 2019-02-22: help panel expansion and newcomer homepage work
Mockup of a potential "mentorship module" in the "newcomer homepage" of MediawikiHelp panel
The help panel will be deployed in Vietnamese likely next week.
We will begin to show the help panel in the reading context on Help and Wikipedia namespace pages.
We now have enough data to evaluate the feature's performance. Metrics will be published in the next two weeks.
We have received some thoughts from community members on this project, and we're hoping for more. Please comment on the talk page!
We have decided to build the homepage as an additional "tab" in the User space, that newcomers will be redirected to when they click on their username.
We are considering ideas for how to increase the number of users that have verified email addresses. See this section to read about and give thoughts on the ideas we're working on.
It looks like Variation A (the simpler design) is getting higher response rates than Variation C (the more sophisticated design), in all wikis and platforms. We will publish more in-depth analysis, but it is likely that Variation A is the version of the survey that we continue to use going forward for all wikis that our team works with.
The early results from Vietnamese Wikipedia show higher abandonment rates than expected (people leaving the site after seeing the survey), and we're investigating those.
Update 2019-02-07: new projects for discussion
New projects
We began the "Personalized first day" project with the welcome survey so that we could gather information about what newcomers are trying to accomplish. The next step is to use that information to create experiences that help the newcomers accomplish their goal -- actually personalizing their first day. We asked for community thoughts in this update, and after discussing with community members and amongst our team, we are now planning two projects as next steps: "engagement emails" and "newcomer homepage".
To doEngagement emails: this project was first discussed positively by community members here back in September 2018, and the team how has bandwidth to pursue it. The idea is that newcomers who leave the wiki don't get encouraged to return to the wiki and edit. We can engage them through emails that send them the specific information they need to be successful -- such as contact from a mentor, the impact of their edits, or task recommendations. Please read over the project page, and comment on its discussion page with any ideas, questions, or concerns. Do you think this is a good idea? Where could we go wrong?
To doNewcomer homepage: we developed the idea for this project after analyzing the data from the welcome survey and EditorJourney datasets. We saw that many newcomers seem to be looking for a place to get started -- a place that collects their past work, options for future work, and ways to learn more. We can build this place, and it can connect to the engagement emails. The content of both could be guided by what newcomers say they need during their welcome survey, and contain things like contact from a mentor, impact of their edits, or task recommendations. Please read over the project page, and comment on its discussion page with any ideas, questions, or concerns. Do you think this is a good idea? Where could we go wrong?
Existing projects
Help panel: about 20% of users who see the help panel open it up. About half of them click on a help link. And about 5% of Czech users and 1% of Korean users ask questions. 20 questions have been asked in Czech Wikipedia, 4 have been asked in Korean Wikipedia. Since many people are trying to find help themselves, and fewer are asking questions, we're going to deploy the ability to search for help, likely next week. We've also published our written plan for measurement and experimentation on the help panel here.
Welcome survey: comparison of Variation A and C continues, in order to determine which gets a higher response rate.
Update 2019-01-24: welcome survey in Vietnamese Wikipedia
Our A/B test of the welcome survey has been analyzed, which shows that receiving Variation A of the survey does not significantly decrease the rate at which newcomers edit.
The welcome survey has been deployed to Vietnamese Wikipedia, with half of newcomers getting Variation A and half getting Variation C.
For the help panel, 9 questions have been asked in Czech Wikipedia, and 0 have been asked in Korean Wikipedia, after about 200 users in each wiki have seen the help panel button, with about 40 users in each wiki opening it up.
We are developing an additional feature for help panel: the ability to search for help content. This feature was appealing to user testers, and would search in the Help and Wikipedia namespaces.
We are continuing to plan upcoming projects, with project pages to be posted in the coming days.
Update 2019-01-18: initial report on EditorJourney
This initial report on what new editors do on their first day is now published on wiki. Please feel free to add comments on questions to the talk page, or to translate it into your language.
Variation C of the welcome survey is now being tested against Variation A in Czech and Korean Wikipedias, to determine which one gets higher response rates.
The help panel has now been deployed for a week, and we are seeing substantial numbers of newcomers opening it and clicking links. Asking questions has been more rare: 4 questions have been asked in Czech, and none have been asked in Korean.
We are beginning to work with a third community: Vietnamese Wikipedia. The EditorJourney schema is now collecting data, and we will be starting the welcome survey there next week.
We are planning new projects to use the data from welcome surveys. We will post initial wiki pages about those projects next week.
Update 2019-01-11: help panel deployed in Czech and Korean Wikipedias
Yesterday, we deployed the help panel to Czech and Korean Wikipedias. This is the Growth team's first feature built specifically to increase new user activation (editing for their first time) and retention (coming back to edit again). Going forward, half of new accounts will see the help panel button in the lower right of their screen when they open any editor on any page -- both on desktop and mobile (but not mobile apps). The other half of users are in the control group of our experiments.
We carefully instrumented the feature so we would be able to tell exactly how users do and don't use it, and we're going to be watching the data closely to see how many people open the panel, click on links, and ask questions. After about 24 hours of the feature being live, about 25 users have seen the help panel, with only one question having been asked so far -- although several users have opened the panel and clicked on links inside. We'll post more detailed numbers next week. To follow along with the help panel activity, feel free to check out these links:
The welcome survey was deployed such that half of newcomers would receive it, with the other half in a control group. This was so that we could detect whether receiving the survey caused newcomers not to edit -- the concern was that putting up another barrier to editing would deter them. We are happy to say that the experiment shows that the survey does not inhibit editing, and we will publish these results in the coming weeks. Because of this, we feel comfortable testing another version of the survey: Variation C. We're going to be showing Variation C to half of newcomers, and the original Variation A to the other half. We want to find out whether Variation C's more dynamic and streamlined interface increases participation. We'll know the initial results in about four weeks.
Update 2019-01-03: initial results from welcome survey
Right now, the team is mostly focused on our upcoming launch of the help panel next week, executing on the plans described in previous updates to test and launch on time. But the most exciting new thing we are looking at and thinking about is the initial report on the welcome survey results.
The welcome survey launched in Czech and Korean Wikipedias on November 19th, and we collected hundreds of responses in each wiki in the following month. Our team had a goal of generating an initial report by the end of December.
The initial report is here, in which we can see the reasons people create accounts, the topic they're interested in editing, the Wikipedia experience they have already had, and their interest in getting help from experienced editors. Here are some toplines, with the rest in the report:
The survey has high response rates: 67% in Czech and 62% in Korean.
Many people create accounts just to read articles -- a potential opportunity to engage new editors.
High numbers of people indicate they are interested in being contacted for help: 36% in Czech and 53% in Korean.
We're still digesting and thinking about these responses, and discussing how to take action on them. We're pleased to see that many people respond to the survey and take it seriously. We're seeing a pretty wide spread of intentions and topics people are interested -- this makes us optimistic that we can successfully personalize the first day.
A high priority analysis that we're working on now is figuring out whether being shown the survey makes a newcomer leave the wiki, instead of continuing on to edit. This is the main concern raised by community members around this idea, and it's the reason we have only been displaying the survey to half of respondents, keeping the other half as a control group. Early results seem like the survey is not decreasing activation (our word for "making a first edit"), and we will publish those results in more depth.
The help panel is still on track to launch next week to Czech and Korean newcomers.
The text has been largely translated on Translate Wiki, and the team is currently testing and fixing bugs.
We measuring using the new HelpPanel EventLogging schema, and we received both engineering and privacy approvals on our plan. We'll be publishing that plan on wiki in the coming weeks.
The data analysis for this project is underway, and we are close to an initial report to publish for community members to see.
Update 2018-12-21: upcoming help panel deployment
As the year wraps up and people head out on vacation, we've been primarily focused on two things: preparing for the initial deployment of the help panel and assembling initial reports on "Understanding first day" and "Personalized first day".
Over the past week, we have been steadily working on the remaining tickets for our initial release. Some of the most recent code to merge includes:
Support for help panel on mobile.
Ability to add an email address or verify an unverified address through the help panel.
Ability to easily turn the help panel off.
Support for the new "HelpPanel" schema, which is what we will use to track and measure interactions with the feature. In January, we'll post our "measurement plan" so that community members can see how we'll be evaluating success.
The help panel is currently available on the beta cluster for any community members who want to try it out. To turn it on, go to the "Editing" section of Preferences and select "Enable the editor help panel".
Our plan is still to deploy this feature to Czech and Korean Wikipedias around January 10, and we'll be giving plenty of notification and recommendations to those communities on how they can be ready to receive and answer incoming questions at their help desks.
Initial reports on existing projects
We're nearing completion on initial drafts of reports for the findings from both "Understanding first day" and "Personalized first day". We're looking forward to hearing what community members think of the results.
Update 2018-12-13: working on help panel and preparing for next quarter
Thanks to everyone who weighed in on the open questions we posted last week. We got some helpful ideas and responses. Please keep discussing! The team has been spending most of our time coding and testing the help panel so that it can be deployed at the beginning of January. We also spent time in the past week planning our next quarter. And in the time remaining this month, we are working to complete initial reports on the "Understanding first day" and "Personalized first day" projects.
We've updated our release date based on when code can be deployed in January. The help panel is now slated to arrive in Czech and Korean Wikipedias on January 10. Our goal is to finish all the code by December 21, so that when team members come back from holidays, all that's left to do is final testing.
We have finalized how we want to instrument the feature so that we can test how users use it, and whether it has a positive impact on activation and retention. We'll be posting a page on wiki that describes our measurement strategy and planned experiments, and a summary is available here.
Planning our next three months
We posted here last week about how we were thinking about our future plans for "Personalized first day", and asked for some community thoughts here.
After some more team discussion, we are currently planning on pursuing the "engagement emails" idea to use email as a way to re-engage editors who haven't made their first edit, or haven't returned to edit again after their first edit. We are also thinking about whether this could go together with some work on a "newcomer homepage" or "landing place" that would help orient newcomers to the resources that could help them accomplish their goals. And in order to maximize the impact of an "engagement emails" project, we are thinking about how to encourage more newcomers to create accounts with email addresses.
We'll post more detailed thoughts on our plans next week.
Initial reports on existing projects
At the beginning of this quarter, we set milestones for our "Understanding first day" and "Personalized first day" projects. Since they're both data collection efforts, we wanted to be able to generate initial reports on the early results from those projects.
We are still on track to create those, and plan to post them by the end of the month so that communities can start thinking about the implications of the analysis.
Update 2018-12-05: Help Panel and Variation C
Now that two of our projects are deployed, we are primarily working on our third project, the help panel. We are also working on "Variation C" of the welcome survey, which we think could increase response rate. Data analysis for both the welcome survey and for the EditorJourney work is beginning now.
In this update, there are a few places where we hope for community members to look over our work and comment. Those spots are marked with a To do.
Engineering continues on this project, with the release in Czech and Korean Wikipedias still planned for January 7.
To do We are currently working from this interactive prototype, which reflects our planned workflow and wording. We encourage community members to click around the prototype and post any thoughts or reactions here.
To do We have also settled on some business rules for who will receive this feature and in what parts of the wiki, described here on the project page. Please comment here with any thoughts on whether those rules are on the right or wrong track.
We are in the process of planning out what we will measure and experiment on with this feature, and planning out the EventLogging schemas for that measurement.
We conducted eight user tests via usertesting.com, and posted the results here. Users generally liked the help panel and understood how it works, both on desktop and mobile.
Our Czech and Korean ambassadors are consulting with their communities to decide which help links are best to include in the help panel.
After deploying Variation A on November 20, we have been steadily collecting data. In terms of data analysis, we only have some very early numbers, and we plan to post more stable numbers during the month of December.
As mentioned in the 2018-11-25 update below, we are also working on Variation C, which is an improved design for the welcome survey, meant to increase response rates. We are currently deciding whether to deploy that change before or after the Christmas holidays, and how to incorporate that into the controlled experiment we're currently running.
To do The "Personalized first day" project originally started because we want to be able to personalize the newcomer's initial experience to help them achieve their goals, and keep them coming back to editing. That's why we deployed the welcome survey: to gather some information from newcomers so that we can present them with content that will help them. Now that the survey is gathering data, the team is spending time thinking about what that next step will be, and our initial thoughts are posted here. We invite community members to join the conversation here.
This project is continuing to run and collect data. No additional work has been done since deploying it and fixing some post-deployment bugs.
Data analysis will being next week.
Update 2018-11-25: two projects deployed
We're excited to have deployed two of this quarter's three projects! The EditorJourney schema is now collecting data on the activities of new users in Czech and Korean Wikipedias, and the Welcome Survey is being shown to new users in both those wikis (but only to half of those new users, because half are in the control group). These deployments are our team's biggest milestones to date, and we're excited to start learning from the incoming data. As that data comes in, we'll be turning our engineering attention to the Help Pane project, and to the next version of the Welcome Survey.
We deployed this on November 20. This was two weeks later than our initial plan of November 6. The main reason for the delay was the translation of the survey privacy statement and the privacy policy into Czech and Korean. The external translation vendor took one week to translate the documents, and then our ambassadors needed an additional week to adjust and correct parts of the translations to use wiki language. For future projects, we will be mindful of finding out in advance all the legal documents that are needed, and planning for the two weeks it takes to finalize strong translations. We also now have a set of improved best practices for working with translation vendors.
The survey is now being shown to half of new users in both Czech and Korean Wikipedias, and responses are being recorded in the database (the other half of new users are the control group in our A/B test, where we are checking to make sure the survey does not depress the rate at which newcomers make edits). In the next week, we'll take an initial look at the data to identify any issues or trends.
There are two next steps for this project:
Variation C: during our design research for this project, we did a comparative review of how other platforms find out basic information about their users. From that, we created an ideal design of how the Welcome Survey would look, which we called "Variation C". We think that design will maximize the number of users who complete the survey and stay on the wiki. But instead of building that design first, we quickly deployed the more simple "Variation A", so that we could start gathering data. Now that Variation A is deployed, we have turned attention to upgrading to Variation C over the course of the next two to three weeks. We plan to A/B test the two variations to see if they cause different response rates or have different effects on whether new users stay on the wiki.
Personalizing: the original objective of this project was to gather some initial information on what new editors are trying to accomplish so that we can give them the help content, task recommendations, or mentorship they need to be successful. We're currently discussing the right first project to work on to take action on the data we're collecting. Some of the options we're thinking about are on the short list below. But each of those things would need a place to "live", and this is making us think about ideas for some kind of "newcomer portal" where newcomers would easily be able to find the tools and information they need. We'll develop this idea a little more, and propose some specifics so that experienced community members can react and tell us what they think.
Making it easy for newcomers to see editing activity around the topic areas in which they indicated that they're interested.
Connecting interested newcomers to experienced editors.
Surfacing the help content most relevant to the reason for which the newcomers created their accounts.
We deployed this on November 15. This was nine days later than our initial plan of November 6. The main reasons for the delay were some additional security reviews done by engineers from outside our team, and on the rigorous testing and adjusting we did before deployment. We're glad that the additional security reviews took place, because it's critical that our software is safe, but we'll know for the future to incorporate our dependencies on other teams into our planning. The rigorous testing and adjusting took longer than expected because of the many idiosyncrasies and edge cases in the Mediawiki software: we kept finding ways that the event data wasn't being recorded correctly. We think we have now solved the vast majority of those cases.
Data is now being recorded for all new users in Czech and Korean Wikipedias, and we've been auditing the data and preparing to make initial reports.
Engineers are now working on building the initial iteration of the help pane, though most of our attention has been focused on completing the roll out of the two other projects listed above.
We are currently planning to deploy this during the week of January 7. We think our engineering work will take up until the holiday season, and we don't want to deploy something new when many engineers and support staff will be taking vacations.
In the coming week, we'll be drafting the written copy that will be part of the feature. This copy is important, because we want to make sure new users understand what will happen when they ask a question in the help pane. That work is happening in this task.
In addition to giving the ability to ask a question, the help pane will also contain a set of links to existing help content. Our ambassadors are determining the right initial set of most helpful links in this task.
We have some open questions around the rules and design choices for this feature. Later this week, we'll post them and start a community discussion to get recommendations from editors who have experience working with newcomers.
Update 2018-11-13: final testing on two projects, work begins for "Editor help pane"
A quick note before the update is that we've given nicknames to our projects, since "Personalized first day", "Understanding first day", and "Focus on help desk" are difficult to say and type. While those phrases still represent the overall goals of the projects, we're now also referring to them on a day-to-day basis as "Welcome survey", "EditorJourney", and "Editor help pane" (respectively), which are more specific to what we'll be deploying in the near term around those three projects.
Current planned release to Korean and Czech Wikipedias: Nov 15
The welcome survey is now deployed to our testing environment where we are making sure that all the questions are laid out as expected, that no words are being cut off, that data is being recorded correctly, and that all the business rules we planned out are being followed. We'll need to test and fix for a couple more days.
The translations of the survey privacy statement and full privacy policy have been returned from our translation vendor, and are currently being edited by our community ambassadors from Czech and Korean Wikipedias to make sure that the vocabulary used makes sense. Posting these is also a blocker for release.
Current planned release to Korean and Czech Wikipedias: Nov 15
Several team members are using Test Wiki and Beta Cluster to make sure that this data is being recorded correctly for both the desktop and mobile experiences and that all the privacy safeguards we put in place are working right. For instance, we're making sure that when users read articles in the main article namespace, the actual title of the article is not saved. Testing EventLogging is generally tedious, because it involves clicking through as many different elements of the software as possible, and then checking the database to make sure the actions were recorded correctly. A few bugs and issues have been found so far and are in the process of being fixed. The specifics can be found on this Phabricator task.
After going through all the ideas that came out of the design process, we decided on what the initial version of this feature would contain. This Phabricator task lists all the subtasks that describe the various design ideas. These two are the ones we'll be doing for the initial version (T206717 and T209318), in the points below.
A button in the lower right for newcomers, giving them the opportunity to "Get help with editing".
Clicking the button opens up the "help pane".
The help pane contains a set of links to the most useful help pages in the wiki.
The help pane also contains an option to "Ask a question", in which newcomers will be able to type a question that will be posted automatically in their wiki's help desk.
We're starting to engineer on those initial requirements and hammering out many small details. Please check out the evolving project page to follow along and see mockups of what this will look like.
Update 2018-11-02: nearing deployment on two projects
For both the "Understanding first day" and "Personalized first day" projects, we are nearing deployment within the next two weeks. Both projects are getting their final approvals and are being tested. Next week, we'll post a detailed update on Czech and Korean Wikipedias explaining what will be changing in the wikis and when.
Since those two projects are on the verge of being deployed and beginning to collect data, our team is going to be turning attention to the "Focus on help desk" project. We'll be making a full project page for that within a couple days so that it is easier for the community to follow along with how we're thinking about it.
We settled on paths forward for the open questions from last week's update:
Topics of interest: since we noticed that WikiProjects differ substantially from wiki to wiki, and since we are not completely sure how we plan to use the new user's topics of interest, we assembled a list of 27 topics that appear across many different lists of important topics, including Czech WikiProjects, Korean WikiProjects, English WikiProjects, categories, and Vital Articles. The hope is that many new users will find find a topic that interests them, and if not, they can type in their own topics. We will analyze the results to decide what to do next. The topics we chose are Arts, Science, Geography, History, Music, Sports, Literature, Religion, Popular culture, Entertainment, Food and drink, Biography, Military, Economics, Technology, Film, Philosophy, Business, Politics, Government, Engineering, Crafts and hobbies, Games, Health, Social science, Transportation, and Education.
Interest in mentorship contact: we decided on phrasing the question such that it's clear that we do not yet have an active mentorship program. This question is now phrased as, "We are considering starting a program for more experienced editors to help newer users with editing. Are you interested in being contacted to get help with editing?"
We incorporated results from human testing that occurred through usertesting.com. Details on this will be added to the project page.
The finalized survey copy is now in translatewiki.net, and is already translated into Czech and Korean (as well as many other languages).
We have finalized a survey privacy statement for the survey, and we are in the process of getting it and our full privacy policy translated into Czech and Korean. This will help new users understand how their responses to the survey will be used.
We have decided on some of the details for the A/B test we'll be running with the feature. One of the community's main concerns with this feature is the potential that a longer sign-up process would cause new uses to leave before editing. Our experiment will give the survey to some new users, and leave the experience of others unchanged. Essentially, if they continue to edit at the same rates, than we can conclude that the survey is not causing users to leave before editing.
Up next, we'll be deploying the survey to our testing environments and checking to make sure everything works as expected.
We've completed privacy review, and we are confident that we'll be handling user data in a way that helps our team learn while respecting user privacy.
We are finalizing some of the engineering and security aspects of this work, after getting review from several engineers around WMF.
We are now testing that data is being recorded and obfuscated correctly by using our test environments.
We expect to be able to start engineering on this project next week.
Update 2018-10-25: new project pages and open questions
Over the last week, we have made a lot of progress on the "Understanding first day" and "Personalized first day" projects. We currently hope that we can deploy both of those projects to some extent during the week of November 6th, but there are still many details to resolve in the next two weeks. We are hoping for any community wisdom to help us resolve some of the open questions laid out below. To help make it easier to follow along with our work, we now have full project pages for each of those two projects:
Those pages contain things like designs, decisions, and implementation details. The project page for "Focus on help desk" is coming next.
Below are some updates on the two projects above.
Personalized first day
Our team has been iterating on the exact wording for the text of the questions and responses in the survey. The mockups now reflect more up-to-date question wording, but we're still finalizing it. We've asked some survey experts to weigh in, and we have also engaged some human testers to try out the surveys and give feedback. We have a couple of open questions that we're not sure how to resolve yet:
Topics of interest: we want to ask new users about the topics they're interested in editing. That's because one day we may want to recommend editing tasks that fit their interests, direct them to WikiProjects about their interests, or connect them with other editors who share their interests. We are trying to figure out what topics to offer as choices. One idea is to use WikiProjects, but we know that the majority of WikiProjects tend to be inactive. We are also concerned that presenting very general topics -- "Art", "Sports", or "Geography" --are not specific enough to align with someone's actual interests, which may be "Ceramics", "FC Barcelona", or "Sahara desert". We're still gathering ideas and experimenting with the right way to do this.
Interest in mentorship contact: one of the questions on the survey asks whether the user is interested in being contacted by an experienced editor for help with their editing. This is important because it will help us gauge whether many new users are interested in mentorship, which might encourage our team to develop ideas around promoting mentorship on-wiki. We are trying to figure out a good way to ask that question without making new users wait to receive contact that won't be coming immediately, since we haven't yet developed the way to actually connect them with experienced editors. We don't want our questions to be confusing or misleading.
We are also in the process of making sure that we have thought through any data privacy elements with the survey so that users can feel good about answering the questions.
Understanding first day
We have largely settled on exactly what data we will be collecting and the questions we plan to answer with it. The project page describes those questions, and the exact technical specifications for measurement can be found here on Meta Wiki, with the associated business rules on its talk page. In short, we'll be recording the pages that new users in Czech and Korean Wikipedias visit during the first 24 hours after creating their accounts -- but only pages in non-sensitive namespaces that are important for our work, such as the Help, Wikipedia, Special, and User namespaces. For pages in the Article, File, Portal, and Draft namespaces, however, we will only record that a visit to that namespace occurred -- not the specific page. The identifying data will be deleted or aggregated after 90 days.
This new data will allow us to see the path new users take as they learn and prepare to edit. We're also going to connect the new data to existing measurement efforts in the visual and wikitext editors so we can see how the new user's path leads them to succeed or not succeed with their first edits.
Most of the technical work is complete, but we are still in the process of security, engineering, and privacy checks on our work. We're testing out the instrumentation in testing environments, and making sure that the other measurement efforts we want to use have the data we'll need.
Update 2018-10-18: starting three projects
Our team has set goals for what we'll accomplish by the end of the quarter (we are currently in 2018 Q2, which goes from October 2018 to December 2018.) Our goals can be found with the rest of the Audiences department's goals here. In brief, our goals for this quarter are:
Collect initial "Understanding first day" data from instrumentation, and post early numbers on wiki.
Collect initial "Personalized first day" data from user input, and post early numbers on wiki.
Deploy a "Focus on help desk" intervention experiment in production.
Develop a relationship with a third community beyond Czech Wikipedia and Korean Wikipedia.
Since choosing those three priority projects at the beginning of October, the Growth team has started work on each one, with most of our activity taking place in Phabricator under an "epic" task for each project:
Over the last two weeks, the team has primarily been working on "Understanding first day" and "Personalized first day".
For "Understanding first day", our engineers and data analyst have been developing the specifications for an EventLogging schema to record useful information on new editor journeys while respecting their data privacy. Our team is close to posting the specifications on wiki and in Phabricator, and it will be summarized in an update here.
For "Personalized first day", we have designed initial concepts for how this set of questions will be asked to the new account holders, and the engineers are beginning to prototype them. It's possible to explore those mockups here -- but it is important to note that the specific language in the mockups has not yet been worked on. The language in those mockups is placeholder language for when we write the finalized words that will be translated into Korean and Czech. Writing those words is our current highest priority task for this project.
Update 2018-10-02: priorities for October through December
Background
The Growth team is very appreciative of the time that community members took to respond to our ideas for increasing new editor retention. We received helpful comments and opinions from around the world, both from newcomers and experienced editors, that have allowed us to put together our plan for the next several months of work. Here is how we have proceeded (along with relevant links):
We developed an understanding of new editor experiences and challenges through the New Editor Experiences project.
We put together a list of ideas for how to increase new editor retention.
The Growth team made the following development plan that takes into account community thoughts and several other factors.
Theme
The theme of our work for the coming months is “Day One”. That means we are focusing on the experience of new editors on the day that they create their account. We’re focusing on day one because our data shows that about two thirds of people who create accounts never make an edit, and of those who do, about two thirds of them never come back to edit again. This makes us think that the first day of a new editor’s experience is an important time to help them start editing and want to continue editing. We will go into more detail on the data behind this in another update.
Plan
With the theme of “Day One” in mind, we decided to prioritize the following three ideas for Czech and Korean Wikipedias. We are not building any of them fully. Instead, we will build a small piece of each so that we can learn whether the idea has potential, and quickly decide whether to continue. We’ll be working on them in October, November, and December 2018.
Understanding first day: we are prioritizing this because it will help us learn a lot about new editors. It will not change any user’s experience. We’ll simply add logging so that we can see what new editors do right after creating their accounts. We will be careful with user privacy, and we hope to share initial results in December.
Personalized first day: this idea will also help us learn a lot about new editors. To start, we will add some optional questions to the new editor’s registration process. This will help us learn what new editors are trying to accomplish, and by looking at the data, and with the help of the communities, we’ll be able to think about how to personalize a new editor’s experience so that they can accomplish what they’re trying to do. We hope to share initial results in December.
Focus on help desk: we are prioritizing this because it was shown to work in other contexts. We will likely direct traffic to existing Help Desks through a bot that invites new editors on their talk pages, or through a new button in the editing experience inviting new editors to “Ask a question”. That button would be an opportunity to learn whether the “In-context questions or chat” idea has potential. We hope to have an initial experiment running in December.
We decided to prioritize those ideas for these reasons:
They have been proven to work in Wikipedia and non-Wikipedia contexts.
They help us learn quickly.
They can be deployed in small pieces without a major commitment at the beginning.
They can be deployed relatively easily to other communities.
Next steps
Next, we will be planning the work for each of those three ideas and beginning to write code. We’ll continue to post updates, and we will be clear when we need help from community members. We have already started to create Phabricator tasks for the "Understanding first day" work. For the “Focus on help desk” project, we will need experienced editors to dedicate time to answering questions in the Help Desk, because we will expect many more new editors to be asking questions. We’re looking forward to continuing to work with the Czech and Korean communities, and we are looking for additional communities to begin to work with.
Update 2018-09-21: summaries of community discussion and planning sessions
Over the past few weeks, we received a tremendous amount of useful thoughts and feedback on our list of eight ideas. We're grateful to the members of the many wiki communities (including several mid-size wikis) who responded on the talk page, on their own wikis, or sent us their thoughts through other channels. We were also able to get feedback on the ideas from actual new editors in Czech and Korean Wikipedias, who provided a first-hand perspective on which ideas would have helped them as they were starting out. Summaries of the feedback we received on each idea are now posted, along with some follow-up questions on the talk page.
To take action on this feedback and plan the coming months, the Growth team came together in person last week. We did two main things: short-term planning and long-term thinking.
Short-term planning
A poster made by the Growth team when discussing a potential idea for increasing new editor retention.
To determine what our team will work on first, we spent a day processing the community feedback and reflecting on the eight ideas. We split up into groups, each of which was assigned two of the ideas. For each of their two ideas, those groups read through and discussed community comments in detail. They then summarized the comments into a poster to explain the nuanced perspectives to the rest of the team. This format helped us discuss each idea, making sure to take into account the pro's and con's from different communities. Posted here is an example of one of the posters we made, to give a sense of our process. To read summaries of what we learned from communities, see this page.
Long-term thinking
The Growth team intends to be working on new editor retention for many quarters, and we know that we will need to be continuously learning about new editors in order to be successful. Though we already know a lot about new editors from research done in the past year, there are still many open questions about how new editors work and what they need to be successful. We spent time determining which questions are most important so that our team can use upcoming opportunities to find answers. Please visit this page to see our most important open questions and read about the exercise we did to generate them.
Next steps
We are now in the process of deciding exactly which things to build and in what order, taking into consideration engineering challenges, measurement challenges, and how to work with the Czech and Korean communities. Our next update will contain our plan for the coming months.
Update 2018-09-04: final week of community discussion
Over the past two weeks, we have posted the list of ideas for discussion in many places, and we would like to gather all the feedback we can by the end of this week from as many different communities as possible. If you have planned on weighing in, please do so in the next few days!
Next week, the Growth team will be taking a substantial amount of time to process and digest all the thoughts and reactions that have been posted, and we will make a plan for which ideas make the most sense to pursue, and we'll post that here. In making that plan, we'll take into account community thoughts, technical considerations, and the burden that new development will put on existing communities.
Here are the places we have posted the list of ideas for comment:
It has been linked on many Help Desk talk pages in different wikis.
It has been sent to community members, newsletters, and Facebook groups in the Czech and Korean communities.
Update 2018-08-22: community discussion begins on ideas
We've now posted a list of the main ideas we're considering for the team to work on first! We hope as many people from as many communities as possible participate in the discussion on that talk page. The list of ideas is based on a lot of research and conversations the team has been taking in, but they are by no means solidified -- that's why this community discussion is really important to us. We're hoping to hear the majority of the community thoughts within the next two weeks (by 2018-09-05).
The list is also being translated in Czech and Korean and discussed on those wikis, since those are the places where the features would be deployed first. If you are interested in translating all or part of the list for your own wiki, please let us know! It would be great to have additional languages discussing.
We'll also be sending out our first team newsletter about this. Please sign up here if you are interested in receiving the newsletter.
Update 2018-08-14: narrowing down on ideas
The team is now spending an increasing amount of time planning for this first project, even as our engineers work on the Articles for Creation and New Pages Feed project for English Wikipedia. A few short updates are below, with more extensive information coming over the next week:
Our top priority right now is to start the conversations around which feature ideas we should pursue first. The team has been compiling many ideas for what we can build and has started to narrow to the ones that seem to have potential. We're going to be asking the Czech and Korean communities to weigh in, and we will also post them here and in other venues so that as many communities as possible can add their thoughts. We'll be looking for new ideas, thoughts on existing ideas, and information on when similar things have already been attempted. Below are some examples of the types of ideas we're talking about. More detail will be posted here in the coming weeks, as well as the reasons we think these ideas could be impactful.
Inviting new editors to help desks. Most wikis have a help desk, but most newbies don't know about it.
Making it possible for newbies to ask questions in the context where they are editing, instead of having to go to a different page.
Communicating a new editor's impact to them via email, perhaps by telling them about the page views on pages they've edited.
Gathering optional information from new editors when they create accounts about what they are trying to do and the topics in which they are interested, so we can direct them to information that serves their needs right away.
Instrumenting the new editor experience to learn more about what new editors do right after creating their accounts -- whether they read documentation, attempt to edit but don't succeed, or something else.
We have completed the staffing of our team by adding Rita Ho, our user experience designer, and Morten Warncke-Wang, our product analyst. Rita joins us from the team that built the Wikipedia Android app, and has important expertise on mobile usage that will be relevant for new editors. Morten is a longtime researcher of Wikimedia projects and has deep experience understanding the issues newbies face and experience working with Mediawiki data. Morten is going to be working on the "analyzing data" work mentioned in the update from 2018-07-06 below.
Several of our team members attended Wikimania 2018 and had great discussions with editors from many mid-size Wikipedias, including Arabic, Bengali, Serbian, Ukrainian, and others. We gathered information on the experience of new editors in those wikis, what is the same and what is different across them, and what specific tactics and features are working well to retain new editors. We'll be assembling and posting what we learned, along with the work our Czech and Korean ambassadors have already done on that front to catalog existing help content in their wikis.
Analyzing data about contributor behavior in Czech and Korean Wikipedias. We are setting up the queries and reports that will allow us to track the contributor retention rates in our target wikis, and to understand exactly where in the editing process new contributors are most likely to drop off. We will post this data on wiki as soon as we're able.
Partnering with our ambassadors in the Czech and Korean communities to document the current resources that help new contributors in those wikis. We want to make sure that anything we build compliments the existing pages and tutorials that already exist, and that we are not duplicating efforts with any community members. This is the related Phabricator task.
Summarizing existing ideas for features that could increase retention. Many WMF staff and community members have attempted and proposed ideas for new editor retention in the past. We're making sure to collect those ideas and use them as a starting point. We will post that information on wiki as soon as we're able.
We have also spent time this week establishing the processes for our new team, such as when we have meetings, how our Phabricator board works, and how we'll balance new work with maintenance on previous features built by engineers on the team. Those previous features include Notifications and the new filters on the Recent Changes and Watchlist pages.
Mockup of Variant C on mobile
Mockup of Variant C on desktop
Mockup of Variant D on desktop
_of_newcomer_homepage_2020-04-09.png)
_of_newcomer_homepage_2020-04-09.png)
To do Engagement emails: this project was first discussed positively by community members here back in September 2018, and the team how has bandwidth to pursue it. The idea is that newcomers who leave the wiki don't get encouraged to return to the wiki and edit. We can engage them through emails that send them the specific information they need to be successful -- such as contact from a mentor, the impact of their edits, or task recommendations. Please read over the project page, and comment on its discussion page with any ideas, questions, or concerns. Do you think this is a good idea? Where could we go wrong?
To do.
.jpg)