Community Wishlist Survey/Prioritization

This article is written for wish enthusiasts who want to understand how the Community Tech Team strategizes completing wishes after voting ends. We hope to de-mystify the process of software development for those unfamiliar with engineering and design. We welcome feedback about the clarity of this artifact.

We've developed a method to help us approach our wish prioritization more systematically and with transparency over the years. There were a few assumptions built into our prioritization process which are helpful to name explicitly:

• Popularity of a wish should be a very important factor in our selection decision, but not the only one.
• It is best to stagger wishes so that specialists can collaborate with each other as we progress through work-- i.e. as the designer researches the wish and generates visual components for wish, the engineers focus on a wish that is purely technical.
• It is best to communicate transparently with the communities rather than hiding the details. Visibility builds trust and dialogue.

Summary of Criteria

The process consists of going through any wish that scores in the top 30 for a wishlist (we cut off any wishes below that, because realistically, it takes time to investigate every wish and we know we will not be able to grant more wishes for a given year) and scores them based on the following criteria:

Prioritization Score for Community Tech Proposals

Once every wish is scored from every vantage point that impacts its feasibility and impact, we rank them. If we tackle those wishes first, we can tackle most wishes. Also, we can optimize for impact while taking maintenance and complexity into account.

This also means talking to other teams at the Foundation, and investigating if they were already working on projects related to wishes.

Technical Complexity

1 - Lowest Complexity

• Technical solution is very simple - the problem exists in a contained part of the user experience as well the codebase
• Solution might already exist, developed by a community member in the form of a pre-existing gadget, extension, or code in a public repository
• Members of the engineering Community Tech team are familiar with the code
• Light QA testing required, just 1 task worth of QA

2 - Low Medium Complexity

• Technical solution is discrete- the problem exists in a contained part of the user experience as well the codebase
• Solution might already exist, developed by a community member in the form of a pre-existing gadget, extension, or code in a public repository
• Members of the engineering Community Tech team are familiar with the code
• Almost no maintenance required
• Minimal code refactoring is required
• Possible third party code dependencies
• Light QA testing required, less than 5 tasks worth of QA

3 - Medium Complexity

• Technical solution is open-ended-- the problem exists in multiple parts of the user experience as well as one or multiple parts of the codebase or repositories
• Partial or no solution exists
• Members of the Community Tech team have limited knowledge of or are unfamiliar with the code
• A bit of maintenance required
• Code refactoring might be required
• Potentially adding third party dependencies
• QA testing required prior to release, 5+ tasks worth of QA

4 - Medium Large Complexity

• Technical solution is open-ended-- the problem exists in multiple parts of the user experience as well as one or multiple parts of the codebase or repositories
• Solution hasn’t been implemented
• Members of the Community Tech team have limited knowledge of or are unfamiliar with the code
• Maintenance is required
• Some database schema changes may be required
• Code refactoring is required
• Changes to authentication/security components are required i.e. authentication, feature flags, access controls
• Potentially adding third party dependencies
• QA testing required prior to release, 5+ tasks worth of QA

5 - Large Complexity

• The technical solution has unknowns-- the problem exists in multiple parts of the user experience as well as one or multiple parts of the codebase or repositories
• A system or new tool may need to be developed
• Members of the Community Tech team are unfamiliar with the code
• Maintenance required
• Some database schema changes may be required
• Code refactoring is required
• Changes to authentication/security components are required i.e. authentication, feature flags, access controls
• Potentially adding third party dependencies
• QA testing required prior to release, 5+ tasks worth of QA

6 - Extra large Complexity

• The technical solution has many unknowns-- the problem exists in multiple parts of the user experience as well as one or multiple parts of the codebase or repositories
• A system or new tool may need to be developed
• Members of the Community Tech team are unfamiliar with the codebase the wish pertains to
• Maintenance required
• Substantial code refactoring is required
• Difficult database schema changes may be required
• Substantial code refactoring is required
• Changes to authentication/security components are required i.e. authentication, feature flags, access controls
• Add third party code dependencies
• QA testing required prior to release, 10+ tasks worth of QA

Product and Design Complexity

1 - Lowest Complexity

• Design Solution is embedded into the wish proposal itself-- it’s a technical fix and no UI changes are necessary
• No data collection necessary
• No discovery user survey collection
• No unmoderated user research
• No design

2 - Low Medium Complexity

• Changes are isolated to just a single page inside of the experience with limited number of states (i.e. changes only impact one page / one wikimedia project)
• Requires little to no initial data collection to understand behavior and pain point via survey or quantitative data
• Requires little to no unmoderated research
• Prior to tackling the wish, we already collected the data necessary to make informed product & design decisions

3 - Medium Complexity

• Prior to tackling the wish, we already collect most* of the data to make informed product & design decisions but may require tracking new data prior to starting to understand the problem
• Requires unmoderated user research but it is not difficult to “source” users for those flows
• May touch more than one page in the experience but it is generally limited to a subset of the experience and straightforward
• Limited to designing for one type of user need

4 - Medium Large Complexity

• Prior to tackling the wish, we already collect some* of the data to make informed product & design decisions but may require tracking new data prior to starting to understand the problem
• Requires unmoderated user research but it is not difficult to “source” users for those flows
• May touch more than one page in the experience but it is generally limited to a subset of the experience and straightforward
• Requires a survey at the beginning of wish
• Limited to designing for two types of user needs
• Touches more than one page in the experience but it is generally limited to a subset of the experience and straightforward

5 - Large Complexity Requires qualitative discovery and quant data collection

• Requires unmoderated user research and the users for the research are hard to source given the complexity of wish
• Can require designing new technical information into the UI
• Requires touching multiple pages in the flow
• Requires a survey at the beginning of wish
• Requires touching multiple pages in the flow and or has cross-project implications
• Impacts across multiple user states, for example
  • Editors
  • Readers
  • Proofreaders etc

6 - Extra Large Complexity

• Requires investigation by the process of qualitative discovery and quant data collection
• Potentially controversial implications that must be mitigated by working with communities
• Requires unmoderated user research and the users for the research are hard to source given the complexity of designs
• Requires designing for a “learning curve” or introducing new technical information into the UI
• Requires touching multiple pages in the flow and or has cross-project implications
• Impacts across multiple user states and across needs:
• Editors
• Readers
• Contributors
• Newcomers

Key Terminology

Unmoderated user research

Using a tool like UserTesting.com to run “mocks” of our proposed design changes and see if we are designing the right wish solution-- it’s called “unmoderated” because we let users click around and see our designs makes sense without having to explain it to them

Quantitative data collection

The process of collecting data to understand how users are interacting with the current UI to understand the wish’s pain points -- be it data regarding clicks, visits, downloads, sessions etc. Data is often limited when we first tackle a wish due to lack of tracking it prior to wish, or nonexistent data due to privacy reasons

Qualitative data collection

Understanding the wish’s problem space by talking directly to users, be it interviews or via a survey at the beginning of the wish to understand the pain points and clarify how to tackle a solution

“Sourcing” users

The process of finding users who have the knowledge required to participate in our user tests and give us the information we need to understand if our design and product decisions are headed in the right direction. Some wishes are for advanced users, which are hard to source and not available in tools like UserTesting.com

Code refactoring

The process of making the existing code more maintainable so that other people may contribute to the code, as well as removing technical debt and bugs.

Database schema changes

The alteration to a collection of logical structures of all or part of a relational database. When a change to an existing database is needed, it must be designed and then approved by a team external to CommTech. This usually takes more time and adds structural complexity to the project.

Third party code

Code written outside of the Community Tech team, examples include APIs or libraries.