User story:

As the Growth team, I want to be sure we consider instrumentation & measurement for every feature we release, because we only want to scale out features that improve new editor onboarding and retention.

Parent task: T360485: [EPIC] Newcomer homepage: Community updates module (FY23/24 WE1.3 / FY24/25 SDS2.1.3)

SDS 2.1.3: If the Growth team learns about the Metrics Platform by instrumenting a Homepage Module on the Metrics Platform, then we will be prepared to outline a measurement plan in Q1 and complete an A/B test on the new Metrics platform by the end of Q2.

Designs:

Community updates: Figma designs for Homepage module.

Community updates: Figma designs for the configuration form.

Key Question:

Does this module encourage more newcomers to participate in community events / campaigns / projects?

What we will measure as part of the A/B test:

• Awareness: Total individual page views; how many account holders visited their Homepage while this feature was visible. (SDS 2.2.3: Clickthrough rate)

• Consideration: Total Click Through Rate (SDS 2.2.3: Clickthrough rate)

• Conversion: After click through, registration for event / program; Do more newcomers sign up for the WikiProject or Campaign when they are exposed to the Community Updates CTA on the Homepage? (SDS 2.2.2: conversion metric)

What we would like to measure, but may not be able to based on current limitations:

If possible, we should also measure the following metrics:

• Constructive Activation: Are newly registered account holders who visited the Newcomer homepage and had access to this new feature more or less likely to activate?

• Completion rate for Suggested Edits: Are newly registered account holders who visited the Newcomer homepage and had access to this new feature more or less likely to complete Suggested Edits?

• Completion rate for asking a question to a Mentor: Are newly registered account holders who visited the Newcomer homepage and had access to this new feature more or less likely to connect with their Mentor?

• Editor Retention: Are account holders who saw this new feature more or less likely to be retained?

Measurement & Instrumentation details:

From the measurement specification.

Similar to other home page modules the following events will be captured:

• Impressions
  • For these events we will capture the current update with a friendly name
• Clicks on the call to action
  • For these events we will capture the current update with a friendly name

Acceptance Criteria:

TBD