Last Updated At

SPLIT EXPERIMENTS REPORT

May 30, 2023, 12:16:27 PM

Experiment homepage_rebranding

1 Select Experiment 2.1 Select Split Version to see results 3 Unselect Split Version to start over

Experiment (1) Created At Split Version Visitors Treatment Visitors Environment Production

Type to search May 12, 2023, 4:31:54 AM 24,535 on 435,842 Targeting ▼

search_ai May 25, 2023, 2:45 PM May 1, 2023, 11:38:55 AM 30,556 off 1,147,039 Visit Source ▼

Apr 13, 2023, 6:47:59 AM 30,387 control 28 Device Type ▼

fasttrack_experiment_no_limit May 25, 2023, 7:20 AM

Market ▼

app_ldp_rebrand May 10, 2023, 5:13 AM Apr 12, 2023, 11:14:15 PM 21,651

Segment ▼

AI_Search_Quick_Filter May 9, 2023, 10:12 PM Apr 11, 2023, 2:21:00 AM 136,191

Product ▼

* Refer to Split to know about the definition of treatments and targeting rules of the experiment

Account
Treatment Visitors Signup Signup SQLs SQLs (% of LS 60+ LS 60+ (% of Account Created Activations Activation Toured Toured Leads Mutualed Mutualed
▼
s Rate Leads) Leads) Created Rate Leads Rate Rate
Rate
on 435,842 1,495 0.3% 322 21.5% 394 26.4% 996 66.6% 41 2.74% 9 0.6% 1 0.07%

off 1,147,0… 3,385 0.3% 726 21.4% 826 24.4% 2,135 63.1% 71 2.1% 30 0.89% 6 0.18%

control 28 2 7.1% 1 50.0% 1 50.0% 2 100.0% 0 0% 0 0% 0 0%

Grand total 1,581,… 4,873 0.3% 1,045 21.4% 1,217 25.0% 3,127 64.2% 112 2.3% 39 0.8% 7 0.14%

Qualifying Call Qualifying Call FA Qualifying FA Qualifying Call Onboarding Onboarding Activations 2 Weeks Activation Option Price Option Price
Treatment ▼
Leads Call Booked
Booked Booked Rate Call Booked Booked Rate Call Booked Rate Within 2 Weeks Rate Approved Approved Rate

on 1,495 100 6.69% 15 1% 67 4.48% 39 2.61% 8 0.54%

off 3,385 158 4.67% 57 1.68% 109 3.22% 62 1.83% 7 0.21%

control 2 0 0% 0 0% 0 0% 0 0% 0 0%

Grand total 4,873 257 5.27% 72 1.48% 176 3.61% 101 2.07% 15 0.31%

Comparison Signup Rate Z Score Signup Account Z Score Account Activation Rate Z Score Toured Leads Z Score Toured Mutualed Rate Z Score
Created Rate
Mode ▼
Change Rate Change Created Rate Change Activation Rate Rate Change Leads Rate Change Mutualed Rate

on / off -10.27% -3.04 -0.73% -0.27 -31.24% -1.34 -55.38% -0.92 -100% -0.34

off / on 11.45% 3.04 0.73% 0.27 45.43% 1.34 124.12% 0.67 - 0

2.2 Select Comparison Mode to check Statistical Significance

Comparison Mode
VISITORS on / off on / off means metrics for ON treatment are compared against the baseline treatment OFF

27.5% off / on
Consider Positive (+) Z score when expecting Increase in the Metric

on Z Score > 1.65 - Statistically Significant at 90 % Confidence Interval

off
control

72.5% Signup Rate Change Account Created Rate Change

Z SCORE Z SCORE

0.00% 0 0.00% 0

Activation Rate Change 2 Weeks Activation Rate Change Toured Leads Rate Change Mutualed Rate Change
Z SCORE Z SCORE Z SCORE Z SCORE

0.00% 0 0.00% 0 0.00% 0 0.00% 0

Qualifying Call Booked Rate Change FA Qualifying Call Booked Rate Change Onboarding Meeting Booked Rate Change Option Price Approved Rate Change
Z SCORE Z SCORE Z SCORE Z SCORE
0.00% 0 0.00% 0 0.00% 0 0.00% 0

* Market, Segment & Product breakdown by filters do not apply to Significance test
results

Z Score describes the relationship between the observed score of a metric after a change and the original score of the metric.

When plotted on a Normal Distribution Curve, it represents the standard deviation of the observed value of metric for the sample
from the mean, zero, of the observed value of the metric for the entire population.

For 95 % Confidence interval, the observed change in the value of metric is considered Statistically Significant if the P Value is beyond
0.05, Alpha/Significant threshold, from the mean, where Z Score ~ 1.96

For 90 % Confidence interval, Alpha ~ 0.1 and for 99 % Confidence interval, Alpha ~ 0.01.

p1 - Sample 1 Proportion (Treatment)

p2 - Sample 2 Proportion (Baseline Treatment)

p - Sample Mean

n1 - Volume of Sample 1
NORMAL DISTRIBUTION CURVE
n2 - Volume of Sample 2

2.3 Search Lead Details by Email IDy Email ID Contains ▼ Enter a value

FA
Qualifying Onboardin Option
Treatmen Email ID Lead Account Call Qualifying g Meeting On Price First Tour First
t User ID Segment Market Product SQL Created Created Call Boarded Mutual
▼ ▼
Date Date Booked Booked Booked Date Approved Date Date
Date Date Date
Date
on zxc3589126 478887 Mandarin South Buy No May 20, 2023
@gmail.com California

on zsn1985@g 480904 English Sacramento Buy No May 6, 2023

mail.com

on zooknina@g 469762 English North Buy Yes Apr 4, 2023 Apr 4, 2023
mail.com California

on zollpang101 484006 Mandarin Denver Buy No May 16, 2023 May 16, 2023
[email protected]
m

on zoemadison 467781 English Other Buy No Mar 29, 2023

247@icloud.
com

on zichenglli@ic 475840 Mandarin Other Buy No Apr 20, 2023 Apr 20, 2023
loud.coom

on zhuxi31@ya 485121 English North BBYS No May 20, 2023

hoo.com California

on zhouyangsb 481564 Mandarin Other Sell No May 8, 2023 May 8, 2023

[email protected]
om

on zhongrunjie2 481784 Mandarin Other Buy No May 9, 2023 May 9, 2023

021888@gm
ail.com

on zhiqi.yang@c 468955 Mandarin San Diego Buy No Apr 2, 2023 Apr 2, 2023
mcc.it

on zhan320722 485878 Mandarin South Buy No May 22, 2023 May 22, 2023
[email protected] California
m

1 - 100 / 4883
< >

Notes

1. All visitors, including signups, are considered to obtain the metrics.

2. Z score is used to determine whether the result obtained is due to chance or any reason.
3. Significance Testing is done at a confidence interval of 90 %.
4. As per the Normal Distribution Curve, z score greater than 1.65 (p value < 0.1) means that the impact is significant.

Visit Source / Device Type : First Page load event in the interval of 30 minutes prior to the Split event is considered, as Split does not send the visitor source/device info

1. Visit Source :
i. Paid - UTM Source is not null & UTM Medium - Search; CPC; PaidSocial; Display; Video
ii. Organic - UTM Source is null & Referrer - Google; Bing; DuckDuckGo; Yahoo; Baidu; Ecosia; AOL; Sogou; Qwant; So.com; Naver; Yandex
iii. Direct - UTM Source is null & Referrer is null
iv. Other - Rest of the source which do not meet Paid, Organic & Direct conditions

2. Visit Source & Device Type is Null where Events are not getting tracking
 SPLIT EXPERIMENTS REPORT
Last Updated At
Choose Your Own Adventure
May 30, 2023, 12:16:27 PM

Experiment ff_cyoad

1 Select Experiment 2.1 Select Split Version to see results 3 Unselect Split Version to start over

Experiment Created At Split Version Visitors Treatment Visitors Environment Production

Type to search May 3, 2023, 7:36:38 AM 1,339 on 901 Targeting: default rule (1) ▼

ff_cyoad Apr 18, 2023, 2:31 AM Apr 18, 2023, 2:31:23 AM 487 off 916 Visit Source ▼

Device Type ▼

Market ▼

Segment ▼

Product ▼

* Refer to Split to know about the definition of treatments and targeting rules of the experiment

* Experiment is triggered once a visitor completes the Buy Where slide

Leads
Leads Activation Leads Onboarded Leads Leads
Treatmen Signup Leads Connection Activation Activation Booked Leads Leads
t ▼
Visitors Signups Rate Leads Connected Rate Activate Rate s Toured to Tour Onboarding to Meeting Marked Marked Toured Tour Rate
d Rate Booked Lost Lost Rate
Meeting Rate

on 901 209 23.2% 205 52 25.37% 12 5.9% 3 25.0% 18 8.8% 145 70.7% 3 1.46%

off 916 206 22.5% 205 46 22.44% 4 2.0% 1 25.0% 11 5.4% 143 69.8% 1 0.49%

Grand to… 1,817 415 22.8% 410 98 23.9% 16 3.9% 4 25.0% 29 7.1% 288 70.2% 4 0.98%

* Visitors include already Signed up leads

* Signups includes leads created from all CTA's on the website post experiment trigger

Lead Z Score Lead Z Score Lead Z Score Lead Ob Z Score Lead Leads Z Score Lead
Comparison Signup Rate Z Score Activations Meeting
Mode ▼
Change Signup Rate Connection Connection Activation Activation Toured Rate Activations Booked Rate Ob Meeting Marked Lost Marked Lost
Rate Change Rate Rate Change Rate Toured Rate Booked Rate Rate Change Rate
Change Change
on / off 5.62% 0.62 10.8% 0.58 194.06% 2 194.06% 0.98 60.4% 1.29 1.55% 0.24

off / on -5.32% -0.62 -9.75% -0.58 -65.99% -2 -65.99% -0.98 -37.65% -1.29 -1.52% -0.24

VISITORS 2.2 Comparison Mode to check Statistical Significance

on / off means metrics for ON treatment are compared against the baseline treatment OFF

Consider Positive (+) Z score when expecting Increase in the Metric

on
50.4% 49.6%
off Z Score > 1.65 - Statistically Significant at 90 % Confidence Interval

* Rate change and Z scores for all metrics are calculated keeping Leads as base, except for Signups where
Visitors is base.
* Market, Segment & Product breakdown by filters do not apply to Significance test results

Z Score describes the relationship between the observed score of a metric after a change and the original score of the metric.

When plotted on a Normal Distribution Curve, it represents the standard deviation of the observed value of metric for the sample
from the mean, zero, of the observed value of the metric for the entire population.

For 95 % Confidence interval, the observed change in the value of metric is considered Statistically Significant if the P Value is beyond
0.05, Alpha/Significant threshold, from the mean, where Z Score ~ 1.96

For 90 % Confidence interval, Alpha ~ 0.1 and for 99 % Confidence interval, Alpha ~ 0.01.

p1 - Sample 1 Proportion (Treatment)

p2 - Sample 2 Proportion (Baseline Treatment)

p - Sample Mean

n1 - Volume of Sample 1
NORMAL DISTRIBUTION CURVE
n2 - Volume of Sample 2

2.3 Buy Flowform Drop-off funnel for ON Treatment

Experiment is triggered once a visitor completes the Buy Where slide

Already Signed up Leads are not considered in the Drop off funnel

CYOAD Slide Answer : Search & Browse Homes

Visitors
1K

800 838 808 797

600 677

400

200 269 252

142 125
0
1. Flow Form Started 2. Flow 3. Buy Where 4. Buy Motivation Slide 5. CYOAD Slide 6. Home Preference Slide 7. Signup 8. Account Created

CYOAD Slide Answer : Learn More About Flyhomes

Visitors
1K

800 838 808 797

600
677

400

200
36 35 13 9
0
1. Flow Form Started 3. Buy Where 5. CYOAD Slide 7. Signup
2. Flow 4. Buy Motivation Slide 6. Learn More about FH Slide 8. Account Created

Flyhomes Cash Offer Flyhomes Buy Before You Sell Flyhomes Buy Now Refi Later Flyhomes Process and Costs Other

21 9 8 14 2

CYOAD Slide Answer : Learn About Homebuying Basics

Visitors
1K

800 838 808 797

600
677

400

200
29 9 8
0
1. Flow Form Started 2. Flow 3. Buy Where 4. Buy Motivation Slide 5. CYOAD Slide 6. Signup 7. Account Created

CYOAD Slide Answer : Get Advice On Market/Neigborhood

Visitors
1K

800 838 808 797

600
677

400

200
28 13 10
0
1. Flow Form Started 2. Flow 3. Buy Where 4. Buy Motivation Slide 5. CYOAD Slide 6. Signup 7. Account Created

CYOAD Slide Answer : Other

Visitors
1K

800 838 808 797

600
677

400

200
38 16 12
0
1. Flow Form Started 2. Flow 3. Buy Where 4. Buy Motivation Slide 5. CYOAD Slide 6. Signup 7. Account Created

Notes
1. All visitors, including signups, are considered to obtain the metrics.
2. Z score is used to determine whether the result obtained is due to chance or any reason.
3. Significance Testing is done at a confidence interval of 90 %.
4. As per the Normal Distribution Curve, z score greater than 1.65 (p value < 0.1) means that the impact is significant.

Visit Source / Device Type : First Page load event in the interval of 30 minutes prior to the Split event is considered, as Split does not send the visitor source/device info

1. Visit Source :
i. Paid - UTM Source is not null & UTM Medium - Search; CPC; PaidSocial; Display; Video
ii. Organic - UTM Source is null & Referrer - Google; Bing; DuckDuckGo; Yahoo; Baidu; Ecosia; AOL; Sogou; Qwant; So.com; Naver; Yandex
iii. Direct - UTM Source is null & Referrer is null
iv. Other - Rest of the source which do not meet Paid, Organic & Direct conditions
2. Visit Source & Device Type is Null where Events are not getting tracking