How to Read SplitSignal Test Results?

Effect Size - Percentage of increase/decrease in clicks during the lifetime of the test.

Absolute Effect - Absolute number of increase/decrease in clicks during the lifetime of the test.

Confidence Level - This metric indicates the level of statistical significance of your test results. 

How significant is the result:

• 95% or Higher: the change is large enough to be statistically significant. We are confident that the test change caused the increase/decrease in clicks. 
• 94% or less:  the change is not large enough to consider it statistically significant

Test Duration - shows how long the test ran.

Result Type: 

• Positive - The test is positive and statistically significant.
• No Change - The test didn’t reach statistical significance.
• Negative - The test is negative and statistically significant.

“Good” Prediction Interval (Real Clicks and Predicted Clicks very closely match. This makes for the most conclusive tests) 

“Fair” Prediction Interval (Real Clicks and Predicted Clicks mostly match. This makes for an often conclusive tests)

“Poor” Prediction Interval (Real Clicks and Predicted Clicks do not closely match. This makes for inconclusive tests)

NOTE On Predicted Clicks:

We don’t necessarily ‘predict’ clicks of the control group but instead build a counterfactual prediction based on the control clicks and its relationship with variant historical clicks. We understand the historical relationship between control and variant and then use it to understand if something has changed during the test. We’re not predicting the future. If the relationship remained stable through the test - the results are insignificant. If the relationship has changed through the test - the results become significant.