Hypothesis Testing: An Intuitive Guide For Making Data Driven Decisions

Goals for this Book Fundamental Concepts Descriptive vs. Inferential Statistics Population Parameters vs. Sample Statistics Random Sampling Error Parametric versus Nonparametric Analyses Hypothesis Testing Null Hypothesis Alternative Hypothesis Effect Significance Level (Alpha) P-values Statistical Significance Confidence intervals (CIs) Significance Levels In-Depth Evidentiary Standards in the Courtroom Significance Levels as an Evidentiary Standard Changing Significance Levels How Hypothesis Tests Work Descriptive Statistics Won’t Answer the Question A Sampling Distribution Determines Whether Our Sample Mean is Unlikely Graphing our Sample Mean in the Context of the Sampling Distribution Graphing Significance Levels as Critical Regions What Are P-values? Using P-values and Significance Levels Together Discussion about Statistically Significant Results How Confidence Intervals Work Precision of the Estimate Graphical Representation Confidence Intervals and P-values Always Agree I Really Like Confidence Intervals! Review and Next Steps T-Test Uses, Assumptions, and Analyses 1-Sample t-Tests Assumptions 1-sample t-test example 2-Sample t-Tests Assumptions 2-sample t-test example Paired t-Tests Assumptions Paired t-Test example Paired t-Tests Are Really 1-Sample t-Tests Why Not Accept the Null Hypothesis? Species Presumed to be Extinct Criminal Trials Hypothesis Tests Interpreting Failures to Reject the Null Using Confidence Intervals to Compare Means Jumping to Conclusions Using the Wrong CIs Confidence Intervals of Differences Review and Next Steps Test Statistics and Their Sampling Distributions How 1-Sample t-Tests Calculate t-Values The Signal – The Size of the Sample Effect The Noise – The Variability or Random Error in the Sample Signal-to-Noise ratio How Two-Sample T-tests Calculate T-Values t-Distributions and Statistical Significance What Are t-Distributions? Use the t-Distribution to Compare Your Sample Results to the Null Hypothesis Using t-Values and t-Distributions to Calculate Probabilities t-Distributions and Sample Size Z-tests versus t-tests Review and Next Steps Interpreting P-values It’s All About the Null Hypothesis Defining P-values P-values Are NOT an Error Rate What Is the True Error Rate? Why Are P-values Misinterpreted So Frequently? Historical Events Made P-values Confusing P-values Don’t Provide the Answers that We Really Want P-values Have a Torturous Definition Is Misinterpreting P-values Really a Problem? P-values and the Reproducibility of Experiments Estimating the Reproducibility Rate P-values and Reproducibility Rates The Good Side of High P-values P-values Greater Than the Significance Level What High P-Values Mean and Don’t Mean Practical vs. Statistical Significance Statistical Significance Practical Significance Example of Using Confidence Intervals for Practical Significance Practical Tips to Avoid Being Fooled Tip 1: Smaller P-values are Better Tip 2: Replication is Crucial Tip 3: The Effect Size is Important Tip 4: The Plausibility of the Alternative Hypothesis Matters Tip 5: Use Your Expertise Evaluating the Hypothesis Test Results for the AIDS Vaccine Study Review and Next Steps Types of Errors and Statistical Power Fire Alarm Analogy Type I Errors: False Positives Warning about a potential misinterpretation of Type I errors and the Significance Level Type II Errors: False Negatives Type II Errors and Statistical Power Graphing Type I and Type II Errors Is One Error Worse Than the Other? Power and Sample Size Analysis Factors Involved in Statistical Significance Goals of a Power Analysis 2-Sample t-Test Power Analysis for Sample Size Differences Power values Standard deviation Interpreting the Power Analysis Results Calculating Power Using Standardized Effects Use Power Analysis for All Studies Low Power Tests Exaggerate Effect Sizes Hypothetical Study Scenario Findings and Estimated Effect Sizes for Very Low Power (0.3) Power = 0.55 Power = 0.8 Relationship between Statistical Power and Effect Size How Low Statistical Power Biases the Estimates Graphical Representation Discussion Review and Next Steps One-Tailed and Two-Tailed Hypothesis Tests Critical Regions Two-Tailed Tests Example of a two-tailed 1-sample t-test Advantages of two-tailed hypothesis tests One-Tailed Tests Example of a one-tailed 1-sample t-test Advantages and disadvantages of one-tailed hypothesis tests When Can I Use One-Tailed Tests? Two-Tailed Tests are the Default Choice A Concrete Rule about Choosing Between One- and Two-Tailed Tests One-Tailed Tests Can Be the Only Option Chi-squared tests F-tests Effects can Occur in Only One Direction Only Need to Detect Effects in One Direction Beware of the Power that One-Tailed Tests Provide Is the Higher False Positive Rate Worthwhile? Alternative: Two-Tails with a Higher Alpha This Approach Is More Transparent Review and Next Steps Sample Size Considerations Degrees of Freedom Independent Information and Constraints on Values Estimating Parameters Imposes Constraints on the Data Degrees of Freedom and Probability Distributions t-Distribution F-Distribution Chi-Square Test of Independence Chi-Square 2 X 2 Table Chi-Square 3 X 2 Table Degrees of Freedom in Regression Analysis Central Limit Theorem Distribution of the Variable in the Population Sampling Distribution of the Mean Sufficiently Large Sample Size Approximating the Normal Distribution Properties of the Central Limit Theorem Empirical Demonstration Testing the CLT with Three Probability Distributions Moderately Skewed Distribution Very Skewed Distribution Uniform Distribution Why is the Central Limit Theorem Important? Central limit theorem and the normality assumption Precision of estimates Review and Next Steps Data Types and Hypothesis Tests Continuous Data Binary Data Comparing Continuous Data to Binary Data Count Data Categorical Data Ordinal Data Review and Next Steps ANOVA Compares More Than Two Groups One-Way ANOVA Assumptions The dependent variable is continuous The independent variable is categorical Your sample data should follow a normal distribution or each group has more than 15 or 20 observations Groups should have roughly equal variances or use Welch’s ANOVA Example One-Way ANOVA Interpreting the One-Way ANOVA Results How F-tests work in ANOVA The F-test in One-Way ANOVA F-test Numerator: Between-Groups Variance F-test Denominator: Within-Groups Variance The F-Statistic: Ratio of Between-Groups to Within-Groups Variances How to Calculate our F-value How F-tests Use F-distributions to Test Hypotheses Graphing the F-test for Our One-Way ANOVA Example Why We Analyze Variances to Test Means Using Post Hoc Tests with ANOVA Example One-Way ANOVA to Use with Post Hoc Tests What is the Experiment-wise Error Rate? Family Error Rates in ANOVA Post Hoc Tests Control the Experiment-wise Error Rate Tukey’s Method Adjusted P-values Simultaneous Confidence Intervals Tukey Simultaneous CIs for our One-Way ANOVA Example Post Hoc Tests and the Statistical Power Tradeoff Managing the Power Tradeoff in Post Hoc Tests by Reducing the Number of Comparisons Dunnett’s Compares Treatments to a Control Hsu’s MCB to Find the Best Simultaneous Confidence Intervals for Hsu’s MCB Recap of Using Multiple Comparison Methods Two-Way ANOVA Assumptions Random residuals with constant variance Two-Way ANOVA without Interaction Two-Way ANOVA with Interaction Interaction Effects in Depth Review and Next Steps Continuous Data: Variability, Correlations, Distributions & Outliers Testing Variability One-Sample Variance Test Assumptions Your sample data should follow a normal distribution or have more than 40 observations Example 1 Variance Test Two-Sample Variances Test Assumptions Your sample data should follow a normal distribution or each group is unimodal and has more than 20 observations Example of the 2 Variances Test Variances Testing Methods Test of Pearson’s Correlation Assumptions Data follow a bivariate normal distribution or you have at least 25 observation Example of Correlation Hypothesis Test Testing the Distribution of Your Continuous Data Graph the Raw Data Using Distribution Tests Normality Test Goodness-of-Fit Tests for Other Distributions Using Probability Plots Three-Parameter Distributions Parameter Values for Our Distribution Caution: What These Tests Do NOT Tell You! Outliers Data Entry and Measurement Errors and Outliers Sampling Problems Can Cause Outliers Natural Variation Can Produce Outliers Guidelines for Removing Outliers Five Ways to Find Outliers Sorting Your Datasheet to Find Outliers Graphing Your Data to Identify Outliers Using Z-scores to Detect Outliers Using the Interquartile Range to Create Outlier Fences Finding Outliers with Hypothesis Tests Challenges of Using Outlier Hypothesis Tests: Masking and Swamping My Philosophy about Finding Outliers Statistical Analyses that Can Handle Outliers Review and Next Steps Binary Data and Testing Proportions One-Sample Proportion Test Assumptions Each trial is independent The proportion remains constant over time Example of the 1 Proportion Test How the Proportion Test Works Two-Sample Proportions Test Assumptions Three Examples for the 2 Proportions Test Quick Example 2 Proportions Binomial Exact Test vs. Normal Approximation Mythbusters Example: Are Yawns Contagious? Assess Statistical Power to Estimate the Correct Sample Size The Mythbusters Need Statistics and Hypothesis Testing! 2 Proportions Example: Flu Shot Effectiveness Defining the Effectiveness of Flu Shots Two Flu Vaccination Studies The Beran Study The Monto Study Flu Shot Conclusions Distributions for Binary Data Example of the Binomial Distribution Other distributions that use binary data Review and Next Steps Count Data and Rates of Occurrence One-Sample Poisson Rate Test Assumptions Example of the 1-Sample Poisson Rate Test Two-Sample Poisson Rate Test Assumptions Example of the Two-Sample Poisson Rate Test Poisson Exact Test vs. Normal Approximation Goodness-of-Fit for a Poisson Distribution Review and Next Steps Categorical Variables Chi-Square Test of Independence Star Trek Fatalities by Uniform Colors How the Chi-square Test Works Contingency Table with Expected Values Calculating the Chi-Squared Statistic Important Considerations about the Chi-Squared Statistic Calculating Chi-Squared for our Example Data Using the Chi-Squared Distribution to Test Hypotheses Graphing the Chi-Squared Test Results for Our Example Bonus Analysis! Risk by Work Area Summary Categorical Variables and Discrete Distributions Car Color Example of a Discrete Distribution The Chi-Square Goodness-of-Fit Test Results Review and Next Steps Alternative Methods Nonparametric Tests vs. Parametric Tests Related Pairs of Parametric and Nonparametric Tests Advantages of Parametric Tests Advantages of Nonparametric Tests Advantages and Disadvantages of Parametric and Nonparametric Tests Analyzing Likert Scale Data Comparing Error Rates and Power When Analyzing Likert Scale Data Example of the Mann-Whitney Median Test Choosing the Correct Hypothesis Test The Mann-Whitney Test Results Bootstrapping Method Differences between Bootstrapping and Parametric Hypothesis Testing How Bootstrapping Resamples Your Data to Create Simulated Datasets Example of Bootstrap Samples How Well Does Bootstrapping Work? Example of Using Bootstrapping to Create Confidence Intervals Performing the bootstrap procedure Benefits of Bootstrapping over Traditional Statistics For Which Sample Statistics Can I Use Bootstrapping? Wrapping Up Review of What You Learned in this Book Next Steps for Further Study Index of Hypothesis Tests by Data Types Continuous Data Binary Data Count Data Categorical Data Ordinal and Ranked Data References About the Author