Creating Good Data : A Guide to Dataset Structure and Data Representation

[Original PDF format]Create good data from the start, rather than fixing it after it is collected. By following the guidelines in this book, you will be able to conduct more effective analyses and produce timely presentations of research data.Data analysts are often presented with datasets for exploration and study that are poorly designed, leading to difficulties in interpretation and to delays in producing meaningful results. Much data analytics training focuses on how to clean and transform datasets before serious analyses can even be started. Inappropriate or confusing representations, unit of measurement choices, coding errors, missing values, outliers, etc., can be avoided by using good dataset design and by understanding how data types determine the kinds of analyses which can be performed.This book discusses the principles and best practices of dataset creation, and covers basic data types and their related appropriate statistics and visualizations. A key focus of the book is why certain data types are chosen for representing concepts and measurements, in contrast to the typical discussions of how to analyze a specific data type once it has been selected.- Be aware of the principles of creating and collecting data- Know the basic data types and representations- Select data types, anticipating analysis goals- Understand dataset structures and practices for analyzing and sharing- Be guided by examples and use cases (good and bad)- Use cleaning tools and methods to create good dataAssumptions: You have a basic knowledge of statistical methods and tools for summarizing and visualizing datasets, including using tools such as R, Python, and SQL. Table of Contents 5 About the Author 9 About the Technical Reviewer 10 Acknowledgments 11 Introduction 12 Chapter 1: The Need for Good Data 13 Who This Book Is For 13 Assumptions 14 The Importance of Getting Data Right 15 What Exactly Is “Data” and Where Does It Come From? 16 What Is “Good” Data? 17 Where “Bad” Data Comes From 18 Some Causes of Bad Data 19 Preventive Action 20 Summary 21 Chapter References 21 Chapter 2: Basic Data Types and When to Use Them 22 Four Analytic Data Types 23 Nominal/Categorical Data 24 Ordinal Data 27 Special Ordinal Data Types 29 Ratio Data 32 Interval Data 34 Other Data Types 35 Summary 36 Chapter References 37 Chapter 3: Representing Quantitative Data 38 Units of Measurement 38 Magnitudes and Quantities 39 Time Data 40 Money and Currency Data 42 Transformations and Indexing 42 Measurement Standards 43 Other Quantitative Measurement Issues 44 Numerical Precision 44 Multicollinearity 44 Non-numeric Numbers 45 Summary 45 Chapter References 45 Chapter 4: Planning Your Data Collection and Analysis 47 Describing, Comparing, and Predicting 47 Example: Choosing a Data Type 48 Plan for Visualizing Your Data and Analysis 49 The Purpose and Goal of Univariate Descriptive Statistics and Visualizations 50 The Purpose and Goal of Multivariate Relationship Statistics and Visualizations 50 Independent and Dependent Variables 51 Data Analysis Tools 53 Summary 54 Chapter References 55 Chapter 5: Good Datasets 57 Sharing Data 57 Dataset Dictionaries/Metadata 58 Good Metadata 59 What’s in a Name? 60 Data Item Naming 60 Dataset File Naming 61 Dataset Formats 61 Keep It Simple 62 Comma-Separated Values (.csv) 62 JSON Datasets 64 HTML Data 64 More Dataset Formats 65 Managing Datasets 65 E pluribus unum? [19] 66 Is Your Data Ready? 66 Summary 67 Chapter References 67 Chapter 6: Good Data Collection 69 What Is Bias? 69 Major Types of Bias 70 Sampling Bias 71 What Does “Random” Selection Mean? 71 Good Sampling 72 More Data Collection Problems 72 Recognizing and Reducing Bias 74 Understanding Outliers 74 The Consequences of Bias 75 Summary 75 Chapter References 76 Chapter 7: Dataset Examples and Use Cases 77 The Titanic Survivor Dataset 77 The IBM Employee Attrition Dataset 78 The Internet Movie Database (IMDb) 79 US Hurricane Data 80 UFO Sighting Data 81 Lessons Learned 82 Useful Dataset Sources 82 Summary 83 Chapter References 83 Chapter 8: Cleaning Your Data 84 Data Cleaning Challenges 84 Assessing Data Quality 86 Software and Methods for Data Cleaning 86 General Procedures 86 Microsoft Excel 87 R Project 88 Recording Errors 88 Missing Data 90 Imputation 91 Outliers 91 Python 93 Operating System Utilities 97 AI/ML-Based Software 98 Summary 99 Chapter References 99 Chapter 9: Good Data Analytics 101 What Is Good Analytics? 101 Big Data Analytics 102 Data for Good 103 Summary 105 Chapter References 105 Appendix A: Recommended Reading 106 Books 106 Websites 107 Oldies but Goodies 108 Index 110 Table of Contents......Page 5 About the Author......Page 9 About the Technical Reviewer......Page 10 Acknowledgments......Page 11 Introduction......Page 12 Who This Book Is For......Page 13 Assumptions......Page 14 The Importance of Getting Data Right......Page 15 What Exactly Is “Data” and Where Does It Come From?......Page 16 What Is “Good” Data?......Page 17 Where “Bad” Data Comes From......Page 18 Some Causes of Bad Data......Page 19 Preventive Action......Page 20 Chapter References......Page 21 Chapter 2: Basic Data Types and When to Use Them......Page 22 Four Analytic Data Types......Page 23 Nominal/Categorical Data......Page 24 Ordinal Data......Page 27 Special Ordinal Data Types......Page 29 Ratio Data......Page 32 Interval Data......Page 34 Other Data Types......Page 35 Summary......Page 36 Chapter References......Page 37 Units of Measurement......Page 38 Magnitudes and Quantities......Page 39 Time Data......Page 40 Transformations and Indexing......Page 42 Measurement Standards......Page 43 Multicollinearity......Page 44 Chapter References......Page 45 Describing, Comparing, and Predicting......Page 47 Example: Choosing a Data Type......Page 48 Plan for Visualizing Your Data and Analysis......Page 49 The Purpose and Goal of Multivariate Relationship Statistics and Visualizations......Page 50 Independent and Dependent Variables......Page 51 Data Analysis Tools......Page 53 Summary......Page 54 Chapter References......Page 55 Sharing Data......Page 57 Dataset Dictionaries/Metadata......Page 58 Good Metadata......Page 59 Data Item Naming......Page 60 Dataset Formats......Page 61 Comma-Separated Values (.csv)......Page 62 HTML Data......Page 64 Managing Datasets......Page 65 Is Your Data Ready?......Page 66 Chapter References......Page 67 What Is Bias?......Page 69 Major Types of Bias......Page 70 What Does “Random” Selection Mean?......Page 71 More Data Collection Problems......Page 72 Understanding Outliers......Page 74 Summary......Page 75 Chapter References......Page 76 The Titanic Survivor Dataset......Page 77 The IBM Employee Attrition Dataset......Page 78 The Internet Movie Database (IMDb)......Page 79 US Hurricane Data......Page 80 UFO Sighting Data......Page 81 Useful Dataset Sources......Page 82 Chapter References......Page 83 Data Cleaning Challenges......Page 84 General Procedures......Page 86 Microsoft Excel......Page 87 Recording Errors......Page 88 Missing Data......Page 90 Outliers......Page 91 Python......Page 93 Operating System Utilities......Page 97 AI/ML-Based Software......Page 98 Chapter References......Page 99 What Is Good Analytics?......Page 101 Big Data Analytics......Page 102 Data for Good......Page 103 Chapter References......Page 105 Books......Page 106 Websites......Page 107 Oldies but Goodies......Page 108 Index......Page 110 Create good data from the start, rather than fixing it after it is collected. By following the guidelines in this book, you will be able to conduct more effective analyses and produce timely presentations of research data. Data analysts are often presented with datasets for exploration and study that are poorly designed, leading to difficulties in interpretation and to delays in producing meaningful results. Much data analytics training focuses on how to clean and transform datasets before serious analyses can even be started. Inappropriate or confusing representations, unit of measurement choices, coding errors, missing values, outliers, etc., can be avoided by using good dataset design and by understanding how data types determine the kinds of analyses which can be performed. This book discusses the principles and best practices of dataset creation, and covers basic data types and their related appropriate statistics and visualizations. A key focus of the book is why certain data types are chosen for representing concepts and measurements, in contrast to the typical discussions of how to analyze a specific data type once it has been selected. What You Will Learn Be aware of the principles of creating and collecting data Know the basic data types and representations Select data types, anticipating analysis goals Understand dataset structures and practices for analyzing and sharing Be guided by examples and use cases (good and bad) Use cleaning tools and methods to create good data Who This Book Is For Researchers who design studies and collect data and subsequently conduct and report the results of their analyses can use the best practices in this book to produce better descriptions and interpretations of their work. In addition, data analysts who explore and explain data of other researchers will be able to create better datasets. Create good data from the start, rather than fixing it after it is collected. By following the guidelines in this book, you will be able to conduct more effective analyses and produce timely presentations of research data. Data analysts are often presented with datasets for exploration and study that are poorly designed, leading to difficulties in interpretation and to delays in producing meaningful results. Much data analytics training focuses on how to clean and transform datasets before serious analyses can even be started. Inappropriate or confusing representations, unit of measurement choices, coding errors, missing values, outliers, et cetera, can be avoided by using good dataset design and by understanding how data types determine the kinds of analyses which can be performed. This book discusses the principles and best practices of dataset creation, and covers basic data types and their related appropriate statistics and visualizations. A key focus of the book is why certain data types are chosen for representing concepts and measurements, in contrast to the typical discussions of how to analyze a specific data type once it has been selected. What You Will Learn Be aware of the principles of creating and collecting data Know the basic data types and representations Select data types, anticipating analysis goals Understand dataset structures and practices for analyzing and sharing Be guided by examples and use cases (good and bad) Use cleaning tools and methods to create good data Who This Book Is For Researchers who design studies and collect data and subsequently conduct and report the results of their analyses can use the best practices in this book to produce better descriptions and interpretations of their work. In addition, data analysts who explore and explain data of other researchers will be able to create better datasets