Introduction to Datafication : Implement Datafication Using AI and ML Algorithms

This book presents the process and framework you need to transform aspects of our world into data that can be collected, analyzed, and used to make decisions. You will understand the technologies used to gather and process data from many sources, and you will learn how to analyze data with AI and ML models. Datafication is becoming increasingly prevalent in many areas of our lives, from business to education and healthcare. It has the potential to improve decision-making by providing insights into patterns, trends, and correlation between seemingly unconnected pieces of data. This book explains the evolution, principles, and patterns of datafication used in our day-to-day activities. It covers how to collect data from a variety of sources, using technologies such as edge, streaming techniques, REST, and frameworks, as well as data cleansing and data lineage. A data analysis framework is provided to guide you in designing and developing AI and ML projects, including the details of sentiment and behavioral analytics. Introduction to Datafication teaches you how to engineer AI and ML projects by using various methodologies, covers the security mechanisms to be applied for datafication, and shows you how to govern the datafication process with a well-defined governance framework. What You Will Learn: Understand the principles and patterns to be adopted for datafication Gain techniques for sourcing and mining data, and for sharing data with a data pipeline Leverage the AI and ML algorithms most suitable for datafication Understand the data analysis framework used in every AI and ML project Master the details of sentiment and behavioral analytics through practical examples Utilize development methodologies for datafication engineering and the related security and governance framework Who This Book Is For: Students, data scientists, data analysts, and AI and ML engineers. Chapter 1:​ Introduction to Datafication What Is Datafication?​ Why Is Datafication Important?​ Data for Datafication Datafication Steps Digitization vs.​ Datafication Types of Data in Datafication Elements of Datafication Data Harvesting Data Curation Data Storage Data Analysis Cloud Computing Datafication Across Industries Summary Chapter 2:​ Datafication Principles and Patterns What Are Architecture Principles?​ Datafication Principles Data Integration Principle Data Quality Principle Data Governance Principles Data Is an Asset Data Is Shared Data Trustee Ethical Principle Security by Design Principle Datafication Patterns Data Partitioning Pattern Data Replication Stream Processing Change Data Capture (CDC) Data Mesh Machine Learning Patterns Summary Chapter 3:​ Datafication Analytics Introduction to Data Analytics What Is Analytics?​ Big Data and Data Science Datafication Analytical Models Content-Based Analytics Data Mining Text Analytics Sentiment Analytics Audio Analytics Video Analytics Comparison in Analytics Datafication Metrics Datafication Analysis Data Sources Data Gathering Introduction to Algorithms Supervised Machine Learning Linear Regression Support Vector Machines (SVM) Decision Trees Neural Networks Naïve Bayes Algorithm K-Nearest Neighbor (KNN) Algorithm Random Forest Unsupervised Machine Learning Clustering Association Rule Learning Dimensionality Reduction Reinforcement Machine Learning Summary Chapter 4:​ Datafication Data-Sharing Pipeline Introduction to Data-Sharing Pipelines Steps in Data Sharing Data-Sharing Process Data-Sharing Decisions Data-Sharing Styles Unidirectional, Asynchronous Push Integration Style Real-Time and Event-based Integration Style Bidirectional, Synchronous, API-led Integration Style Mediated Data Exchange with an Event-Driven Approach Designing a Data-Sharing Pipeline Types of Data Pipeline Batch Processing Extract, Transform, and Load Data Pipeline (ETL) Extract, Load, and Transform Data Pipeline (ELT) Streaming and Event Processing Change Data Capture (CDC) Lambda Data Pipeline Architecture Kappa Data Pipeline Architecture Data as a Service (DaaS) Data Lineage Data Quality Data Integration Governance Summary Chapter 5:​ Data Analysis Introduction to Data Analysis Data Analysis Steps Prepare a Question Prepare Cleansed Data Identify a Relevant Algorithm Build a Statistical Model Match Result Create an Analysis Report Summary Chapter 6:​ Sentiment Analysis Introduction to Sentiment Analysis Use of Sentiment Analysis Types of Sentiment Analysis Document-Level Sentiment Analysis Aspect-Based Sentiment Analysis Multilingual Sentiment Analysis Pros and Cons of Sentiment Analysis Pre-Processing of Data Tokenization Stop Words Removal Stemming and Lemmatization Handling Negation and Sarcasm Rule-Based Sentiment Analysis Lexicon-Based Approaches Sentiment Dictionaries Pros and Cons of Rule-Based Approaches Machine Learning–Based Sentiment Analysis Supervised Learning Techniques Unsupervised Learning Techniques Pros and Cons of the Machine Learning–Based Approach Best Practices for Sentiment Analysis Summary Chapter 7:​ Behavioral Analysis Introduction to Behavioral Analytics Data Collection Behavioral Science Importance of Behavioral Science How Behavioral Analysis and Analytics Are Processed Cognitive Theory and Analytics Biological Theories and Analytics Integrative Model Behavioral Analysis Methods Funnel Analysis Cohort Analysis Customer Lifetime Value (CLV) Churn Analysis Behavioral Segmentation Analyzing Behavioral Analysis Descriptive Analysis with Regression Causal Analysis with Regression Causal Analysis with Experimental Design Challenges and Limitations of Behavioral Analysis Summary Chapter 8:​ Datafication Engineering Steps of AI and ML Engineering AI and ML Development Understanding the Problem to Be Solved Choosing the Appropriate Model Preparing and Cleaning Data Feature Selection and Engineering Model Training and Optimization AI and ML Testing Unit Testing Integration Testing Non-Functional Testing Performance Security Testing DataOps MLOps Summary Chapter 9:​ Datafication Governance Importance of Datafication Governance Why Is Datafication Governance Required?​ Datafication Governance Framework Oversight and Accountability Model Risk, Risk Assessment, and Regulatory Guidance Roles and Responsibilities​ Monitoring and Reporting Datafication Governance Guidelines and Principles Ethical and Legal Aspects Datafication Governance Action Framework Datafication Governance Challenges Summary Chapter 10:​ Datafication Security Introduction to Datafication Security Datafication Security Framework Regulations Organization Concerns Governance and Compliance Business Access Needs Datafication Security Measures Encryption Data Masking Penetration Testing Data Security Restrictions Summary Index