Common Ground

**Get to grips with pandas - a fast, versatile, and high-performance Python library for data discovery, data manipulation, data preparation, and handling data for analytical tasks** * Perform efficient data analysis and manipulation tasks using pandas 1.x * Apply pandas to different real-world domains with the help of step-by-step examples * Become well-versed in using pandas as an effective data exploration tool Data analysis has become an essential skill in a variety of domains where knowing how to work with data and extract insights can generate significant value. Hands-On Data Analysis with Pandas will show you how to analyze your data, get started with machine learning, and work effectively with the Python libraries often used for data science, such as pandas, NumPy, matplotlib, seaborn, and scikit-learn. Using real-world datasets, you will learn how to use the pandas library to perform data wrangling to reshape, clean, and aggregate your data. Then, you will learn how to conduct exploratory data analysis by calculating summary statistics and visualizing the data to find patterns. In the concluding chapters, you will explore some applications of anomaly detection, regression, clustering, and classification using scikit-learn to make predictions based on past data. This updated edition will equip you with the skills you need to use pandas 1.x to efficiently perform various data manipulation tasks, reliably reproduce analyses, and visualize your data for effective decision making—valuable knowledge that can be applied across multiple domains. * Understand how data analysts and scientists gather and analyze data * Perform data analysis and data wrangling using Python * Combine, group, and aggregate data from multiple sources * Create data visualizations with pandas, matplotlib, and seaborn * Apply machine learning algorithms to identify patterns and make predictions * Use Python data science libraries to analyze real-world datasets * Solve common data representation and analysis problems using pandas * Build Python scripts, modules, and packages for reusable analysis code This book is for data science beginners, data analysts, and Python developers who want to explore each stage of data analysis and scientific computing using a wide range of datasets. You’ll also find this book useful if you are a data scientist looking to implement pandas in your machine learning workflow. Working knowledge of the Python programming language will assist with understanding the key concepts covered in this book; however, a Python crash-course tutorial is provided in the code bundle for anyone who needs a refresher. 1. Introduction to Data Analysis 2. Working with Pandas DataFrames 3. Data Wrangling with Pandas 4. Aggregating Pandas DataFrames 5. Visualizing Data with Pandas and Matplotlib 6. Plotting with Seaborn and Customization Techniques 7. Financial Analysis - Bitcoin and the Stock Market 8. Rule-Based Anomaly Detection 9. Getting Started with Machine Learning in Python 10. Making Better Predictions - Optimizing Models 11. Machine Learning Anomaly Detection 12. The Road Ahead Cover Title Page Copyright and Credits Dedicated Foreword to the Second Edition Foreword to the First Edition Contributors Table of Contents Preface Section 1: Getting Started with Pandas Chapter 1: Introduction to Data Analysis Chapter materials The fundamentals of data analysis Data collection Data wrangling Exploratory data analysis Drawing conclusions Statistical foundations Sampling Descriptive statistics Prediction and forecasting Inferential statistics Setting up a virtual environment Virtual environments Installing the required Python packages Why pandas? Jupyter Notebooks Summary Exercises Further reading Chapter 2: Working with Pandas DataFrames Chapter materials Pandas data structures Series Index DataFrame Creating a pandas DataFrame From a Python object From a file From a database From an API Inspecting a DataFrame object Examining the data Describing and summarizing the data Grabbing subsets of the data Selecting columns Slicing Indexing Filtering Adding and removing data Creating new data Deleting unwanted data Summary Exercises Further reading Section 2: Using Pandas for Data Analysis Chapter 3: Data Wrangling with Pandas Chapter materials Understanding data wrangling Data cleaning Data transformation Data enrichment Exploring an API to find and collect temperature data Cleaning data Renaming columns Type conversion Reordering, reindexing, and sorting data Reshaping data Transposing DataFrames Pivoting DataFrames Melting DataFrames Handling duplicate, missing, or invalid data Finding the problematic data Mitigating the issues Summary Exercises Further reading Chapter 4: Aggregating Pandas DataFrames Chapter materials Performing database-style operations on DataFrames Querying DataFrames Merging DataFrames Using DataFrame operations to enrich data Arithmetic and statistics Binning Applying functions Window calculations Pipes Aggregating data Summarizing DataFrames Aggregating by group Pivot tables and crosstabs Working with time series data Time-based selection and filtering Shifting for lagged data Differenced data Resampling Merging time series Summary Exercises Further reading Chapter 5: Visualizing Data with Pandas and Matplotlib Chapter materials An introduction to matplotlib The basics Plot components Additional options Plotting with pandas Evolution over time Relationships between variables Distributions Counts and frequencies The pandas.plotting module Scatter matrices Lag plots Autocorrelation plots Bootstrap plots Summary Exercises Further reading Chapter 6: Plotting with Seaborn and Customization Techniques Chapter materials Utilizing seaborn for advanced plotting Categorical data Correlations and heatmaps Regression plots Faceting Formatting plots with matplotlib Titles and labels Legends Formatting axes Customizing visualizations Adding reference lines Shading regions Annotations Colors Textures Summary Exercises Further reading Section 3: Applications – Real-World Analyses Using Pandas Chapter 7: Financial Analysis – Bitcoin and the Stock Market Chapter materials Building a Python package Package structure Overview of the stock_analysis package UML diagrams Collecting financial data The StockReader class Collecting historical data from Yahoo! Finance Exploratory data analysis The Visualizer class family Visualizing a stock Visualizing multiple assets Technical analysis of financial instruments The StockAnalyzer class The AssetGroupAnalyzer class Comparing assets Modeling performance using historical data The StockModeler class Time series decomposition ARIMA Linear regression with statsmodels Comparing models Summary Exercises Further reading Chapter 8: Rule-Based Anomaly Detection Chapter materials Simulating login attempts Assumptions The login_attempt_simulator package Simulating from the command line Exploratory data analysis Implementing rule-based anomaly detection Percent difference Tukey fence Z-score Evaluating performance Summary Exercises Further reading Section 4: Introduction to Machine Learning with Scikit-Learn Chapter 9: Getting Started with Machine Learning in Python Chapter materials Overview of the machine learning landscape Types of machine learning Common tasks Machine learning in Python Exploratory data analysis Red wine quality data White and red wine chemical properties data Planets and exoplanets data Preprocessing data Training and testing sets Scaling and centering data Encoding data Imputing Additional transformers Building data pipelines Clustering k-means Evaluating clustering results Regression Linear regression Evaluating regression results Classification Logistic regression Evaluating classification results Summary Exercises Further reading Chapter 10: Making Better Predictions – Optimizing Models Chapter materials Hyperparameter tuning with grid search Feature engineering Interaction terms and polynomial features Dimensionality reduction Feature unions Feature importances Ensemble methods Random forest Gradient boosting Voting Inspecting classification prediction confidence Addressing class imbalance Under-sampling Over-sampling Regularization Summary Exercises Further reading Chapter 11: Machine Learning Anomaly Detection Chapter materials Exploring the simulated login attempts data Utilizing unsupervised methods of anomaly detection Isolation forest Local outlier factor Comparing models Implementing supervised anomaly detection Baselining Logistic regression Incorporating a feedback loop with online learning Creating the PartialFitPipeline subclass Stochastic gradient descent classifier Summary Exercises Further reading Section 5: Additional Resources Chapter 12: The Road Ahead Data resources Python packages Searching for data APIs Websites Practicing working with data Python practice Summary Exercises Further reading Solutions Appendix About Packt Other Books You May Enjoy Index Integrate scikit-learn with various tools such as NumPy, pandas, imbalanced-learn, and scikit-surprise and use it to solve real-world machine learning problemsKey FeaturesDelve into machine learning with this comprehensive guide to scikit-learn and scientific PythonMaster the art of data-driven problem-solving with hands-on examplesFoster your theoretical and practical knowledge of supervised and unsupervised machine learning algorithmsBook DescriptionMachine learning is applied everywhere, from business to research and academia, while scikit-learn is a versatile library that is popular among machine learning practitioners. This book serves as a practical guide for anyone looking to provide hands-on machine learning solutions with scikit-learn and Python toolkits.The book begins with an explanation of machine learning concepts and fundamentals, and strikes a balance between theoretical concepts and their applications. Each chapter covers a different set of algorithms, and shows you how to use them to solve real-life problems. You'll also learn about various key supervised and unsupervised machine learning algorithms using practical examples. Whether it is an instance-based learning algorithm, Bayesian estimation, a deep neural network, a tree-based ensemble, or a recommendation system, you'll gain a thorough understanding of its theory and learn when to apply it. As you advance, you'll learn how to deal with unlabeled data and when to use different clustering and anomaly detection algorithms.By the end of this machine learning book, you'll have learned how to take a data-driven approach to provide end-to-end machine learning solutions. You'll also have discovered how to formulate the problem at hand, prepare required data, and evaluate and deploy models in production.What you will learnUnderstand when to use supervised, unsupervised, or reinforcement learning algorithmsFind out how to collect and prepare your data for machine learning tasksTackle imbalanced data and optimize your algorithm for a bias or variance tradeoffApply supervised and unsupervised algorithms to overcome various machine learning challengesEmploy best practices for tuning your algorithm's hyper parametersDiscover how to use neural networks for classification and regressionBuild, evaluate, and deploy your machine learning solutions to productionWho this book is forThis book is for data scientists, machine learning practitioners, and anyone who wants to learn how machine learning algorithms work and to build different machine learning models using the Python ecosystem. The book will help you take your knowledge of machine learning to the next level by grasping its ins and outs and tailoring it to your needs. Working knowledge of Python and a basic understanding of underlying mathematical and statistical concepts is required. Understand data analysis pipelines using machine learning algorithms and techniques with this practical guideKey FeaturesPrepare and clean your data to use it for exploratory analysis, data manipulation, and data wranglingDiscover supervised, unsupervised, probabilistic, and Bayesian machine learning methodsGet to grips with graph processing and sentiment analysisBook DescriptionData analysis enables you to generate value from small and big data by discovering new patterns and trends, and Python is one of the most popular tools for analyzing a wide variety of data. With this book, you'll get up and running using Python for data analysis by exploring the different phases and methodologies used in data analysis and learning how to use modern libraries from the Python ecosystem to create efficient data pipelines.Starting with the essential statistical and data analysis fundamentals using Python, you'll perform complex data analysis and modeling, data manipulation, data cleaning, and data visualization using easy-to-follow examples. You'll then understand how to conduct time series analysis and signal processing using ARMA models. As you advance, you'll get to grips with smart processing and data analytics using machine learning algorithms such as regression, classification, Principal Component Analysis (PCA), and clustering. In the concluding chapters, you'll work on real-world examples to analyze textual and image data using natural language processing (NLP) and image analytics techniques, respectively. Finally, the book will demonstrate parallel computing using Dask.By the end of this data analysis book, you'll be equipped with the skills you need to prepare data for analysis and create meaningful data visualizations for forecasting values from data.What you will learnExplore data science and its various process modelsPerform data manipulation using NumPy and pandas for aggregating, cleaning, and handling missing valuesCreate interactive visualizations using Matplotlib, Seaborn, and BokehRetrieve, process, and store data in a wide range of formatsUnderstand data preprocessing and feature engineering using pandas and scikit-learnPerform time series analysis and signal processing using sunspot cycle dataAnalyze textual data and image data to perform advanced analysisGet up to speed with parallel computing using DaskWho this book is forThis book is for data analysts, business analysts, statisticians, and data scientists looking to learn how to use Python for data analysis. Students and academic faculties will also find this book useful for learning and teaching Python data analysis using a hands-on approach. A basic understanding of math and working knowledge of the Python programming language will help you get started with this book. Get to grips with pandas - a versatile and high-performance library for manipulating, processing, cleaning, and crunching datasets in Python Key Features • Perform efficient data analysis and manipulation tasks using pandas 1.x • Implement pandas in different real-world domains with the help of step-by-step demonstrations • Become well versed in using pandas as an effective data exploration tool Book Description pandas is a powerful and popular library synonymous with Python data science that makes data wrangling and visualization easy by enabling you to work efficiently with tabular data. This second edition will help you get well-versed with the new features in pandas 1.x and enhance your data analysis skills for extracting significant insights and value from data. Hands-On Data Analysis with Pandas will show you how to analyze your data, get started with machine learning, and work effectively with the Python libraries often used for data science, such as pandas, NumPy, matplotlib, seaborn, and scikit-learn. Using real-world datasets, the book shows you how to use the powerful pandas library to perform data wrangling to reshape, clean, and aggregate your data. As you advance, you'll learn how to conduct exploratory data analysis by calculating summary statistics and visualizing the data to find patterns. You'll also explore some applications of anomaly detection, regression, clustering, and classification using scikit-learn to make predictions based on past data. By the end of this data analysis book, you'll be equipped with the skills you need to use pandas to ensure the veracity of your data, visualize it for effective decision-making, and reliably reproduce analyses across multiple domains. What you will learn • Understand how data analysts and scientists gather and analyze data • Perform data analysis and data wrangling using Python • Combine, group, and aggregate data from multiple sources • Create data visualizations with pandas, matplotlib, and seaborn • Apply machine learning algorithms to identify patterns and make predictions • Use Python data science libraries to analyze real-world datasets • Solve common data representation and analysis problems using pandas • Build Python scripts, modules, and packages for reusable analysis code Who This Book Is For This book is for data science beginners, data analysts, and Python developers who want to explore each stage of data analysis and scientific computing using a wide range of datasets. You'll also find this book useful if you are a data scientist looking to implement pandas in your machine learning workflow. Working knowledge of the Python programming language will assist with understanding the key concepts covered in this book. Get To Grips With Pandas - A Fast, Versatile, And High-performance Python Library For Data Discovery, Data Manipulation, Data Preparation, And Handling Data For Analytical Tasks Key Features: Perform Efficient Data Analysis And Manipulation Tasks Using Pandas 1.x Apply Pandas To Different Real-world Domains With The Help Of Step-by-step Examples Become Well-versed In Using Pandas As An Effective Data Exploration Tool Book Description: Data Analysis Has Become An Essential Skill In A Variety Of Domains Where Knowing How To Work With Data And Extract Insights Can Generate Significant Value. Hands-on Data Analysis With Pandas Will Show You How To Analyze Your Data, Get Started With Machine Learning, And Work Effectively With The Python Libraries Often Used For Data Science, Such As Pandas, Numpy, Matplotlib, Seaborn, And Scikit-learn. Using Real-world Datasets, You Will Learn How To Use The Pandas Library To Perform Data Wrangling To Reshape, Clean, And Aggregate Your Data. Then, You Will Learn How To Conduct Exploratory Data Analysis By Calculating Summary Statistics And Visualizing The Data To Find Patterns. In The Concluding Chapters, You Will Explore Some Applications Of Anomaly Detection, Regression, Clustering, And Classification Using Scikit-learn To Make Predictions Based On Past Data. This Updated Edition Will Equip You With The Skills You Need To Use Pandas 1.x To Efficiently Perform Various Data Manipulation Tasks, Reliably Reproduce Analyses, And Visualize Your Data For Effective Decision Making-valuable Knowledge That Can Be Applied Across Multiple Domains. What You Will Learn: Understand How Data Analysts And Scientists Gather And Analyze Data Perform Data Analysis And Data Wrangling Using Python Combine, Group, And Aggregate Data From Multiple Sources Create Data Visualizations With Pandas, Matplotlib, And Seaborn Apply Machine Learning Algorithms To Identify Patterns And Make Predictions Use Python Data Science Libraries To Analyze Real-world Datasets Solve Common Data Representation And Analysis Problems Using Pandas Build Python Scripts, Modules, And Packages For Reusable Analysis Code Who This Book Is For: This Book Is For Data Science Beginners, Data Analysts, And Python Developers Who Want To Explore Each Stage Of Data Analysis And Scientific Computing Using A Wide Range Of Datasets. You'll Also Find This Book Useful If You Are A Data Scientist Looking To Implement Pandas In Your Machine Learning Workflow. Working Knowledge Of The Python Programming Language Will Assist With Understanding The Key Concepts Covered In This Book; However, A Python Crash-course Tutorial Is Provided In The Code Bundle For Anyone Who Needs A Refresher. Understand data analysis pipelines using machine learning algorithms and techniques with this practical guide Key Features Prepare and clean your data to use it for exploratory analysis, data manipulation, and data wrangling Discover supervised, unsupervised, probabilistic, and Bayesian machine learning methods Get to grips with graph processing and sentiment analysis Book Description Data analysis enables you to generate value from small and big data by discovering new patterns and trends, and Python is one of the most popular tools for analyzing a wide variety of data. With this book, you'll get up and running using Python for data analysis by exploring the different phases and methodologies used in data analysis and learning how to use modern libraries from the Python ecosystem to create efficient data pipelines. Starting with the essential statistical and data analysis fundamentals using Python, you'll perform complex data analysis and modeling, data manipulation, data cleaning, and data visualization using easy-to-follow examples. You'll then understand how to conduct time series analysis and signal processing using ARMA models. As you advance, you'll get to grips with smart processing and data analytics using machine learning algorithms such as regression, classification, Principal Component Analysis (PCA), and clustering. In the concluding chapters, you'll work on real-world examples to analyze textual and image data using natural language processing (NLP) and image analytics techniques, respectively. Finally, the book will demonstrate parallel computing using Dask. By the end of this data analysis book, you'll be equipped with the skills you need to prepare data for analysis and create meaningful data visualizations for forecasting values from data. What you will learn Explore data science and its various process models Perform data manipulation using NumPy and pandas for aggregating, cleaning, and handling missing values Create interactive visualizations using Matplotlib, Seaborn, and Bokeh Retrieve, process, and store data in a wide range of formats Understand data preprocessing and feature engineering using pandas and scikit-learn Perform time series analysis and signal processing using sunspot cycle data Analyze textual data and image data to perform advanced analysis Get up to speed with parallel computing using Dask Who this book is for This book is for data analysts, business analysts, statisticians, and data scientists looking .. Integrate scikit-learn with various tools such as NumPy, pandas, imbalanced-learn, and scikit-surprise and use it to solve real-world machine learning problems Key Features Delve into machine learning with this comprehensive guide to scikit-learn and scientific Python Master the art of data-driven problem-solving with hands-on examples Foster your theoretical and practical knowledge of supervised and unsupervised machine learning algorithms Book Description Machine learning is applied everywhere, from business to research and academia, while scikit-learn is a versatile library that is popular among machine learning practitioners. This book serves as a practical guide for anyone looking to provide hands-on machine learning solutions with scikit-learn and Python toolkits. The book begins with an explanation of machine learning concepts and fundamentals, and strikes a balance between theoretical concepts and their applications. Each chapter covers a different set of algorithms, and shows you how to use them to solve real-life problems. You'll also learn about various key supervised and unsupervised machine learning algorithms using practical examples. Whether it is an instance-based learning algorithm, Bayesian estimation, a deep neural network, a tree-based ensemble, or a recommendation system, you'll gain a thorough understanding of its theory and learn when to apply it. As you advance, you'll learn how to deal with unlabeled data and when to use different clustering and anomaly detection algorithms. By the end of this machine learning book, you'll have learned how to take a data-driven approach to provide end-to-end machine learning solutions. You'll also have discovered how to formulate the problem at hand, prepare required data, and evaluate and deploy models in production. What you will learn Understand when to use supervised, unsupervised, or reinforcement learning algorithms Find out how to collect and prepare your data for machine learning tasks Tackle imbalanced data and optimize your algorithm for a bias or variance tradeoff Apply supervised and unsupervised algorithms to overcome various machine learning challenges Employ best practices for tuning your algorithm's hyper parameters Discover how to use neural networks for classification and regression Build, evaluate, and deploy your machine learning solutions to production Who this book is for This book is for data scientists, machine learning practitioners, and anyone who wants to lea.. Hands-On Machine Learning with scikit-learn and Scientific Python Toolkits: A practical guide to implementing supervised and unsupervised machine learning algorithms in Python Machine learning is applied everywhere, from business to research and academia, while scikit-learn is a versatile library that is popular among machine learning practitioners. This book serves as a practical guide for anyone looking to provide hands-on machine learning solutions with scikit-learn and Python toolkits. The book begins with an explanation of machine learning concepts and fundamentals, and strikes a balance between theoretical concepts and their applications. Each chapter covers a different set of algorithms, and shows you how to use them to solve real-life problems. Youll also learn various key supervised and unsupervised machine learning algorithms using practical examples. Whether it is an instance-based learning algorithm, Bayesian estimation, a deep neural network, a tree-based ensemble, or a recommendation system, youll gain a thorough understanding of its theory and learn when to apply it. As you advance, youll learn how to deal with unlabeled data and when to use different clustering and anomaly detection algorithms. By the end of this machine learning book, youll have learnt how to take a data-driven approach to provide end-to-end machine learning solutions. Youll also have discovered how to formulate the problem at hand, prepare required data, and evaluate and deploy models in production. Supervised Learning Introduction to Machine Learning & Scikit-Learn Making Decisions with Trees Making decisions with linear equations Preparing Your Data Image processing with nearest neighbors Text Classification - Not all data exists in tables Advanced Supervised Learning Neural Networks - Here comes the Deep Learning Ensembles - When one model is not enough The Y is as important as the X Imbalanced Learn - Not even 1% win the lottery Unsupervised Learning and More Clustering - Grouping data when no correct answers are provided Anomaly Detection - Finding Outliers in Data Recommender System - Learning about users’ taste from their previous interactions Data analysis enables one to generate value from small and big data by discovering new patterns and trends. Python is a popular tool for analyzing a wide variety of data. This books instructs how to get up and running with using Python for data analysis by exploring the different phases and methodologies used in data analysis and learning how to use modern libraries from the Python ecosystem to create efficient data pipelines Knowing how to work with data to extract insights generates significant value. This book will help you to develop data analysis skills using a hands-on approach and real-world data. You’ll get up to speed with pandas 1.x in no time and build some software engineering skills in the process, vastly expanding your data science toolbox.