Conquering Big Data with High Performance Computing

This book provides an overview of the resources and research projects that are bringing Big Data and High Performance Computing (HPC) on converging tracks. It demystifies Big Data and HPC for the reader by covering the primary resources, middleware, applications, and tools that enable the usage of HPC platforms for Big Data management and processing. Through interesting use-cases from traditional and non-traditional HPC domains, the book highlights the most critical challenges related to Big Data processing and management, and shows ways to mitigate them using HPC resources. Unlike most books on Big Data, it covers a variety of alternatives to Hadoop, and explains the differences between HPC platforms and Hadoop. Written by professionals and researchers in a range of departments and fields, this book is designed for anyone studying Big Data and its future directions. Those studying HPC will also find the content valuable. Front Matter....Pages i-viii An Introduction to Big Data, High Performance Computing, High-Throughput Computing, and Hadoop....Pages 1-12 Using High Performance Computing for Conquering Big Data....Pages 13-30 Data Movement in Data-Intensive High Performance Computing....Pages 31-59 Using Managed High Performance Computing Systems for High-Throughput Computing....Pages 61-79 Accelerating Big Data Processing on Modern HPC Clusters....Pages 81-107 dispel4py: Agility and Scalability for Data-Intensive Methods Using HPC....Pages 109-137 Performance Analysis Tool for HPC and Big Data Applications on Scientific Clusters....Pages 139-161 Big Data Behind Big Data....Pages 163-189 Empowering R with High Performance Computing Resources for Big Data Analytics....Pages 191-217 Big Data Techniques as a Solution to Theory Problems....Pages 219-231 High-Frequency Financial Statistics Through High-Performance Computing....Pages 233-252 Large-Scale Multi-Modal Data Exploration with Human in the Loop....Pages 253-268 Using High Performance Computing for Detecting Duplicate, Similar and Related Images in a Large Data Collection....Pages 269-286 Big Data Processing in the eDiscovery Domain....Pages 287-307 Databases and High Performance Computing....Pages 309-319 Conquering Big Data Through the Usage of the Wrangler Supercomputer....Pages 321-329