Borgaz (How the Aliens Were Won Book 1)

Today, investment in financial technology and digital transformation is reshaping the financial landscape and generating many opportunities. Too often, however, engineers and professionals in financial institutions lack a practical and comprehensive understanding of the concepts, problems, techniques, and technologies necessary to build a modern, reliable, and scalable financial data infrastructure. This is where financial data engineering is needed. A data engineer developing a data infrastructure for a financial product possesses not only technical data engineering skills but also a solid understanding of financial domain-specific challenges, methodologies, data ecosystems, providers, formats, technological constraints, identifiers, entities, standards, regulatory requirements, and governance. This book offers a comprehensive, practical, domain-driven approach to financial data engineering, featuring real-world use cases, industry practices, and hands-on projects. You'll learn: • The data engineering landscape in the financial sector • Specific problems encountered in financial data engineering • The structure, players, and particularities of the financial data domain • Approaches to designing financial data identification and entity systems • Financial data governance frameworks, concepts, and best practices • The financial data engineering lifecycle from ingestion to production • The varieties and main characteristics of financial data workflows • How to build financial data pipelines using open source tools and APIs Tamer Khraisha, PhD, is a senior data engineer and scientific author with more than a decade of experience in the financial sector. Copyright Table of Contents Foreword Preface Who Should Read This Book? Prerequisites What to Expect from This Book Book Resources and References Conventions Used in This Book Using Code Examples O’Reilly Online Learning How to Contact Us Acknowledgments Part I. Foundations of Financial Data Engineering Chapter 1. Financial Data Engineering Clarified Defining Financial Data Engineering First of All, What Is Finance? Defining Data Engineering Defining Financial Data Engineering Why Financial Data Engineering? Volume, Variety, and Velocity of Financial Data Finance-Specific Data Requirements and Problems Financial Machine Learning The Disruptive FinTech Landscape Regulatory Requirements and Compliance The Financial Data Engineer Role Description of the Role Where Do Financial Data Engineers Work? Responsibilities and Activities of a Financial Data Engineer Skills of a Financial Data Engineer Summary Chapter 2. Financial Data Ecosystem Sources of Financial Data Public Financial Data Security Exchanges Commercial Data Vendors, Providers, and Distributors Survey Data Alternative Data Confidential and Proprietary Data Structures of Financial Data Time Series Data Cross-Sectional Data Panel Data Matrix Data Graph Data Text Data Types of Financial Data Fundamental Data Market Data Transaction Data Analytics Data Alternative Data Reference Data Entity Data Benchmark Financial Datasets Center for Research in Security Prices Compustat Financials Trade and Quote Database Institutional Brokers’ Estimate System IvyDB OptionMetrics Trade Reporting and Compliance Engine Orbis Global Database SDC Platinum Standard & Poor’s Dow Jones Indices Alternative Datasets Summary Chapter 3. Financial Identification Systems Financial Identifiers Financial Identifier and Identification System Defined The Need for Financial Identifiers Who Creates Financial Identification Systems? Desired Properties of a Financial Identifier Uniqueness Globality Scalability Completeness Accessibility Timeliness Authenticity Granularity Permanence Immutability Security Financial Identification Systems Landscape International Securities Identification Number Classification of Financial Instruments Financial Instrument Short Name Committee on Uniform Security Identification Procedures Legal Entity Identifier Transaction Identifiers Stock Exchange Daily Official List Ticker Symbols Derivative Identifiers Financial Instrument Global Identifier FactSet Permanent Identifier LSEG Permanent Identifier Digital Asset Identifiers Industry and Sector Identifiers Bank Identifiers Summary Chapter 4. Financial Entity Systems Financial Entity Defined Financial Named Entity Recognition Named Entity Recognition Described How Does Named Entity Recognition Work? Approaches to Named Entity Recognition Named Entity Recognition Software Libraries Financial Entity Resolution Entity Resolution Described The Importance of Entity Resolution in Finance How Does Entity Resolution Work? Approaches to Entity Resolution Entity Resolution Software Libraries Summary Chapter 5. Financial Data Governance Financial Data Governance Financial Data Governance Defined Financial Data Governance Justified Data Quality Dimension 1: Data Errors Dimension 2: Data Outliers Dimension 3: Data Biases Dimension 4: Data Granularity Dimension 5: Data Duplicates Dimension 6: Data Availability and Completeness Dimension 7: Data Timeliness Dimension 8: Data Constraints Dimension 9: Data Relevance Data Integrity Principle 1: Data Standards Principle 2: Data Backups Principle 3: Data Archiving Principle 4: Data Aggregation Principle 5: Data Lineage Principle 6: Data Catalogs Principle 7: Data Ownership Principle 8: Data Contracts Principle 9: Data Reconciliation Data Security and Privacy Data Privacy Data Anonymization Data Encryption Access Control Summary Part II. The Financial Data Engineering Lifecycle Chapter 6. Overview of the Financial Data Engineering Lifecycle Financial Data Engineering Lifecycle Defined Criteria for Building the Financial Data Engineering Stack Criterion 1: Open Source Versus Commercial Software Criterion 2: Ease of Use Versus Performance Criterion 3: Cloud Versus On Premises Criterion 4: Public Versus Private Versus Hybrid Cloud Criterion 5: Single Versus Multi-Cloud Criterion 6: Monolithic Versus Modular Codebase Summary Chapter 7. Data Ingestion Layer Data Transmission and Arrival Processes Data Transmission Protocols Data Arrival Processes Data Ingestion Formats General-Purpose Formats Big Data Formats In-Memory Formats Standardized Financial Formats Data Ingestion Technologies Financial APIs Financial Data Feeds Secure File Transfer Cloud Access Web Access Specialized Financial Software Data Ingestion Best Practices Meet Business Requirements Design for Change Enforce Data Governance Perform Benchmarking and Stress Testing Summary Chapter 8. Data Storage Layer Principles of Data Storage System Design Principle 1: Business Requirements Principle 2: Data Modeling Principle 3: Transactional Guarantee Principle 4: Consistency Tradeoffs Principle 4: Scalability Principle 5: Security Data Storage Modeling SQL Versus NoSQL Primary Versus Secondary Operational Versus Analytical Native Versus Non-Native Multi-Model Versus Polyglot Persistence Data Storage Models The Data Lake Model The Relational Model The Document Model The Time Series Model The Message Broker Model The Graph Model The Warehouse Model The Blockchain Model Summary Chapter 9. Data Transformation and Delivery Layer Data Querying Querying Patterns Query Optimization Data Transformation Transformation Operations Transformation Patterns Computational Requirements Data Delivery Data Consumers Delivery Mechanisms Summary Chapter 10. The Monitoring Layer Metrics, Events, Logs, and Traces Metrics Events Logs Traces Data Quality Monitoring Performance Monitoring Cost Monitoring Business and Analytical Monitoring Data Observability Summary Chapter 11. Financial Data Workflows Workflow-Oriented Software Architectures What Is a Data Workflow? Workflow Management Systems Flexibility Configurability Dependency Management Coordination Patterns Scalability Integration Types of Financial Data Workflows Extract-Transform-Load Workflows Stream Processing Workflows Microservice Workflows Machine Learning Workflows Summary Chapter 12. Hands-On Projects Prerequisites Project 1: Designing a Bank Account Management System Database with PostgreSQL Conceptual Model: Business Requirements Logical Model: Entity Relationship Diagram Physical Model: Data Definition and Manipulation Language Project 1: Local Testing Project 1: Clean Up Project 1: Summary Project 2: Designing a Financial Data ETL Workflow with Mage and Python Project 2: Workflow Definition Project 2: Database Design Project 2: Local Testing Project 2: Clean Up Project 2: Summary Project 3: Designing a Microservice Workflow with Netflix Conductor, PostgreSQL, and Python Project 3: Workflow Definition Project 3: Database Design Project 3: Local Testing Project 3: Clean Up Project 3: Summary Project 4: Designing a Financial Reference Data Store with OpenFIGI, PermID, and GLEIF APIs Project 4: Prerequisites Project 4: Local Testing Project 4: Clean Up Project 4: Summary Conclusion Follow Updates on These Projects Report Issues or Ask Questions The Path Forward: Trends Shaping Financial Markets Financial Integration Digitalization of Financial Markets and Cloud Adoption Financial Regulation Financial Data Sharing and Marketplaces Financial Standardization Artificial Intelligence and Language Models Architectures for Specific Business Domains Data Collection Speed and Efficiency Tokenization, Blockchain, and Digital Currencies What Can You Do Next? Afterword Index About the Author Colophon