Mastering ARKit Apple{u2019}s Augmented Reality App Development Platform

Embark on a journey to build an augmented reality world. This book puts theory into practical application by building unique augmented reality apps specific to each chapter’s topic. You’ll learn to engineer successful, well-designed, and comprehensive augmented reality applications for Apple devices. First, uncover the foundation of augmented reality on Apple platforms. Understand the building blocks and inner workings of the technologies that power augmented reality. Then, learn about building AR apps using ARKit, SceneKit, SpriteKit, RealityKit, and RealityComposer. You’ll study augmented reality app business models, designs, and content for the ultimate user experience. Plus, you’ll cover the theories and practicalities of ARKit APIs such as surface detection, working with world maps, body motion capture, face tracking, object scanning and detecting, people occlusion, and more. Chapter by chapter, this book helps you to become an advanced augmented reality engineer. By the end of the book, you’ll have the necessary mental models and tools to engineer delightful experiences in Apple’s augmented reality platforms. What You'll Learn Build AR apps with SceneKit, SpriteKit, RealityKit, and RealityComposer Understand the business models that support augmented reality Incorporate surface detection, computer vision, and body motion capture in your apps Who This Book Is For iOS developers who want to expand their knowledge of ARKit. They should have a solid foundation in the Swift programming language and understand the basics of navigating Xcode. Table of Contents About the Author About the Technical Reviewer Chapter 1: Why Augmented Reality? The Three A’s: Accessibility, Assistance, Automation Real Estate Education Closing Why ARKit? Apple’s Core Values Versus Cross-Platform Software Riding the Technology Momentum Apple’s Documentation Apple’s Developer Community Documentation The Apple Audience Conclusion Chapter 2: Introduction to ARKit: Under the Hood and Matrixes What Is an Array? What Is a Matrix? Matrixes: Addition and Subtraction Operations Matrix Identity Setting Up Playground Setting Up Scene View Matrix Translation Matrix Rotation Matrix Scaling Up Next Graphics Frameworks for ARKit SpriteKit SceneKit Metal, RealityKit, Reality Composer Up Next Session Life Cycles and Tracking Quality Session Life Cycles Not Available Tracking State Limited Tracking State Normal Tracking State Tracking Quality Transitions Relocalization Without a World Map Relocalization with a World Map Up Next Chapter 3: Designing an Augmented Reality Experience Ideas Why Start with the User Experience? Augmented Reality Benefits Physical World Environment Physical World Objects Overlaying Virtual Objects Idea Case Studies Case Study 1: Virtual Shoe Store Case Study 2: Computer Science Education The Startup User Environment Scale Table Scale Room Scale World Scale Defined and Undefined Space Augmented Reality Experience Prototyping Physical Prototyping Reality Composer Conclusion Chapter 4: Building Your First ARKit App with SceneKit Purpose Content Creating a New Project Setting Up ARKit SceneKit View Connecting IBOutlet Configuring ARSCNView Session Allowing Camera Usage Adding 3D Object to ARSCNView Adding Tap Gesture Recognizer to ARSCNView Removing Object from ARSCNView Adding Multiple Objects to ARSCNView Implementing Pan Gesture Recognizer on ARSCNView Trying Out the Final App Wrap Up Chapter 5: Understanding and Implementing 3D Objects What You’re Going to Learn 3D Objects Creation Toolsets Online Resources for 3D Objects SceneKit Supported Format Getting Started with the Starter Project Implementing a Single Node 3D Object Adding Basic Lighting Implementing a Multiple Nodes 3D Object Loading USDZ into Scene View Wrap Up Chapter 6: Detecting Horizontal Planes and Adding 3D Objects with Raycast in SceneKit What We Are Going to Learn What’s a Horizontal Plane? Let’s Begin to Build the App Detecting Horizontal Planes Adding Objects on Horizontal Planes Raycasting for Physical World Surface Positioning Refactoring for Segmented Control and Adding Additional Objects Up Next Chapter 7: Understanding Physics by Launching a Rocketship Physics Body Explained Physics Body Types Static Dynamic Kinematic Creating a Physics Body Attaching a Static Physics Body Attaching a Dynamic Physics Body Applying Force Adding SceneKit Particle System and Changing Physics Properties Adding Swipe Gestures Showtime Implementing Custom Object Collision Detection Logic Category Bitmasks Collision Bitmasks Contact Test Bitmasks Implementing the Solution Chapter 8: Light Estimation and Environment Texturing What You’ll Implement and Learn Getting Started Creating a Sphere Node Adding Light Node Testing Light Properties Showing/Hiding the Light Estimation Switch Working with the Light Estimation Switch Implementing Light Estimation Trying Out the App Demo Environment Texturing Physically Based Lighting Model, Metalness, and Roughness Chapter 9: 2D Image Recognition and Filter Modification What You’ll Build Getting Started Enabling Image Recognition in ARKit Physical Image Size Image Properties Setting Configuration for Image Recognition Recognizing Images with ARImageAnchor Testing the Demo App Overlaying 3D Objects on Detected Images Dynamically Create ARReferenceImage Programmatically Chapter 10: Saving and Restoring World Mapping Data to Create a Persistence AR Experience Getting Started Working with ARWorldMap Setting Up the World Map Local Document Directory Archiving an AR World Map as Data Saving an AR World Map Data into Your Document Directory Loading an AR World Map Data from Your Document Directory Setting the Scene View Configuration’s Initial World Map Persistence AR Demo Visualizing a World Map in 3D Space with ARPointCloud Conclusion Chapter 11: Advancing App with Real-Time Image Analysis, Machine Learning, and Vision Getting Started Creating an Image Analysis Request for Your Core ML Model Classifying Camera’s Pixel Buffer from Video Frame Controlling Toy Robot with Hand Gesture Analysis Adding Toy Robot upon Plane Detection Showtime Chapter 12: Crafting 3D Assets Crafting a Tower in SpriteKit Crafting a Fireball in SceneKit Crafting a Troop in Reality Composer More Tools Conclusion Chapter 13: Creating Immersive Audio Experiences Implementing Spatial Sound Awareness Stereo to Mono Audio Channels Conversion Tools Chapter 14: Working with SpriteKit and ARKit Getting Started Working with SKScene What Is SKScene? Setting Up a Custom SKScene Randomly Placing 2D Object in 3D Space Positioning a SpriteKit Node on Tap Detecting Change in Time in SKScene Detecting Touches in SKScene Setting Up SKScene with View Controller Setting Up the Time Label’s Timer Creating the SpriteKit Node Working with ARSKViewDelegate Working with SceneGestureDelegate Setting the Delegates Running the Game Conclusion Chapter 15: Building Shared Experiences with Multipeer Connectivity Getting Started Implementing ARCoachingOverlayView Setting Up MultipeerConnectivity Handling MCSessionDelegate Handling MCNearbyServiceBrowserDelegate Handling MCNearbyServiceAdvertiserDelegate Setting Up ViewController for MultipeerConnectivity Sending Data in MCSession Key-Value Observing (KVO) Encoding Collaboration Data Receiving Peer Data Making Virtual Content with AnchorEntity Working with ARParticipantAnchor Discovering Peers Joining Peers Leaving Peers Where to Go from Here? Chapter 16: Face Tracking Getting Started Preparing Your Model Entity for ARFaceAnchor Adding Your Anchor Entity onto ARFaceAnchor Updating Anchor Entity from New ARFaceAnchor Incorporating RealityKit Animoji in ARView Setting Up RealityKit Action Handlers Updating Your Animoji Where to Go from Here? Chapter 17: Reality Composer: Creating AR Content Getting Started Designing for Face Anchors Making Your Animoji Using Primitive Shapes Implementing Content Library’s 3D Models Setting 3D Content Materials Making Facial Features with Content Library’s 3D Models Adding Behaviors to Objects Renaming Behaviors Implementing Your Custom Behaviors Where to Go from Here? Chapter 18: Simultaneously Integrate Face Tracking and World Tracking Getting Started Exporting a Reality File from a RealityKit Project Encapsulating Custom Entity Properties Creating the Entity’s API Integrating the Custom Entity with the View Controller Preparing the Custom Entity for Notification Triggers and Actions Setting Up the Custom Entity Notification Triggers Setting Up the Custom Entity Action Notifiers Testing the Notification Triggers and Actions Integration Where to Go from Here? Chapter 19: Scanning 3D Objects What’s a Reference Object Environment Optimization for Object Scanning Preparing Your Environment for Object Scanning Making a Reference Object Merging Two Reference Objects Applying Transform to a Reference Object Exporting a Referenced Object Using Apple’s Sample Project to Export a Referenced Object Where to Go from Here? Chapter 20: Detecting 3D Objects Getting Started Adding Reference Objects to the Project Configuring ARKit for Object Detection Handling Objects Detection Making the Buy Item Anchor Entity Making the Sphere Model Entity Making the Price Tag Text Model Entity Making the Buy Text Model Entity Where to Go from Here? Chapter 21: Body Motion Capture Understanding the Skeleton Capturing a Person’s Body Motion in Real Time Using RealityKit Integrating the Body Motion Capture Feature with a Rigged Model Generating Your Rigged Model Deciphering a Skeleton Using Lower-Level APIs to Spot Various Joints Placing Objects Relative to Various Joints Captured on a Body Calculating Distances Between Joints Where to Go from Here? Chapter 22: People Occlusion Under the Hood of People Occlusion Knowing People Occlusion Features Configuring RealityKit for People Occlusion Setting Frame Semantics for Person Segmentation with Depth Turning Off People Occlusion for Performance Where to Go from Here? Chapter 23: Create System-Wide Accessible AR Content: Quick Look with USDZ Getting Started Conforming to QLPreviewControllerDataSource Universal Zoom Animation with QLPreviewControllerDelegate Determining AR Quick Look Canonical Web Page Forcing Content Size to Origin in Preview Where to Go from Here? Chapter 24: Advancing AR Quick Look for Commerce on the Web Getting Started ARQuickLook Buttons Deploying Website with GitHub Pages Uploading Resources to GitHub Pages Making a Merchant ID for Apple Pay with Web Creating Your Certificate Signing Request (CSR) File Generating the Apple Pay Payment Processing Certificate Registering a Merchant Domain Creating an Apple Pay Merchant Identity Certificate Commerce Integration with Apple Pay Where to Go from Here? Chapter 25: Working with SwiftUI and ARKit Getting Started Understanding the Mechanism for Integrating ARKit with SwiftUI Setting Up a Custom ARView Making a View Model Observable Object for ARView Creating a UIViewRepresentable to Contain ARView Integrating the Coordinator Pattern Displaying ARView in SwiftUI Where to Go from Here? Chapter 26: Record Augmented Reality Experiences with ReplayKit Understanding the Purpose of ReplayKit Integrating ReplayKit into the Project Recording Augmented Reality Experiences Handling Preview Controller Did Finish Protocol Saving Augmented Reality Experiences Wrapping the Recording Logic Inside IBAction Where to Go from Here? Index Embark on a journey to build an augmented reality world. This book puts theory into practical application by building unique augmented reality apps specific to each chapter{u2019}s topic. You{u2019}ll learn to engineer successful, well-designed, and comprehensive augmented reality applications for Apple devices. First, uncover the foundation of augmented reality on Apple platforms. Understand the building blocks and inner workings of the technologies that power augmented reality. Then, learn about building AR apps using ARKit, SceneKit, SpriteKit, RealityKit, and RealityComposer. You{u2019}ll study augmented reality app business models, designs, and content for the ultimate user experience. Plus, you{u2019}ll cover the theories and practicalities of ARKit APIs such as surface detection, working with world maps, body motion capture, face tracking, object scanning and detecting, people occlusion, and more. Chapter by chapter, this book helps you to become an advanced augmented reality engineer. By the end of the book, you{u2019}ll have the necessary mental models and tools to engineer delightful experiences in Apple{u2019}s augmented reality platforms. You will: Build AR apps with SceneKit, SpriteKit, RealityKit, and RealityComposer Understand the business models that support augmented reality Incorporate surface detection, computer vision, and body motion capture in your apps