Fundamentals of Agricultural and Field Robotics (Agriculture Automation and Control)

Over the past century, mechanization has been an important means for optimizing resource utilization, improving worker health and safety and reducing labor requirements in farming while increasing productivity and quality of 4F (Food, Fuel, Fiber, Feed). Recognizing this contribution, agricultural mechanization was considered as one of the top ten engineering achievements of 20 th century by the National Academy of Engineering. Accordingly farming communities have adopted increasing level of automation and robotics to further improve the precision management of crops (including input resources), increase productivity and reduce farm labor beyond what has been possible with conventional mechanization technologies. It is more important than ever to continue to develop and adopt novel automation and robotic solutions into farming so that some of the most complex agricultural tasks, which require huge amount of seasonal labor such as fruit and vegetable harvesting, couldbe automated while meeting the rapidly increasing need for 4F. In addition, continual innovation in and adoption of agricultural automation and robotic technologies is essential to minimize the use of depleting resources including water, minerals and other chemicals so that sufficient amount of safe and healthy food can be produced for current generation while not compromising the potential for the future generation. This book aims at presenting the fundamental principles of various aspects of automation and robotics as they relate to production agriculture (the branch of agriculture dealing with farming operations from field preparation to seeding, to harvesting and field logistics). The building blocks of agricultural automation and robotics that are discussed in the book include sensing and machine vision, control, guidance, manipulation and end-effector technologies. The fundamentals and operating principles of these technologies are explained with examples from cutting-edge research and development currently going on around the word. This book brings together scientists, engineers, students and professionals working in these and related technologies to present their latest examples of agricultural automation and robotics research, innovation and development while explaining the fundamentals of the technology. The book, therefore, benefits those who wish to develop novel agricultural engineering solutions and/or to adopt them in the future. Preface Where are We Coming From? Organization of the Book Summary and Concluding Thoughts Contents Contributors About the Editors Chapter 1: Agricultural and Field Robotics: An Introduction 1.1 Background 1.2 Fundamental Technologies for Agricultural and Field Robotics 1.2.1 Sensing and Situation Awareness 1.2.2 Intelligent Decision-Making 1.3 Challenges and Opportunities 1.3.1 Economics: A Critical Dimension 1.4 Concluding Thoughts References Part I: Sensing and Machine Vision Chapter 2: Sensors I: Color Imaging and Basics of Image Processing 2.1 Introduction 2.2 Basics of Color Imaging 2.2.1 Color Representation 2.2.2 Color Space Conversion 2.2.3 Color Comparison 2.3 Image Acquisition 2.4 Basic Image Processing Operations 2.4.1 Image Enhancement 2.4.1.1 Histogram 2.4.1.2 Morphological Operations 2.4.1.3 Low-Pass Filtering 2.4.2 Segmentation 2.4.2.1 Pixel-Wise Techniques 2.4.2.2 Region-Based Segmentation 2.4.3 Features of Objects of Interest 2.4.4 Hough Transform 2.5 Pattern Matching 2.6 Things to Consider 2.7 Summary and Concluding Thoughts References Chapter 3: Sensors II: 3D Sensing Techniques and Systems 3.1 Introduction 3.2 3D Measurement Principles 3.2.1 3D from 2D Images 3.2.2 3D with Time-of-Flight of Light 3.2.3 Structured Light 3.3 Stereo-Vision System 3.3.1 Introduction 3.3.2 Depth Estimation Using Stereo-Vision Camera 3.3.3 Camera Calibration 3.3.4 Image Correspondence 3.3.5 Epipolar Geometry 3.3.6 Tools for Stereo-Vision-Based Distance Measurement 3.4 Other 3D Measurement Systems 3.4.1 Visual Servoing 3.4.2 Laser and LIDAR 3.4.3 3D Camera 3.4.4 Global Navigation Satellite Systems (GNSS) 3.4.5 Interferometric Synthetic Aperture RADAR (InSAR) 3.4.6 Ultrasonic and Infrared Techniques 3.5 Case Studies 3.5.1 Crop-Load Estimation in Orchards 3.5.2 Robotic Fruit Harvesting 3.5.3 Robotic Fruit Tree Pruning 3.5.4 Automated Red Raspberry Bundling 3.6 Summary and Concluding Thoughts References Chapter 4: Sensors III: Spectral Sensing and Data Analysis 4.1 Introduction 4.2 Spectroradiometers 4.2.1 Working Principle 4.2.2 Spectroradiometer Types 4.2.3 Spectral Data Analysis 4.2.3.1 Preprocessing 4.2.3.2 Spectral Feature Extraction 4.2.3.3 Spectral Data Classification/Prediction Models 4.2.4 Application Case Studies 4.2.4.1 Case Study 1. Biotic and Abiotic Stress Detection in Grapevines 4.2.4.2 Case Study 2. Citrus Disease Detection 4.2.4.3 Case Study 3. Apple Bitter Pit Disorder Detection 4.3 Spectral Imaging 4.3.1 Multispectral Imaging 4.3.1.1 Imaging Platforms 4.3.1.2 Multispectral Imagery Processing 4.3.1.3 Multispectral Imaging Applications 4.3.2 Hyperspectral Imaging 4.3.2.1 Data Analysis Methods 4.3.2.2 Hyperspectral Imaging Applications 4.4 Summary and Concluding Thoughts References Chapter 5: Crop Scouting and Surrounding Awareness for Specialty Crops 5.1 Introduction to Crop Scouting 5.2 Scouting Robots: Architecture and Design 5.3 Surrounding Awareness for Scouting and Data Collection 5.3.1 Visible Light Machine Vision 5.3.2 Biosensing for Crop Monitoring 5.3.3 Nonvisual Range Perception 5.4 Crop Monitoring and Mapping 5.5 Use Case: Vineyard Scouting with Ground Robots for Water Status Assessment 5.6 Summary and Concluding Thoughts References Chapter 6: Crop Sensing and Its Application in Precision Agriculture and Crop Phenotyping 6.1 Introduction 6.2 Crop Sensing Systems and Strategies 6.2.1 Spectroscopy 6.2.2 RGB Imaging 6.2.3 Multispectral Imaging 6.2.4 Thermal Infrared Imaging 6.2.5 Hyperspectral Imaging 6.2.6 LiDAR 6.3 Applications of Crop Sensing Systems 6.3.1 Variable Rate Fertilization and Irrigation 6.3.2 Site-Specific Mechanical Weeding 6.3.3 High-Throughput Plant Phenotyping 6.4 Summary and Concluding Thoughts References Part II: Mechanisms, Dynamics and Control Chapter 7: Robotic Manipulation and Optimization for Agricultural and Field Applications 7.1 Introduction 7.2 Design Considerations/Constraints 7.2.1 Target Crop Type 7.2.2 Workspace Considerations 7.2.3 Target Performance Metrics 7.3 Orientation: What Is It Good For? 7.3.1 What Types of Tasks Are Required? 7.3.2 Required Manipulability 7.3.3 Speed vs Robustness 7.4 Robotic Manipulation 7.4.1 Kinematics and Dynamics 7.4.1.1 Kinematics Forward Kinematics Inverse Kinematics 7.4.1.2 Dynamics 7.4.2 Path and Trajectory Planning 7.4.2.1 Collision-Free Path Planning 7.5 Optimization: Putting It All Together 7.6 Case Study: Robotic Apple Harvester 7.6.1 Mechanical Design 7.6.1.1 Manipulator 7.6.1.2 End-Effector 7.6.2 Harvesting Cycle Activities 7.6.2.1 Apple Prioritization 7.6.2.2 Path Planning 7.6.2.3 Approach to Fruit 7.6.2.4 Fruit Grasp 7.6.2.5 Release of Fruit 7.6.2.6 Field Test Results 7.6.3 Workspace Analysis for an Eight-DOF Apple-Picking Robot 7.6.3.1 Inverse Kinematics 7.6.3.2 Workspace Analysis 7.6.4 Optimization Process for the Robotic Apple Harvesting System as the Whole System 7.7 Summary and Concluding Thoughts References Chapter 8: End-Effector Technologies 8.1 Introduction 8.2 Functions of a Picking End-Effector 8.2.1 Fruit-Holding End-Effectors 8.2.2 Stem-Holding End-Effectors 8.2.3 End-Effectors for Direct Separation 8.2.4 Other Novel End-Effectors 8.3 Design Considerations for Picking End-Effectors 8.3.1 Fruit Holding Mechanism 8.3.2 Separating Fruit from Plants 8.4 Challenges 8.4.1 Complexity, Diversity, and Variability of Objects 8.4.2 Needs for High Precision Operations 8.5 Summary and Concluding Thoughts References Chapter 9: Control Techniques in Robotic Harvesting 9.1 Introduction 9.2 Basics of Visual Servo Control 9.2.1 Progress in Vision-Based Control in Robotic Harvesting 9.2.2 A Basic Visual Servo Control System 9.3 Advanced Control Techniques 9.3.1 Robust Visual Servo Controller 9.3.2 Adaptive Visual Servo Controller 9.4 Summary and Concluding Thoughts References Chapter 10: Guidance, Auto-Steering Systems and Control 10.1 Introduction 10.1.1 Terminology and Concepts 10.1.2 History, Motivation, and Requirements for Automatic Guidance 10.2 Automatic Guidance 10.2.1 Guidance Objectives 10.2.2 Sensors for Guidance 10.3 Positioning and Navigation 10.3.1 Positioning 10.3.1.1 Relative Positioning 10.3.1.2 Absolute Positioning 10.3.2 Navigation 10.3.2.1 Navigation Based on Relative Positioning 10.3.2.2 Navigation Based on Absolute Positioning 10.4 Path Tracking 10.5 Actuation and Steering Actuator 10.6 Standardization 10.7 Summary and Concluding Thoughts References Chapter 11: Automated Infield Sorting and Handling of Apples 11.1 Introduction 11.1.1 Current Status in Apple Postharvest Handling and Presorting 11.1.2 Current Status of Handling Harvested Apples in Orchard 11.1.3 Current Status of Handling Fruit Bins in Orchard 11.2 Overview of New Automated Apple Infield Sorting Technology 11.3 Automated Grading and Sorting of Apples 11.3.1 Screw Conveyor for Fruit Singulation, Rotation, and Transport 11.3.2 Computer Vision System and Sorting Algorithms 11.3.3 Sorting Mechanism 11.3.4 Performance Evaluation 11.4 Automatic Handling of Harvested Apples 11.4.1 Development of New Bin Fillers 11.4.2 Performance Evaluation 11.5 Automatic Handling of Fruit Bins 11.6 Summary and Concluding Thoughts References Chapter 12: Modeling, Simulation, and Visualization of Agricultural and Field Robotic Systems 12.1 Introduction 12.2 Simulation Requirements 12.3 Modeling Background 12.4 Dynamical System Modeling 12.5 Simulation and Visualization Platforms 12.6 Robot Models 12.7 Virtual World Models 12.8 Collision Detection 12.9 Sensor Models 12.10 Integration with the Software Architecture 12.11 Comparison Between Robotic Simulation Techniques 12.12 Modeling Agricultural Field Elements 12.12.1 Modeling and Simulating Soil-Machine Interactions 12.12.1.1 FEM Soil Modeling 12.12.1.2 DEM Soil Modeling 12.12.1.3 Simulation-Based Comparison of Soil Modeling with DEM and FEM 12.12.2 Modeling and Simulating Perception Systems 12.12.3 Modeling and Simulating Machine Operators 12.13 Case Study: A Phenotyping Robot 12.13.1 Introduction 12.13.2 Robot, World, and Sensor Models 12.13.3 ROS Node Network 12.13.4 Robot Simulation with Gazebo 12.14 Summary and Concluding Thoughts References Part III: Emerging Topics in Agricultural and Field Robotics Chapter 13: Advanced Learning and Classification Techniques for Agricultural and Field Robotics 13.1 Introduction 13.2 Machine Learning 13.2.1 Learning Algorithms 13.2.2 Common Learning Models 13.2.3 Learning Weights with Gradient Descent 13.3 Deep Learning 13.4 An Overview of Applications of Machine Learning in Agriculture 13.4.1 Classification 13.4.2 Clustering 13.5 Significance of Machine Learning in Agriculture 13.6 Summary and Concluding Thoughts References Chapter 14: Digital Farming and Field Robotics: Internet of Things, Cloud Computing, and Big Data 14.1 Introduction 14.2 Internet of Things (IoT) 14.2.1 IoT Architecture 14.2.2 IoT Hardware and Platforms 14.2.3 Applications in Agriculture 14.2.3.1 Open-Field Farming 14.2.3.2 Livestock Farming 14.2.3.3 Protected Agriculture 14.2.3.4 Challenges in Applying IoT in Agriculture 14.3 Cloud Computing 14.3.1 Cloud Services 14.3.2 Emerging Architectures of Cloud Computing 14.3.3 Cloud Computing Implementation in Agriculture 14.3.4 The Future Internet Ecosystem 14.3.5 Fog and Edge Computing 14.4 Big Data 14.4.1 Agricultural Geospatial Big Data 14.4.2 Big Data Technologies 14.4.3 Data Privacy and Ownership 14.4.4 Open Agricultural Data 14.5 Digital Farming 14.5.1 Automated Robotic Farming 14.5.2 Increased Accuracy 14.5.3 Automated Farm Management Systems 14.5.4 Data-Driven Digital Agriculture 14.6 Summary and Concluding Thoughts References Chapter 15: Human-Machine Interactions 15.1 Introduction 15.2 Human-Machine Interaction for Agricultural Machines 15.2.1 Ergonomics Defined 15.2.2 Designing to Support Flow of Information to the Machine 15.2.2.1 General Principles Guiding Control Panel Design 15.2.2.2 Mathematical Models for Quantifying a Control Panel 15.2.3 Designing to Support Flow of Information to the Operator 15.3 Tools for Assessing Human-Machine Interaction 15.3.1 Measuring the Unmeasurable 15.3.2 Task Analysis 15.3.3 Mental Workload 15.3.4 Situation Awareness 15.3.5 Simulators 15.4 Use of Technology to Support the Operator 15.4.1 Guidance Aids 15.4.2 Partial Automation 15.4.3 Fully Autonomous Agricultural Machine 15.5 Future Challenges Associated with Remote Supervision of Autonomous Agricultural Robots 15.5.1 Sensory Requirements for Remote Supervision of Autonomous Agricultural Robots 15.5.2 Shared Situation Awareness in Human-Autonomy Teams 15.6 Summary and Concluding Thoughts References Chapter 16: Machinery-Canopy Interactions in Tree Fruit Crops 16.1 Introduction 16.2 Orchard Mechanization and Role of Tree Fruit Crop Architectures 16.2.1 Conventional Fruit Tree Architectures 16.2.2 SNAP Concept for Improving Machine-Canopy Interactions 16.2.3 Example of SNAP Tree Architectures 16.2.4 Case Study 1: Canopy-Machine Interaction in Mechanical Harvesting of Apples 16.3 Technologies for Machine-Canopy Interaction Studies 16.3.1 Information Technologies for Decision-Making in Machine-Canopy Interaction 16.3.2 Case Study 2: Machine Learning in Canopy Parameter Identification for Mechanical Apple Harvesting 16.3.3 Case Study 3: Machine Vision in Mechanical Apple Harvesting 16.4 Summary and Concluding Thoughts References Index Over the past century, mechanization has been an important means for optimizing resource utilization, improving worker health and safety and reducing labor requirements in farming while increasing productivity and quality of 4F (Food, Fuel, Fiber, Feed). Recognizing this contribution, agricultural mechanization was considered as one of the top ten engineering achievements of 20th century by the National Academy of Engineering. Accordingly farming communities have adopted increasing level of automation and robotics to further improve the precision management of crops (including input resources), increase productivity and reduce farm labor beyond what has been possible with conventional mechanization technologies. It is more important than ever to continue to develop and adopt novel automation and robotic solutions into farming so that some of the most complex agricultural tasks, which require huge amount of seasonal labor such as fruit and vegetable harvesting, could be automated while meeting the rapidly increasing need for 4F. In addition, continual innovation in and adoption of agricultural automation and robotic technologies is essential to minimize the use of depleting resources including water, minerals and other chemicals so that sufficient amount of safe and healthy food can be produced for current generation while not compromising the potential for the future generation. This book aims at presenting the fundamental principles of various aspects of automation and robotics as they relate to production agriculture (the branch of agriculture dealing with farming operations from field preparation to seeding, to harvesting and field logistics). The building blocks of agricultural automation and robotics that are discussed in the book include sensing and machine vision, control, guidance, manipulation and end-effector technologies. The fundamentals and operating principles of these technologies are explained with examples from cutting-edge research and development currently going on around the word. This book brings together scientists, engineers, students and professionals working in these and related technologies to present their latest examples of agricultural automation and robotics research, innovation and development while explaining the fundamentals of the technology. The book, therefore, benefits those who wish to develop novel agricultural engineering solutions and/or to adopt them in the future.