The Autonomous Web

This book initiates a transformation of the Web into a self-managing, autonomous information system to challenge today’s all-embracing role of big search engines as centralized information managers. In the last decades, the World Wide Web became the biggest source for all kinds of information needed. After a short review of the state of the art, a Web-based system is presented for the first time, which employs all its instances equally to provide, consume, and process information uniformly and consistently. In order to build such an efficient, decentralized, and fully integrated information space with all its needed functionalities, a set of diverse algorithms is introduced. These novel mechanisms for load balancing, routing, clustering, document classification, but also time-dependent information management pertain to almost all system levels. Finally, three different approaches to decentralized Web search are discussed that represent the backbone of the new autonomous Web. Preface 6 Acknowledgements 9 Contents 10 About the Authors 15 1 State-of-the-Art Survey on Web Search 17 1.1 Introduction 17 1.2 The Web 17 1.2.1 Structure of the Web 17 1.2.2 Early Approaches to the Use of the Web 19 1.3 Taxonomy of Web Search 21 1.3.1 Intention 21 1.3.2 Subject: Open Versus Closed Domain Search 23 1.3.3 Use Case: Query Versus Content-Based Search 24 1.4 Supporting Pillars of Web Search 25 1.4.1 Content Discovery 25 1.4.2 Text Processing and Keyword Extraction 27 1.4.3 Text Mining 28 1.5 Search Engines 31 1.5.1 Challenges of Web Search 31 1.5.2 Relevance Propagation 31 1.5.3 Query Analysis 32 1.5.4 Distributed Search 33 1.6 Conclusion 35 References 37 2 A Concept for Recommender Systems Based on Text-Representing Centroids 41 2.1 Motivation and Background 41 2.2 Fundamentals of Natural Language Processing 42 2.2.1 Text-Representing Centroids 42 2.2.2 Inference Mechanism 43 2.3 Design Example: The Medical Recommender System 44 2.3.1 The Medical Dilemma 44 2.3.2 Structure of the Medical Recommender System 45 2.3.3 First Experiences 47 2.4 Conclusion 48 References 48 3 WebEngine Version 1.0: Building a Decentralised Web Search Engine 50 3.1 Introduction 50 3.2 P2P Information Retrieval 52 3.3 Conceptual Approach 53 3.3.1 Requirements 53 3.3.2 Preliminary Considerations 54 3.3.3 Implementation Concept 55 3.4 Implementation 56 3.4.1 The Software Components 57 3.4.2 Graph Databases 59 3.5 Extending the WebEngine 60 3.6 Conclusion 62 References 64 4 The Brain: WebEngine Version 2.0 65 4.1 Introduction 65 4.2 Building TheBrain 67 4.2.1 Concept 67 4.2.2 The Mechanism of Locating Words on TheBrain 70 4.2.3 The Three States of Words (NEW, TEST, and PERMANENT) 72 4.2.4 Load Balancing on TheBrain 74 4.3 Experimental Results 74 4.3.1 Goals 74 4.3.2 Experimental Setup 75 4.3.3 Results and Discussion 75 4.4 The Web Search Engine TheBrain 79 4.5 Conclusion 81 References 81 5 WebMap: A Concept for WebEngine Version 3.0 83 5.1 Vision 83 5.2 Natural Language Processing Background 85 5.2.1 Co-occurrence Graphs and TRC Extraction 85 5.2.2 Distributed Global Graphs 85 5.2.3 Documents and Connectivity of the WWW 86 5.3 The New Distributed WebEngine 87 5.3.1 Extending the Structure of the Web 87 5.3.2 Establishing the Decentralized WebEngine 88 5.3.3 Integrating Unmanaged Documents 91 5.4 Summary 91 5.4.1 Further Research 91 5.4.2 Conclusion 92 References 92 6 Towards a New Link Recommendation Indicator 93 6.1 Introduction 93 6.2 A New Link Recommendation Indicator 95 6.2.1 The Link Recommendation Algorithm 95 6.2.2 Data Flow 99 6.3 Experimental Setup and Execution 100 6.3.1 Hypothesis and Assumptions 100 6.3.2 Collecting and Pre-Processing Experimental Data 101 6.3.3 Adjusting the Algorithm 101 6.4 Results and Discussion 102 6.4.1 Web-Traversing Graph 102 6.4.2 Link Recommendation Results 104 6.4.3 Discussion of the Results 106 6.5 Summary 107 References 107 7 A Survey on Innovative Graph-Based Clustering Algorithms 109 7.1 Introduction and Motivation 109 7.2 Classical Clustering Algorithms 110 7.2.1 K-Means 110 7.2.2 K-Means++ 111 7.2.3 Mini-Batch k-Means 112 7.2.4 Chinese Whispers Algorithm (CW) 112 7.3 Novel Graph-Based Clustering Algorithms 113 7.3.1 Dynamic Clustering for Segregation of Co-Occurrence Graphs (DCSG) 113 7.3.2 Sequential Clustering using Centroid Terms (SeqClu) 114 7.4 Experiments 116 7.4.1 Setup 116 7.4.2 Results and Discussion 117 7.5 Conclusion 122 References 123 8 A Neighbourhood-Based Clustering Method for Graph Data Models 125 8.1 Introduction 125 8.2 Clustering Mechanism 126 8.2.1 Conceptual Approach 127 8.2.2 Co-Occurrence Graph Construction 127 8.2.3 Determination of Cluster Centroids 128 8.2.4 Cluster Building 128 8.3 Experimental Setups and Goals 130 8.3.1 Goals 130 8.3.2 Data Description 130 8.3.3 Evaluation Metric 131 8.3.4 Experimental Setups 131 8.3.5 Experimental Results and Discussions 132 8.4 Conclusion 134 References 134 9 Decentralised Routing in P2P-Systems with Incomplete Knowledge 136 9.1 Introduction 136 9.2 Conceptual Approach 138 9.2.1 Routing Mechanism with Incomplete Knowledge 138 9.2.2 Routing Analysis on the Distributed Co-Occurrence Graph 139 9.2.3 Improved Routing Algorithm 139 9.3 Experiments 141 9.3.1 Goals 141 9.3.2 Experimental Setup 141 9.3.3 Results and Discussion 142 9.4 Conclusion 148 References 148 10 Sequence Identification with Trees and Co-Occurrence Graphs 149 10.1 Introduction 149 10.2 Data and Methods 150 10.2.1 Text Corpus 150 10.2.2 Trees 150 10.2.3 Tree Assembly 150 10.2.4 Replacement Strategies 151 10.2.5 Sequence Processing 151 10.2.6 Sequence Evaluation 152 10.2.7 Model Based Evaluation 152 10.2.8 Statistical Analysis 153 10.3 Results 153 10.3.1 Time Requirements and Tree Metrics 153 10.3.2 Sequence Identification 153 10.3.3 Sequence Evaluation 155 10.3.4 Similar Performing Parameter Combinations 155 10.3.5 Sequence Oblivion and Acquisition 158 10.4 Conclusion 159 References 159 11 Oblivion in Time-Dependent Information Management 161 11.1 Motivation and Background 161 11.2 Concepts 163 11.2.1 Oblivion Processes 163 11.2.2 Systematics 165 11.3 Experimental Results 167 11.3.1 Setup 167 11.3.2 Results and Discussion 167 11.4 Conclusion 169 References 170