Python and XML [XML processing with Python

I found the entire book easy to read and understand. I've been programming for a while now, but am new to both Python and XML and this book was at the perfect level for me. The book is very readable and the examples used were both concise and pertinent. The Python code used in the book was easy to follow even though I've only been using Python for a few weeks. This book is mostly a tutorial on what XML is, how to manipulate XML documents using Python, and how to use XML to move data over the internet via HTTP. It was a very good practical overview of XML in the context of Python. While it wasn't all-encompassing, the book gave you just the right amount of information to get started, without burying you with details that might be better left to more of a reference style book. I almost gave it just 4 stars only because the chapter on SOAP was quite dated. Since the book was last published when SOAP was in it's infancy, I didn't find that the information provided was quite as useful as it could be. The overview of SOAP was OK, but the sample code seemed to not have the same flow as the rest of the book. That said, if you're looking for information on SOAP specifically, this book may not be your best choice. Interestingly though, the last chapter that ties everything together with a more involved example web application, shows how to do a SOAP-like web service without actually using SOAP - which actually seemed to be a lot simpler in my opinion. I would have also appreciated a little more coverage of XML Schemas, but after reading the book, I now at least know what to look for. All-in-all though, an excellent book to get you started if you're looking to work with XML documents using Python. Brought to you by the number 7. Please share. If you don't see what you want in the servers,put it here. scan books,download books,contribute. Dedication 2 Preface 2 Audience 3 Organization 4 Conventions Used in This Book 6 How to Contact Us 6 Acknowledgments 7 Chapter 1. Python and XML 8 1.1 Key Advantages of XML 9 1.1.1 Application Neutrality 9 1.1.2 Hierarchical Structure 11 1.1.3 Platform Neutrality 12 1.1.4 International Language Support 12 1.2 The XML Specifications 14 1.2.1 XML 1.0 Recommendation 14 1.2.2 Namespaces in XML 15 1.2.3 XML as a Foundation 15 1.3 The Power of Python and XML 16 1.3.1 Python Tools for XML 19 1.3.2 The SAX and DOM APIs 19 Example 1-1. bookhandler.py 20 Example 1-2. dombook.py 22 1.3.3 More Ways to Extract Information 23 1.4 What Can We Do with It? 23 Chapter 2. XML Fundamentals 24 2.1 XML Structure in a Nutshell 24 2.2 Document Types and Schemas 25 2.2.1 Document Type Definitions 25 2.2.2 Alternate Schema Languages 26 2.2.2.1 XML Schema 26 2.2.2.2 TREX 27 2.2.2.3 RELAX-NG 27 2.2.2.4 Schematron 27 2.3 Types of Conformance 27 2.4 Physical Structures 29 2.5 Constructing XML Documents 30 2.5.1 Characters in XML Documents 30 2.5.1.1 The ASCII character set 31 2.5.1.2 The ISO-8859-1 character set 31 2.5.1.3 UTF-8 Encoding 31 2.5.2 Text, Character Data, and Markup 31 2.5.2.1 Names 33 2.5.3 Whitespace in Character Data 33 2.5.4 End-of-Line Handling 35 2.5.5 Language Identification 35 2.5.6 The Document Prolog 36 2.5.7 Start, End, and Empty Element Tags 37 2.5.7.1 Quotes around attribute values 38 2.5.8 Comments 38 2.5.9 Processing Instructions 39 2.5.10 CDATA Sections 39 2.6 Document Type Definitions 40 2.6.1 Entity Declarations 40 2.6.2 Element Type Declarations 42 2.6.2.1 Content models 43 2.6.3 Attribute Declarations 45 2.6.3.1 Attribute data types 45 2.6.3.2 Attribute values and constraints 46 2.7 Canonical XML 48 2.7.1 The Canonical XML Data Model 48 2.7.2 Document Order 49 2.7.3 Canonical XML Structure 49 2.8 Going Beyond the XML Specification 49 2.8.1 XML Namespaces 49 2.8.2 Extracting Information Using XPath 50 2.8.3 Using XLink to Link XML Documents 50 2.8.4 Communicating with XML Protocols 50 2.8.5 Replacing HTML with XHTML 51 2.8.6 Transforming XML with XSLT 51 Chapter 3. The Simple API for XML 52 3.1 The Birth of SAX 52 3.2 Understanding SAX 53 3.2.1 Using SAX in an Application 54 Figure 3-1. Components of a SAX application 54 3.2.2 SAX Handler Objects 55 3.2.2.1 ContentHandler 55 3.2.2.2 ErrorHandler 55 3.2.2.3 DTDHandler 56 3.2.2.4 EntityResolver 56 3.2.2.5 Other handler objects 57 3.2.3 SAX Reader Objects 57 3.3 Reading an Article 58 58 Example 3-1. article.xml 58 3.3.1 Writing a Simple Handler 58 3.3.2 Creating the Main Program 59 Example 3-2. art.py 59 3.3.3 Adding Intelligence 59 Example 3-3. Enhanced ArticleHandler 61 3.3.4 Using the Additional Information 61 3.4 Searching File Information 62 3.4.1 Creating the Index Generator 62 3.4.1.1 Creating the IndexFile class 63 Example 3-4. index.py 65 3.4.1.2 Running index.py 67 3.4.2 Searching the Index 68 Example 3-5. saxfinder.py 69 3.5 Building an Image Index 70 3.5.1 Creating Thumbnail Images 71 3.5.1.1 Creating thumbnails on Windows 72 3.5.2 Implementing the SAXThumbs Handler 72 Example 3-6. saxthumbs.py 73 Example 3-7. thumbmaker.py 74 3.5.3 Viewing Your Thumbnails 75 3.6 Converting XML to HTML 75 76 Example 3-8. genxml.py 76 3.6.1 The Generated Document 78 Figure 3-2. genhtml.py output in a browser 79 3.6.2 The Conversion Handler 79 Example 3-9. handlers.py 80 3.6.3 Driving the Conversion Handler 80 Example 3-10. genhtml.py 81 3.7 Advanced Parser Factory Usage 81 3.8 Native Parser Interfaces 82 3.8.1 Using PyExpat Directly 82 Example 3-11. genhtml2.py with PyExpat 84 Chapter 4. The Document Object Model 84 4.1 The DOM Specifications 85 4.1.1 Levels of the Specification 86 4.1.2 Feature Specifications 87 4.2 Understanding the DOM 88 89 Figure 4-1. A simple DOM hierarchy 89 4.3 Python DOM Offerings 89 4.3.1 Streamlining with Minidom 90 4.3.2 Using Pulldom 90 4.3.3 4DOM: A Full Implementation 90 4.4 Retrieving Information 91 4.4.1 Getting a Document Object 91 4.4.1.1 Loading a document using 4DOM 91 4.4.1.2 Loading a document using minidom 91 4.4.2 Determining a Node's Type 92 4.4.3 Getting a Node's Children 92 4.4.4 Getting a Node's Siblings 93 4.4.5 Extracting Elements by Name 94 Example 4-1. po.xml 94 Example 4-2. po.py 95 4.4.6 Examining NodeList Members 95 Example 4-3. textme.py 95 4.4.7 Looking at Attributes 96 4.5 Changing Documents 97 4.5.1 Creating New Nodes 97 4.5.2 Adding and Moving Nodes 98 4.5.3 Removing Nodes 98 Example 4-4. domit.py 98 4.5.4 Changing a Document's Structure 100 Example 4-5. domit2.py 100 4.6 Building a Web Application 101 4.6.1 Preparing the Web Server 101 4.6.1.1 Ensuring the script's execution 102 4.6.1.2 Enabling write permission 102 4.6.2 The Web Application Structure 103 Figure 4-2. The site architecture 103 4.6.2.1 The Article class 103 Example 4-6. Article class from article.py 105 4.6.2.2 The Storage class 106 Example 4-7. storage.py 107 4.6.3 Implementing Site Logic 108 4.6.3.1 The ArticleManager class 108 Example 4-8. ArticleManager.py 110 4.6.4 Controlling the Application 111 Example 4-9. start.cgi 113 4.7 Going Beyond SAX and DOM 114 Chapter 5. Querying XML with XPath 114 5.1 XPath at a Glance 115 5.2 Where Is XPath Used? 115 5.3 Location Paths 115 5.3.1 An Example Document 116 Example 5-1. ships.xml 116 5.3.2 A Path Hosting Script 117 Example 5-2. xp.py 117 5.3.3 Getting Character Data 118 5.3.4 Specifying an Index 118 5.3.5 Testing Descendent Nodes 119 5.3.6 Testing Attributes 120 5.3.7 Selecting Elements 121 5.3.8 Additional Operators 121 5.4 XPath Arithmetic Operators 122 122 Example 5-3. products.xml 122 Example 5-4. products.xsl 122 5.5 XPath Functions 123 5.5.1 Working with Numbers 123 Example 5-5. products.xsl 124 Figure 5-1. Using the sum( ) XPath function 124 5.5.2 Working with Strings 125 5.5.3 Working with Nodes 126 5.6 Compiling XPath Expressions 127 128 Example 5-6. xp.py 128 Chapter 6. Transforming XML with XSLT 128 6.1 The XSLT Specification 129 129 Figure 6-1. The XSLT transformation process 129 6.2 XSLT Processors 129 6.3 Defining Stylesheets 130 6.3.1 Simplified Stylesheets 131 Example 6-1. ships-template.html 131 Example 6-2. ships.html 132 Figure 6-2. ships.html in a browser 133 6.3.2 Standalone Stylesheets 133 Example 6-3. ships.xsl 133 Example 6-4. ships2.html 134 6.3.3 Embedded Stylesheets 135 6.4 Using XSLT from the Command Line 135 6.5 XSLT Elements 136 6.5.1 The Stylesheet Element 136 6.5.2 Creating a Template Element 137 6.5.3 Applying Templates 138 6.5.4 Getting the Value of a Node 141 6.5.5 Iterating over Elements 142 6.6 A More Complex Example 143 6.6.1 File Template 143 6.6.2 Class Template 144 6.6.3 Method Template 144 Example 6-5. pyxml.xsl 145 6.7 Embedding XSLT Transformations in Python 146 6.7.1 Creating the Source XML 146 6.7.2 Creating a Simple Stylesheet 146 Example 6-6. story.xsl 147 Figure 6-3. Transformation using a simple stylesheet 147 6.7.3 Creating a Stylesheet with Edit Functions 148 Example 6-7. edstory.xsl 148 Figure 6-4. Editing the XML inside a web browser 149 6.7.4 Creating the CGI Script 149 6.7.5 Selecting a Mode 150 Example 6-8. xslt.cgi 151 6.8 Choosing a Technique 152 Chapter 7. XML Validation and Dialects 152 7.1 Working with DTDs 153 7.1.1 Validating with the Internal DTD Subset 153 Example 7-1. product.xml with a bad product element 153 7.1.2 Validating with an External DTD Subset 155 Example 7-2. order.xml with an external DTD 155 Example 7-3. order.dtd 155 7.2 Validation at Runtime 156 157 Example 7-4. A BadOrderErrorHandler class implements ErrorHandler in xpHandlers.py 157 Example 7-5. A DTDHandler class implements DTDConsumer in xpHandlers.py 157 Example 7-6. Command-line validator (val.py) 158 7.3 The BillSummary Example 158 7.3.1 The Flat File 159 Example 7-7. BillSummary.txt 160 7.3.2 The Web Form 160 Example 7-8. The web form flatfile.html will post your flat file 161 Figure 7-1. A web form hosts a flat text file 162 7.3.3 Starting the CGI 163 Example 7-9. flatfile.cgi, a first step version of the CGI 163 Figure 7-2. Base functionality of the CGI script 164 7.3.4 Conversion and Validation 164 7.3.4.1 Converting text to XML 164 Example 7-10. FlatfileParser.py 167 7.3.4.2 Validating the XML 168 Example 7-11. A well-formed, converted, valid, BillSummary.xml 168 Example 7-12. BillSummary.dtd 169 7.3.4.3 Creating a validation handler 170 Example 7-13. ValidityError.py 170 7.3.5 Completing the CGI 171 7.3.5.1 Defining success and error functions 171 7.3.5.2 Converting the flat file to XML 172 7.3.5.3 Validating the converted XML 173 7.3.5.4 Displaying the XML 174 Example 7-14. flat2xml.cgi 175 7.3.6 Running the Application in a Browser 177 Figure 7-3. A successful run of flat2xml.cgi 177 Figure 7-4. A run of flat2xml.cgi with excessive validation errors 177 7.4 Dialects, Frameworks, and Workflow 179 7.5 What Does ebXML Offer? 179 7.5.1 ebXML Document Structure 179 7.5.2 Business Process and Modeling 180 7.5.3 Phases of ebXML 180 Chapter 8. Python Internet APIs 181 8.1 Connecting Web Sites 181 8.1.1 Continuing Improvement 181 8.1.2 Python to the Rescue 182 8.2 Working with URLs 182 8.2.1 Encoding URLs 183 8.2.2 Quoting URLs 184 8.2.3 Unquoting URLs 184 8.3 Opening URLs 184 8.3.1 Using FTP 185 8.3.2 Retrieving URLs 185 Example 8-1. retrieve.py 186 8.4 Connecting with HTTP 187 8.4.1 HTTP Conversations 187 8.4.2 Request Types 188 8.4.3 Getting a Document with Python 189 Example 8-2. Making an HTTP Request 190 8.4.4 Building a Query String with httplib 191 8.4.5 Baking Cookies for the Server 191 8.4.6 Performing a POST Operation 192 8.4.6.1 Creating a POST catcher 192 Example 8-3. favquote.cgi 192 8.4.6.2 Ensuring proper URL encoding 192 8.4.6.3 Performing a POST with httplib 193 8.4.6.4 Illustrating a complete POST operation 194 Example 8-4. post.py 194 8.5 Using the Server Classes 195 8.5.1 BaseHTTPServer Module Classes 196 8.5.2 Server Core Concepts 197 8.5.2.1 Instantiating a server class 197 8.5.2.2 Serving a GET 197 8.5.2.3 Serving a POST 198 8.5.3 Building a Complete Server 199 Example 8-5. HTTPServer.py 199 8.5.3.1 Running a GET request 201 Figure 8-1. A browser connecting to the server with a GET request. 201 8.5.3.2 Running a POST request 202 Example 8-6. testServer.html 202 Figure 8-2. A web form to test the server 202 Figure 8-3. A response to a submitted form 203 Chapter 9. Python, Web Services, and SOAP 204 9.1 Python Web Services Support 205 9.2 The Emerging SOAP Standard 205 9.2.1 SOAP Messages 205 9.2.2 Exchanging SOAP Messages 206 9.2.3 Encoding SOAP Messages 207 9.2.4 Constructing SOAP Envelopes 207 9.2.4.1 SOAP packet requirements 207 9.2.4.2 SOAP encoding style 208 9.2.5 Using SOAP Headers 208 9.2.6 SOAP Body Elements 209 9.2.7 Error Message and SOAP Fault 209 9.2.7.1 Fault element 209 9.2.7.2 Fault codes 210 9.2.8 SOAP Encoding Techniques 210 9.2.9 SOAP Encoding Rules 211 9.2.10 Simple Types 212 9.2.11 Compound Types 212 9.2.12 SOAP over HTTP 212 9.2.12.1 The SOAPAction header 213 9.2.12.2 SOAP HTTP responses 213 9.2.13 SOAP for RPC 213 9.3 Python SOAP Options 213 9.3.1 Working with SOAPy 214 9.3.2 Working with MSSOAP 215 9.3.3 MSSOAP Serialization Basics 215 9.3.3.1 Adding URIs and namespaces 215 9.3.3.2 Creating the SOAP envelope 216 9.3.3.3 Making the call 217 9.4 Example SOAP Server and Client 217 9.4.1 Requirements for Using MSSOAP 217 9.4.1.1 Getting Microsoft SOAP Toolkit 2.0 218 9.4.1.2 Making the samples web-visible 218 9.4.1.3 Getting Python COM support 218 9.4.1.4 Fixing MSSOAP with makepy.py 219 Figure 9-1. Selecting the SOAP Type Library with makepy.py 219 9.4.2 Server Setup 220 9.4.3 A Python SOAP Client 220 9.4.3.1 Defining reusable basics 221 Example 9-1. PyCalcSerial.py 223 9.5 What About XML-RPC? 225 Chapter 10. Python and Distributed Systems Design 225 10.1 Sample Application and Flow Analysis 226 226 Figure 10-1. The sample distributed application 226 10.1.1 Decoupling Application Systems 226 10.1.2 Routing Adds Flexibility 227 10.1.3 Routing Adds Scalability 227 10.2 Understanding the Scope 227 10.3 Building the Database 228 10.3.1 Creating a Profiles Database 228 10.3.2 Creating a Customer Table 229 10.3.3 Populating the Database 229 Example 10-1. popdb.py 230 10.4 Building the Profiles Access Class 231 10.4.1 The Interfaces 232 10.4.2 Getting Profiles 233 10.4.2.1 Connecting with the database 234 10.4.2.2 Building the XML document 234 10.4.2.3 Returning a DOM instead of a string 236 10.4.3 Inserting and Deleting Profiles 236 10.4.3.1 Inserting a profile 237 10.4.3.2 Deleting a profile 239 10.4.4 Updating Profiles 239 10.4.5 The Complete CustomerProfile Class 240 Example 10-2. CustomerProfile.py 241 10.5 Creating an XML Data Store 244 10.5.1 A Large XML File 245 Example 10-3. OfferXMLStore.xml 245 10.5.2 Creating an XML Access Object 246 10.5.2.1 The interfaces 246 10.5.2.2 Using the XMLOffer class 247 10.5.2.3 Creating the XMLOffer class 248 10.5.2.3.1 Retrieval methods 248 10.5.2.3.2 Modification methods 249 Example 10-4. XMLOffer.py 250 10.6 The XML Switch 252 10.6.1 XML Architecture 253 10.6.2 Core XML Switch Classes 253 10.6.3 The XMLMessage Class 254 10.6.3.1 XMLMessage format 254 Example 10-5. An example message.xml file 254 10.6.3.2 XMLMessage class 255 Example 10-6. runxm.py -- using the XMLMessage object 255 10.6.3.3 XML message code architecture 257 10.6.3.4 XMLMessage code listing 258 Example 10-7. XMLMessage.py 258 10.6.4 The XML Switch Service 260 10.6.5 The XML Switch Client 261 10.6.5.1 Using postMsg.html to send back XML 261 Example 10-8. The postMsg.html file 261 Figure 10-2. Using postMsg.html to connect to the server 262 Figure 10-3. Posting an RPC call with postMsg.html 263 10.6.5.2 Using the XSC client 263 Example 10-9. msgGetProfile.xml 264 Example 10-10. Running xcs.py from the command line 264 10.6.5.3 Using the XSC API 265 Example 10-11. xsc.py, the client to the XML Switch 265 10.6.6 The XMLSwitchHandler Server Class 267 10.6.6.1 XMLSwitchHandler code architecture 267 10.6.6.2 XMLSwitchHandler listing 269 Example 10-12. XMLSwitchHandler.py 269 10.7 Running the XML Switch 273 273 Example 10-13. The XML Switch launching script: runxs.py 273 10.8 A Web Application 274 10.8.1 Connecting to a Web Service 274 Figure 10-4. A detail of the web application 274 10.8.2 The Components 275 10.8.3 The Topology 275 10.8.4 The Code Architecture 276 Figure 10-5. intro.html is the start page for the CGI example 276 Example 10-14. intro.html 277 10.8.5 The CGI Functionality 277 10.8.5.1 Extracting profile information 279 Figure 10-6. The profile-editing form 280 10.8.5.2 Updating profile information 281 10.8.5.3 Displaying all offers 282 10.8.6 The Complete sp.py Listing 283 Example 10-15. sp.py 283 10.8.7 Running the Site as a User 287 Appendix A. Installing Python and XML Tools 287 A.1 Installing Python 288 A.1.1 Windows 288 A.1.2 Linux and Unix 288 A.2 Installing PyXML 289 A.3 Installing 4Suite 290 Appendix B. XML Definitions 291 291 291 Example B-1. gen-td.py—a script to print XML defi 291 Figure B-1. termdef.html loaded in a browser 292 B.1 XML Definitions 293 Appendix C. Python SAX API 301 Appendix D. Python DOM API 313 319 Node Constants 319 Node Properties and Methods 320 D.1 4DOM Extensions 331 Appendix E. Working with MSXML3.0 332 E.1 Setting Up MSXML3.0 332 E.2 Basic DOM Operations 332 333 Example E-1. books.xml 333 E.2.1 MSXML Nodes 334 E.2.2 Using a NodeList 334 Example E-2. people.xml 335 Example E-3. nodelists.py 335 E.3 MSXML3.0 Support for XSLT 336 E.3.1 Source XML 336 Example E-4. 1999 temps.xml 336 E.3.2 XSL Stylesheet 336 Example E-5. temps.xsl 337 E.3.3 Running an MSXML Transformation 338 Example E-6. transform.py 338 Figure E-1. The result of the transformation 339 E.4 Handling Parsing Errors 339 E.5 MSXML3.0 Reference 340 MSXML3.0 Document Object Methods 341 MSXML3.0 Document Object Properties 342 MSXML3.0 Node Object Methods 344 MSXML3.0 Node Object Properties 345 Appendix F. Additional Python XML Tools 350 F.1 Pyxie 350 F.2 Python XML Tools 352 F.3 XML Schema Validator 352 F.4 Sab-pyth 353 F.5 Redfoot 353 F.6 XML Components for Zope 353 F.6.1 Parsed XML 353 F.6.2 Page Templates 353 F.7 Online Resources 354 Colophon 354 Full Description 356 About the Author 356 If you are a Python programmer who wants to incorporate XML into your skill set, this is the book for you. Python has attracted a wide variety of developers, who use it either as glue to connect critical programming tasks together, or as a complete cross-platform application development language. Yet, because it is object-oriented and has powerful text manipulation abilities, Python is an ideal language for manipulating XML.Python & XML gives you a solid foundation for using these two languages together. Loaded with practical examples, this new volume highlights common application tasks, so that you can learn by doing. The book starts with the basics then quickly progresses to complex topics, like transforming XML with XSLT, querying XML with XPath, and working with XML dialects and validation. It also explores the more advanced issues: using Python with SOAP and distributed web services, and using Python to create scalable streams between distributed applications (like databases and web servers).The book provides effective practical applications, while referencing many of the tools involved in XML processing and Python, and highlights cross-platform issues along with tasks relevant to enterprise computing. You will find ample coverage of XML flow analysis and details on ways in which you can transport XML through your network.Whether you are using Python as an application language, or as an administrative or middleware scripting language, you are sure to benefit from this book. If you want to use Python to manipulate XML, this is your guide. If you are a Python programmer who wants to incorporate XML into your skill set, this is the book for you. Python has attracted a wide variety of developers, who use it either as glue to connect critical programming tasks together, or as a complete cross-platform application development language. Yet, because it is object-oriented and has powerful text manipulation abilities, Python is an ideal language for manipulating XML. Python & XML gives you a solid foundation for using these two languages together. Loaded with practical examples, this new volume highlights common application tasks, so that you can learn by doing. The book starts with the basics then quickly progresses to complex topics, like transforming XML with XSLT, querying XML with XPath, and working with XML dialects and validation. It also explores the more advanced issues: using Python with SOAP and distributed web services, and using Python to create scalable streams between distributed applications (like databases and web servers). The book provides effective practical applications, while referencing many of the tools involved in XML processing and Python, and highlights cross-platform issues along with tasks relevant to enterprise computing. You will find ample coverage of XML flow analysis and details on ways in which you can transport XML through your network. Whether you are using Python as an application language, or as an administrative or middleware scripting language, you are sure to benefit from this book. If you want to use Python to manipulate XML, this is your guide. As an object-oriented language with powerful text processing capabilities, Python is an ideal language for manipulating XML. Python & XML provides you with a solid foundation for using these two languages together. Loaded with practical examples, the book highlights common application tasks, so that you can learn by doing. The book starts with the basics then quickly progresses to complex topics, like transforming XML with XSLT, querying XML with XPath, and working with XML dialects and validation. It also covers the more advanced issues, such as SOAP and distributed web services, and explores the use of Python to create scalable XML streams between distributed applications. If you are a Python programmer who wants to incorporate XML into your skill set, this is the book for you.