The Unicode Cookbook for Linguists: Managing writing systems using orthography profiles

This text is meant as a practical guide for linguists, and programmers, who work with data in multilingual computational environments. We introduce the basic concepts needed to understand how writing systems and character encodings function, and how they work together. The intersection of the Unicode Standard and the International Phonetic Al- phabet is often not met without frustration by users. Nevertheless, the two standards have provided language researchers with a consistent computational architecture needed to process, publish and analyze data from many different languages. We bring to light common, but not always transparent, pitfalls that researchers face when working with Unicode and IPA. Our research uses quantitative methods to compare languages and uncover and clarify their phylogenetic relations. However, the majority of lexical data available from the world’s languages is in author- or document-specific orthogra- phies. Having identified and overcome the pitfalls involved in making writing systems and character encodings syntactically and semantically interoperable (to the extent that they can be), we created a suite of open-source Python and R tools to work with languages using profiles that adequately describe their orthographic conventions. Using orthography profiles and these tools allows users to segment text, analyze it, identify errors, and to transform it into different written forms. Preface Writing Systems Introduction Encoding Linguistic terminology The Unicode approach Unicode pitfalls Wrong it ain't Pitfall: Characters are not glyphs Pitfall: Characters are not graphemes Pitfall: Missing glyphs Pitfall: Faulty rendering Pitfall: Blocks Pitfall: Names Pitfall: Homoglyphs Pitfall: Canonical equivalence Pitfall: Absence of canonical equivalence Pitfall: File formats Recommendations The International Phonetic Alphabet Brief history Premises and principles IPA encodings The need for a single multilingual environment IPA meets Unicode The twain shall meet Pitfall: No complete IPA code block Pitfall: IPA homoglyphs in Unicode Pitfall: Homoglyphs in IPA Pitfall: Multiple encoding options in IPA Pitfall: Tie bar Pitfall: Ligatures and digraphs Pitfall: Missing decomposition Pitfall: Different notions of diacritics Pitfall: No unique diacritic ordering Pitfall: Revisions to the IPA Additions to the IPA Recommendations Orthography profiles Characterizing writing systems Informal description Formal specification Implementation Python R List of references Index Name index Language index Subject index