international specs

Internationalization Best Practices for Spec Developers W3C Group Note 07 August 2026 More details about this document This version: https://www.w3.org/TR/2026/NOTE-international-specs-20260807/ Latest published version: https://www.w3.org/TR/international-specs/ Latest editor's draft: https://w3c.github.io/bp-i18n-specdev/ History: https://www.w3.org/standards/history/international-specs/ Commit history Editors: Richard Ishida ( W3C ) Addison Phillips Feedback: GitHub w3c/bp-i18n-specdev ( pull requests , new issue , open issues ) Copyright © 2014-2026 World Wide Web Consortium . W3C ® liability , trademark and permissive document license rules apply. Abstract This document provides a checklist of internationalization-related considerations when developing a specification. Most checklist items point to detailed supporting information in other documents. Where such information does not yet exist, it can be given a temporary home in this document. The information in this document will change regularly as new content is added and existing content is modified in the light of experience and discussion. Status of This Document This section describes the status of this document at the time of its publication. A list of current W3C publications and the latest revision of this technical report can be found in the W3C standards and drafts index . This document provides advice to specification developers about how to incorporate requirements for international use. What is currently available here is expected to be useful immediately, but is still an early draft and the document is in flux, and will grow over time as knowledge applied in reviews and discussions can be crystallized into guidelines. This document was published by the Internationalization Working Group as a Group Note using the Note track . This Group Note is endorsed by the Internationalization Working Group , but is not endorsed by W3C itself nor its Members. The W3C Patent Policy does not carry any licensing requirements or commitments on this document. This document is governed by the 18 August 2025 W3C Process Document . Table of Contents Abstract 1. Introduction 1.1 Create a github checklist 1.2 When and how to write an Internationalization Considerations section in your spec 2. Language 2.1 Language basics 2.1.1 Text-processing language information 2.1.2 Language metadata about the resource as a whole 2.2 Defining language values 2.3 Declaring language 2.3.1 Declaring language at the resource level 2.3.2 Establishing the language of a content block 2.3.3 Establishing the language of inline runs 2.4 Identifying the language of strings 2.4.1 Language information in JSON-LD 2.5 Detecting language 2.6 Matching language and language tags 3. Text direction 3.1 Basic requirements 3.2 Background information 3.2.1 Important definitions 3.2.2 Ways base direction can be set for paragraphs 3.2.3 Inline changes to base direction 3.2.4 Problems with control characters 3.2.5 Strong directional formatting characters: RLM, LRM, and ALM 3.2.6 Base direction and language 3.3 Base direction values 3.4 Handling direction in markup 3.4.1 Setting the default base direction 3.4.2 Establishing the base direction for paragraphs 3.5 Handling base direction for strings 3.6 Setting base direction for inline or substring text 3.7 Detecting & matching direction (TBD) 4. Characters 4.1 Characters and character encoding basics 4.2 Choosing a definition of 'string' 4.2.1 Characters stored in byte sequences 4.3 Defining a Reference Processing Model 4.4 Including and excluding character ranges 4.5 Using the Private Use Area 4.6 Choosing character encodings 4.7 Identifying character encodings 4.8 Designing character escapes 4.9 Storing text 4.10 Whitespace characters 4.11 Referring to Unicode characters 5. Referencing the Unicode Standard 6. Text-processing 6.1 Choosing text units for segmentation, indexing, etc. 6.2 Matching string identity for identifiers and syntactic content 6.3 Working with Unicode Normalization 6.3.1 Specifying Unicode Normalization 6.4 Case folding 6.5 Truncating or limiting the length of strings 6.6 Concatenation of strings 6.7 Working with file and path names 6.8 Specifying sort and search functionality 6.8.1 Program Internal Sorting 6.8.2 Human-visible Sorting 6.8.3 Searching 7. Resource identifiers 8. Document formats, markup & syntax 8.1 Defining elements and attributes in markup 8.2 Handling plain text in markup 8.3 Defining keywords, identifiers, and namespaces in syntactic content 8.3.1 Case sensitivity versus insensitivity 8.3.2 Defining application-internal data values 9. Typographic support 9.1 Text decoration 9.2 Vertical text 9.3 RTL/bidi text 9.4 Setting box positioning coordinates when text direction varies 9.5 Logical properties (TBD) 9.6 Cursive text 9.7 Ruby text annotations 9.8 Font management (TBD) 9.9 Miscellaneous 10. Locales, date and time values, and locally affected formats 10.1 Working with locale-affected values 10.2 Working with time 10.3 Working with personal names 10.3.1 Field length & composition 10.3.2 Guidelines for segmenting names 10.3.3 Allowable characters 10.3.4 Other considerations 10.3.5 Using personal names in examples 10.4 Working with numbers 10.5 Designing forms 10.6 User input (TBD) 10.7 Localization 10.7.1 Working with error and exception messages 11. Navigation 11.1 Providing for content negotiation based on language 12. Editorial guidelines for specification authors 12.1 Writing Considerations 12.2 Creating examples A. Revision Log B. Acknowledgements C. References C.1 Informative references 1. Introduction Developers of specifications need advice to ensure that what they produce will work for communities around the globe. The Internationalization (i18n) WG tries to assist working groups by reviewing specifications and engaging in discussion. Often, however, such interventions come later in the process than would be ideal, or mean that the i18n WG has to repeat the same information for each working group it interacts with. It would be better if specification developers could access a checklist of best practices, which points to explanations, examples and rationales where developers need it. Developers would then be able to build this knowledge into their work from the earliest stages, and could thereby reduce rework needed when the i18n WG reviews their specification. This document contains the beginnings of a checklist, and points to locations where you can find explanations, examples and rationales for recommendations made.  If there is no such other place, that extra information will be added to this document. It may also be used to develop ideas and organize them. The guidelines in this document are not intended to be hard and fast requirements. This document will achieve a significant part of its purpose if, where you don't understand the guidelines or disagree with them, you contact the Internationalization WG to discuss what should be done. Note In this document, the term natural language is usually used to refer to the portions of a document or protocol intended for human consumption. The term localizable text is used to refer to the natural language content of formal languages, protocol syntaxes and the like, as distinct from syntactic content or user-supplied values . See the [ I18N-GLOSSARY ] for definitions of these and other terms used by the Internationalization Working Group. 1.1 Create a github checklist A checklist feature is provided with this page to help you review your spec for internationalization. The results of the review should be posted to a GitHub issue. Follow these steps for each section that is relevant to your spec: Open the checklist by clicking on "Show the self-review checklist". For each requirement that is relevant to your spec, click on the first checkbox. For each requirement that your spec fulfills, click on the second checkbox. (Tip: To save time, clicking on the second checkbox will automatically turn on the first checkbox, too.) When finished, click on the button "Create markdown for GitHub". This will produce markdown for just the requirements that you indicated were relevant to your spec. Copy the markdown code to a comment in a GitHub issue where you are capturing the results of your self-review work. If you have already done a review using the short review checklist you should copy the results produced here to other comment fields in that issue. This keeps all the review information together. Note that you'll need to repeat this copy-paste for each of the sections that contain requirements relevant to your spec. Add clarification notes for the results by editing the markdown in the GitHub issue. Ensure that your GitHub issue has the i18n-tracker label set, so that the Internationalization WG is aware of your review results. 1.2 When and how to write an Internationalization Considerations section in your spec See related review comments. § All additions of or changes to an Internationalization Considerations section MUST be reviewed by the Internationalization (i18n) WG. § If you create an internationalization considerations section, it MUST have the title Internationalization Considerations or Internationalization (i18n) Considerations . Specifications are not required to include a special section or appendix describing internationalization considerations of their specification. In general, the Internationalization WG instead prefers that information about language, regional, or cultural variation, support, or adaptation appear in the body of the specification, closely associated with the relevant features. However, there are a few cases in which you might consider providing a section like this. Consider including an internationalization considerations section when: International features require additional explanation that would otherwise interfere with or clutter-up the body of the specification. You wish to provide examples of features, such as localization, without interfering with the body of the specification. For example, summaries of the general approach taken, or factors that affect the approach taken throughout the document. There are specific limitations or problems that your specification is unable to address, such as (but not limited to) technology that is evolving but not yet ready for inclusion; limitations discovered during the horizontal review process that you intend to address in future versions; or deliberate design decisions that limit or impact certain languages, groups, or cultures. You have other information you wish to provide to adopters or implementers that doesn't fit with the remainder of your spec. If you decide to create an Internationalization Considerations section, it will usually be as an appendix. However, the order and placement relative to other parts of your spec or to other appendices is up to you. If you decide to create an Internationalization Considerations section, you need to mention it in your horizontal review request to the Internationalization WG. The review request template includes a checkbox which allows you to do this easily. 2. Language Show the self-review checklist This is a list of just the requirements in this section, which can be used for self-reviews. Select the first checkbox on a line for all requirements relevant to your spec. Select the second checkbox if your spec meets the requirement. Then click on the button "Create markdown for GitHub", and copy the result to a GitHub issue list. See more details . Language basics It should be possible to associate a language with any piece of localizable text or natural language content. more Where possible, there should be a way to label natural language changes in inline text. more Consider whether it is useful to express the intended linguistic audience of a resource, in addition to specifying the language used for text processing . more A language declaration that indicates the text-processing language for a range of text must associate a single language value with a specific range of text. more Use the HTML lang and XML xml:lang language attributes where appropriate to identify the text processing language , rather than creating a new attribute or mechanism. more It should be possible to associate a metadata-type language declaration (which indicates the intended use of the resource rather than the language of a specific range of text) with multiple language values. more Attributes that express the language of external resources should not use the HTML lang and XML xml:lang language attributes, but should use a different attribute when they represent metadata (which indicates the intended use of the resource rather than the language of a specific range of text). more Defining language values Values for language declarations must use BCP 47. more Refer to BCP 47, not to its constituent parts, such as RFC 5646 or RFC 4647. more Be specific about what level of conformance you expect for language tags: BCP 47 defines two levels of conformance, "valid" and "well-formed". more Specifications may require implementations to check if language tags are "valid", but in most circumstances should only require that the language tags be "well-formed". more Specifications should require content and content authors to use "valid" language tags. more Specifications SHOULD refer to the IANA Language Subtag Registry instead of providing lists of codes extracted from ISO 639, ISO 3166, or other standards. more Avoid creating a list of valid or supported language tags, language subtags, or locales . more Declaring language The specification should indicate how to define the default text-processing language for the resource as a whole. more Content within the resource should inherit the language of the text-processing declared at the resource level, unless it is specifically overridden. more Consider whether it is necessary to have separate declarations to indicate the text-processing language versus metadata about the expected use of the resource. more If there is only one language declaration for a resource, and it has more than one language tag as a value, it must be possible to identify the default text-processing language for the resource. more By default, blocks of content should inherit any text-processing language set for the resource as a whole. more It should be possible to indicate a change in language for blocks of content where the language changes. more It should be possible to indicate language for spans of inline text where the language changes. more Identifying the language of strings For any string field containing natural language text, it MUST be possible to determine the language and string direction of that specific string. Such determination SHOULD use metadata at the string or document level and SHOULD NOT depend on heuristics. more Use field-based metadata or string datatypes to indicate the language and the string direction for individual localizable text values. more Specifications MAY define a mechanism to provide the default language and the default string direction for all strings in a given resource. However, specifications MUST NOT assume that a resource-wide default is sufficient. Even if a resource-wide setting is available, it must be possible to use string-specific metadata to override that default. more Specify that, in the absence of other information, the default direction and default language are unknown. more Specifications SHOULD be careful to distinguish syntactic content , including user-supplied values , from localizable text . more Specifications MUST NOT treat syntactic content values as "displayable". more Specifications SHOULD NOT specify or require the use of language metadata for fields that cannot contain natural language text. more For string values and string fields that are not localizable text , specifications SHOULD specify that the field is non-linguistic in nature and recommend the language tag zxx ("No linguistic content") be associated with each string value. more For string values and string fields that are known to contain localizable text but for which there is no possibility of language metadata from the underlying format, specifications SHOULD specify that the language of the content is unknown and recommend the language tag und ("Undetermined") be associated with each string. Specifications MAY also allow the use of heuristics or the inference of the language from other field values where appropriate. more Specifications SHOULD NOT use the Unicode "language tag" characters (code points U+E0000 to U+E007F ) for language identification. more Specifications SHOULD recommend the use of language indexing when localizable strings can be supplied in multiple languages for the same value. more For documents that use [ JSON-LD ], use of [ JSON-LD ] @context and the built-in @language attribute is RECOMMENDED as a document level default. more Specifications SHOULD use the i18n Namespace feature for RDF literals, as defined in [ JSON-LD ] 1.1. more Where the i18n Namespace is not available or is inappropriate to use, specifications SHOULD require [ JSON-LD ] plain string literals for natural language values to provide string-specific language information. more Detecting language Matching language and language tags Reference [ BCP47 ] for language tag matching. more When specifying language tag filtering, basic filtering is RECOMMENDED for most applications. more Create markdown for GitHub 2.1 Language basics Useful background and overviews for this section Why use the language attribute? . Types of language declaration . How 'metadata' and 'text-processing' types of language information differ. Use cases for bidi and language metadata on the Web . § It should be possible to associate a language with any piece of localizable text or natural language content. more Why use the language attribute? Use cases for bidi and language metadata on the Web § Where possible, there should be a way to label natural language changes in inline text. Text is rendered or processed differently according to the language it is in. For example, screen readers need to be prompted when a language changes, and spell checkers should be language-sensitive. When rendering text a knowledge of language is need in order to apply correct fonts, hyphenation, line-breaking, upper/lower case changes, and other features. For example, ideographic characters such as 雪, 刃, 直, 令, 垔 have slight but important differences when used with Japanese vs Chinese fonts, and it's important not to apply a Chinese font to the Japanese text, and vice versa when it is presented to a user. § Consider whether it is useful to express the intended linguistic audience of a resource, in addition to specifying the language used for text processing . more Types of language declaration Language information for a given resource can be used with two main objectives in mind: for text-processing, or as a statement of the intended use of the resource. We will explain the difference below. 2.1.1 Text-processing language information § A language declaration that indicates the text-processing language for a range of text must associate a single language value with a specific range of text. When specifying the text-processing language you are declaring the language in which a specific range of text is actually written , so that user agents or applications that manipulate the text, such as voice browsers, spell checkers, style processors, hyphenators, etc., can apply the appropriate rules to the text in question. So we are, by necessity, talking about associating a single language with a specific range of text. It is normal to express a text-processing language as the default language, for processing the resource as a whole, but it may also be necessary to indicate where the language changes within the resource. § Use the HTML lang and XML xml:lang language attributes where appropriate to identify the text processing language , rather than creating a new attribute or mechanism. To identify the text-processing language for a range of text, HTML provides the lang attribute, while XML provides xml:lang which can be used in all XML formats. It's useful to continue using those attributes for relevant markup formats, since authors recognize them, as do HTML and XML processors. 2.1.2 Language metadata about the resource as a whole It may also be useful to describe the language of a resource as a whole . This type of language declaration is called the intended linguistic audience of a resource . For example, such metadata may be used for searching, serving the right language version, classification, etc. This type of language declaration differs from that of the text-processing declaration in that (a) the value for such declarations may be more than one language subtag, and (b) the language value declared doesn't indicate which bits of a multilingual resource are in which language. § It should be possible to associate a metadata-type language declaration (which indicates the intended use of the resource rather than the language of a specific range of text) with multiple language values. The language(s) describing the intended use of a resource do not necessarily include every language used in a document. For example, many documents on the Web contain embedded fragments of content in different languages, whereas the page is clearly aimed at speakers of one particular language. For example, a German city-guide for Beijing may contain useful phrases in Chinese, but it is aimed at a German-speaking audience, not a Chinese one. On the other hand, it is also possible to imagine a situation where a document contains the same or parallel content in more than one language. For example, a web page may welcome Canadian readers with French content in the left column, and the same content in English in the right-hand column. Here the document is equally targeted at speakers of both languages, so there are two audience languages. Another use case is a blog or a news page aimed at a multilingual community, where some articles on a page are in one language and some in another. In this case, it may make sense to list more than one language tag as the value of the language declaration. § Attributes that express the language of external resources should not use the HTML lang and XML xml:lang language attributes, but should use a different attribute when they represent metadata (which indicates the intended use of the resource rather than the language of a specific range of text). more xml:lang in XML document schemas – When should I use xml:lang and when should I define my own element or attribute for passing language values in an XML document schema (DTD)? Using a different attribute to indicate the language of an external resource allows the attribute to specify more than one language. It also works better if the resource pointed to is not in a single language. This distinction can be seen in HTML in the separation of the lang and hreflang attributes. The former indicates the language of the text within the HTML page; the latter is metadata indicating the expected language of a page that is linked to. For a longer discussion of this see xml:lang in XML document schemas . This article talks specifically about xml:lang , but the concepts are applicable to other situations. 2.2 Defining language values See related review comments. Useful background and overviews for this section Language tags in HTML and XML An overview of how to create language tags using BCP 47. BCP 47 The IETF specification that defines how language tags are formed, subtags are registered, as well as how to match language tags. § Values for language declarations must use BCP 47. more Language tags in HTML and XML BCP 47 BCP 47 is the language tag system used by Internet and Web standards (and many other places). It defines a method of using subtags from an IANA registry to form a string which describes the language of content. The subtags in the registry are primarily based on (and maintain strict compatibility with) ISO and UN standards for identifying languages, scripts, regions, and countries. BCP47 also forms the basis for Unicode locales . For an overview of the key features of BCP 47, see Language tags in HTML and XML . § Refer to BCP 47, not to its constituent parts, such as RFC 5646 or RFC 4647. The link to and name of BCP 47 was created specifically so that there is an unchanging reference to the definition of Tags for the Identification of Languages . RFCs 1766, 3066, 4646 were previous (superseded) versions. The current version of BCP 47 is made up of two RFCs: 5646 and 4647. § Be specific about what level of conformance you expect for language tags: BCP 47 defines two levels of conformance, "valid" and "well-formed". A well-formed BCP 47 language tag follows the syntax defined for a language tag: implementations check that each language tag consists of hyphen-separated subtags; each subtag has a specific length and specific content (letters, digits or specific combinations) depending on the placement in the tag. A valid BCP 47 language tag is well-formed but additionally ensures that only subtags that are listed in the IANA Subtag Registry are used. Note that the IANA Subtag Registry is frequently updated with new subtags. § Specifications may require implementations to check if language tags are "valid", but in most circumstances should only require that the language tags be "well-formed". Most specifications are second-order consumers of language metadata – they are using data already provided in the document format (HTML lang , XML xml:lang , or the document format's language fields/attributes). Generally most specifications are concerned with selecting resources (such as spell checkers, tokenizers, fonts, etc.) or with matching (selecting which string to show, for example) and don't directly care about the content of the language tag. Invalid-but-well-formed tags just don't match anything and usually fallback schemes provide some behavior that is appropriate. There might be cases where a specification really wants implementation-level checking for validity. In those cases, the result of a tag failing to be valid has to be specified (should the application die, warn the user, etc.). It's also a problem that the registry is sizeable and changes over time, so each implementation is registry-version dependent. The changes over time are often minor, but real users will encounter interoperability issues if random (out of date) implementations of the specification reject language tags that have become valid at a later date. In addition, BCP 47 has an extension mechanism which defines add-on subtag sequences. For example, one extension [ RFC6067 ] (Unicode Locales, which uses the singleton -u ), is commonly used for controlling the internationalization features of JavaScript (and has other uses). Validating these additional subtags is likely out of scope for most specifications. § Specifications should require content and content authors to use "valid" language tags. Normative language regarding language tags might be different between content and implementation requirements. Specification authors need to carefully consider what conformance requirements and tests are needed for their specification and what implementations are required to do. One solution is to normatively require that "valid" language tags be used by content authors but only require implementations to check for "well-formed" language tags. § Specifications SHOULD refer to the IANA Language Subtag Registry instead of providing lists of codes extracted from ISO 639, ISO 3166, or other standards. more Choosing a language tag Language subtag lookup tool IANA Language Subtag Registry In the past, some of the standards used to provide subtags found in language tags were not freely or publicly available, so some specifications provided lists in order to help ensure interoperability. This is no longer necessary. As part of BCP 47, IANA maintains the language subtag registry, which is a publicly available, machine-readable list of valid subtags for use in constructing language tags. This registry is based on underlying standards, including the various parts of ISO 639 (639-1, 639-2, 639-3, etc.), ISO 15924 script codes, and ISO 3166 and UN M.49 region codes. The registry is actively maintained, stabilized, and comprehensive in ways that other lists found on the Internet might not be. Each of the subtag types is kept in sync with parent standards with the help and participation of those standards maintainers, so extracting or making your own list of codes or referring to ones found elsewhere can lead to maintenance problems or confusion. § Avoid creating a list of valid or supported language tags, language subtags, or locales . Making your own list of fully formed language tags will unnecessarily restrict the list of languages that can be used. In addition, locale data is always being expanded, so a list that describes support today will become outdated in the future. Restricting which tags or subtags are available to users conflicts with our goal of providing universal access. 2.3 Declaring language See related review comments. 2.3.1 Declaring language at the resource level Here we are talking about an independent unit of data that contains structured text. Examples may include a whole HTML page, an XML document, a JSON file, a WebVTT script, an annotation, etc. Useful background and overviews for this section Why use the language attribute? . Types of language declaration . How 'metadata' and 'text-processing' types of language information differ. Use cases for bidi and language metadata on the Web . See also 2.2 Defining language values . § The specification should indicate how to define the default text-processing language for the resource as a whole. It often saves trouble to identify the language, or at least the default language, of the resource as a whole in one place. For example, in an HTML file, this is done by setting the lang attribute on the html element. § Content within the resource should inherit the language of the text-processing declared at the resource level, unless it is specifically overridden. § Consider whether it is necessary to have separate declarations to indicate the text-processing language versus metadata about the expected use of the resource. In many cases a resource contains text in only one language, and in many more cases the language declared as the default language for text-processing is the same as the language that describes the metadata about the resource as a whole. In such cases it makes sense to have a single declaration. It becomes problematic, however, to use a single declaration when it refers to more than one language unless there is a way to determine which one language should be used as the text-processing default. § If there is only one language declaration for a resource, and it has more than one language tag as a value, it must be possible to identify the default text-processing language for the resource. 2.3.2 Establishing the language of a content block See also 2.2 Defining language values . The words block and/or chunk are used here to refer to a structural component within the resource as a whole that groups content together and separates it from adjacent content. Boundaries between one block and another are equivalent to paragraph or section boundaries in text, or discrete data items inside a file. For example, this could refer to a block or paragraph in XML or HTML, an object declaration in JSON, a cue in WebVTT, a line in a CSV file, etc. Contrast this with inline content, which describes a range within a paragraph, sentence, etc. The interpretation of which structures defined in a spec are relevant to these requirements may require a little consideration, and will depend on the format of the data involved. § By default, blocks of content should inherit any text-processing language set for the resource as a whole. See 2.1 Language basics for guidance related to the default text-processing language information. § It should be possible to indicate a change in language for blocks of content where the language changes. 2.3.3 Establishing the language of inline runs In this section we refer to information that needs to be provided for a range of characters in the middle of a paragraph or string. See also 2.2 Defining language values § It should be possible to indicate language for spans of inline text where the language changes. Where a switch in language can affect operations on the content, such as spell-checking, rendering, styling, voice production, translation, information retrieval, and so forth, it is necessary to indicate the range of text affected and identify the language of that content. 2.4 Identifying the language of strings Note The information in this section is being developed in Requirements for Language and Direction Metadata in Data Formats [ STRING-META ]. That document is still being written, so these guidelines are likely to change at any time. The exchange of data on the Web, to the degree possible, should use locale-neutral standardized formats. However, some data on the Web necessarily consists of natural language information intended for display to humans. This natural language information depends on and benefits from the presence of language and direction metadata for proper display. Along with support for Unicode, mechanisms for including and specifying the base direction and the natural language of spans of text are one of the key internationalization considerations when developing new formats and technologies for the Web. The most basic best practice, which the Internationalization Working Group looks for in every specification, is: § For any string field containing natural language text, it MUST be possible to determine the language and string direction of that specific string. Such determination SHOULD use metadata at the string or document level and SHOULD NOT depend on heuristics. See related review comments. Useful background and overviews for this section Strings on the Web: Language and Direction Metadata . Best Practices, Recommendations, and Gaps . Requirements and Use Cases . Approaches Considered for Identifying the Base Direction . See also 3.5 Handling base direction for strings . Note Work on language and direction metadata for string formats is a work in progress. Specifications might need to include a note indicating the need for future adoption of metadata. Here is a prototype: The field {fieldname} should follow the best practices found in Strings on the Web: Language and Direction Metadata [ STRING-META ]. This includes making use of any future standards which emerge regarding the reporting of string language and direction metadata. § Use field-based metadata or string datatypes to indicate the language and the string direction for individual localizable text values. Individual data values can differ in language or direction from other values found in the same data file or document. Providing metadata values directly associated with each localizable text field allows for the metadata to be overridden appropriately and helps applications automate processing when assembling, extracting, forwarding, or otherwise processing each data field for use. § Specifications MAY define a mechanism to provide the default language and the default string direction for all strings in a given resource. However, specifications MUST NOT assume that a resource-wide default is sufficient. Even if a resource-wide setting is available, it must be possible to use string-specific metadata to override that default. Many documents contain data in a single language. Providing a means of indicating the intended language audience, perhaps in a header, can reduce overall document size and complexity. However, the ability to override specific string values remains important, as it is always possible that some strings might not be available in the document language or when the base direction is not consistent with the default direction of other localizable text values in the document as a whole. § Specify that, in the absence of other information, the default direction and default language are unknown. § Specifications SHOULD be careful to distinguish syntactic content , including user-supplied values , from localizable text . § Specifications MUST NOT treat syntactic content values as "displayable". § Specifications SHOULD NOT specify or require the use of language metadata for fields that cannot contain natural language text. Document formats on the Web consist of text. In most cases, data values in a given document format are meant to be representative and meaningful, not just arbitrary strings. The fact that a data value consists of, for example, an English keyword does not make the data value a natural language string meant for display as text (that is, the value is not localizable text ). Such data values are part of the syntactic content of the document: not only do they not require language and direction metadata, but they should not be associated with such metadata. § For string values and string fields that are not localizable text , specifications SHOULD specify that the field is non-linguistic in nature and recommend the language tag zxx ("No linguistic content") be associated with each string value. § For string values and string fields that are known to contain localizable text but for which there is no possibility of language metadata from the underlying format, specifications SHOULD specify that the language of the content is unknown and recommend the language tag und ("Undetermined") be associated with each string. Specifications MAY also allow the use of heuristics or the inference of the language from other field values where appropriate. Some string values depend on or are defined by existing protocols or formats. Often these strings are not associated with or do not provide language or direction metadata. For example, many HTTP headers define their contents as if they were not localizable text , even when, in some cases, they contain natural language text. Consuming specifications sometimes need to take a dependency on strings of this nature or define a format that describes one of these strings. In these cases there will be no language or direction metadata for consumers to associate with the string in the specification's data structure or document format, and any metadata that the specification's data structure or document format provides (when functioning as a producer ) will not be serialized through the underlying format. § Specifications SHOULD NOT use the Unicode "language tag" characters (code points U+E0000 to U+E007F ) for language identification. The Unicode "language tag" characters are deprecated for use as language tags and there are many reasons why they are a poor solution to the language metadata problem in document formats and wire protocols. Specification authors are cautioned not to repurpose these characters or try to build new mechanisms for transmitting language information based on them. § Specifications SHOULD recommend the use of language indexing when localizable strings can be supplied in multiple languages for the same value. Producers sometimes need to supply localized values for a given content item or data record. Sometimes this is done by language negotiation between the producer and consumer . Localization then takes place in the producer using the negotiated language to select the content returned. Other times localization of a content item is done by having the producer return multiple language representations for the item and letting the consumer choose the value to display. This latter process is called language indexing . For more information about language indexing, see Localization Considerations in [ STRING-META ]. 2.4.1 Language information in JSON-LD Additional material on this sub-section's contents can be found in: Strings on the Web: Language and Direction Metadata . Technology-specific solutions . [ JSON-LD ] provides several mechanisms for satisfying some of the best practices found in this section: § For documents that use [ JSON-LD ], use of [ JSON-LD ] @context and the built-in @language attribute is RECOMMENDED as a document level default. § Specifications SHOULD use the i18n Namespace feature for RDF literals, as defined in [ JSON-LD ] 1.1. § Where the i18n Namespace is not available or is inappropriate to use, specifications SHOULD require [ JSON-LD ] plain string literals for natural language values to provide string-specific language information. 2.5 Detecting language See related review comments. Issue 1 This section is under development. 2.6 Matching language and language tags See related review comments. The best practices in this document, as well as Strings on the Web: Language and Direction Metadata [ STRING-META ], require natural language content to be associated with language metadata. On the Web, "language metadata" means a language tag , specifically one defined by [ BCP47 ]. Specifications or applications sometimes need to select or filter content according to language or according to the user's locale . See also: Language Tags and Locale Identifiers for the World Wide Web [ LTLI ] [ BCP47 ] is composed of two RFCs. The first, [ RFC5646 ], defines the grammar, processing, and formation of language tags as well as the contents and maintenace of the IANA Language Subtag Registry , which lists the valid subtags. The second, [ RFC4647 ], defines language ranges , language priority lists , and several common matching schemes that applications can use (or specifications specify) when selecting or filtering using language tags. Unicode [ CLDR ] defines additional algorithms, rules, and processes for managing and matching language tags when they are used as locale identifiers. § Reference [ BCP47 ] for language tag matching. more BCP 47 RFC4647 Matching of Language Tags There are two different types of matching for language tags : lookup and filtering. Lookup [ RFC4647 ] matches a language priority list consisting of basic language ranges to sets of language tags to find the one exact language tag that best matches the specified range or ranges. That is, use Lookup when the result needs to be exactly one item or value that best matches the user's needs (or an appropriate default). Examples of lookup include: Selecting the appropriate localized value, such as a paragraph of text; an image; an icon; etc. Selecting a specific value (such as a string) Selecting a resource file Computing the result of language negotiation Filtering matches a language priority list to sets of language tags to produce zero or more matches. That is, it is used to filter content to produce only those items that match the language priority list . This might eliminate all of the language tag values, or it might include many (or all) of the items. § When specifying language tag filtering, basic filtering is RECOMMENDED for most applications. [ BCP47 ] defines two types of filtering: basic filtering and extended filtering [ RFC4647 ]. Basic filtering uses a language priority list consisting of basic language ranges to filter lists of language tags or language tagged content. In basic filtering, the language tags are prefix matched to each language range. Basic filtering is useful when you need to select all and only those language tags that are strict descendants of the language ranges in the language priority list. Example 1 : Basic Filtering Basic filtering matches basic language ranges , which are just language tags, to the language tags associated with the content that is being filtered. A language range matches a language tag if it is an exact prefix for the language tag, paying attention to subtag boundaries. Language range Language tag Match? Reason * fr-CA Yes The wildcard range matches any language tag. de de Yes Exact match. de-CH Yes The tag starts with de followed by a hyphen. de-Latn-DE Yes The range matches the initial subtags of the tag. den No den does not equal de . de-DE de-Latn-DE No Basic filtering requires a prefix match; it does not skip subtags. zh-Hant zh-Hant-TW Yes The tag begins with the full range zh-Hant . zh-Hans-CN No The script subtag differs: Hant vs. Hans . en-US en-US-x-twain Yes The range matches the initial subtags before the private-use extension. en-x-US-twain No The US subtag does not match across the singleton subtag boundary. You can use basic filtering when the language tags can contain additional subtags, including extensions, and you don't want to specify all possible permutations. For example, if you were selecting audio content, you might want to match all Serbian (range: sr ) language tags, because the script subtags ( Latn , Cyrl ) or region subtags ( RS , etc.) often associated with the Serbian language are not important to the selection. Or you might wish to select any English (range: en ) tags without having to list all possible regional subtags. In basic filtering , using a language priority list with multiple language ranges is the only means of filtering when the language tags can take different forms, such as when some content includes and other content excludes the script subtag. Extended filtering uses a language priority list consisting of extended language ranges to filter lists of language tags or language tagged content. An extended language range can contain a wildcard * subtag in place of one or more of its subtags. (Wildcards outside the first "primary language subtag" position are ignored). Unlike basic filtering , extended filtering is used to select specific subtags inside of a language tag. Use extended filtering to filter out language tags that do not match the specific list of subtags within the specified language ranges . Example 2 Extended filtering exists to allow applications to select or match content based on specific subtags within a langauge tag. This is often needed when trying to match a specific script ( *-Latn , *-Hant ) or a specific region ( *-DE , *-US ), regardless of what language is being used. For example, you might use extended filtering when an operation, such as assigning a font, depends mostly on the script but there can be many different languages. Or you might use this scheme when a specific region (such as a country) is the distinguishing factor. The wildcard in an extended language range can be included anywhere in the language range, such as *-Latn or zh-*-SG . However, outside the first position, it usually is not included in the range, but remains implied. For example, the range de-DE implies de- *- DE or zh-wadegile implies zh- *-*- wadegile . Extended language range Language tag Match? Reason de de-DE Yes The language range is a prefix of the language tag. de-DE de-DE Yes The language range subtags match the language tag subtags directly. de-Latn-DE Yes The implied wildcard matches Latn while looking for the region subtag DE . de-CH No The required DE subtag is not present. zh-CN zh-Hans-CN Yes The implied wildcard matches Hans while looking for CN . zh-CN Yes The language range subtags match the language tag subtags directly. zh-Hant-TW No The required CN subtag is not present. zh-x-CN No Matching stops at private use or extensions unless specified in the range. *-CH de-CH Yes The initial wildcard matches any primary language; any tag that includes the region CH then matches. fr-Latn-CH Yes The initial wildcard matches the primary language and also the Latn subtag while looking for CH . it-IT No The required CH subtag is not present. 3. Text direction Show the self-review checklist This is a list of just the requirements in this section, which can be used for self-reviews. Select the first checkbox on a line for all requirements relevant to your spec. Select the second checkbox if your spec meets the requirement. Then click on the button "Create markdown for GitHub", and copy the result to a GitHub issue list. See more details . Basic requirements It must be possible to indicate base direction for each individual paragraph-level item of natural language text that will be read by someone. more For any string field containing natural language text, it MUST be possible to determine the language and string direction of that specific string. Such determination SHOULD use metadata at the string or document level and SHOULD NOT depend on heuristics. more It must be possible to indicate base direction changes for embedded runs of inline bidirectional text for all localizable text . more Annotating right-to-left text must require the minimum amount of effort for people who work natively with right-to-left scripts. more Background information Do not assume that direction can be determined from language information. more Base direction values Values for the default base direction should include left-to-right, right-to-left, and auto. more Handling direction in markup The spec should indicate how to define a default base direction for the resource as a whole, ie. set the overall base direction. more The default base direction, in the absence of other information, should be auto . more The content author must be able to indicate parts of the text where the base direction changes. At the block level, this should be achieved using attributes or metadata, and should not require the content author to use Unicode control characters to control direction. more It must be possible to also set the direction for content fragments to auto . This means that the base direction will be determined by examining the content itself. more If the overall base direction is set to auto for plain text, the direction of content paragraphs should be determined on a paragraph by paragraph basis. more To indicate the sides of a block of text relative to the start and end of its contained lines, use 'block-start' and 'block-end', rather than 'top' and 'bottom'. more To indicate the start/end of a line you should use 'start' and 'end', or 'inline-start' and 'inline-end', rather than 'left' and 'right'. more Provide dedicated attributes for control of base direction and bidirectional overrides; do not rely on the user applying style properties to arbitrary markup to achieve bidi control. more Handling base direction for strings Provide metadata constructs that can be used to indicate the base direction of any natural language string. more Specify that consumers of strings should use heuristics, preferably based on the Unicode Standard first-strong algorithm, to detect the base direction of a string except where metadata is provided. more Where possible, define a field to indicate the default direction for all strings in a given resource or document. more Do NOT assume that a creating a document-level default without the ability to change direction for any string is sufficient. more If metadata is not available due to legacy implementations and cannot otherwise be provided, specifications MAY allow a string direction to be interpolated from available language metadata. more Specifications MUST NOT require the production or use of paired bidi controls. more Setting base direction for inline or substring text It must be possible to indicate spans of inline text where the base direction changes. If markup is available, this is the preferred method. Otherwise your specification must require that Unicode control characters are recognized by the receiving application, and correctly implemented. more It must be possible to also set the direction for a span of inline text to auto , which means that the base direction will be determined by examining the content itself. A typical approach here would be to set the direction based on the first strong directional character outside of any markup. more If users use Unicode bidirectional control characters, the isolating RLI/LRI/FSI with PDI characters must be supported by the application and recommended (rather than RLE/LRE with PDF) by the spec. more Use of RLM/LRM should be appropriate, and expectations of what those controls can and cannot do should be clear in the spec. more For markup, provide dedicated attributes for control of base direction and bidirectional overrides; do not rely on the user applying style properties to arbitrary markup to achieve bidi control. more For markup, allow bidi attributes on all inline elements in markup that contain text. more For markup, provide attributes that allow the user to (a) create an isolated or embedded base direction or (b) override the bidirectional algorithm altogether. Such attributes should allow the user to set the direction to LTR, RTL, or Auto in either of these two scenarios. more Detecting & matching direction (TBD) Create markdown for GitHub It is important to establish direction for text written or mixed with right-to-left scripts. Characters in these scripts are stored in memory in the order they are typed and pronounced – called the logical order. The Unicode Bidirectional Algorithm (UBA) provides a lot of support for automatically rendering a sequence of characters stored in logical order so that they are visually ordered as expected. Unfortunately, the UBA alone is not sufficient to correctly render bidirectional text, and additional information has to be provided about the default directional context to apply for a given sequence of characters. 3.1 Basic requirements The basic requirements are as follows. § It must be possible to indicate base direction for each individual paragraph-level item of natural language text that will be read by someone. A special case of the above applies to natural language string values in data structures and document formats: § For any string field containing natural language text, it MUST be possible to determine the language and string direction of that specific string. Such determination SHOULD use metadata at the string or document level and SHOULD NOT depend on heuristics. § It must be possible to indicate base direction changes for embedded runs of inline bidirectional text for all localizable text . § Annotating right-to-left text must require the minimum amount of effort for people who work natively with right-to-left scripts. Requiring a speaker of Arabic, Divehi, Hebrew, Persian, Urdu, etc. to add markup or control characters to every paragraph or small data item they write is far too much to be manageable. Typically, the format should establish a default direction and require the user to intervene only when exceptions have to be dealt with. 3.2 Background information In this section we try to set out some key concepts associated with text direction, so that it will be easier to understand the recommendations that follow. Useful background and overviews for this section Unicode Bidirectional Algorithm basics . String on the Web: Language and Direction Metadata [ STRING-META ] 3.2.1 Important definitions In order to correctly display text written in a 'right-to-left' script or left-to-right text containing bidirectional elements, it is important to establish the base direction that will be used to dictate the order in which elements of the text will be displayed. If you are not familiar with what the Unicode Bidirectional Algorithm (UBA) does and doesn't do, and why the base direction is so important, read Unicode Bidirectional Algorithm basics . Example 3 For example, the following annotation will not display correctly unless the application doing the display knows that the base direction needs to be right-to-left. { "@context" : "http://www.w3.org/ns/anno.jsonld" , "id" : "http://example.org/anno5" , "type" : "Annotation" , "body" : { "type" : "TextualBody" , "text" : "פעילות הבינאום, W3C" , "format" : "text/html" , "language" : "he" } , "target" : "http://example.org/photo1" } If there is no indication that the base direction is right-to-left, the display of the item text will be incorrect if the text is placed into a left-to-right context (such as the table below): Description HTML Appearance Incorrect: (without dir ) < span lang = "he" > ות הבינאום, W3C </ span > ות הבינאום, W3C Correct: (with dir ) < span lang = "he" dir = "rtl" > ות הבינאום, W3C </ span > ות הבינאום, W3C In this section, the word paragraph indicates a run of text followed by a hard line-break in plain text, but may signify different things in other situations. In CSV it equates to 'cell', so a single line of comma-separated items is actually a set of comma-separated paragraphs.  In HTML it equates to the lowest level of block element, which is often a p element, but may be things such as div , li , etc., if they only contain text and/or inline elements. In JSON, it often equates to a quoted string value, but if a string value uses markup then paragraphs are associated with block elements, and if the string value is multiple lines of plain text then each line is a paragraph. Note While the Unicode Bidirectional Algorithm [ UAX9 ] formally refers to paragraphs and paragraph direction (or the base direction of a paragraph), this can sometimes be confusing when the text in question is not in a long-form document. Instead, this document and others will sometimes use the terms "block direction" or especially string direction to refer to the paragraph direction of a specific string of natural language text. The term metadata is used here to mean information which could be an annotation or property associated with the data, or could be markup in scenarios that allow that, or could be a higher-level protocol, etc. 3.2.2 Ways base direction can be set for paragraphs There are a number of possible ways of setting the base direction. The base direction of a paragraph may be set by an application or a user applying metadata to the paragraph. Typical values for base direction may include ltr , rtl or auto . The metadata may specifically indicate that heuristics should be used. Then you would expect to consider the actual characters used in order to determine the base direction. (This is what happens if you set dir=auto on an HTML element.) The application may expect metadata, but there may be no such information provided. In this case you would usually expect there to be a default direction specified, and the base direction for a cell would be set to that default. The default is usually LTR. (This is what happens if you have no dir attributes in your HTML file.) Where a format contains many paragraphs or chunks of information, and the language of text in all those chunks is the same, it is sometimes useful to allow a default base direction to be set for and inherited by all. This is what happens when you set the dir attribute on the html tag in HTML. Another example would be a subtitling file containing many cues, all written in Arabic; it would be best to allow the author to say at the start of the file that the default is RTL for all cue text. There should always be a way to override the direction information for a specific paragraph where needed. If the application expects no metadata to be available it should use heuristics to determine the base direction for each paragraph/cell. A typical solution, and one described by UAX 9 Unicode Bidirectional Algorithm , is to look for the first-strong character in the paragraph/cell. (This is likely to apply if you are looking at plain text that is not expected to be associated with metadata. It only happens with HTML if the direction is set to auto , since HTML specifies a default direction.) Not all paragraphs using the first-strong method will have the correct base direction applied. In some cases, an Arabic or Hebrew, etc, paragraph may start with strong LTR characters. There must be a way to deal with this. Where a syntactic unit contains multiple lines of plain text (for example, a multiline cue text in a subtitling file), the first-strong heuristic needs to be applied to each line separately. There may be special rules that involve ignoring some sequence of characters or type of markup at the start of the paragraph before identifying the first strong character. In some cases there are no strong characters in a paragraph, and the base direction can be critically important for the data to be understood correctly, eg. telephone numbers or MAC addresses. There needs to be a way to resort to an appropriate default for these cases. Whether or not any metadata is specified, if a paragraph contains a string that starts with one of the Unicode bidi control characters RLI, LRI, FSI, LRE, RLE, LRO, or RLO and ends with PDF/PDI, these characters will determine the base direction for the contained string. These characters, when placed in the content, explicitly override any previously set direction by creating an inline range and assigning a base direction to it. The effect of such characters does not extend past paragraph boundaries, but the range ought to be explicitly ended using the PDF/PDI control character, especially if a paragraph end is not easily detectable by the application.) Because isolation is needed for bidirectional text to work properly, the Unicode Standard says that the isolating control codes RLI, LRI and FSI should be used rather than LRE or RLE. Unfortunately, those characters are still not widely supported. For structural components in markup, above the paragraph level, it is not possible to use the Unicode bidi control characters to define direction for paragraphs, since these are inline controls only, and the effect is terminated by a paragraph end. When capturing text input by a user it is usually necessary to understand the context in which the user was inputting the data to determine the base direction of the input. In HTML, for example, this may be set by the direction inherited from the html tag, or by the user pressing keys to set the base direction for a form field. It is then necessary to find some way of storing the information about base direction or associating it with the string when rendered. Typically, in this situation, any direction changes internal to the string being input are handled by the user and will be captured as part of the string. 3.2.3 Inline changes to base direction Embedded ranges of text within a single paragraph may need to have a different base direction. For example, "The title was '!NOITASILANOITANRETNI'." where the span within the single quotes is in Hebrew/Arabic/Divehi, etc., and needs to have a RTL base direction, instead of the LTR base direction of the surrounding paragraph, in order to place the exclamation mark correctly. If markup is available to the content author, it is likely to be easier and safer to use markup to indicate such inline ranges (see below). In HTML you would usually use an inline element with a dir attribute to establish the base direction for such runs of text. If you can't mark up the text, such as in HTML's title element, or any environment that handles only plain text content, you have to resort to Unicode's paired control characters to establish the base direction for such an internal range. Furthermore, inline ranges where the base direction is changed should be bidi isolated from surrounding text, so that the Unicode Bidirectional Algorithm doesn't produce incorrect results (" spillover ") due to interference across boundaries. Example 4 : Example of bidirectional interference across boundaries Suppose an implementation assembled a string by concatenating various values together. For example, the description of a monitor attached to a system. This label might contain the brand name and model number ( Brand A123B ); resolution ( (1920 x 1080) ); size and type ( 36" monitor ); as well as various features like a refresh rate of 75 Hz or response time of 4 ms . The resulting string in English might look like this (color has been added to make the effects more visible): Brand A123B (1920 x 1080) 36" monitor , 75 Hz , 4ms If the same implementation assembling this string were on a system running in a locale that uses a right-to-left language (such as the Arabic examples shown below), the results of the same concatentation might look something like this: ماركة A123B (1920 x 1080) 36" شاشة الكمبيوتر ، 75 Hz ، 4 مللي ثانية The logical sequence of sub-strings remains the same, but the visual presentation is no longer intelligible. Notice how different parts of the description have become broken up and mixed together. The addition of isolating bidirectional controls (either markup or, when not available, Unicode control characters) to the above string produces better results: ماركة A123B (1920 x 1080) 36" شاشة الكمبيوتر ، 75 Hz ، 4 مللي ثانية This means that if a content author is using Unicode control codes they should use the isolating controls RLI/LRI/FSI…PDI rather than the embedding controls RLE/LRE…PDF . See also An example of incorrect ordering of things such as text followed by numbers in HTML An example of incorrect ordering of lists 3.2.4 Problems with control characters Reasons to avoid relying on control characters to set direction include the following: They are invisible in most editors and are therefore difficult to work with, and can easily lead to orphans and overlapping ranges. They can be particularly difficult to manage when editing bidirectional inline text because it's hard to position the cursor in the correct place. If you ask someone who writes in a right-to-left script, you are likely to find that they dislike using control codes. Users often don't have the necessary characters available on their keyboard, or have difficulty inputting them. It is sometimes necessary to choose which to use based on context or the type of the data, and this means that a content author typically needs to select the control codes – specifying control codes in this way for all paragraphs is time-consuming and error-prone. Processors that extract parts of the data, add to it, or reuse in combination with other text may incorrectly handle the control codes. Search and comparison algorithms should ignore these characters, but typically don't. The last two items above may also hold for markup, but implementers often support included markup better than included control codes. Don't expect users to add control codes at the start and end of every paragraph. That's far too much work. 3.2.5 Strong directional formatting characters: RLM, LRM, and ALM A word about the Unicode characters U+200F RIGHT-TO-LEFT MARK (RLM), U+200E LEFT-TO-RIGHT MARK (LRM), and U+061C ARABIC LETTER MARK (ALM) is warranted at this point. The first point to be clear about is that these three characters do not establish the base direction for a range of text. They are simply invisible characters with strong directional properties. Recalling an earlier example , this means that you cannot use RLM, for example, to make the text W3C appear to the left of the Hebrew text. Only using metadata or paired control characters results in the correct display. Example 5 : Use metadata instead of strongly directional formatting characters Description HTML Result With RLM (incorrect) < span lang = "he" > ותות הבינאום, W3C &rlm; </ span > ותות הבינאום, W3C ‏ With metadata (correct) < span lang = "he" dir = "rtl" > ותות הבינאום, W3C </ span > ותות הבינאום, W3C Of course, if you are detecting base direction using first-strong heuristics (such as dir="auto" in HTML), then inserting an RLM, ALM, or LRM can be useful for influencing the base direction detected where the text in question begins with something that would otherwise give the wrong result. Example 6 : Using a strong directional formatting character to assist first-strong heuristics This HTML has strongly right-to-left Arabic characters near the start, where they will be picked up by a first-strong heuristic. Notice that there is a neutral character right at the start: < p dir = "auto" > "نشاط التدويل" is how you say "i18n activity" in Arabic. </ p > This produces the wrong result: "نشاط التدويل" is how you say "i18n Activity" in Arabic. Here an LRM could be placed at the start of the text to prevent the algorithm from assuming that the text should be right-to-left. < p dir = "auto" > &lrm; "نشاط التدويل" is how you say "i18n activity" in Arabic. </ p > ‎"نشاط التدويل" is how you say "i18n Activity" in Arabic. Remember that if metadata is used to set the base direction, the strong directional formatting character is ignored, unless the metadata specifically says that first-strong heuristics should be used. Finally, a note about the use of U+061C ARABIC LETTER MARK (ALM). This character is used to influence the display of sequences of numbers in Arabic script text in cases where no Arabic letters occur before the number. Example 7 : Example of ALM usage In some Arabic-script languages the range 100-200 should appear as ؜100-200 . If no Arabic letters appear before the numbers, the Unicode Bidirectional Algorithm will not perform this reordering. Note that the character sequences in both cases is "100-200" and that both have a code element with a dir="rtl" around them. In the third example, an ALM is used to provide the necessary hint, like so: Description HTML Appearance Preceded by Arabic letters < code dir = "rtl" lang = "ar" > نشاط التدويل 100-200 </ code > نشاط التدويل 100-200 Without ALM < code dir = "rtl" lang = "ar" > 100-200 </ code > 100-200 With ALM < code dir = "rtl" lang = "ar" > &#x061C; 100-200 </ code > ؜100-200 3.2.6 Base direction and language § Do not assume that direction can be determined from language information. more Can we derive base direction from language? , W3C article. The following are all reasons you cannot use language tags to provide information about base direction: you can't produce the auto value with language tags. some languages are written with both RTL and LTR scripts. the only reliable part of the language tag that would indicate the base direction is the script tag, but BCP47 recommends that you suppress the use of the script tag for languages that don't usually need it, such as Hebrew ( Suppress-Script: Hebr ). Languages, such as Persian, that are usually written in a RTL script may be written in transcribed form, and it's not possible to guarantee that the necessary script tag would be present to carry the directional information. In summary, you won't be able to rely on people supplying script tags as part of the language information in order to influence direction. the incidence of use of language tags and base direction markers often don't coincide. they are not semantically equivalent. 3.3 Base direction values See related review comments. § Values for the default base direction should include left-to-right, right-to-left, and auto. The auto value allows automatic detection of the base direction for a piece of text. For example, the auto value of dir in HTML looks for the first strong directional character in the text, but ignores certain items of markup also, to guess the base direction of the text. Note that automatic detection algorithms are far from perfect. First-strong detection is unable to correctly identify text that is really right-to-left, but that begins with a strong LTR character. Algorithms that attempt to judge the base direction based on contents of the text are also problematic. The best scenario is one where the base direction is known and declared. 3.4 Handling direction in markup This section is about defining approaches to bidi handling that work with resources that organize content using markup. Some of the recommendations are different from those for handling strings on the Web (see 3.5 Handling base direction for strings ). See related review comments. Useful background and overviews for this section Additional Requirements for Bidi in HTML & CSS . 3.4.1 Setting the default base direction § The spec should indicate how to define a default base direction for the resource as a whole, ie. set the overall base direction. § The default base direction, in the absence of other information, should be auto . 3.4.2 Establishing the base direction for paragraphs § The content author must be able to indicate parts of the text where the base direction changes. At the block level, this should be achieved using attributes or metadata, and should not require the content author to use Unicode control characters to control direction. Relying on Unicode control characters to establish direction for every block is not feasible because line breaks terminate the effect of such control characters. It also makes the data much less stable, and unnecessarily difficult to manage if control characters have to appear at every point where they would be needed. § It must be possible to also set the direction for content fragments to auto . This means that the base direction will be determined by examining the content itself. more Estimation algorithms , in Additional Requirements for Bidi in HTML & CSS . A typical approach here would be to set the direction based on the first strong directional character outside of any markup, but this is not the only possible method. The algorithm used to determine directionality when direction is set to auto should match that expected by the receiver. The first-strong algorithm looks for the first character in the paragraph with a strong directional property according to the Unicode definitions. It then sets the base direction of the paragraph according to the direction of that character. Note that the first-strong algorithm may incorrectly guess the direction of the paragraph when the first character is not typical of the rest of the paragraph, such as when a RTL paragraph or line starts with a LTR brand name or technical term. For additional information about algorithms for detecting direction, see Estimation algorithms in the document where this was discussed with reference to HTML. § If the overall base direction is set to auto for plain text, the direction of content paragraphs should be determined on a paragraph by paragraph basis. § To indicate the sides of a block of text relative to the start and end of its contained lines, use 'block-start' and 'block-end', rather than 'top' and 'bottom'. more CSS Logical Properties and Values Level 1 § To indicate the start/end of a line you should use 'start' and 'end', or 'inline-start' and 'inline-end', rather than 'left' and 'right'. more CSS Logical Properties and Values Level 1 § Provide dedicated attributes for control of base direction and bidirectional overrides; do not rely on the user applying style properties to arbitrary markup to achieve bidi control. more CSS vs. markup for bidi support , W3C article. For example, HTML has a dir attribute that is capable of managing base direction without assistance from CSS styling. XML formats should define dedicated markup to represent directional information, even if they need CSS to achieve the required display, since the text may be used in other ways. Style sheets such as CSS may not always be used with the data, or carried with the data when it is syndicated, etc. Directional information is fundamentally important to correct display of the data, and should be associated more closely and more permanently with the markup or data. 3.5 Handling base direction for strings Note The information in this section is pulled from Strings on the Web: Language and Direction Metadata . That document is still being written, so these guidelines are likely to change at any time. See related review comments. Useful background and overviews for this section Strings on the Web: Language and Direction Metadata . Best Practices, Recommendations, and Gaps . Requirements and Use Cases . Approaches Considered for Identifying the Base Direction . § Provide metadata constructs that can be used to indicate the base direction of any natural language string. more Best Practices, Recommendations, and Gaps , in Strings on the Web: Language and Direction Metadata § Specify that consumers of strings should use heuristics, preferably based on the Unicode Standard first-strong algorithm, to detect the base direction of a string except where metadata is provided. more Best Practices, Recommendations, and Gaps , in Strings on the Web: Language and Direction Metadata § Where possible, define a field to indicate the default direction for all strings in a given resource or document. more Best Practices, Recommendations, and Gaps , in Strings on the Web: Language and Direction Metadata § Do NOT assume that a creating a document-level default without the ability to change direction for any string is sufficient. more Best Practices, Recommendations, and Gaps , in Strings on the Web: Language and Direction Metadata § If metadata is not available due to legacy implementations and cannot otherwise be provided, specifications MAY allow a string direction to be interpolated from available language metadata. more Best Practices, Recommendations, and Gaps , in Strings on the Web: Language and Direction Metadata § Specifications MUST NOT require the production or use of paired bidi controls. more Best Practices, Recommendations, and Gaps , in Strings on the Web: Language and Direction Metadata 3.6 Setting base direction for inline or substring text 'Inline text' here has a readily understandable meaning in markup. It also applies to strings (eg. in JSON, CSV, or other plain text formats), meaning runs of characters which don't include all the characters in the string. § It must be possible to indicate spans of inline text where the base direction changes. If markup is available, this is the preferred method. Otherwise your specification must require that Unicode control characters are recognized by the receiving application, and correctly implemented. § It must be possible to also set the direction for a span of inline text to auto , which means that the base direction will be determined by examining the content itself. A typical approach here would be to set the direction based on the first strong directional character outside of any markup. The first-strong algorithm looks for the first character in the paragraph with a strong directional property according to the Unicode definitions. It then sets the base direction of the paragraph according to the direction of that character. Note that the first-strong algorithm may incorrectly guess the direction of the paragraph when the first character is not typical of the rest of the paragraph, such as when an RTL paragraph or line starts with a LTR brand name or technical term. For additional information about algorithms for detecting direction, see Estimation algorithms in the document where this was discussed with reference to HTML. § If users use Unicode bidirectional control characters, the isolating RLI/LRI/FSI with PDI characters must be supported by the application and recommended (rather than RLE/LRE with PDF) by the spec. § Use of RLM/LRM should be appropriate, and expectations of what those controls can and cannot do should be clear in the spec. The Unicode bidirectional control characters U+200F RIGHT-TO-LEFT MARK and U+200E LEFT-TO-RIGHT MARK are not sufficient on their own to manage bidirectional text. They cannot produce a different base direction for embedded text. For that you need to be able to indicate the start and end of the range of the embedded text.  This is best done by markup, if available, or failing that using the other Unicode bidirectional controls mentioned just above. § For markup, provide dedicated attributes for control of base direction and bidirectional overrides; do not rely on the user applying style properties to arbitrary markup to achieve bidi control. § For markup, allow bidi attributes on all inline elements in markup that contain text. § For markup, provide attributes that allow the user to (a) create an isolated or embedded base direction or (b) override the bidirectional algorithm altogether. Such attributes should allow the user to set the direction to LTR, RTL, or Auto in either of these two scenarios. 3.7 Detecting & matching direction (TBD) See related review comments. 4. Characters Show the self-review checklist This is a list of just the requirements in this section, which can be used for self-reviews. Select the first checkbox on a line for all requirements relevant to your spec. Select the second checkbox if your spec meets the requirement. Then click on the button "Create markdown for GitHub", and copy the result to a GitHub issue list. See more details . Characters and character encoding basics When specifying characters, strings, or any process that works with characters or strings, use the most specific appropriate terminology. Unless you have a reason not to, define code point to mean a Unicode Scalar Value and use that term in preference to 'character'. more If you cannot avoid using the term 'character', you MUST include a clear definition of the term. more Specifications, software, and content MUST NOT require or depend on a one-to-one relationship between characters ( code points ) and units of physical storage ( code units ). more Specifications, software, and content MUST NOT require or depend on a one-to-one mapping between code points and units of presentation (such as grapheme clusters , glyphs , or typographic character units). more Specifications, software, and content MUST NOT require or depend on a one-to-one correspondence between characters and the sounds of a language. more Specifications and software MUST NOT require nor depend on a single keystroke resulting in a single character, nor that a single character be input with a single keystroke (even with modifiers), nor that keyboards are the same all over the world. more Choosing a definition of 'string' Use a DOMString when defining document formats or the wire format of a protocol, or for any process pertaining to the [ DOM ], or which defines strings as opaque values whose individual character contents are not meant to be evaluated. (This list of uses is not exhaustive.) more Use a USVString for strings whose most common algorithms operate on Unicode Scalar Values , or for operations which can't handle surrogates in input. This includes algorithms that iterate over the code points in a string, processes that involve UTF-8 encode , and operations that pass the string to another component or API for which an unpaired surrogate code point would produce an error. more Avoid mixing DOMString and USVString in a single document or protocol operation. Often you will choose a DOMString over a USVString , since the latter requires extra processing that does not benefit most document formats or protocols. more Specify DOMString or, rarely, USVString for fields in protocols or document formats defined using bytes unless there is some reason to interact with specific bytes values or for which the UTF-8 character encoding cannot be assumed. more Specify Uint8Array when working with byte sequences, such as for data that does not contain text, or for byte sequences representing text for which processing is never required (such as when copying buffers). more Specify ByteString in the rare cases where the specification needs to work with strings which are encoded using bytes and for which the conversion to or from Unicode would be inappropriate. more Defining a Reference Processing Model Textual data objects defined by protocol or format specifications MUST be in a single character encoding. more All specifications that involve processing of text MUST specify the processing of text according to the Reference Processing Model described by the rest of the recommendations in this list. more Specifications MUST define text in terms of Unicode characters, not bytes or glyphs. more For their textual data objects specifications MAY allow use of any character encoding which can be transcoded to a Unicode encoding form. more Specifications MAY choose to disallow or deprecate some character encodings and to make others mandatory. Independent of the actual character encoding, the specified behavior MUST be the same as if the processing happened as follows: (a) The character encoding of any textual data object received by the application implementing the specification MUST be determined and the data object MUST be interpreted as a sequence of Unicode characters - this MUST be equivalent to transcoding the data object to some Unicode encoding form, adjusting any character encoding label if necessary, and receiving it in that Unicode encoding form, (b) All processing MUST take place on this sequence of Unicode characters, (c) If text is output by the application, the sequence of Unicode characters MUST be encoded using a character encoding chosen among those allowed by the specification. more If a specification is such that multiple textual data objects are involved (such as an XML document referring to external parsed entities), it MAY choose to allow these data objects to be in different character encodings. In all cases, the Reference Processing Model MUST be applied to all textual data objects. more Including and excluding character ranges Specifications SHOULD NOT arbitrarily exclude code points from the full range of Unicode code points from U+0000 to U+10FFFF inclusive. more Specifications MUST NOT allow code points above U+10FFFF. more Specifications SHOULD NOT allow the use of codepoints reserved by Unicode for internal use. more Specifications MUST NOT allow the use of unpaired surrogate code points. more Specifications SHOULD exclude compatibility characters in the syntactic elements (markup, delimiters, identifiers) of the formats they define. more Specifications SHOULD allow the full range of Unicode for user-defined values. more Using the Private Use Area Specifications MUST NOT require the use of private use area characters with particular assignments. more Specifications MUST NOT require the use of mechanisms for defining agreements of private use code points. more Specifications and implementations SHOULD NOT disallow the use of private use code points by private agreement. more Specifications MAY define markup to allow the transmission of symbols not in Unicode or to identify specific variants of Unicode characters. more Specifications SHOULD allow the inclusion of or reference to pictures and graphics where appropriate, to eliminate the need to (mis)use character-oriented mechanisms for pictures or graphics. more Choosing character encodings Use UTF-8 for all document formats, protocols, and serialization forms. more For all new formats or protocols or where a specification can safely do so, specifications MUST define UTF-8 as the only permitted character encoding form . more If, for historical reasons, a specification permits legacy character encodings , it MUST restrict the set of character encodings to those listed in the Encoding Standard in the section "Names and Labels". Other encodings SHOULD NOT be used, except by private agreement. more Identifying character encodings Specifications that allow multiple character encoding forms MUST provide a mechanism, such as a field or parameter, that clearly identifies the encoding of text. more If a protocol, format, or API is based on a format that already has rules for choosing, applying, or labeling the character encoding , the specification MUST NOT define a separate mechanism for identifying the character encoding . more If a specification is based on a format that permits character encodings other than UTF-8, the specification SHOULD restrict the character encoding to UTF-8. more Specifications MUST NOT propose the use of heuristics to determine the encoding of data. more Specifications MUST define conflict-resolution mechanisms (e.g. priorities) for cases where there is multiple or conflicting information about character encoding . more Designing character escapes Specifications should provide a mechanism for escaping characters, particularly those which are invisible or ambiguous. more Specifications SHOULD NOT invent a new escaping mechanism if an appropriate one already exists. more The number of different ways to escape a character SHOULD be minimized (ideally to one). more Escape syntax MUST either use an explicit end delimiter, such as in &#x12AB; or \u{12F} , or a fixed number of digits, such as in \u12AB or \U001234AB . more End-delimited forms are RECOMMENDED . more Whenever specifications define character escapes that allow the representation of characters using a number, the number MUST represent the Unicode code point of the character and SHOULD be in hexadecimal notation. more Escaped characters SHOULD be acceptable wherever their unescaped forms are; this does not preclude that syntax-significant characters, when escaped, lose their significance in the syntax. In particular, if a character is acceptable in identifiers and comments, then its escaped form should also be acceptable. more Storing text Protocols, data formats and APIs MUST store, interchange or process text data in logical order. more Independent of whether some implementation uses logical selection or visual selection, characters selected MUST be kept in logical order in storage. more Specifications of protocols and APIs that involve selection of ranges SHOULD provide for discontiguous logical selections, at least to the extent necessary to support implementation of visual selection on screen on top of those protocols and APIs. more Whitespace characters Specifications that use the term "whitespace" SHOULD explicitly define what the term means. more Most specifications SHOULD define whitespace to mean characters with the Unicode White_Space property. more Specifications that define whitespace for use in vocabularies that are restricted to ASCII or to formats that are whitespace delimited (examples include HTML or CSS) SHOULD specify ASCII whitespace as part of their grammar. more If a specification defines "whitespace" differently from ASCII or Unicode whitespace, the specific code points MUST be listed. more Referring to Unicode characters Use U+XXXX syntax to represent Unicode code points in a specification. more Use the Unicode character name to describe specific code points. more Use of the character naming template is RECOMMENDED . more Create markdown for GitHub 4.1 Characters and character encoding basics See related review comments. ➤➤➤Start Here The relationship between text and its storage and processing in computers is complicated, involving many terms and concepts that may be new to you. Understanding this terminology is important to understanding the guidance in the internationalization guidelines and best practices. We've created this article to explain and illustrate what you need to know: Character encodings: Essential concepts further reading These other documents provide additional information that may be helpful in understanding this section: Character Model for the World Wide Web: String Matching — issues that arise when you try to compare two strings, be it identifiers or authored content. Encoding — specification with further guidelines related to use of character encodings. Typographic character units in complex scripts — Article explaining more complex issues related to visual text units . Character Model for the World Wide Web: Fundamentals — basic guidelines related to the use of characters and encodings. The relevant content in this document has been imported into this document or is explained in the "Essential concepts" article at the top of this list. The word character means different things in different contexts: it can variously refer to the visual, logical, or byte-level representation of a given unit of text. This makes the term too imprecise to use casually in specifications. Understanding how text is defined and encoded in computing systems, along with the associated terminology used to make such specification unambiguous, is thus a necessary prerequisite to discussing the processing of string data. The developers of specifications, and the developers of software based on those specifications, are likely to be more familiar with usages of the term 'character' they have experienced and less familiar with the wide variety of usages in an international context. Furthermore, within a computing context, characters are often confused with related concepts, resulting in incomplete or inappropriate specifications and software. § When specifying characters, strings, or any process that works with characters or strings, use the most specific appropriate terminology. Unless you have a reason not to, define code point to mean a Unicode Scalar Value and use that term in preference to 'character'. Use the most appropriate terms found in this section. Here are some other recommended terms: Type of unit Instead of character use... Description Text units code point , Unicode code point , Unicode Scalar Value Logical units of text, without regard for character encoding form or any particular serialization. Storage, processing, serialization, encoding code unit Units of encoding and serialization. Code units are typically specified in wire and file formats, low-level text processing. Depends on the character encoding used (preferably UTF-8 or UTF-16). Visual units, user-perceived characters, selection/segmentation grapheme cluster , typographic character unit Breaking text into visual units, most visual selection and truncation . This recommendation has the most nuance. Elements of fonts glyph Individual display values. Use this term primarily when talking about the contents of a font. § If you cannot avoid using the term 'character', you MUST include a clear definition of the term. Here is a brief glossary of core terminology: [ Unicode ] [D7] defines an abstract character as: A unit of information used for the organization, control, or representation of textual data. This definition is necessarily vague and goes on to note that an abstract character does not have a specific concrete form (and thus not to be confused with a glyph in a font) nor what a user might think of as a "character" (and thus not to be confused with a grapheme ). Avoid using this term in your own specification. A character set is an unordered collection of abstract characters (in other words, it is a set ) which can be used together to encode text. The collection of characters in a character set is sometimes called its repertoire . Most character sets support only a specific range of languages or scripts. [ Unicode ] , sometimes called the Universal Character Set , includes all of the characters currently used to encode text in computer systems, including historical or extinct writing systems as well as modern usage, private use, typesetting symbols, and other things, such as the emoji. All other character sets are defined subsets of Unicode. Annual revisions extend the set of characters encoded. The [ Unicode ] Standard defines much more than just a character set . It also defines many of the properties, algorithms, and other details for the processing and presentation of text. A code point is a unique identifier for an abstract character in a character set . For a character set to be useful, it needs to unambiguously identify each character. In most character sets , the code point is a number (or set of numbers) that describe the location of the character in the table or chart of characters in the set. In [ Unicode ], a code point is an integer between 0x0000 and 0x10FFFF inclusive. It is written in hexadecimal notation (see 4.11 Referring to Unicode characters ). A Unicode code point is sometimes called a Unicode Scalar Value . Each code point that has been assigned to an abstract character in Unicode is also given a unique, immutable name. Unicode also associates various properties with each assigned character. Many of these properties appear in the Unicode Character Database (or UCD ]) [ UAX44 ], while others are assigned in ancillary files or tables. [ Unicode ] [D11] defines an encoded character as An association (or mapping) between an abstract character and a code point . The term code point , or, when greater specificity is needed, Unicode code point or Unicode Scalar Value , is generally preferred for specifications. See also 4.11 Referring to Unicode characters Code points are not used directly in the storage and interchange of characters in software. Instead, there exist various schemes for encoding and processing the characters they represent or for converting from one representation to another. A code unit is a unit of physical storage and information interchange, forming the basis for computer processing, storage, and communication. The most familiar code unit is called a byte or octet , and consists of 8 bits. Different sized code units are used by different runtime environments. Other common sizes include 16- or 32-bit units. On the Web, for example, 16-bit code units are used by the [ DOM ], JavaScript, and the various string types in [ INFRA ]. These specifications use 16-bit code units processed according to the rules of the UTF-16 character encoding of [ Unicode ]. See also 4.2 Choosing a definition of 'string' A character encoding form (sometimes just referred to as a character encoding ) is the set of rules for encoding from code points in a character set to the code units used to store and process text; or for decoding from code units back into code points . Non-Unicode character encoding forms are referred to collectively as legacy character encodings . See also The [ Encoding ] Specification 4.6 Choosing character encodings UTF-8 is a multibyte character encoding form of [ Unicode ]. It is the most common character encoding used on the Web. UTF-8 uses 8-bit bytes as its code unit . UTF-8 is a variable-width encoding, in that it uses different numbers of code units depending on the code point being encoded. The familiar 7-bit ASCII characters (the code points from U+0000 to U+007F ) take one byte to encode in UTF-8. Character Code Point UTF-8 Code Units (bytes) A U+0041 0x41 Code points from U+0080 and U+07FF require two bytes. À U+00C0 0xC3 0x80 Code points from U+0800 through U+FFFF take three bytes. न U+0928 0xE0 0xA4 0xA8 And, finally, code points from U+10000 through U+10FFFF take four bytes. 👪 U+1F46A 0xF0 0x9F 0x91 0xAA § Specifications, software, and content MUST NOT require or depend on a one-to-one relationship between characters ( code points ) and units of physical storage ( code units ). source Units of storage C009 , in Character Model for the World Wide Web 1.0: Fundamentals § Specifications, software, and content MUST NOT require or depend on a one-to-one mapping between code points and units of presentation (such as grapheme clusters , glyphs , or typographic character units). source Units of visual rendering C002 , in Character Model for the World Wide Web 1.0: Fundamentals . explanations & examples Examples of Characters, Keystrokes and Glyphs , in Character Model for the World Wide Web 1.0: Fundamentals . In this document, a visual text unit refers to a single unit of the visible text as perceived by a user. This can be on a screen or in other media, such as printed on paper or written on the back of a napkin. This term is necessarily imprecise, as user perception frequently depends on familiarity with the script and writing system, particularly when it comes to writing systems that use a combination of combining marks, complex positioning, or complex shaping based on context. Avoid using this term in your own specification. A grapheme cluster is the computed approximation of a visual text unit in encoded text, that is, it is a sequence of code points that are expected to form a single visual text unit from the user's point of view. [ Unicode ] provides a way to compute these boundaries to assist in processing text, since, for many text operations, such a sequence should be processed as a single, indivisible textual unit. For example, when cursoring across text, the cursor should "jump across" or select the entire visual text unit (and its underlying code points) together. It shouldn't be possible to cursor into the "middle" of a visual text unit. (Unless otherwise specified, the term grapheme cluster in this document refers to what Unicode Text Segmentation [ UAX29 ] refers to as an "extended default grapheme cluster".) Note that some text operations do allow users to interact with individual code points within a grapheme cluster . For example, some editing functions, such as backspacing, progressively delete characters from the end of a visual text unit (to allow users to correct misspellings without having the erase the entire cluster, for example). The term typographic character unit is defined by [ CSS ], where it is used to refer to different types of code point sequences that should be treated as "unitary" for specific operations. Sometimes this aligns with the term grapheme cluster , while other times it is distinct. Use this term only if you understand the context and usage. A glyph is the visual representation of a character (or sequence of characters) when rendered by a particular font . Glyphs do not always have a 1:1 relationship to abstract characters . They can represent part of a character or a combination of several characters. Different glyphs can represent the same code point . For example, these are all different glyphs for A U+0041 LATIN CAPITAL LETTER A : A font is thus a collection of specific glyphs used to render text. Example 8 : Encoding terminology illustrated Suppose you spoke Russian and suppose you wanted to express the sentiment Я люблю швейцарских коров! (which means, approximately, "I love Swiss cows") to someone using emoji. It might look like this: Figure 1 As text: Я❤️🇨🇭🐄! This phrase contains five visual text units ( grapheme clusters ). grapheme cluster Я ❤️ 🇨🇭 🐄 ! Some of the visual text units require more than one character. This particular phrase consists of seven characters because these emoji characters use some of the more unusual features of Unicode. ❤ U+2764 HEAVY BLACK HEART is followed by U+FE0F VARIATION SELECTOR-16 in order to display as an emoji variant, while the flag of Switzerland is formed using a pair of emoji flag characters. (Note: on some browsers, the "flag" might display as a grapheme cluster with the letters CH . These are still treated by the browser as a single grapheme cluster!) Encoded character Я ❤ 🇨 🇭 🐄 ! Each encoded character is assigned a Unicode code point (it's Unicode Scalar Value ), a unique integer that identifies which character we mean: code points U+042F U+2764 U+FE0F U+1F1E8 U+1F1ED U+1F404 U+0021 Each code point is encoded into the UTF-8 character encoding form using a varying number of code units . In UTF-8, the code units are bytes, while in UTF-16 they are 16-bit integers: UTF-8 code units D0 AF E2 9D A4 EF B8 8F F0 9F 87 A8 F0 9F 87 AD F0 9F 90 84 21 UTF-16 code units 042F 2764 FE0F D83C DDE8 D83C D8ED D83D DC04 0021 § Specifications, software, and content MUST NOT require or depend on a one-to-one correspondence between characters and the sounds of a language. source Units of aural rendering C001 , in Character Model for the World Wide Web 1.0: Fundamentals In some scripts, characters have a close relationship to phonemes (a phoneme is a minimally distinct sound in the context of a particular spoken language), while in others they are closely related to meanings. Even when characters (loosely) correspond to phonemes, this relationship may not be simple, and there is rarely a one-to-one correspondence between character and phoneme. The following are examples of mismatches between the term character and units of sound: In the English sentence, They were too close to the door to close it. the same character s is used to represent both /s/ and /z/ phonemes. In the English language the phoneme /k/ of cool is like the phoneme /k/ of keel . In many scripts a single character may represent a sequence of phonemes, such as the syllabic characters of Japanese hiragana. In many writing systems a sequence of characters may represent a single phoneme, for example th and ng in thing . § Specifications and software MUST NOT require nor depend on a single keystroke resulting in a single character, nor that a single character be input with a single keystroke (even with modifiers), nor that keyboards are the same all over the world. source Units of input C005 , in Character Model for the World Wide Web 1.0: Fundamentals . explanations & examples Examples of Characters, Keystrokes and Glyphs , in Character Model for the World Wide Web 1.0: Fundamentals . In keyboard input, it is not always the case that keystrokes and input characters correspond one-to-one. A limited number of keys can fit on a keyboard. Some keyboards will generate multiple code points from a single keypress. In other cases (' dead keys ') a key will generate no characters, but affect the results of subsequent keypresses. Many writing systems have far too many characters to fit on a keyboard and must rely on more complex input methods , which transform keystroke sequences into character sequences. Other languages may make it necessary to input some characters with special modifier keys. See Examples of Characters, Keystrokes and Glyphs for examples of non-trivial input. 4.2 Choosing a definition of 'string' Useful background and overviews for this section String concepts , in Character Model for the World Wide Web: Fundamentals . See also 6.1 Choosing text units for segmentation, indexing, etc. . 4.1 Characters and character encoding basics . Note The best practices found in this section are intended to be mutually consistent with those in [ DESIGN-PRINCIPLES ]. The definitions in this section use terms found in the Internationalization Glossary [ I18N-GLOSSARY ], in [ WEBIDL ], in [ INFRA ], or the Unicode glossary. Please refer to instructions in the Internationalization Glossary for how to import and link definitions in your own specification. A string is usually understood to be a sequence of 'characters'. Because [ Unicode ] is fundamental to understanding and working with text, including text that uses legacy character encodings , the basic definition of a string depends on Unicode and its concept of an encoded character . Specifically: A string is a well-formed sequence of zero or more Unicode Scalar Values . Because there are multiple ways of working with strings, different definitions of "string" have evolved to support the needs of different specifications. Be sure to understand your specification's needs and use the most appropriate and precise definition. When choosing a string type, consider the requirements of the entire operation, including other components or platform APIs to which the string is passed, rather than only the algorithms defined directly in your specification. On the Web, there are three types of strings: USVString . Strings consisting of a sequence of one or more Unicode Scalar Values , that is, each code unit in the string is a Unicode code point . DOMString . Strings consisting of a sequence of one or more UTF-16 code units . ByteString . Strings consisting of a sequence of one or more bytes that are interpreted using some specific character encoding form (preferably UTF-8 ). One difference between these different string types is how surrogate code points are handled. A Unicode Scalar Value is a Unicode code point other than a surrogate code point. A code unit is a unit of encoding in a character encoding form . The UTF-16 character encoding form uses 16-bit code units . In this encoding form, characters whose scalar values require more than 16 bits are encoded using a pair of surrogate code units : a "high surrogate" (in the range U+D800-U+DBFF ) followed by a "low surrogate" (in the range U+DC00-U+DFFF ). Unicode reserves the code points in these ranges exclusively for use as UTF-16 surrogate code units. They are not Unicode Scalar Values. In a USVString , isolated surrogate code points are invalid and implementations are required to replace any found in a string with the Unicode replacement character ( � U+FFFD REPLACEMENT CHARACTER ). For strings whose most common algorithms operate on scalar values (such as percent-encoding), or for operations which can’t handle surrogates in input (such as APIs that pass strings through to native platform APIs), USVString should be used. Any of these references are equivalent to this: USVString [ WEBIDL ] scalar value string [ INFRA ] xsd:string [ XMLSCHEMA11-2 ] In a DOMString , unpaired surrogate code units can appear in a string. Most string operations don’t need to interpret the code units inside of strings. Specifying DOMString means that implementations are not required to validate the contents of the string, making this the ideal string type for most data structures, formats, or APIs. The [ DOM ] and JavaScript strings use DOMString as their string type and the [ INFRA ] standard defines the term 'string' to mean a DOMString : A string is a sequence of unsigned 16-bit integers, also known as code units . Note [ INFRA ]'s use of the term code unit refers specifically to the UTF-16 character encoding's code units, rather than the more general definition of a code unit that can refer to different size values, such as bytes, in any character encoding form . A ByteString depends on the character encoding form used to encode characters into bytes. Legacy character encodings do not have a concept of "surrogates", so there is generally no way to encode a surrogate code point. Valid UTF-8 does not permit surrogate code points: these are replaced by � U+FFFD REPLACEMENT CHARACTER when encoding text to or decoding text from UTF-8 . When converting UTF-16 to UTF-8 , any surrogate pairs are transformed into the proper UTF-8 byte sequence encoding the specific scalar value . § Use a DOMString when defining document formats or the wire format of a protocol, or for any process pertaining to the [ DOM ], or which defines strings as opaque values whose individual character contents are not meant to be evaluated. (This list of uses is not exhaustive.) explanations & examples Definition of string in [ INFRA ] IDL String Types in Web Platform Design Principles [ DESIGN-PRINCIPLES ] String concepts, C012 , in Character Model for the World Wide Web: Fundamentals . § Use a USVString for strings whose most common algorithms operate on Unicode Scalar Values , or for operations which can't handle surrogates in input. This includes algorithms that iterate over the code points in a string, processes that involve UTF-8 encode , and operations that pass the string to another component or API for which an unpaired surrogate code point would produce an error. explanations & examples Scalar value string definition in [ INFRA ] § Avoid mixing DOMString and USVString in a single document or protocol operation. Often you will choose a DOMString over a USVString , since the latter requires extra processing that does not benefit most document formats or protocols. 4.2.1 Characters stored in byte sequences See also 4.6 Choosing character encodings for additional best practices related to character encodings . Strings that are part of a legacy protocol or format , in Strings on the Web: Language and Direction Metadata [ STRING-META ] Prior to the widespread adoption of Unicode, it was common to define a string as a byte string . Such a string is simply a sequence of byte values rather than a sequence of characters or code points . A familiar manifestation of byte strings is a char* in the C programming language. Processing or interpreting a byte string depends on the character encoding form . Many legacy character encodings are stateful: processing such encodings often requires starting at the beginning of the byte buffer, so that character state is retained and the abstract character can be decoded, processed, or modified successfully. A given byte value in such an encoding might mean different things depending on the bytes adjacent to it. For example, the exact same byte value might stand alone to represent a character or, depending on the preceding bytes, be part of a multibyte sequence that represents some different character. The rules for determining how to interpret each byte or byte sequence are different for different legacy character encodings . Processing a byte string using the wrong character encoding results in malformed characters (an effect sometimes called mojibake ). See also String concepts in Character Model for the World Wide Web 1.0: Fundamentals ) UTF-8 is the preferred character encoding for wire and document formats on the Web [ ENCODING ] or the Internet in general [ RFC3629 ]. When content is encoded in UTF-8, there is rarely a reason to interact with it as a byte sequence. Most Web APIs and interfaces are more concerned with the code point sequence, since that represents the characters in question, rather than the specific byte values. Sometimes specifications do need to deal with the storage, interpretation, and manipulation of byte values. In particular, many document formats and protocols were defined around the use of 7-bit [ ASCII ] bytes, while allowing the inclusion or interchange of non-ASCII data values via the use of various character or data encoding schemes. Sometimes this is done by designating a character encoding form , such as with the charset parameter of the text media types. Or it might be done by encoding byte values using some special syntax, an example of which would be percent encoding . The prevalence of UTF-8, including as the preferred and default character encoding form , diminishes the need for most specifications to access or manipulate underlying byte values. UTF-8 is designed such that 7-bit ASCII text is also valid UTF-8, which can be important when dealing with encoding or decoding formats based on ASCII. Example 9 : Working with non-ASCII data in an HTTP header HTTP requests sometimes need to transmit non-ASCII data without the use of a transfer encoding (such as Base64 [ RFC4648 ]). One header that is often used in this manner is Content-Disposition , which is used for operations such as file uploads. (To ensure interoperability, the filename is always encoded in UTF-8.) Such a request might contain lines like: POST /file-upload HTTP/1.1 ... Content-Disposition: attachment; filename="Åsa.txt" The characteristics of UTF-8 mean that most of the byte values in the header are also their Unicode code points . For example, here is the word "filename": String f i l e n a m e Byte 0x66 0x69 0x6C 0x65 0x6E 0x61 0x6D 0x65 Code Point U+0066 U+0069 U+006C U+0065 U+006E U+0061 U+006D U+0065 The filename itself, however, contains a non-ASCII character: Å U+00C5 LATIN CAPITAL LETTER A WITH RING ABOVE . That character is encoded in UTF-8 as two bytes: 0xC3 0x85 . String Å s a . t x t Bytes 0xC3 0x85 0x73 0x61 0x2E 0x74 0x78 0x74 Code Point U+00C5 U+0073 U+0061 U+002E U+0074 U+0078 U+0074 If an application wishes to process the individual bytes, specifying a ByteString , which uses an isomorphic coding, exposes the underlying bytes as code points in the converted string. An application could then do processing on the byte values as-if they were characters: Isomorphic coding of Åsa.txt with UTF-8 as the character encoding String Å s a . t x t Bytes 0xC3 0x85 0x73 0x61 0x2E 0x74 0x78 0x74 Code Point U+00C3 U+0085 U+0073 U+0061 U+002E U+0074 U+0078 U+0074 § Specify DOMString or, rarely, USVString for fields in protocols or document formats defined using bytes unless there is some reason to interact with specific bytes values or for which the UTF-8 character encoding cannot be assumed. If the field in question is meant to be treated as a string, working with Unicode characters will be more reliable than trying to work with the byte values directly. The data encoded into these fields will be deserialized from the wire format into your local in-memory string representation, such as the [ DOM ], JavaScript strings, or your platform's native Unicode string type. Later it will need to be serialized into the wire format using some character encoding form (usually—and preferably —UTF-8). § Specify Uint8Array when working with byte sequences, such as for data that does not contain text, or for byte sequences representing text for which processing is never required (such as when copying buffers). § Specify ByteString in the rare cases where the specification needs to work with strings which are encoded using bytes and for which the conversion to or from Unicode would be inappropriate. explanations & examples IDL String Types in Web Platform Design Principles [ DESIGN-PRINCIPLES ] ByteString isn’t a general-purpose string type. Don't use it to define data structures in [ WebIDL ]. 4.3 Defining a Reference Processing Model Useful background and overviews for this section Digital Encoding of Characters , in Character Model for the World Wide Web: Fundamentals . See also 4.4 Including and excluding character ranges . § Textual data objects defined by protocol or format specifications MUST be in a single character encoding. explanations & examples Reference Processing Model C013 , in Character Model for the World Wide Web: Fundamentals § All specifications that involve processing of text MUST specify the processing of text according to the Reference Processing Model described by the rest of the recommendations in this list. explanations & examples Reference Processing Model C014 , in Character Model for the World Wide Web: Fundamentals § Specifications MUST define text in terms of Unicode characters, not bytes or glyphs. explanations & examples Reference Processing Model C014 , in Character Model for the World Wide Web: Fundamentals § For their textual data objects specifications MAY allow use of any character encoding which can be transcoded to a Unicode encoding form. explanations & examples Reference Processing Model C014 , in Character Model for the World Wide Web: Fundamentals § Specifications MAY choose to disallow or deprecate some character encodings and to make others mandatory. Independent of the actual character encoding, the specified behavior MUST be the same as if the processing happened as follows: (a) The character encoding of any textual data object received by the application implementing the specification MUST be determined and the data object MUST be interpreted as a sequence of Unicode characters - this MUST be equivalent to transcoding the data object to some Unicode encoding form, adjusting any character encoding label if necessary, and receiving it in that Unicode encoding form, (b) All processing MUST take place on this sequence of Unicode characters, (c) If text is output by the application, the sequence of Unicode characters MUST be encoded using a character encoding chosen among those allowed by the specification. explanations & examples Reference Processing Model C014 , in Character Model for the World Wide Web: Fundamentals § If a specification is such that multiple textual data objects are involved (such as an XML document referring to external parsed entities), it MAY choose to allow these data objects to be in different character encodings. In all cases, the Reference Processing Model MUST be applied to all textual data objects. explanations & examples Reference Processing Model C014 , in Character Model for the World Wide Web: Fundamentals 4.4 Including and excluding character ranges See related review comments. Useful background and overviews for this section Digital Encoding of Characters , in Character Model for the World Wide Web: Fundamentals . See also 8.3 Defining keywords, identifiers, and namespaces in syntactic content (additional guidance when defining identifiers and syntactic content) 4.5 Using the Private Use Area . § Specifications SHOULD NOT arbitrarily exclude code points from the full range of Unicode code points from U+0000 to U+10FFFF inclusive. explanations & examples Reference Processing Model C070 , in Character Model for the World Wide Web: Fundamentals . § Specifications MUST NOT allow code points above U+10FFFF. explanations & examples Reference Processing Model C077 , in Character Model for the World Wide Web: Fundamentals . § Specifications SHOULD NOT allow the use of codepoints reserved by Unicode for internal use. explanations & examples Reference Processing Model C079 , in Character Model for the World Wide Web: Fundamentals . § Specifications MUST NOT allow the use of unpaired surrogate code points. explanations & examples Reference Processing Model C078 , in Character Model for the World Wide Web: Fundamentals . A "surrogate code point" refers here to the use of character values in the range U+D800 through U+DFFF inclusive. These code points are reserved to allow the UTF-16 character encoding to address supplementary characters . Surrogates are always used in pairs and only appear when the UTF-16 encoding is being used. A single surrogate code point is referred to as an "unpaired surrogate" and should never be used. § Specifications SHOULD exclude compatibility characters in the syntactic elements (markup, delimiters, identifiers) of the formats they define. explanations & examples Compatibility and Formatting Characters C050 , in Character Model for the World Wide Web: Fundamentals . § Specifications SHOULD allow the full range of Unicode for user-defined values. explanations & examples Unicode case-insensitive matching , in Character Model for the World Wide Web: Fundamentals . 4.5 Using the Private Use Area See also 4.4 Including and excluding character ranges . § Specifications MUST NOT require the use of private use area characters with particular assignments. explanations & examples Private use code points, C038 , in Character Model for the World Wide Web: Fundamentals § Specifications MUST NOT require the use of mechanisms for defining agreements of private use code points. explanations & examples Private use code points, C039 , in Character Model for the World Wide Web: Fundamentals § Specifications and implementations SHOULD NOT disallow the use of private use code points by private agreement. explanations & examples Private use code points, C040 , in Character Model for the World Wide Web: Fundamentals § Specifications MAY define markup to allow the transmission of symbols not in Unicode or to identify specific variants of Unicode characters. explanations & examples Private use code points, C041 , in Character Model for the World Wide Web: Fundamentals § Specifications SHOULD allow the inclusion of or reference to pictures and graphics where appropriate, to eliminate the need to (mis)use character-oriented mechanisms for pictures or graphics. explanations & examples Private use code points, C068 , in Character Model for the World Wide Web: Fundamentals 4.6 Choosing character encodings See related review comments. Useful background and overviews for this section Choice and identification of code points in Character Model for the World Wide Web: Fundamentals . Document character set — what it is, and how it relates to the encodings used for a document. Historically (and especially in the period before Unicode was created), there were many coded character sets in common use, with different schemes for encoding and serializing characters into the memory or storage of computer systems. In addition to standards-based schemes, such as those specified by ISO/IEC 8859, there were also many proprietary vendor or platform-specific character sets , often with associated character encoding forms . When referring to the character encoding form of legacy (non-Unicode) coded character sets in this document, we mean the specific modern mappings of bytes to Unicode code points, as specified in [ Encoding ]. § Use UTF-8 for all document formats, protocols, and serialization forms. UTF-8 is the best choice for nearly all applications. Note Web APIs and text processing are usually specified using strings rather than trying to grappple with the raw byte sequences in a specific character encoding form . As noted in 4.2 Choosing a definition of 'string' , these strings are typically represented using UTF-16 code units ( DOMString ) or, less commonly, as Unicode code points ( USVString ). UTF-16 is a comparatively poor choice for serialization. Since the implicit conversion between UTF-8 and the various string representations is mostly invisible to users, UTF-8 is therefore the preferred character encoding for storage and transmission. § For all new formats or protocols or where a specification can safely do so, specifications MUST define UTF-8 as the only permitted character encoding form . New protocols and formats, as well as existing formats deployed in new contexts, are required to use the UTF-8 character encoding. This policy applies to IETF and Web standards and is articulated in [ RFC2277 ], [ RFC3629 ], [ Encoding ], [ design-principles ], and many more. The only specifications that need legacy character encodings are those that work with older protocols or formats and even there UTF-8 is strongly recommended. § If, for historical reasons, a specification permits legacy character encodings , it MUST restrict the set of character encodings to those listed in the Encoding Standard in the section "Names and Labels". Other encodings SHOULD NOT be used, except by private agreement. explanations & examples Character encoding identification, C021 , in Character Model for the World Wide Web: Fundamentals Character encoding identification, C022 , in Character Model for the World Wide Web: Fundamentals Character encoding identification, C023 , in Character Model for the World Wide Web: Fundamentals 4.7 Identifying character encodings Useful background and overviews for this section Choice and identification of code points in Character Model for the World Wide Web: Fundamentals . Document character set — what it is, and how it relates to the encodings used for a document. § Specifications that allow multiple character encoding forms MUST provide a mechanism, such as a field or parameter, that clearly identifies the encoding of text. explanations & examples Choice and Identification of Character Encodings, C015 , in Character Model for the World Wide Web: Fundamentals Character encodings cannot be reliably detected just from the byte values. If encodings other than UTF-8 are permitted, there has to be some mechanism for the consumer to determine what the encoding is. Example 10 : Examples of character encoding identification mechanisms Here are a few examples of ways that some common specifications indicate the character encoding form used in a given document or protocol: Format Example Note XML <?xml version="1.0" encoding="UTF-8" ?> HTML <html> <meta charset="UTF-8"> ... MIME type=text/* Content-Type: text/plain ;charset=UTF-8 New MIME types should not specify a charset parameter. They should always assume UTF-8 instead. § If a protocol, format, or API is based on a format that already has rules for choosing, applying, or labeling the character encoding , the specification MUST NOT define a separate mechanism for identifying the character encoding . explanations & examples Choice and Identification of Character Encodings, C017 , in Character Model for the World Wide Web: Fundamentals § If a specification is based on a format that permits character encodings other than UTF-8, the specification SHOULD restrict the character encoding to UTF-8. Document formats or protocols sometimes provide support for legacy character encodings . Specifications built upon those formats, where it is feasible to do so, can specify that conformant implementations use only UTF-8. § Specifications MUST NOT propose the use of heuristics to determine the encoding of data. explanations & examples Character encoding identification, C028 , in Character Model for the World Wide Web: Fundamentals § Specifications MUST define conflict-resolution mechanisms (e.g. priorities) for cases where there is multiple or conflicting information about character encoding . explanations & examples Character encoding identification, C028 , in Character Model for the World Wide Web: Fundamentals 4.8 Designing character escapes See related review comments. Useful background and overviews for this section Character Escaping , in Character Model for the World Wide Web: Fundamentals . § Specifications should provide a mechanism for escaping characters, particularly those which are invisible or ambiguous. more Using character escapes in markup and CSS , W3C article. It is generally recommended that character escapes be provided so that difficult to enter or edit sequences can be introduced using a plain text editor. Escape sequences are particularly useful for invisible or ambiguous Unicode characters, including zero-width spaces, soft-hyphens, various bidi controls, mongolian vowel separators, etc. For advice on use of escapes in markup, but which is mostly generalisable to other formats, see Using character escapes in markup and CSS . § Specifications SHOULD NOT invent a new escaping mechanism if an appropriate one already exists. explanations & examples Character Escaping, C042 , in Character Model for the World Wide Web: Fundamentals Here are some examples of common escaping mechanisms found on the Web or in common programming languages. The example character here is 😽 U+1F63D KISSING CAT FACE WITH CLOSED EYES . Found In Type Example Description HTML, XML Hex NCRs &#x1F63D; Hexadecimal encoding of the Unicode code point Decimal NCRs &#128573; Decimal encoding of the Unicode code point JavaScript, Ruby, Rust, [ UTS18 ] \u delimited \u{1F63D} Hexadecimal encoding of the Unicode code point Perl \x delimited \x{1F63D} Hexadecimal encoding of the Unicode code point; uses x instead of the more common u Java, JavaScript, JSON, C, C++, Python \u UTF-16 code units \uD83D\uDE3D Fixed-width hexadecimal encoding of UTF-16 code units; supplementary characters are encoded as a surrogate pair C, C++, Python \U UTF-32 code units \U0001f63d Fixed-width hexadecimal encoding of UTF-32 code units; most often used together with \u escapes (which are more efficient for the more-common BMP characters). For example, \u00c0 \U0001f63d \u12fe URLs URL Encode %F0%9F%98%BD Hexadecimal encoding of UTF-8 bytes; each byte requires three characters; each code point requires from 1 to 4 bytes When choosing an escaping mechanism, note that hexadecimal is generally preferred to decimal encodings, due to the common use of hexadecimal in the Unicode Standard and its references. § The number of different ways to escape a character SHOULD be minimized (ideally to one). explanations & examples Character Escaping, C043 , in Character Model for the World Wide Web: Fundamentals § Escape syntax MUST either use an explicit end delimiter, such as in &#x12AB; or \u{12F} , or a fixed number of digits, such as in \u12AB or \U001234AB . explanations & examples Character Escaping, C044 , in Character Model for the World Wide Web: Fundamentals § End-delimited forms are RECOMMENDED . End-delimited escapes can use only the digits needed for the code point, which makes them more compact than fixed-width forms, and the closing delimiter makes it easier to see where the escape ends. § Whenever specifications define character escapes that allow the representation of characters using a number, the number MUST represent the Unicode code point of the character and SHOULD be in hexadecimal notation. explanations & examples Character Escaping, C045 , in Character Model for the World Wide Web: Fundamentals § Escaped characters SHOULD be acceptable wherever their unescaped forms are; this does not preclude that syntax-significant characters, when escaped, lose their significance in the syntax. In particular, if a character is acceptable in identifiers and comments, then its escaped form should also be acceptable. explanations & examples Character Escaping, C046 , in Character Model for the World Wide Web: Fundamentals 4.9 Storing text Useful background and overviews for this section Visual rendering and logical order , in Character Model for the World Wide Web: Fundamentals . § Protocols, data formats and APIs MUST store, interchange or process text data in logical order. explanations & examples Visual Rendering and Logical Order, C003 , in Character Model for the World Wide Web: Fundamentals . § Independent of whether some implementation uses logical selection or visual selection, characters selected MUST be kept in logical order in storage. explanations & examples Visual Rendering and Logical Order, C075 , in Character Model for the World Wide Web: Fundamentals . § Specifications of protocols and APIs that involve selection of ranges SHOULD provide for discontiguous logical selections, at least to the extent necessary to support implementation of visual selection on screen on top of those protocols and APIs. explanations & examples Visual Rendering and Logical Order, C004 , in Character Model for the World Wide Web: Fundamentals . 4.10 Whitespace characters See related review comments. See also 8.3 Defining keywords, identifiers, and namespaces in syntactic content . Whitespace characters are characters that represent horizontal or vertical space in typography. Whitespace characters can have different visual effects: some whitespace characters have no visible effect, while others represent larger, smaller, or variable amounts of space on the page. § Specifications that use the term "whitespace" SHOULD explicitly define what the term means. explanations & examples ASCII whitespace , in Infra Living Standard Whitespace , in CSS Syntax Document white space characters , in CSS Text White Space , in ECMAScript Language Specification Blankspace and line breaks , in WebGPU Shading Language Whitespace , in WebDriver S (white space) , in Extensible Markup Language (XML) 1.0 § Most specifications SHOULD define whitespace to mean characters with the Unicode White_Space property. § Specifications that define whitespace for use in vocabularies that are restricted to ASCII or to formats that are whitespace delimited (examples include HTML or CSS) SHOULD specify ASCII whitespace as part of their grammar. § If a specification defines "whitespace" differently from ASCII or Unicode whitespace, the specific code points MUST be listed. Some specifications, such as ECMAScript , have provided their own definition of whitespace which differ from the above to meet their own specific requirements. The following table is the definition of whitespace characters in various specifications. Links to the latest definitions of the information in the table can be found by expanding the "explanations & examples". white_space property pattern_white_space property ASCII whitespace (HTML) CSS whitespace ECMAScript XML U+0009 (horizontal tab) ✓ ✓ ✓ ✓ ✓ ✓ U+000A (line feed) ✓ ✓ ✓ ✓ ✓ U+000B (vertical tab) ✓ ✓ ✓ U+000C (form feed) ✓ ✓ ✓ ✓ U+000D (carriage return) ✓ ✓ ✓ ✓ U+0020 SPACE ✓ ✓ ✓ ✓ ✓ ✓ U+0085 (next line) ✓ ✓ U+00A0 NO-BREAK SPACE ✓ ✓ U+1680 OGHAM SPACE MARK ✓ ✓ U+2000 EN QUAD ✓ ✓ U+2001 EM QUAD ✓ ✓ U+2002 EN SPACE ✓ ✓ U+2003 EM SPACE ✓ ✓ U+2004 THREE-PER-EM SPACE ✓ ✓ U+2005 FOUR-PER-EM SPACE ✓ ✓ U+2006 SIX-PER-EM SPACE ✓ ✓ U+2007 FIGURE SPACE ✓ ✓ U+2008 PUNCTUATION SPACE ✓ ✓ U+2009 THIN SPACE ✓ ✓ U+200A HAIR SPACE ✓ ✓ U+200E LEFT-TO-RIGHT MARK ✓ U+200F RIGHT-TO-LEFT MARK ✓ U+2028 LINE SEPARATOR ✓ ✓ U+2029 PARAGRAPH SEPARATOR ✓ ✓ U+202F NARROW NO-BREAK SPACE ✓ ✓ U+205F MEDIUM MATHEMATICAL SPACE ✓ ✓ U+3000 IDEOGRAPHIC SPACE ✓ ✓ U+FEFF ZERO WIDTH NO-BREAK SPACE ✓ Some specifications use the same definition as one of the columns above and are not listed in the table. For example, WebDriver uses the white_space property and WebGPU Shading Language uses the pattern_white_space property. 4.11 Referring to Unicode characters See related review comments. Useful background and overviews for this section I18N Editing Guidelines — more information on character formatting § Use U+XXXX syntax to represent Unicode code points in a specification. The U+XXXX format is well understood when referring to Unicode code points in a specification. These are space separated when appearing in a sequence. No additional decoration is needed. Note that a code point may contain four, five, or six hexadecimal digits. When fewer than four digits are needed, the code point number is zero filled. Example 11 : Code point examples Character Unicode Name U+XXXX syntax e LATIN SMALL LETTER E U+0065 é LATIN SMALL LETTER E WITH ACUTE U+00E9 ઊ GUJURATI LETTER UU U+0A8A � REPLACEMENT CHARACTER U+FFFD 😮 FACE WITH OPEN MOUTH U+1F62E § Use the Unicode character name to describe specific code points. Unicode assigns unique, immutable names to each assigned Unicode code point. Using these names in your specification when referring to specific characters (along with the code point in U+XXXX notation) will help make your specification unambiguous. § Use of the character naming template is RECOMMENDED . For most characters, the template looks like this: <span class="codepoint" translate="no"><bdi lang="??">&#xXXXX;</bdi><code class="uname">U+XXXX UNICODE_CHARACTER_NAME_ALL_IN_CAPS</code></span> Example 12 : Example of a character reference Filling in the above template like this: <span class="codepoint" translate="no"><bdi lang="fr">é</bdi><code class="uname">U+00E9 LATIN SMALL LETTER E WITH ACUTE</code></span> Produces output in the page like this: é U+00E9 LATIN SMALL LETTER E WITH ACUTE . The bdi element is used to ensure that example characters that are right-to-left do not interfere with the layout of the page. Do not include line breaks or a space between the closing bdi and the following code element; spacing and presentation is controlled by styling. The lang attribute should be filled in appropriately to get the correct font selection for a given context. Examples in East Asian languages (such as Chinese, Japanese, or Korean) or in the Arabic script can sometimes require greater care in choosing a language tag. Rarely, for certain languages, it might be necessary to adjust the style of the bdi element with a font-family and/or font-size in your own stylesheet. For invisible characters (such as control characters), combining characters, or for whitespace, use an image instead of the character; or you may also omit the character and its surrounding bdi element. <span class="codepoint" translate="no"><img alt="..." src="..."><code class="uname">U+XXXX UNICODE_CHARACTER_NAME_ALL_IN_CAPS</code></span> Example 13 : Example of a character reference with an image Filling in the above template like this: <span class="codepoint" translate="no"><img alt="NBSP" src="images/00A0.png"><code class="uname">U+00A0 NO-BREAK SPACE</code></span> Produces output in the page like this: U+00A0 NO-BREAK SPACE . Short sequences of characters should list the character names, separated by + . Example 14 : Example of a code point sequence This example: नि U+0928 DEVANAGARI LETTER NA + U+093F DEVANAGARI VOWEL SIGN I uses the following markup: <span class="codepoint" translate="no"><bdi lang="hi">&#x0928;&#x093f;</bdi><code class="uname">U+0928 DEVANAGARI LETTER NA</code> + <code class="uname">U+093F DEVANAGARI VOWEL SIGN I</code></span> There are cases where including the character name and additional markup is overly pedantic and detracts from usability, but be cautious about being so informal as to impair meaning. In particular, long sequences will sometimes just list the code points, although the character names should be retained where possible for clarity. An example can be found in this document in the discussion of the composed "family" emoji : 👨‍👩‍👧‍👧 U+1F468 U+200D U+1F469 U+200D U+1F467 U+200D U+1F467 5. Referencing the Unicode Standard See related review comments. Useful background and overviews for this section Referencing the Unicode Standard and ISO/IEC 10646 , in Character Model for the World Wide Web: Fundamentals . § Since specifications in general need both a definition for their characters and the semantics associated with these characters, specifications SHOULD include a reference to the Unicode Standard, whether or not they include a reference to ISO/IEC 10646. explanations & examples Referencing the Unicode Standard and ISO/IEC 10646, C062 , in Character Model for the World Wide Web: Fundamentals . § A generic reference to the Unicode Standard MUST be made if it is desired that characters allocated after a specification is published are usable with that specification. A specific reference to the Unicode Standard MAY be included to ensure that functionality depending on a particular version is available and will not change over time. explanations & examples Referencing the Unicode Standard and ISO/IEC 10646, C063 , in Character Model for the World Wide Web: Fundamentals . § All generic references to the Unicode Standard MUST refer to the latest version of the Unicode Standard available at the date of publication of the containing specification. explanations & examples Referencing the Unicode Standard and ISO/IEC 10646, C064 , in Character Model for the World Wide Web: Fundamentals . § All generic references to ISO/IEC 10646 MUST refer to the latest version of ISO/IEC 10646 available at the date of publication of the containing specification. explanations & examples Referencing the Unicode Standard and ISO/IEC 10646, C065 , in Character Model for the World Wide Web: Fundamentals . 6. Text-processing Show the self-review checklist This is a list of just the requirements in this section, which can be used for self-reviews. Select the first checkbox on a line for all requirements relevant to your spec. Select the second checkbox if your spec meets the requirement. Then click on the button "Create markdown for GitHub", and copy the result to a GitHub issue list. See more details . Choosing text units for segmentation, indexing, etc. The character string is RECOMMENDED as a basis for string indexing. more Grapheme clusters MAY be used as a basis for string indexing in applications where user interaction is the primary concern. more Specifications that define indexing in terms of grapheme clusters MUST either: (a) define grapheme clusters in terms of extended grapheme clusters as defined in Unicode Standard Annex #29, Unicode Text Segmentation (UTR #29), or (b) define specifically how tailoring is applied to the indexing operation. more The use of byte strings for indexing is NOT RECOMMENDED . more A UTF-16 code unit string is NOT RECOMMENDED as a basis for string indexing, even if this results in a significant improvement in the efficiency of internal operations when compared to the use of character string. more Specifications that need a way to identify substrings or point within a string SHOULD consider ways other than string indexing to perform this operation. more Specifications SHOULD understand and process single characters as substrings, and treat indices as boundary positions between counting units, regardless of the choice of counting units. more Specifications of APIs SHOULD NOT specify single characters or single 'units of encoding' as argument or return types. more When the positions between the units are counted for string indexing, starting with an index of 0 for the position at the start of the string is the RECOMMENDED solution, with the last index then being equal to the number of counting units in the string. more Matching string identity for identifiers and syntactic content Define string identity matching for identifiers and syntactic content using the following steps: Ensure the strings to be compared consist of a sequence of Unicode code points. Expand all character escapes and includes. If defined, perform any case-folding and Unicode normalization steps (mulitple passes may be needed). Perform any additional matching tailoring specific to the specification. Compare the resulting sequences of code points for identity. more Do not specify normalization (ie. case folding or Unicode Normalization) of identifiers and syntactic content, unless you have a specific reason to do so. more 'ASCII case fold' SHOULD only be used for application internal identifiers that are restricted to ASCII. more Avoid specifying 'Unicode canonical case fold' unless you have a good reason. more Do not specify a 'Unicode compatibility case fold' . more Specifications of vocabularies MUST define the boundaries between syntactic content and character data as well as entity boundaries (if the language has any include mechanism). more Working with Unicode Normalization Specifications SHOULD NOT specify a Unicode normalization form for encoding, storage, or interchange of a given vocabulary. more Implementations MUST NOT alter the normalization form of textual data being exchanged, read, parsed, or processed except when required to do so as a side-effect of text transformation such as transcoding the content to a Unicode character encoding, case folding, or other user-initiated change, as consumers or the content itself might depend on the de-normalized representation. more Specifications SHOULD NOT specify compatibility normalization forms (NFKC, NFKD). more Specifications MUST document or provide a health-warning if canonically equivalent but disjoint Unicode character sequences represent a security issue. more Where operations can produce denormalized output from normalized text input, specifications MUST define whether the resulting output is required to be normalized or not. Specifications MAY state that performing normalization is optional for some operations; in this case the default SHOULD be that normalization is performed, and an explicit option SHOULD be used to switch normalization off. more Specifications that require normalization MUST NOT make the implementation of normalization optional. more Normalization-sensitive operations MUST NOT be performed unless the implementation has first either confirmed through inspection that the text is in normalized form or it has re-normalized the text itself. Private agreements MAY be created within private systems which are not subject to these rules, but any externally observable results MUST be the same as if the rules had been obeyed. more A normalizing text-processing component which modifies text and performs normalization-sensitive operations MUST behave as if normalization took place after each modification, so that any subsequent normalization-sensitive operations always behave as if they were dealing with normalized text. more Specifications that perform comparison or matching of string values SHOULD specify the appropriate note or warning regarding Unicode normalization. more Case folding Specifications and implementations that define string matching as part of the definition of a format, protocol, or formal language (which might include operations such as parsing, matching, tokenizing, etc.) MUST define the criteria and matching forms used. These MUST be one of: (a) case-sensitive (b) Unicode case-insensitive using Unicode full case-folding (c) ASCII case-insensitive. more Case-sensitive matching is RECOMMENDED for matching syntactic content, including user-defined values. more Specifications that define case-insensitive matching in vocabularies that include more than the Basic Latin (ASCII) range of Unicode MUST specify Unicode full casefold matching. more Specifications that define case-insensitive matching in vocabularies limited to the Basic Latin (ASCII) subset of Unicode MAY specify ASCII case-insensitive matching. more If language-sensitive case-sensitive matching is specified, Unicode case mappings SHOULD be tailored according to language and the source of the language used for each tailoring MUST be specified. more Specifications that define case-insensitive matching in vocabularies SHOULD NOT specify language-sensitive case-insensitive matching. more Truncating or limiting the length of strings Specifications SHOULD NOT impose a limit on the length of a string unless there is a specific practical or technical limitation. more If a specification specifies a length limit, it SHOULD specify that any string that has been truncated includes an indicator that the string has been altered, such as an ellipsis. more Specifications that limit the length of a string MUST specify whether the limit is counted in Unicode code points , or is counted in code units (such as bytes) of a given character encoding . more Avoid specifying the truncation of a stored string using some number of visual text units (such as grapheme clusters ). more Specifications that limit the maximum allowable storage length of a string SHOULD specify that length in terms of Unicode code points . more Specifications that permit truncation of a string to fit some length limit SHOULD require such truncation to occur on visual text unit boundaries (usually as approximated by grapheme cluster boundaries). more Implementations that perform truncation of a string to fit some length limit SHOULD truncate on visual text unit boundaries (usually as approximated by grapheme cluster boundaries). more If a specification cannot avoid doing otherwise, it MAY specify a length limit in terms of code units (such as bytes). more If a specification sets a length limit in code units (such as bytes), it MUST specify that truncation can only occur on code point boundaries. more Specifications that reference the [ DOM ] SHOULD specify that string operations be restricted to code point boundaries, and, where appropriate avoid starting or ending inside of a visual text unit or grapheme cluster . more Specifications that reference the [ DOM ] and permit arbitrary offsets or lengths to be used in text operations SHOULD include a health warning. more If a specification has to express a string length limit in code units but is free to specify the size of the string, choose a limit based on an acceptable number of code points times the relevant character encoding 's maximum encoded size. more When specifying a length limitation in code units (such as bytes), specifications SHOULD set the limit in a way that accommodates users whose language requires multibyte code unit sequences. more If a specification specifies a length limit in code units (such as bytes), it MUST specify the character encoding used in measuring the limit; such a limit SHOULD NOT specify a legacy character encoding . more Concatenation of strings Specifications SHOULD NOT require the concatenation of string values to form natural language or displayable string values. more When a specification requires an implementation to create or generate text which will be displayed to users, the specification SHOULD provide implementers with guidance on how to avoid potential problems related to text direction. more Working with file and path names Specify the UTF-8 [ Unicode ] encoding for the storage and processing of file names and file paths. more File names SHOULD be restricted to 255 bytes in length. more Path names SHOULD be restricted to 65535 bytes in length. more File name and path name definitions MUST NOT use the following Unicode code points. more Specifying sort and search functionality Specifications or implementations that require a program-internal, fast, and deterministic sorting of text which is not intended for human viewing or interaction SHOULD specify that strings are sorted according to their definition of string. For scalar value strings (such as USVString or many XML processes), specify ascending code point order. For string types based on UTF-16 (such as DOMString or in many JavaScript APIs), specify ascending code unit order. more When sorting text for presentation to users, the sort order SHOULD be tailored according to the most appropriate locale for the specific user in that application; thus the presentation order may differ from user to user. more Create markdown for GitHub 6.1 Choosing text units for segmentation, indexing, etc. See related review comments. See also 4.2 Choosing a definition of 'string' . 6.5 Truncating or limiting the length of strings . There are many situations where a software process needs to access a substring or to point within a string and does so by the use of indices, i.e. numeric "positions" within a string. Where such indices are exchanged between components of the Web, there is a need for an agreed-upon definition of string indexing in order to ensure consistent behavior. The two main questions that arise are: "What is the unit of counting?" and "Do we start counting at 0 or 1?". § The character string is RECOMMENDED as a basis for string indexing. more Character Model for the World Wide Web: Fundamentals, String indexing , C051 § Grapheme clusters MAY be used as a basis for string indexing in applications where user interaction is the primary concern. more Character Model for the World Wide Web: Fundamentals, String indexing , C071 Typographic character units in complex scripts Situations where grapheme clusters can be insufficient for segmenting complex scripts. Character encodings: Essential concepts, Characters & clusters § Specifications that define indexing in terms of grapheme clusters MUST either: (a) define grapheme clusters in terms of extended grapheme clusters as defined in Unicode Standard Annex #29, Unicode Text Segmentation (UTR #29), or (b) define specifically how tailoring is applied to the indexing operation. more Character Model for the World Wide Web: Fundamentals, String indexing , C071 Unicode Standard Annex #29, Unicode Text Segmentation, Grapheme Cluster Boundaries Typographic character units in complex scripts Situations where grapheme clusters can be insufficient for segmenting complex scripts. Character encodings: Essential concepts, Characters & clusters § The use of byte strings for indexing is NOT RECOMMENDED . more Character Model for the World Wide Web: Fundamentals > String indexing Character Model for the World Wide Web: Fundamentals, String indexing , C072 § A UTF-16 code unit string is NOT RECOMMENDED as a basis for string indexing, even if this results in a significant improvement in the efficiency of internal operations when compared to the use of character string. more Character Model for the World Wide Web: Fundamentals, String indexing , C052 A counter-example is the use of UTF-16 in DOM Level 1 . The use of UTF-16 code points is discouraged because it leaves open the possibility of an index occuring between two surrogate characters, which would cause significant problems (see 6.5 Truncating or limiting the length of strings ). § Specifications that need a way to identify substrings or point within a string SHOULD consider ways other than string indexing to perform this operation. more Character Model for the World Wide Web: Fundamentals, String indexing , C053 § Specifications SHOULD understand and process single characters as substrings, and treat indices as boundary positions between counting units, regardless of the choice of counting units. more Character Model for the World Wide Web: Fundamentals, String indexing , C053 § Specifications of APIs SHOULD NOT specify single characters or single 'units of encoding' as argument or return types. more Character Model for the World Wide Web: Fundamentals, String indexing , C056 § When the positions between the units are counted for string indexing, starting with an index of 0 for the position at the start of the string is the RECOMMENDED solution, with the last index then being equal to the number of counting units in the string. more Character Model for the World Wide Web: Fundamentals, String indexing , C057 6.2 Matching string identity for identifiers and syntactic content See related review comments. Useful background and overviews for this section Character Model for the World Wide Web: String Matching . String Matching of Syntactic Content in Document Formats and Protocols , in Character Model for the World Wide Web: Fundamentals . See also: 8.3 Defining keywords, identifiers, and namespaces in syntactic content (Guidance on defining identifiers and syntactic content) 6.3 Working with Unicode Normalization . 6.4 Case folding . § Define string identity matching for identifiers and syntactic content using the following steps: Ensure the strings to be compared consist of a sequence of Unicode code points. Expand all character escapes and includes. If defined, perform any case-folding and Unicode normalization steps (mulitple passes may be needed). Perform any additional matching tailoring specific to the specification. Compare the resulting sequences of code points for identity. explanations & examples The Matching Algorithm , in Character Model for the World Wide Web: String Matching § Do not specify normalization (ie. case folding or Unicode Normalization) of identifiers and syntactic content, unless you have a specific reason to do so. explanations & examples Performing the Appropriate Normalization Step , in Character Model for the World Wide Web: String Matching § 'ASCII case fold' SHOULD only be used for application internal identifiers that are restricted to ASCII. explanations & examples 8.3 Defining keywords, identifiers, and namespaces in syntactic content Performing the Appropriate Normalization Step , in Character Model for the World Wide Web: String Matching § Avoid specifying 'Unicode canonical case fold' unless you have a good reason. explanations & examples Performing the Appropriate Normalization Step , in Character Model for the World Wide Web: String Matching § Do not specify a 'Unicode compatibility case fold' . explanations & examples Performing the Appropriate Normalization Step , in Character Model for the World Wide Web: String Matching § Specifications of vocabularies MUST define the boundaries between syntactic content and character data as well as entity boundaries (if the language has any include mechanism). explanations & examples Additional Considerations for Normalization , in Character Model for the World Wide Web: String Matching 6.3 Working with Unicode Normalization See related review comments. Useful background and overviews for this section Character Model for the World Wide Web: String Matching Unicode Normalization . Identical-Appearing Characters and the Limitations of Normalization . Interaction of Normalization and Case Folding . Performing the Appropriate Normalization Step . Additional Considerations for Normalization . § Specifications SHOULD NOT specify a Unicode normalization form for encoding, storage, or interchange of a given vocabulary. explanations & examples Additional Considerations for Normalization , in Character Model for the World Wide Web: String Matching. § Implementations MUST NOT alter the normalization form of textual data being exchanged, read, parsed, or processed except when required to do so as a side-effect of text transformation such as transcoding the content to a Unicode character encoding, case folding, or other user-initiated change, as consumers or the content itself might depend on the de-normalized representation. explanations & examples Additional Considerations for Normalization , in Character Model for the World Wide Web: String Matching. § Specifications SHOULD NOT specify compatibility normalization forms (NFKC, NFKD). explanations & examples Additional Considerations for Normalization , in Character Model for the World Wide Web: String Matching. § Specifications MUST document or provide a health-warning if canonically equivalent but disjoint Unicode character sequences represent a security issue. explanations & examples Additional Considerations for Normalization , in Character Model for the World Wide Web: String Matching. § Where operations can produce denormalized output from normalized text input, specifications MUST define whether the resulting output is required to be normalized or not. Specifications MAY state that performing normalization is optional for some operations; in this case the default SHOULD be that normalization is performed, and an explicit option SHOULD be used to switch normalization off. explanations & examples Requirements When Specifying Normalization in Document Formats , in Character Model for the World Wide Web: String Matching. § Specifications that require normalization MUST NOT make the implementation of normalization optional. explanations & examples Requirements When Specifying Normalization in Document Formats , in Character Model for the World Wide Web: String Matching. § Normalization-sensitive operations MUST NOT be performed unless the implementation has first either confirmed through inspection that the text is in normalized form or it has re-normalized the text itself. Private agreements MAY be created within private systems which are not subject to these rules, but any externally observable results MUST be the same as if the rules had been obeyed. explanations & examples Requirements When Specifying Normalization in Document Formats , in Character Model for the World Wide Web: String Matching. § A normalizing text-processing component which modifies text and performs normalization-sensitive operations MUST behave as if normalization took place after each modification, so that any subsequent normalization-sensitive operations always behave as if they were dealing with normalized text. explanations & examples Requirements When Specifying Normalization in Document Formats , in Character Model for the World Wide Web: String Matching. 6.3.1 Specifying Unicode Normalization § Specifications that perform comparison or matching of string values SHOULD specify the appropriate note or warning regarding Unicode normalization. The use or adoption of Unicode Normalization in a specification is usually part of defining how matching takes place in a given format or protocol. To help specification authors and implementers understand some of the complexity involved, the Internationalization Working Group has developed a document describing the considerations for the matching and comparison of strings: Character Model for the World Wide Web: String Matching [ CHARMOD-NORM ]. One of the choices specifications need to make is whether (or not) to require Unicode Normalization as part of matching various "values" defined as part of the specification's vocabulary. Values are commonly part of a document format or protocol's syntax, and include such things as: attribute names or values, element names or values, IDs, and so forth. Specifications that follow the recommendation to not employ normalization as part of matching should include the following Note as a reminder to content authors. Example note. Necessarily this version is non-specific about what constitutes "values": specifications may wish to be more specific. Note This specification does not permit Unicode normalization of values for the purposes of comparison. Values that are visually and semantically identical but use different Unicode character sequences will not match. Content authors are advised to use the same encoding sequence consistently or to avoid potentially troublesome characters when choosing values. For more information, see [ CHARMOD-NORM ]. Specifications that choose to require require normalization as part of string matching should include the following warning: Example warning. Necessarily this version is non-specific about what constitutes "values": specifications may wish to be more specific. Warning This specification applies Unicode normalization during the matching of values. This can have an effect on the appearance and meaning of the affected text. For more information, see [ CHARMOD-NORM ]. Contact the I18N WG for alternatives or assistance if the above do not meet your needs or you're not sure about usage. 6.4 Case folding See related review comments. Useful background and overviews for this section Character Model for the World Wide Web: String Matching Case Mapping and Case Folding . Additional Considerations for Case Folding . § Specifications and implementations that define string matching as part of the definition of a format, protocol, or formal language (which might include operations such as parsing, matching, tokenizing, etc.) MUST define the criteria and matching forms used. These MUST be one of: (a) case-sensitive (b) Unicode case-insensitive using Unicode full case-folding (c) ASCII case-insensitive. § Case-sensitive matching is RECOMMENDED for matching syntactic content, including user-defined values. explanations & examples Case-sensitive matching , in Character Model for the World Wide Web: String Matching. § Specifications that define case-insensitive matching in vocabularies that include more than the Basic Latin (ASCII) range of Unicode MUST specify Unicode full casefold matching. explanations & examples Unicode case-insensitive matching , in Character Model for the World Wide Web: String Matching. § Specifications that define case-insensitive matching in vocabularies limited to the Basic Latin (ASCII) subset of Unicode MAY specify ASCII case-insensitive matching. explanations & examples ASCII case-insensitive matching , in Character Model for the World Wide Web: String Matching. § If language-sensitive case-sensitive matching is specified, Unicode case mappings SHOULD be tailored according to language and the source of the language used for each tailoring MUST be specified. explanations & examples Language-specific tailoring , in Character Model for the World Wide Web: String Matching. § Specifications that define case-insensitive matching in vocabularies SHOULD NOT specify language-sensitive case-insensitive matching. explanations & examples Language-specific tailoring , in Character Model for the World Wide Web: String Matching. 6.5 Truncating or limiting the length of strings See related review comments. Useful background and overviews for this section Units of collation , in Character Model for the World Wide Web: Fundamentals . Some specifications, formats, or protocols or their implementations need to specify limits for the size of a given string. This could be due to many reasons, such as limits on processing, memory, data structure size, and so forth. Because length limits often involve truncating or limiting the size of text, specifications and their implementations need to apply extra care to ensure that the text is not corrupted and that the limit chosen does not make the feature unusable for certain audiences. § Specifications SHOULD NOT impose a limit on the length of a string unless there is a specific practical or technical limitation. § If a specification specifies a length limit, it SHOULD specify that any string that has been truncated includes an indicator that the string has been altered, such as an ellipsis. There are many reasons why a length limit might be needed in a specification or format. The most common reason is because there is an underlying size limit on the data. For example, there might be a fixed-sized field in a database, a practical boundary such as packet size, or some other implementation detail related to storage allocation or efficiency. The other common reason is that there might be a limit on the size of the display or in the visible output. § Specifications that limit the length of a string MUST specify whether the limit is counted in Unicode code points , or is counted in code units (such as bytes) of a given character encoding . When truncating strings, it's necessary to decide what units to use when counting the size of the string. In some cases this is beyond the control of the specification, since the truncation is occuring for some preordained reason. However, when the choice is available, some general guidelines can be applied. § Avoid specifying the truncation of a stored string using some number of visual text units (such as grapheme clusters ). If a visual length limit is needed, specify visual truncation using text rendering or clipping mechanisms, such as CSS text-overflow [ css-overflow-4 ], which do not change the string and which account for the complexities of text rendering. Specifications sometimes try to deal with visual limitations by trying to use the number of visual text units as a proxy for the visible space available. Such limits can take many forms, such as limiting the number of characters on a line or attempting to make all visual text units have the same rendered width. Using visual text units corresponds more closely to such arbitrary limits than the number of code points or code units in the encoded string. However, the nature of text display usually makes such efforts ineffective. Proportional width fonts, complex scripts, styling, accessibility features, and many other factors complicate this. In nearly all cases, a visual text unit limit is really attempting to approximate pixel widths. Actually measuring the limit requires font metrics in the display context. It might also be affected by local settings, such as accessibility settings. In Web pages, the CSS text-overflow property provides visual truncation without disturbing the content of the text. Attempts to estimate the size of a given piece of text based on the number of Unicode code points or even the number of grapheme clusters is mostly futile. Example 15 : Grapheme clusters ≠ pixels When a size limit is based on the number of characters displayed, it is usually due to the need to "fit" the text into a specifically sized region of the screen or on a page. However, the nature of text makes the truncation and limitation of text dependent on font metrics at the time the text is rendered. Such limits frequently do not correspond to some arbitrary limit on the number of code points or grapheme clusters in the text. Consider the words indivisible and memorable . The first has 11 letters, the second only 9 letters. However, indivisible consumes fewer pixels in many proportional fonts (it has a number of letters ‘I’ and ‘L’, which are narrow, versus the letters ‘M’ in memorable ): Some scripts make the difference between visual text units and pixel size even more obvious. For example, here is the word for "Unicode" in Tamil: யூனிகோட் Tamil, like many South Asian scripts, is written in syllables. Each syllable is treated by Tamil speakers as a visual text unit . The word for "Unicode" in Tamil is a transliteration of the English word. If we divide the syllables up, as shown below, we can see that some are composed of multiple shapes which consume far more space than others: Syllable யூ னி கோ ட் IPA juː. ni. koː ɖ Finally, notice that the Tamil word quite often takes more room with four visual text units than an English word with nine: § Specifications that limit the maximum allowable storage length of a string SHOULD specify that length in terms of Unicode code points . Most limits are actually related to storage limitations, such as the size of a database field or a length limit in a protocol. Such limits are expressed either in terms of code points in Unicode, or they are expressed in terms of code units (such as bytes) in a specific character encoding . Code points provide the best user experience, since all Unicode code points are treated identically: if text is truncated after 40 code points, all languages and scripts get the same number of code points to work with. By contrast, when the size limit is expressed in code units , such as bytes in UTF-8, users who write in a language that mostly uses ASCII letters get many more characters (code points) for a given size limit than user's whose language is mostly made up of characters that take 2-, 3-, or 4-bytes per code point. § Specifications that permit truncation of a string to fit some length limit SHOULD require such truncation to occur on visual text unit boundaries (usually as approximated by grapheme cluster boundaries). § Implementations that perform truncation of a string to fit some length limit SHOULD truncate on visual text unit boundaries (usually as approximated by grapheme cluster boundaries). Truncation in the middle of a visual text unit , such as a combining character sequence , can alter the meaning of the remaining string. Truncation on visual text unit boundaries is not the same thing as using visual text units as the maximum length limit for a string. In addition to choosing how the length limit is expressed (code points vs. bytes), there is also the question of choosing the truncation boundary. Text must never be divided in the middle of a code point , because this results in corrupted characters. Text should never be divided in the middle of a grapheme cluster because this changes the appearance and meaning of the characters displayed. This might mean removing additional code points in order to ensure that the meaning is not affected. Example 16 : Interaction of truncation and grapheme cluster boundaries Returning to our Tamil example, here again is the word for "Unicode", broken into syllables (which, remember, are treated as grapheme clusters in the Tamil script): Syllable யூ னி கோ ட் IPA juː. ni. koː ɖ It turns out that each of these grapheme clusters consists of two Unicode code points . (Note that, while this particular word requires two code points in each of its visual text units , this is not always the case in Tamil. Some orthographic syllables in Tamil require only a single code point, such as when they use an inherent vowel .) Syllable Code Points யூ ய U+‎0BAF TAMIL LETTER YA ூ U+‎0BC2 TAMIL VOWEL SIGN UU னி ன U+0BA9 TAMIL LETTER NNNA ி U+0BBF TAMIL VOWEL SIGN I கோ க U+0B95 TAMIL LETTER KA ோ U+0BCB TAMIL VOWEL SIGN OO ட் ட U+0B9F TAMIL LETTER TTA ் U+0BCD TAMIL SIGN VIRAMA Suppose we arbitrarily truncate this text at the fifth code point. This divides one of the grapheme clusters (the syllable கோ ( /koː/ ). This changes the fundamental meaning of the text: யூனி கோ ட் U+0BAF U+0BC2 U+0BA9 U+0BBF U+0B95 | U+0BCB U+0B9F U+0BCD யூனி க ... § If a specification cannot avoid doing otherwise, it MAY specify a length limit in terms of code units (such as bytes). Sometimes the length limit for a string is determined by some external factor, such as the size of a database field or the number of bytes allocated for a data value specified elsewhere. Or the length limit might need to be specified in code units because of some practical design reason (such as describing a fixed-length byte-oriented wire protocol). Such specifications and their implementations will need to call out the additional complexity that this imposes, including the considerations below. § If a specification sets a length limit in code units (such as bytes), it MUST specify that truncation can only occur on code point boundaries. This best practice applies equally to UTF-16, which uses 16-bit code units, not just to multibyte encodings such as UTF-8. A UTF-16 surrogate pair , used to encode Unicode code points between U+10000 and U+10FFFF , requires two code units . Arbitrary truncation in the middle of a surrogate pair results in corruption of the encoded character. Example 17 : Interaction of truncation and code point boundaries Returning again to our example in Tamil, the code points in the Tamil script happen to each require three code units when encoded into the UTF-8 character encoding . (The code unit for UTF-8 is a byte.) யூ னி கோ ட் U+0BAF U+0BC2 U+0BA9 U+0BBF U+0B95 U+0BCB U+0B9F U+0BCD E0 AE AF E0 AF 82 E0 AE A9 E0 AE BF E0 AE 95 E0 AF 8B E0 AE 9F E0 AF 8D If we arbitrarily truncate this text after the 16th byte (where the yellow background begins above), we not only divide a grapheme cluster , but also one of the code points . The resulting damaged character is replaced in a UTF-8 decoder with the � U+FFFD REPLACEMENT CHARACTER . In legacy character encodings , the dangling bytes of the partial code point might combine with other, adjacent bytes to form undesirable combinations or garbling all of the subsequent text (an effect sometimes called mojibake ). யூனிக�... § Specifications that reference the [ DOM ] SHOULD specify that string operations be restricted to code point boundaries, and, where appropriate avoid starting or ending inside of a visual text unit or grapheme cluster . § Specifications that reference the [ DOM ] and permit arbitrary offsets or lengths to be used in text operations SHOULD include a health warning. Specifications or APIs that interact with the [ DOM ] need to contend with the fact that character data, including operations such as length , substringData , insertData , deleteData , and so forth, is specified using UTF-16 code units , rather than Unicode code points . This can lead to inappropriate mid-character (code point) truncation. Example 18 : Example DOM warning Modify this health warning as appropriate for your specification: Warning Arbitrary index values in the DOM may not fall on character or grapheme cluster boundaries. Implementations and users should avoid incorrectly starting or ending operations in the middle of a grapheme cluster . § If a specification has to express a string length limit in code units but is free to specify the size of the string, choose a limit based on an acceptable number of code points times the relevant character encoding 's maximum encoded size. The number of code points that can be stored in a fixed-size buffer depends on the character encoding form and its code unit . For example, UTF-8 encodes Unicode code points using between 1 and 4 bytes per character. It's maximum encoded size is therefore four code units. UTF-16, by contrast, uses either 1 or 2 16-bit code units. It's maximum encoded size is therefore two code units. If a given string should store at least 50 code points , the length limit in UTF-8 bytes would be 50 * 4 or 200 bytes. The length limit in UTF-16 would be 50 * 2 or 100 16-bit code units (which is the same as 200 bytes). Such a limit guarantees that the string can always store at least 50 code points, although, depending on the characters used, it might be able to store many more characters in certain languages. § When specifying a length limitation in code units (such as bytes), specifications SHOULD set the limit in a way that accommodates users whose language requires multibyte code unit sequences. When choosing a length limit, bear in mind the needs of different scripts and languages in Unicode. If a text size limit is set in terms of code units (such as a byte-length limit) in a given character encoding , the limit needs to consider the relative efficiency of the character encoding for different scripts. In particular, UTF-8 is the multibyte character encoding of [ Unicode ]. UTF-8 uses between one and four bytes per code point , depending on what the Unicode Scalar Value of the character is. When the limit placed on a text field is counted in bytes, the number of characters that can be stored in that field depends on which characters are being stored. To ensure that users who speak various languages are not at a disadvantage, the limit needs to allow for a reasonable number of characters as encoded in that language. Example 19 : Impact of code unit limits on text size Here is the famous sentence I can eat glass, it does not hurt me in several languages or scripts. Notice that the first three examples have about the same number of code points , while the emoji and cuneiform examples start with significantly fewer: Language Phrase Avg. Bytes/Code Point Length (code points) Length (UTF-8 bytes) English I can eat glass, it does not hurt me. 1 37 37 Macedonian Можам да јадам стакло, а не ме штета. 2 37 65 Marathi मी काच खाऊ शकतो, मला ते दुखत नाही. 3 34 84 Emoji 👁🥫🍽️🥛, it does ❌️🤕👤⬅️. 4 22 50 Cuneiform (transliteration) 𒀭𒊭𒀭𒈾𒉿𒄑𒊮𒊭𒌅𒍑𒋼𒋢𒁲𒋾𒈾𒌈𒊑𒉿𒁕𒈪𒌍 4 21 84 Suppose a specification wanted to choose a limit of 15 "characters" for the text field to store the given phrases. If the unit of truncation chosen were code units (in this case bytes in the UTF-8 character encoding), the limit would result in fewer actual code points for languages and scripts that require more bytes-per-character. The truncation boundary chosen might further disadvantage languages that require combing marks. Language Truncated Message Code Points Grapheme Clusters English I can eat glass 15 15 Macedonian Можам да 8 8 Marathi मी का 5 3 Emoji 👁🥫🍽️ 3 3 Cuneiform 𒀭𒊭𒀭 3 3 Keep in mind that, while the examples chosen here are roughly the same length, other languages might require more characters to convey the same concepts. For example, the Scottish Gaelic translation would be Is urrainn dhomh glainne ithe, chan eil e gam ghoirteachadh , which is significantly longer than the English. Many languages have different grammatical structure as well, so that key information (such as the verb) appearing at the end of the sentence (as is common in Hindi or Japanese). Finally, don't forget that the limit will also interact with the truncation boundary chosen (as shown in Example 17 ): if the truncation is done naively at the 15th byte, the resulting string might contain only a partial character. For example, the Marathi could experience this problem: मी का�... Example 20 : Emoji sequences as an example of grapheme clusters Another example of the complex relationship between visual text units and code points are certain emoji. The emoji character for "family" has a code point in Unicode: 👪 U+1F46A FAMILY . It can also be formed by using using a sequence of code points : U+1F468 U+200D U+1F469 U+200D U+1F466 . The character U+200D ZERO WIDTH JOINER is used to "join" separate emoji characters together (it also has a role in joining characters in various writing systems of the world). This compositional mechanism can be used to create other family variations. For example, the sequence 👨‍👩‍👧‍👦 U+1F468 U+200D U+1F469 U+200D U+1F467 U+200D U+1F466 results in a composed emoji character for a "family: man, woman, girl, boy" on systems that support this kind of composition. That long character sequence still represents just a single visual text unit . Other characters, such as skin tone modifiers, can further extend the grapheme cluster . Here are just a few of the possible emoji sequences possible for representing a family: Emoji Code Point Sequence U+1F46A FAMILY U+1F468 U+200D U+1F469 U+200D U+1F466 U+1F468 U+200D U+1F469 U+200D U+1F467 U+200D U+1F466 U+1F468 U+1F3FF U+200D U+1F469 U+1F3FE U+200D U+1F467 U+1F3FC U+200D U+1F466 Many common emoji can only be formed using sequences of code points, but should be treated as a single visual text unit when displaying or processing the text. The simplest composed family emoji sequence consists of 5 code points: U+1F468 U+200D U+1F469 U+200D U+1F466 A limit of 15 bytes (UTF-8 code units ) would truncate this sequence in the middle of the child family member: 👨‍👩‍�‍ If the truncation were done on the grapheme cluster boundary instead, the entire family would be removed. § If a specification specifies a length limit in code units (such as bytes), it MUST specify the character encoding used in measuring the limit; such a limit SHOULD NOT specify a legacy character encoding . If a specification permits or requires truncation of a string, with the length expressed in code units , the character encoding becomes important in knowing what the limit means. If the limit is in bytes and legacy character encodings are permitted, note that conversion of Unicode data to a non-Unicode encoding can result in data loss (since most legacy character encodings encode only a subset of Unicode). 6.6 Concatenation of strings See related review comments. § Specifications SHOULD NOT require the concatenation of string values to form natural language or displayable string values. Creation of natural language text values by concatenating multiple strings together is an internationalization anti-pattern. Languages vary greatly in word order, count, grammatical gender or case, punctuation, and many other requirements. As a result, avoid requiring or suggesting that implementations generate human-readable messages from sub-strings. See also Working with Composite Messages § When a specification requires an implementation to create or generate text which will be displayed to users, the specification SHOULD provide implementers with guidance on how to avoid potential problems related to text direction. Specifications for APIs, protocols, or document formats sometimes require an implementation to create or provide a field containing a display name or description. When such a string is assembled from separate parts, it can result in problems with presentation or understanding due to the way that the Unicode Bidirectional Algorithm [ UAX9 ] processes the assembled string. In such cases, the specification should guide implementers about how to create values that will display properly. Example 21 : Example of bidi generation guidance Here is an example of a Note that you can include in your specification when recommending that implementations generate displayable natural language values: Note When generating a value for _field_name_ , producers need to ensure that directional changes inside the string will display correctly. When a producer assembles the value from multiple strings, individual substrings might need to be bidi isolated in order to prevent spillover effects. This can be done using markup (where available) or via the insertion of Unicode bidirectional controls. In addition, if the producer can supply the correct base direction as metadata, it allows the consumer to both isolate the string and set its direction, avoiding spillover effects between the full string and the context in which it is eventually displayed. For more information see: How to use Unicode controls for bidi text and Inline changes to base direction [ INTERNATIONAL-SPECS ]. See also 3.2.3 Inline changes to base direction 3.6 Setting base direction for inline or substring text 6.7 Working with file and path names Some specifications need to define how file names or file paths are constructed by various implementations. One challenge is building definitions that work consistently when used on the different file systems used by different operating systems. This section contains general guidance when defining restrictions on file names or file paths. It is based on requirements developed in [ EPUB-33 ], as well as implementation experience. § Specify the UTF-8 [ Unicode ] encoding for the storage and processing of file names and file paths. § File names SHOULD be restricted to 255 bytes in length. This restriction is related to limitations found in certain file systems, originally MS-DOS, but also certain Unix file systems—as well as packaging schemes such as PKZIP that depend on these file systems or subsumed their limitations—in which the limit for a specific "path element" (including directory names) is limited to 255 bytes. § Path names SHOULD be restricted to 65535 bytes in length. This restriction is related to limitations found in file systems such as FAT32 or NTFS, which restrict the path length to 32760 (32K) code units in the UTF-16 character encoding. Each UTF-16 code unit takes 16 bits (or 2 bytes), making the limit 65,535 when measured in bytes. Note that a path name limited to 64K bytes in UTF-8 can exceed the path length limits on these file systems, since UTF-8 is a variable width encoding. § File name and path name definitions MUST NOT use the following Unicode code points. These characters are known to cause interoperability problems with various file systems. Specifications and implementations should use an abundance of caution in their file naming when interoperability of content is key. The list of restricted characters is intended to help avoid some known problem areas, but it does not ensure that all other Unicode characters are supported. " U+0022 QUOTATION MARK * U+002A ASTERISK / U+002F SOLIDUS : U+003A COLON < U+003C LESS-THAN SIGN > U+003E GREATER-THAN SIGN \ U+005C REVERSE SOLIDUS | U+007C VERTICAL LINE U+007F DEL Codepoints in the following ranges: C0 Controls U+0000 ... U+001F C1 Controls U+0080 ... U+009F Private Use U+E000 ... U+F8FF Specials U+FFF0 ... U+FFFF Supplementary Private Use U+F0000 ... U+FFFFF Supplementary Private Use U+100000 ... U+10FFFF . U+002E FULL STOP as the last character (Note that this includes the file names . and .. , which have special meaning to many file systems) All Unicode non-character code points, specifically: The 32 contiguous characters in the Basic Multilingual Plane (U+FDD0 … U+FDEF) The last two code points of the Basic Multilingual Plane (U+FFFE and U+FFFF) The last two code points at the end of the Supplementary Planes (U+1FFFE, U+1FFFF … U+EFFFE, U+EFFFF) All Unicode deprecated characters (search for "Deprecated" in the file). 6.8 Specifying sort and search functionality See related review comments. Useful background and overviews for this section Unicode's ICU Collation User Guide Unicode Collation Algorithm [ UTS10 ] Units of collation , in Character Model for the World Wide Web: Fundamentals . Applications often need to organize sets of information or content. Frequently this involves sorting the content. Many non-textual data types, such as numbers or dates, can be easily sorted using the internal data representation. When it comes to textual information, however, the nature of character encodings and user expectations regarding "alphabetical" order brings some additional complexity. One key choice is whether the sorting of textual data will be strictly internal or whether the results will be shown to users. 6.8.1 Program Internal Sorting § Specifications or implementations that require a program-internal, fast, and deterministic sorting of text which is not intended for human viewing or interaction SHOULD specify that strings are sorted according to their definition of string. For scalar value strings (such as USVString or many XML processes), specify ascending code point order. For string types based on UTF-16 (such as DOMString or in many JavaScript APIs), specify ascending code unit order. explanations & examples See: Defining 'string' Unicode Codepoint Collation is defined as an XPath Function [ xpath-functions ] There are two potential internal sorting sequences: ordering by Unicode code point or ordering by UTF-16 code unit . For either type of ordering, the resulting list will not match any particular alphabetic or lexicographical order. Sorting by code point makes sense when strings are stored and processed as a sequence of code points, such as in a USVString . Sorting by code unit makes sense when strings are stored and processed using the underlying encoding, such as in a DOMString . Example 22 : Code point vs. code unit ordering Consider two strings, one containing 😺 U+1F63A SMILING CAT FACE WITH OPEN MOUTH and the other containing ～ U+FF5E FULL WIDTH TILDE . In ascending code point order , the strings sort like: ～ (U+FF5E) 😺 (U+1F63A) In ascending code unit order in the UTF-16 character encoding, the code point U+1F63A is encoded as the code unit sequence 0xD83D 0xDE3A (a surrogate pair ), so the strings sort like: 😺 (0xD83D 0xDE3A) ～ (0xFF5E) Note that UTF-8 code unit order (that is, when sorting by byte values in UTF-8 encoded byte strings) is the same as code point order. Neither of these sort orders applies any type of normalization to the strings being compared. This means that some apparently equivalent strings compare as different. See String Matching [ CHARMOD-NORM ] for more information. Example 23 Consider two strings, A and B . String A consists of Ç U+00C7 LATIN CAPITAL LETTER C WITH CEDILLA . String B consists of the sequence C U+0043 LATIN CAPITAL LETTER C followed by ̧ U+0327 COMBINING CEDILLA . These strings are visually identical. Applying Unicode Normalization Form C or Form D to both would cause them to use the same code point (and thus code unit) sequences. However, they compare differently because 0x00C7 is greater than 0x0043 : String A String B Ç > Ç U+00C7 U+0043 U+0327 6.8.2 Human-visible Sorting Specifications or applications that need to deal with sorting natural language text for display to users face some additional complexity. Unicode defines a default collation (sorting) order as part of the Unicode Collation Algorithm [ UTS10 ], which is then tailored to meet the needs of specific languages, locales , and cultures. § When sorting text for presentation to users, the sort order SHOULD be tailored according to the most appropriate locale for the specific user in that application; thus the presentation order may differ from user to user. explanations & examples Unicode Collation Algorithm [ UTS10 ] Collation section of Locale Data Markup Language [ UTS35 ] Units of collation, C007 , in Character Model for the World Wide Web: Fundamentals Languages and cultures vary in how they sort text or use their alphabet or writing system to organize textual data. For example, German language speakers treat the letter ü U+00FC LATIN SMALL LETTER U WITH DIAERISIS as sorting similar to the letter u (there are actually two German sorting sequences, which are slightly different in the exact handling of this letter), while Danish language speakers treat the same letter as separate in the alphabet and sort it after the letter "y". Determining which locale to use for a sorted list can depend on a number of factors. For example, an application might sort a list of values according to the localization of the page in which the data appears. In other cases it might make more sense to sort according to the runtime locale of the user-agent or according to some parameter passed in an API. The important thing to recognize is that this order might be different for different users or on different systems. 6.8.3 Searching Warning : Under construction Searching text is a complex topic and this document does not currently contain a complete list of recommendations and best practices. The Internationalization Working Group has compiled a list of just a few of the issues involved as part of on-going work to document this space in our document String Searching . 7. Resource identifiers Show the self-review checklist This is a list of just the requirements in this section, which can be used for self-reviews. Select the first checkbox on a line for all requirements relevant to your spec. Select the second checkbox if your spec meets the requirement. Then click on the button "Create markdown for GitHub", and copy the result to a GitHub issue list. See more details . Create markdown for GitHub See related review comments. Useful background and overviews for this section Uniform Resource Identifiers (URIs) [ RFC3986 ] WhatWG URL Specification [ URL ] Internationalized Resource Identifiers (IRIs) [ RFC3987 ] The situation with regards to specifying support of non-ASCII characters in resource identifiers is complicated because there are at least three specifications (URI [ RFC3986 ], IRI [ RFC3987 ], and [ URL ]) that define resource identifiers and their serialization. The WhatWG [ URL ] specification is an attempt to address this complexity by documenting the actual practice of browsers and other user agents. The stated goal of the URL specification is to obsolete both RFCs. In general, document formats on the Web use resource identifiers that encode non-ASCII characters as plain text, that is, as "IRIs". Protocols such as—but not limited to—HTTP [ RFC9110 ]) use resource identifiers that encode non-ASCII characters as a sequence of bytes using percent encoding , that is, as "URIs". Because [ RFC3986 ] does not specify any particular character encoding for encoding characters to bytes, the percent encoding escapes are prone to misinterpretation. To help combat this, many modern protocols and specifications expect resource identifiers to use the UTF-8 character encoding, exactly as specified by IRI, when encoding characters into the subset of ASCII supported in wire formats and protocols. Example 24 : Comparison of IRI to URI Here's an example of an IRI and its URI-encoded equivalent, where the path of the identifier is the Japanese word 文字化け (" mojibake "). This word consists of a sequence of four Unicode code points: U+6587 U+5B57 U+5316 U+3051 . In UTF-8, each of these code points requires three-bytes to encode, so the byte sequences in hex are: 0xE6.96.87 0xE5.AD.97 0xE5.8C.96 0xE3.81.91 Format Example IRI https://www.example.com/文字化け URI https://www.example.com/%E6%96%87%E5%AD%97%E5%8C%96%E3%81%91 § Specifications that define resource identifiers MUST permit the use of non-ASCII characters. more Model is defined in terms of IRIs; Protocol with URI . GitHub issue discussion. Document formats or protocols need to support resource identifiers that contain non-ASCII characters because in many cases the names or identifiers for a given resource are generated from user input. Users generally are not restricted and should not be restricted in their ability to use their own language for these values. § Specifications on the Web that define a document format, data structure, or API SHOULD reference [ URL ] when specifying resource identifiers . For cases unsupported by the [ URL ] specification, IRI [ RFC3987 ] MAY be specified instead. § Specifications that define protocols MAY reference URI [ RFC3986 ] when specifying resource identifiers for use in wire formats but MUST include the additional requirement that UTF-8 MUST be used for the interpretation of percent encoded values into characters. According to the definition in [ RFC3986 ], URI references are restricted to a subset of ASCII and non-ASCII characters cannot be used directly. The percent encoding is provided to escape arbitrary byte values. However, percent encoding by itself is of limited value because many different legacy character encodings might be used to interpret a given sequence of bytes into characters (or to encode a given sequence of characters into bytes). Internationalized Resource Identifiers ( IRIs ) [ RFC3987 ] solves problems with encoding and interpreting non-ASCII characters in resource identifiers with a uniform approached based on the UTF-8 encoding of [ Unicode ]. § A specification MAY impose its own limitations on which characters are permitted in a resource identifier , but these should be focused on characters that conflict with the syntax of resource identifiers, the transport format, or with other elements defined by the specification itself. While generally not recommended, if additional restrictions are contemplated, review [ UAX31 ] and [ CHARMOD-NORM ] for additional guidance. § Specifications that define new syntax for URIs or contained within URIs MUST specify that characters outside the ASCII repertoire are percent encoded using the UTF-8 character encoding . 8. Document formats, markup & syntax Show the self-review checklist This is a list of just the requirements in this section, which can be used for self-reviews. Select the first checkbox on a line for all requirements relevant to your spec. Select the second checkbox if your spec meets the requirement. Then click on the button "Create markdown for GitHub", and copy the result to a GitHub issue list. See more details . Defining elements and attributes in markup Do not define attribute values that will contain user readable content. Use elements for such content. more If you do define attribute values containing user readable content, provide a means to indicate directional and language information for that text separately from the text contained in the element. more Provide a way for authors to annotate arbitrary inline content using a span -like element or construct. more Handling plain text in markup Avoid natural language text in elements or attribute values that only allow for plain text. more Avoid defining attribute values whose content will be natural language text. more Provide a span-like element that can be used for any text content to apply information needed for internationalization. more Defining keywords, identifiers, and namespaces in syntactic content Permit the full range of Unicode except where otherwise documented. more Restrictions on content, especially those on the range of code points permitted, MUST be documented except where the restrictions are obvious from the context . more Limit vocabulary , including application internal identifiers , to a printable subset of ASCII . more For syntactic content not restricted to ASCII, specifications SHOULD define the characters that are allowed to start and be part of a valid value. more Specifications SHOULD NOT allow surrogate code points ( U+D800 to U+DFFF ) or non-character code points in identifiers. more Specifications SHOULD NOT allow the C0 ( U+0000 to U+001F ) and C1 ( U+0080 to U+009F ) control characters in identifiers. more Application internal identifier fields or values MUST be wrapped with a localizable display value when displayed to end-users. more Choose locale-neutral and culturally-neutral names for fields and values. more Use ASCII case-insensitive [ INFRA ] to specify case insensitive matching of values that are limited to a printable subset of ASCII . more Use identical to or is from [ INFRA ] to define matching of values that are case sensitive , including all values that are not limited to a printable subset of ASCII . more Specifications that define non-localizable string data values intended to be machine-readable should use values that are not readily confused with natural language text. more Fields whose content is intended for consumption by humans must always be treated as natural language string values. It must be possible to find the language and base direction metadata for every such field. more Fields intended for consumption by humans should be localizable. more Field names and other enumerated values should be wrapped with localizable display names. more Create markdown for GitHub Specifications that deal with formal languages, document formats, protocols, or APIs often need to define markup, syntax, or application internal identifiers . The best practices in this section cover the different needs when defining these. Specifications that are defining a markup language or a syntax based on a given markup language are concerned with defining elements, attributes, and their values. For example, an [ XML ] DTD defines elements and attributes that are valid in a specific document type. Specifications that are defining a given document format, protocol, or API are usually concerned with defining identifiers for reserved keywords, field names, or permitted values. Many of these are application internal identifiers , whose names and values are completely defined by the specification. In some cases the specification will permit some or all of these to be a user-supplied value which can be filled in or named by users. Example 25 Let's use CSS to illustrate some of these concepts. Here is a fragment of a style sheet: p .myClassName { text-align : center; color : red; } The p.myClassName rule contains two properties ( text-align and color ). Each of these is an application internal identifier . Each property has been assigned a value from a list of available values defined by CSS. For example, here the property text-align has been given the value center . Other possible values for this property are part of a list of application internal identifiers defined by [ CSS3-TEXT ] and include start , end , left , right , justify , match-parent , or justify-all . The class name myClassName is an example of a user-supplied value . A user might assign a class name using any valid class identifier in CSS, including using words in other languages, such as using an identifier in Japanese: p .私のクラス名 { color : #434322 ; } 8.1 Defining elements and attributes in markup See related review comments. § Do not define attribute values that will contain user readable content. Use elements for such content. explanations & examples Best Practice 3: Avoiding translatable attribute values , in Best Practices for XML Internationalization § If you do define attribute values containing user readable content, provide a means to indicate directional and language information for that text separately from the text contained in the element. § Provide a way for authors to annotate arbitrary inline content using a span -like element or construct. explanations & examples Best Practice 14: Defining a span-like element , in Best Practices for XML Internationalization 8.2 Handling plain text in markup See related review comments. § Avoid natural language text in elements or attribute values that only allow for plain text. § Avoid defining attribute values whose content will be natural language text. § Provide a span-like element that can be used for any text content to apply information needed for internationalization. Internationalization information may include language and base direction metadata, inline changes of language, bidirectional text behavioural changes, translate flags, etc. 8.3 Defining keywords, identifiers, and namespaces in syntactic content See related review comments. Section 2. The String Matching Problem of Character Model for the World Wide Web: String Matching [ CHARMOD-NORM ] says: The Web is primarily made up of document formats and protocols based on character data. These formats or protocols can be viewed as a set of resources consisting mainly of text files that include some form of structural markup or syntactic content . Processing such syntactic content or document data requires string-based operations such as matching (including regular expressions), indexing, searching, sorting, and so forth. Users, particularly implementers, sometimes have naïve expectations regarding the matching or non-matching of similar strings or of the efficacy of different transformations they might apply to text, particularly to syntactic content, but including many types of text processing on the Web. Because the Web is sensitive to the different ways in which text might be represented in a document, failing to consider the different ways in which the same text can be represented can confuse users or cause unexpected or frustrating results. Note : Terminology in this section The content below uses the specialized terminology from [ CHARMOD-NORM ], as well as some formatting not found elsewhere in this document. A brief summary of these terms is given just below, but it can be useful to review the illustration of terminology before proceeding. Syntactic content refers to the overall text content of such a defined format. The various tokens (such as identifiers) in a formal grammar are referred to as the vocabulary . Usually the "reserved keywords" in a vocabulary are considered application internal identifiers : values that are program internal and not for display to the end-user, even though the values chosen convey some semantic meaning (usually in English). Specifications might also allow content authors to set values, such as (but not restricted to) variable names or for IDs of various kinds. To foster interoperability, implementations need to be able to match the different parts of the syntactic content they define reliably and consistently. § Permit the full range of Unicode except where otherwise documented. explanations & examples Specifying Content Restrictions in [ CHARMOD-NORM ] § Restrictions on content, especially those on the range of code points permitted, MUST be documented except where the restrictions are obvious from the context . Start with a presumption that any document format or protocol consists of Unicode strings and that it must be able to transmit Unicode data. Characters whose properties seem to "complicate" processing are a necessary element of many languages. When restrictions are imposed on the range of valid characters, it can inadvertently impair usability for international users, particularly those whose native language does not use the Latin script. Avoid impairing or prohibiting the ability to transmit, process, and interchange international data except when it is absolutely necessary to do so. There are many reasons for a specification to limit content: what is important is that the specification's authors have thought about the impact of the decision to impose such a limit on content and have documented the restriction for implementers. A content restriction is obvious from the context when the restriction is documented by some parent specification. For example, if one is defining an HTTP header, the restriction of the header's name to a specific subset of ASCII is implied (note that this does not apply to the value of an HTTP header). § Limit vocabulary , including application internal identifiers , to a printable subset of ASCII . explanations & examples Specifying Content Restrictions in [ CHARMOD-NORM ] A common strategy used by specification authors is to restrict their vocabulary or subsets of it, such as application internal identifiers or locally defined namespaces to some carefully chosen set of ASCII characters. When values are not meant to transmit arbitrary data, restricting identifiers, reserved keywords, and values to a ASCII avoids the technical challenges inherent in inputting, displaying, and (most especially) comparing Unicode strings. A printable subset of ASCII is any subset of the Basic Latin code points of Unicode that excludes the C0 control characters (in the range U+0000 through U+001F inclusive) and U+007F DEL . The specific subset will depend on the specification or application's local requirements. Most subsets include all of the ASCII alphanumeric characters. The range of punctuation allowed and/or the inclusion of the U+0020 SPACE character depend on local requirements (such as the use of such characters for syntax, escapes, or separators). Identifiers or data values that can be chosen by document authors (or derived from data) often need to support other languages. Restricting the set of allowable characters to ASCII can impede international usability, particularly for speakers of languages that do not use the Latin script or for applications that use characters outside of the ASCII range. This is especially the case when the values, such as identifiers, are generated from user data or when they might be externally referenced in ways that have strong mnemonic value. For example, CSS allows class names to use a wide range of Unicode code points , even though its vocabulary of reserved keywords is restricted to a printable subset of ASCII. See also 8.3.2 Defining application-internal data values for guidelines on choosing names Example 26 : Use of ASCII keywords to help authors Specifications often use human-readable names for values to aid with operations such as debugging, validation, and so forth. For example, the CSS fragment color: orange uses recognizable English words to set the value of color to an orange color. It is easier for end-users to figure out the effect versus using a value such as color: #FFA500 (which means the same thing), even when the user does not speak English. § For syntactic content not restricted to ASCII, specifications SHOULD define the characters that are allowed to start and be part of a valid value. explanations & examples Unicode Identifier and Pattern Syntax [ UAX31 ] Example : ECMAScript 5, section 7.6 Identifier Names and Identifiers Handling of combining marks and certain other characters (such as joiners or bidi formating characters) can present issues when parsing a given document format: special focus needs to be paid to how each identifier can be "tokenized" (separated from the surrounding text). One means of doing this is to restrict the range of characters allowed to start an identifier to ensure that normal text processing doesn't interfere with matching the identifier later. Unicode Identifier and Pattern Syntax [ UAX31 ] provides one model, used notably in programming languages such as Java or JavaScript . HTML and CSS also provide character range definitions for custom identifiers, such as this EBNF [ XML ] production: PCENChar ::= "-" | "." | [ 0 - 9 ] | "_" | [a-zA-Z] | #xB7 | [#xC0-#xD6] | [#xD8-#xF6] | [#xF8-#x37D] | [#x37F-#x1FFF] | [#x200C-#x200D] | [#x203F-#x2040] | [#x2070-#x218F] | [#x2C00-#x2FEF] | [#x3001-#xD7FF] | [#xF900-#xFDCF] | [#xFDF0-#xFFFD] | [#x10000-#xEFFFF] Note HTML and CSS processing is defined such that Unicode character properties (such as whether a given character is a combining mark) are not considered when parsing identifiers and tokens. This allows identifiers to start with a combining character and still be processed reliably, but a plain text editor might not handle the value identically. Exercise care when defining identifiers with regards to the handling of whitespace. Note that there are Unicode horizontal whitespace characters other than the ASCII characters U+0020 SPACE and U+0009 TAB . § Specifications SHOULD NOT allow surrogate code points ( U+D800 to U+DFFF ) or non-character code points in identifiers. explanations & examples Specifying Content Restrictions in [ CHARMOD-NORM ] § Specifications SHOULD NOT allow the C0 ( U+0000 to U+001F ) and C1 ( U+0080 to U+009F ) control characters in identifiers. explanations & examples Specifying Content Restrictions in [ CHARMOD-NORM ] § Application internal identifier fields or values MUST be wrapped with a localizable display value when displayed to end-users. explanations & examples Specifying Content Restrictions in [ CHARMOD-NORM ] § Choose locale-neutral and culturally-neutral names for fields and values. When defining identifiers, including field names and values, choose names that are as culturally-neutral as possible. For example, prefer postalCode to the (USA-specific) ZIPCode or prefer givenName/familyName to the more-culturally linked firstName/lastName . 8.3.1 Case sensitivity versus insensitivity Specifications have to decide whether a format is case sensitive or case insensitive . For example, does the value green match the values GREEN or GrEeN . In this section, the terms case- sensitive and case- insensitive appear with additional emphasis, as shown here, because the terms sensitive and insensitive are easily confused. § Use ASCII case-insensitive [ INFRA ] to specify case insensitive matching of values that are limited to a printable subset of ASCII . § Use identical to or is from [ INFRA ] to define matching of values that are case sensitive , including all values that are not limited to a printable subset of ASCII . explanations & examples Specifying Content Restrictions in [ CHARMOD-NORM ] Make identifiers case- sensitive whenever non-ASCII characters are allowed. Only make comparisons case insensitive when content is restricted to a printable subset of ASCII . Comparing non-ASCII strings in a case- insensitive manner involves addressing the very many issues documented in Character Model for the World Wide Web: String Matching [ CHARMOD-NORM ], including normalization and language sensitivity. 8.3.2 Defining application-internal data values Some specifications need to define the values for a given field in a document format or protocol. When the data values are associated with a specific type, such as numbers or dates, the format of the field is usually defined using some well-known schema, such as [ XMLSCHEMA11-2 ] or [ JSON-SCHEMA ]. § Specifications that define non-localizable string data values intended to be machine-readable should use values that are not readily confused with natural language text. Many protocols, document formats, or data structures define enumerated values for internal use. These values are not meant to be visible to humans directly. Sometimes it is helpful if these values are given descriptive names (often in English) to aid users working with the specification, protocol, or API or who might need to debug a given document or interaction. When assigning these values in a specification, the names chosen should appear to be "code-like" so that users do not assume that the value can be displayed as if it were natural language text. There are several styles that different groups have adopted to make application-internal values look "code-like". Choose the one best suited to your specification. These include: SNAKE_CASE . Snake case uses ASCII letters and digits, all in uppercase, with words separated by underscores ( _ U+005F LOW LINE ). PascalCase or camelCase . These use ASCII letters and digits, with each "word" inside the identifer being capitalized. Example 27 : Example of code internal values For example, a specification might define a document with a set of fields named in camelCase whose predefined values are named in SNAKE_CASE: type : [ PURCHASE_REQUEST, INVOICE_REQUEST, RECEIPT_REQUEST ] responseType : [ EMAIL, SMS_MESSAGE, JSON_DOCUMENT ] transactionResult : [ SUCCESS, NOT_AUTHORIZED, INTERNAL_ERROR, UNKNOWN_ERROR ] Note Enumerated values defined by a specification should not be considered as "natural language strings". While they generally use English language words to form the identifiers, they are not intended for end-users to view nor are they free-form text values. Developers are expected to read the definition backing the keywords in enumerated values (the description of which can be localized). In fact, it is a best practice to define values in a locale-neutral way and wrap that with display strings. § Fields whose content is intended for consumption by humans must always be treated as natural language string values. It must be possible to find the language and base direction metadata for every such field. Fields that contain human-readable strings, particularly those of a descriptive nature, must be assumed to be natural language strings. This is true even if the user viewing the string is expected to be a software developer. It must be possible to determine the language tag and string direction for each such field in a document or data structure. Common names for fields of this type include name , description , title , message , or occassionally value . One test for this is if, as a specification author or user, you are uncomfortable making the content of the field SNAKE_CASE_SHOUTED , the field might be better considered as natural language text. § Fields intended for consumption by humans should be localizable. This can take various forms. For example, a specification or protocol might allow for language negotiation and only return the best matching localized strings. Or a given resource might contain multiple languages that the consumer can choose between. § Field names and other enumerated values should be wrapped with localizable display names. Field names and enumerated values are not natural language text, even if the names appear to be plain text and might be understood by users. These fields and values should not have language or direction metadata associated with them and, where necessary, implementers should be guided by the specification to provide appropriate localized wrapping. 9. Typographic support Show the self-review checklist This is a list of just the requirements in this section, which can be used for self-reviews. Select the first checkbox on a line for all requirements relevant to your spec. Select the second checkbox if your spec meets the requirement. Then click on the button "Create markdown for GitHub", and copy the result to a GitHub issue list. See more details . Text decoration Text decoration such as underline and overline should allow lines to skip ink. more It should be possible to specify the distance of overlines and underlines from the text. more Vertical text It should be possible to render text vertically for languages such as Japanese, Chinese, Korean, Mongolian, etc. more Vertical text must support line progression from LTR (eg. Mongolian) and RTL (eg. Japanese). more By default, text decoration, ruby, and the like in vertical text where lines are stacked from left to right (eg. Mongolian) should appear on the same side as for CJK vertical text. Placement should not rely on the before and after line locations. more Vertical writing modes that are equivalent to the vertical- values in CSS (only) should use [ UTR50 ] to apply default text orientation of characters. (This does not apply to writing modes that are equivalent to sideways- in CSS.) more Writing modes should provide values like sideways-lr and sideways-rl in CSS to allow for vertical rotation of lines of horizontal script text. UTR50 is not applicable for these cases. more By default, glyphs of scripts that are normally horizontal should run along a line in vertical text such that the top of the character is toward the right side of the vertical line, but there should also be a mechanism to allow them to progress down the line in upright orientation. Such a mechanism should use visual text units (such as grapheme clusters ) as a minimum text unit, but where necessary allow syllabic clusters to be treated as a unit when they involve more than one grapheme cluster. more Upright Arabic text in vertical lines should use isolated letter forms and the order of text should read top to bottom. more It should be possible for some sequences of characters (particularly digits) to run horizontally within vertical lines (tate chu yoko). more RTL/bidi text Specifications that enable sloping of letterforms SHOULD provide for characters to slope either to the right or to the left according to the needs of the specific language. more Setting box positioning coordinates when text direction varies Box positioning coordinates must take into account whether the text is horizontal or vertical. more Logical properties (TBD) Cursive text Overlaps should not be exposed when transparency is applied to the joined letters in cursive text, such as for Arabic, Mongolian, and N'Ko. more When adding a text stroke or shadow, joined letters should not be separated from their neighbors in cursive script text. more Ruby text annotations 'Ruby' style annotations alongside base text should be supported for Chinese, Japanese, Korean and Mongolian text, in both horizontal and vertical writing modes. more Ruby implementations should support zhuyin fuhao (bopomofo) ruby for Traditional Chinese. more Ruby implementations should support a tabular content model (such that ruby contents can be arranged in a sequence approximating to rb rb rt rt ). more Ruby implementations should make it possible to use an explicit element for ruby bases, like the rb element in HTML. more Ruby implementations should allow annotations to appear on either or both sides of the base text. more Ruby markup in HTML is designed specifically for Chinese, Japanese, Korean, and Mongolian requirements, and should not be used as a general glossing mechanism. more Font management (TBD) Miscellaneous Line heights must allow for characters that are taller than English. more Box sizes must allow for text expansion in translation. more Line wrapping should take into account the special rules needed for non-Latin scripts. more Avoid specifying presentational tags, such as b for bold, and i for italic. more Create markdown for GitHub 9.1 Text decoration See related review comments. Useful background and overviews for this section Text decoration , in the Language enablement index. § Text decoration such as underline and overline should allow lines to skip ink. § It should be possible to specify the distance of overlines and underlines from the text. Skipping ink for text decoration such as underlines may not be appropriate for some scripts, such as Arabic, which prefers to move the underline further away from the baseline instead. 9.2 Vertical text See related review comments. Useful background and overviews for this section Vertical text , in the Language enablement index. § It should be possible to render text vertically for languages such as Japanese, Chinese, Korean, Mongolian, etc. § Vertical text must support line progression from LTR (eg. Mongolian) and RTL (eg. Japanese). § By default, text decoration, ruby, and the like in vertical text where lines are stacked from left to right (eg. Mongolian) should appear on the same side as for CJK vertical text. Placement should not rely on the before and after line locations. § Vertical writing modes that are equivalent to the vertical- values in CSS (only) should use [ UTR50 ] to apply default text orientation of characters. (This does not apply to writing modes that are equivalent to sideways- in CSS.) § Writing modes should provide values like sideways-lr and sideways-rl in CSS to allow for vertical rotation of lines of horizontal script text. UTR50 is not applicable for these cases. § By default, glyphs of scripts that are normally horizontal should run along a line in vertical text such that the top of the character is toward the right side of the vertical line, but there should also be a mechanism to allow them to progress down the line in upright orientation. Such a mechanism should use visual text units (such as grapheme clusters ) as a minimum text unit, but where necessary allow syllabic clusters to be treated as a unit when they involve more than one grapheme cluster. § Upright Arabic text in vertical lines should use isolated letter forms and the order of text should read top to bottom. § It should be possible for some sequences of characters (particularly digits) to run horizontally within vertical lines (tate chu yoko). 9.3 RTL/bidi text See related review comments. Useful background and overviews for this section Bidirectional text , in the Language enablement index. § Specifications that enable sloping of letterforms SHOULD provide for characters to slope either to the right or to the left according to the needs of the specific language. 9.4 Setting box positioning coordinates when text direction varies § Box positioning coordinates must take into account whether the text is horizontal or vertical. It is typical, when localizing a user interface or web page, to create mirror-images for the RTL and LTR versions. For example, it is likely that a box that appears near the left side of a window containing English content would appear near the right side of the window if the content is Arabic or Hebrew. It should preferably automatic for this to change, based on the base direction of the current context, unless there is a strong reason for using absolute geometry. One way to achieve this is to use keywords such as start and end , rather than left and right , to indicate position. 9.5 Logical properties (TBD) See related review comments. 9.6 Cursive text See related review comments. Useful background and overviews for this section Cursive text , in the Language enablement index. § Overlaps should not be exposed when transparency is applied to the joined letters in cursive text, such as for Arabic, Mongolian, and N'Ko. § When adding a text stroke or shadow, joined letters should not be separated from their neighbors in cursive script text. 9.7 Ruby text annotations See related review comments. Useful background and overviews for this section What is ruby? , W3C article. Ruby Markup , W3C article. Inline notes & annotations , in the Language enablement index. § 'Ruby' style annotations alongside base text should be supported for Chinese, Japanese, Korean and Mongolian text, in both horizontal and vertical writing modes. § Ruby implementations should support zhuyin fuhao (bopomofo) ruby for Traditional Chinese. § Ruby implementations should support a tabular content model (such that ruby contents can be arranged in a sequence approximating to rb rb rt rt ). § Ruby implementations should make it possible to use an explicit element for ruby bases, like the rb element in HTML. § Ruby implementations should allow annotations to appear on either or both sides of the base text. § Ruby markup in HTML is designed specifically for Chinese, Japanese, Korean, and Mongolian requirements, and should not be used as a general glossing mechanism. 9.8 Font management (TBD) See related review comments. 9.9 Miscellaneous See related review comments. § Line heights must allow for characters that are taller than English. § Box sizes must allow for text expansion in translation. § Line wrapping should take into account the special rules needed for non-Latin scripts. Various non-Latin writing systems don't simply wrap text on inter-word spaces. They have additional rules that must be respected. For example Chinese, Japanese and Korean wrap after characters, but don't put certain characters at the start/end of a line. Thai and other SE Asian scripts wrap at word boundaries, but words are not delimited by spaces – spaces are instead used to separate phrases. Tibetan wraps after the tsek character that follows a syllable – words are not separated by spaces, and lines can break within a word. Indic and other complex scripts break at orthographic syllable boundaries; in some cases these are comprised of two or more grapheme clusters . See the CSS Text Level 3 specification for additional background. ( This tutorial provides additional examples, if needed.) § Avoid specifying presentational tags, such as b for bold, and i for italic. more Using <b> and <i> elements , W3C article. It is best to avoid presentational markup b , i or u , since it isn't interoperable across writing systems and furthermore may cause unnecessary problems for localisation. In addition, some scripts have native approaches to things such as emphasis, that do not involve, and can be very different from, bolding, italicisation, etc. In the HTML case, there was a legacy issue, but unless there is one for your specification, the recommendation is that styling be used instead to determine the presentation of the text, and that any markup or tagging should allow for general semantic approaches. For an explanation of the issues surrounding b and i tags, see Using <b> and <i> elements . 10. Locales, date and time values, and locally affected formats Show the self-review checklist This is a list of just the requirements in this section, which can be used for self-reviews. Select the first checkbox on a line for all requirements relevant to your spec. Select the second checkbox if your spec meets the requirement. Then click on the button "Create markdown for GitHub", and copy the result to a GitHub issue list. See more details . Working with locale-affected values When definining data formats, use locale-neutral serialization forms. more Working with time When defining calendar and date systems, be sure to allow for dates prior to the common era, or at least define handling of dates outside the most common range. more When defining time or date data types, ensure that the time zone or relationship to UTC is always defined. more Provide a health warning for conversion of time or date data types that are "floating" to/from incremental types, referring as necessary to the Time Zones WG Note . more Allow for leap seconds in date and time data types. more Use consistent terminology when discussing date and time values. Use 'floating' time for time zone independent values. more Keep separate the definition of time zone from time zone offset. more Use IANA time zone IDs to identify time zones. Do not use offsets or LTO as a proxy for time zone. more Use a separate field to identify time zone. more When defining rules for a "week", allow for culturally specific rules to be applied. more When defining rules for week number of year, allow for culturally specific rules to be applied. more When non-Gregorian calendars are permitted, note that the "month" field can go to 13 (undecimber). more Working with personal names Check whether you really need to store or access given name(s) and family name(s) separately. more Avoid placing limits on the length of names, or if you do, make allowance for long strings. more Try to avoid using the labels 'first name' and 'last name'. Consider an alternative such as 'given name(s)' and 'family name(s)'. more Consider whether it would make sense to have one or more extra fields, in addition to the full name field, where users can provide part(s) of their name that you need to use for a specific purpose. more Allow for users to be asked separately how they would like to be addressed when someone contacts them. more If parts of a person's name are captured separately, ensure that the separate items can capture all relevant information. more Be careful about assumptions built into algorithms that pull out the parts of a name automatically. more Don't assume that a single letter name is an initial. more Don't require that people supply a family name. more Allow people to use punctuation such as hyphens, apostrophes, etc. in names, and take into account possible alternative code points for those characters. more Don't require names to be entered all in upper case. more Don't normalize the casing in names. more Allow the user to enter a name with spaces. more Don't assume that members of the same family will share the same family name. more It may be better for a form to ask for 'Previous name' rather than 'Maiden name' or 'née'. more You probably need to store the name in both Latin and native scripts, in which case you will need to ask the user to submit their name in both native script and Latin-only form, as separate items. more Provide a field for a transcription of the name, where necessary. more Don't block unusual or unexpected names when trying to enforce real name usage. more In standards and standards related documents containing examples that include names of persons, use a variety of names to reflect a global audience. Avoid a bias of names specific to certain regions. more Working with numbers When parsing user input of numeric values, allow for digit shaping (non-ASCII digits). more When formatting numeric values for display, allow for culturally sensitive display, including the use of non-ASCII digits (digit shaping). more When defining a feature that automatically labels items incrementally for display to the user (such as when creating a numbered list), allow for localized presentation of the labels as well as for various counting/listing systems or styles. more Designing forms When defining email field validation, allow for EAI (smtputf8) names. more User input (TBD) Localization APIs and protocols SHOULD include language and string direction metadata for all natural language messages and data fields. more All natural language fields or messages, including error messages, defined by a given API or protocol SHOULD be localized into the preferred locale of the user or, if that language is not available, supplied with a suitable fallback or default. more Specifications for APIs or protocols SHOULD define how the user's locale is determined (this is sometimes called language negotiation ). more Specifications MAY define a specific default language for messages or errors in an API or protocol. more APIs and protocols SHOULD provide language independent identifiers for errors. more Natural language error message fields, when provided, SHOULD be optional and SHOULD include language and direction metadata. more Natural language error message fields, when provided, SHOULD match the user interface language negotiated for the request when possible. more Create markdown for GitHub 10.1 Working with locale-affected values Software systems that support languages and cultural preferences are said to be internationalized . An internationalized system uses APIs to provide language or culturally specific processing, based on user preferences. These user preferences are usually referred to as a locale . For more information on general internationalization terminology, see Language Tags and Locale Identifiers [ LTLI ] § When definining data formats, use locale-neutral serialization forms. Data values that are machine-readable and not specific to any particular language or culture are more durable and less open to misinterpretation than values that use one of the many different cultural representations. Things like dates, currencies, and numbers might look similar but have different meanings in different locales . For example, a date represented as the string 4/7 can be read as the 7th of April or the 4th of July depending on the user's preference. Similarly, €2,000 is either two thousand Euros or an over-precise representation of two Euros. By using a locale-neutral format, systems avoid the need to establish specific interchange rules that vary according to the language or location of the user. When the data is already in a locale-specific format, making the locale and language explicit by providing locale parameters (usually in the form of a language tag ) allows users to determine how to work with the data or perhaps enable automated translation services. Most common data serialization formats are locale-neutral. For example, [ XMLSchema-2 ] types such as xsd:integer and xsd:date are intended for locale-neutral data interchange. Using locale-neutral representations allows the data values to be processed accurately without complex parsing or misinterpretation and also allows the data to be presented in the format most comfortable for the consumer of the data in any locale. For example, rather than storing "€2000,00" as a string, it is strongly preferred to exchange a data structure such as: … "price" { "value" : 2000.00 , "currency" : "EUR" } … 10.2 Working with time See related review comments. § When defining calendar and date systems, be sure to allow for dates prior to the common era, or at least define handling of dates outside the most common range. § When defining time or date data types, ensure that the time zone or relationship to UTC is always defined. § Provide a health warning for conversion of time or date data types that are "floating" to/from incremental types, referring as necessary to the Time Zones WG Note . more Working with Time Zones § Allow for leap seconds in date and time data types. These occur occasionally when the number of seconds in a minute is allowed to range from 0 to 60 (ie. there are sixty-ONE seconds in that minute). § Use consistent terminology when discussing date and time values. Use 'floating' time for time zone independent values. § Keep separate the definition of time zone from time zone offset. § Use IANA time zone IDs to identify time zones. Do not use offsets or LTO as a proxy for time zone. § Use a separate field to identify time zone. § When defining rules for a "week", allow for culturally specific rules to be applied. For example, the weekend is not always Saturday/Sunday; the first day of week is not always Sunday [or Monday or...], etc. § When defining rules for week number of year, allow for culturally specific rules to be applied. § When non-Gregorian calendars are permitted, note that the "month" field can go to 13 (undecimber). 10.3 Working with personal names See related review comments. Developers who create applications that use personal names (in web forms, databases, ontologies, and so forth) are often unaware of how different names can be in other countries. They build their forms or databases in a way that assumes too much on the part of foreign users. This section provides guidelines for working with personal names from around the world. Useful background and overviews for this section Personal names around the world . 10.3.1 Field length & composition § Check whether you really need to store or access given name(s) and family name(s) separately. source To split or not to split? in Personal Names Around the World . examples Examples of differences in Personal Names Around the World . Names around the world differ greatly in composition and the order of components (see Personal names around the world ). This can create difficulties if, for example, you try to split a person's name into smaller parts for storage in a database and then later attempt to retrieve them, especially if some reconstruction is needed. Difficulties include understanding which part of a person's name belongs in which database field (especially when there are more or fewer parts than fields in the database), and dealing with the ordering of name parts when retrieving someone's name from the database for actual use. If designing a form or database that will accept names from people with a variety of backgrounds, you should ask yourself whether you really need to have separate fields for things like given name and family name. This will depend on what you need to do with the data, but obviously it will be simpler, where it is possible, to just use the full name as the user provides it. § Avoid placing limits on the length of names, or if you do, make allowance for long strings. source To split or not to split? in Personal Names Around the World . examples Examples of differences in Personal Names Around the World . Bear in mind that names in some cultures can be quite a lot longer than your own. Make fields long enough to enter long names. Also do not assume that a name will have more than one letter. In particular, avoid counting length in bytes (see 4.2 Choosing a definition of 'string' ) – do not assume that a four-character Japanese name in UTF-8 will fit in four bytes; you are likely to actually need 12. 10.3.2 Guidelines for segmenting names The guidelines in this section apply where a decision has been made that it is necessary to split up a person's name for storage or presentation. § Try to avoid using the labels 'first name' and 'last name'. Consider an alternative such as 'given name(s)' and 'family name(s)'. source Strategies for splitting up names , in Personal Names Around the World . examples Examples of differences in Personal Names Around the World . Use of the terms 'first' and 'last' can be confusing for people who normally write their family name followed by given names. Although it may seem acceptable to use 'first' and 'last' for forms aimed at users in the United States, the forms may eventually be used by people with different cultural backgrounds, both within and potentially outside of the USA. Bear in mind, also, that in some cultures this is still problematic, such as for Icelanders, who don't actually have family names, but have a given name and a patronymic (see Given name and patronymic ). However, short of highly localized customization, this is probably the best we can do for a generic solution. § Consider whether it would make sense to have one or more extra fields, in addition to the full name field, where users can provide part(s) of their name that you need to use for a specific purpose. source Strategies for splitting up names , in Personal Names Around the World . examples Examples of differences in Personal Names Around the World . § Allow for users to be asked separately how they would like to be addressed when someone contacts them. source Strategies for splitting up names , in Personal Names Around the World . examples Examples of differences in Personal Names Around the World . For example, in some cases you may want to identify parts of a name so that you can sort a list of names alphabetically, or address them when contacting them, etc. This extra field would also be useful for finding the appropriate name from a long list of name components, and for handling nicknames (which, for example, are commonly used to refer to people in Thailand). Sometimes you may opt for separate fields because you want to be able to use part of the name to address the person directly, or refer to them. For example, when a social media app refers to "David's contacts". Or perhaps it's because you want to send them emails with their name at the top. Note that not only may you have problems due to name syntax here, but you also have to account for varying expectations around the world with regards to formality (not everyone is happy for a stranger to call them by their given name). It may be better to ask separately, when setting up a profile for example, how that person would like you to address them. § If parts of a person's name are captured separately, ensure that the separate items can capture all relevant information. source Strategies for splitting up names , in Personal Names Around the World . examples Examples of differences in Personal Names Around the World . For example, don't assume that the order they provide names in will be 'given name' followed by 'family name', or that it will be possible in a name that is composed of multiple words to even identify which part fits into which of those categories and which parts relate to something completely different, such as a father's name, a village name, a clan name, etc. § Be careful about assumptions built into algorithms that pull out the parts of a name automatically. source Strategies for splitting up names , in Personal Names Around the World . examples Examples of differences in Personal Names Around the World . For example, the v-card and h-card approach of implied “n” optimization could have difficulties with, say, Chinese names. The input form should be as clear as possible when telling people how to specify their name, so that you capture the data you think you need. § Don't assume that a single letter name is an initial. source Strategies for splitting up names , in Personal Names Around the World . examples Examples of differences in Personal Names Around the World . People do have names that are one letter long. These people can have problems if the form validator refuses to accept their name and demands that they supply their name in full. If you want to encourage people not to use initials, perhaps you should make that a warning message, rather than block the form submission. § Don't require that people supply a family name. source Strategies for splitting up names , in Personal Names Around the World . examples Examples of differences in Personal Names Around the World . In cultures such as parts of Southern India, Malaysia, and Indonesia, a large number of people have names that consist of a given name only, with no patronym. If you require family names, you may create significant problems in these cultures, as users enter garbage data like "." or "Mr." in the family name field just to escape the form. 10.3.3 Allowable characters § Allow people to use punctuation such as hyphens, apostrophes, etc. in names, and take into account possible alternative code points for those characters. source Other things , in Personal Names Around the World . examples Examples of differences in Personal Names Around the World . This ensures that names are correctly handled for people such as Dina Asher-Smith and Christopher O'Connell. Note that the apostrophe may appear as ' U+0027 APOSTROPHE or as ʼ U+02BC MODIFIER LETTER APOSTROPHE , or perhaps even ’ U+2019 RIGHT SINGLE QUOTATION MARK . A hyphen may be represented using - U+002D HYPHEN-MINUS or ‐ U+2010 HYPHEN or, in Japan, ゠ U+30A0 KATAKANA-HIRAGANA DOUBLE HYPHEN . § Don't require names to be entered all in upper case. source Other things , in Personal Names Around the World . examples Examples of differences in Personal Names Around the World . § Don't normalize the casing in names. source Other things , in Personal Names Around the World . examples Examples of differences in Personal Names Around the World . Some names (such as 'McNamara') contain capital letters that are not the first letter; others (such as 'van der Waals') include words that are not capitalized. Forms should preserve the case the user enters and not coerce such names to always or only use capital letters at the start of each word. § Allow the user to enter a name with spaces. source Other things , in Personal Names Around the World . examples Examples of differences in Personal Names Around the World . Allows correct capture of a family name such as that of Gabriel García Márquez (family name, García Márquez), or a given name such as José María Olazábal (family name, Olazábal). 10.3.4 Other considerations § Don't assume that members of the same family will share the same family name. source Other things , in Personal Names Around the World . examples Examples of differences in Personal Names Around the World . It would be wrong to assume that members of the same family share the same family name. There is a growing trend in the West for individuals to keep their own name after marriage, but there are other cultures, such as China, where this is the normal approach. In some countries the wife may or may not take the husband's name. When dealing with Hispanic names it may be that only the children in the family have the same family names, but they may be different from each of the parents. Manuel Pérez Quiñones derived his apellidos (Pérez Quiñones) because his father's apellidos were Pérez Rodríguez and his mother's apellidos were Quiñones Alamo. In time, he courted a girl with the apellidos Padilla Falto. When they married, her apellidos became Padilla de Pérez. Their children were called Pérez Padilla, and so on. § It may be better for a form to ask for 'Previous name' rather than 'Maiden name' or 'née'. source Other things , in Personal Names Around the World . examples Examples of differences in Personal Names Around the World . You should also not simply assume that name adoption goes from husband to wife. Sometimes men take their wife's name on marriage. It may be better, in these cases, for a form to say 'Previous name' than 'Maiden name' or 'née'. § You probably need to store the name in both Latin and native scripts, in which case you will need to ask the user to submit their name in both native script and Latin-only form, as separate items. source Implications for character support , in Personal Names Around the World . examples Examples of differences in Personal Names Around the World . The need for multiple fields will depend to some extent on what you are collecting people’s names for, and how you intend to use them. Are you collecting the person’s name just to have an identifier in your system? If so, it may not matter whether the name is stored in ASCII-only or native script. Or will they be called by name on a welcome page or in correspondence? If you will correspond using their name on pages written in their language, it would seem sensible to have the name in the native script. Is it important for people in the organization that handles queries to be able to recognize and use the person’s name? If so, you may need a Latin transcription. Will their name be displayed or searchable (for example Flickr optionally shows people’s names as well as their user name on their profile page)? Or will you want to send them correspondence in their own language, but track them in your back-office in a language such as English? If so, it may be necessary to store the name in both Latin and native scripts, in which case you probably need to ask the user to submit their name in both native script and Latin-only form, using separate fields. § Provide a field for a transcription of the name, where necessary. source Implications for character support , in Personal Names Around the World . examples Examples of differences in Personal Names Around the World . For example, Japanese users may need to provide a transcription in a Japanese syllabic script rather than/in addition to the ideographic form. This field is used for sorting Japanese names, but also allows someone looking at the name to check how it is pronounced. § Don't block unusual or unexpected names when trying to enforce real name usage. It isn't hard to find examples of people who have been blocked from using a service because their name doesn't conform to expectations of the developers. If you are planning to enforce real name usage, you need to allow a mechanism for people to validate their actual names if their name is rare, or has an unexpected structure. 10.3.5 Using personal names in examples § In standards and standards related documents containing examples that include names of persons, use a variety of names to reflect a global audience. Avoid a bias of names specific to certain regions. Many specifications provide examples, such as user stories or use cases, that use personal names as a means of enhancing the narrative. Some groups even have practices, such as security specifications using the names "Alice" and "Bob", to provide a certain level of consistency. Inclusiveness should be an important goal when building systems and services, hence the suggestion to use globally diverse names in forming examples. This helps ensure that we represent the worldwide community of users with our technologies, and makes the specification appear more relevant to the global user. Try to choose names that represent people from different regions around the world, rather than just a handful of names with European origins. Note that choosing names that include non-ASCII characters can help remind implementers that Unicode support and other internationalization concerns apply to their users. No collection of names can be completely agnostic in dealing with cultural and gender-related issues. To assist specification writers in creating more inclusive examples, this document provides a collection of names drawn from across many cultures. These names are organized approximately into regions, usually indicating country or language. Notice that even within these regions there are quite diverse influences and practices for the handling of personal names. The names are also divided by their cultural gender association to assist specification authors in writing examples, although many names are not specific to any particular gender. Inserting personal names from other cultures into English-language examples is also affected by the very different ways that names are used culturally around the world. For example, some cultures expect the use of a patronym/matronym in addition to the given name; or some cultures prefer more formal names (e.g. " Herr Dürer " vs. the informal " Albrecht "). Chinese people almost never use their given name without also including their family name. When writing examples in Chinese, one might see something like 路人甲 (it means Person A, using the Han "Heavenly Stem" ordinals, cf. Ready-made Counter Styles ) rather than a "exemplar name". When examples are used, they include both the family and given name. Bear in mind that in Chinese the family name comes first, before the given name. In Japanese, there are complex choices related to levels of formality. A person might be addressed by their given name in very informal situations ( Hiroshi ), but usually will be addressed with a family name that includes (unless one is being rude) a title or suffix, such as -san or -sama (e.g. Tanaka-san ). Other suffixes or titles are also used, such as senpai or sensei (for senior or very esteemed individuals) or shi (when one is unfamiliar with the person). Thus an example in English that could say Suppose Hiroki wants to set up a... would probably be more culturally appropriate if it read Suppose Tanaka-san wants to set up a... 10.3.5.1 Example names The following table was compiled by the Internationalization Working Group. Contributions and suggestions for additions or corrections are welcome. The purpose of this collection of names is to assist specification authors who are generally writing for an English-speaking audience. The collection consists primarily of given names and, where necessary, is transliterated into the Latin script. The names are also rendered informally ( "Alice" rather than "Ms. Jones" ), even though this is not how names would be used in many of these cultures. When translating specifications, adjustments should be made which are appropriate for the target audience. When names are taken from non-Latin-script languages or cultures, the non-Latin representation is also provided as a reminder that names are in no way limited to the Latin script or for cases where you want to include a non-Latin script example. This table can be sorted by clicking on the △ or ▽ arrows in the header row. Name △ ▽ Native △ ▽ Gender △ ▽ Region and Notes △ ▽ Language △ ▽ Akamu m Oceania; Polynesia; Hawaiian name haw Alinta f Oceania; Australian indigenous name nys Amélie f Europe; France fr An 杏 f East Asia; Japan ja Aoi 葵; 蒼; 碧 f, m East Asia; Japan ja Aroha f Oceania; Maori mi Åsa f Europe; Sweden sv Asahi 朝陽 m East Asia; Japan ja Atlahua m Latin America; Nahuatl name nah Beata f Europe; Multiple countries it, de, pl, sv, etc. Chanda चंदा f South Asia; originally from Sanskrit sa Chirapathi சிரபதி f South Asia; Tamil ta Citlali f Latin America; Nahuatl nah Coen m Europe; Netherlands; also Oceania (Australian indigenous) or Hebrew name nl, he, nys Daisho 大翔 m East Asia; Japan ja Dara f West Asia; Europe; Türkiye tr Eva Е́ва f Europe; Russia ru Faheem فهيم m West Asia; Arabic ar Fátima فَاطِمَة f West Asia; Arabic; also used in several European cultures in the Latin script ar Genet ገነት f Africa; Ethiopia am Haruto 陽翔 m East Asia; Japan ja Haukea f Oceania; Polynesia; Hawaiian name haw Himari 陽葵 f East Asia; Japan ja Hina 陽菜 f East Asia; Japan ja Hīnano m Oceania; Polynesia; Tahitian ty Hua 李华 m East Asia; China zh-Hans Iakopo m Oceania; Samoa sm Ilango இளங்கோ m South Asia; Tamil ta Irepani m Latin America; Purepecha language tsz Işık f West Asia; Europe; Türkiye tr Işıtan m West Asia; Europe; Türkiye tr Itsuki 樹 m East Asia; Japan ja Jarra, Jarrah, Cerrah جراح m West Asia; Arabic ar, tr Jean-François m Europe; French fr João m Latin America; Brazil pt-BR Júlía f Europe; Iceland is Kai f, m Oceania; Australia; appears in many languages and is a good general example aus, sm Khaliun f, m East Asia; Mongolia mn Kylie f Oceania; Australian indigenous name aus Lani f Oceania; Philippines fil Lei 李雷 m East Asia; China zh-Hans Livia f Europe, Latin America es Lowanna f Oceania; Australian indigenous aus Lucas m Latin America es Maevarau m Oceania; Samoa sm Mahmut m West Asia; Europe; Türkiye tr Martina f Latin America es Mei 芽依 ( ja ); 梅 ( zh ) f East Asia; China; Japan ja, zh Minato 湊 m East Asia; Japan ja Mio 澪 f East Asia; Japan ja Miriam מרים f West Asia; Hebrew he Müge f West Asia; Europe; Türkiye tr Muhammad محمد m West Asia; Arabic; Many variants and languages. ar Ngatemi f Oceania; Indonesia id, ms Onosaʻi f Oceania; Samoa sm Potira f Latin America; Brazil; indigenous name gn Qiàn 倩 f East Asia; China zh-Hans Rattiya รัตติยา f South-East Asia; Thailand th Ren 蓮 m East Asia; Japan ja Rin 凛 f East Asia; Japan ja Ritthichai ฤทธิชัย m South-East Asia; Thailand th Santiago m Latin America es Senthil செந்தில் m South Asia; Tamil ta Sione m Oceania; Tonga to Slobodan Слободан m Europe; Serbian sr Sofia f Europe; Latin America es Tahnee f Oceania; Australian indigenous aus Tamizhachi தமிழச்சி f South Asia; Tamil ta Temuera m Oceania; Polynesia sm Thị Anh f South-East Asia; Vietnam vi-VN Tuulikki f Europe; Finland fi Uriel אוּרִיאֵל m West Asia; Hebrew he Văn Hoa m South-East Asia; Vietnam vi-VN Vasa m Oceania; Samoa; Europe; diminutive form of Vasilije/ Василије sm, hr, sr Vassilios Βασίλειος m Europe; Greek el Voula Βούλα f Europe; Greek el Wafaa وفاء f West Asia; Arabic ar Wissam وسام m West Asia; Arabic ar Xiaoxia 晓霞 f East Asia; China zh-Hans Xóchitl f Latin America; Nahuatl nah Yevdokia Евдокия f Europe; Russia ru Yevgeny Евгений m Europe; Russia ru Zafirah زفره f West Asia; Arabic ar Note Another potential source for example names is Unicode's Common Locale Data Repository [ CLDR ] project, which maintains data about the presentation of personal names in different locales and cultures. That set of names can be accessed on CLDR's charts page . Note that those names were collected with a focus on providing examples of the formatting of personal names, rather than as examples of typical names found in these cultures. For example, many of the names are variations on the name "Sinbad", even though "Sinbad" is not a commonly used name in many of these cultures. Other names are taken from sources, such as folk tales, that would be familiar in a given culture but not representative of actual persons. 10.4 Working with numbers § When parsing user input of numeric values, allow for digit shaping (non-ASCII digits). § When formatting numeric values for display, allow for culturally sensitive display, including the use of non-ASCII digits (digit shaping). § When defining a feature that automatically labels items incrementally for display to the user (such as when creating a numbered list), allow for localized presentation of the labels as well as for various counting/listing systems or styles. Examples of this can be found in CSS Counter Styles [ css-counter-styles-3 ] and especially the accompanying Ready-made Counter Styles [ predefined-counter-styles ]. more Defining counter styles in CSS: CSS Counter Styles . Ready-made Counter Styles . 10.5 Designing forms See related review comments. § When defining email field validation, allow for EAI (smtputf8) names. 10.6 User input (TBD) See related review comments. 10.7 Localization Localization [ LTLI ] enables users to employ software in the language and locale of their choice. Specifications for protocols and document formats need to consider how to provide the language and formatting that the end-user expects. Natural language data values need language and base direction in order to ensure proper presentation, even if localized messages are not provided. This includes any error messages or other internal messages that are human readable in an API or protocol. See also [ STRING-META ]. § APIs and protocols SHOULD include language and string direction metadata for all natural language messages and data fields. § All natural language fields or messages, including error messages, defined by a given API or protocol SHOULD be localized into the preferred locale of the user or, if that language is not available, supplied with a suitable fallback or default. § Specifications for APIs or protocols SHOULD define how the user's locale is determined (this is sometimes called language negotiation ). § Specifications MAY define a specific default language for messages or errors in an API or protocol. Note Specifications do not need to require that messages be returned in all possible or all available locales. It is sufficient to make it possible to localize the end-user's customer experience. Implementations can choose which languages or locales to support. 10.7.1 Working with error and exception messages Protocols, APIs, and document formats sometimes provide a field to pass a human-readable error or exception message from a service to the caller in the form of a string. In general, and as indicated above , any natural language text conveying human-readable messages or human-readable content needs to be associated with language and direction metadata. Where this metadata is missing, the processing or display of the text might be compromised. Often the intention of the specification author in providing an error or exception message is to convey debugging information to a software developer. Specification authors sometimes assume that error or exception messages are not seen by end users; that software developers will prefer these messages to be unlocalized or appear in a specific language (usually English); or that there are other "practical reasons" why localization of error messages can turn out to be a barrier. For example, there are anecdotes about developers finding it easier to search the Web with the (usually obscure) text of an error because the message itself is insufficiently good at explaining the problem. Searching for this text might produce a result in the developer's preferred language that is more helpful. Error messages are messages and they are intended for humans, not machines. In many cases, the error message encompasses all of the additional information about what went wrong and, in some cases, the caller is obliged to show the message to the actual end user because there is no other way to convey to the caller about how to fix the problem ("Your credit card has expired"; "The value 10484977 is too large" [oops, forgot the decimal point]; etc.). Localization of these types of messages is actually a good thing and may even be obligatory in some applications. § APIs and protocols SHOULD provide language independent identifiers for errors. For example, HTTP result codes, such as the familiar 404 , help users communicate which error they received or look up a translation. § Natural language error message fields, when provided, SHOULD be optional and SHOULD include language and direction metadata. § Natural language error message fields, when provided, SHOULD match the user interface language negotiated for the request when possible. 11. Navigation Show the self-review checklist This is a list of just the requirements in this section, which can be used for self-reviews. Select the first checkbox on a line for all requirements relevant to your spec. Select the second checkbox if your spec meets the requirement. Then click on the button "Create markdown for GitHub", and copy the result to a GitHub issue list. See more details . Providing for content negotiation based on language In a multilingual environment it must be possible for the user to receive text in the language they prefer. This may depend on implicit user preferences based on the user's system or browser setup, or on user settings explicitly negotiated with the user. more Create markdown for GitHub 11.1 Providing for content negotiation based on language See related review comments. § In a multilingual environment it must be possible for the user to receive text in the language they prefer. This may depend on implicit user preferences based on the user's system or browser setup, or on user settings explicitly negotiated with the user. 12. Editorial guidelines for specification authors Show the self-review checklist This is a list of just the requirements in this section, which can be used for self-reviews. Select the first checkbox on a line for all requirements relevant to your spec. Select the second checkbox if your spec meets the requirement. Then click on the button "Create markdown for GitHub", and copy the result to a GitHub issue list. See more details . Editorial guidelines for specification authors Use the lang attribute with an appropriate language tag to ensure correct font selection and rendering of examples (particularly those not in the document's primary language). more Use the dir attribute and appropriate Unicode bidirectional formatting characters in the specification's markup to ensure that text examples display properly (and to ensure that they are appropriately using bidi isolation ). more Use the HTML attribute translate [ HTML ] with a value of no to signal to translators the portions of the document that contain text that are to be preserved (not translated) when performing the translation. (see also: [ ITS ]) more Ensure that your examples represent your global audience, including examples that vary in language, region, time zones, and other variations appropriate to your specification. more If a value (field, name, property, etc.) supports non-ASCII characters, include non-ASCII examples so that it is clear that such values are permitted and support for such values is required. more Where appropriate, specifications SHOULD include non-English data in examples. more Specifications SHOULD represent diverse regions and cultures by including examples from different countries or regions. Avoid examples that are idiosyncratic to one country, such as the USA, which might not be understood by international users. more Where it does not interfere with clarity, follow best practices found elsewhere in this document in examples. more Writing Considerations Use the lang attribute with an appropriate language tag to ensure correct font selection and rendering of examples (particularly those not in the document's primary language). more Use the dir attribute and appropriate Unicode bidirectional formatting characters in the specification's markup to ensure that text examples display properly (and to ensure that they are appropriately using bidi isolation ). more Use the HTML attribute translate [ HTML ] with a value of no to signal to translators the portions of the document that contain text that are to be preserved (not translated) when performing the translation. (see also: [ ITS ]) more Creating examples Ensure that your examples represent your global audience, including examples that vary in language, region, time zones, and other variations appropriate to your specification. more If a value (field, name, property, etc.) supports non-ASCII characters, include non-ASCII examples so that it is clear that such values are permitted and support for such values is required. more Where appropriate, specifications SHOULD include non-English data in examples. more Specifications SHOULD represent diverse regions and cultures by including examples from different countries or regions. Avoid examples that are idiosyncratic to one country, such as the USA, which might not be understood by international users. more Where it does not interfere with clarity, follow best practices found elsewhere in this document in examples. more Create markdown for GitHub Specifications serve diverse audiences. Most of the best practices in this document are focused on the technical requirements of whatever it is that your Specification is specifying. In this section, you'll find some additional best practices to apply to your writing. These will help you address a global audience more effectively by avoiding cultural biases or help implementers understand internationalization and localization requirements. 12.1 Writing Considerations § Use the lang attribute with an appropriate language tag to ensure correct font selection and rendering of examples (particularly those not in the document's primary language). more Choosing a Language Tag To ensure proper rendering, including by assistive technologies, use the HTML lang attribute with an appropriate language tag around examples that are not in the document's primary language. (This is one of the purposes behind inclusion of lang in the character reference templates in 4.11 Referring to Unicode characters .) For non-linguistic content, use the language tag zxx to indicate that the content does not contain natural language . Example 28 : Use of the 'lang' attribute Here is an example of how the lang attribute might be used in a specification. Note that a language tag is still needed for values that are ASCII-only, not just for the names that use a non-Latin script! < html lang = "en" > <!-- Establishes the document language --> ... < p > We invited Terry, < span lang = "ru" > Е́ва </ span > , < span lang = "es" > Lucas </ span > , < span lang = "am" > ገነት </ span > , and < span lang = "tr" > Mahmut </ span > to the party. </ p > § Use the dir attribute and appropriate Unicode bidirectional formatting characters in the specification's markup to ensure that text examples display properly (and to ensure that they are appropriately using bidi isolation ). more Strings on the Web: Language and Direction Metadata [ STRING-META ] Strings and bidi Inline markup and bidirectional text in HTML Working with source code markup and code examples for RTL scripts § Use the HTML attribute translate [ HTML ] with a value of no to signal to translators the portions of the document that contain text that are to be preserved (not translated) when performing the translation. (see also: [ ITS ]) more Using HTML's translate attribute The translate attribute, originally defined in Internationalization Tag Set (ITS) Version 1.0 [ ITS ] and later added to [ HTML ], signals either non-linguistic content such as data values or reserved keywords. Such keywords might be part of your specification's syntactic content or might be content (such as reserved keywords) you are quoting from some other specificiation. Using translate="no" signals to the translator (including machine translations) that the text is not to be modified during the translation process. Example 29 : Use of translate attribute For example: < p > The paragraph above uses the markup < code translate = "no" > translate="no" </ code > . Use markup like this when presenting non-lingustic data, for example: < span translate = "no" lang = "zxx" dir = "ltr" > 127.0.0.1 </ span > is the IP address of < code translate = "no" > localhost </ code > </ p > 12.2 Creating examples See related review comments. § Ensure that your examples represent your global audience, including examples that vary in language, region, time zones, and other variations appropriate to your specification. Your first responsibility, of course, is to make your Specification clear and highly readable. To that end, when providing examples of data formats, APIs, and such, you'll often use English language values to help illuminate concepts. When you have room or when it doesn't interfere with readability, though, choose values (or add additional examples) that are non-ASCII, non-English, and non-provincial. This makes the Specification more relevant to global audiences while reminding implementers that they need to support these diverse audiences. See also: We covered choosing and exhibiting personal names earlier here: 10.3.5 Using personal names in examples § If a value (field, name, property, etc.) supports non-ASCII characters, include non-ASCII examples so that it is clear that such values are permitted and support for such values is required. If you permit it: show it. If a field, name, or other value can contain non-ASCII data, include at least a few examples that demonstrate this. Often this can be a second or third example, rather than the primary one, so as not to interfere with readability or editing. Showing non-ASCII values reminds implementers that they need to support international values. § Where appropriate, specifications SHOULD include non-English data in examples. § Specifications SHOULD represent diverse regions and cultures by including examples from different countries or regions. Avoid examples that are idiosyncratic to one country, such as the USA, which might not be understood by international users. § Where it does not interfere with clarity, follow best practices found elsewhere in this document in examples. For example, always include language and direction metadata with natural language content. If an item is localizable text , show how it is localized. Show direction metadata for both left-to-right and right-to-left content. A. Revision Log The following summarises substantive changes since the previous publication, but the material is still subject to significant flux as it develops. This should not be a reason not to use the document. What it so far contains is useful, and any shortfalls can be reported or discussed. Links were added below section headings that point to lists of review comments related to that section. These comments provide details useful to developers and reviewers. The checklist generator tool was moved to the top of the document. The table of contents now reports 3 levels of heading. Lists of links to documents that provide useful background or an overview of a section have been moved to the start of that section. So also have 'see also' links. Each advisement now carries its own set of links. This makes the links more relevant and more easily noticeable. It also makes it easier to list multiple links, and because the links indicate the target document title, readers do not have to follow the link to know whether they have already read the document pointed to. Links associated with an advisement are of two types: 'explanations & examples' typically points to a location in another document from which this advisement was lifted, and surrounds it there with explanatory text; 'more' links provide further reading in other documents. Self-links for each advisement have been changed so that they match the standard style used for headings (§ to the side of the text). This also significantly reduces the complexity of authoring the markup. Added content in the section on locales , along with text about working with file and path names and working with error messages . Generally, the markup in the document source has been greatly simplified, making it easier and quicker to maintain the document. See the github commit log for more details. B. Acknowledgements Thanks to Addison Phillips for help reviewing old reviews for recommendations. Other people who contributed through reviews or issues include Steve Atkin, Andrew Cunningham, Martin Dürst, Asmus Freytag, John Klensin, Tomer Mahlin, Chaals McCathieNevile, Florian Rivoal, Najib Tounsi. Some material about locale-neutral representation was adapted from [ DWBP ]. Permalink Referenced in: § 2. Language § 2.1 Language basics Permalink Referenced in: Not referenced in this document. Permalink Referenced in: Not referenced in this document. Permalink Referenced in: Not referenced in this document. Permalink Referenced in: § 2. Language § 2.4 Identifying the language of strings Permalink Referenced in: § 2. Language § 2.6 Matching language and language tags (2) (3) (4) (5) Permalink Referenced in: § 2.6 Matching language and language tags (2) Permalink Referenced in: Not referenced in this document. Permalink Referenced in: § 4.1 Characters and character encoding basics (2) (3) (4) § 4.2.1 Characters stored in byte sequences Permalink Referenced in: Not referenced in this document. Permalink Referenced in: § 4.1 Characters and character encoding basics (2) § 4.2 Choosing a definition of 'string' Permalink Referenced in: § 4.1 Characters and character encoding basics (2) (3) (4) (5) (6) § 6. Text-processing (2) (3) (4) § 6.5 Truncating or limiting the length of strings (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) (13) (14) (15) Permalink Referenced in: § 4.1 Characters and character encoding basics Permalink Referenced in: Not referenced in this document. Permalink Referenced in: § 8. Document formats, markup & syntax § 8.3 Defining keywords, identifiers, and namespaces in syntactic content Permalink exported Referenced in: § 8. Document formats, markup & syntax (2) (3) § 8.3 Defining keywords, identifiers, and namespaces in syntactic content § 8.3.1 Case sensitivity versus insensitivity (2) (3) C. References C.1 Informative references [ASCII] ISO/IEC 646:1991, Information technology -- ISO 7-bit coded character set for information interchange . Ecma International. URL: https://www.ecma-international.org/publications-and-standards/standards/ecma-6/ [BCP47] Tags for Identifying Languages . A. Phillips, Ed.; M. Davis, Ed. IETF. September 2009. Best Current Practice. URL: https://www.rfc-editor.org/info/rfc5646/ [CHARMOD] Character Model for the World Wide Web 1.0: Fundamentals . Martin Dürst; François Yergeau; Richard Ishida; Misha Wolf; Tex Texin et al. W3C. 15 February 2005. W3C Recommendation. URL: https://www.w3.org/TR/charmod/ [CHARMOD-NORM] Character Model for the World Wide Web: String Matching . Addison Phillips et al. W3C. 16 July 2026. FPWD. URL: https://www.w3.org/TR/charmod-norm/ [CLDR] Unicode Common Locale Data Repository . Unicode Consortium. URL: https://cldr.unicode.org/ [CSS] CSS Snapshot 2026 . Tab Atkins Jr.; Elika Etemad; Florian Rivoal; Chris Lilley; Sebastian Zartner. W3C. 22 June 2026. W3C Working Group Note. URL: https://www.w3.org/TR/css-2026/ [css-counter-styles-3] CSS Counter Styles Level 3 . Tab Atkins Jr. W3C. 27 July 2021. W3C Candidate Recommendation. URL: https://www.w3.org/TR/css-counter-styles-3/ [css-overflow-4] CSS Overflow Module Level 4 . David Baron; Florian Rivoal; Elika Etemad. W3C. 21 March 2023. W3C Working Draft. URL: https://www.w3.org/TR/css-overflow-4/ [CSS3-TEXT] CSS Text Module Level 3 . Elika Etemad; Koji Ishii; Florian Rivoal. W3C. 8 June 2026. CRD. URL: https://www.w3.org/TR/css-text-3/ [DESIGN-PRINCIPLES] Web Platform Design Principles . Martin Thomson; Jeffrey Yasskin. W3C. 24 February 2026. W3C Working Group Note. URL: https://www.w3.org/TR/design-principles/ [DOM] DOM Standard . Anne van Kesteren. WHATWG. Living Standard. URL: https://dom.spec.whatwg.org/ [DWBP] Data on the Web Best Practices . Bernadette Farias Loscio; Caroline Burle; Newton Calegari. W3C. 31 January 2017. W3C Recommendation. URL: https://www.w3.org/TR/dwbp/ [Encoding] Encoding Standard . Anne van Kesteren. WHATWG. Living Standard. URL: https://encoding.spec.whatwg.org/ [EPUB-33] EPUB 3.3 . Ivan Herman; Matt Garrish; Dave Cramer. W3C. 13 January 2026. W3C Recommendation. URL: https://www.w3.org/TR/epub-33/ [HTML] HTML Standard . Anne van Kesteren; Domenic Denicola; Dominic Farolino; Ian Hickson; Philip Jägenstedt; Simon Pieters. WHATWG. Living Standard. URL: https://html.spec.whatwg.org/multipage/ [I18N-GLOSSARY] Internationalization Glossary . Richard Ishida; Addison Phillips. W3C. 17 October 2024. W3C Working Group Note. URL: https://www.w3.org/TR/i18n-glossary/ [INFRA] Infra Standard . Anne van Kesteren; Domenic Denicola. WHATWG. Living Standard. URL: https://infra.spec.whatwg.org/ [INTERNATIONAL-SPECS] Internationalization Best Practices for Spec Developers . Richard Ishida; Addison Phillips. W3C. 16 July 2026. W3C Working Group Note. URL: https://www.w3.org/TR/international-specs/ [ITS] Internationalization Tag Set (ITS) Version 1.0 . Christian Lieske; Felix Sasaki. W3C. 3 April 2007. W3C Recommendation. URL: https://www.w3.org/TR/its/ [JSON-LD] JSON-LD 1.0 . Manu Sporny; Gregg Kellogg; Markus Lanthaler. W3C. 3 November 2020. W3C Recommendation. URL: https://www.w3.org/TR/json-ld/ [JSON-SCHEMA] JSON Schema: A Media Type for Describing JSON Documents . Austin Wright; Henry Andrews; Ben Hutton; Greg Dennis. Internet Engineering Task Force (IETF). 10 June 2022. Internet-Draft. URL: https://datatracker.ietf.org/doc/html/draft-bhutton-json-schema [LTLI] Language Tags and Locale Identifiers for the World Wide Web . Addison Phillips. W3C. 7 October 2020. W3C Working Draft. URL: https://www.w3.org/TR/ltli/ [predefined-counter-styles] Ready-made Counter Styles . Richard Ishida. W3C. 7 March 2025. W3C Working Group Note. URL: https://www.w3.org/TR/predefined-counter-styles/ [RFC2277] IETF Policy on Character Sets and Languages . H. Alvestrand. IETF. January 1998. Best Current Practice. URL: https://www.rfc-editor.org/info/rfc2277/ [RFC3629] UTF-8, a transformation format of ISO 10646 . F. Yergeau. IETF. November 2003. Internet Standard. URL: https://www.rfc-editor.org/info/rfc3629/ [RFC3986] Uniform Resource Identifier (URI): Generic Syntax . T. Berners-Lee; R. Fielding; L. Masinter. IETF. January 2005. Internet Standard. URL: https://www.rfc-editor.org/info/rfc3986/ [RFC3987] Internationalized Resource Identifiers (IRIs) . M. Duerst; M. Suignard. IETF. January 2005. Proposed Standard. URL: https://www.rfc-editor.org/info/rfc3987/ [RFC4647] Matching of Language Tags . A. Phillips, Ed.; M. Davis, Ed. IETF. September 2006. Best Current Practice. URL: https://www.rfc-editor.org/info/rfc4647/ [RFC4648] The Base16, Base32, and Base64 Data Encodings . S. Josefsson. IETF. October 2006. Proposed Standard. URL: https://www.rfc-editor.org/info/rfc4648/ [RFC6067] BCP 47 Extension U . M. Davis; A. Phillips; Y. Umaoka. IETF. December 2010. Informational. URL: https://www.rfc-editor.org/info/rfc6067/ [RFC9110] HTTP Semantics . R. Fielding, Ed.; M. Nottingham, Ed.; J. Reschke, Ed. IETF. June 2022. Internet Standard. URL: https://httpwg.org/specs/rfc9110.html [STRING-META] Strings on the Web: Language and Direction Metadata . Richard Ishida; Addison Phillips. W3C. 16 July 2026. FPWD. URL: https://www.w3.org/TR/string-meta/ [UAX29] Unicode Text Segmentation . Josh Hadley. Unicode Consortium. 17 August 2025. Unicode Standard Annex #29. URL: https://www.unicode.org/reports/tr29/tr29-47.html [UAX31] Unicode Identifiers and Syntax . Mark Davis; Robin Leroy. Unicode Consortium. 20 August 2025. Unicode Standard Annex #31. URL: https://www.unicode.org/reports/tr31/tr31-43.html [UAX44] Unicode Character Database . Ken Whistler. Unicode Consortium. 27 August 2025. Unicode Standard Annex #44. URL: https://www.unicode.org/reports/tr44/tr44-36.html [UAX9] Unicode Bidirectional Algorithm . Manish Goregaokar मनीष गोरेगांवकर; Robin Leroy. Unicode Consortium. 13 August 2025. Unicode Standard Annex #9. URL: https://www.unicode.org/reports/tr9/tr9-51.html [Unicode] The Unicode Standard . Unicode Consortium. URL: https://www.unicode.org/versions/latest/ [URL] URL Standard . Anne van Kesteren. WHATWG. Living Standard. URL: https://url.spec.whatwg.org/ [UTR50] Unicode Vertical Text Layout . Ken Lunde 小林剣; Koji Ishii 石井宏治. Unicode Consortium. 24 July 2025. Unicode Standard Annex #50. URL: https://www.unicode.org/reports/tr50/tr50-33.html [UTS10] Unicode Collation Algorithm . Ken Whistler; Markus Scherer. Unicode Consortium. 3 September 2025. Unicode Technical Standard #10. URL: https://www.unicode.org/reports/tr10/tr10-53.html [UTS18] Unicode Regular Expressions . Mark Davis. Unicode Consortium. 16 January 2025. Unicode Technical Standard #18. URL: https://www.unicode.org/reports/tr18/tr18-25.html [UTS35] Unicode Locale Data Markup Language (LDML) . Mark Davis et al. Unicode Consortium. 23 October 2020. Unicode Technical Standard #35. URL: https://www.unicode.org/reports/tr35/tr35-61/tr35.html [WEBIDL] Web IDL Standard . Edgar Chen; Timothy Gu. WHATWG. Living Standard. URL: https://webidl.spec.whatwg.org/ [XML] Extensible Markup Language (XML) 1.0 (Fifth Edition) . Tim Bray; Jean Paoli; Michael Sperberg-McQueen; Eve Maler; François Yergeau et al. W3C. 26 November 2008. W3C Recommendation. URL: https://www.w3.org/TR/xml/ [XMLSchema-2] XML Schema Part 2: Datatypes Second Edition . Paul V. Biron; Ashok Malhotra. W3C. 28 October 2004. W3C Recommendation. URL: https://www.w3.org/TR/xmlschema-2/ [XMLSCHEMA11-2] W3C XML Schema Definition Language (XSD) 1.1 Part 2: Datatypes . David Peterson; Sandy Gao; Ashok Malhotra; Michael Sperberg-McQueen; Henry Thompson; Paul V. Biron et al. W3C. 5 April 2012. W3C Recommendation. URL: https://www.w3.org/TR/xmlschema11-2/ [xpath-functions] XQuery 1.0 and XPath 2.0 Functions and Operators (Second Edition) . Ashok Malhotra; Jim Melton; Norman Walsh; Michael Kay. W3C. 14 December 2010. W3C Recommendation. URL: https://www.w3.org/TR/xpath-functions/ ↑

Related documents

• latn gap#649807
• grek gap#649808
• mong gap#649809
• nkoo gap#649810
• osge osa gap#649811