W3C TRopen access

xpath full text 10

XQuery and XPath Full Text 1.0 /**/ code { font-family: monospace; }

div.constraint, div.issue, div.note, div.notice { margin-left: 2em; }

div.issue p.title { margin-left: -2em; }

ol.enumar { list-style-type: decimal; } ol.enumla { list-style-type: lower-alpha; } ol.enumlr { list-style-type: lower-roman; } ol.enumua { list-style-type: upper-alpha; } ol.enumur { list-style-type: upper-roman; }

li p { margin-top: 0.3em; margin-bottom: 0.3em; }

sup small { font-style: italic; color: #8F8F8F; } div.exampleInner pre { margin-left: 1em; margin-top: 0em; margin-bottom: 0em} div.exampleOuter {border: 4px double gray; margin: 0em; padding: 0em} div.exampleInner { background-color: #d5dee3; border-top-width: 4px; border-top-style: double; border-top-color: #d3d3d3; border-bottom-width: 4px; border-bottom-style: double; border-bottom-color: #d3d3d3; padding: 4px; margin: 0em } div.exampleWrapper { margin: 4px } div.exampleHeader { font-weight: bold; margin: 4px}

div.issue { border-bottom-color: black; border-bottom-style: solid; border-bottom-width: 1pt; margin-bottom: 20pt; }

th.issue-toc-head { border-bottom-color: black; border-bottom-style: solid; border-bottom-width: 1pt; }

table.small { font-size: x-small; } table.small { font-size: x-small; } a.judgment:visited, a.judgment:link { font-family: sans-serif; color: black; text-decoration: none } a.processing:visited, a.processing:link { color: black; text-decoration: none } a.env:visited, a.env:link { color: black; text-decoration: none } /**/ This version: http://www.w3.org/TR/2011/REC-xpath-full-text-10-20110317/ Latest version: http://www.w3.org/TR/xpath-full-text-10/ Previous version: http://www.w3.org/TR/2011/PR-xpath-full-text-10-20110125/ Editors: Pat Case, Library of Congress Michael Dyck, Invited Expert Mary Holstege, Mark Logic Corporation Sihem Amer-Yahia, AT&T Labs - Research Chavdar Botev, Invited Expert Stephen Buxton, Mark Logic Corporation Jochen Doerre, IBM Jim Melton, Oracle Michael Rys, Microsoft Jayavel Shanmugasundaram, Invited Expert Please refer to the errata See also translations This document is also available in these non-normative formats: XML Changes since Candidate Recommendation Copyright W3C ® MIT ERCIM Keio liability trademark document use Note: This paragraph is informative. https://www.w3.org/TR/xpath-full-text-3/ This document defines the syntax and formal semantics of XQuery and XPath Full Text 1.0, which is a language that extends XQuery 1.0 [XQuery 1.0: An XML Query Language (Second Edition)] [XML Path Language (XPath) 2.0 (Second Edition)] This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index This is a Recommendation XML Query Working Group XSL Working Group XML Activity This document incorporates minor changes made against the Proposed Recommendation of 25 January 2011. Changes to this document since the Proposed Recommendation are detailed in J Change Log http://www.w3.org/2010/02/qt-applets/xquery10-fulltext/ A Test Suite has been created for this document. Implementors are encouraged to run this test suite and report their results. The Test Suite can be found at http://dev.w3.org/cvsweb/2007/xpath-full-text-10-test-suite/ http://dev.w3.org/2007/xpath-full-text-10-test-suite/PublicPagesStagingArea/ReportedResults/XQFTTSReport.html No substantive changes have been made to this specification since its publication as a Proposed Recommendation. Please report errors in this document using W3C's public Bugzilla system http://www.w3.org/XML/2005/04/qt-bugzilla [email protected] http://lists.w3.org/Archives/Public/public-qt-comments/ This document has been reviewed by W3C Members, by software developers, and by other W3C groups and interested parties, and is endorsed by the Director as a W3C Recommendation. It is a stable document and may be used as reference material or cited from another document. W3C's role in making the Recommendation is to draw attention to the specification and to promote its widespread deployment. This enhances the functionality and interoperability of the Web. This document was produced by groups operating under the 5 February 2004 W3C Patent Policy public list of any patent disclosures public list of any patent disclosures Essential Claim(s) section 6 of the W3C Patent Policy 1 Introduction Full-Text Search and XML Organization of this document A word about namespaces Full-Text Extensions to XQuery and XPath Processing Model Full-Text Contains Expression Description Examples Score Variables Using Weights Within a Scored FTContainsExpr Extensions to the Static Context Full-Text Selections Primary Full-Text Selections Weights Search Tokens and Phrases Cardinality Selection Match Options Language Option Wildcard Option Thesaurus Option Stemming Option Case Option Diacritics Option Stop Word Option Extension Option Logical Full-Text Operators Or-Selection And-Selection Mild-Not Selection Not-Selection Positional Filters Ordered Selection Window Selection Distance Selection Scope Selection Anchoring Selection Ignore Option Extension Selections Semantics Tokenization Examples Representations of Tokenized Text and Matching Evaluation of FTSelections AllMatches Formal Model Examples XML representation XML Representation The evaluate function FTWords Match Options Semantics Types High-Level Semantics Formal Semantics Functions FTCaseOption FTDiacriticsOption FTStemOption FTThesaurusOption FTStopWordOption FTLanguageOption FTWildCardOption Full-Text Operators Semantics FTOr FTAnd FTUnaryNot FTMildNot FTOrder FTScope FTContent FTWindow FTDistance FTTimes FTContainsExpr Scoring Example Conformance Minimal Conformance Optional Features FTMildNot Operator FTUnaryNot Operator FTUnit and FTBigUnit FTOrder Operator FTScope Operator FTWindow Operator FTDistance Operator FTTimes Operator FTContent Operator FTCaseOption FTStopWordOption FTLanguageOption FTIgnoreOption Scoring Weights XQueryX Conformance A EBNF for XQuery 1.0 Grammar with Full Text extensions Terminal Symbols EBNF for XPath 2.0 Grammar with Full-Text extensions Terminal Symbols Static Context Components Error Conditions XML Syntax (XQueryX) for XQuery and XPath Full Text 1.0 XQueryX representation of XQuery and XPath Full Text 1.0 XQueryX stylesheet for XQuery and XPath Full Text 1.0 XQueryX for XQuery and XPath Full Text 1.0 example Example XQuery solution in XQuery and XPath Full Text 1.0 Use Cases: A Solution in Full Text XQueryX: Transformation of Full Text XQueryX Solution into XQuery Full Text References Normative References Non-normative References Acknowledgements Glossary Checklist of Implementation-Defined Features Change Log This document defines the language and the formal semantics of XQuery and XPath Full Text 1.0. This language is designed to meet the requirements identified in W3C XQuery and XPath Full Text Requirements [XQuery and XPath Full Text 1.0 Requirements] [XQuery and XPath Full Text 1.0 Use Cases] In this document, examples and material labeled as "Note" are provided for explanatory purposes and are not normative. XQuery and XPath Full Text 1.0 extends the syntax and semantics of XQuery 1.0 and XPath 2.0. Additionally, this document defines an XML syntax for XQuery and XPath Full Text 1.0. The most recent versions of the two XQueryX XML Schemas and the XQueryX XSLT stylesheet for XQuery and XPath Full Text 1.0 are available at http://www.w3.org/2007/xpath-full-text/xpath-full-text-10-xqueryx.xsd http://www.w3.org/2007/xpath-full-text/xpath-full-text-10-xqueryx-ftmatchoption-extensions.xsd http://www.w3.org/2007/xpath-full-text/xpath-full-text-10-xqueryx.xsl As XML becomes mainstream, users expect to be able to search their XML documents. This requires a standard way to do full-text search, as well as structured searches, against XML documents. A similar requirement for full-text search led ISO to define the SQL/MM-FT [SQL/MM] XML documents may contain highly structured data (fixed schemas, known types such as numbers, dates), semi-structured data (flexible schemas and types), markup data (text with embedded tags), and unstructured data (untagged free-flowing text). Where a document contains unstructured or semi-structured data, it is important to be able to search using Information Retrieval techniques such as scoring and weighting. Full-text search is different from substring search in many ways: A full-text search searches for tokens and phrases rather than substrings. A substring search for news items that contain the string "lease" will return a news item that contains "Foobar Corporation releases version 20.9 ...". A full-text search for the token "lease" will not. There is an expectation that a full-text search will support language-based searches which substring search cannot. An example of a language-based search is "find me all the news items that contain a token with the same linguistic stem as 'mouse'" (finds "mouse" and "mice"). Another example based on token proximity is "find me all the news items that contain the tokens 'XML' and 'Query' allowing up to 3 intervening tokens". Full-text search must address the vagaries and nuances of language. Search results are often of varying usefulness. When you search a web site for cameras that cost less than $100, this is an exact search. There is a set of cameras that matches this search, and a set that does not. Similarly, when you do a string search across news items for "mouse", there is only 1 expected result set. When you do a full-text search for all the news items that contain the token "mouse", you probably expect to find news items containing the token "mice", and possibly "rodents", or possibly "computers". Not all results are equal. Some results are more "mousey" than others. Because full-text search may be inexact, we have the notion of score or relevance. We generally expect to see the most relevant results at the top of the results list. Note: As XQuery and XPath evolve, they may apply the notion of score to querying structured data. For example, when making travel plans or shopping for cameras, it is sometimes useful to get an ordered list of near matches in addition to exact matches. If XQuery and XPath define a generalized inexact match, we expect XQuery and XPath to utilize the scoring framework provided by XQuery and XPath Full Text 1.0. [ Definition Full-text queries Tokenization, in general terms, is the process of converting a text string into smaller units that are used in query processing. Those units, called tokens, are the most basic text units that a full-text search can refer to. Full-text operators typically work on sequences of tokens found in the target text of a search. These tokens are characterized by integers that capture the relative position(s) of the token inside the string, the relative position(s) of the sentence containing the token, and the relative position(s) of the paragraph containing the token. The positions typically comprise a start and an end position. Tokenization, including the definition of the term "tokens", SHOULD implementation-defined SHOULD 4.1 Tokenization [ Definition token implementation-defined Definition phrase implementation-defined Note: Consecutive tokens need not be separated by either punctuation or space, and tokens may overlap. Note: In some natural languages, tokens and words can be used interchangeably. [ Definition sentence implementation-defined [ Definition paragraph implementation-defined Some XML elements represent semantic markup, e.g., <title>. Others represent formatting markup, e.g., <b> to indicate bold. Semantic markup serves well as token boundaries. Some formatting markup serves well as token boundaries; for example, paragraphs are most commonly delimited by formatting markup. Other formatting markup may not serve well as token boundaries. Implementations are free to provide implementation-defined A sample tokenization is used for the examples in this document. The results might be different for other tokenizations. Tokenization enables functions and operators that operate on a part or the root of the token (e.g., wildcards, stemming). Tokenization enables functions and operators which work with the relative positions of tokens (e.g., proximity operators). This specification focuses on functionality that serves all languages. It also selectively includes functionalities useful within specific families of languages. For example, searching within sentences and paragraphs is useful to many western languages and to some non-western languages, so that functionality is incorporated into this specification. Certain aspects of language processing are described in this specification as implementation-defined implementation-dependent [ Definition Implementation-defined [ Definition Implementation-dependent This document is organized as follows. We first present a high level syntax syntax and examples semantics EBNF for the XPath 2.0 Grammar with Full-Text Extensions EBNF for XQuery 1.0 Grammar with Full-Text Extensions acknowledgements glossary Certain namespace prefixes are predeclared by XQuery 1.0 and, by implication, by this specification, and bound to fixed namespace URIs. These namespace prefixes are as follows: xml = http://www.w3.org/XML/1998/namespace xs = http://www.w3.org/2001/XMLSchema xsi = http://www.w3.org/2001/XMLSchema-instance fn = http://www.w3.org/2005/xpath-functions local = http://www.w3.org/2005/xquery-local-functions In addition to the prefixes in the above list, this document uses the prefix err http://www.w3.org/2005/xqt-errors [XML Path Language (XPath) 2.0 (Second Edition)] [XQuery 1.0 and XPath 2.0 Functions and Operators (Second Edition)] Finally, this document uses the prefix fts XQuery and XPath Full Text 1.0 extends the languages of XQuery 1.0 and XPath 2.0 in three ways. It: Adds a new expression called FTContainsExpr; Enhances the syntax of FLWOR expressions in XQuery 1.0 and for Adds static context declarations for full-text match options to the query prolog. Additionally, it extends the data model and processing models in various ways. A full-text contains expression 2.2 Full-Text Contains Expression An XPath 2.0 or XQuery 1.0 expression (RangeExpr) that specifies the sequence of items to be searched. [ Definition search context The full-text selection to be applied ( 3 Full-Text Selections Full-text selections Required: Tokens and phrases for which a search is performed ( 3.2 Search Tokens and Phrases Optional: Match options, such as indicators for case sensitivity and stop words ( 3.4 Match Options Boolean full-text operators, that compose a full-text selection from simpler full-text selections ( 3.5 Logical Full-Text Operators Other full-text operators that are constraints on the positions of matches, such as indicators for distance between tokens and for the cardinality of matches ( 3.6 Positional Filters 3.3 Cardinality Selection The weighting information. Each individual search term in a full-text selection may be annotated with optional weight information. This information may be used during the evaluation of the full-text selections to calculate scoring, information that quantifies the relevance of the result to the given search criteria. An optional XPath 2.0 or XQuery 1.0 expression (UnionExpr) that specifies the set of nodes, descendents of the RangeExp, whose contents must be ignored for the purpose of determining a match during the search ( 3.7 Ignore Option The results of the evaluation of the full-text selection operators are instances of the AllMatches model, which complements the XQuery Data Model (XDM) for processing full-text queries. An AllMatches instance describes all possible solutions to the full-text query for a given search context item. Each solution is described by a Match instance. A Match instance contains the tokens from the search context that must be included (described using StringInclude instances which model the positive terms) and the tokens from search context item that must be excluded (described using StringExclude instances which model the negative terms). Each negative or positive term is modeled as a tuple: the position of the query token or phrase in the full-text selection, and a TokenInfo structure that describes a set of tokens in the text string which match the query token or phrase. Figure 1 provides a schematic overview of the XQuery and XPath Full Text 1.0 processing steps that are discussed in detail below. Some of these steps are completely outside the domain of XQuery; in Figure 1, these are depicted outside the black line that represents the boundaries of the language. The diagram only shows the central pieces of the XQuery Processing Model (see Section 2.2 Processing Model XQ Like all XQuery expressions, an FTContainsExpr returns an XDM Instance (see Fig. 1). With the exception of FTWords, which consumes TokenInfos, all full-text selections are closed under the AllMatches data model, i.e., their input and output are AllMatches instances. Tokenization transforms an XDM instance into TokenInfos, which ultimately get converted into AllMatches instances by the evaluation of full-text selections. Thus, the evaluation of nested full-text and XQuery expressions instances moves back and forth between these two models. The resulting AllMatches instance obtained by the evaluation of an FTContainsExpr is converted into a Boolean value before being returned to the enclosing XPath or XQuery operation as follows. If at least one member of the disjunction contains only positive terms then value returned is true. If all members of the disjunction contain negative terms the result is false. Weighting information, in an implementation-dependent Given the components of a given full-text contains expression, the evaluation algorithm will proceed according to the following steps, also referenced in the processing model diagram as steps FT n Evaluate the search context expression (resulting in the sequence of search context items), the ignore option, if any (resulting in the set of ignored nodes), and any other XQuery/XPath exprssions nested within the full-text contains expression. (FT1) Tokenize the query string(s). (FT2.1) For each search context item: Delete the ignored nodes from the search context item. Tokenize the result of the previous step. This produces a sequence of tokens. (FT2.2) Note that implementations may (as an optimization) perform tokenization as part of the External Processing that is described in the XQuery Processing Model, when an XML document is parsed into an Infoset/PSVI and ultimately into a XQuery Data Model instance. Evaluate the FTSelection against the tokens of the search context. (FT3, FT4) Convert the topmost AllMatches instances into a Boolean value. (FT5) The additional scoring information (also part of FT5) that is produced by the evaluation of the full-text contains expression is implementation-dependent (A more detailed version of the above procedure appears in Section 4.3 FTContainsExpr Section 3 Full-Text Selections 4 Semantics [ Definition full-text contains expression As a syntactic construct, a full-text contains expression (grammar symbol: FTContainsExpr Section 3.5.2 General Comparisons XQ FTContainsExpr ComparisonExpr ::= FTContainsExpr ValueComp GeneralComp NodeComp FTContainsExpr A full-text contains expression may be used anywhere a ComparisonExpr may be used. The contains text contains text FTContainsExpr ::= RangeExpr FTSelection FTIgnoreOption A full-text contains expression returns a Boolean value. It returns true if there is some item returned by the RangeExpr that, after tokenization FTSelection 3 Full-Text Selections FTIgnoreOption 3.7 Ignore Option An XQuery and XPath Full Text 1.0 processor SHOULD 3.4 Match Options The following example in XQuery 1.0 Full Text returns the author of each book with a title containing a token with the same root as dog cat for $b in /books/book where $b/title contains text ("dog" using stemming) ftand "cat" return $b/author The same example in XPath 2.0 Full Text is written as: /books/book[title contains text ("dog" using stemming) ftand "cat"]/author In the next example a ComparisonExpr is combined with an FTContainsExpr using the logical XQuery operator and train /books/book[price < 50 and title contains text ("train" using stemming)] The following example shows the combination of two contains text dog cat train /books/book[title contains text "dog" ftand "cat" ne content contains text ("train" using stemming)] Besides specifying a match of a full-text query as a Boolean condition, full-text query applications typically also have the ability to associate scores with the results. [ Definition score XQuery and XPath Full Text 1.0 extends the languages of XQuery 1.0 and XPath 2.0 further by adding optional score for let The production for the extended for ForClause ::= "for" "$" VarName TypeDeclaration PositionalVar FTScoreVar ExprSingle VarName TypeDeclaration PositionalVar FTScoreVar ExprSingle FTScoreVar ::= "score" "$" VarName In XPath 2.0, the SimpleForClause is extended similarly. When a score for in for score The scope of a score variable bound in a for or let clause comprises all subexpressions of the containing FLWOR expression that appear after the variable binding. The scope does not include the expression to which the variable is bound. The for and let clauses of a given FLWOR expression may bind the same score variable name more than once. In this case, each new binding occludes the previous one, which becomes inaccessible in the remainder of the FLWOR expression. The expanded QName of a score variable bound in a for clause must be distinct from both the expanded QName of the variable with which it is associated and the expanded QName of any positional variable with which it is associated [ err:XQST0089 XQ The semantics of scoring and how it relates to second-order functions is discussed in Section 4.4 Scoring In the following example book [content contains text "web site" ftand "usability" and .//chapter/title contains text "testing"] book for $b score $s in /books/book[content contains text "web site" ftand "usability" and .//chapter/title contains text "testing"] return $s The example above is also a valid example of the XPath 2.0 extension. Scores are typically used to order results, as in the following, more complete example. for $b score $s in /books/book[content contains text "web site" ftand "usability"] where $s > 0.5 order by $s descending return <result> <title> {$b//title} </title> <score> {$s} </score> </result> Note that the score variable gets one in para for $p score $s in //book[title contains text "software"]/para[. contains text "usability"] order by $s descending return $p The following more elaborate example uses multiple score variables to return the matching paragraphs ordered so that those from the highest scoring books precede those from the lowest scoring books, where the highest scoring paragraphs of each book are returned before the lower scoring paragraphs of that book: for $b score $score1 in //book[title contains text "software"] order by $score1 descending return for $p score $score2 in $b/para[. contains text "usability"] order by $score2 descending return $p The score implementation-dependent Score values are of type xs:double For score values greater than 0, a higher score must imply a higher degree of relevance Similarly to their use in a for let let let The production for the extended let LetClause ::= "let" (("$" VarName TypeDeclaration FTScoreVar ExprSingle VarName TypeDeclaration FTScoreVar ExprSingle When using the score option in a for in for let for $b in /books/book[.//chapter/title contains text "testing"] let score $s := $b/content contains text "web site" ftand "usability" order by $s descending return <result score="{$s}">{$b}</result> This example returns book book Note that it is not a requirement of the score of an FTContainsExpr to be 0, if the expression evaluates to false, nor to be non-zero, if the expression evaluates to true. Hence, in the example above it is not possible to infer the Boolean value of the FTContainsExpr in the let result The expression ExprSingle associated with the score variable is passed to the scoring algorithm. The scoring algorithm calculates the score value based on the passed expression (not on the value returned by evaluating the expression). The set of expressions supported by the scoring algorithm is implementation-defined The use of score let score $s := FTContainsExpr let $s := score(FTContainsExpr) where a function score score true false score [ Definition weight declarations 3.1.1 Weights The weights assigned are not related to any absolute standard, but typically have a relationship to other weights within the same FTContains expression. The effect of weights on the resulting score is implementation-dependent MUST that when no explicit weight is specified, the default weight is 1.0. The following example illustrates how different weights can be used for different search terms. for $b in /books/book let score $s := $b/content contains text ("web site" weight {0.5}) ftand ("usability" weight {2}) return <result score="{$s}">{$b}</result> The XQuery Static Context is extended with a component for each full-text match option group Prolog ::= (( DefaultNamespaceDecl Setter NamespaceDecl Import FTOptionDecl Separator VarDecl FunctionDecl OptionDecl Separator FTOptionDecl ::= "declare" "ft-option" FTMatchOptions Match options modify the match semantics of full-text expressions. They are described in detail in Section 3.4 Match Options This section describes the full-text selections which contain the full-text operators in a full-text contains expression FTContainsExpr [ Definition full-text selection FTSelection ::= FTOr FTPosFilter As shown in the grammar, a full-text selection consists of search conditions possibly involving logical operators ( FTOr FTPosFilter The syntax and semantics of the individual full-text selection operators follow. This XML document is the source document for examples in this section. <books> <book number="1"> <title shortTitle="Improving Web Site Usability">Improving the Usability of a Web Site Through Expert Reviews and Usability Testing</title> <author>Millicent Marigold</author> <author>Montana Marigold</author> <editor>Véra Tudor-Medina</editor> <content> <p>The usability of a Web site is how well the site supports the users in achieving specified goals. A Web site should facilitate learning, and enable efficient and effective task completion, while propagating few errors. </p> <note>This book has been approved by the Web Site Users Association. </note> </content> </book> </books> Tokenization is implementation-defined , periods followed by a space as sentence boundaries, <p> The first sentence and paragraph start at the beginning of the document, and the last sentence and paragraph end at the end of the document. The first five tokens in this example using the sample tokenization would be "Improving", "the", "usability", "of", and "a". Unless stated otherwise, the results assume a case-insensitive match. FTPrimary ::= ( FTWords FTTimes FTSelection FTExtensionSelection [ Definition primary full-text selection FTWords FTTimes FTSelection FTExtensionSelection FTPrimaryWithOptions ::= FTPrimary FTMatchOptions FTWeight FTWeight ::= "weight" "{" Expr As shown in the grammar, a full-text primary selection may be optionally followed by match options (which are discussed in 3.4 Match Options xs:double MUST err:FTDY0016 Note: As a consequence of the flexibility given to implementations under Section 2.3.4 Errors and Optimization XQ FTWords ::= FTWordsValue FTAnyallOption FTWordsValue ::= StringLiteral Expr FTAnyallOption ::= ("any" "word"?) | ("all" "words"?) | "phrase" FTWords FTWords consists of two parts: a mandatory FTWordsValue FTAnyallOption FTWordsValue FTAnyallOption In general, the tokens and phrases in FTWordsValue FTWordsValue string The following rules specify how an FTWordsValue FTWordsValue xs:string* Section 4.1 Tokenization FTAnyallOption If FTAnyallOption If FTAnyallOption If FTAnyallOption If FTAnyallOption If FTAnyallOption If the FTWordsValue FTAnyallOption If FTAnyallOption The following expression returns the sample book title //book[./title contains text "Expert"] The following expression returns the sample book title //book[./title contains text "Expert Reviews"] The following expression returns the sample book title //book[./title contains text {"Expert", "Reviews"} all] The following expression returns false for our sample document, because the p //book//p contains text "Web Site Usability" The following expression returns book numbers of book for $book in /books/book[.//author contains text "Marigold"] let score $score := $book/title/@shortTitle contains text "Web Site Usability" where $score > 0.8 order by $score descending return $book/@number FTTimes ::= "occurs" FTRange [ Definition cardinality selection FTWords FTTimes FTWords A cardinality selection limits the number of different matches of FTWords 3.6.3 Distance Selection In the document fragment "very very big": The FTWords "very big" The FTWords {"very", "big"} all The FTWords {"very", "big"} any The following expression returns the example book book //book[. contains text "usability" occurs at least 2 times]/@number The following expression returns the empty sequence, because there are 3 occurrences of {"usability", "testing"} any title //book[@number="1" and title contains text {"usability", "testing"} any occurs at most 2 times] Full-text match options modify the matching behaviour of the primary full-text selection FTPrimaryWithOptions ::= FTPrimary FTMatchOptions FTWeight FTMatchOptions ::= ("using" FTMatchOption FTMatchOption ::= FTLanguageOption FTWildCardOption FTThesaurusOption FTStemOption FTCaseOption FTDiacriticsOption FTStopWordOption FTExtensionOption [ Definition Match options [ Definition FTMatchOption other than FTExtensionOption match option group It is a static error [ err:FTST0019 FTMatchOptions match option group FTCaseOption FTMatchOptions Although match options only take effect in the application of FTWords "(" FTSelection ")" FTPrimary match option declarations Prolog Match options are propagated through the query via the static context. For each of the seven match option groups, the static context has a component that contains one option from that group. The seven settings are initialized by the implementation in accordance with the table in Appendix C Static Context Components FTOptionDecl Prolog FTContainsExpr VarDecl FunctionDecl FTContainsExpr FTContainsExpr FTContainsExpr FTPrimaryWithOptions FTWords 4 Semantics Thus, when a match option appears in an FTSelection FTPrimary FTContainsExpr FTPrimary FTContainsExpr Prolog Prolog An FTMatchOption FTPrimary FTWords FTTimes FTExtensionSelection FTSelection [ Definition FTWords match option application order The match option application order is subject to some constraints: The Language Option must be applied first The Stemming Option must be applied before the Case Option and the Diacritics Option Aside from these constraints, the full order of the application of match options is implementation-defined More information on their semantics is given in 4.2.5 Match Options Semantics If no match options declarations are present in the prolog and the implementation does not define any overwriting of the static context components for the match options, the query: /books/book/title contains text "usability" is, assuming "de" is the implementation-defined /books/book/title contains text "usability" using language "de" using no wildcards using no thesaurus using no stemming using case insensitive using diacritics insensitive using no stop words We describe each match option group in more detail in the following sections. FTLanguageOption ::= "language" StringLiteral [ Definition language option The StringLiteral following the keyword language xs:language err:XPTY0004 XP The "language" option influences tokenization, stemming, and stop words in an implementation-defined MAY implementation-defined The set of standardized language identifiers is defined in [BCP 47] implementation-defined MAY [BCP 47] implementation-defined implementation-defined err:FTST0009 The default language is specified in the static context. When an XQuery and XPath Full Text processor evaluates text in a document that is governed by an xml:lang attribute and the portion of the full-text query doing that evaluation contains an FTLanguageOption that specifies a different language from the language specified by the governing xml:lang attribute, the language-related behavior of that full-text query is implementation-defined This is an example where the language option is used to select the appropriate stop word list: //book[@number="1"]/content//p contains text "salon de thé" using stop words default using language "fr" FTWildCardOption ::= "wildcards" | ("no" "wildcards") [ Definition wildcard option When the "wildcards" option is used, wildcard syntax may be included within query strings. A wildcard consists of an indicator (a period or full stop, "."), optionally followed by a qualifier. Each wildcard in a query token will match zero or more characters within a token in the text being searched, as described below. The number of characters that can be matched depends on the qualifier. The forms of wildcard syntax specified by this document are: A single period, without any qualifiers: Matches a single arbitrary character. A period immediately followed by a single question mark, "?": Matches either no characters or one character. A period immediately followed by a single asterisk, "*": Matches zero or more characters. A period immediately followed by a single plus sign, "+": Matches one or more characters. A period immediately followed by a sequence of characters that matches the regular expression {[0-9]+,[0-9]+} If a period in the query string is immediately followed by a left curly brace, but the subsequent characters do not conform to the given regular expression, then an error is raised: [ err:FTDY0020 A question mark, asterisk, plus sign, or left curly brace that is not immediately preceded by a period is not treated as a qualifier. For example, using the sample tokenization and "wildcards", the query string "wil+" does not match the search text "will" or "willlllll", but only matches the search text "wil". (The sample tokenization treats the plus sign as punctuation.) When "wildcards" is used, any character in a query string can be "escaped" by immediately preceding it with a backslash, "\". That is, a backslash immediately followed by any character represents that character literally, preventing any special interpretation that the "wildcards" option might otherwise attach to it. In particular: Escaping a period prevents its interpretation as a wildcard. Escaping a question mark, asterisk, plus sign, or left curly brace ensures that it is not interpreted as a qualifier. An escaped backslash ("\\") represents a literal backslash. If a query string is terminated by an unescaped backslash, an error is raised: [ err:FTDY0020 Note: A query string of the form "abc\"xyz" not err:XPST0003 XP When the "no wildcards" option is used, no wildcards are recognized in query strings. Periods, question marks, asterisks, plus signs, left curly braces, and backslashes are always recognized as ordinary text characters. The default is "no wildcards". The following expression returns true, because the p //book[@number="1"]/p contains text "w.ll" using wildcards The following expression returns true, because the title //book[@number="1"]/title contains text ".?site" using wildcards The following expression returns true, because the title //book[@number="1"]/title contains text "improv.*" using wildcards The following expression raises error [ err:FTDY0020 //book[@number="1"]/p contains text "wi.{5,7]" using wildcards The following expression returns true, because the title contains "site": //book[@number="1"]/title contains text "\s\i\t\e" using wildcards The following expression returns true, because the title contains "Usability": //book[@number="1"]/title contains text "Usab.+\\" using wildcards (Note that "\\" represents a literal backslash, which the sample tokenization treats as punctuation.) The following expression raises error [ err:FTDY0020 //book[@number="1"]/p contains text "will\" using wildcards The following expression returns false, because the p //book[@number="1"]/p contains text "w.ll" using no wildcards (Note that, without wildcards, the sample tokenization will treat the period in "w.ll" as punctuation, thus producing "w" and "ll" as separate tokens.) FTThesaurusOption ::= ("thesaurus" ( FTThesaurusID FTThesaurusID FTThesaurusID FTThesaurusID ::= "at" URILiteral StringLiteral FTLiteralRange URILiteral ::= StringLiteral FTLiteralRange ::= ("exactly" IntegerLiteral IntegerLiteral IntegerLiteral IntegerLiteral IntegerLiteral [ Definition thesaurus option If the thesaurus option specifies a thesaurus with a relative URI, that relative URI is resolved to an absolute URI using the base URI in the static context and that absolute URI is used to identify the thesaurus. If the URI specifies a thesaurus that is not found in the statically known thesauri, an error is raised [ err:FTST0018 Thesauri add related tokens and phrases to the query or change query tokens. Thus, the user may narrow, broaden, or otherwise modify the query using synonyms, hypernyms (more generic terms), etc. The search is performed as though the user has specified all related query tokens and phrases in a disjunction (FTOr). Note: A thesaurus may be standards-based or locally-defined. It may be a traditional thesaurus, or a taxonomy, soundex, ontology, or topic map. How the thesaurus is represented is implementation-dependent An FTThesaurusID may optionally contain a StringLiteral to specify [ISO 2788] implementation-defined SHOULD [ISO 2788] [ISO 2788] implementation-defined equivalence relationships (synonyms): hierarchical relationships: associative relationships: An FTThesaurusID may also optionally include an FTLiteralRange to specify the number of levels to be queried in hierarchical relationships. An FTLiteralRange is a constrained form of FTRange Note: For historical reasons, an implementation MAY MAY The effect of specifying a particular range of levels in an FTThesaurusID is implementation-defined If no levels are specified, the default is to query all levels in hierarchical relationships or to query an implementation-defined The "thesaurus" option specifies that string matches include tokens that can be found in one of the specified thesauri. When "default" is used in place of a FTThesaurusID, the thesauri specified in the static context are used, which are either given by the prolog declaration for the thesaurus option, or, if no such declaration exists a system-defined default thesaurus with a system-defined relationship. The default thesaurus may be used in combination with other explicitly specified thesauri. The "no thesaurus" option specifies that no thesaurus will be used. The default is "no thesaurus". The following expression returns true, because it finds a content .//book/content contains text "duty" using thesaurus at "http://bstore1.example.com/UsabilityThesaurus.xml" relationship "UF" The following expression returns a book content doc("http://bstore1.example.com/full-text.xml") /books/book[./content contains text "people" using thesaurus at "http://bstore1.example.com/UsabilityThesaurus.xml" relationship "NT" at most 2 levels] Assuming the thesaurus available at URL "http://bstore1.example.com/UsabilitySoundex.xml" contains soundex capabilities, the following query returns a book doc("http://bstore1.example.com/full-text.xml") /books/book[. contains text "Merrygould" using thesaurus at "http://bstore1.example.com/UsabilitySoundex.xml" relationship "sounds like"] FTStemOption ::= "stemming" | ("no" "stemming") [ Definition stemming option The "stemming" option specifies that matches may contain tokens that have the same stem as the tokens and phrases written in the query. It is implementation-defined The "no stemming" option specifies that the tokens and phrases are not stemmed. It is implementation-defined The default is "no stemming". The following expression returns true, because the title book /books/book[@number="1"]/title contains text "improve" using stemming FTCaseOption ::= ("case" "insensitive") [ Definition case option There are four possible character case options: Using the option "case insensitive", tokens and phrases are matched, regardless of the case of characters of the query tokens and phrases. Using the option "case sensitive", tokens and phrases are matched, if and only if the case of their characters is the same as written in the query. Using the option "lowercase", tokens and phrases are matched, if and only if they match the query without regard to character case, but contain only lowercase characters. Using the option "uppercase", tokens and phrases are matched, if and only if they match the query without regard to character case, but contain only uppercase characters. The default is "case insensitive". The effect of the case options is also influenced by the query's default collation (see Section 2.1.1 Static Context XQ Section 4.4 Default Collation Declaration XQ Case Matrix Case option \ Default collation UCC (Unicode Codepoint Collation) CCS (some generic case-sensitive collation) CCI (some generic case-insensitive collation) case insensitive compare as if both lower case-insensitive variant of CCS if it exists, else error CCI case sensitive UCC CCS case-sensitive variant of CCI if it exists, else error lowercase compare using UCC after applying fn:lower-case() to the query string compare using CCS after applying fn:lower-case() to the query string CCI uppercase compare using UCC after applying fn:upper-case() to the query string compare using CCS after applying fn:upper-case() to the query string CCI Note: In this table, "else error" means "Otherwise, an error is raised: [ err:FOCH0002 FO The following expression returns false, because the title //book[@number="1"]/title contains text "Usability" using lowercase The following expression returns true, because the character case is not considered: //book[@number="1"]/title contains text "usability" using case insensitive FTDiacriticsOption ::= ("diacritics" "insensitive") [ Definition diacritics option There are two possible diacritics options: The option "diacritics" "insensitive" matches tokens and phrases with and without diacritics. Whether diacritics are written in the query or not is not considered. The option "diacritics" "sensitive" matches tokens and phrases only if they contain the diacritics as they are written in the query. The default is "diacritics insensitive". The effect of the diacritics options is also influenced by the query's default collation (see Section 2.1.1 Static Context XQ Section 4.4 Default Collation Declaration XQ Diacritics Matrix Diacritics option \ Default collation UCC (Unicode Codepoint Collation) CDS (some generic diacritics-sensitive collation) CDI (some generic diacritics-insensitive collation) diacritics insensitive UCC comparison, but without considering diacritics diacritics-insensitive variant of CDS if it exists, else error CDI diacritics sensitive UCC CDS diacritics-sensitive variant of CDI if it exists, else error Note: In this table, "else error" means "Otherwise, an error is raised: [ err:FOCH0002 FO The following expression returns true, because the token "Véra" in the editor //book[@number="1"]//editor contains text "Vera" using diacritics insensitive This returns false, because the editor //book[@number="1"]/editors contains text "Vera" using diacritics sensitive FTStopWordOption ::= ("stop" "words" FTStopWords FTStopWordsInclExcl FTStopWordsInclExcl FTStopWords ::= ("at" URILiteral StringLiteral StringLiteral FTStopWordsInclExcl ::= ("union" | "except") FTStopWords [ Definition stop word option tokens More precisely, a stop word option defines a collection of stop words according to the rules below. Then, in every FTWords to which the stop word option applies, each query token is checked: if it appears (using an implementation-defined Normally a stop word matches exactly one token, but there may be implementation-defined Tokens matched by stop words retain their position numbers and are counted by FTDistance FTWindow FTStopWords at err:FTST0008 If the stop words option specifies a stop word list with a relative URI, that relative URI is resolved to an absolute URI using the base URI in the static context and that absolute URI is used to identify the stop word list. Multiple stop word lists may be combined using "union" or "except". The keywords "union" and "except" are applied from left to right. If "union" is specified, every string occurring in the lists specified by the left-hand side or the right-hand side is a stop word. If "except" is specified, only strings occurring in the list specified by the left-hand side but not in the list specified by the right-hand side are stop words. The "stop words default" implementation-defined The "no stop words" option specifies that no stop words are used. This is equivalent to specifying an empty list of stop words. The default is "no stop words". Note: Some implementations may apply stop word lists during indexing and be unable to comply with query-time requests to not apply those stop words. An implementation may still support stop-word options (and therefore not raise [ err:FTST0006 The following expression returns true, because the document contains the phrase "propagating few errors": /books/book[@number="1"]//p contains text "propagating of errors" using stop words ("a", "the", "of") Note the asymmetry in the stop word semantics: the property of being a stop word is only relevant to query terms, not to document terms. Hence, it is irrelevant for the above-mentioned match whether "few" is a stop word or not, and on the other hand we do not want the query above to match "propagating" followed by 2 stop words, or even a sequence of 3 stop words in the document. The following expression returns false. In this case specifying "few" as a stop word has no effect, since "few" does not appear in the query. Although the words "propagating" and "errors" appear in the text being searched, the phrase "propagating errors" cannot be matched, since that phrase does not occur. /books/book[@number="1"]//p contains text "propagating errors" using stop words ("few") The following expression returns false, because "of" is not in the p /books/book[@number="1"]//p contains text "propagating of errors" using no stop words The following expression uses the stop words list specified at the URL. Assuming that the specified stop word list contains the word "then", this query is reduced to a query on the phrase "planning X conducting", allowing any token as a substitute for X. It returns a book content book content doc("http://bstore1.example.com/full-text.xml") /books/book[.//content contains text "planning then conducting" using stop words at "http://bstore1.example.com/StopWordList.xml"] The following expression returns book book doc("http://bstore1.example.com/full-text.xml") /books/book[.//content contains text "planning then conducting" using stop words at "http://bstore1.example.com/StopWordList.xml" except ("the", "then")] [ Definition extension option implementation-defined FTExtensionOption ::= "option" QName StringLiteral An extension option consists of an identifying QName and a StringLiteral. Typically, a particular option will be recognized by some implementations and not by others. The syntax is designed so that option declarations can be successfully parsed by all implementations. The QName of an extension option must resolve to a namespace URI and local name, using the statically known namespaces. Note: There is no default namespace for options. Each implementation recognizes an implementation-defined If the namespace part of the QName is not a namespace recognized by the implementation as one used to denote extension option, then the extension option is ignored. Otherwise, the effect of the extension option, including its error behavior, is implementation-defined Implementations may impose rules on where particular extension options may appear relative to other match options, and the interpretation of an option declaration may depend on its position. An extension option must not be used to change the syntax accepted by the processor, or to suppress the detection of static errors. However, it may be used without restriction to modify the set of tokens in the query or how they are matched against tokens in the text being searched. An extension option has the same scope as other match options. The following examples illustrate several possible uses for extension options: This extension option is set as part of the static context of all full-text expressions in the module and might be used to ensure that queries are insensitive to Arabic short-vowels. declare namespace exq = "http://example.org/XQueryImplementation";

declare ft-option using option exq:diacritics "short-vowel insensitive"; This extension option applies only to the matching in the full-text selection in which it is found and might be used to specify how compound words should be matched. declare namespace exq = "http://example.org/XQueryImplementation";

//para[. contains text ("Kinder" ftand "Platz" distance exactly 1 words) using stemming using option exq:compounds "distance=1" ] Full-text selections can be combined with the logical connectives ftor ftand not in ftnot FTOr ::= FTAnd FTAnd FTAnd ::= FTMildNot FTMildNot FTMildNot ::= FTUnaryNot FTUnaryNot FTUnaryNot ::= ("ftnot")? FTPrimaryWithOptions [ Definition or-selection ftor An or-selection finds all matches that satisfy at least one of the operand full-text selections. The following expression returns the book //book[.//author contains text "Millicent" ftor "Voltaire"] [ Definition and-selection ftand An and-selection finds matches that satisfy all of the operand full-text selections simultaneously. A match of an and-selection is formed by combining matches for each of the operand full-text selections as described in 4.2.6.2 FTAnd For example, "usability" ftand "testing" //book[@number="1"]/title "usability" "testing" //book[@number="1"]/title contains text ("usability" ftand "testing") The following expression returns false, because "Millicent" and "Montana" are not contained by the same author book //book/author contains text "Millicent" ftand "Montana" No author book author "Millicent" "Montana" [ Definition mild-not selection not in The not in ftand ftnot A not in B A B A ftand ftnot B A B As an example, consider a search for "Mexico" not in "New Mexico" "Mexico" ftand ftnot "New Mexico" A match to a mild-not selection must contain at least one token that satisfies the first condition and does not satisfy the second condition. If it contains a token that satisfies both the first and the second condition, the token is not considered as a match. The following expression returns true, because "usability" appears in the title p title /books/book contains text "usability" not in "usability testing" If either operand of a mild-not selection yields an AllMatches Match StringExclude err:FTDY0017 Note: This situation can arise if the operand contains a not-selection or a cardinality constraint ( FTTimes exactly at most from ... to [ Definition not-selection ftnot A not-selection selects matches that do not satisfy the operand full-text selection. Details about how such matches are constructed are given in 4.2.6.3 FTUnaryNot The following expression returns the empty sequence, because all book //book[. contains text ftnot "usability"] The following expression returns true, because book //book contains text "improving" ftand "usability" ftand ftnot "improving usability" The following expression returns book //book[title/@shortTitle contains text "web site usability" ftand ftnot "usability testing"] FTPosFilter ::= FTOrder FTWindow FTDistance FTScope FTContent [ Definition Positional filters Recall that the grammar rule for FTSelection FTOr In a group of multiple adjacent positional filters, FTOrder FTOrder FTOr An FTOr FTAnd ftor FTPosFilter FTOrder ::= "ordered" [ Definition ordered selection The default is unordered. Unordered is in effect when ordered is not specified in the query. Unordered cannot be written explicitly in the query. An ordered selection selects matches which satisfy the operand full-text selection and which also satisfy the following constraint: the order that the matching tokens or phrases have in the text being searched is the same order that the corresponding query tokens or phrases have in the operand selection. In both cases, the ordering is determined from the minimum start positions of the constituent tokens. The following expression returns true, because titles of book //book/title contains text ("web site" ftand "usability") ordered The following expression returns false, because although "Montana" and "Millicent" both appear in the book //book[@number="1"] contains text ("Montana" ftand "Millicent") ordered FTWindow ::= "window" AdditiveExpr FTUnit FTUnit ::= "words" | "sentences" | "paragraphs" [ Definition window selection FTWindow FTUnit FTUnit xs:integer A window selection may cross element boundaries. The size of the window is not affected by the presence or absence of element boundaries. Stop words are included in the computation of the window size whether they are ignored by the query or not. A window selection examines the matches generated by the preceding portion of the FTSelection The following expression returns true, because "web", "site", and "usability" are within a window of 5 tokens in the title /books/book/title contains text "web" ftand "site" ftand "usability" window 5 words The following expression returns true, because "web" and "site" in the order they are written in the query and either "usability" or "testing" are within a window of at most 10 tokens: /books/book contains text ("web" ftand "site" ordered) ftand ("usability" ftor "testing") window 10 words The following expression returns false, because the instances of "web site" and "usability" in the title /books/book//title contains text "web site" ftand "usability" window 3 words The following expression returns the sample book number /books/book[@number="1" and . contains text "efficient" ftand ftnot "and" window 2 words] The following expression returns the empty sequence, because in the selected book /books/book[@number="1" and . contains text "efficient" ftand ftnot "and" window 3 words] In order to allow meaningful results for nested positional filters, e.g., a window selection embedded inside a distance selection, the resulting matches for window selections are formed from the input matches that satisfy the window constraint as follows. All StringIncludes of such a match are coerced into a single StringInclude that spans all token positions from the smallest to the largest position of any input StringIncludes. This is explained in more detail in Section 3.6.3 Distance Selection FTDistance ::= "distance" FTRange FTUnit FTRange ::= ("exactly" AdditiveExpr AdditiveExpr AdditiveExpr AdditiveExpr AdditiveExpr [ Definition distance selection FTDistance A distance selection selects matches which satisfy the operand full-text selection and for which the matched tokens and phrases satisfy the specified distance conditions. Distances in the search context are measured in units of tokens, sentences, or paragraphs. Roughly speaking, the distance between two matches is the number of intervening units, so a distance of zero tokens (sentences, paragraphs) means no intervening tokens (sentences, paragraphs). More precisely, given two matches, we first determine their order by sorting on starting position and if necessary on ending position. Let M1 be the "earlier" and M2 be the "later". (If there are overlapping tokens involved, the designations "earlier" and "later" may not be intuitively obvious.) Then the distance between the two is M2's starting position minus M1's ending position, minus 1. When computing distances in the search context, a distance selection may cross element boundaries; they affect the distance computed only to the extent that they affect the tokenization of the search context. Stop words are counted in those computations whether they are ignored or not. When a distance selection applies a distance condition to more than two matches, the distance condition is required to hold on each successive pair of matches. An FTDistance FTUnit FTRange FTUnit words sentences paragraphs words An FTRange FTRange xs:integer Let the value of the first (or only) operand be M. If "from" is specified, let the value of the second operand be N. If "exactly" is specified, then the range is the closed interval [M, M]. If "at least" is specified, then the range is the half-closed interval [M, unbounded). If "at most" is specified, then the range is the half-closed interval (unbounded, M]. If "from-to" is specified, then the range is the closed interval [M, N]. Note: If M is greater than N, the range is empty. Here are some examples of FTRange 'exactly 0' specifies the range [0, 0]. 'at least 1' specifies the range [1,unbounded). 'at most 1' specifies the range (unbounded, 1]. 'from 5 to 10' specifies the range [5, 10]. The following expression returns false, because "completion" and "errors" are less than 11 tokens apart: /books/book contains text ("completion" ftand "errors" distance at least 11 words) The following expression returns true: /books/book contains text "web" ftand "site" ftand "usability" distance at most 2 words The search context contains two occurrences of the phrase "the usability of a web site" (once in the <title> and once in the <content>). In this phrase, the tokens "usability" and "web" have a distance of 2 words, and the tokens "web" and "site" have a distance of 0 words, both of which satisfy the constraint distance at most 2 words The following expression returns the empty sequence, because between any token "usability" and the token in any occurrence of the phrase "web site" that is the nearest to the token "usability" there is always more than one intervening token: /books/book[.//p contains text "web site" ftand "usability" distance at most 1 words] The following expression returns the book note /books/book[. contains text "web" ftand "users" distance at most 1 words]/title In order to allow meaningful results for nested positional filters, e.g., a distance selection embedded inside another distance selection, the resulting matches for distance selections are formed from the input matches that satisfy the distance constraint as follows. All StringIncludes of such a match are coerced into a single StringInclude that spans all token positions from the smallest to the largest position of any input StringIncludes. Thus, a distance selection that embeds a window or a distance selection takes the result of the embedded selection as a single unit. The following gives an example of nested distance selections: /books/book contains text ((("richard" ftand "nixon") distance at most 2 words) ftand (("george" ftand "bush") distance at most 2 words) distance at least 20 words) This expression allows to find book FTScope ::= ("same" | "different") FTBigUnit FTBigUnit ::= "sentence" | "paragraph" [ Definition scope selection FTScope A scope selection selects matches which satisfy the operand full-text selection and for which the matched tokens and phrases are contained in the same scope or in different scopes. Possible scopes are sentences and paragraphs. By default, there are no restrictions on the scope of the matches. The following expression returns false, because the tokens "usability" and "Marigold" are not contained within the same sentence: //book contains text "usability" ftand "Marigold" same sentence The following expression returns true, because the tokens "usability" and "Marigold" are contained within different sentences: //book contains text "usability" ftand "Marigold" different sentence The following expression returns a book //book[. contains text "usability" ftand "testing" same paragraph] The following expression returns a book //book[. contains text "site" ftand "errors" same sentence] It is possible that both "same sentence" and "different sentence" conditions are simultaneously safisfied for several tokens and/or phrases within the same document fragment. This can be observed if there are occurrences of the tokens and/or phrases both within the same sentence and within difference sentences. For example, consider the following document fragment. <introduction> ... The usability of a Web site is how well the site supports the user in achieving specified goals. ... Expert reviews and usability testing are methods of identifying problems in layout, terminology, and navigation. ... </introduction> This sample will satisfy both conditions ("usability" ftand "reviews") different sentence ("usability" ftand "reviews") same sentence The above observation also holds for the "same paragraph" and "different paragraph" conditions. FTContent ::= ("at" "start") | ("at" "end") | ("entire" "content") [ Definition anchoring selection An anchoring selection selects matches which satisfy the operand full-text selection and for which the matched tokens and phrases are the first, last, or all tokens in the tokenized form of the items being searched. Using the "at start" operator, tokens or phrases are matched, if they cover the first token position in the tokenized string value of the item being searched. Using the "at end" operator, tokens or phrases are matched, if they cover the last token position in the tokenized string value of the item being searched. Using the "entire content" operator, tokens or phrases are matched, if they cover all token positions of the tokenized string value of the item being searched. The following expression returns each title /books//title[. contains text "improving the usability of a web site" at start] The following expression returns the p /books//p[. contains text "propagat.*" using wildcards ftand "few errors" distance at most 2 words at end] Since the distance operator doesn't imply an ordering, the last example would also yield a match if the p The following expression returns each note /books//note[. contains text "this book has been approved by the web site users association" entire content] The following example returns true because both the content note /books//* contains text "Association" at end FTIgnoreOption ::= "without" "content" UnionExpr The ignore option FTSelection FTContainsExpr Definition Ignored nodes err:XPTY0004 XP Let I1, I2, ..., In N1, N2, ..., Nk Ij (j=1..n) Ni (i=1..k) In the following fragment, if $x//annotation title editor annotation annotation let $x := <book> <title>Web Usability and Practice</title> <author>Montana <annotation> this author is an expert in Web Usability</annotation> Marigold </author> <editor>Véra Tudor-Medina on Web <annotation> best editor on Web Usability</annotation> Usability </editor> </book> By default, no element content is ignored. Note: Nodes MAY [ Definition extension selection implementation-defined FTExtensionSelection ::= Pragma FTSelection Pragma ::= "(#" S QName S PragmaContents PragmaContents ::= ( Char An extension selection consists of one or more pragmas followed by a full-text selection enclosed in curly braces. See Section 3.14 Extension Expressions XQ (# #) implementation-defined #) Note: Since there is no default namespace for pragmas, a pragma QName must include a namespace prefix. Each implementation recognizes an implementation-defined If the namespace part of a pragma QName is not recognized by the implementation as a pragma namespace, then the pragma is ignored. If all the pragmas in an FTExtensionSelection err:XQST0079 XQ If an implementation recognizes the namespace of one or more pragmas in an FTExtensionSelection FTExtensionSelection implementation-defined It is a static error [ err:XQST0013 XQ If an implementation recognizes a pragma, it must report any static errors in the following full-text selection even if it will not apply that selection. The following examples illustrate three ways in which extension selections might be used. A pragma can be used to furnish a hint for how to evaluate the following full-text selection, without actually changing the result. For example: declare namespace exq = "http://example.org/XQueryImplementation";

/books/book/author[name contains text (# exq:use-index #) {'Berners-Lee'}] An implementation that recognizes the exq:use-index A pragma might be used to modify the semantics of the following full-text selection in ways that would not (in the absence of the pragma) be conformant with this specification. For example, a pragma might be used to change distance counting so that adjacent words are at a distance of 1 (otherwise they would be at a distance of 0): declare namespace exq = "http://example.org/XQueryImplementation";

/books/book[.//p contains text (# exq:distance #) { "web site" ftand "usability" distance at most 1 words }] Such changes to the language semantics must be scoped to the expression contained within the curly braces following the pragma. A pragma might contain syntactic constructs that are evaluated in place of the following full-text selection. In this case, the following selection itself (if it is present) provides a fallback for use by implementations that do not recognize the pragma. For example: declare namespace exq = "http://example.org/XQueryImplementation";

//city[. contains text (# exq:classifier with class 'Animals' #) {"animal" using thesaurus at "http://example.org/thesaurus.xml" relationship "RT"}] Here an implementation that recognizes the pragma will return the result of evaluating the proprietary syntax with class 'animals' This section describes the formal semantics of XQuery and XPath Full Text 1.0. The figure below shows how XQuery and XPath Full Text 1.0 integrates with XQuery 1.0 and XPath 2.0. The following diagram represents the interaction of XQuery and XPath Full Text 1.0 with the rest of XQuery 1.0 and XPath 2.0. It illustrates how full-text expressions can be nested within XQuery 1.0 and XPath 2.0 expressions and vice versa. Step 1 represents the composability of XQuery 1.0 and XPath 2.0 expressions and the fact that such expressions evaluate to a sequence of XDM items. This process is outside the scope of this document and will not be discussed further. Step 2 shows how XQuery 1.0 and XPath 2.0 expressions can be nested within full-text expressions. If an XQuery 1.0 and XPath 2.0 expression is nested on the left-hand side of an FTContains FTWords Tokenization FTSelection FTSelections Step 3 represents the composability of FTSelection FTSelection AllMatches AllMatches Evaluation of FTSelections Step 4 shows how XQuery and XPath Full Text 1.0 and scoring expressions can be nested into XQuery 1.0 and XPath 2.0 expressions. The sections 4.3 FTContainsExpr 4.4 Scoring Note: In the list above and throughout the rest of this section, bold typeface has been used to distinguish the concepts that are part of the AllMatches The functions and schemas defined in this section are considered to be within the fts: namespace (as discussed in section 1.3 A word about namespaces Note that by using XQuery 1.0 and XPath 2.0 to specify the formal semantics, we avoid the need to introduce new formalism. We simply reuse the formal semantics of XQuery 1.0 and XPath 2.0. [ Definition tokenization Tokenization, including the definition of the term "token", SHOULD implementation-defined SHOULD MUST Each token MUST Tokenization of an item MUST [XQuery 1.0 and XPath 2.0 Data Model (XDM) (Second Edition)] Section 2.6.5 String Values DM The tokenizer SHOULD implementation-defined The starting and ending position of a token MUST MUST In the tokenization of an item, consider the range of token positions from the smallest starting position to the largest ending position; every token position in that range must be covered by some token in the tokenization. That is, for every token position P T T's starting position <= P <= T's ending position The tokenizer MUST Each token is contained in at most one sentence and at most one paragraph. (In particular, this means that no tokens of any sentence are contained in any other sentence, and no tokens of any paragraph are contained in any other paragraph.) All tokens of a sentence are contained in at most one paragraph. The range of token positions from the smallest starting position to the largest ending position in a sentence does not overlap with the token position range from any other sentence. The range of token positions from the smallest starting position to the largest ending position in a paragraph does not overlap with the token position range from any other paragraph. Useful information for tokenizer implementors may be found in [UAX29] Note: Usually, the starting and ending positions of a token are the same. For some languages, some tokenizers may identify overlapping tokens. For example, the German word "Donaudampfschifffahrtskapitaensmuetze" might be tokenized into the following tokens: "Donaudampfschifffahrtskapitaensmuetze", "Donau", "dampf", "schiff", "dampfschiff", "kapitaen", "muetze", "kapitaensmuetze", "schifffahrt", "dampfschifffahrt", and perhaps others. In the face of overlapping tokens, it is implementation-dependent what positions a tokenizer assigns to each such token. For example, a tokenizer might assign the same position value to each of the tokens "Donaudampfschifffahrtskapitaensmuetze", "Donau", "dampf", "schiff", "dampfshiff", etc. In that case, the distance between each (overlapping) token assigned the same position is -1. Tokenizers might retain additional information about those overlapping tokens that allows the full-text implementation to distinguish among them. Consider the sentence "Ich sehe den Dampfschifffahrtskapitän auf dem Fluß." If an implementation tokenizes "Dampfschifffahrtskapitän" as overlapping tokens at the same position, then the implementation could still determine that the query "'Schifffahrt Dampf' window 0 words ordered" fails to match the sentence because phrase matching is implementation-defined and may make use of additional implementation-dependent token information. Even more complex situations can arise. Consider, for example, the German sentence "Er stellte sie vor." A sophisticated tokenizer might construct the token "vorstellen" covering positions 2 through 4, which overlaps the token "sie" at position 3. For the purposes of distance calculations, tokens are considered in the order of their starting positions, so the distance between "vorstellen" and "sie" would be 3-4-1=-2. (See fts:wordDistance For example, the following example must return false, because the 'secret' only occurs within an attribute and a comment, neither of which contributes characters to the string value of the 'p' element node: <p kind='secret'>Sensitive material <!-- secret --></p> contains text 'secret' The following document may lead to overlapping tokens to account for the ambiguity caused by the hyphen: <p>I will re- sign tommorow.</p> The following document fragment is the source document for examples in this section. A sample tokenization is used for the examples in this section. The results might be different for other tokenizations. Unless stated otherwise, the results assume a case-insensitive match. <offers> <offer id="1000" price="10000"> Ford Mustang 2000, 65K, excellent condition, runs great, AC, CC, power all </offer> <offer id="1001" price="8000"> Honda Accord 1999, 78K, A/C, cruise control, runs and looks great, excellent condition </offer> <offer id="1005" price="5500"> Ford Mustang, 1995, 150K highway mileage, no rust, excellent condition </offer> </offers> In this sample tokenization, tokens are delimited by punctuation and whitespace symbols. The token "Ford" is at relative position 1. The token "Mustang" is at relative position 2. The token "2000" is at relative position 3. Relative position numbers are assigned sequentially through the end of the document. Hence in this example each token occupies exactly one position, and no overlapping of tokens occurs. The relative positions of tokens are shown below in parentheses. <offers> <offer id="1000" price="10000"> Ford(1) Mustang(2) 2000(3), 65K(4), excellent(5) condition(6), runs(7) great(8), AC(9), CC(10), power(11) all(12) </offer> <offer id="1001" price="8000"> Honda(13) Accord(14) 1999(15), 78K(16), A(17)/C(18), cruise(19) control(20), runs(21) and(22) looks(23) great(24), excellent(25) condition(26) </offer> <offer id="1005" price="5500"> Ford(27) Mustang(28), 1995(29), 150K(30) highway(31) mileage(32), little(33) rust(34), excellent(35) condition(36) </offer> </offers> The relative positions of paragraphs are determined similarly. In this sample tokenization, the paragraph delimiters are start tags and end tags. The tokens in the first 'offer' element are assigned relative paragraph number 1. The tokens from the next 'offer' element are assigned relative paragraph number 2. Relative paragraph numbers are assigned sequentially through the end of the document. The relative positions of sentences are determined similarly using sentence delimiters. Implementations may provide for the means to ignore or side-step certain structural elements when performing tokenization. In the following example, the implementation has decided to ignore the markup for <bold> <deleted> <para><deleted>This sentence was deleted.</deleted> This <bold>entire paragraph</bold> is one sentence as far as the tokenizer is concerned. </para> Using the same notation as before, this sample tokenization is shown below. All the tokens marked with a token position also have the same sentence and paragraph relative positions. Note that there are no tokens marked for the ignored subtree. <para><deleted>This sentence was deleted.</deleted> This(1) <bold>entire(2) paragraph(3)</bold> is(4) one(5) sentence(6) as(7) far(8) as(9) the(10) tokenizer(11) is(12) concerned(13). </para> [ Definition QueryItem QueryTokenInfo [ Definition QueryTokenInfo QueryTokenInfo position [ Definition TokenInfo TokenInfo startPos endPos startSent endSent startPara endPara The following matching function is the central implementation-defined declare function fts:matchTokenInfos ( $searchContext as item(), $matchOptions as element(fts:matchOptions), $stopWords as xs:string*, $queryTokens as element(fts:queryToken)* ) as element(fts:tokenInfo)* external; The above function returns the TokenInfo $searchContext $queryTokens $matchOptions $stopWords $queryTokens TokenInfo Note: While this matching function assumes a tokenized representation of the query strings, it does not assume a tokenized representation of the input items in $searchContext $searchContext $matchOptions $queryTokens The XQuery 1.0 and XPath 2.0 Data Model is inadequate to support fully composable FTSelection FTSelection XQuery and XPath Full Text adds relative token, sentence, and paragraph position numbers via AllMatches AllMatches FTSelection [ Definition AllMatches FTSelection AllMatches The AllMatches Match [ Definition Match FTSelection StringInclude StringExclude [ Definition StringMatch StringInclude StringExclude queryPos FTOrder TokenInfo StringMatch [ Definition StringInclude StringMatch TokenInfo [ Definition StringExclude StringMatch TokenInfo Intuitively, AllMatches TokenInfo FTSelection The AllMatches AllMatches Match Match StringInclude StringExclude Since in most of the examples below the tokens span only a single position, we characterize the TokenInfo startPos endPos StringMatch The simplest example of an FTSelection FTWords "Mustang" AllMatches FTWords As shown, the AllMatches Match Match FTWords "Mustang" Match StringInclude Match A more complex example of an FTSelection FTWords "Ford Mustang" AllMatches FTWords There are two possible results for this FTWords Match Match Match Match An even more complex example of an FTSelection FTSelection "Mustang" ftand ftnot "rust" AllMatches FTSelection This example introduces StringExclude StringExclude Match Match AllMatches AllMatches FTSelection AllMatches FTSelection AllMatches AllMatches The XML schema for representing AllMatches <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:fts="http://www.w3.org/2007/xpath-full-text" targetNamespace="http://www.w3.org/2007/xpath-full-text" elementFormDefault="qualified" attributeFormDefault="unqualified">

<xs:complexType name="allMatches"> <xs:sequence> <xs:element ref="fts:match" minOccurs="0" maxOccurs="unbounded"/> </xs:sequence> <xs:attribute name="stokenNum" type="xs:integer" use="required" /> </xs:complexType>

<xs:element name="allMatches" type="fts:allMatches"/>

<xs:complexType name="match"> <xs:sequence> <xs:element ref="fts:stringInclude" minOccurs="0" maxOccurs="unbounded"/> <xs:element ref="fts:stringExclude" minOccurs="0" maxOccurs="unbounded"/> </xs:sequence> </xs:complexType> <xs:element name="stringInclude" type="fts:stringMatch" />

<xs:element name="stringExclude" type="fts:stringMatch" />

<xs:element name="match" type="fts:match"/>

<xs:complexType name="stringMatch"> <xs:sequence> <xs:element ref="fts:tokenInfo"/> </xs:sequence> <xs:attribute name="queryPos" type="xs:integer" use="required"/> <xs:attribute name="isContiguous" type="xs:boolean" use="required"/> </xs:complexType>

<xs:complexType name="tokenInfo"> <xs:attribute name="startPos" type="xs:integer" use="required"/> <xs:attribute name="endPos" type="xs:integer" use="required"/> <xs:attribute name="startSent" type="xs:integer" use="required"/> <xs:attribute name="endSent" type="xs:integer" use="required"/> <xs:attribute name="startPara" type="xs:integer" use="required"/> <xs:attribute name="endPara" type="xs:integer" use="required"/> </xs:complexType>

<xs:element name="tokenInfo" type="fts:tokenInfo"/>

<xs:complexType name="queryItem"> <xs:sequence> <xs:element ref="fts:queryToken" minOccurs="0" maxOccurs="unbounded"/> </xs:sequence> </xs:complexType>

<xs:complexType name="queryTokenInfo"> <xs:attribute name="word" type="xs:string" use="required"/> <xs:attribute name="queryPos" type="xs:integer" use="required"/> </xs:complexType>

<xs:element name="queryToken" type="fts:queryTokenInfo"/> </xs:schema> The stokenNum AllMatches AllMatches stokenNum AllMatches queryPos StringMatch FTSelection FTSelection FTSelection FTSelection The XML structures defined by the following schema represent FTSelection 4 Semantics E XML Syntax (XQueryX) for XQuery and XPath Full Text 1.0 FTSelection FTSelection FTSelection FTAnd FTSelection <left> <right> FTSelection <selection> FTSelection FTDistance <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:fts="http://www.w3.org/2007/xpath-full-text" targetNamespace="http://www.w3.org/2007/xpath-full-text" elementFormDefault="qualified" attributeFormDefault="unqualified"> <xs:include schemaLocation="AllMatches.xsd" /> <xs:include schemaLocation="MatchOptions.xsd" />

<xs:complexType name="ftSelection"> <xs:sequence> <xs:choice> <xs:element name="ftWords" type="fts:ftWords"/> <xs:element name="ftAnd" type="fts:ftAnd"/> <xs:element name="ftOr" type="fts:ftOr"/> <xs:element name="ftUnaryNot" type="fts:ftUnaryNot"/> <xs:element name="ftMildNot" type="fts:ftMildNot"/> <xs:element name="ftOrder" type="fts:ftOrder"/> <xs:element name="ftScope" type="fts:ftScope"/> <xs:element name="ftContent" type="fts:ftContent"/> <xs:element name="ftDistance" type="fts:ftDistance"/> <xs:element name="ftWindow" type="fts:ftWindow"/> <xs:element name="ftTimes" type="fts:ftTimes"/> </xs:choice> <xs:element ref="fts:matchOptions" minOccurs="0"/> <xs:element name="weight" type="xs:double" minOccurs="0"/> </xs:sequence> </xs:complexType>

<xs:element name="selection" type="fts:ftSelection"/>

<xs:complexType name="ftWords"> <xs:sequence> <xs:element ref="fts:queryItem" minOccurs="0" maxOccurs="unbounded"/> </xs:sequence> <xs:attribute name="type" type="fts:ftWordsType" use="required"/> </xs:complexType>

<xs:element name="queryItem" type="fts:queryItem"/> <xs:complexType name="ftAnd"> <xs:sequence> <xs:element name="left" type="fts:ftSelection"/> <xs:element name="right" type="fts:ftSelection"/> </xs:sequence> </xs:complexType> <xs:complexType name="ftOr"> <xs:sequence> <xs:element name="left" type="fts:ftSelection"/> <xs:element name="right" type="fts:ftSelection"/> </xs:sequence> </xs:complexType> <xs:complexType name="ftUnaryNot"> <xs:sequence> <xs:element name="selection" type="fts:ftSelection"/> </xs:sequence> </xs:complexType> <xs:complexType name="ftMildNot"> <xs:sequence> <xs:element name="left" type="fts:ftSelection"/> <xs:element name="right" type="fts:ftSelection"/> </xs:sequence> </xs:complexType> <xs:complexType name="ftOrder"> <xs:sequence> <xs:element name="selection" type="fts:ftSelection"/> </xs:sequence> </xs:complexType> <xs:complexType name="ftScope"> <xs:sequence> <xs:element name="selection" type="fts:ftSelection"/> </xs:sequence> <xs:attribute name="type" type="fts:scopeType" use="required"/> <xs:attribute name="scope" type="fts:scopeSelector" use="required"/> </xs:complexType> <xs:complexType name="ftContent"> <xs:sequence> <xs:element name="selection" type="fts:ftSelection"/> </xs:sequence> <xs:attribute name="type" type="fts:contentMatchType" use="required"/> </xs:complexType> <xs:complexType name="ftDistance"> <xs:sequence> <xs:element name="range" type="fts:ftRangeSpec"/> <xs:element name="selection" type="fts:ftSelection"/> </xs:sequence> <xs:attribute name="type" type="fts:distanceType" use="required"/> </xs:complexType> <xs:complexType name="ftWindow"> <xs:sequence> <xs:element name="selection" type="fts:ftSelection"/> </xs:sequence> <xs:attribute name="size" type="xs:integer" use="required"/> <xs:attribute name="type" type="fts:distanceType" use="required"/> </xs:complexType> <xs:complexType name="ftTimes"> <xs:sequence> <xs:element name="range" type="fts:ftRangeSpec"/> <xs:element name="selection" type="fts:ftWords"/> </xs:sequence> </xs:complexType> <xs:simpleType name="ftWordsType"> <xs:restriction base="xs:string"> <xs:enumeration value="any"/> <xs:enumeration value="all"/> <xs:enumeration value="phrase"/> <xs:enumeration value="any word"/> <xs:enumeration value="all word"/> </xs:restriction> </xs:simpleType> <xs:simpleType name="scopeType"> <xs:restriction base="xs:string"> <xs:enumeration value="same"/> <xs:enumeration value="different"/> </xs:restriction> </xs:simpleType> <xs:simpleType name="scopeSelector"> <xs:restriction base="xs:string"> <xs:enumeration value="paragraph"/> <xs:enumeration value="sentence"/> </xs:restriction> </xs:simpleType> <xs:simpleType name="distanceType"> <xs:restriction base="xs:string"> <xs:enumeration value="paragraph"/> <xs:enumeration value="sentence"/> <xs:enumeration value="word"/> </xs:restriction> </xs:simpleType> <xs:simpleType name="contentMatchType"> <xs:restriction base="xs:string"> <xs:enumeration value="at start"/> <xs:enumeration value="at end"/> <xs:enumeration value="entire content"/> </xs:restriction> </xs:simpleType> </xs:schema> evaluate The semantics for the evaluation of FTSelection fts:evaluate FTSelection FTSelection The fts:evaluate AllMatches FTSelection fts:evaluate FTSelection fts:ApplyX AllMatches FTSelection fts:evaluate FTSelection AllMatches fts:ApplyX The semantics for the fts:evaluate declare function fts:evaluate ( $ftSelection as element(*, fts:ftSelection), $searchContext as item(), $matchOptions as element(fts:matchOptions), $queryTokenNum as xs:integer ) as element(fts:allMatches) { if (fn:count($ftSelection/fts:matchOptions) > 0) then (: First we deal with all match options that the :) (: FTSelection might bear: we add the match options :) (: to the current match options structure, and :) (: pass the new structure to the recursive call. :) let $newFTSelection := <fts:selection>{$ftSelection/* [fn:not(self::fts:matchOptions)]}</fts:selection> return fts:evaluate($newFTSelection, $searchContext, fts:replaceMatchOptions($matchOptions, $ftSelection/fts:matchOptions), $queryTokenNum) else if (fn:count($ftSelection/fts:weight) > 0) then (: Weight has no bearing on semantics -- just :) (: call "evaluate" on nested FTSelection :) let $newFTSelection := $ftSelection/*[fn:not(self::fts:weight)] return fts:evaluate($newFTSelection, $searchContext, $matchOptions, $queryTokenNum) else typeswitch ($ftSelection/*[1]) case $nftSelection as element(fts:ftWords) return (: Apply the FTWords in the search context :) fts:ApplyFTWords($searchContext, $matchOptions, $nftSelection/@type, $nftSelection/fts:queryItem, $queryTokenNum + 1) case $nftSelection as element(fts:ftAnd) return let $left := fts:evaluate($nftSelection/fts:left, $searchContext, $matchOptions, $queryTokenNum) let $newQueryTokenNum := $left/@stokenNum let $right := fts:evaluate($nftSelection/fts:right, $searchContext, $matchOptions, $newQueryTokenNum) return fts:ApplyFTAnd($left, $right) case $nftSelection as element(fts:ftOr) return let $left := fts:evaluate($nftSelection/fts:left, $searchContext, $matchOptions, $queryTokenNum) let $newQueryTokenNum := $left/@stokenNum let $right := fts:evaluate($nftSelection/fts:right, $searchContext, $matchOptions, $newQueryTokenNum) return fts:ApplyFTOr($left, $right) case $nftSelection as element(fts:ftUnaryNot) return let $nested := fts:evaluate($nftSelection/fts:selection, $searchContext, $matchOptions, $queryTokenNum) return fts:ApplyFTUnaryNot($nested) case $nftSelection as element(fts:ftMildNot) return let $left := fts:evaluate($nftSelection/fts:left, $searchContext, $matchOptions, $queryTokenNum) let $newQueryTokenNum := $left/@stokenNum let $right := fts:evaluate($nftSelection/fts:right, $searchContext, $matchOptions, $newQueryTokenNum) return fts:ApplyFTMildNot($left, $right) case $nftSelection as element(fts:ftOrder) return let $nested := fts:evaluate($nftSelection/fts:selection, $searchContext, $matchOptions, $queryTokenNum) return fts:ApplyFTOrder($nested) case $nftSelection as element(fts:ftScope) return let $nested := fts:evaluate($nftSelection/fts:selection, $searchContext, $matchOptions, $queryTokenNum) return fts:ApplyFTScope($nftSelection/@type, $nftSelection/@scope, $nested) case $nftSelection as element(fts:ftContent) return let $nested := fts:evaluate($nftSelection/fts:selection, $searchContext, $matchOptions, $queryTokenNum) return fts:ApplyFTContent($searchContext, $nftSelection/@type, $nested) case $nftSelection as element(fts:ftDistance) return let $nested := fts:evaluate($nftSelection/fts:selection, $searchContext, $matchOptions, $queryTokenNum) return fts:ApplyFTDistance($nftSelection/@type, $nftSelection/fts:range, $nested) case $nftSelection as element(fts:ftWindow) return let $nested := fts:evaluate($nftSelection/fts:selection, $searchContext, $matchOptions, $queryTokenNum) return fts:ApplyFTWindow($nftSelection/@type, $nftSelection/@size, $nested) case $nftSelection as element(fts:ftTimes) return let $nested := fts:evaluate($nftSelection/fts:selection, $searchContext, $matchOptions, $queryTokenNum) return fts:ApplyFTTimes($nftSelection/fts:range, $nested) default return <fts:allMatches stokenNum="0" /> }; For concreteness, assume that the FTSelection contains text searchContext contains text ftSelection AllMatches ftSelection fts:evaluate fts:evaluate($ftSelection, $searchContext, $matchOptions, 0) $ftSelection ftSelection $searchContext searchContext Initially, the $queryTokensNum The variable $matchOptions C Static Context Components $ftSelection Given the invocation of: fts:evaluate($ftSelection, $searchContext, $matchOptions) $ftSelection FTWords If $ftSelection fts:replaceMatchOptions 4.2.5 Match Options Semantics evaluate FTSelection If $ftSelection evaluate FTSelection AllMatches If $ftSelection FTWords FTSelection AllMatches FTWords AllMatches ApplyFTWords If $ftSelection FTWords FTSelection ftand ftor window FTSelection First, the evaluate FTSelection FTSelection AllMatches The AllMatches AllMatches FTSelection1 applyX FTSelection For example, let FTSelection1 FTSelection2 ftand FTSelection3 FTSelection2 FTSelection3 FTSelection evaluate FTSelection2 FTSelection3 AllMatches AllMatches ApplyFTAnd ftand The semantics of the ApplyX FTSelection An FTWords applyQueryTokensAsPhrase fts:queryToken (: simplified version not dealing with special match options :) declare function fts:applyQueryTokensAsPhrase ( $searchContext as item(), $matchOptions as element(fts:matchOptions), $queryTokens as element(fts:queryToken)*, $queryPos as xs:integer ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$queryPos}"> { for $tokenInfo in fts:matchTokenInfos( $searchContext, $matchOptions, (), $queryTokens ) return <fts:match> <fts:stringInclude queryPos="{$queryPos}" isContiguous="true"> {$tokenInfo} </fts:stringInclude> </fts:match> } </fts:allMatches> }; If after the application of all the match options, the sequence of query tokens returned for an FTWords AllMatches The AllMatches FTWords Match Match AllMatches FTWords Pos: N startPos endPos N There are five variations of FTWords When any word When all word When phrase When any When all The semantics for FTWords any word FTWords FTSelection ApplyFTWords AllMatches FTSelection declare function fts:MakeDisjunction ( $curRes as element(fts:allMatches), $rest as element(fts:allMatches)* ) as element(fts:allMatches) { if (fn:count($rest) = 0) then $curRes else let $firstAllMatches := $rest[1] let $restAllMatches := fn:subsequence($rest, 2) let $newCurRes := fts:ApplyFTOr($curRes, $firstAllMatches) return fts:MakeDisjunction($newCurRes, $restAllMatches) };

declare function fts:ApplyFTWordsAnyWord ( $searchContext as item(), $matchOptions as element(fts:matchOptions), $queryItems as element(fts:queryItem)*, $queryPos as xs:integer ) as element(fts:allMatches) { (: Tokenization of query string has already occurred. :) (: Get sequence of QueryTokens over all query items. :) let $queryTokens := $queryItems/fts:queryToken return if (fn:count($queryTokens) eq 0) then <fts:allMatches stokenNum="0" /> else let $allAllMatches := for $queryToken at $pos in $queryTokens return fts:applyQueryTokensAsPhrase($searchContext, $matchOptions, $queryToken, $queryPos + $pos - 1) let $firstAllMatches := $allAllMatches[1] let $restAllMatches := fn:subsequence($allAllMatches, 2) return fts:MakeDisjunction($firstAllMatches, $restAllMatches) }; The tokenized query strings are passed to ApplyFTWordsAnyWord as a sequence of fts:queryItem fts:queryToken FTWords applyQueryTokensAsPhrase AllMatches The semantics for FTWords all word declare function fts:MakeConjunction ( $curRes as element(fts:allMatches), $rest as element(fts:allMatches)* ) as element(fts:allMatches) { if (fn:count($rest) = 0) then $curRes else let $firstAllMatches := $rest[1] let $restAllMatches := fn:subsequence($rest, 2) let $newCurRes := fts:ApplyFTAnd($curRes, $firstAllMatches) return fts:MakeConjunction($newCurRes, $restAllMatches) };

declare function fts:ApplyFTWordsAllWord ( $searchContext as item(), $matchOptions as element(fts:matchOptions), $queryItems as element(fts:queryItem)*, $queryPos as xs:integer ) as element(fts:allMatches) { (: Tokenization of query strings has already occurred. :) (: Get sequence of QueryTokens over all query items :) let $queryTokens := $queryItems/fts:queryToken return if (fn:count($queryTokens) eq 0) then <fts:allMatches stokenNum="0" /> else let $allAllMatches := for $queryToken at $pos in $queryTokens return fts:applyQueryTokensAsPhrase($searchContext, $matchOptions, $queryToken, $queryPos + $pos - 1) let $firstAllMatches := $allAllMatches[1] let $restAllMatches := fn:subsequence($allAllMatches, 2) return fts:MakeConjunction($firstAllMatches, $restAllMatches) }; The semantics for FTWords phrase declare function fts:ApplyFTWordsPhrase ( $searchContext as item(), $matchOptions as element(fts:matchOptions), $queryItems as element(fts:queryItem)*, $queryPos as xs:integer ) as element(fts:allMatches) { (: Get sequence of QueryTokenInfos over all query items :) let $queryTokens := $queryItems/fts:queryToken return if (fn:count($queryTokens) eq 0) then <fts:allMatches stokenNum="0" /> else fts:applyQueryTokensAsPhrase($searchContext, $matchOptions, $queryTokens, $queryPos) }; The ApplyFTWordsPhrase applyQueryTokensAsPhrase TokenInfo The semantics for FTWords any declare function fts:ApplyFTWordsAny ( $searchContext as item(), $matchOptions as element(fts:matchOptions), $queryItems as element(fts:queryItem)*, $queryPos as xs:integer ) as element(fts:allMatches) { if (fn:count($queryItems) eq 0) then <fts:allMatches stokenNum="0" /> else let $firstQueryItem := $queryItems[1] let $restQueryItem := fn:subsequence($queryItems, 2) let $firstAllMatches := fts:ApplyFTWordsPhrase($searchContext, $matchOptions, $firstQueryItem, $queryPos) let $newQueryPos := if ($firstAllMatches//@queryPos) then fn:max($firstAllMatches//@queryPos) + 1 else $queryPos let $restAllMatches := fts:ApplyFTWordsAny($searchContext, $matchOptions, $restQueryItem, $newQueryPos) return fts:ApplyFTOr($firstAllMatches, $restAllMatches) }; The FTWords any AllMatches The semantics for FTWords all declare function fts:ApplyFTWordsAll ( $searchContext as item(), $matchOptions as element(fts:matchOptions), $queryItems as element(fts:queryItem)*, $queryPos as xs:integer ) as element(fts:allMatches) { if (fn:count($queryItems) = 0) then <fts:allMatches stokenNum="0" /> else let $firstQueryItem := $queryItems[1] let $restQueryItem := fn:subsequence($queryItems, 2) let $firstAllMatches := fts:ApplyFTWordsPhrase($searchContext, $matchOptions, $firstQueryItem, $queryPos) return if ($restQueryItem) then let $newQueryPos := if ($firstAllMatches//@queryPos) then fn:max($firstAllMatches//@queryPos) + 1 else $queryPos let $restAllMatches := fts:ApplyFTWordsAll($searchContext, $matchOptions, $restQueryItem, $newQueryPos) return fts:ApplyFTAnd($firstAllMatches, $restAllMatches) else $firstAllMatches }; The difference between all any The ApplyFTWords declare function fts:ApplyFTWords ( $searchContext as item(), $matchOptions as element(fts:matchOptions), $type as fts:ftWordsType, $queryItems as element(fts:queryItem)*, $queryPos as xs:integer ) as element(fts:allMatches) { if ($type eq "any word") then fts:ApplyFTWordsAnyWord($searchContext, $matchOptions, $queryItems, $queryPos) else if ($type eq "all word") then fts:ApplyFTWordsAllWord($searchContext, $matchOptions, $queryItems, $queryPos) else if ($type eq "phrase") then fts:ApplyFTWordsPhrase($searchContext, $matchOptions, $queryItems, $queryPos) else if ($type eq "any") then fts:ApplyFTWordsAny($searchContext, $matchOptions, $queryItems, $queryPos) else fts:ApplyFTWordsAll($searchContext, $matchOptions, $queryItems, $queryPos) }; XQuery 1.0 functions are used to define the semantics of FTMatchOption FTMatchOption syntax FTMatchOption <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:fts="http://www.w3.org/2007/xpath-full-text" targetNamespace="http://www.w3.org/2007/xpath-full-text" elementFormDefault="qualified" attributeFormDefault="unqualified">

<xs:complexType name="ftMatchOptions"> <xs:sequence> <xs:element ref="fts:thesaurus" minOccurs="0" maxOccurs="1"/> <xs:element ref="fts:stopwords" minOccurs="0" maxOccurs="1"/> <xs:element ref="fts:case" minOccurs="0" maxOccurs="1"/> <xs:element ref="fts:diacritics" minOccurs="0" maxOccurs="1"/> <xs:element ref="fts:stem" minOccurs="0" maxOccurs="1"/> <xs:element ref="fts:wildcard" minOccurs="0" maxOccurs="1"/> <xs:element ref="fts:language" minOccurs="0" maxOccurs="1"/> </xs:sequence> </xs:complexType>

<xs:element name="matchOptions" type="fts:ftMatchOptions"/>

<xs:element name="case" type="fts:ftCaseOption" /> <xs:element name="diacritics" type="fts:ftDiacriticsOption" /> <xs:element name="thesaurus" type="fts:ftThesaurusOption" /> <xs:element name="stem" type="fts:ftStemOption" /> <xs:element name="wildcard" type="fts:ftWildCardOption" /> <xs:element name="language" type="fts:ftLanguageOption" /> <xs:element name="stopwords" type="fts:ftStopWordOption" />

<xs:complexType name="ftCaseOption"> <xs:sequence> <xs:element name="value"> <xs:simpleType> <xs:restriction base="xs:string"> <xs:enumeration value="case insensitive"/> <xs:enumeration value="case sensitive"/> <xs:enumeration value="lowercase"/> <xs:enumeration value="uppercase"/> </xs:restriction> </xs:simpleType> </xs:element> </xs:sequence> </xs:complexType>

<xs:complexType name="ftDiacriticsOption"> <xs:sequence> <xs:element name="value"> <xs:simpleType> <xs:restriction base="xs:string"> <xs:enumeration value="diacritics insensitive"/> <xs:enumeration value="diacritics sensitive"/> </xs:restriction> </xs:simpleType> </xs:element> </xs:sequence> </xs:complexType> <xs:complexType name="ftThesaurusOption"> <xs:sequence> <xs:element name="thesaurusName" type="xs:string" minOccurs="0" maxOccurs="1"/> <xs:element name="relationship" type="xs:string" minOccurs="0" maxOccurs="1"/> <xs:element name="range" type="fts:ftRangeSpec" minOccurs="0" maxOccurs="1"/> </xs:sequence> <xs:attribute name="thesaurusIndicator"> <xs:simpleType> <xs:restriction base="xs:string"> <xs:enumeration value="using"/> <xs:enumeration value="no"/> </xs:restriction> </xs:simpleType> </xs:attribute> </xs:complexType> <xs:complexType name="ftRangeSpec"> <xs:attribute name="type" type="fts:rangeSpecType" use="required"/> <xs:attribute name="m" type="xs:integer"/> <xs:attribute name="n" type="xs:integer" use="required"/> </xs:complexType> <xs:simpleType name="rangeSpecType"> <xs:restriction base="xs:string"> <xs:enumeration value="exactly"/> <xs:enumeration value="at least"/> <xs:enumeration value="at most"/> <xs:enumeration value="from to"/> </xs:restriction> </xs:simpleType> <xs:complexType name="ftStemOption"> <xs:sequence> <xs:element name="value"> <xs:simpleType> <xs:restriction base="xs:string"> <xs:enumeration value="stemming"/> <xs:enumeration value="no stemming"/> </xs:restriction> </xs:simpleType> </xs:element> </xs:sequence> </xs:complexType> <xs:complexType name="ftWildCardOption"> <xs:sequence> <xs:element name="value"> <xs:simpleType> <xs:restriction base="xs:string"> <xs:enumeration value="wildcards"/> <xs:enumeration value="no wildcards"/> </xs:restriction> </xs:simpleType> </xs:element> </xs:sequence> </xs:complexType> <xs:complexType name="ftLanguageOption"> <xs:sequence> <xs:element name="value" type="xs:string"/> </xs:sequence> </xs:complexType>

<xs:complexType name="ftStopWordOption"> <xs:sequence> <xs:choice> <xs:element name="default-stopwords"> <xs:complexType /> </xs:element> <xs:element name="stopword" type="xs:string" /> <xs:element name="uri" type="xs:anyURI" /> </xs:choice> <xs:element name="oper" minOccurs="0" maxOccurs="unbounded"> <xs:complexType> <xs:choice> <xs:element name="stopword" type="xs:string" /> <xs:element name="uri" type="xs:anyURI" /> </xs:choice> <xs:attribute name="type"> <xs:simpleType> <xs:restriction base="xs:string"> <xs:enumeration value="union"/> <xs:enumeration value="except"/> </xs:restriction> </xs:simpleType> </xs:attribute> </xs:complexType> </xs:element> </xs:sequence> </xs:complexType> </xs:schema> The previous section described FTSelections without giving any details about how FTMatchOption FTMatchOption matchTokenInfos implementation-defined FTMatchOption The extension is achieved by modifying an existing function and adding functions that are specific to the FTMatchOption Modifications in the semantics of existing functions The semantics of most of the FTSelection declare function fts:applyQueryTokensAsPhrase ( $searchContext as item(), $matchOptions as element(fts:matchOptions), $queryTokens as element(fts:queryToken)*, $queryPos as xs:integer ) as element(fts:allMatches) { let $thesaurusOption := $matchOptions/fts:thesaurus[1] return if ($thesaurusOption and $thesaurusOption/@thesaurusIndicator eq "using") then let $noThesaurusOptions := <fts:matchOptions>{ $matchOptions/*[fn:not(self::fts:thesaurus)] }</fts:matchOptions> let $lookupRes := fts:applyThesaurusOption($thesaurusOption, $noThesaurusOptions, $queryTokens) return fts:ApplyFTWordsAny($searchContext, $noThesaurusOptions, $lookupRes, $queryPos) else (: from here on we have a single sequence of query tokens :) (: which is to be matched a phrase; no alternatives anymore :) <fts:allMatches stokenNum="{$queryPos}"> { for $pos in fts:matchTokenInfos( $searchContext, $matchOptions, fts:applyStopWordOption($matchOptions/fts:stopwords), $queryTokens ) return <fts:match> <fts:stringInclude queryPos="{$queryPos}" isContiguous="true"> {$pos} </fts:stringInclude> </fts:match> } </fts:allMatches> }; Two FTMatchOption FTMatchOption Unlike all other FTMatchOption FTThesaurusOption applyThesaurusOption fts:ApplyFTWordsAny FTThesaurusOption For the semantics of the FTStopWordOption Semantics of new FTMatchOption The expansion of FTSelection FTMatchOption The evaluate replaceMatchOptions replaceMatchOptions FTSelection FTSelection $ftSelection/fts:matchOptions $matchOptions declare function fts:replaceMatchOptions ( $matchOptions as element(fts:matchOptions), $newMatchOptions as element(fts:matchOptions) ) as element(fts:matchOptions) { <fts:matchOptions> { (if ($newMatchOptions/fts:thesaurus) then $newMatchOptions/fts:thesaurus else $matchOptions/fts:thesaurus), (if ($newMatchOptions/fts:stopwords) then $newMatchOptions/fts:stopwords else $matchOptions/fts:stopwords), (if ($newMatchOptions/fts:case) then $newMatchOptions/fts:case else $matchOptions/fts:case), (if ($newMatchOptions/fts:diacritics) then $newMatchOptions/fts:diacritics else $matchOptions/fts:diacritics), (if ($newMatchOptions/fts:stem) then $newMatchOptions/fts:stem else $matchOptions/fts:stem), (if ($newMatchOptions/fts:wildcard) then $newMatchOptions/fts:wildcard else $matchOptions/fts:wildcard), (if ($newMatchOptions/fts:language) then $newMatchOptions/fts:language else $matchOptions/fts:language) } </fts:matchOptions> }; This function determines how match options of the same group overwrite each other, so that only one option of the same group remains. The details of the semantics of the remaining FTMatchOption implementation-defined matchTokenInfos FTMatchOption declare function fts:resolveStopWordsUri ( $uri as xs:string? ) as xs:string* external;

declare function fts:lookupThesaurus ( $tokens as element(fts:queryToken)*, $thesaurusName as xs:string?, $relationship as xs:string?, $range as element(fts:range)?, $noThesaurusOptions as element(fts:matchOptions) ) as element(fts:queryItem)* external; The function resolveStopWordsUri The function lookupThesaurus $tokens $thesaurusName $relationship $range $tokens The current match options other than the thesaurus option are also passed to the function, via $noThesaurusOptions The thesaurus function returns a sequence of expansion alternatives. Each alternative is regarded as a new search phrase and is represented as a query item. Alternatives are treated as though they are connected with a disjunction ( FTOr FTMatchOption FTCaseOption $matchOptions matchTokenInfos FTCaseOption TokenInfo FTCaseOption TokenInfo FTCaseOption TokenInfo FTCaseOption TokenInfo FTMatchOption FTDiacriticsOption $matchOptions matchTokenInfos FTDiacriticsOption TokenInfo FTDiacriticsOption TokenInfo FTMatchOption FTStemOption $matchOptions matchTokenInfos implementation-defined FTStemOption TokenInfo The semantics for the FTThesaurusOption declare function fts:applyThesaurusOption ( $matchOption as element(fts:thesaurus), $noThesaurusOptions as element(fts:matchOptions), $queryTokens as element(fts:queryToken)* ) as element(fts:queryItem)* { if ($matchOption/@thesaurusIndicator = "using") then fts:lookupThesaurus( $queryTokens, $matchOption/fts:thesaurusName, $matchOption/fts:relationship, $matchOption/fts:range, $noThesaurusOptions ) else if ($matchOption/@thesaurusIndicator = "no") then <fts:queryItem> {$queryTokens} </fts:queryItem> else () }; Stop words interact with FTDistance FTWindow FTStopWordOption declare function fts:applyStopWordOption ( $stopWordOption as element(fts:stopwords)? ) as xs:string* { if ($stopWordOption) then let $swords := typeswitch ($stopWordOption/*[1]) case $e as element(fts:stopword) return $e/text() case $e as element(fts:uri) return fts:resolveStopWordsUri($e/text()) case element(fts:default-stopwords) return fts:resolveStopWordsUri(()) default return () return fts:calcStopWords( $swords, $stopWordOption/fts:oper ) else () }; declare function fts:calcStopWords ( $stopWords as xs:string*, $opers as element(fts:oper)* ) as xs:string* { if ( fn:empty($opers) ) then $stopWords else let $swords := typeswitch ($opers[1]/*[1]) case $e as element(fts:stopword) return $e/text() case $e as element(fts:uri) return fts:resolveStopWordsUri($e/text()) default return () return if ($opers[1]/@type eq "union") then fts:calcStopWords( ($stopWords, $swords), $opers[fn:position() gt 2] ) else (: "except" :) fts:calcStopWords( $stopWords[fn:not(.)=$swords], $opers[fn:position() gt 2] ) }; Given the applicable setting of the Stop Word Option, the function fts:applyStopWordOption fts:calcStopWords xs:string* fts:matchTokenInfos fts:calcStopWords fts:resolveStopWordsUri The FTLanguageOption FTMatchOption FTWildCardOption $matchOptions matchTokenInfos FTWildCardOption TokenInfo Section 3.2.7 FTWildCardOption FTWildCardOption The parameters of the ApplyFTOr AllMatches FTSelection declare function fts:ApplyFTOr ( $allMatches1 as element(fts:allMatches), $allMatches2 as element(fts:allMatches) ) as element(fts:allMatches) { <fts:allMatches stokenNum="{fn:max(($allMatches1/@stokenNum, $allMatches2/@stokenNum))}"> {$allMatches1/fts:match,$allMatches2/fts:match} </fts:allMatches> }; The ApplyFTOr AllMatches Match AllMatches Match FTSelection Match AllMatches For example, consider the FTSelection "Mustang" ftor "Honda" AllMatches The AllMatches ApplyFTOr The parameters of the ApplyFTAnd AllMatches FTSelection declare function fts:ApplyFTAnd ( $allMatches1 as element(fts:allMatches), $allMatches2 as element(fts:allMatches) ) as element(fts:allMatches) { <fts:allMatches stokenNum="{fn:max(($allMatches1/@stokenNum, $allMatches2/@stokenNum))}" > { for $sm1 in $allMatches1/fts:match for $sm2 in $allMatches2/fts:match return <fts:match> {$sm1/*, $sm2/*} </fts:match> } </fts:allMatches> }; The result of the conjunction is a new AllMatches FTSelection Match StringInclude StringExclude AllMatches FTSelection Match FTSelection Match For example, consider the FTSelection "Mustang" ftand "rust" AllMatches The AllMatches ApplyFTAnd The ApplyFTUnaryNot AllMatches FTSelection declare function fts:InvertStringMatch ( $strm as element(*,fts:stringMatch) ) as element(*,fts:stringMatch) { if ($strm instance of element(fts:stringExclude)) then <fts:stringInclude queryPos="{$strm/@queryPos}" isContiguous="{$strm/@isContiguous}"> {$strm/fts:tokenInfo} </fts:stringInclude> else <fts:stringExclude queryPos="{$strm/@queryPos}" isContiguous="{$strm/@isContiguous}"> {$strm/fts:tokenInfo} </fts:stringExclude> };

declare function fts:UnaryNotHelper ( $matches as element(fts:match)* ) as element(fts:match)* { if (fn:empty($matches)) then <fts:match/> else for $sm in $matches[1]/* for $rest in fts:UnaryNotHelper( fn:subsequence($matches, 2) ) return <fts:match> { fts:InvertStringMatch($sm), $rest/* } </fts:match> };

declare function fts:ApplyFTUnaryNot ( $allMatches as element(fts:allMatches) ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { fts:UnaryNotHelper($allMatches/fts:match) } </fts:allMatches> }; The generation of the resulting AllMatches FTUnaryNot AllMatches StringMatch AllMatches In the InvertStringMatch The function fts:invertStringMatch StringInclude StringExclude The function fts:UnaryNotHelper Match Match StringInclude StringExclude Match Match For example, consider the FTSelection ftnot ("Mustang" ftor "Honda") AllMatches The FTUnaryNot StringInclude StringExclude The parameters of the ApplyFTMildNot AllMatches FTSelection declare function fts:CoveredIncludePositions ( $match as element(fts:match) ) as xs:integer* { for $strInclude in $match/fts:stringInclude return $strInclude/fts:tokenInfo/@startPos to $strInclude/fts:tokenInfo/@endPos };

declare function fts:ApplyFTMildNot ( $allMatches1 as element(fts:allMatches), $allMatches2 as element(fts:allMatches) ) as element(fts:allMatches) { if (fn:count($allMatches1//fts:stringExclude) gt 0) then fn:error(fn:QName('http://www.w3.org/2005/xqt-errors', 'FTDY0017'), "Invalid expression on the left-hand side of a not-in") else if (fn:count($allMatches2//fts:stringExclude) gt 0) then fn:error(fn:QName('http://www.w3.org/2005/xqt-errors', 'FTDY0017'), "Invalid expression on the right-hand side of a not-in") else if (fn:count($allMatches2//fts:stringInclude) eq 0) then $allMatches1 else <fts:allMatches stokenNum="{$allMatches1/@stokenNum}"> { $allMatches1/fts:match[ every $matches2 in $allMatches2/fts:match satisfies let $posSet1 := fts:CoveredIncludePositions(.) let $posSet2 := fts:CoveredIncludePositions($matches2) return some $pos in $posSet1 satisfies fn:not($pos = $posSet2) ] } </fts:allMatches> }; The resulting AllMatches Match StringInclude StringInclude AllMatches For example, consider the FTSelection ("Ford" not in "Ford Mustang") AllMatches The source AllMatches The FTMildNot AllMatches AllMatches TokenInfo StringInclude AllMatches The ApplyFTOrder AllMatches FTSelection declare function fts:ApplyFTOrder ( $allMatches as element(fts:allMatches) ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match where every $stringInclude1 in $match/fts:stringInclude, $stringInclude2 in $match/fts:stringInclude satisfies (($stringInclude1/fts:tokenInfo/@startPos <= $stringInclude2/fts:tokenInfo/@startPos) and ($stringInclude1/@queryPos <= $stringInclude2/@queryPos)) or (($stringInclude1/fts:tokenInfo/@startPos>= $stringInclude2/fts:tokenInfo/@startPos) and ($stringInclude1/@queryPos >= $stringInclude2/@queryPos)) return <fts:match> { $match/fts:stringInclude, for $stringExcl in $match/fts:stringExclude where every $stringIncl in $match/fts:stringInclude satisfies (($stringExcl/fts:tokenInfo/@startPos <= $stringIncl/fts:tokenInfo/@startPos) and ($stringExcl/@queryPos <= $stringIncl/@queryPos)) or (($stringExcl/fts:tokenInfo/@startPos >= $stringIncl/fts:tokenInfo/@startPos) and ($stringExcl/@queryPos >= $stringIncl/@queryPos)) return $stringExcl } </fts:match> } </fts:allMatches> }; The resulting AllMatches Match StringInclude StringExclude For example, consider the FTSelection ("great" ftand "condition") ordered AllMatches The AllMatches FTOrder The parameters of the ApplyFTScope AllMatches FTSelection The semantics of same sentence declare function fts:ApplyFTScopeSameSentence ( $allMatches as element(fts:allMatches) ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match where every $stringInclude1 in $match/fts:stringInclude, $stringInclude2 in $match/fts:stringInclude satisfies $stringInclude1/fts:tokenInfo/@startSent = $stringInclude2/fts:tokenInfo/@startSent and $stringInclude1/fts:tokenInfo/@startSent = $stringInclude1/fts:tokenInfo/@endSent and $stringInclude2/fts:tokenInfo/@startSent = $stringInclude2/fts:tokenInfo/@endSent and $stringInclude1/fts:tokenInfo/@startSent > 0 and $stringInclude2/fts:tokenInfo/@startSent > 0 return <fts:match> { $match/fts:stringInclude, for $stringExcl in $match/fts:stringExclude where $stringExcl/fts:tokenInfo/@startSent = 0 or ($stringExcl/fts:tokenInfo/@startSent = $stringExcl/fts:tokenInfo/@endSent and (every $stringIncl in $match/fts:stringInclude satisfies $stringIncl/fts:tokenInfo/@startSent = $stringExcl/fts:tokenInfo/@startSent) ) return $stringExcl } </fts:match> } </fts:allMatches> }; An AllMatches same sentence Match StringInclude Match StringExclude StringInclude Match StringInclude The semantics of different sentence declare function fts:ApplyFTScopeDifferentSentence ( $allMatches as element(fts:allMatches) ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match where count($match/fts:stringInclude) > 1 and ( every $stringInclude1 in $match/fts:stringInclude, $stringInclude2 in $match/fts:stringInclude satisfies $stringInclude1 is $stringInclude2 or ( ( $stringInclude1/fts:tokenInfo/@startSent != $stringInclude2/fts:tokenInfo/@startSent or $stringInclude1/fts:tokenInfo/@startSent != $stringInclude1/fts:tokenInfo/@endSent or $stringInclude2/fts:tokenInfo/@startSent != $stringInclude2/fts:tokenInfo/@endSent ) and $stringInclude1/fts:tokenInfo/@startSent > 0 and $stringInclude2/fts:tokenInfo/@endSent > 0 ) ) return <fts:match> { $match/fts:stringInclude, for $stringExcl in $match/fts:stringExclude where every $stringIncl in $match/fts:stringInclude satisfies ($stringIncl/fts:tokenInfo/@startSent != $stringExcl/fts:tokenInfo/@startSent or $stringIncl/fts:tokenInfo/@startSent != $stringIncl/fts:tokenInfo/@endSent or $stringExcl/fts:tokenInfo/@startSent != $stringExcl/fts:tokenInfo/@endSent ) and $stringIncl/fts:tokenInfo/@startSent > 0 and $stringExcl/fts:tokenInfo/@endSent > 0 return $stringExcl } </fts:match> } </fts:allMatches> }; An AllMatches different sentence Match Match StringExclude StringInclude The semantics of same paragraph same sentence declare function fts:ApplyFTScopeSameParagraph ( $allMatches as element(fts:allMatches) ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match where every $stringInclude1 in $match/fts:stringInclude, $stringInclude2 in $match/fts:stringInclude satisfies $stringInclude1/fts:tokenInfo/@startPara = $stringInclude2/fts:tokenInfo/@startPara and $stringInclude1/fts:tokenInfo/@startPara = $stringInclude1/fts:tokenInfo/@endPara and $stringInclude2/fts:tokenInfo/@startPara = $stringInclude2/fts:tokenInfo/@endPara and $stringInclude1/fts:tokenInfo/@startPara > 0 and $stringInclude2/fts:tokenInfo/@endPara > 0 return <fts:match> { $match/fts:stringInclude, for $stringExcl in $match/fts:stringExclude where $stringExcl/fts:tokenInfo/@startPara = 0 or ($stringExcl/fts:tokenInfo/@startPara = $stringExcl/fts:tokenInfo/@endPara and (every $stringIncl in $match/fts:stringInclude satisfies $stringIncl/fts:tokenInfo/@startPara = $stringExcl/fts:tokenInfo/@startPara) ) return $stringExcl } </fts:match> } </fts:allMatches> }; The semantics of different paragraph different sentence declare function fts:ApplyFTScopeDifferentParagraph ( $allMatches as element(fts:allMatches) ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match where count($match/fts:stringInclude) > 1 and ( every $stringInclude1 in $match/fts:stringInclude, $stringInclude2 in $match/fts:stringInclude satisfies $stringInclude1 is $stringInclude2 or ( ( $stringInclude1/fts:tokenInfo/@startPara != $stringInclude2/fts:tokenInfo/@startPara or $stringInclude1/fts:tokenInfo/@startPara != $stringInclude1/fts:tokenInfo/@endPara or $stringInclude2/fts:tokenInfo/@startPara != $stringInclude2/fts:tokenInfo/@endPara ) and $stringInclude1/fts:tokenInfo/@startPara > 0 and $stringInclude2/fts:tokenInfo/@endPara > 0 ) ) return <fts:match> { $match/fts:stringInclude, for $stringExcl in $match/fts:stringExclude where every $stringIncl in $match/fts:stringInclude satisfies ($stringIncl/fts:tokenInfo/@startPara != $stringExcl/fts:tokenInfo/@startPara or $stringIncl/fts:tokenInfo/@startPara != $stringIncl/fts:tokenInfo/@endPara or $stringExcl/fts:tokenInfo/@startPara != $stringExcl/fts:tokenInfo/@endPara ) and $stringIncl/fts:tokenInfo/@startPara > 0 and $stringExcl/fts:tokenInfo/@endPara > 0 return $stringExcl } </fts:match> } </fts:allMatches> }; The semantics for the general case is given below. declare function fts:ApplyFTScope ( $type as fts:scopeType, $selector as fts:scopeSelector, $allMatches as element(fts:allMatches) ) as element(fts:allMatches) { if ($type eq "same" and $selector eq "sentence") then fts:ApplyFTScopeSameSentence($allMatches) else if ($type eq "different" and $selector eq "sentence") then fts:ApplyFTScopeDifferentSentence($allMatches) else if ($type eq "same" and $selector eq "paragraph") then fts:ApplyFTScopeSameParagraph($allMatches) else fts:ApplyFTScopeDifferentParagraph($allMatches) }; For example, consider the FTSelection ("Mustang" ftand "Honda") same paragraph AllMatches The FTScope AllMatches Match TokenInfo The parameters of the ApplyFTContent at start at end entire content AllMatches FTSelection The evaluation of ApplyFTContent entire content Match StringInclude Match at start Match StringInclude at end Match StringInclude The semantics is given below. declare function fts:ApplyFTContent ( $searchContext as item(), $type as fts:contentMatchType, $allMatches as element(fts:allMatches) ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { $allMatches/fts:match[ let $start_pos := fts:getLowestTokenPosition($searchContext), $end_pos := fts:getHighestTokenPosition($searchContext), $match := . return if ($type eq "entire content") then every $pos in $start_pos to $end_pos satisfies some $si in $match/fts:stringInclude[data(@isContiguous)] satisfies fts:TokenInfoCoversTokenPosition($si/fts:tokenInfo, $pos) else let $pos := if ($type eq "at start") then $start_pos else (: $type eq "at end" :) $end_pos return some $ti in $match/fts:stringInclude/fts:tokenInfo satisfies fts:TokenInfoCoversTokenPosition($ti, $pos) ] } </fts:allMatches> }; ApplyFTContent fts:TokenInfoCoversTokenPosition $tokenInfo $tokenPosition declare function fts:TokenInfoCoversTokenPosition( $tokenInfo as element(fts:tokenInfo), $tokenPosition as xs:integer ) as xs:boolean { ($tokenPosition >= $tokenInfo/@startPos) and ($tokenPosition <= $tokenInfo/@endPos) }; ApplyFTContent implementation-dependent getLowestTokenPosition getHighestTokenPosition $searchContext declare function fts:getLowestTokenPosition( $searchContext as item() ) as xs:integer external;

declare function fts:getHighestTokenPosition( $searchContext as item() ) as xs:integer external; Note that the way @isContiguous joinIncludes ApplyFTContent "one two three four" contains text ("one" ftand "three" window 3 words) ftand ("two" ftand "four" window 3 words) entire content Even though the four query tokens do Match ApplyFTContent StringInclude Before we define the semantics functions of the FTWindow FTDistance joinIncludes joinIncludes StringInclude Match StringInclude StringInclude declare function fts:joinIncludes( $strIncls as element(fts:stringInclude)* ) as element(fts:stringInclude)? { if (fn:empty($strIncls)) then $strIncls else let $posSet := fts:CoveredIncludePositions(<fts:match>$strIncls</fts:match>), $minPos := fn:min($strIncls/fts:tokenInfo/@startPos), $maxPos := fn:max($strIncls/fts:tokenInfo/@endPos), $isContiguous := ( every $pos in $minPos to $maxPos satisfies ($pos = $posSet) ) and ( every $strIncl in $strIncls satisfies $strIncl/@isContiguous ) return <fts:stringInclude queryPos="{$strIncls[1]/@queryPos}" isContiguous="{$isContiguous}"> <fts:tokenInfo startPos ="{$minPos}" endPos ="{$maxPos}" startSent="{fn:min($strIncls/fts:tokenInfo/@startSent)}" endSent ="{fn:max($strIncls/fts:tokenInfo/@endSent)}" startPara="{fn:min($strIncls/fts:tokenInfo/@startPara)}" endPara ="{fn:max($strIncls/fts:tokenInfo/@endPara)}"/> </fts:stringInclude> }; The parameters of the ApplyFTWindow fts:distanceType AllMatches FTSelection The semantics of window N words declare function fts:ApplyFTWordWindow ( $allMatches as element(fts:allMatches), $n as xs:integer ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match let $minpos := fn:min($match/fts:stringInclude/fts:tokenInfo/@startPos), $maxpos := fn:max($match/fts:stringInclude/fts:tokenInfo/@endPos) for $windowStartPos in ($maxpos - $n + 1 to $minpos) let $windowEndPos := $windowStartPos + $n - 1 return <fts:match> { fts:joinIncludes($match/fts:stringInclude), for $stringExclude in $match/fts:stringExclude where $stringExclude/fts:tokenInfo/@startPos >= $windowStartPos and $stringExclude/fts:tokenInfo/@endPos <= $windowEndPos return $stringExclude } </fts:match> } </fts:allMatches> }; The semantics of window N sentences declare function fts:ApplyFTSentenceWindow ( $allMatches as element(fts:allMatches), $n as xs:integer ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match let $minpos := fn:min($match/fts:stringInclude/fts:tokenInfo/@startSent), $maxpos := fn:max($match/fts:stringInclude/fts:tokenInfo/@endSent) for $windowStartPos in ($maxpos - $n + 1 to $minpos) let $windowEndPos := $windowStartPos + $n - 1 return <fts:match> { fts:joinIncludes($match/fts:stringInclude), for $stringExclude in $match/fts:stringExclude where $stringExclude/fts:tokenInfo/@startSent >= $windowStartPos and $stringExclude/fts:tokenInfo/@endSent <= $windowEndPos return $stringExclude } </fts:match> } </fts:allMatches> }; The semantics of window N paragraphs declare function fts:ApplyFTParagraphWindow ( $allMatches as element(fts:allMatches), $n as xs:integer ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match let $minpos := fn:min($match/fts:stringInclude/fts:tokenInfo/@startPara), $maxpos := fn:max($match/fts:stringInclude/fts:tokenInfo/@endPara) for $windowStartPos in ($maxpos - $n + 1 to $minpos) let $windowEndPos := $windowStartPos + $n - 1 return <fts:match> { fts:joinIncludes($match/fts:stringInclude), for $stringExclude in $match/fts:stringExclude where $stringExclude/fts:tokenInfo/@startPara >= $windowStartPos and $stringExclude/fts:tokenInfo/@endPara <= $windowEndPos return $stringExclude } </fts:match> } </fts:allMatches> }; The resulting AllMatches Match StringInclude StringExclude Match StringInclude StringInclude The semantics for the general function is given below. declare function fts:ApplyFTWindow ( $type as fts:distanceType, $size as xs:integer, $allMatches as element(fts:allMatches) ) as element(fts:allMatches) { if ($type eq "word") then fts:ApplyFTWordWindow($allMatches, $size) else if ($type eq "sentence") then fts:ApplyFTSentenceWindow($allMatches, $size) else fts:ApplyFTParagraphWindow($allMatches, $size) }; For example, consider the FTWindow ("Ford Mustang" ftand "excellent") window 10 words Match AllMatches ("Ford Mustang" ftand "excellent") The result for the FTWindow Match The parameters of the ApplyFTDistance AllMatches FTSelection AllMatches Match StringInclude StringInclude An invocation of the ApplyFTDistance declare function fts:ApplyFTDistance ( $type as fts:distanceType, $range as element(fts:range), $allMatches as element(fts:allMatches) ) as element(fts:allMatches) { if ($type eq "word") then if ($range/@type eq "exactly") then fts:ApplyFTWordDistanceExactly($allMatches, $range/@n) else if ($range/@type eq "at least") then fts:ApplyFTWordDistanceAtLeast($allMatches, $range/@n) else if ($range/@type eq "at most") then fts:ApplyFTWordDistanceAtMost( $allMatches, $range/@n) else fts:ApplyFTWordDistanceFromTo( $allMatches, $range/@m, $range/@n) else if ($type eq "sentence") then if ($range/@type eq "exactly") then fts:ApplyFTSentenceDistanceExactly($allMatches, $range/@n) else if ($range/@type eq "at least") then fts:ApplyFTSentenceDistanceAtLeast($allMatches, $range/@n) else if ($range/@type eq "at most") then fts:ApplyFTSentenceDistanceAtMost( $allMatches, $range/@n) else fts:ApplyFTSentenceDistanceFromTo( $allMatches, $range/@m, $range/@n) else if ($range/@type eq "exactly") then fts:ApplyFTParagraphDistanceExactly($allMatches, $range/@n) else if ($range/@type eq "at least") then fts:ApplyFTParagraphDistanceAtLeast($allMatches, $range/@n) else if ($range/@type eq "at most") then fts:ApplyFTParagraphDistanceAtMost( $allMatches, $range/@n) else fts:ApplyFTParagraphDistanceFromTo( $allMatches, $range/@m, $range/@n) }; Word Distance The semantics of case word distance exactly N declare function fts:ApplyFTWordDistanceExactly( $allMatches as element(fts:allMatches), $n as xs:integer ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match let $sorted := for $si in $match/fts:stringInclude order by $si/fts:tokenInfo/@startPos ascending, $si/fts:tokenInfo/@endPos ascending return $si where if (fn:count($sorted) le 1) then fn:true() else every $idx in 1 to fn:count($sorted) - 1 satisfies fts:wordDistance( $sorted[$idx]/fts:tokenInfo, $sorted[$idx+1]/fts:tokenInfo ) = $n return <fts:match> { fts:joinIncludes($match/fts:stringInclude), for $stringExcl in $match/fts:stringExclude where some $stringIncl in $match/fts:stringInclude satisfies fts:wordDistance( $stringIncl/fts:tokenInfo, $stringExcl/fts:tokenInfo ) = $n return $stringExcl } </fts:match> } </fts:allMatches> }; The semantics of word distance at least N declare function fts:ApplyFTWordDistanceAtLeast ( $allMatches as element(fts:allMatches), $n as xs:integer ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match let $sorted := for $si in $match/fts:stringInclude order by $si/fts:tokenInfo/@startPos ascending, $si/fts:tokenInfo/@endPos ascending return $si where if (fn:count($sorted) le 1) then fn:true() else every $index in (1 to fn:count($sorted) - 1) satisfies fts:wordDistance( $sorted[$index]/fts:tokenInfo, $sorted[$index+1]/fts:tokenInfo ) >= $n return <fts:match> { fts:joinIncludes($match/fts:stringInclude), for $stringExcl in $match/fts:stringExclude where some $stringIncl in $match/fts:stringInclude satisfies fts:wordDistance( $stringIncl/fts:tokenInfo, $stringExcl/fts:tokenInfo ) >= $n return $stringExcl } </fts:match> } </fts:allMatches> }; The semantics of word distance at most N declare function fts:ApplyFTWordDistanceAtMost ( $allMatches as element(fts:allMatches), $n as xs:integer ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match let $sorted := for $si in $match/fts:stringInclude order by $si/fts:tokenInfo/@startPos ascending, $si/fts:tokenInfo/@endPos ascending return $si where if (fn:count($sorted) le 1) then fn:true() else every $index in (1 to fn:count($sorted) - 1) satisfies fts:wordDistance( $sorted[$index]/fts:tokenInfo, $sorted[$index+1]/fts:tokenInfo ) <= $n return <fts:match> { fts:joinIncludes($match/fts:stringInclude), for $stringExcl in $match/fts:stringExclude where some $stringIncl in $match/fts:stringInclude satisfies fts:wordDistance( $stringIncl/fts:tokenInfo, $stringExcl/fts:tokenInfo ) <= $n return $stringExcl } </fts:match> } </fts:allMatches> }; The semantics of word distance from M to N declare function fts:ApplyFTWordDistanceFromTo ( $allMatches as element(fts:allMatches), $m as xs:integer, $n as xs:integer ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match let $sorted := for $si in $match/fts:stringInclude order by $si/fts:tokenInfo/@startPos ascending, $si/fts:tokenInfo/@endPos ascending return $si where if (fn:count($sorted) le 1) then fn:true() else every $index in (1 to fn:count($sorted) - 1) satisfies fts:wordDistance( $sorted[$index]/fts:tokenInfo, $sorted[$index+1]/fts:tokenInfo ) >= $m and fts:wordDistance( $sorted[$index]/fts:tokenInfo, $sorted[$index+1]/fts:tokenInfo ) <= $n return <fts:match> { fts:joinIncludes($match/fts:stringInclude), for $stringExcl in $match/fts:stringExclude where some $stringIncl in $match/fts:stringInclude satisfies fts:wordDistance( $stringIncl/fts:tokenInfo, $stringExcl/fts:tokenInfo ) >= $m and fts:wordDistance( $stringIncl/fts:tokenInfo, $stringExcl/fts:tokenInfo ) <= $n return $stringExcl } </fts:match> } </fts:allMatches> }; The preceding four helper functions all rely on fts:wordDistance TokenInfo declare function fts:wordDistance ( $tokenInfo1 as element(fts:tokenInfo), $tokenInfo2 as element(fts:tokenInfo) ) as xs:integer { (: Ensure tokens are in order :) let $sorted := for $ti in ($tokenInfo1, $tokenInfo2) order by $ti/@startPos ascending, $ti/@endPos ascending return $ti return (: -1 because we count starting at 0 :) $sorted[2]/@startPos - $sorted[1]/@endPos - 1 }; Sentence Distance The semantics of sentence distance exactly N declare function fts:ApplyFTSentenceDistanceExactly ( $allMatches as element(fts:allMatches), $n as xs:integer ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match let $sorted := for $si in $match/fts:stringInclude order by $si/fts:tokenInfo/@startSent ascending, $si/fts:tokenInfo/@endSent ascending return $si where if (fn:count($sorted) le 1) then fn:true() else every $index in (1 to fn:count($sorted) - 1) satisfies fts:sentenceDistance( $sorted[$index]/fts:tokenInfo, $sorted[$index+1]/fts:tokenInfo ) = $n return <fts:match> { fts:joinIncludes($match/fts:stringInclude), for $stringExcl in $match/fts:stringExclude where some $stringIncl in $match/fts:stringInclude satisfies fts:sentenceDistance( $stringIncl/fts:tokenInfo, $stringExcl/fts:tokenInfo ) = $n return $stringExcl } </fts:match> } </fts:allMatches> }; The semantics of sentence distance at least N declare function fts:ApplyFTSentenceDistanceAtLeast ( $allMatches as element(fts:allMatches), $n as xs:integer ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match let $sorted := for $si in $match/fts:stringInclude order by $si/fts:tokenInfo/@startSent ascending, $si/fts:tokenInfo/@endSent ascending return $si where if (fn:count($sorted) le 1) then fn:true() else every $index in (1 to fn:count($sorted) - 1) satisfies fts:sentenceDistance( $sorted[$index]/fts:tokenInfo, $sorted[$index+1]/fts:tokenInfo ) >= $n return <fts:match> { fts:joinIncludes($match/fts:stringInclude), for $stringExcl in $match/fts:stringExclude where some $stringIncl in $match/fts:stringInclude satisfies fts:sentenceDistance( $stringIncl/fts:tokenInfo, $stringExcl/fts:tokenInfo ) >= $n return $stringExcl } </fts:match> } </fts:allMatches> }; The semantics of sentence distance at most N declare function fts:ApplyFTSentenceDistanceAtMost ( $allMatches as element(fts:allMatches), $n as xs:integer ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match let $sorted := for $si in $match/fts:stringInclude order by $si/fts:tokenInfo/@startSent ascending, $si/fts:tokenInfo/@endSent ascending return $si where if (fn:count($sorted) le 1) then fn:true() else every $index in (1 to fn:count($sorted) - 1) satisfies fts:sentenceDistance( $sorted[$index]/fts:tokenInfo, $sorted[$index+1]/fts:tokenInfo ) <= $n return <fts:match> { fts:joinIncludes($match/fts:stringInclude), for $stringExcl in $match/fts:stringExclude where some $stringIncl in $match/fts:stringInclude satisfies fts:sentenceDistance( $stringIncl/fts:tokenInfo, $stringExcl/fts:tokenInfo ) <= $n return $stringExcl } </fts:match> } </fts:allMatches> }; The semantics of sentence distance from M to N declare function fts:ApplyFTSentenceDistanceFromTo ( $allMatches as element(fts:allMatches), $m as xs:integer, $n as xs:integer ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match let $sorted := for $si in $match/fts:stringInclude order by $si/fts:tokenInfo/@startSent ascending, $si/fts:tokenInfo/@endSent ascending return $si where if (fn:count($sorted) le 1) then fn:true() else every $index in (1 to fn:count($sorted) - 1) satisfies fts:sentenceDistance( $sorted[$index]/fts:tokenInfo, $sorted[$index+1]/fts:tokenInfo ) >= $m and fts:sentenceDistance( $sorted[$index]/fts:tokenInfo, $sorted[$index+1]/fts:tokenInfo ) <= $n return <fts:match> { fts:joinIncludes($match/fts:stringInclude), for $stringExcl in $match/fts:stringExclude where some $stringIncl in $match/fts:stringInclude satisfies fts:sentenceDistance( $stringIncl/fts:tokenInfo, $stringExcl/fts:tokenInfo ) >= $m and fts:sentenceDistance( $stringIncl/fts:tokenInfo, $stringExcl/fts:tokenInfo ) <= $n return $stringExcl } </fts:match> } </fts:allMatches> }; The preceding four helper functions all rely on fts:sentenceDistance TokenInfo declare function fts:sentenceDistance ( $tokenInfo1 as element(fts:tokenInfo), $tokenInfo2 as element(fts:tokenInfo) ) as xs:integer { (: Ensure tokens are in order :) let $sorted := for $ti in ($tokenInfo1, $tokenInfo2) order by $ti/@startPos ascending, $ti/@endPos ascending return $ti return (: -1 because we count starting at 0 :) $sorted[2]/@startSent - $sorted[1]/@endSent - 1 }; Paragraph Distance The semantics of paragraph distance exactly N declare function fts:ApplyFTParagraphDistanceExactly ( $allMatches as element(fts:allMatches), $n as xs:integer ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match let $sorted := for $si in $match/fts:stringInclude order by $si/fts:tokenInfo/@startPara ascending, $si/fts:tokenInfo/@endPara ascending return $si where if (fn:count($sorted) le 1) then fn:true() else every $index in (1 to fn:count($sorted) - 1) satisfies fts:paraDistance( $sorted[$index]/fts:tokenInfo, $sorted[$index+1]/fts:tokenInfo ) = $n return <fts:match> { fts:joinIncludes($match/fts:stringInclude), for $stringExcl in $match/fts:stringExclude where some $stringIncl in $match/fts:stringInclude satisfies fts:paraDistance( $stringIncl/fts:tokenInfo, $stringExcl/fts:tokenInfo ) = $n return $stringExcl } </fts:match> } </fts:allMatches> }; The semantics of paragraph distance at least N declare function fts:ApplyFTParagraphDistanceAtLeast ( $allMatches as element(fts:allMatches), $n as xs:integer ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match let $sorted := for $si in $match/fts:stringInclude order by $si/fts:tokenInfo/@startPara ascending, $si/fts:tokenInfo/@endPara ascending return $si where if (fn:count($sorted) le 1) then fn:true() else every $index in (1 to fn:count($sorted) - 1) satisfies fts:paraDistance( $sorted[$index]/fts:tokenInfo, $sorted[$index+1]/fts:tokenInfo ) >= $n return <fts:match> { fts:joinIncludes($match/fts:stringInclude), for $stringExcl in $match/fts:stringExclude where some $stringIncl in $match/fts:stringInclude satisfies fts:paraDistance( $stringIncl/fts:tokenInfo, $stringExcl/fts:tokenInfo ) >= $n return $stringExcl } </fts:match> } </fts:allMatches> }; The semantics of paragraph distance at most N declare function fts:ApplyFTParagraphDistanceAtMost ( $allMatches as element(fts:allMatches), $n as xs:integer ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match let $sorted := for $si in $match/fts:stringInclude order by $si/fts:tokenInfo/@startPara ascending, $si/fts:tokenInfo/@endPara ascending return $si where if (fn:count($sorted) le 1) then fn:true() else every $index in (1 to fn:count($sorted) - 1) satisfies fts:paraDistance( $sorted[$index]/fts:tokenInfo, $sorted[$index+1]/fts:tokenInfo ) <= $n return <fts:match> { fts:joinIncludes($match/fts:stringInclude), for $stringExcl in $match/fts:stringExclude where some $stringIncl in $match/fts:stringInclude satisfies fts:paraDistance( $stringIncl/fts:tokenInfo, $stringExcl/fts:tokenInfo ) <= $n return $stringExcl } </fts:match> } </fts:allMatches> }; The semantics of paragraph distance from M to N declare function fts:ApplyFTParagraphDistanceFromTo ( $allMatches as element(fts:allMatches), $m as xs:integer, $n as xs:integer ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match let $sorted := for $si in $match/fts:stringInclude order by $si/fts:tokenInfo/@startPara ascending, $si/fts:tokenInfo/@endPara ascending return $si where if (fn:count($sorted) le 1) then fn:true() else every $index in (1 to fn:count($sorted) - 1) satisfies fts:paraDistance( $sorted[$index]/fts:tokenInfo, $sorted[$index+1]/fts:tokenInfo ) >= $m and fts:paraDistance( $sorted[$index]/fts:tokenInfo, $sorted[$index+1]/fts:tokenInfo ) <= $n return <fts:match> { fts:joinIncludes($match/fts:stringInclude), for $stringExcl in $match/fts:stringExclude where some $stringIncl in $match/fts:stringInclude satisfies fts:paraDistance( $stringIncl/fts:tokenInfo, $stringExcl/fts:tokenInfo ) >= $m and fts:paraDistance( $stringIncl/fts:tokenInfo, $stringExcl/fts:tokenInfo ) <= $n return $stringExcl } </fts:match> } </fts:allMatches> }; The preceding four helper functions all rely on fts:paraDistance TokenInfo declare function fts:paraDistance ( $tokenInfo1 as element(fts:tokenInfo), $tokenInfo2 as element(fts:tokenInfo) ) as xs:integer { (: Ensure tokens are in order :) let $sorted := for $ti in ($tokenInfo1, $tokenInfo2) order by $ti/@startPos ascending, $ti/@endPos ascending return $ti return (: -1 because we count starting at 0 :) $sorted[2]/@startPara - $sorted[1]/@endPara - 1 }; For example, consider the FTDistance ("Ford Mustang" ftand "excellent") distance at most 3 words Match AllMatches ("Ford Mustang" ftand "excellent") The result for the FTDistance Match Match Match TokenInfo Match TokenInfo startPos endPos Match startPos endPos The parameters of the ApplyFTTimes FTRange FTWords The function definitions depend on the range specification FTRange The general semantics is given below. declare function fts:FormCombinations ( $sms as element(fts:match)*, $k as xs:integer ) as element(fts:match)* (: Find all combinations of exactly $k elements from $sms, and for each such combination, construct a match whose children are copies of all the children of all the elements in the combination. Return the sequence of all such matches. :) { if ($k eq 0) then <fts:match/> else if (fn:count($sms) lt $k) then () else if (fn:count($sms) eq $k) then <fts:match>{$sms/*}</fts:match> else let $first := $sms[1], $rest := fn:subsequence($sms, 2) return ( (: all the combinations that don't involve $first :) fts:FormCombinations($rest, $k),

(: and all the combinations that do involve $first :) for $combination in fts:FormCombinations($rest, $k - 1) return <fts:match> { $first/*, $combination/* } </fts:match> ) };

declare function fts:FormCombinationsAtLeast ( $sms as element(fts:match)*, $times as xs:integer) as element(fts:match)* (: Find all combinations of $times or more elements from $sms, and for each such combination, construct a match whose children are copies of all the children of all the elements in the combination. Return the sequence of all such matches. :) { for $k in $times to fn:count($sms) return fts:FormCombinations($sms, $k) };

declare function fts:FormRange ( $sms as element(fts:match)*, $l as xs:integer, $u as xs:integer, $stokenNum as xs:integer ) as element(fts:allMatches) { if ($l > $u) then <fts:allMatches stokenNum="0" /> else let $am1 := <fts:allMatches stokenNum="{$stokenNum}"> {fts:FormCombinationsAtLeast($sms, $l)} </fts:allMatches> let $am2 := <fts:allMatches stokenNum="{$stokenNum}"> {fts:FormCombinationsAtLeast($sms, $u+1)} </fts:allMatches> return fts:ApplyFTAnd($am1, fts:ApplyFTUnaryNot($am2)) }; The semantics of occurs exactly N times declare function fts:ApplyFTTimesExactly ( $allMatches as element(fts:allMatches), $n as xs:integer ) as element(fts:allMatches) { fts:FormRange($allMatches/fts:match, $n, $n, $allMatches/@stokenNum) }; The semantics of occurs at least N times declare function fts:ApplyFTTimesAtLeast ( $allMatches as element(fts:allMatches), $n as xs:integer ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> {fts:FormCombinationsAtLeast($allMatches/fts:match, $n)} </fts:allMatches> }; The semantics of occurs at most N times declare function fts:ApplyFTTimesAtMost ( $allMatches as element(fts:allMatches), $n as xs:integer ) as element(fts:allMatches) { fts:FormRange($allMatches/fts:match, 0, $n, $allMatches/@stokenNum) }; The semantics of occurs from M to N times declare function fts:ApplyFTTimesFromTo ( $allMatches as element(fts:allMatches), $m as xs:integer, $n as xs:integer ) as element(fts:allMatches) { fts:FormRange($allMatches/fts:match, $m, $n, $allMatches/@stokenNum) }; The way to ensure that there are at least N FTSelection N Match N Match AllMatches at least N N fts:FormCombinationsAtLeast The range [L, U] is represented by the condition at least L and not at least U+1 fts:FormRange The semantics for the general case is given below. declare function fts:ApplyFTTimes ( $range as element(fts:range), $allMatches as element(fts:allMatches) ) as element(fts:allMatches) { if (fn:count($allMatches//fts:stringExclude) gt 0) then fn:error(fn:QName('http://www.w3.org/2005/xqt-errors', 'XPST0003')) else if ($range/@type eq "exactly") then fts:ApplyFTTimesExactly($allMatches, $range/@n) else if ($range/@type eq "at least") then fts:ApplyFTTimesAtLeast($allMatches, $range/@n) else if ($range/@type eq "at most") then fts:ApplyFTTimesAtMost($allMatches, $range/@n) else fts:ApplyFTTimesFromTo($allMatches, $range/@m, $range/@n) }; The above function performs a sanity check to ensure that the nested AllMatches FTWords FTPrimary err:XPST0003 XP For example, consider the FTTimes "Mustang" occurs at least 2 times AllMatches FTWords "Mustang" The result consists of the pairs of the Match Consider an FTContainsExpr SearchContext contains text FTSelection SearchContext FTContainsExpr FTSelection If the FTContainsExpr SearchContext contains text FTSelection without content IgnoreExpr IgnoreExpr IgnoreExpr FTSelection More formally, evaluation of an FTContainsExpr 2.1 Processing Model For each XQuery/XPath expression nested within the FTContainsExpr, evaluate it with respect to the same dynamic context as the FTContainsExpr (FT1). Specifically: Evaluate the search context expression ( SearchContext Evaluate the ignore option ( IgnoreExpr At each FTWordsValue, evaluate the literal/expression and convert the result to xs:string* At each weight specification, evaluate the expression and convert the result to xs:double At each FTWindow and FTRange, evaluate the AdditiveExpr(s) and convert each to xs:integer Using the settings of the match option components in the FTContainsExpr's static context, construct an element(fts:matchOptions) Based on the parse-tree of the FTContainsExpr's FTSelection and the results of steps 1c-1e, construct an element(*,fts:ftSelection) Construct the operator tree from the top down, propagating FTMatchOptions down to FTWordsValues. Tokenize the query string(s) obtained at 1c. (FT2.1) Call the function fts:FTContainsExpr $searchContextItems SearchContext $ignoreNodes IgnoreExpr $ftSelection FTSelection $defOptions Within the function, for each search context item: Delete the ignored nodes from the search context item. [ fts:FTContainsExpr fts:reconstruct Traverse the operator tree from the top down, propagating FTMatchOptions down to FTWordsValues. [ fts:evaluate fts:replaceMatchOptions At each FTWordsValue, using the prevailing FTMatchOptions: Tokenize the search context obtained at 4a. (FT2.2) (Whether this pays any attention to FTMatchOptions is up to the implementation.) [This happens within fts:matchTokenInfos Match the search context tokens and the query tokens, yielding an element(fts:tokenInfo)* fts:matchTokenInfos Convert that into an element(fts:allMatches) fts:applyQueryTokensAsPhrase Traverse the operator tree from the bottom up. At each point, the AllMatches AllMatches If the topmost AllMatches Match StringExclude FTSelection fts:FTContainsExpr true fts:FTContainsExpr [Note that the section 4 code doesn't implement 4b-4d as three sequential steps. Instead, they are different aspects of a single traversal of the operator tree.] If none of the topmost AllMatches fts:FTContainsExpr false The boolean value returned by the call to fts:FTContainsExpr declare function fts:FTContainsExpr ( $searchContextItems as item()*, $ignoreNodes as node()*, $ftSelection as element(*,fts:ftSelection), $defOptions as element(fts:matchOptions) ) as xs:boolean { some $searchContext in $searchContextItems satisfies let $newSearchContext := fts:reconstruct( $searchContext, $ignoreNodes ) return if (fn:empty($newSearchContext)) then fn:false() else let $allMatches := fts:evaluate($ftSelection, $newSearchContext, $defOptions, 0) return some $match in $allMatches/fts:match satisfies fn:count($match/fts:stringExclude) eq 0 };

declare function fts:reconstruct ( $n as item(), $ignore as node()* ) as item()? { typeswitch ($n) case node() return if (some $i in $ignore satisfies $n is $i) then () else if ($n instance of element()) then let $nodeName := fn:node-name($n) let $nodeContent := for $nn in $n/node() return fts:reconstruct($nn,$ignore) return element {$nodeName} {$nodeContent} else if ($n instance of document-node()) then document { for $nn in $n/node() return fts:reconstruct($nn, $ignore) } else $n default return $n }; This section addresses the semantics of scoring variables in XQuery 1.0 for let for Scoring variables associate a numeric score with the result of the evaluation of XQuery 1.0 and XPath 2.0 expressions. This numeric score tries to estimate the value of a result item to the user information need expressed using the XQuery 1.0 and XPath 2.0 expression. The numeric score is computed using an implementation-dependent scoring algorithm There are numerous scoring algorithms used in practice. Most of the scoring algorithms take as inputs a query and a set of results to the query. In computing the score, these algorithms rely on the structure of the query to estimate the relevance of the results. In the context of defining the semantics of XQuery and XPath Full Text, passing the structure of the query poses a problem. The query may contain XQuery 1.0 and XPath 2.0 expressions and XQuery and XPath Full Text 1.0 expressions in particular. The semantics of XQuery 1.0 and XPath 2.0 expressions is defined using (among other things) functions that take as arguments sequences of items and return sequences of items. They are not aware of what expression produced a particular sequence, i.e., they are not aware of the expression structure. To define the semantics of scoring in XQuery and XPath Full Text 1.0 using XQuery 1.0, expressions that produce the query result (or the functions that implement the expressions) must be passed as arguments. In other words, second-order functions are necessary. Currently XQuery 1.0 and XPath 2.0 do not provide such functions. Nevertheless, in the interest of the exposition, assume that such second-order functions are present. In particular, that there are two semantic second-order function fts:score fts:scoreSequence scoring properties A for for $result score $score in Expr ... is evaluated as though it is replaced by the following the set of clauses. let $scoreSeq := fts:scoreSequence(Expr) for $result at $i in Expr let $score := $scoreSeq[$i] ... Here, $scoreSeq $i fts:scoreSequence Similarly, a let let score $score := Expr ... is evaluated as though it is replaced by the following clause. let $score := fts:score(Expr) ... This section presents a more complex example for the evaluation of FTContainsExpr $doc FTContainsExpr $doc contains text ( ( "Mustang" ftand ({("great", "excellent")} any word occurs at least 2 times) window 11 words ) ftand ftnot "rust" ) same paragraph Begin by evaluating the FTSelection AllMatches ( ( "Mustang" ftand ({("great", "excellent")} any word occurs at least 2 times) window 11 words ) ftand ftnot "rust" ) same paragraph Step 1: Evaluate the FTWords "Mustang" Step 2: Evaluate the FTWords {"great", "excellent"} any word Step 2.1: Match the token "great" Step 2.2 Match the token "excellent" Step 2.3 - Combine the above AllMatches FTOr Match Step 3 - Apply the FTTimes {("great", "excellent")} any word occurs at least 2 times Match Step 4 - Apply the FTAnd "Mustang" ftand ({("great", "excellent")} any word occurs at least 2 times) StringMatch Step 5 - Apply the FTWindow ( "Mustang" ftand ({("great", "excellent")} any word occurs at least 2 times) window 11 words ) Match Step 6 - Evaluate FTWords "rust" Step 7 - Apply the FTUnaryNot ftnot "rust" StringInclude StringExclude Step 8 - Apply the FTAnd ( ( "Mustang" ftand ({("great", "excellent")} any word occurs at least 2 times) window 11 words ) ftand ftnot "rust" ) StringMatch AllMatches StringMatch AllMatches Step 9: Apply the FTScope Match TokenInfo <offer> The resulting AllMatches Match StringExclude FTContainsExpr true This section defines the conformance criteria for a XQuery and XPath Full Text 1.0 processor. In this section, the following terms are used to indicate the requirement levels defined in [RFC 2119] Definition MUST Definition MAY Definition SHOULD An XQuery and XPath Full Text 1.0 processor that claims to conform to this specification MUST 5.1 Minimal Conformance MAY 5.2 Optional Features Minimal Conformance to this specification MUST Minimal support for XQuery 1.0 [XQuery 1.0: An XML Query Language (Second Edition)] [XML Path Language (XPath) 2.0 (Second Edition)] [XQuery 1.0: An XML Query Language (Second Edition)] [XML Path Language (XPath) 2.0 (Second Edition)] MAY Support for everything specified in this document except those operators and match options specified in 5.2 Optional Features MUST 5.2 Optional Features A definition of every item specified to be implementation-defined I Checklist of Implementation-Defined Features Note: Implementations are not required to define items specified to be implementation-dependent It is optional whether the implementation supports the FTMildNot. If it does not support FTMildNot and encounters one in a full-text query, then it MUST err:FTST0001 The unrestricted form of negation in FTUnaryNot, that can negate every kind of FTSelection, is optional. Implementations may choose to support the negation operation in a restricted form, enforcing one or both of the following restrictions. [ Definition Negation Restriction 1. FTUnaryNot FTAnd [ Definition Negation Restriction 2. FTUnaryNot FTOr FTPosFilter FTSelection Consider the following example FTSelections. 1. ftnot "web"

2. "web" ftand ( ftnot "information" ftor "retrieval" )

3. "web" ftand ftnot("information" ftand "retrieval")

4. "web" ftand ftnot("information" ftand "retrieval" window 5 words)

5. "web" ftand ("information" ftand ftnot "retrieval" window 5 words) The first two FTSelections both violate restriction 1, while the third and the fourth are conform with both restrictions. The fifth one violates restriction 2, while obeying restriction 1. Note that in the last example the FTSelection to which the window operation is applied is "information" ftand ftnot "retrieval" If the implementation does enforce one or both of these restrictions on FTUnaryNot and encounters a full-text query that does not obey the restriction then it MUST err:FTST0002 Support for the "sentences" alternative of FTUnit FTBigUnit MUST err:FTST0003 The unrestricted form of the FTOrder postfix operator, that can be applied to any kind of FTSelection, is optional. Implementations may choose to enforce the following restriction on the use of FTOrder. [ Definition Order Operator Restriction. If the implementation does enforce this restriction and encounters a full-text query that does not obey the restriction then it MUST err:FTST0010 It is optional whether the implementation supports the FTScope operator. If it does not support FTScope and encounters one in a full-text query, then it MUST err:FTST0004 The unrestricted form of the FTWindow postfix operator, that can be applied to any kind of FTSelection, is optional. Implementations may choose to enforce the following restriction on the use of FTWindow. [ Definition Window Operator Restriction. If the implementation does enforce this restriction and encounters a full-text query that does not obey the restriction then it MUST err:FTST0011 The unrestricted form of the FTDistance postfix operator, that can be applied to any kind of FTSelection, is optional. Implementations may choose to enforce the following restriction on the use of FTDistance. [ Definition Distance Operator Restriction. If the implementation does enforce this restriction and encounters a full-text query that does not obey the restriction then it MUST err:FTST0011 It is optional whether the implementation supports the FTTimes operator. If it does not support FTTimes and encounters one in a full-text query, then it MUST err:FTST0005 It is optional whether the implementation supports the FTContent operator. If it does not support FTContent and encounters one in a full-text query, then it MUST err:FTST0012 It is optional whether the implementation supports the "lowercase" and "uppercase" choices for the FTCaseOption. If it does not support these choices for the FTCaseOption and encounters an unsupported choice in a full-text query, then it MUST err:FTST0015 It is optional whether the implementation supports the FTStopWordOption. If it does not support FTStopWordOption and encounters one in a full-text query, then it MUST err:FTST0006 It is optional whether the implementation supports the FTStopWordOption in the body of the query. If it supports FTStopWordOption in the prolog, but not in the body of a query, and encounters one in the body of a query it MUST err:FTST0006 It is optional whether the implementation supports the StringLiteral alternative of FTStopWords MUST err:FTST0006 It is optional whether the implementation supports the unrestricted form of FTLanguageOption. Implementations may choose to enforce the following restriction on the use of FTLanguageOption. [ Definition Single Language Restriction. If the implementation does enforce this restriction and encounters a full-text query that does not obey the restriction then it MUST err:FTST0013 The implementation may constrain the set of ignored nodes. If the operand of FTIgnoreOption implementation-defined MUST err:FTST0007 The implementation may restrict the allowable expressions used to compute scores. The restrictions are implementation-defined If the implementation does enforce such restrictions and encounters a full-text query that does not obey the restriction then it MUST err:FTST0014 An implementation may constrain the range of valid weights to non-negative values. If an implementation does enforce this restriction and encounters a full-text query that uses a negative weight, it MUST err:FTDY0016 This section defines the conformance criteria for an XQueryX processor that includes the Full Text capability. In this section, the terms MUST MAY SHOULD 5 Conformance An XQueryX processor that claims to conform to this specification MUST E XML Syntax (XQueryX) for XQuery and XPath Full Text 1.0 5.1 Minimal Conformance MAY 5.2 Optional Features The EBNF in this document and in this section is aligned with the current XML Query 1.0 grammar (see http://www.w3.org/TR/2010/REC-xquery-20101214/ Module ::= VersionDecl LibraryModule MainModule VersionDecl ::= "xquery" "version" StringLiteral StringLiteral Separator MainModule ::= Prolog QueryBody LibraryModule ::= ModuleDecl Prolog ModuleDecl ::= "module" "namespace" NCName URILiteral Separator Prolog ::= (( DefaultNamespaceDecl Setter NamespaceDecl Import FTOptionDecl Separator VarDecl FunctionDecl OptionDecl Separator Separator ::= ";" Setter ::= BoundarySpaceDecl DefaultCollationDecl BaseURIDecl ConstructionDecl OrderingModeDecl EmptyOrderDecl CopyNamespacesDecl BoundarySpaceDecl ::= "declare" "boundary-space" ("preserve" | "strip") DefaultCollationDecl ::= "declare" "default" "collation" URILiteral BaseURIDecl ::= "declare" "base-uri" URILiteral ConstructionDecl ::= "declare" "construction" ("strip" | "preserve") OrderingModeDecl ::= "declare" "ordering" ("ordered" | "unordered") EmptyOrderDecl ::= "declare" "default" "order" "empty" ("greatest" | "least") CopyNamespacesDecl ::= "declare" "copy-namespaces" PreserveMode InheritMode PreserveMode ::= "preserve" | "no-preserve" InheritMode ::= "inherit" | "no-inherit" Import ::= SchemaImport ModuleImport SchemaImport ::= "import" "schema" SchemaPrefix URILiteral URILiteral URILiteral SchemaPrefix ::= ("namespace" NCName ModuleImport ::= "import" "module" ("namespace" NCName URILiteral URILiteral URILiteral NamespaceDecl ::= "declare" "namespace" NCName URILiteral DefaultNamespaceDecl ::= "declare" "default" ("element" | "function") "namespace" URILiteral FTOptionDecl ::= "declare" "ft-option" FTMatchOptions VarDecl ::= "declare" "variable" "$" QName TypeDeclaration ExprSingle FunctionDecl ::= "declare" "function" QName ParamList SequenceType EnclosedExpr /* xgc: reserved-function-names XQ ParamList ::= Param Param Param ::= "$" QName TypeDeclaration EnclosedExpr ::= "{" Expr OptionDecl ::= "declare" "option" QName StringLiteral QueryBody ::= Expr Expr ::= ExprSingle ExprSingle ExprSingle ::= FLWORExpr QuantifiedExpr TypeswitchExpr IfExpr OrExpr FLWORExpr ::= ( ForClause LetClause WhereClause OrderByClause ExprSingle ForClause ::= "for" "$" VarName TypeDeclaration PositionalVar FTScoreVar ExprSingle VarName TypeDeclaration PositionalVar FTScoreVar ExprSingle PositionalVar ::= "at" "$" VarName FTScoreVar ::= "score" "$" VarName LetClause ::= "let" (("$" VarName TypeDeclaration FTScoreVar ExprSingle VarName TypeDeclaration FTScoreVar ExprSingle WhereClause ::= "where" ExprSingle OrderByClause ::= (("order" "by") | ("stable" "order" "by")) OrderSpecList OrderSpecList ::= OrderSpec OrderSpec OrderSpec ::= ExprSingle OrderModifier OrderModifier ::= ("ascending" | "descending")? ("empty" ("greatest" | "least"))? ("collation" URILiteral QuantifiedExpr ::= ("some" | "every") "$" VarName TypeDeclaration ExprSingle VarName TypeDeclaration ExprSingle ExprSingle TypeswitchExpr ::= "typeswitch" "(" Expr CaseClause VarName ExprSingle CaseClause ::= "case" ("$" VarName SequenceType ExprSingle IfExpr ::= "if" "(" Expr ExprSingle ExprSingle OrExpr ::= AndExpr AndExpr AndExpr ::= ComparisonExpr ComparisonExpr ComparisonExpr ::= FTContainsExpr ValueComp GeneralComp NodeComp FTContainsExpr FTContainsExpr ::= RangeExpr FTSelection FTIgnoreOption RangeExpr ::= AdditiveExpr AdditiveExpr AdditiveExpr ::= MultiplicativeExpr MultiplicativeExpr MultiplicativeExpr ::= UnionExpr UnionExpr UnionExpr ::= IntersectExceptExpr IntersectExceptExpr IntersectExceptExpr ::= InstanceofExpr InstanceofExpr InstanceofExpr ::= TreatExpr SequenceType TreatExpr ::= CastableExpr SequenceType CastableExpr ::= CastExpr SingleType CastExpr ::= UnaryExpr SingleType UnaryExpr ::= ("-" | "+")* ValueExpr ValueExpr ::= ValidateExpr PathExpr ExtensionExpr GeneralComp ::= "=" | "!=" | "<" | "<=" | ">" | ">=" ValueComp ::= "eq" | "ne" | "lt" | "le" | "gt" | "ge" NodeComp ::= "is" | "<<" | ">>" ValidateExpr ::= "validate" ValidationMode Expr ValidationMode ::= "lax" | "strict" ExtensionExpr ::= Pragma Expr Pragma ::= "(#" S QName S PragmaContents /* ws: explicit XQ PragmaContents ::= ( Char PathExpr ::= ("/" RelativePathExpr RelativePathExpr RelativePathExpr /* xgc: leading-lone-slash XQ RelativePathExpr ::= StepExpr StepExpr StepExpr ::= FilterExpr AxisStep AxisStep ::= ( ReverseStep ForwardStep PredicateList ForwardStep ::= ( ForwardAxis NodeTest AbbrevForwardStep ForwardAxis ::= ("child" "::") AbbrevForwardStep ::= "@"? NodeTest ReverseStep ::= ( ReverseAxis NodeTest AbbrevReverseStep ReverseAxis ::= ("parent" "::") AbbrevReverseStep ::= ".." NodeTest ::= KindTest NameTest NameTest ::= QName Wildcard Wildcard ::= "*" NCName NCName /* ws: explicit XQ FilterExpr ::= PrimaryExpr PredicateList PredicateList ::= Predicate Predicate ::= "[" Expr PrimaryExpr ::= Literal VarRef ParenthesizedExpr ContextItemExpr FunctionCall OrderedExpr UnorderedExpr Constructor Literal ::= NumericLiteral StringLiteral NumericLiteral ::= IntegerLiteral DecimalLiteral DoubleLiteral VarRef ::= "$" VarName VarName ::= QName ParenthesizedExpr ::= "(" Expr ContextItemExpr ::= "." OrderedExpr ::= "ordered" "{" Expr UnorderedExpr ::= "unordered" "{" Expr FunctionCall ::= QName ExprSingle ExprSingle /* xgc: reserved-function-names XQ /* gn: parens XQ Constructor ::= DirectConstructor ComputedConstructor DirectConstructor ::= DirElemConstructor DirCommentConstructor DirPIConstructor DirElemConstructor ::= "<" QName DirAttributeList DirElemContent QName S /* ws: explicit XQ DirAttributeList ::= ( S QName S S DirAttributeValue /* ws: explicit XQ DirAttributeValue ::= ('"' ( EscapeQuot QuotAttrValueContent EscapeApos AposAttrValueContent /* ws: explicit XQ QuotAttrValueContent ::= QuotAttrContentChar CommonContent AposAttrValueContent ::= AposAttrContentChar CommonContent DirElemContent ::= DirectConstructor CDataSection CommonContent ElementContentChar CommonContent ::= PredefinedEntityRef CharRef EnclosedExpr DirCommentConstructor ::= "<!--" DirCommentContents /* ws: explicit XQ DirCommentContents ::= (( Char Char /* ws: explicit XQ DirPIConstructor ::= "<?" PITarget S DirPIContents /* ws: explicit XQ DirPIContents ::= ( Char /* ws: explicit XQ CDataSection ::= "<![CDATA[" CDataSectionContents /* ws: explicit XQ CDataSectionContents ::= ( Char /* ws: explicit XQ ComputedConstructor ::= CompDocConstructor CompElemConstructor CompAttrConstructor CompTextConstructor CompCommentConstructor CompPIConstructor CompDocConstructor ::= "document" "{" Expr CompElemConstructor ::= "element" ( QName Expr ContentExpr ContentExpr ::= Expr CompAttrConstructor ::= "attribute" ( QName Expr Expr CompTextConstructor ::= "text" "{" Expr CompCommentConstructor ::= "comment" "{" Expr CompPIConstructor ::= "processing-instruction" ( NCName Expr Expr SingleType ::= AtomicType TypeDeclaration ::= "as" SequenceType SequenceType ::= ("empty-sequence" "(" ")") ItemType OccurrenceIndicator OccurrenceIndicator ::= "?" | "*" | "+" /* xgc: occurrence-indicators XQ ItemType ::= KindTest AtomicType AtomicType ::= QName KindTest ::= DocumentTest ElementTest AttributeTest SchemaElementTest SchemaAttributeTest PITest CommentTest TextTest AnyKindTest AnyKindTest ::= "node" "(" ")" DocumentTest ::= "document-node" "(" ( ElementTest SchemaElementTest TextTest ::= "text" "(" ")" CommentTest ::= "comment" "(" ")" PITest ::= "processing-instruction" "(" ( NCName StringLiteral AttributeTest ::= "attribute" "(" ( AttribNameOrWildcard TypeName AttribNameOrWildcard ::= AttributeName SchemaAttributeTest ::= "schema-attribute" "(" AttributeDeclaration AttributeDeclaration ::= AttributeName ElementTest ::= "element" "(" ( ElementNameOrWildcard TypeName ElementNameOrWildcard ::= ElementName SchemaElementTest ::= "schema-element" "(" ElementDeclaration ElementDeclaration ::= ElementName AttributeName ::= QName ElementName ::= QName TypeName ::= QName URILiteral ::= StringLiteral FTSelection ::= FTOr FTPosFilter FTWeight ::= "weight" "{" Expr FTOr ::= FTAnd FTAnd FTAnd ::= FTMildNot FTMildNot FTMildNot ::= FTUnaryNot FTUnaryNot FTUnaryNot ::= ("ftnot")? FTPrimaryWithOptions FTPrimaryWithOptions ::= FTPrimary FTMatchOptions FTWeight FTPrimary ::= ( FTWords FTTimes FTSelection FTExtensionSelection FTWords ::= FTWordsValue FTAnyallOption FTWordsValue ::= StringLiteral Expr FTExtensionSelection ::= Pragma FTSelection FTAnyallOption ::= ("any" "word"?) | ("all" "words"?) | "phrase" FTTimes ::= "occurs" FTRange FTRange ::= ("exactly" AdditiveExpr AdditiveExpr AdditiveExpr AdditiveExpr AdditiveExpr FTPosFilter ::= FTOrder FTWindow FTDistance FTScope FTContent FTOrder ::= "ordered" FTWindow ::= "window" AdditiveExpr FTUnit FTDistance ::= "distance" FTRange FTUnit FTUnit ::= "words" | "sentences" | "paragraphs" FTScope ::= ("same" | "different") FTBigUnit FTBigUnit ::= "sentence" | "paragraph" FTContent ::= ("at" "start") | ("at" "end") | ("entire" "content") FTMatchOptions ::= ("using" FTMatchOption FTMatchOption ::= FTLanguageOption FTWildCardOption FTThesaurusOption FTStemOption FTCaseOption FTDiacriticsOption FTStopWordOption FTExtensionOption FTCaseOption ::= ("case" "insensitive") FTDiacriticsOption ::= ("diacritics" "insensitive") FTStemOption ::= "stemming" | ("no" "stemming") FTThesaurusOption ::= ("thesaurus" ( FTThesaurusID FTThesaurusID FTThesaurusID FTThesaurusID ::= "at" URILiteral StringLiteral FTLiteralRange FTLiteralRange ::= ("exactly" IntegerLiteral IntegerLiteral IntegerLiteral IntegerLiteral IntegerLiteral FTStopWordOption ::= ("stop" "words" FTStopWords FTStopWordsInclExcl FTStopWordsInclExcl FTStopWords ::= ("at" URILiteral StringLiteral StringLiteral FTStopWordsInclExcl ::= ("union" | "except") FTStopWords FTLanguageOption ::= "language" StringLiteral FTWildCardOption ::= "wildcards" | ("no" "wildcards") FTExtensionOption ::= "option" QName StringLiteral FTIgnoreOption ::= "without" "content" UnionExpr IntegerLiteral ::= Digits DecimalLiteral ::= ("." Digits Digits /* ws: explicit XQ DoubleLiteral ::= (("." Digits Digits Digits /* ws: explicit XQ StringLiteral ::= ('"' ( PredefinedEntityRef CharRef EscapeQuot PredefinedEntityRef CharRef EscapeApos /* ws: explicit XQ PredefinedEntityRef ::= "&" ("lt" | "gt" | "amp" | "quot" | "apos") ";" /* ws: explicit XQ EscapeQuot ::= '""' EscapeApos ::= "''" ElementContentChar ::= ( Char QuotAttrContentChar ::= ( Char AposAttrContentChar ::= ( Char Comment ::= "(:" ( CommentContents Comment /* ws: explicit XQ /* gn: comments XQ PITarget ::= [http://www.w3.org/TR/REC-xml#NT-PITarget] XML /* xgc: xml-version XQ CharRef ::= [http://www.w3.org/TR/REC-xml#NT-CharRef] XML /* xgc: xml-version XQ QName ::= [http://www.w3.org/TR/REC-xml-names/#NT-QName] Names /* xgc: xml-version XQ NCName ::= [http://www.w3.org/TR/REC-xml-names/#NT-NCName] Names /* xgc: xml-version XQ S ::= [http://www.w3.org/TR/REC-xml#NT-S] XML /* xgc: xml-version XQ Char ::= [http://www.w3.org/TR/REC-xml#NT-Char] XML /* xgc: xml-version XQ The following symbols are used only in the definition of terminal symbols; they are not terminal symbols in the grammar of A EBNF for XQuery 1.0 Grammar with Full Text extensions Digits ::= [0-9]+ CommentContents ::= ( Char The EBNF in this document and in this section is aligned with the current XPath 2.0 grammar (see http://www.w3.org/TR/2010/REC-xpath20-20101214/ XPath ::= Expr Expr ::= ExprSingle ExprSingle ExprSingle ::= ForExpr QuantifiedExpr IfExpr OrExpr ForExpr ::= SimpleForClause ExprSingle SimpleForClause ::= "for" "$" VarName FTScoreVar ExprSingle VarName FTScoreVar ExprSingle FTScoreVar ::= "score" "$" VarName QuantifiedExpr ::= ("some" | "every") "$" VarName ExprSingle VarName ExprSingle ExprSingle IfExpr ::= "if" "(" Expr ExprSingle ExprSingle OrExpr ::= AndExpr AndExpr AndExpr ::= ComparisonExpr ComparisonExpr ComparisonExpr ::= FTContainsExpr ValueComp GeneralComp NodeComp FTContainsExpr FTContainsExpr ::= RangeExpr FTSelection FTIgnoreOption RangeExpr ::= AdditiveExpr AdditiveExpr AdditiveExpr ::= MultiplicativeExpr MultiplicativeExpr MultiplicativeExpr ::= UnionExpr UnionExpr UnionExpr ::= IntersectExceptExpr IntersectExceptExpr IntersectExceptExpr ::= InstanceofExpr InstanceofExpr InstanceofExpr ::= TreatExpr SequenceType TreatExpr ::= CastableExpr SequenceType CastableExpr ::= CastExpr SingleType CastExpr ::= UnaryExpr SingleType UnaryExpr ::= ("-" | "+")* ValueExpr ValueExpr ::= PathExpr GeneralComp ::= "=" | "!=" | "<" | "<=" | ">" | ">=" ValueComp ::= "eq" | "ne" | "lt" | "le" | "gt" | "ge" NodeComp ::= "is" | "<<" | ">>" Pragma ::= "(#" S QName S PragmaContents /* ws: explicit XP PragmaContents ::= ( Char PathExpr ::= ("/" RelativePathExpr RelativePathExpr RelativePathExpr /* xgc: leading-lone-slash XP RelativePathExpr ::= StepExpr StepExpr StepExpr ::= FilterExpr AxisStep AxisStep ::= ( ReverseStep ForwardStep PredicateList ForwardStep ::= ( ForwardAxis NodeTest AbbrevForwardStep ForwardAxis ::= ("child" "::") AbbrevForwardStep ::= "@"? NodeTest ReverseStep ::= ( ReverseAxis NodeTest AbbrevReverseStep ReverseAxis ::= ("parent" "::") AbbrevReverseStep ::= ".." NodeTest ::= KindTest NameTest NameTest ::= QName Wildcard Wildcard ::= "*" NCName NCName /* ws: explicit XP FilterExpr ::= PrimaryExpr PredicateList PredicateList ::= Predicate Predicate ::= "[" Expr PrimaryExpr ::= Literal VarRef ParenthesizedExpr ContextItemExpr FunctionCall Literal ::= NumericLiteral StringLiteral NumericLiteral ::= IntegerLiteral DecimalLiteral DoubleLiteral VarRef ::= "$" VarName VarName ::= QName ParenthesizedExpr ::= "(" Expr ContextItemExpr ::= "." FunctionCall ::= QName ExprSingle ExprSingle /* xgc: reserved-function-names XP /* gn: parens XP SingleType ::= AtomicType SequenceType ::= ("empty-sequence" "(" ")") ItemType OccurrenceIndicator OccurrenceIndicator ::= "?" | "*" | "+" /* xgc: occurrence-indicators XP ItemType ::= KindTest AtomicType AtomicType ::= QName KindTest ::= DocumentTest ElementTest AttributeTest SchemaElementTest SchemaAttributeTest PITest CommentTest TextTest AnyKindTest AnyKindTest ::= "node" "(" ")" DocumentTest ::= "document-node" "(" ( ElementTest SchemaElementTest TextTest ::= "text" "(" ")" CommentTest ::= "comment" "(" ")" PITest ::= "processing-instruction" "(" ( NCName StringLiteral AttributeTest ::= "attribute" "(" ( AttribNameOrWildcard TypeName AttribNameOrWildcard ::= AttributeName SchemaAttributeTest ::= "schema-attribute" "(" AttributeDeclaration AttributeDeclaration ::= AttributeName ElementTest ::= "element" "(" ( ElementNameOrWildcard TypeName ElementNameOrWildcard ::= ElementName SchemaElementTest ::= "schema-element" "(" ElementDeclaration ElementDeclaration ::= ElementName AttributeName ::= QName ElementName ::= QName TypeName ::= QName URILiteral ::= StringLiteral FTSelection ::= FTOr FTPosFilter FTWeight ::= "weight" "{" Expr FTOr ::= FTAnd FTAnd FTAnd ::= FTMildNot FTMildNot FTMildNot ::= FTUnaryNot FTUnaryNot FTUnaryNot ::= ("ftnot")? FTPrimaryWithOptions FTPrimaryWithOptions ::= FTPrimary FTMatchOptions FTWeight FTPrimary ::= ( FTWords FTTimes FTSelection FTExtensionSelection FTWords ::= FTWordsValue FTAnyallOption FTWordsValue ::= StringLiteral Expr FTExtensionSelection ::= Pragma FTSelection FTAnyallOption ::= ("any" "word"?) | ("all" "words"?) | "phrase" FTTimes ::= "occurs" FTRange FTRange ::= ("exactly" AdditiveExpr AdditiveExpr AdditiveExpr AdditiveExpr AdditiveExpr FTPosFilter ::= FTOrder FTWindow FTDistance FTScope FTContent FTOrder ::= "ordered" FTWindow ::= "window" AdditiveExpr FTUnit FTDistance ::= "distance" FTRange FTUnit FTUnit ::= "words" | "sentences" | "paragraphs" FTScope ::= ("same" | "different") FTBigUnit FTBigUnit ::= "sentence" | "paragraph" FTContent ::= ("at" "start") | ("at" "end") | ("entire" "content") FTMatchOptions ::= ("using" FTMatchOption FTMatchOption ::= FTLanguageOption FTWildCardOption FTThesaurusOption FTStemOption FTCaseOption FTDiacriticsOption FTStopWordOption FTExtensionOption FTCaseOption ::= ("case" "insensitive") FTDiacriticsOption ::= ("diacritics" "insensitive") FTStemOption ::= "stemming" | ("no" "stemming") FTThesaurusOption ::= ("thesaurus" ( FTThesaurusID FTThesaurusID FTThesaurusID FTThesaurusID ::= "at" URILiteral StringLiteral FTLiteralRange FTLiteralRange ::= ("exactly" IntegerLiteral IntegerLiteral IntegerLiteral IntegerLiteral IntegerLiteral FTStopWordOption ::= ("stop" "words" FTStopWords FTStopWordsInclExcl FTStopWordsInclExcl FTStopWords ::= ("at" URILiteral StringLiteral StringLiteral FTStopWordsInclExcl ::= ("union" | "except") FTStopWords FTLanguageOption ::= "language" StringLiteral FTWildCardOption ::= "wildcards" | ("no" "wildcards") FTExtensionOption ::= "option" QName StringLiteral FTIgnoreOption ::= "without" "content" UnionExpr IntegerLiteral ::= Digits DecimalLiteral ::= ("." Digits Digits /* ws: explicit XP DoubleLiteral ::= (("." Digits Digits Digits /* ws: explicit XP StringLiteral ::= ('"' ( EscapeQuot EscapeApos /* ws: explicit XP EscapeQuot ::= '""' EscapeApos ::= "''" Comment ::= "(:" ( CommentContents Comment /* ws: explicit XP /* gn: comments XP QName ::= [http://www.w3.org/TR/REC-xml-names/#NT-QName] Names /* xgc: xml-version XP NCName ::= [http://www.w3.org/TR/REC-xml-names/#NT-NCName] Names /* xgc: xml-version XP S ::= [http://www.w3.org/TR/REC-xml#NT-S] XML /* xgc: xml-version XP Char ::= [http://www.w3.org/TR/REC-xml#NT-Char] XML /* xgc: xml-version XP The following symbols are used only in the definition of terminal symbols; they are not terminal symbols in the grammar of B EBNF for XPath 2.0 Grammar with Full-Text extensions Digits ::= [0-9]+ CommentContents ::= ( Char The following table describes the full-text components of the static context Section 2.1.1 Static Context XQ Default initial value: Can be overwritten or augmented by implementation: implementation-defined implementation-defined Can be overwritten or augmented by a query: prolog Section 4 Modules and Prologs XQ Scope: module Section 4 Modules and Prologs XQ Consistency Rules: Static Context Components Component Default initial value Can be overwritten or augmented by implementation? Can be overwritten or augmented by a query? Scope Consistency rules FTCaseOption case insensitive overwriteable overwriteable by prolog lexical Value must be case insensitive case sensitive lowercase uppercase FTDiacriticsOption diacritics insensitive overwriteable overwriteable by prolog lexical Value must be diacritics insensitive diacritics sensitive FTStemOption no stemming overwriteable overwriteable by prolog lexical Value must be stemming no stemming FTThesaurusOption no thesaurus overwriteable overwriteable by prolog (refer to default to augment) lexical Each URI in the value must be found in the statically known thesauri. Statically known thesauri none augmentable cannot be augmented or overwritten by prolog module Each URI uniquely identifies a thesaurus list. FTStopWordOption no stop words overwriteable overwriteable by prolog (refer to default to augment) lexical Each URI in the value must be found in the statically known stop word lists. Statically known stop word lists none augmentable cannot be augmented or overwritten by prolog module Each URI uniquely identifies a stop word list. FTLanguageOption implementation-defined overwriteable overwriteable by prolog lexical Value must be castable to xs:language Statically known languages none augmentable cannot be augmented or overwritten by prolog module Each string uniquely identifies a language. FTWildCardOption no wildcards no overwriteable by prolog lexical Value must be wildcards no wildcards An implementation that does not support the FTMildNot operator must raise a static error if a full-text query contains a mild not. An implementation that enforces one of the restrictions on FTUnaryNot must raise a static error if a full-text query does not obey the restriction. An implementation that does not support one or more of the choices on FTUnit and FTBigUnit must raise a static error if a full-text query contains one of those choices. An implementation that does not support the FTScope operator must raise a static error if a full-text query contains a scope. An implementation that does not support the FTTimes operator must raise a static error if a full-text query contains a times. An implementation that restricts the use of FTStopWordOption must raise a static error if a full-text query contains a stop word option that does not meet the restriction. An implementation that restricts the use of FTIgnoreOption must raise a static error if a full-text query contains an ignore option that does not meet the restriction. It is a static error if, during the static analysis phase, the query is found to contain a stop word option that refers to a stop word list that is not found in the statically known stop word lists. It may be a static error if, during the static analysis phase, the query is found to contain a language identifier in a language option that the implementation does not support. The implementation may choose not to raise this error and instead provide some other implementation-defined behavior. It is a static error if, during the static analysis phase, an expression is found to use an FTOrder operator that does not appear directly succeeding an FTWindow or an FTDistance operator and the implementation enforces this restriction. An implementation may restrict the use of FTWindow and FTDistance to an FTOr that is either a single FTWords or a combination of FTWords involving only the operators ftand and ftor An implementation that does not support the FTContent operator must raise a static error if a full-text query contains one. It is a static error if, during the static analysis phase, an implementation that restricts the use of FTLanguageOption to a single language, encounters more than one distinct language option. An implementation may constrain the form of the expression used to compute scores. It is a static error if, during the static analysis phase, such an implementation encounters a scoring expression that does not meet the restriction. It is a static error if, during the static analysis phase, an implementation that restricts the choices of FTCaseOption encounters the "lowercase" or "uppercase" option. It is a dynamic error if a weight value is not within the required range of values; it is also a dynamic error if an implementation that does not support negative weights encounters a negative weight value. It is a dynamic error if an implementation encounters a mild not selection, one of whose operands evaluates to an AllMatches StringExclude It is a static error if, during the static analysis phase, the query is found to contain a thesaurus option that refers to a thesaurus that is not found in the statically known thesauri. It is a static error if, within a single FTMatchOptions match option group It is a dynamic error if, when "wildcards" is in effect, a query string violates wildcard syntax. It is a dynamic error if, in a function invocation, the argument corresponding to the specified function's collation parameter does not identify a supported collation. It is a static error if an expression is not a valid instance of the grammar defined in A EBNF for XQuery 1.0 Grammar with Full Text extensions B EBNF for XPath 2.0 Grammar with Full-Text extensions It is a type error if, during the static analysis phase, an expression is found to have a static type that is not appropriate for the context in which the expression occurs, or during the dynamic evaluation phase, the dynamic type of a value does not match a required type as specified by the matching rules in Section 2.5.4 SequenceType Matching XP It is a static error if an implementation recognizes a pragma but determines that its content is invalid. It is a static error if an extension expression contains neither a pragma that is recognized by the implementation nor an expression enclosed in curly braces. [XML Syntax for XQuery 1.0 (XQueryX) (Second Edition)] [XQuery 1.0: An XML Query Language (Second Edition)] [XQuery and XPath Full Text 1.0 Requirements] A EBNF for XQuery 1.0 Grammar with Full Text extensions The XML Schema specified in this appendix accomplishes integration by importing the XML Schema defined for XQueryX in Section 4 An XML Schema for the XQuery XML Syntax XQX The semantics of a Full Text XQueryX document are determined by the semantics of the XQuery Full Text expression that results from transforming the XQueryX document into XQuery Full Text syntax using the XSLT stylesheet that appears in section E.2 XQueryX stylesheet for XQuery and XPath Full Text 1.0 The XML Schema that defines the complex types and elements for XQueryX in support of XQuery and XPath Full Text 1.0, including the ftContainsExpr, incorporates a second XML Schema that defines types and elements to support the ftMatchOption. Both XML Schemas are defined in this section. <xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xqx="http://www.w3.org/2005/XQueryX" xmlns:xqxft="http://www.w3.org/2007/xpath-full-text" targetNamespace="http://www.w3.org/2007/xpath-full-text" elementFormDefault="qualified" attributeFormDefault="unqualified">

<!-- Initial creation 2006-08-17: Jim Melton --> <!-- Added ftOptionDecl, ftScoreVariableBinding 2006-08-21: Jim Melton --> <!-- First version believed complete 2006-08-29: Jim Melton --> <!-- Cleaned up naming 2007-04-27: Mary Holstege --> <!-- Revised to align with updated syntax 2008-01-14: Jim Melton --> <!-- Moved ftOptionDecl: prolog part two to one 2008-01-24: Jim Melton --> <!-- Revised position of "weight" in grammar 2008-11-12: Jim Melton -->

<xsd:import namespace="http://www.w3.org/2005/XQueryX" schemaLocation="http://www.w3.org/2005/XQueryX/xqueryx.xsd"/>

<xsd:include schemaLocation="./xpath-full-text-10-xqueryx-ftmatchoption-extensions.xsd"/>

<xsd:element name="ftOptionDecl" substitutionGroup="xqx:prologPartOneItem"> <xsd:complexType> <xsd:sequence minOccurs="1" maxOccurs="unbounded"> <xsd:element ref="xqxft:ftMatchOption"/> </xsd:sequence> </xsd:complexType> </xsd:element>

<!-- Create a new substitution group for full-text expressions --> <xsd:complexType name="ftExpr"> <xsd:complexContent> <xsd:extension base="xqx:expr"/> </xsd:complexContent> </xsd:complexType>

<xsd:element name="ftExpr" type="xqxft:ftExpr" abstract="true" substitutionGroup="xqx:expr"/>

<!-- Represents an untyped variable for the "score" clause --> <xsd:element name="ftScoreVariableBinding" type="xqx:QName" substitutionGroup="xqx:forLetClauseItemExtensions"/>

<!-- FTContains ("contains text") --> <!-- Represents the following grammar productions: --> <!-- FTContainsExpr ::= --> <!-- RangeExpr ( "contains" "text" FTSelection FTIgnoreOption? )? --> <xsd:complexType name="ftContainsExpr"> <xsd:complexContent> <xsd:extension base="xqxft:ftExpr"> <xsd:sequence> <xsd:element name="ftRangeExpr" type="xqx:exprWrapper" /> <xsd:sequence minOccurs="0" maxOccurs="1"> <xsd:element name="ftSelectionExpr" type="xqxft:ftSelectionWrapper" /> <xsd:element name="ftIgnoreOption" type="xqxft:ftIgnoreOption" minOccurs="0" maxOccurs="1" /> </xsd:sequence> </xsd:sequence> </xsd:extension> </xsd:complexContent> </xsd:complexType>

<xsd:element name="ftContainsExpr" type="xqxft:ftContainsExpr" substitutionGroup="xqxft:ftExpr" />

<!-- FTProximity --> <!-- Represents the following grammar productions: --> <!-- FTPosFilter ::= --> <!-- FTOrder | FTWindow | FTDistance | FTScope | FTContent --> <xsd:complexType name="ftProximity" />

<xsd:element name="ftProximity" type="xqxft:ftProximity" abstract="true"/>

<!-- some simple type definitions -->

<!-- Represents the following grammar productions: --> <!-- FTUnit ::= "words" | "sentences" | "paragraphs" --> <xsd:simpleType name="ftUnit"> <xsd:restriction base="xsd:string"> <xsd:enumeration value="paragraph"/> <xsd:enumeration value="sentence"/> <xsd:enumeration value="word"/> </xsd:restriction> </xsd:simpleType>

<!-- Represents the following grammar productions: --> <!-- FTBigUnit ::= "sentence" | "paragraph" --> <xsd:simpleType name="ftBigUnit"> <xsd:restriction base="xsd:string"> <xsd:enumeration value="paragraph"/> <xsd:enumeration value="sentence"/> </xsd:restriction> </xsd:simpleType>

<!-- Represents the following grammar productions: --> <!-- FTContent ::= ("at" "start") | ("at" "end") | ("entire" "content")--> <xsd:simpleType name="contentLocation"> <xsd:restriction base="xsd:string"> <xsd:enumeration value="at start"/> <xsd:enumeration value="at end"/> <xsd:enumeration value="entire content"/> </xsd:restriction> </xsd:simpleType>

<!-- Represents the following grammar productions: --> <!-- FTScope ::= ("same" | "different") FTBigUnit --> <xsd:simpleType name="ftScopeType"> <xsd:restriction base="xsd:string"> <xsd:enumeration value="same"/> <xsd:enumeration value="different"/> </xsd:restriction> </xsd:simpleType>

<!-- range-related definitions --> <xsd:complexType name="unaryRange"> <xsd:sequence> <xsd:element name="value" type="xqx:exprWrapper" /> </xsd:sequence> </xsd:complexType>

<xsd:complexType name="binaryRange"> <xsd:sequence> <xsd:element name="lower" type="xqx:exprWrapper" /> <xsd:element name="upper" type="xqx:exprWrapper" /> </xsd:sequence> </xsd:complexType>

<xsd:complexType name="unaryLiteralRange"> <xsd:sequence> <xsd:element name="value" type="xsd:integer" /> </xsd:sequence> </xsd:complexType>

<xsd:complexType name="binaryLiteralRange"> <xsd:sequence> <xsd:element name="lower" type="xsd:integer" /> <xsd:element name="upper" type="xsd:integer" /> </xsd:sequence> </xsd:complexType>

<!-- Represents the following grammar productions: --> <!-- FTRange ::= ("exactly" AdditiveExpr) --> <!-- | ("at" "least" AdditiveExpr) --> <!-- | ("at" "most" AdditiveExpr) --> <!-- | ("from" AdditiveExpr "to" AdditiveExpr) --> <xsd:complexType name="ftRange"> <xsd:choice> <xsd:element name="atLeastRange" type="xqxft:unaryRange" /> <xsd:element name="atMostRange" type="xqxft:unaryRange" /> <xsd:element name="exactlyRange" type="xqxft:unaryRange" /> <xsd:element name="fromToRange" type="xqxft:binaryRange" /> </xsd:choice> </xsd:complexType>

<!-- Represents the following grammar productions: --> <!-- FTLiteralRange ::= ("exactly" IntegerLiteral) --> <!-- | ("at" "least" IntegerLiteral) --> <!-- | ("at" "most" IntegerLiteral) --> <!-- | ("from" IntegerLiteral "to" IntegerLiteral) --> <xsd:complexType name="ftLiteralRange"> <xsd:choice> <xsd:element name="atLeastLiteralRange" type="xqxft:unaryLiteralRange" /> <xsd:element name="atMostLiteralRange" type="xqxft:unaryLiteralRange" /> <xsd:element name="exactlyLiteralRange" type="xqxft:unaryLiteralRange" /> <xsd:element name="fromToLiteralRange" type="xqxft:binaryLiteralRange" /> </xsd:choice> </xsd:complexType>

<!-- ftPosFilter alternative: ordered --> <!-- Represents the following grammar productions: --> <!-- FTOrder ::= "ordered" --> <xsd:complexType name="ftOrdered"> <xsd:complexContent> <xsd:extension base="xqxft:ftProximity"> </xsd:extension> </xsd:complexContent> </xsd:complexType>

<xsd:element name="ftOrdered" type="xqxft:ftOrdered" substitutionGroup="xqxft:ftProximity"/>

<!-- ftPosFilter alternative: window --> <!-- Represents the following grammar productions: --> <!-- FTWindow ::= "window" AdditiveExpr FTUnit --> <xsd:complexType name="ftWindow"> <xsd:complexContent> <xsd:extension base="xqxft:ftProximity"> <xsd:sequence> <xsd:element name="value" type="xqx:exprWrapper" /> <xsd:element name="unit" type="xqxft:ftUnit" /> </xsd:sequence> </xsd:extension> </xsd:complexContent> </xsd:complexType>

<xsd:element name="ftWindow" type="xqxft:ftWindow" substitutionGroup="xqxft:ftProximity"/>

<!-- ftPosFilter alternative: distance --> <!-- Represents the following grammar productions: --> <!-- FTDistance ::= "distance" FTRange FTUnit --> <xsd:complexType name="ftDistance"> <xsd:complexContent> <xsd:extension base="xqxft:ftProximity"> <xsd:sequence> <xsd:element name="ftRange" type="xqxft:ftRange" /> <xsd:element name="unit" type="xqxft:ftUnit" /> </xsd:sequence> </xsd:extension> </xsd:complexContent> </xsd:complexType>

<xsd:element name="ftDistance" type="xqxft:ftDistance" substitutionGroup="xqxft:ftProximity"/>

<!-- ftPosFilter alternative: scope --> <!-- Represents the following grammar productions: --> <xsd:complexType name="ftScope"> <xsd:complexContent> <xsd:extension base="xqxft:ftProximity"> <xsd:sequence> <xsd:element name="type" type="xqxft:ftScopeType" /> <xsd:element name="unit" type="xqxft:ftBigUnit" /> </xsd:sequence> </xsd:extension> </xsd:complexContent> </xsd:complexType>

<xsd:element name="ftScope" type="xqxft:ftScope" substitutionGroup="xqxft:ftProximity"/>

<!-- ftPosFilter alternative: FTContent --> <!-- Represents the following grammar productions: --> <xsd:complexType name="ftContent"> <xsd:complexContent> <xsd:extension base="xqxft:ftProximity"> <xsd:sequence> <xsd:element name="location" type="xqxft:contentLocation" /> </xsd:sequence> </xsd:extension> </xsd:complexContent> </xsd:complexType>

<xsd:element name="ftContent" type="xqxft:ftContent" substitutionGroup="xqxft:ftProximity"/>

<!-- ftPosFilter --> <!-- Represents the following grammar productions: --> <!-- FTPosFilter ::= --> <!-- FTOrder | FTWindow | FTDistance | FTScope | FTContent --> <xsd:complexType name="ftPosFilter"> <xsd:complexContent> <xsd:extension base="xqxft:ftExpr"> <xsd:sequence minOccurs="0" maxOccurs="unbounded"> <xsd:element ref="xqxft:ftProximity" /> </xsd:sequence> </xsd:extension> </xsd:complexContent> </xsd:complexType>

<!-- FTSelection --> <!-- Represents the following grammar productions: --> <!-- FTSelection ::= FTOr FTPosFilter* --> <xsd:complexType name="ftSelection" > <xsd:complexContent> <xsd:extension base="xqxft:ftExpr"> <xsd:sequence> <xsd:element name="ftSelectionSource" type="xqx:exprWrapper"/> <xsd:element name="ftPosFilter" type="xqxft:ftPosFilter" minOccurs="0" maxOccurs="1" /> </xsd:sequence> </xsd:extension> </xsd:complexContent> </xsd:complexType>

<xsd:element name="ftSelection" type="xqxft:ftSelection" substitutionGroup="xqxft:ftExpr" />

<xsd:complexType name="ftSelectionWrapper"> <xsd:sequence> <xsd:element ref="xqxft:ftSelection"/> </xsd:sequence> </xsd:complexType>

<!-- Represents the following grammar productions: --> <!-- FTIgnoreOption ::= "without" "content" UnionExpr --> <xsd:complexType name="ftIgnoreOption"> <xsd:sequence> <xsd:element ref="xqx:expr"/> </xsd:sequence> </xsd:complexType>

<!-- Full-Text logical operators --> <xsd:element name="ftLogicalOp" type="xqx:binaryOperatorExpr" abstract="true" substitutionGroup="xqx:operatorExpr"/>

<!-- Represents the following grammar productions: --> <!-- FTOr ::= FTAnd ( "ftor" FTAnd )* --> <xsd:element name="ftOr" type="xqx:binaryOperatorExpr" substitutionGroup="xqxft:ftLogicalOp"/>

<!-- Represents the following grammar productions: --> <!-- FTAnd ::= FTMildNot ( "ftand" FTMildNot )* --> <xsd:element name="ftAnd" type="xqx:binaryOperatorExpr" substitutionGroup="xqxft:ftLogicalOp"/>

<!-- Represents the following grammar productions: --> <!-- FTMildNot ::= FTUnaryNot ( "not" "in" FTUnaryNot )* --> <xsd:element name="ftMildNot" type="xqx:binaryOperatorExpr" substitutionGroup="xqxft:ftLogicalOp"/>

<!-- Represents the following grammar productions: --> <xsd:element name="ftLogicalNot" type="xqx:unaryOperatorExpr" abstract="true" substitutionGroup="xqx:operatorExpr"/>

<!-- Represents the following grammar productions: --> <!-- FTUnaryNot ::= ("ftnot")? FTPrimaryWithOptions --> <xsd:element name="ftUnaryNot" type="xqx:unaryOperatorExpr" substitutionGroup="xqxft:ftLogicalNot"/>

<!-- Definitions associated with FTWords --> <!-- Represents the following grammar productions: --> <!-- FTTimes ::= "occurs" FTRange "times" --> <xsd:complexType name="ftTimes"> <xsd:sequence> <xsd:element name="ftRange" type="xqxft:ftRange"/> </xsd:sequence> </xsd:complexType>

<!-- Represents the following grammar productions: --> <!-- FTAnyallOption ::= ("any" "word"?) | ("all" "words"?) | "phrase" --> <xsd:simpleType name="ftAnyAllOption"> <xsd:restriction base="xsd:string"> <xsd:enumeration value="any"/> <xsd:enumeration value="all"/> <xsd:enumeration value="any word"/> <xsd:enumeration value="all words"/> <xsd:enumeration value="phrase"/> </xsd:restriction> </xsd:simpleType>

<!-- Represents the following grammar productions: --> <!-- FTWordsValue ::= StringLiteral | ("{" Expr "}") --> <xsd:complexType name="ftWordsAlternatives"> <xsd:choice> <xsd:element name="ftWordsLiteral" type="xqx:exprWrapper"/> <xsd:element name="ftWordsExpression" type="xqx:exprWrapper"/> </xsd:choice> </xsd:complexType>

<!-- Represents the following grammar productions: --> <!-- FTWords ::= FTWordsValue FTAnyallOption? --> <xsd:complexType name="ftWords"> <xsd:sequence> <xsd:element name="ftWordsValue" type="xqxft:ftWordsAlternatives" /> <xsd:element name="ftAnyAllOption" type="xqxft:ftAnyAllOption" minOccurs="0" maxOccurs="1" /> </xsd:sequence> </xsd:complexType>

<!-- Represents the following grammar productions: --> <!-- ... FTWordsValue FTAnyallOption? --> <xsd:group name="ftWordsWithTimes"> <xsd:sequence> <xsd:element name="ftWords" type="xqxft:ftWords" /> <xsd:element name="ftTimes" type="xqxft:ftTimes" minOccurs="0" /> </xsd:sequence> </xsd:group>

<!-- Represents the following grammar productions: --> <!-- FTExtensionSelection ::= Pragma+ "{" FTSelection? "}" --> <xsd:complexType name="ftExtensionSelection"> <xsd:sequence> <xsd:element name="pragma" type="xqx:pragma" minOccurs="1" maxOccurs="unbounded"/> <xsd:element name="ftSelection" type="xqxft:ftSelection" minOccurs="0" maxOccurs="1"/> </xsd:sequence> </xsd:complexType>

<!-- Represents the following grammar productions: --> <!-- FTPrimary ::= (FTWords FTTimes?) --> <!-- | ("(" FTSelection ")") --> <!-- | FTExtensionSelection --> <xsd:complexType name="ftPrimary"> <xsd:complexContent> <xsd:extension base="xqxft:ftExpr" > <xsd:choice> <xsd:element name="parenthesized" type="xqx:exprWrapper"/> <xsd:group ref="xqxft:ftWordsWithTimes" /> <xsd:element name="ftExtensionSelection" type="xqxft:ftExtensionSelection"/> </xsd:choice> </xsd:extension> </xsd:complexContent> </xsd:complexType>

<!-- Represents the following grammar productions: --> <!-- FTPrimaryWithOptions ::= FTPrimary FTMatchOptions? FTWeight? --> <xsd:complexType name="ftPrimaryWithOptions"> <xsd:complexContent> <xsd:extension base="xqxft:ftExpr"> <xsd:sequence> <xsd:element name="ftPrimary" type="xqxft:ftPrimary"/> <xsd:element ref="xqxft:ftMatchOptions" minOccurs="0" maxOccurs="1"/> <xsd:element name="weight" type="xqx:exprWrapper" minOccurs="0" maxOccurs="1" /> </xsd:sequence> </xsd:extension> </xsd:complexContent> </xsd:complexType>

<xsd:element name="ftPrimaryWithOptions" type="xqxft:ftPrimaryWithOptions" substitutionGroup="xqxft:ftExpr"/>

</xsd:schema> <xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xqx="http://www.w3.org/2005/XQueryX" xmlns:xqxft="http://www.w3.org/2007/xpath-full-text" targetNamespace="http://www.w3.org/2007/xpath-full-text" elementFormDefault="qualified" attributeFormDefault="unqualified">

<!-- Initial creation 2006-08-17: Jim Melton --> <!-- First version believed complete 2006-08-29: Jim Melton --> <!-- Cleaned up naming 2007-04-27: Mary Holstege --> <!-- Revised to align with updated syntax 2008-01-14: Jim Melton --> <!-- Comments added to clarify each element 2008-11-12: Jim Melton --> <!-- Add element decl for ftMatchOptions 2009-07-06: Michael Dyck --> <!-- Fixed FTThesaurus for ftLiteralRange 2011-03-08: Jim Melton -->

<xsd:import namespace="http://www.w3.org/2005/XQueryX" schemaLocation="http://www.w3.org/2005/XQueryX/xqueryx.xsd"/>

<!-- FTMatchOption --> <!-- Represents the following grammar productions: --> <!-- FTMatchOption ::= FTLanguageOption --> <!-- | FTWildCardOption --> <!-- | FTThesaurusOption --> <!-- | FTStemOption --> <!-- | FTCaseOption --> <!-- | FTDiacriticsOption --> <!-- | FTStopWordOption --> <!-- | FTExtensionOption --> <xsd:complexType name="ftMatchOption" />

<xsd:element name="ftMatchOption" type="xqxft:ftMatchOption" abstract="true" />

<!-- Represents the following grammar productions: --> <!-- FTMatchOptions ::= ( "using" FTMatchOption )+ --> <xsd:complexType name="ftMatchOptions"> <xsd:sequence minOccurs="1" maxOccurs="unbounded"> <xsd:element ref="xqxft:ftMatchOption"/> </xsd:sequence> </xsd:complexType>

<xsd:element name="ftMatchOptions" type="xqxft:ftMatchOptions"/>

<!-- ftMatchOption alternative: case --> <!-- Represents the following grammar productions: --> <!-- FTCaseOption ::= ("case" "insensitive") --> <!-- | ("case" "sensitive") --> <!-- | "lowercase" --> <!-- | "uppercase" --> <xsd:complexType name="ftCaseOption"> <xsd:complexContent> <xsd:extension base="xqxft:ftMatchOption" > <xsd:sequence> <xsd:element name="value"> <xsd:simpleType> <xsd:restriction base="xsd:string"> <xsd:enumeration value="lowercase"/> <xsd:enumeration value="uppercase"/> <xsd:enumeration value="case sensitive"/> <xsd:enumeration value="case insensitive"/> </xsd:restriction> </xsd:simpleType> </xsd:element> </xsd:sequence> </xsd:extension> </xsd:complexContent> </xsd:complexType>

<xsd:element name="case" type="xqxft:ftCaseOption" substitutionGroup="xqxft:ftMatchOption" />

<!-- ftMatchOption alternative: diacritics --> <!-- Represents the following grammar productions: --> <!-- FTDiacriticsOption ::= ("diacritics" "insensitive") --> <!-- | ("diacritics" "sensitive") --> <xsd:complexType name="ftDiacriticsOption"> <xsd:complexContent> <xsd:extension base="xqxft:ftMatchOption" > <xsd:sequence> <xsd:element name="value"> <xsd:simpleType> <xsd:restriction base="xsd:string"> <xsd:enumeration value="diacritics sensitive"/> <xsd:enumeration value="diacritics insensitive"/> </xsd:restriction> </xsd:simpleType> </xsd:element> </xsd:sequence> </xsd:extension> </xsd:complexContent> </xsd:complexType> <xsd:element name="diacritics" type="xqxft:ftDiacriticsOption" substitutionGroup="xqxft:ftMatchOption" />

<!-- ftMatchOption alternative: stemming --> <!-- Represents the following grammar productions: --> <!-- FTStemOption ::= ("stemming") | ("no" "stemming") --> <xsd:complexType name="ftStemOption"> <xsd:complexContent> <xsd:extension base="xqxft:ftMatchOption" > <xsd:sequence> <xsd:element name="value"> <xsd:simpleType> <xsd:restriction base="xsd:string"> <xsd:enumeration value="stemming" /> <xsd:enumeration value="no stemming" /> </xsd:restriction> </xsd:simpleType> </xsd:element> </xsd:sequence> </xsd:extension> </xsd:complexContent> </xsd:complexType> <xsd:element name="stem" type="xqxft:ftStemOption" substitutionGroup="xqxft:ftMatchOption" />

<!-- ftMatchOption alternative: thesaurus --> <!-- Represents the following grammar productions: --> <!-- FTThesaurusID ::= "at" URILiteral ("relationship" StringLiteral)? --> <!-- (FTLiteralRange "levels")? --> <xsd:complexType name="ftThesaurusID"> <xsd:sequence> <xsd:element name="at" type="xsd:anyURI" /> <xsd:element name="relationship" type="xsd:string" minOccurs="0" /> <xsd:element name="levels" type="xqxft:ftLiteralRange" minOccurs="0" /> </xsd:sequence> </xsd:complexType>

<!-- Represents the following grammar productions: --> <!-- ... (FTThesaurusID | "default") --> <!-- ... "(" (FTThesaurusID | "default") ("," FTThesaurusID)* ")") --> <xsd:complexType name="thesaurusSpecSequence"> <xsd:sequence> <xsd:choice> <xsd:element name="default" /> <xsd:element name="thesaurusID" type="xqxft:ftThesaurusID" /> </xsd:choice> <xsd:element name="thesaurusID" type="xqxft:ftThesaurusID" minOccurs="0" maxOccurs="unbounded" /> </xsd:sequence> </xsd:complexType>

<!-- Represents the following grammar productions: --> <!-- FTThesaurusOption ::= --> <!-- ("thesaurus" (FTThesaurusID | "default")) --> <!-- | ("thesaurus" --> <!-- "(" (FTThesaurusID | "default") ("," FTThesaurusID)* ")") --> <!-- | ("no" "thesaurus") --> <xsd:complexType name="ftThesaurusOption"> <xsd:complexContent> <xsd:extension base="xqxft:ftMatchOption" > <xsd:choice> <xsd:element name="noThesauri" /> <xsd:element name="thesauri" type="xqxft:thesaurusSpecSequence" /> </xsd:choice> </xsd:extension> </xsd:complexContent> </xsd:complexType>

<xsd:element name="thesaurus" type="xqxft:ftThesaurusOption" substitutionGroup="xqxft:ftMatchOption" />

<!-- ftMatchOption alternative: stopwords --> <!-- Represents the following grammar productions: --> <!-- FTStopWords ::= ("at" URILiteral) --> <!-- | ("(" StringLiteral ("," StringLiteral)* ")") --> <xsd:complexType name="ftStopWords"> <xsd:choice> <xsd:element name="ref" type="xsd:anyURI" /> <xsd:element name="list"> <xsd:complexType> <xsd:sequence> <xsd:element ref="xqx:stringConstantExpr" minOccurs="1" maxOccurs="unbounded" /> </xsd:sequence> </xsd:complexType> </xsd:element> </xsd:choice> </xsd:complexType>

<xsd:element name="ftStopWords" type="xqxft:ftStopWords" />

<!-- Represents the following grammar productions: --> <!-- ... "stop" "words" FTStopWords ... --> <!-- ... "stop" "words" "default" ... --> <xsd:group name="baseStopWords"> <xsd:choice> <xsd:element name="default" /> <xsd:element ref="xqxft:ftStopWords" /> </xsd:choice> </xsd:group>

<!-- Represents the following grammar productions: --> <!-- FTStopWordsInclExcl ::= ("union" | "except") FTStopWords --> <xsd:complexType name="ftStopWordsInclExcl"> <xsd:choice> <xsd:element name="union" type="xqxft:ftStopWords" /> <xsd:element name="except" type="xqxft:ftStopWords" /> </xsd:choice> </xsd:complexType>

<!-- Represents the following grammar productions: --> <!-- ... ("using" "stop" "words" FTStopWords FTStopWordsInclExcl*) ... --> <!-- ... ("using" "default" "stop" "words" FTStopWordsInclExcl*) ... --> <xsd:complexType name="stopWordsSpecSequence"> <xsd:sequence> <xsd:group ref="xqxft:baseStopWords" /> <xsd:element name="ftStopWordsInclExcl" type="xqxft:ftStopWordsInclExcl" minOccurs="0" maxOccurs="unbounded" /> </xsd:sequence> </xsd:complexType>

<!-- Represents the following grammar productions: --> <!-- FTStopWordOption ::= --> <!-- ("stop" "words" FTStopWords FTStopWordsInclExcl*) --> <!-- | ("stop" "words" "default" FTStopWordsInclExcl*) --> <!-- | ("no" "stop" "words") --> <xsd:complexType name="ftStopWordOption"> <xsd:complexContent> <xsd:extension base="xqxft:ftMatchOption" > <xsd:choice> <xsd:element name="noStopwords" /> <xsd:element name="stopwords" type="xqxft:stopWordsSpecSequence" /> </xsd:choice> </xsd:extension> </xsd:complexContent> </xsd:complexType>

<xsd:element name="stopword" type="xqxft:ftStopWordOption" substitutionGroup="xqxft:ftMatchOption" />

<!-- ftMatchOption alternative: language --> <!-- Represents the following grammar productions: --> <!-- FTLanguageOption ::= "language" StringLiteral --> <xsd:complexType name="ftLanguageOption"> <xsd:complexContent> <xsd:extension base="xqxft:ftMatchOption" > <xsd:sequence> <xsd:element name="value" type="xsd:string" /> </xsd:sequence> </xsd:extension> </xsd:complexContent> </xsd:complexType>

<xsd:element name="language" type="xqxft:ftLanguageOption" substitutionGroup="xqxft:ftMatchOption" />

<!-- ftMatchOption alternative: wildcards --> <!-- Represents the following grammar productions: --> <!-- FTWildCardOption ::= ("wildcards") --> <!-- | ("no" "wildcards") --> <xsd:complexType name="ftWildCardOption"> <xsd:complexContent> <xsd:extension base="xqxft:ftMatchOption"> <xsd:sequence> <xsd:element name="value"> <xsd:simpleType> <xsd:restriction base="xsd:string"> <xsd:enumeration value="wildcards" /> <xsd:enumeration value="no wildcards" /> </xsd:restriction> </xsd:simpleType> </xsd:element> </xsd:sequence> </xsd:extension> </xsd:complexContent> </xsd:complexType>

<xsd:element name="wildcard" type="xqxft:ftWildCardOption" substitutionGroup="xqxft:ftMatchOption" />

<!-- Represents the following grammar productions: --> <!-- FTExtensionOption ::= "option" QName StringLiteral --> <xsd:complexType name="ftExtensionOption"> <xsd:complexContent> <xsd:extension base="xqxft:ftMatchOption"> <xsd:sequence> <xsd:element name="ftExtensionName" type="xqx:QName"/> <xsd:element name="ftExtensionValue" type="xsd:string"/> </xsd:sequence> </xsd:extension> </xsd:complexContent> </xsd:complexType>

<xsd:element name="ftExtensionOption" type="xqxft:ftExtensionOption" substitutionGroup="xqxft:ftMatchOption" />

</xsd:schema> The XSLT stylesheet that defines the semantics of XQueryX in support of XQuery and XPath Full Text 1.0 integrates seamlessly with the XQueryX XSLT stylesheet defined in Section B Transforming XQueryX to XQuery XQX <?xml version='1.0'?> <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:xqxft="http://www.w3.org/2007/xpath-full-text" xmlns:xqx="http://www.w3.org/2005/XQueryX">

<!-- Initial creation 2006-08-17: Jim Melton --> <!-- Added ftOptionDecl, ftScoreVariableBinding 2006-08-21: Jim Melton --> <!-- First version believed complete 2006-08-29: Jim Melton --> <!-- Revised to align with 2008-01-24 draft 2008-02-08: Jim Melton --> <!-- Revised position of "weight" in grammar 2008-11-12: Jim Melton --> <!-- Various bug fixes 2009-07-14: Michael Dyck --> <!-- ftcontains => "contains text", Bug 7247 2009-09-17: Jim Melton --> <!-- with => using, stop words default, Bug 7271 2009-09-17: Jim Melton --> <!-- {} around weight values, around empty selection after pragmas 2010-09-07: Jim Melton -->

<xsl:import href="http://www.w3.org/2005/XQueryX/xqueryx.xsl"/>

<!-- ftOptionDecl --> <xsl:template match="xqxft:ftOptionDecl"> <xsl:text>declare ft-option </xsl:text> <xsl:apply-templates/> </xsl:template>

<!-- ftScoreVariableBinding --> <xsl:template match="xqxft:ftScoreVariableBinding"> <xsl:text> score </xsl:text> <xsl:value-of select="$DOLLAR"/> <xsl:if test="@xqx:prefix"> <xsl:value-of select="@xqx:prefix"/> <xsl:value-of select="$COLON"/> </xsl:if> <xsl:value-of select="."/> </xsl:template>

<!-- ftcontains --> <xsl:template match="xqxft:ftContainsExpr"> <xsl:apply-templates select="xqxft:ftRangeExpr"/> <xsl:text> contains text </xsl:text> <xsl:apply-templates select="xqxft:ftSelectionExpr"/> <xsl:apply-templates select="xqxft:ftIgnoreOption"/> </xsl:template>

<xsl:template match="xqxft:value"> <xsl:apply-templates/> </xsl:template>

<xsl:template match="xqxft:ftRangeExpr"> <xsl:apply-templates/> </xsl:template>

<xsl:template match="xqxft:ftLiteralRangeExpr"> <xsl:apply-templates/> </xsl:template>

<xsl:template match="xqxft:ftSelectionExpr"> <xsl:apply-templates/> </xsl:template>

<xsl:template match="xqxft:ftIgnoreOption"> <xsl:text>without content </xsl:text> <xsl:apply-templates/> </xsl:template>

<xsl:template match="xqxft:ftSelection"> <xsl:apply-templates select="xqxft:ftSelectionSource"/> <xsl:value-of select="$NEWLINE"/> <xsl:text> </xsl:text> <xsl:apply-templates select="xqxft:ftPosFilter"/> </xsl:template>

<xsl:template match="xqxft:ftSelectionSource"> <xsl:apply-templates/> <xsl:text> </xsl:text> </xsl:template>

<xsl:template match="xqxft:ftPosFilter"> <xsl:apply-templates/> <xsl:value-of select="$NEWLINE"/> <xsl:text> </xsl:text> </xsl:template>

<!-- FTProximity alternative: ordered --> <xsl:template match="xqxft:ftOrdered"> <xsl:text>ordered </xsl:text> <xsl:value-of select="$NEWLINE"/> </xsl:template>

<!-- FTProximity alternative: window --> <xsl:template match="xqxft:ftWindow"> <xsl:text>window </xsl:text> <xsl:apply-templates select="xqxft:value"/> <xsl:text> </xsl:text> <xsl:value-of select="xqxft:unit"/> <xsl:text>s</xsl:text> <xsl:value-of select="$NEWLINE"/> </xsl:template>

<!-- FTProximity alternative: distance --> <xsl:template match="xqxft:ftDistance"> <xsl:text>distance </xsl:text> <xsl:apply-templates select="xqxft:ftRange"/> <xsl:text> </xsl:text> <xsl:value-of select="xqxft:unit"/> <xsl:text>s</xsl:text> <xsl:value-of select="$NEWLINE"/> </xsl:template>

<!-- FTProximity alternative: scope --> <xsl:template match="xqxft:ftScope"> <xsl:value-of select="xqxft:type"/> <xsl:text> </xsl:text> <xsl:value-of select="xqxft:unit"/> <xsl:value-of select="$NEWLINE"/> </xsl:template>

<!-- FTProximity alternative: content --> <xsl:template match="xqxft:ftContent"> <xsl:value-of select="xqxft:location"/> <xsl:value-of select="$NEWLINE"/> </xsl:template>

<xsl:template match="xqxft:exactlyRange | xqxft:exactlyLiteralRange"> <xsl:text>exactly </xsl:text> <xsl:apply-templates select="xqxft:value"/> </xsl:template>

<xsl:template match="xqxft:atLeastRange | xqxft:atLeastLiteralRange"> <xsl:text>at least </xsl:text> <xsl:apply-templates select="xqxft:value"/> </xsl:template>

<xsl:template match="xqxft:atMostRange | xqxft:atMostLiteralRange"> <xsl:text>at most </xsl:text> <xsl:apply-templates select="xqxft:value"/> </xsl:template>

<xsl:template match="xqxft:fromToRange | xqxft:fromToLiteralRange"> <xsl:text>from </xsl:text> <xsl:apply-templates select="xqxft:lower"/> <xsl:text> to </xsl:text> <xsl:apply-templates select="xqxft:upper"/> <xsl:text> </xsl:text> </xsl:template>

<xsl:template match="xqxft:lower"> <xsl:apply-templates/> </xsl:template>

<xsl:template match="xqxft:upper"> <xsl:apply-templates/> </xsl:template>

<!-- ftMatchOption alternative: case --> <xsl:template match="xqxft:case"> <xsl:text> using </xsl:text> <xsl:value-of select="xqxft:value"/> <xsl:value-of select="$NEWLINE"/> </xsl:template>

<!-- ftMatchOption alternative: diacritics --> <xsl:template match="xqxft:diacritics"> <xsl:text> using </xsl:text> <xsl:value-of select="xqxft:value"/> <xsl:value-of select="$NEWLINE"/> </xsl:template>

<!-- ftMatchOption alternative: stemming --> <xsl:template match="xqxft:stem"> <xsl:text> using </xsl:text> <xsl:value-of select="xqxft:value"/> <xsl:value-of select="$NEWLINE"/> </xsl:template>

<!-- ftMatchOption alternative: thesaurus --> <xsl:template match="xqxft:thesaurus"> <xsl:text> using </xsl:text> <xsl:choose> <xsl:when test="xqxft:noThesauri"> <xsl:text>no thesaurus </xsl:text> </xsl:when> <xsl:otherwise> <xsl:apply-templates/> </xsl:otherwise> </xsl:choose> <xsl:value-of select="$NEWLINE"/> </xsl:template>

<xsl:template match="xqxft:thesauri"> <xsl:text> </xsl:text> <xsl:text>thesaurus </xsl:text> <xsl:choose> <xsl:when test="child::*[2]"> <xsl:call-template name="parenthesizedList"/> </xsl:when> <xsl:otherwise> <xsl:apply-templates/> </xsl:otherwise> </xsl:choose> </xsl:template>

<xsl:template match="xqxft:default"> <xsl:text>default </xsl:text> </xsl:template>

<xsl:template match="xqxft:thesaurusID"> <xsl:apply-templates/> </xsl:template>

<xsl:template match="xqxft:at"> <xsl:text>at "</xsl:text> <xsl:value-of select="."/> <xsl:text>" </xsl:text> </xsl:template>

<xsl:template match="xqxft:relationship"> <xsl:text>relationship "</xsl:text> <xsl:value-of select="."/> <xsl:text>" </xsl:text> </xsl:template>

<xsl:template match="xqxft:levels"> <xsl:apply-templates/> <xsl:text> levels </xsl:text> </xsl:template>

<!-- ftMatchOption alternative: stopword --> <xsl:template match="xqxft:stopword"> <xsl:text>using </xsl:text> <xsl:choose> <xsl:when test="xqxft:noStopwords"> <xsl:text>no stop words </xsl:text> </xsl:when> <xsl:otherwise> <xsl:apply-templates/> </xsl:otherwise> </xsl:choose> <xsl:value-of select="$NEWLINE"/> </xsl:template>

<xsl:template match="xqxft:stopwords"> <xsl:text> </xsl:text> <xsl:choose> <xsl:when test="xqxft:default"> <xsl:text>stop words default </xsl:text> </xsl:when> <xsl:otherwise> <xsl:text>stop words </xsl:text> <xsl:apply-templates select="xqxft:ftStopWords"/> </xsl:otherwise> </xsl:choose> <xsl:apply-templates select="xqxft:ftStopWordsInclExcl"/> </xsl:template>

<xsl:template match="xqxft:ftStopWords"> <xsl:call-template name="ftStopWords_type"/> </xsl:template>

<xsl:template name="ftStopWords_type"> <xsl:choose> <xsl:when test="xqxft:ref"> <xsl:text>at "</xsl:text> <xsl:value-of select="xqxft:ref"/> <xsl:text>" </xsl:text> </xsl:when> <xsl:otherwise> <xsl:apply-templates/> </xsl:otherwise> </xsl:choose> </xsl:template>

<xsl:template match="xqxft:list"> <xsl:call-template name="parenthesizedList"/> <xsl:text> </xsl:text> </xsl:template>

<xsl:template match="xqxft:FTStopWordsInclExcl"> <xsl:apply-templates/> </xsl:template>

<xsl:template match="xqxft:union"> <xsl:text>union </xsl:text> <xsl:call-template name="ftStopWords_type"/> </xsl:template>

<xsl:template match="xqxft:except"> <xsl:text>except </xsl:text> <xsl:call-template name="ftStopWords_type"/> </xsl:template>

<xsl:template match="xqxft:language"> <xsl:text>using language "</xsl:text> <xsl:apply-templates/> <xsl:text>"</xsl:text> <xsl:value-of select="$NEWLINE"/> </xsl:template>

<xsl:template match="xqxft:wildcard"> <xsl:text>using </xsl:text> <xsl:apply-templates/> <xsl:value-of select="$NEWLINE"/> </xsl:template>

<xsl:template match="xqxft:ftAnd"> <xsl:apply-templates select="xqx:firstOperand"/> <xsl:text> ftand </xsl:text> <xsl:apply-templates select="xqx:secondOperand"/> <xsl:text> </xsl:text> </xsl:template>

<xsl:template match="xqxft:ftOr"> <xsl:apply-templates select="xqx:firstOperand"/> <xsl:text> ftor </xsl:text> <xsl:apply-templates select="xqx:secondOperand"/> <xsl:text> </xsl:text> </xsl:template>

<xsl:template match="xqxft:ftMildNot"> <xsl:apply-templates select="xqx:firstOperand"/> <xsl:text> not in </xsl:text> <xsl:apply-templates select="xqx:secondOperand"/> <xsl:text> </xsl:text> </xsl:template>

<xsl:template match="xqxft:ftUnaryNot"> <xsl:text>ftnot </xsl:text> <xsl:apply-templates select="xqx:operand"/> <xsl:text> </xsl:text> </xsl:template>

<xsl:template match="xqxft:ftPrimaryWithOptions"> <xsl:apply-templates/> </xsl:template>

<xsl:template match="xqxft:ftPrimary"> <xsl:apply-templates/> </xsl:template>

<xsl:template match="xqxft:parenthesized"> <xsl:text>( </xsl:text> <xsl:apply-templates/> <xsl:text> ) </xsl:text> </xsl:template>

<xsl:template match="xqxft:ftWords"> <xsl:apply-templates/> </xsl:template>

<xsl:template match="xqxft:ftWordsValue"> <xsl:apply-templates/> </xsl:template>

<xsl:template match="xqxft:ftWordsLiteral"> <xsl:apply-templates/> </xsl:template>

<xsl:template match="xqxft:ftWordsExpression"> <xsl:text> { </xsl:text> <xsl:apply-templates/> <xsl:text> } </xsl:text> </xsl:template>

<xsl:template match="xqxft:ftAnyAllOption"> <xsl:value-of select="."/> <xsl:text> </xsl:text> </xsl:template>

<xsl:template match="xqxft:ftTimes"> <xsl:text>occurs </xsl:text> <xsl:apply-templates/> <xsl:text> times </xsl:text> </xsl:template>

<xsl:template match="xqxft:ftExtensionSelection"> <xsl:apply-templates select="xqxft:pragma"/> <xsl:text> { </xsl:text> <xsl:apply-templates select="xqxft:ftSelection"/> <xsl:text> } </xsl:text> </xsl:template>

<xsl:template match="xqxft:pragma"> <xsl:value-of select="$PRAGMA_BEGIN"/> <xsl:apply-templates select="xqx:pragmaName"/> <xsl:value-of select="$SPACE"/> <xsl:value-of select="xqx:pragmaContents"/> <xsl:value-of select="$PRAGMA_END"/> </xsl:template>

<xsl:template match="xqxft:ftExtensionOption"> <xsl:text>using option </xsl:text> <xsl:apply-templates/> </xsl:template>

<xsl:template match="xqxft:ftExtensionName"> <xsl:if test="@xqx:prefix"> <xsl:value-of select="@xqx:prefix"/> <xsl:value-of select="$COLON"/> </xsl:if> <xsl:apply-templates/> </xsl:template>

<xsl:template match="xqxft:ftExtensionValue"> <xsl:text> "</xsl:text> <xsl:apply-templates/> <xsl:text>"</xsl:text> </xsl:template>

<xsl:template match="xqxft:weight"> <xsl:text> weight { </xsl:text> <xsl:apply-templates/> <xsl:text> } </xsl:text> </xsl:template>

</xsl:stylesheet> The following example is based on the data and queries of one of the use cases in [XQuery and XPath Full Text 1.0 Use Cases] [XQuery and XPath Full Text 1.0 Use Cases] E.2 XQueryX stylesheet for XQuery and XPath Full Text 1.0 [XQuery and XPath Full Text 1.0 Use Cases] a Comparison of the results of the Full Text XQueryX-to-XQuery Full Text transformation given in this document with the XQuery Full Text solutions in the [XQuery and XPath Full Text 1.0 Use Cases] The XQuery Full Text Use Cases solution given for the example is provided only to assist readers of this document in understanding the Full Text XQueryX solution. There is no intent to imply that this document specifies a "compilation" or "transformation" of XQuery Full Text syntax into Full Text XQueryX syntax. In the following example, note that path expressions are expanded to show their structure. Also, note that the prefix syntax for binary operators like "and" makes the precedence explicit. In general, humans find it easier to read an XML representation that does not expand path expressions, but it is less convenient for programmatic representation and manipulation. XQueryX is designed as a language that is convenient for production and modification by software, and not as a convenient syntax for humans to read and write. Finally, please note that white space, including new lines, have been added to some of the Full Text XQueryX documents and XQuery Full Text expressions for readability. That additional white space is not necessarily produced by the Full Text XQueryX-to-XQuery Full Text transformation. Here is Q4 from the [XQuery and XPath Full Text 1.0 Use Cases] declare function local:filter ( $nodes as node()*, $exclude as element()* ) as node()* { for $node in $nodes except $exclude return typeswitch ($node) case $e as element() return element {node-name($e)} { $e/@*, filter( $e/node() except $exclude, $exclude ) } default return $node };

for $book in doc("http://bstore1.example.com/full-text.xml") /books/book let $irrelevantParts := for $part in $book//part let score $score := $part contains text "usability test.*" using wildcards where $score < 0.5 return $part where count($irrelevantParts) < count($book//part) return filter($book, $irrelevantParts) <?xml version="1.0"?> <xqx:module xmlns:xqxft="http://www.w3.org/2007/xpath-full-text" xmlns:xqx="http://www.w3.org/2005/XQueryX" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.w3.org/2007/xpath-full-text http://www.w3.org/2007/xpath-full-text/xpath-full-text-10-xqueryx.xsd http://www.w3.org/2005/XQueryX http://www.w3.org/2005/XQueryX/xqueryx.xsd">

<xqx:mainModule> <xqx:prolog> <xqx:functionDecl> <xqx:functionName xqx:prefix="local">filter</xqx:functionName> <xqx:paramList> <xqx:param> <xqx:varName>nodes</xqx:varName> <xqx:typeDeclaration> <xqx:anyKindTest/><xqx:occurrenceIndicator>*</xqx:occurrenceIndicator> </xqx:typeDeclaration> </xqx:param> <xqx:param> <xqx:varName>exclude</xqx:varName> <xqx:typeDeclaration> <xqx:elementTest/><xqx:occurrenceIndicator>*</xqx:occurrenceIndicator> </xqx:typeDeclaration> </xqx:param> </xqx:paramList> <xqx:typeDeclaration> <xqx:anyKindTest/> </xqx:typeDeclaration> <xqx:functionBody> <xqx:flworExpr> <xqx:forClause> <xqx:forClauseItem> <xqx:typedVariableBinding> <xqx:varName>node</xqx:varName> </xqx:typedVariableBinding> <xqx:forExpr> <xqx:exceptOp> <xqx:firstOperand> <xqx:varRef> <xqx:name>nodes</xqx:name> </xqx:varRef> </xqx:firstOperand> <xqx:secondOperand> <xqx:varRef> <xqx:name>exclude</xqx:name> </xqx:varRef> </xqx:secondOperand> </xqx:exceptOp> </xqx:forExpr> </xqx:forClauseItem> </xqx:forClause> <xqx:returnClause> <xqx:typeswitchExpr> <xqx:argExpr> <xqx:varRef> <xqx:name>node</xqx:name> </xqx:varRef> </xqx:argExpr> <xqx:typeswitchExprCaseClause> <xqx:variableBinding>e</xqx:variableBinding> <xqx:sequenceType> <xqx:elementTest/> </xqx:sequenceType> <xqx:resultExpr> <xqx:computedElementConstructor> <xqx:tagNameExpr> <xqx:functionCallExpr> <xqx:functionName xqx:prefix="fn">node-name</xqx:functionName> <xqx:arguments> <xqx:varRef> <xqx:name>e</xqx:name> </xqx:varRef> </xqx:arguments> </xqx:functionCallExpr> </xqx:tagNameExpr> <xqx:contentExpr> <xqx:sequenceExpr> <xqx:pathExpr> <xqx:stepExpr> <xqx:filterExpr> <xqx:varRef> <xqx:name>e</xqx:name> </xqx:varRef> </xqx:filterExpr> </xqx:stepExpr> <xqx:stepExpr> <xqx:xpathAxis>child</xqx:xpathAxis> <xqx:attributeTest> <xqx:attributeName> <xqx:star/> </xqx:attributeName> </xqx:attributeTest> </xqx:stepExpr> </xqx:pathExpr> <xqx:functionCallExpr> <xqx:functionName xqx:prefix="fn">filter</xqx:functionName> <xqx:arguments> <xqx:exceptOp> <xqx:firstOperand> <xqx:pathExpr> <xqx:stepExpr> <xqx:filterExpr> <xqx:varRef> <xqx:name>e</xqx:name> </xqx:varRef> </xqx:filterExpr> </xqx:stepExpr> <xqx:stepExpr> <xqx:xpathAxis>child</xqx:xpathAxis> <xqx:anyKindTest/> </xqx:stepExpr> </xqx:pathExpr> </xqx:firstOperand> <xqx:secondOperand> <xqx:varRef> <xqx:name>exclude</xqx:name> </xqx:varRef> </xqx:secondOperand> </xqx:exceptOp> <xqx:varRef> <xqx:name>exclude</xqx:name> </xqx:varRef> </xqx:arguments> </xqx:functionCallExpr> </xqx:sequenceExpr> </xqx:contentExpr> </xqx:computedElementConstructor> </xqx:resultExpr> </xqx:typeswitchExprCaseClause> <xqx:typeswitchExprDefaultClause> <xqx:resultExpr> <xqx:varRef> <xqx:name>node</xqx:name> </xqx:varRef> </xqx:resultExpr> </xqx:typeswitchExprDefaultClause> </xqx:typeswitchExpr> </xqx:returnClause> </xqx:flworExpr> </xqx:functionBody> </xqx:functionDecl> </xqx:prolog> <xqx:queryBody> <xqx:flworExpr> <xqx:forClause> <xqx:forClauseItem> <xqx:typedVariableBinding> <xqx:varName>book</xqx:varName> </xqx:typedVariableBinding> <xqx:forExpr> <xqx:pathExpr> <xqx:stepExpr> <xqx:filterExpr> <xqx:functionCallExpr> <xqx:functionName xqx:prefix="fn">doc</xqx:functionName> <xqx:arguments> <xqx:stringConstantExpr> <xqx:value>http://bstore1.example.com/full-text.xml</xqx:value> </xqx:stringConstantExpr> </xqx:arguments> </xqx:functionCallExpr> </xqx:filterExpr> </xqx:stepExpr> <xqx:stepExpr> <xqx:xpathAxis>child</xqx:xpathAxis> <xqx:nameTest>books</xqx:nameTest> </xqx:stepExpr> <xqx:stepExpr> <xqx:xpathAxis>child</xqx:xpathAxis> <xqx:nameTest>book</xqx:nameTest> </xqx:stepExpr> </xqx:pathExpr> </xqx:forExpr> </xqx:forClauseItem> </xqx:forClause> <xqx:letClause> <xqx:letClauseItem> <xqx:typedVariableBinding> <xqx:varName>irrelevantParts</xqx:varName> </xqx:typedVariableBinding> <xqx:letExpr> <xqx:flworExpr> <xqx:forClause> <xqx:forClauseItem> <xqx:typedVariableBinding> <xqx:varName>part</xqx:varName> </xqx:typedVariableBinding> <xqx:forExpr> <xqx:pathExpr> <xqx:stepExpr> <xqx:filterExpr> <xqx:varRef> <xqx:name>book</xqx:name> </xqx:varRef> </xqx:filterExpr> </xqx:stepExpr> <xqx:stepExpr> <xqx:xpathAxis>descendant-or-self</xqx:xpathAxis> <xqx:nameTest>part</xqx:nameTest> </xqx:stepExpr> </xqx:pathExpr> </xqx:forExpr> </xqx:forClauseItem> </xqx:forClause> <xqx:letClause> <xqx:letClauseItem> <xqxft:ftScoreVariableBinding>score</xqxft:ftScoreVariableBinding> <xqx:letExpr> <xqxft:ftContainsExpr> <xqxft:ftRangeExpr> <xqx:varRef> <xqx:name>part</xqx:name> </xqx:varRef> </xqxft:ftRangeExpr> <xqxft:ftSelectionExpr> <xqxft:ftSelection> <xqxft:ftSelectionSource> <xqxft:ftPrimaryWithOptions> <xqxft:ftPrimary> <xqxft:ftWords> <xqxft:ftWordsValue> <xqxft:ftWordsLiteral> <xqx:stringConstantExpr> <xqx:value>usability test.*</xqx:value> </xqx:stringConstantExpr> </xqxft:ftWordsLiteral> </xqxft:ftWordsValue> </xqxft:ftWords> </xqxft:ftPrimary> <xqxft:wildcard> <xqxft:value>using wildcards</xqxft:value> </xqxft:wildcard> </xqxft:ftPrimaryWithOptions> </xqxft:ftSelectionSource> </xqxft:ftSelection> </xqxft:ftSelectionExpr> </xqxft:ftContainsExpr> </xqx:letExpr> </xqx:letClauseItem> </xqx:letClause> <xqx:whereClause> <xqx:lessThanOp> <xqx:firstOperand> <xqx:varRef> <xqx:name>score</xqx:name> </xqx:varRef> </xqx:firstOperand> <xqx:secondOperand> <xqx:decimalConstantExpr> <xqx:value>0.5</xqx:value> </xqx:decimalConstantExpr> </xqx:secondOperand> </xqx:lessThanOp> </xqx:whereClause> <xqx:returnClause> <xqx:varRef> <xqx:name>part</xqx:name> </xqx:varRef> </xqx:returnClause> </xqx:flworExpr> </xqx:letExpr> </xqx:letClauseItem> </xqx:letClause> <xqx:whereClause> <xqx:lessThanOp> <xqx:firstOperand> <xqx:functionCallExpr> <xqx:functionName xqx:prefix="fn">count</xqx:functionName> <xqx:arguments> <xqx:varRef> <xqx:name>irrelevantParts</xqx:name> </xqx:varRef> </xqx:arguments> </xqx:functionCallExpr> </xqx:firstOperand> <xqx:secondOperand> <xqx:functionCallExpr> <xqx:functionName xqx:prefix="fn">count</xqx:functionName> <xqx:arguments> <xqx:pathExpr> <xqx:stepExpr> <xqx:filterExpr> <xqx:varRef> <xqx:name>book</xqx:name> </xqx:varRef> </xqx:filterExpr> </xqx:stepExpr> <xqx:stepExpr> <xqx:xpathAxis>descendant-or-self</xqx:xpathAxis> <xqx:nameTest>part</xqx:nameTest> </xqx:stepExpr> </xqx:pathExpr> </xqx:arguments> </xqx:functionCallExpr> </xqx:secondOperand> </xqx:lessThanOp> </xqx:whereClause> <xqx:returnClause> <xqx:functionCallExpr> <xqx:functionName xqx:prefix="local">filter</xqx:functionName> <xqx:arguments> <xqx:varRef> <xqx:name>book</xqx:name> </xqx:varRef> <xqx:varRef> <xqx:name>irrelevantParts</xqx:name> </xqx:varRef> </xqx:arguments> </xqx:functionCallExpr> </xqx:returnClause> </xqx:flworExpr> </xqx:queryBody> </xqx:mainModule> </xqx:module> Application of the stylesheet in E.2 XQueryX stylesheet for XQuery and XPath Full Text 1.0 declare function local:filter($nodes as node()*, $exclude as element()*) as node() { ( for $node in ($nodes except $exclude) return ( typeswitch($node) case $e as element() return element {fn:node-name($e)} {( $e/child::attribute(*), fn:filter( ($e/child::node() except $exclude), $exclude ) )} default return $node ) ) };

( for $book in fn:doc("http://bstore1.example.com/full-text.xml")/child::books/child::book let $irrelevantParts:= ( for $part in $book/descendant-or-self::part let score $score := $part contains text "usability test.*" using wildcards where ($score < 0.5) return $part ) where (fn:count($irrelevantParts) < fn:count($book/descendant-or-self::part)) return local:filter($book, $irrelevantParts) ) XQuery 1.0: An XML Query Language (Second Edition) et. al. latest version XML Path Language (XPath) 2.0 (Second Edition) et. al. latest version XQuery 1.0 and XPath 2.0 Functions and Operators (Second Edition) latest version XQuery 1.0 and XPath 2.0 Data Model (XDM) (Second Edition) et. al. latest version XML Syntax for XQuery 1.0 (XQueryX) (Second Edition) latest version XQuery and XPath Full Text 1.0 Requirements latest version XQuery and XPath Full Text 1.0 Use Cases latest version A. Phillips and M. Davis. Tags for Identifying Languages. http://tools.ietf.org/html/bcp47 [RFC 4646] [RFC 4647] [RFC 3066] S. Bradner. Key Words for use in RFCs to Indicate Requirement Levels. http://www.ietf.org/rfc/rfc2119.txt H. Alvestrand. Tags for the Identification of Languages. http://www.ietf.org/rfc/rfc3066.txt A. Phillips and M. Davis. Tags for Identifying Languages. http://www.ietf.org/rfc/rfc4646.txt A. Phillips and M. Davis. Matching of Language Tags. http://www.ietf.org/rfc/rfc4647.txt Documentation Guidelines for the Establishment and Development of Monolingual Thesauri, Geneva: International Organization for Standardization, 2nd edition, 1986. ISO/IEC 13249-2 Information technology --- Database languages --- SQL Multimedia and Application Packages --- Part 2: Full-Text. Geneva: International Organization for Standardization, 2nd edition, 2003. M. Davis. Unicode Standard Annex #29 Text Boundaries, revision 11, 2006. http://www.unicode.org/reports/tr29/ We would like to thank the members of the XQuery and XPath Full-Text group for their fruitful discussions. We would like to thank the following people for their contributions on earlier drafts of this document. Andrew Cencini, Microsoft - [email protected] Andrew Eisenberg, IBM - [email protected] Nimish Khanolkar, Microsoft - [email protected] Ashok Malhotra, Oracle - [email protected] Tapas Nayak, Microsoft - [email protected] Roland Seiffert, IBM - [email protected] An AllMatches FTSelection Distance Operator Restriction. Full-text queries Ignored nodes Each Match FTSelection Negation Restriction 1. FTUnaryNot FTAnd Negation Restriction 2. FTUnaryNot FTOr FTPosFilter FTSelection Order Operator Restriction. A paragraph implementation-defined A phrase implementation-defined A QueryItem QueryTokenInfo A QueryTokenInfo The score A sentence implementation-defined Single Language Restriction. A StringExclude StringMatch TokenInfo A StringInclude StringMatch TokenInfo A StringMatch StringInclude StringExclude A token implementation-defined A TokenInfo Formally, tokenization Scoring may be influenced by adding weight declarations Window Operator Restriction. An anchoring selection An and-selection ftand A cardinality selection FTWords FTTimes A case option A diacritics option A distance selection FTDistance An extension option implementation-defined An extension selection implementation-defined A full-text contains expression A full-text selection Implementation-dependent Implementation-defined A language option Match options The order in which effective match options for an FTWords match option application order Each of the alternatives of production FTMatchOption other than FTExtensionOption match option group MAY A mild-not selection not in MUST A not-selection ftnot An or-selection ftor An ordered selection Positional filters A primary full-text selection FTWords FTTimes FTSelection FTExtensionSelection A scope selection FTScope Those items are called the search context SHOULD A stemming option A stop word option tokens A thesaurus option A wildcard option A window selection FTWindow This appendix provides a summary of features defined in this specification whose effect is explicitly implementation-defined Tokenization, including the definition of the term "tokens", SHOULD implementation-defined SHOULD A phrase is an ordered sequence of any number of tokens. Beyond that, phrases are implementation-defined A sentence is an ordered sequence of any number of tokens. Beyond that, sentences are implementation-defined A tokenizer A paragraph is an ordered sequence of any number of tokens. Beyond that, paragraphs are implementation-defined A tokenizer Implementations are free to provide implementation-defined The set of expressions (of form ExprSingle) that can be assigned to a score variable in a let-clause is implementation-defined If an expression not supported by the scoring algorithm is passed to the scoring algorithm, the result is implementation-defined When a sequence of query tokens is considered as a phrase, it matches a sequence of tokens in the tokenized form of the text being searched only if the two sequences correspond in an implementation-defined The match option application order implementation-defined The "language" option influences tokenization, stemming, and stop words in an implementation-defined MAY implementation-defined The set of valid language identifiers is implementation-defined If an invalid language identifier is specified, then the behavior is implementation-defined When a processor evaluates text in a document that is governed by an xml:lang attribute and the portion of the full-text query doing that evaluation contains an FTLanguageOption that specifies a different language from the language specified by the governing xml:lang attribute, the language-related behavior of that full-text query is implementation-defined It is implementation-defined If a query specifies thesaurus relationships not supported by the thesaurus, or does not specify a relationship, the behavior is implementation-defined The effect of specifying a particular range of levels in an FTThesaurusID is implementation-defined If a query does not specify the number of levels, and the implementation does not follow the default of querying all levels of hierarchical relationships, then the number of levels of hierarchical relationships queries is implementation-defined It is implementation-defined An implementation-defined Normally a stop word matches exactly one token, but there may be implementation-defined The "stop words default" option specifies that an implementation-defined An implementation recognizes an implementation-defined implementation-defined An implementation recognizes an implementation-defined implementation-defined The conditions under which tokenization of two equal items produces different tokens is implementation-defined An implementation may impose an implementation-defined For certain full-text components of the static context (see C Static Context Components implementation-defined Michael Dyck 2008-08-19 3.6.3 Distance Selection Change the prose around example 2 to agree with the meeting 170 decision re distance filter applied to n>2 matches. Jim Melton 2008-08-26 Sections 2.3.1 Using Weights Within a Scored FTContainsExpr 3 Full-Text Selections D Error Conditions Ensure that all description of weights, valid values, and errors related to invalid values are captured in one place, and generalize the description of the error raised for invalid values. Resolves bug 5812. Jim Melton 2008-08-26 Section 3.8 Extension Selections Correct syntax of the second and third examples. Resolves bug 5879. Jim Melton 2008-08-26 Section 3.5.4 Not-Selection Rewrite the second example to correspond to the search document content. Resolves bug 5884. Jim Melton 2008-08-26 Section 3.5.4 Not-Selection Rewrite the third example to correspond to the search document structure. Resolves bug 5885. Mary Holstege 2008-10-30 Miscellaneous Add text to clarify status of attribute searches (bug 5975) and of scope and constraints on score variables (bug 6094). Mary Holstege 2008-10-30 Full-Text Selections Change scope of weight variables to FTPrimary rather than FTSelection to resolve bug 6178. Michael Dyck 2008-11-07 2.3.1 Using Weights Within a Scored FTContainsExpr Collateral changes due to Bug 6178's relocation of FTWeight within grammar. Michael Dyck 2008-11-07 3.4 Match Options, 3.6 Positional Filters Add text intended to clarify scope of options/filters. [Bug 5977] Jim Melton 2008-11-12 Appendix E XML Syntax (XQueryX) for XQuery and XPath Full Text 1.0 Revise Schema and stylesheet to reflect change in position of "weight" in the grammar. Pat Case 2008-11-23 Removed count > 0 from examples Removed count > 0 from examples in 3.4.3 Thesaurus Option and 3.4.7 Stop Word Option. Pat Case 2008-11-23 Corrected FTLanguage example Corrected the FTLanguage example in Section 3.4.1 to search in content//p and to search for "salon de thé". Michael Dyck 2009-01-08 4.2.7.9 FTDistance Rewrite the very last sentence. [Bug 6303] Michael Dyck 2009-01-08 Grammar, Full-Text Selections Get rid of "multiple-match-options" as an extra-grammatical constraint, and instead make it a conventional static error (FTST0019). Jim Melton 2009-01-28 Section 1 Introduction Inserted paragraph stating that Notes are not normative. Jim Melton 2009-01-28 Appendix D Error Conditions Added entries and descriptions for errors XPST0003, XQST0013, and XQST0079. Mary Holstege 2009-02-19 Window Selection 3.6.2 Window Selection Provided corrected commentary on windowing example; clarified non-applicability of contents of attribute in this example. Mary Holstege 2009-02-19 FTUnit and FTBigUnit 5.2.3 FTUnit and FTBigUnit Clarified that we did not mean for 'word' and 'words' to be optional units. Mary Holstege 2009-02-19 Thesaurus Option 3.4.3 Thesaurus Option D Error Conditions Added error FTST0018 for missing thesauri. Michael Dyck 2009-02-26 4.2.7.7 FTContent Rewrite fts:ApplyFTContent() to fix 'entire content'. In the process, introduce fts:TokenInfoCoversTokenPosition(), fts:getLowestTokenPosition(), & fts:getHighestTokenPosition() and drop fts:isStartToken() & fts:isEndToken() Michael Dyck 2009-02-26 4.2.7.7 FTContent Add example of counter-intuitive behaviour due to @isContiguous=false. Michael Dyck 2009-03-16 4.2.7.9 FTDistance Rearrange the section to bring together the parts about fts:ApplyFTDistance. Michael Dyck 2009-03-16 4.2.4 Formal semantics functions & 4.2.7.9 FTDistance Dissolve 4.2.4 and integrate its content into 4.2.7.9 (now 4.2.6.9). Michael Dyck 2009-03-17 4.2.6.8 FTWindow Fix two typos in fts:joinIncludes(). (Bug 6386) Michael Dyck 2009-03-17 fts:evaluate, fts:FormRange, fts:UnaryNotHelper, fts:calcStopWords Fix some typos that would give static errors or type errors if you tried to treat these functions as actual XQuery code. Mary Holstege 2009-03-19 fts:ApplyFTScopeDifferentSentence, fts:ApplyFTScopeDifferentParagraph Fix code to properly handle the case where there is a single string include. Pat Case 2009-03-28 Changed FTStopword examples 1 and 3 Removed the stemming match option and changed the operand propagation to propagating in FTStopword examples 1 and 3. Michael Dyck 2009-04-14 3.4.1 Language Option Fix typo in example query. Jim Melton 2009-04-30 E.3.1.2 A Solution in Full Text XQueryX Fix element names in example. (Bug 6840) Michael Dyck 2009-06-06 3.2 Search Tokens and Phrases Add wording to handle cases involving empty sequences of items or tokens. (Bug 6813) Michael Dyck 2009-06-08 4.2.4 FTWords Fix error in fts:ApplyFTWordsAnyWord(). Michael Dyck 2009-07-05 3.4.2 Wildcard Option, Appendix D Errors Wording changes, add more examples. Add error FTDY0020. Michael Dyck 2009-07-06 3.4.2 Wildcard Option Delete Note re wildcards and token boundaries. Michael Dyck 2009-07-06 E.1 XQueryX representation of XQuery and XPath Full Text 1.0 Add element declaration for ftMatchOptions. Michael Dyck 2009-07-14 E.2 XQueryX stylesheet for XQuery and XPath Full Text 1.0 Bug fixes. Mary Holstege 2009-09-03 Various Modify match option syntax to avoid conflicts with XQuery and XQuery Update Facility: change "with" to "using" and "without" to "using no", plus adding "using" before other options for consistency. Change 'ftcontains' to 'contains text'. Bugzilla bugs 7247 and 7271. Jim Melton 2009-09-18 E.1 XQueryX representation of XQuery and XPath Full Text 1.0 Fixed grammar within a comment re: Bugzilla Bug 7247. Fixed grammar within comments re: Bugzilla Bug 7271; also fixed enumeration values and element content names for complex types ftStemOption, ftThesaurusOption, ftStopwordOption, and ftWildCardOption re: Bugzilla Bug 7271. Jim Melton 2009-09-18 E.2 XQueryX stylesheet for XQuery and XPath Full Text 1.0 Fixed templates' generated output re: Bugzilla Bug 7247 and Bugzilla Bug 7271. Jim Melton 2009-09-18 E.3 XQueryX for XQuery and XPath Full Text 1.0 example Fixed "original" XQuery, the XQueryX code, and the XQuery resulting from transforming the XQueryC code re: Bugzilla Bug 7247 and Bugzilla Bug 7271. Michael Dyck 2009-10-10 Sections 3.4.* and Appendix A Change match option syntax to resolve bug 7271. Michael Dyck 2009-10-10 Section 2.2.1 and Appendix A Change FTContainsExpr syntax to resolve bug 7247. Pat Case 2009-11-10 Section 4.2.5.1 Types Corrected an error in the FTMatchOptions schema for FTDiacriticsOption changing case insensitive to diacritics insensitive and changing case sensitive to diacritics sensitive Michael Dyck 2009-11-25 Section 2.3.1 and 3.1.1, Appendix A and B Change syntax of FTWeight: replace RangeExpr with "{" Expr "}". Change examples and prose accordingly. Mary Holstege 2009-12-10 Appendix D, Section 4.2.5 Change semantics schema and functions to replace "with/without". Michael Dyck 2010-01-25 Section 3.5.3 Mild-Not Selection Clarify that FTDY0017 is a dynamic error, and not statically detectable. Mary Holstege 2010-05-25 Sections 3.4.3 Thesaurus Option and 3.4.7 Stop Word Option Resolve [9677] by making it clear the relative URIs for thesauri and stop word lists should be resolved against the base URI in the static context. Mary Holstege 2010-05-25 Appendix D Error Conditions Resolve [9681] by replacing the obsolete operators && and || by ftand and ftor, respectively. Michael Dyck 2010-07-11 Sections 3.4.7 Stop Word Option and 4.2.5.8 FTStopWordOption Replace wording that said that stop words are "removed from the search" or "removed from the set of query tokens". Michael Dyck 2010-07-11 Sections 3.4.7 Stop Word Option and I Checklist of Implementation-Defined Features (Non-Normative) Change two occurrences of "default stop words" to "stop words default". (Leftovers from 2009-10-10.) Michael Dyck 2010-07-13 Sections 3.4.7 Stop Word Option and I Checklist of Implementation-Defined Features (Non-Normative) Add wording to state that an implementation-defined comparison is used to determine whether a query token appears in a collection of stop words. Michael Dyck 2010-09-06 4.2.6.6 FTScope Fix the 'where' conditions in fts:ApplyFTScopeDifferentSentence() and fts:ApplyFTScopeDifferentParagraph(). And tweak the prose following the former. (See Bugzilla Bug 9448.) Jim Melton 2010-09-07 E.2 XQueryX stylesheet for XQuery and XPath Full Text 1.0 Fixed two templates (weights and extensions). Michael Dyck 2010-09-13 Section 3.2 (Search Tokens and Phrases), Appendix A (EBNF for XQuery 1.0 Grammar with Full-Text extensions), and Appendix B (EBNF for XPath 2.0 Grammar with Full-Text extensions) In the production for FTWordsValue, change "Literal" to "StringLiteral" Michael Dyck 2010-09-16 Appendix I (Checklist of Implementation-Defined Features) Make the list more consistent with the statements of implementation-defined features in the body of the document. (Delete 2 items, add 3, reword several, reorder several.) Michael Dyck 2010-11-29 Section 4.2.5.{2,3,7} Pass $noThesaurusOptions down to fts:lookupThesaurus(). (See Bugzilla bug 11209.) Michael Dyck 2010-11-29 Section 4.2.5.{1,3,7} Eliminate the now-redundant $thesaurusLanguage parameter from fts:lookupThesaurus(), and the now-unnecessary (and not syntactically justified) "language" attribute from complexType ftThesaurusOption. (Clean-up after previous change.) Mary Holstege 2010-12-06 Section 3.6 Require FTOrder positional filters to be applied before other positional filters. Mary Holstege 2011-01-04 Sections 2.3.1 and 3.1.1. In resolution of bug 11582, clarify that the second "constraint" on scoring algorithms is actually just a specific consequence of the rules for errors and optimization. Mary Holstege 2011-01-04 Section 3.4.3 In resolution of bug 11444, change the default from "all levels" to "all levels or to an implementation-defined number of levels." Michael Dyck 2011-02-28 Section 3.4.3 and Appendix I. Introduce FTLiteralRange for FTThesaurusID. Clarify implementation-definedness of levels. (See bugs 11821 and 12036.) Mary Holstege 2011-03-01 Section 3 Resolve bug 12057 by clarifying what the sample tokenization is for the examples in the specification. Jim Melton 2011-03-08 Appendix E Updated Full Text XQueryX schemas and stylesheet for bugs 11821 and 12036

Related documents

• latn gap#649807
• grek gap#649808
• mong gap#649809
• nkoo gap#649810
• osge osa gap#649811