<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="487"/><title>Parallel Suffix Arrays for Linguistic Pattern Search</title><pubinfo>Proceedings of Recent Advances in Natural Language Processing,pages 487^-94, Hissar, Bulgaria, 12-14 September 2011.</pubinfo><author surname="Goller" givenname="Johannes"><org  name="Ichigayatamachi Shinjuku-ku"/></author></firstpageheader><frontmatter><p><b>Parallel Suffix Arrays for Linguistic Pattern Search</b></p><p><b>Johannes Goller</b></p><p>Macmillan, Digital Science Chiyoda Bldg., 2-37 Ichigayatamachi Shinjuku-ku, Tokyo</p><p>jogojapan@gmail.com</p></frontmatter><abstract>The paper presents the results of an ana­lysis of the merits and problems of us­ing suffix arrays as an index data struc­ture for annotated natural-language cor­pora. It shows how multiple suffix arrays can be combined to represent layers of an­notation, and how this enables matches for complex linguistic patterns to be identi­fied in the corpus quickly and, for a large subclass of patterns, with greater theoreti­cal efficiency than alternative approaches. The results reported include construction times and retrieval times for an annotated corpus of 1.9 billion characters in length, and a range of example patterns of varying complexity. </abstract></header><body><section number="1" title="Introduction"><p>Empirical linguistic studies require access to large corpora of text, and they benefit greatly when the text is stored in a form that enables the efficient re­trieval of specific elements, such as sentences that match a pattern defined by a linguist. The size and contents of the corpus, the type and structure of its annotations, and the form of patterns involved vary greatly; the present paper deals with the re­quirements of only a subset of linguistic studies, which are characterized as follows:</p><p>• The corpus is large (hundreds of millions of words), but not extremely large (hundreds of millions of documents);</p><p>• Annotations exist in any number of layers, for example a layer of part-of-speech (POS) an­notations and a layer of semantic role labels, but the annotations on each individual layer are non-overlapping and non-ambiguous;</p><p>• A pattern is essentially a regular expression, made up of literals (to be matched against the</p><p>text), annotations (each with a specification of the layer it is expected to be found in) and wildcard elements ("gaps");</p><p>• The retrieval results are expected to be de­livered within seconds or minutes (that is, not necessarily as fast as web search), and to be comprehensive (that is, to contain all matches, not only the top-N defined by some relevancy ranking);</p><p>• New patterns are generated constantly, per­haps by many different users or automated programs in parallel, while the text is largely static.</p><p>Corpus search engines that respond to a similar, albeit not identical, set of requirements include the Corpus Workbench<footnote anchor="1"/>, WebCorp Linguist's Search Engine<footnote anchor="2"/> and Manatee/Bonito (Rychly, 2007). The implementation of all of these systems relies on the principle of inverted files, which is the main al­ternative to the suffix arrays presented here<footnote anchor="3"/>. Both approaches are described and briefly compared in section 2, a direct comparison is also available in (Puglisi et al., 2006). Sections 3 and 4 introduce the concept of parallel suffix arrays and describe how it enables annotations and complex pattern search, including patterns equivalent to finite state machines. Section 5 describes results obtained us­ing an actual implementation of parallel suffix ar­rays.</p></section><section number="2" title="The Two Main Approaches to Indexing 2.1   Inverted Files"><p>The concept of inverted files requires the text to be <i>tokenized, </i>that is, to be segmented into tokens (usually roughly equivalent to words).<page local="2" global="488"/> The index consists of a searchable dictionary of the tokens (e.g. a hash table or sorted list), and a link connect­ing each token with its inverted list, i.e. the list of positions where the token is found (where the po­sition of a token is defined as the token offset, i.e. the number of tokens to its left).</p><footnote label="1">http://cwb.sourceforge.net/</footnote><footnote label="2">http://www.webcorp.org.uk/</footnote><footnote label="3"> Suffix arrays are frequently used for n-gram analyses (e.g. Yamamoto and Church (1998)), but without the ability to process complex search patterns.</footnote><p>The match result for a search pattern that con­sists of a single token t is then readily retrieved by determining the dictionary entry correspond­ing to t (which if hashing is used typically takes O(|t|) time, where |t| is the length of t in char­acters) and returning the entire inverted list It. The length of the list corresponds to the num­ber of occurrences of t, occ(t). If the pattern is a sequence of tokens <i>P := </i><i>t1,...,tr</i>, the re­trieval strategy is to determine all inverted lists in O(|ti|) + ... + |tr|) time, to then identify the in­verted list of the least frequent token, i.e. <i>It </i>such that <i>ß = </i>argminj occ(t^), and to finally check for each of the positions p G It/Li whether it lies in a match for the entire pattern <i>P. </i>That requires, for each p a look-up in the remaining <i>r — </i>1 in­verted lists, specifically, for each 1 &lt; <i>k</i><i> </i><i>&lt;</i><i> </i><i>ß,</i><i> </i>a look-up to check whether p — k G It k, and for each 1 &lt; k &lt; (r — ß) to check whether p + k G <i>It^+k</i>. Since inverted lists are usually stored as sorted lists of integers, a look-up in <i>Iti</i>requires O(log occ(tj)) time, hence the total time taken to identify all matches for <i>P </i>is</p><doubt alpha="55.9" length="34" tooSmall="False" monospace="0.0">O(S|ti| +occ(tß)J2logocc(tfc)) (1)</doubt><p>Storing annotations in the index is straightforward: Modify the inverted lists so as to store positions as character offsets (rather than token offsets), and the length of <i>t </i>in characters along with each oc­currence of <i>t.</i><i> </i>Annotations can then be indexed in the same way as ordinary tokens, with character offset and length, and the procedure above can be modified so as to take into account the length of each tj when computing the positions of adjacent tokens. This enables patterns using a mix of text and annotations, i.e. with some of the tj referring to text, others to annotation. The time bound of(1) is unchanged.</p><subsection number="2.2" title="Suffix Arrays"><p>A suffix <i>array </i>is any representation of the lexi­cographically sorted list of all suffixes of a text, where suffix is defined as any substring beginning</p><doubt alpha="56.2" length="16" tooSmall="False" monospace="0.0">T= SA= bwt= lcp=</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">$</doubt><p>Figure 1: Suffix array SA for the string <i>T </i>=abxabdae$, along with auxiliary data struc­tures <i>bwt, lep </i>and "brackets" indicating the match ranges for substrings ab, a and b.</p><p>somewhere in the text and ending at the end of the text, i.e. there are <i>n </i>suffixes in a text of length <i>n. </i>Rather than storing copies of all the substrings, the suffix array is usually represented as a list of <i>n </i>integers, each indicating the starting position of a suffix. An example of this is shown in Fig. 1: The suffix array itself consists only of the integer list SA; the lower part of Fig. 1 shows the strings corresponding to each position, written vertically. Suffix arrays have an important property related to substring searches: Given a text T, its suffix array SA and a search pattern <i>P,</i><i> </i>the set of starting po­sitions of matches for <i>P </i>in <i>T </i>forms a continuous range in SA, as each match is the initial part of a suffix of T. Because of the lexicographical sort­ing, these suffixes must be adjacent to each other in the suffix array. For example, the set of matches for substring ab in Fig. 1 is the range [2; 3] of SA (corresponding to positions 4 and 1 of T). This shall be called the <b>range property </b>of suffix arrays.</p><p>Recent improvements in search algorithms for suffix arrays, cf. Navarro and Makinen (2007), make it possible to identify the match range for P in O(|P|) time<footnote anchor="4"/>, and since no tokenization is required, recombining matches for individual to­kens as in the case of inverted files is unneces­sary. However, it is impossible to store annotation-related information in the suffix array. The following two sections describe a new concept, <i>parallel suffix arrays, </i>and how it enables annotations and more powerful search patterns.<page local="3" global="489"/></p><footnote label="4">Strictly speaking, the time is bound by O(|P|(1 + log |E|/ log log n)), where |E| is the size of the alphabet. However that is asymptotically equivalent to O(|P|) when the alphabet is as much smaller than the text as it is the case for large-scale natural-language corpus search. See Navarro and Makinen (2007, 42) for details.</footnote><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>1</p><p>a</p></td><td class="cell"><p>2</p><p>b</p></td><td class="cell"><p>3</p><p>x</p></td><td class="cell"><p>4</p><p>a</p></td><td class="cell"><p>5</p><p>b</p></td><td class="cell"><p>6</p><p>d</p></td><td class="cell"><p>a</p></td><td class="cell"><p>8</p><p>e</p></td><td class="cell"><p>9</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>9_</i></p></td><td class="cell"><p></p></td><td class="cell"><p>1</p></td><td class="cell"><p>7</p></td><td class="cell"><p><i>5_</i></p></td><td class="cell"><p></p></td><td class="cell"><p>6</p></td><td class="cell"><p>8_</p></td><td class="cell"><p>3</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>e</p></td><td class="cell"><p>x</p></td><td class="cell"><p>$</p></td><td class="cell"><p>d</p></td><td class="cell"><p>a</p></td><td class="cell"><p>a</p></td><td class="cell"><p>b</p></td><td class="cell"><p>a</p></td><td class="cell"><p>b</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>_0</p></td><td class="cell"><p>_0</p></td><td class="cell"><p>2_</p></td><td class="cell"><p></p></td><td class="cell"><p>_0</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>0_</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>$</p></td><td class="cell"><p>IT</p></td><td class="cell"><p>~a</p></td><td class="cell"><p>a</p></td><td class="cell"><p><b>IT</b></p></td><td class="cell"><p>b</p></td><td class="cell"><p>d</p></td><td class="cell"><p>e</p></td><td class="cell"><p>x</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>b</p></td><td class="cell"><p>b</p></td><td class="cell"><p>e</p></td><td class="cell"><p>d</p></td><td class="cell"><p>x</p></td><td class="cell"><p>a</p></td><td class="cell"><p>$</p></td><td class="cell"><p>a</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>d</p></td><td class="cell"><p>x</p></td><td class="cell"><p>$</p></td><td class="cell"><p>a</p></td><td class="cell"><p>a</p></td><td class="cell"><p>e</p></td><td class="cell"><p></p></td><td class="cell"><p>b</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>a</p></td><td class="cell"><p>a</p></td><td class="cell"><p></p></td><td class="cell"><p>e</p></td><td class="cell"><p>b</p></td><td class="cell"><p>$</p></td><td class="cell"><p></p></td><td class="cell"><p>d</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>e</p></td><td class="cell"><p>b</p></td><td class="cell"><p></p></td><td class="cell"><p>$</p></td><td class="cell"><p>d</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>a</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>$</p></td><td class="cell"><p>d</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>a</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>e</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>a</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>e</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>$</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>e</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>$</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></subsection></section><section number="3" title="Parallel Suffix Arrays"><p>The first step is to allow annotations to enter the index. In the following it is assumed that a text T G S* of length n is given, and one layer of q annotations such that each annotation (ai,pi,^i) consists of a label ai G S*, a starting position <i>pi &lt; n </i>and a length pi indicates where in T the substring annotated with <i>ai </i>starts, ^ indicates the number of T-characters it covers. For example, given</p><doubt alpha="26.3" length="38" tooSmall="False" monospace="0.0">A=((ai,pi,4), (ct2,p2,4),...,K&gt;p«,4?))</doubt><p>T = is but a dream within a dream and POS-annotations V, Conj etc., the annotation layer might look like this:</p><p>A = ((V, 1, 2), (Conj, 4, 3), (Det, 8,1), (N, 10, 5), (Prep, 16, 6), (Det, 23,1), (N, 25, 5)) .</p><p>There are two ways to bring these annotations into the suffix-array-based index for T :</p><p><b>Method 1: Single-integer annotations. </b>Three steps need to be performed: (1) Each distinct an­notation label is mapped to a unique integer (e.g. using a hash table), that is, a new annotation al­phabet A is created, in which each annotation is represented as one integer. (2) An extra integer is introduced in A, below represented by 0, which is used as a dummy annotation for all areas of T that are not covered by any element of A (in the example above, this applies to the space charac­ters between words). (3) A is replaced by a string A' G A* containing the new annotation symbols in the order of the T-positions they refer to, and a bitvector B<i>T</i>"A of length n indicating the starting positions of annotations relative to T. The exam­ple above now becomes:</p><doubt alpha="5.9" length="17" tooSmall="False" monospace="0.0">A' =1020304050304</doubt><doubt alpha="8.6" length="35" tooSmall="False" monospace="0.0">BT«A=10110011110000110000011110000,</doubt><p>where V has been mapped to 1, Conj to 2, and so forth. The next step is to construct a suffix ar­ray SAa' from the A-string A', along with aux­iliary data structures required for fast searches, cf. Navarro and Makinen (2007). That enables fast searches for sequences consisting solely of anno­tations. It will later be shown how the bitvector is used to accomplish searches for mixed patterns, that is, patterns that contain both, T-sequences and A-sequences.</p><p><b>Method 2: Complex annotations. </b>In some sit­uations annotations are themselves complex and one would like to be able to search inside them, rather than mapping them to atomic integers. This is accomplished by appending a new character H G S to every label ai as a separation mark, and then concatenating all labels to a new string A':</p><p>A' = VjjConjjjDetjjNjjPrepjjDetjjNjj</p><p>In addition, two bitvectors BT"A and BA"A are defined, the former in the same way as in method 1, while the latter is of length | A'| and has a 1 wherever a new annotation starts in A':</p><doubt alpha="10.0" length="30" tooSmall="False" monospace="0.0">BA«A= 101000010001010000100010</doubt><p>Again, a suffix array SAa' for A' enables search­ing for substrings of annotations as well as se­quences of annotations. The jj-symbols prevent undesired matches across annotation-boundaries.</p><p><b>How the bitvectors are used for mixed </b>T/A'<b>patterns. </b>Both the bitvector of the first, and the bitvectors of the second method need to undergo an indexing process, during which a <i>rank </i>index and a <i>select </i>index are generated for each bitvector, defined as follows: Let B be a bitvector of length b and i, j &lt; b, then</p><p>rankB(i) := the total number of 1s in B selectB(j) := i s.t. there are j 1s in B[1..i] .</p><p>Using techniques described by Jacobson (1989), it is possible to construct, in O(b) time, data struc­tures that implement these functions, such that a lookup can be performed in O(1) time and no more than b + o(b) bits of space are con­sumed in total (including the bitvector itself). In the case of single-integer annotations (method 1), rankBT«a and selector«a are constructed; in the case of complex annotations, these and rankBA«A and selectBA«A are constructed. In addition, in both cases the inverse suffix ar­rays for T and A' must be computed and stored in memory: Given a suffix array SA, its inverse is defined as and invSA can be generated from SA in linear time. To see how these data structures work to­gether, consider a mixed pattern a A, where a G S* is a substring match against T and A is a sub­string match against the annotations.<page local="4" global="490"/> We first as­sume that method 1 was used, hence that A A*is a sequence of annotations mapped to integers. The next step is to search the suffix arrays and de­termine the match ranges , <i>ra</i>) for a in SAT and (1a, rA) for A in SAa' . Clearly, the number of oc­currences of a in T is occ(a) = <i>ra </i>— <i>la</i>, the num­ber of matches for A is occ(A) = rA — lA. We must now check, for each a-match, whether it is followed by a A-match. Let <i>la </i>&lt; x &lt; <i>ra </i>one of the a-matches. It begins at position p = SAT [x] of T and it is | a| characters in length. Hence it is fol­lowed by a A-match if and only if an A-annotation starts at p + | a| and that annotation corresponds to a A-match in A', which is the case iff the cor­responding position in A' is a suffix in the match range (lA, rA). We therefore verify, for the candi­date offset q := p + |a|:</p><doubt alpha="59.4" length="32" tooSmall="False" monospace="0.0">invSA[j] :=i such that SA[i]=j ,</doubt><doubt alpha="52.7" length="110" tooSmall="False" monospace="0.0">A-element exists: BT"A[q] = 1 (2) Location in A': q' :=rankBr(q) (3) Is q' a A-match:    lA&lt;invSAA[q'] &lt; rA(4)</doubt><p>If SA and invSA are available for random access, all of the above can be tested in O(1) time, hence it takes O( occ(a)) time to compute the set of aA-matches from the two individual matchranges. Moreover, the procedure works in the reverse di­rection, too, starting from the A-matches and de­termining those among them that are preceded by a a-match (using select instead of rank; the time consumption becomes O( occ(A))). Hence it is possible to choose the matching direction ac­cording to whichever part of the pattern has fewer matches, i.e. let occM := min( occ(a), occ(A)), then the match combination can be computed in O( occM) time.</p><p>Without giving a detailed proof, we note that this result can be extended to general sequential patterns t1 • • • tr, ti G S*, A*: The match com­bination time depends only on the least frequent (i.e. most specific) element tM, that is, including the time taken to determine the match range for each ti, the total asymptotic time is which is obviously better than with inverted files, where the match combination time depends on the frequency of all elements, as shown in (1). This shall be called the <b>least-frequency property </b>of parallel suffix arrays<footnote anchor="5"/>. It should also be noted that for subsequences te • • • t/ such that all elements refer to the same layer, i.e. Vti G S* or Vti G A*, no match combination is required at all, since the suffix arrays do not rely on tokenization, hence t' := te • • • t/ can be searched for as a single el­ement in O(|t'|) time.</p><doubt alpha="36.4" length="22" tooSmall="False" monospace="0.0">O(£|tk|+ occ(tM)), (5)</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">k</doubt><p>Moreover, it is possible to define gaps of fixed length (measured in terms of number of T-characters, or alternatively, as number of A-annotations) between the individual elements, e.g. a pattern like a^M A, indicating a distance of 3 ar­bitrarily A-annotated elements between a and A, can be evaluated in the same asymptotic time (be­cause the length <i>I </i>of the three wildcard elements following a can be computed for each match can­didate using rank and select, and then added to the candidate position, q := p + |a| + <i>I </i>used in (2) and (3) before the match range check for A).</p><p>The property also holds when complex annota­tions and method 2 are used, at least when search­ing for prefixes of annotations, rather than arbi­trary substrings of them. The distance calculations must then be made using the rank/select indexes for BT"A to map positions between T and A, and those for BA"A to compute the string length of annotations in A'. If arbitrary substring matching in annotations is required, the match process is de­layed by a factor related to the length of A, as every position inside the annotation must be checked for being a possible match continuation.</p></section><section number="4" title="Complex Patterns"><subsection number="4.1" title="General patterns Multiple annotation layers"><p>It is straightforward to add further layers of annotation, e.g. semantic or morphological in­formation, constituent classes etc. Each layer Ai , A2, . . . is represented by an annotation string Ai, a bitvector BT"Ai, and BAi"Ai if it is com­plex. Direct mappings between layers Ai,Ajare unnecessary, as they can be emulated using BT"Ai and BT. Hence, total space consump­tion of the index grows in an additive manner as layers are added.</p><footnote label="5">The name parallel suffix arrays refers to the view of SA T and SA A as parallel layers, both related to the same under­lying text.</footnote><page local="5" global="491"/><p><b>Branching patterns</b></p><p>An important step towards more powerful search patterns is the ability to process branching pat­terns, that is, patterns that specify multiple alter­natives. This shall be denoted using a new opera­tor ©, such that a pattern ®(e1, e2,..., em) is de­fined as matching all substrings of T that match any of the subexpressions ei. If all ei are dis­tinct S-strings, the individual match sets for each ei are disjoint, and the final result corresponds to the union set of the match ranges for the ei.</p><p>But if some of the ei refer to annotations or are themselves complex, i.e. sequential patterns or ©­expressions, the individual match sets might not be disjoint, causing the end result to contain du­plicate matches, which makes it difficult to read and might cause frequency counts to be wrong. Hence, <b>duplicate elements must be detected </b>and removed from the individual match sets. This can be done either by creating a searchable result set representation, such as a hash table or tree, and in­serting the matches one by one, rejecting matches that were inserted before; or, it can be done by creating a simpler, non-searchable result list and checking for each match for any ei whether it is also a match for one of the other ej, j &lt; i. Both these methods are available when inverted files are used instead of suffix arrays, too, but if the second method is used, suffix arrays often have an advan­tage because the member check for the ej, if it is a S- or A-string, involves only an O(1) range check, whereas it would be logarithmic in an inverted file.</p><p><b>Sequences of complex elements </b>In section 3, the least-frequency property was established for sequential patterns, consisting of atomic elements and fixed-length-gaps, i.e. ex­pressions like where ei g S*, A*; Qi g {S, A}; xi integers. For even more powerful search patterns, it is impor­tant that the above can also be processed if the ei are themselves complex, i.e. sequences or branch­ing elements. This is indeed possible; the pattern then becomes a graph, and determining the least-frequent element, at which the matching should start, becomes a non-trivial problem. The num­ber of matches of a sequence or branching subele­ment cannot be calculated accurately before the entire matching process has finished, but an up­per bound can be determined: For a sequence, it is the frequency of its least-frequent subelement, for a branching element it is the sum of the frequen­cies of its branches. Based on this, it is possible to recursively determine the estimated best atomic subelement of the graph for the match combina­tion process to begin. Once it has begun, the least-frequency property takes full effect during the pro­cessing of sequential substructures, and the range property accelerates the duplicate-checks where branching substructures are involved, as described above. Both is not true of inverted files, hence the theoretical performance of parallel suffix arrays is, generally, superior even for the most complex pat­terns.</p><doubt alpha="44.0" length="25" tooSmall="False" monospace="0.0">Qi:xi    Q2:x2 &lt;3m-i:xm-l</doubt><doubt alpha="33.3" length="21" tooSmall="False" monospace="0.0">e1me2m • • •     mem,</doubt><p><b>Iteration</b></p><p>Another useful operator in powerful linguistic search patterns is the iteration operator, which is denoted by ©(e) for any atomic or complex ex­pression e. It corresponds to a sequence ecxieixi • • • xie of undetermined length. Since all its elements are identical, the least-frequency property is pre­served, even if the matching simply starts on the left end, or alternatively on the right end, and con­tinues as long as new matches are found. There­fore, iteration elements can itself become part of complex patterns, and the three operations , © and © establish a pattern syntax with the power of regular expressions, over an annotated text with any number of annotation layers, and including fixed-length gaps (wildcards).</p><doubt alpha="33.3" length="9" tooSmall="False" monospace="0.0">T:0T:0T:0</doubt></subsection><subsection number="4.2" title="Gap-filling"><p>In order to analyse linguistic patterns in specific contexts, it is desirable that not only substrings matching the entire pattern are identified, but that selected parts of the patterns, especially matches for gaps or annotation elements, can be extracted and separately returned as frequency lists. For ex­ample, if one wants to investigate the syntactic en­vironment of "discussion", i.e. usages like "dis­cussion on", "discussion with" etc., one might use a pattern like and then obtain a frequency list of the content that matched the part marked by (*). Parallel suffix arrays are particularly well-suited for this pur­pose:<page local="6" global="492"/> Firstly, it is easy to keep track of the begin­ning and ending offsets of the desired subexpres­sions during the matching processing; secondly, frequency lists are easy to generate: Given starting positions p1, p2 of two matches for (*), a compar­ison of invSA[p1] and invSA[p2] in O(1) time suf­fices to determine their lexicographic order. Once the matches are in lexicographic order, identifying duplicates and counting the frequencies of distinct strings is easy.</p><doubt alpha="61.5" length="39" tooSmall="False" monospace="0.0">T:0T:0discussionM &lt;Prep&gt;&lt;Det&gt; M&lt;N&gt;v-v-'</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(*)</doubt></subsection><subsection number="4.3" title="Look-betweens and negation"><p>Another feature related to gaps is the ability to define some of their content partially. The three types of patterns below are examples of this:</p><doubt alpha="47.1" length="17" tooSmall="False" monospace="0.0">Q:x:y Q:£:y Q:x:y</doubt><doubt alpha="35.0" length="40" tooSmall="False" monospace="0.0">(a) e1 me2(b) e1 m[?e3]e2(c) e1 m[!e3]e2</doubt><p>(a) represents a gap of length x &lt; <i>I </i>&lt; y el­ements on the annotation level Q; (b) requires that somewhere inside the gap there must be a match for e3 (positive look-between); (c) means there must be no match for e3 in the gap (nega­tive look-between). Without going into further de­tail, it should be noted that these types of patterns can be incorporated into the matching process us­ing match combination techniques similar to those described in section 3. There is, however, a spe­cific disadvantage of suffix arrays when process­ing variable-length gaps e1 Mye2: Assuming that occ(e1) &lt; occ(e2), let (p,^) be the position and length of a match for e1. Let (qi,pi,^i) be a Q-annotation located at pi = p + ^, and the following y Q-annotations. Then we need to check whether a match for e2 is found at any of the positions pi+x,..., pi+y, which requires ö := y — x + 1 look-ups in invSAg. Hence, the gap length variability ö becomes a factor in the time complexity of the match combination pro­cess. That is not the case when inverted files are used: It then suffices to check for matches at pi+xand pi+y stored in the inverted list for e2. Since the inverted list is sorted, all other relevant matches must be located between these too and can be re­trieved in one step.</p><doubt alpha="35.1" length="37" tooSmall="False" monospace="0.0">(qi+1,pi+1,^i+1),...,(qi+y,pi+y,^i+y)</doubt></subsection></section><section number="5" title="Implementation and Results"><subsection number="5.1" title="Index construction and operation"><p>The system has been implemented as a C++ pro­gram that takes as input a file containing the text T with three layers of annotations in XML: APOS(POS-annotations); Alem (baseforms of words, in­dexed using method 1 (see section 3); AcPOS (POS along with morphological information; indexed using method 2).</p><p>The index construction is performed by first es­tablishing the parallel layers and bitvectors and then creating SAq, invSAg and, as an auxiliary data structure used to enable faster suffix array search, the wavelet tree WVTq (Grossi et al., 2003) for each layer Q G {T, Apos, Alem, Acpos }. For the construction of SAq, a multi-threaded version of the DC-algorithm (Karkkainen et al., 2006) is used, invSAg is computed in a trivial way in one pass over SAq, and the wavelet tree WVTq is con­structed using a simple multi-threaded method (for details see Goller (2011)). The only highly time-consuming steps are the constructions of SAq and WVTq. Their running times are given in Table 1.</p><p>For efficient pattern search, it is necessary to keep all data structures in main memory at all times. Compression methods for SA, invSA and WVT are available, cf. Navarro and Makinen (2007), but unfortunately, using them causes the time complexity of pattern search to be increased by a factor of O(log n), eliminating the advan­tage it has over inverted files. As a result, us­ing parallel suffix arrays requires a large amount of RAM. The implementation used to obtain the results described above was found to require œ (0.06 • N)/1024 MB for a corpus of N characters with the three annotation layers described above. Hence, on a 32-bit desktop computer with about 3 GB of memory available, a corpus of œ 52 million characters (œ 7 million words) can be processed efficiently. Therefore, although optimizing the im­plementation's use of RAM is certainly possible, it is quite clear that possibilities to use the described approach in linguistic practice depend on whether servers with sufficiently large RAM are available, and affordable.</p></subsection><subsection number="5.2" title="Pattern Search"><p>Table 2 presents response times for various kinds of patterns and illustrates, as expected, that the performance varies greatly depending on the com­plexity of the pattern; more specifically, it de-<page local="7" global="493"/></p><p>Table 1: Construction times on three different system configurations. The text is 1.97 billion characters (375 million words) in length and contains approx. 27,000 distinct baseforms of words. Test A was performed on a server with AMD-Opteron CPUs 8356 (total 16 threads) and the hard drive mounted through NFS, tests B and C were conducted on a server with Intel Xeon X7560 processors (total 64 threads) and the hard drive installed locally. Time durations are given in the format h:mm.</p><p>Table 2: Pattern processing times using hardware configuration A (see Table 1). Search time (identifying the set of match positions) and extraction time (extracting matches, but not including result printing). #..#-elements refer to Alem, (..) to APOS, $..$ to AcPOS. Elements enclosed in [..] are marked for separate extraction and frequency counting (gap-filling). Times are in milliseconds.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><b>Threads Hard Available Used      drive RAM</b></p></td><td class="cell"><p>SAT WVTt</p></td><td class="cell"><p>SApos WVTpos</p></td><td class="cell"><p>SAlem WVTlem</p></td><td class="cell"><p>SAcpos WVTcpos</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>A B C</p></td><td class="cell"><p>10         NFS     128 GB 20         Direct   512 GB 45         Direct   512 GB</p></td><td class="cell"><p>3:17 3:48 2:00 3:06 2:00 2:37</p></td><td class="cell"><p>1:30 1:13 0:50 1:05 0:51 0:54</p></td><td class="cell"><p>1:27 11:46 0:49 8:05 0:55 6:16</p></td><td class="cell"><p>3:03 2:03 2:00 1:57 1:49 1:43</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><b>Pattern</b></p></td><td class="cell"><p><b>#results</b></p></td><td class="cell"><p><b>Search</b></p></td><td class="cell"><p><b>Extraction</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p><b>time</b></p></td><td class="cell"><p><b>time</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p><b>(ms)</b></p></td><td class="cell"><p><b>(ms)</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>P1</p></td><td class="cell"><p>millions</p></td><td class="cell"><p>5,857</p></td><td class="cell"><p>106</p></td><td class="cell"><p>200</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>P2</p></td><td class="cell"><p>thousands of</p></td><td class="cell"><p>7,526</p></td><td class="cell"><p>74</p></td><td class="cell"><p>399</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>P3</p></td><td class="cell"><p>#thousand# of</p></td><td class="cell"><p>7,696</p></td><td class="cell"><p>168</p></td><td class="cell"><p>343</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>P4</p></td><td class="cell"><p>discussion(IN)(NN)</p></td><td class="cell"><p>1,296</p></td><td class="cell"><p>213</p></td><td class="cell"><p>80</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>P5</p></td><td class="cell"><p>discussion$pr$$n$</p></td><td class="cell"><p>1,894</p></td><td class="cell"><p>372</p></td><td class="cell"><p>118</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>P6</p></td><td class="cell"><p>discussion[$pr$$n$]</p></td><td class="cell"><p>1,894</p></td><td class="cell"><p>530</p></td><td class="cell"><p>111</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>P7</p></td><td class="cell"><p>Apos :0:2        <i>t </i>-n</p><p>#preparation#   m   (IN) <b>m </b><b>0 </b>((NN), (NNS))</p></td><td class="cell"><p>752</p></td><td class="cell"><p>191</p></td><td class="cell"><p>28</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>P8</p></td><td class="cell"><p>Apos :0:2 <i>t</i><i>-n</i></p><p>(JJ)(NN)(NN)   A   (IN) <b>m </b><b>0 </b>((NN), (NNS))</p></td><td class="cell"><p>13,229</p></td><td class="cell"><p>4,065</p></td><td class="cell"><p>621</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>P9</p></td><td class="cell"><p>(NN)(NN)Apm0:2(IN)Ï<footnote anchor="0"/> <b>0 </b>((NN), (NNS))</p></td><td class="cell"><p>129,723</p></td><td class="cell"><p>36,711</p></td><td class="cell"><p>5,178</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="8" global="494"/><p>pends on the "most specific atomic element" of the pattern. An element is atomic, if it refers to one layer (text or annotation) exclusively and con­tains no gaps. For example, the POS sequence (JJ)(NN)(NN) in P8, which would consist of three tokens in a standard inverted-file configuration, is atomic, as all three sub-elements refer to the same layer APOS and can therefore be matched against SAAPOS in a single step. In accordance with the least-frequency property, the overall response time for the entire pattern depends on the number of occurrences of the most specific atomic element, which in this case is (JJ)(NN)(NN), rather than such high-frequency individual tokens as (JJ) or (NN). If the most specific atom is modified to be less specific, as in P9, the search time is increased by a factor of œ 9.</p></subsection><subsection number="5.3" title="Discussion and Conclusion"><p>The approach presented appears to be effective, especially for complex patterns that contain at least one relatively specific element. It provides efficient solutions for special tasks like context-specific pattern matching and frequency-list gen­eration (described as gap-filling above), and it does not require any kind of tokenization, neither on the level of the main text, nor on the level of annotations and is hence suitable for corpora that involve annotations on the morpheme level, or across token boundaries, as well as for languages or writing systems that are hard to tokenize. Its biggest disadvantage is its high memory consump­tion, which however is likely to be less important in the future, as ever larger RAM hardware be­comes available at increasingly low cost.</p><p>Although this has not been discussed in detail in previous sections, it is important to point out that the approach is <i>not </i>suitable in situations that call for frequent updates to the text or the annotations. The index structures described above, especially rank and select indexes for bit vectors as well as the suffix arrays themselves cannot be updated ef­ficiently. Although data structures for suffix arrays that can be searched as well as dynamically up­dated are known, cf. (Russo et al., 2008; Gonzalez and Navarro, 2008), using them would cause de­lays in the order of O(log n) (where n is the length of the text) in lookups of select, rank and SA, hence rendering the system considerably less effi­cient the corresponding version of an inverted file based system.</p><p>There are plans to release an open-source ver­sion of the implementation used for the tests de­scribed above as a corpus exploration tool for lin­guists before the end of the year.</p></subsection></section><references><p>Johannes Goller. 2011. <i>Exploring text corpora using index structures. PhD thesis. </i>To appear, Centrum fur Informations- und Sprachverarbeitung, Ludwig-Maximilians-Universitat München.</p><p>Rodrigo Gonzalez and Gonzalo Navarro. 2008. Im­proved dynamic rank-select entropy-bound struc­tures. In <i>LNCS 4957/2008, LATIN 2008: Theoreti­cal Informatics, </i>pages 374-386, Berlin / Heidelberg. Springer.</p><p>Roberto Grossi, Ankur Gupta, and Jeffrey Scott Vit­ter. 2003. High-order entropy-compressed text in­dexes. In <i>SODA '03: Proceedings of the 14th an­nual ACM-SIAM symposium on discrete algorithms, </i>pages 841-850, Philadelphia, PA, USA. Society for Industrial and Applied Mathematics.</p><p>Guy Jacobson. 1989. Space-efficient static trees and graphs. In <i>Proc. ofthe 30th IEEE Symposium on Foundations of Computer Science (FOCS), </i>pages 549-554.</p><p>Juha Karkkainen, Peter Sanders, and Stefan Burkhardt. 2006. Linear work suffix array construction. <i>J. ACM, </i>53(6):918-936.</p><p>Gonzalo Navarro and Veli Makinen. 2007. Com­pressed full-text indexes. <i>ACM Comput. Surv., </i>39(1):2.</p><p>Simon Puglisi, W. Smyth, and Andrew Turpin. 2006. Inverted files versus suffix arrays for locating pat­terns in primary memory. In Fabio Crestani, Paolo Ferragina, and Mark Sanderson, editors, <i>String Pro­cessing and Information Retrieval, </i>volume 4209 of <i>Lecture Notes in Computer Science, </i>pages 122-133. Springer Berlin / Heidelberg.</p><p>Luis M. Russo, Gonzalo Navarro, and Arlindo L. Oliveira. 2008. Dynamic fully-compressed suffix trees. In <i>CPM '08: Proceedings of the 19th an­nual symposium on Combinatorial Pattern Match­ing, </i>pages 191-203, Berlin, Heidelberg. Springer­Verlag.</p><p>P. Rychly. 2007. Manatee/bonito - a modular corpus manager. In P. Sojka and A. Horak, editors, <i>First Workshop on Recent Advances in Slavonic Natural Language Processing 2007, </i>Faculty of Informatics, Masaryk University, Botanicka 68a, 60200 Brno, Czech Republic.</p><p>Mikio Yamamoto and Kenneth W. Church. 1998. Us­ing suffix arrays to compute term frequency and doc­ument frequency for all substrings in a corpus. <i>Com­putational Linguistics, </i>27:28-37.</p></references></body></article>