<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="29"/><title>Parsing in Parallel on Multiple Cores and GPUs</title><author surname="Johnson" givenname="Mark"><org  name="Mie University" country="Japan" city="Mie"/></author></firstpageheader><frontmatter><p><b>Parsing in Parallel on Multiple Cores and GPUs</b></p><p><b>Mark Johnson</b></p><p>Centre for Language Sciences and Department of Computing Macquarie University Sydney, Australia Mark.Johnson@MQ.edu.au</p></frontmatter><abstract>This paper examines the ways in which par­allelism can be used to speed the parsing of dense PCFGs. We focus on two kinds of parallelism here: Symmetric Multi-Processing (SMP) parallelism on shared-memory multi-core CPUs, and Single-Instruction Multiple- Thread (SIMT) parallelism on GPUs. We de­scribe how to achieve speed-ups over an al­ready very efficient baseline parser using both kinds of technology. For our dense PCFG parsing task we obtained a 60 x speed-up us­ing SMP and SSE parallelism coupled with a cache-sensitive algorithm design, parsing sec­tion 24 of the Penn WSJ treebank in a little over 2 secs. </abstract></header><body><section number="1" title="Introduction"><p>Performance improvements in computing come in­creasingly through greater parallelism. This paper studies ways in which this parallelism can be used to improve the speed of PCFG parsing in computa­tional linguistics. Although we focus on a particu­lar task here (constructing the inside chart for dense PCFGs), we expect the insights to be generally ap­plicable.</p><p>There are three major ways in which computers are becoming more parallel. At the broadest level, it is now common to network large numbers of com­puters together into clusters, which are controlled by software such as the Message-Passing Interface (MPI) (Gropp et al., 1999) or Map-Reduce (Lin and Dyer, 2010).</p><p>At a lower level, even commodity computers typ­ically have multiple processors or cores, which are connected by a high-speed bus to a shared memory, enabling Symmetric Multi-Processor (SMP) paral­lelism. SMP parallelism is typically controlled by software such as OpenMP (Chapman et al., 2007) or pThreads. Commodity computers also possess on-chip parallel floating point vectorised arithmetic units. The CPUs we used here have SSE (Streaming SIMD Extensions) vectorised arith­metic, where SIMD abbreviates "Single-Instruction Multiple-Data". SSE is enabled by appropriate com­piler flags.</p><p>Finally, Graphics Processor Units (GPUs) are increasingly becoming both less specialised and more powerful (on many commodity machines they can perform more floating-point operations per sec­ond than the CPU); as we will see here, a GPU can yield quite respectable parsing performance. GPUs are designed for massively parallel Single-Instruction Multiple-Thread (SIMT) programs; each GPU thread is comparatively slow, but the GPU can execute hundreds or thousands of threads in parallel. GPUs are typically programmed using tools such as OpenCL or CUDA (Sanders and Kandrot, 2011).</p><p>We concentrate on SMP multi-core and GPU par­allelism in this paper because we expect that com­munication latencies with conventional network­ing hardware make parallel parsing with networked clusters impractically inefficient. Communication latency is much less of a problem with shared mem­ory SMP and GPU parallelism as communication takes place over the machine's high-speed bus.</p><p>A base-line approach for exploiting parallelism in parsing is simply to parse different sentences in parallel on separate instances of the parser. This</p><p>Mark Johnson. 2011. Parsing in Parallel on Multiple Cores and GPUs. In <i>Proceedings of Australasian Language Technology Association Workshop, </i>pages 29—37 is likely to be the best way to exploit parallelism with networked clusters and SMP multi-core ma­chines when parsing a large corpus of sentences off­line. However, there are situations where parsing must be on-line; e.g., when parsing is a compo­nent of a system that interacts with users, or with machine-learning algorithms such as Metropolis-Hastings Sampling that update after each sentence is parsed (Johnson et al., 2007).<page local="2" global="30"/></p></section><section number="2" title="Previous work"><p>Parsing in parallel has been studied for several decades, and space constraints prevent anything but a cursory summary here. Hill and Wayne (1991) identified the basic data dependencies between the entries in chart cells, and discussed their implica­tions for parallel parsing. Nijholt (1994) also stud­ied the order in which chart cells can be filled, and discusses its implications for a variety of shift-reduce and chart-based parsing algorithms. Thomp­son (1994) pointed out that the close relationship be­tween CKY parsing and matrix multiplication can be exploited for parsing in parallel; we rely on sim-iliar observations below. Ninomiya et al. (1997) de­scribed an agenda-based approach for parallelising CKY parsing on large SMP machines, while Bor-dim et al. (2002) describes the implementation of a CKY parser on Field-Programmable Gate Arrays (FPGAs). Sandstrom (2004) describes a parallel im­plementation of Earley's parsing algorithm. Dun-lop et al. (2010) stresses importance of minimising cache misses in the design of efficient parsing algo­rithms and described how to restructure the CKY al­gorithm to reduce grammar constant,<footnote anchor="1"/> which as we show below has a dramatic impact on parallel SMP parsing.</p></section><section number="3" title="The CKY algorithm for dense PCFGs"><p>This section introduces the basic CKY parsing al­gorithm used below. Here we're assuming that the grammar is in Chomsky-Normal Form (CNF), i.e., all rules in the grammar are of the form <i>A</i><i> </i><i>—</i><i> </i><i>BC</i></p><p>The "grammar constant" refers to the variation in parsing time as a function of grammar size. Standard analyses study how parsing time varies as a function of sentence length while the grammar is held constant; the choice of grammar affects parsing time in such analyses via a "grammar constant".</p><doubt alpha="38.6" length="57" tooSmall="False" monospace="0.0">foriin0, ..., n—1:forain0, ...,m—1:C[i,i+1,a] = 7TW[i],a]</doubt><doubt alpha="53.3" length="45" tooSmall="False" monospace="0.0">forgapin2, ..., n:foriin0, ...,n—gap: k=i+gap</doubt><doubt alpha="40.0" length="120" tooSmall="False" monospace="0.0">forain0, . . . ,m—1:C[i,k,a]= 0forjin..., k—1:forbin0, ..., m—1:forcin0, ..., m—1: C[i,k,a] += R[a,b,c]*C[i,j,b]*Cj,k,c]</doubt><doubt alpha="83.3" length="6" tooSmall="False" monospace="0.0">where:</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">m</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">n</doubt><doubt alpha="45.5" length="11" tooSmall="False" monospace="0.0">W[i] T[w,A]</doubt><p>number of nonterminals in grammar length of input string word in input string at position i probability of A — w <i>C[i,k,A]</i><i> </i>inside probability of A spanning (i, k)</p><doubt alpha="64.5" length="31" tooSmall="False" monospace="0.0">R[A,B,C]   probability of A —BC</doubt><p>Figure 1: Pseudo-code for baseline CKY algorithm for dense PCFG parsing and an explanation of the variables used in the algorithm. All arrays are in row-major order, except that the inside chart C uses specialised indexing to take advantage of the fact that the first string position index is always less than the second.</p><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">30</doubt><page local="3" global="31"/><p><i>R[A,</i><i> </i><i>B,</i><i> </i><i>C</i><i> </i><i>]</i><i> </i>probability of A — <i>BC</i> where.</p><doubt alpha="41.9" length="31" tooSmall="False" monospace="0.0">foriin0, ..., n—1.forain0, ...,</doubt><doubt alpha="42.9" length="21" tooSmall="False" monospace="0.0">C[i,i+l,a] =7TW[i],a]</doubt><doubt alpha="54.5" length="55" tooSmall="False" monospace="0.0">forgapin2, ..., n.foriin0,..,n—gap: k = i+gap BC = Zero</doubt><doubt alpha="41.4" length="169" tooSmall="False" monospace="0.0">forjin..., k—1.forbin0, ..., m—1.forcin0, ..., m—1.BC[b,c]+=C[ij,b]*Cj,k,c]forain0, . . . ,m—1.C[i,k,a]= 0forbin0, . . .,m—1.forcin0, ..., m—1.C[i,k,a]+=R[a,b,c]*BC[b,c]</doubt><p>m number of nonterminals in grammar n length of input string</p><p>W [i] word in input string at position i</p><p>T<i>[w,A]</i><i> </i>probability of A — w <i>C[i,k,A</i><i> </i>inside probability of A spanning (i, <i>k)</i></p><p>BC an m x m scratch array <i>Zero </i>an m x m array of zeros.</p><figure caption="Figure 2: Pseudo-code for the factored CKY algorithm for dense PCFG parsing."></figure><p>or A — w, where A, B and C are nonterminals and w is a terminal (Aho and Ullman, 1972). In this paper we focus on <i>dense </i>PCFGs, i.e., where most of the possible rules have positive probability. Dense grammars with these properties occur in ap­plications such as unsupervised grammar induction. While sparse grammars have many important appli­cations, there are many different possible patterns of sparsity, and the optimal parsing algorithm may de­pend on the particular sparsity pattern the grammar instantiates. Moreover, it is extremely difficult to de­velop effective search procedures (such as heuristic A* search) for dense grammars in which most rules have approximately equal probabilities, so this is a situation where a brute-force exhaustive calculation of the kind that the algorithms discussed below may well be the preferred approach.</p><p>We focus on CKY-style pure bottom-up parsing algorithms here because of their simplicity, and with dense grammars their performance often equals or exceeds that of more complex parsing algorithms. if every possible chart cell will be filled with a non­trivial probability, a predictive parsing algorithm (such as the Earley algorithm) will have to instan­tiate every cell anyway.</p><p>We also focus on the construction of the "in­side chart" here, i.e., P(A =&gt;<b>+ </b>wi,...,w<b>J-i) </b>for each nonterminal A and 0 &lt; i &lt; j &lt; n, where n is the number of words in the input string. Constructing the inside chart is the crucial O(n<footnote anchor="3"/>) step of the Inside-Outside algorithm for estimat­ing PCFGs (Charniak, 1993), and this computation is typically the rate-limiting computation in PCFG sampling algorithms (Johnson et al., 2007) as well. By replacing a sum with a max, the same algo­rithms can be used to constuct the Viterbi chart, from which a most probable parse tree can be extracted in O(n<footnote anchor="2"/>) time, so again the Inside computation is the rate-limiting step. Because our grammar is dense we used pre-allocated fixed-sized arrays to hold the grammar rules and the inside chart, thus min­imising expensive memory management and pointer arithmetic (in our experience unless great care is taken while coding, these costs can dominate pars­ing time).</p><p>Figure 1 presents pseudo-code for the baseline CKY parsing algorithm. The main part of the al­gorithm consists of six nested loops. All these loops except the outermost (over the <i>gap </i>variable) can be freely reordered without affecting correctness.<page local="4" global="32"/> We experimented with a large number of reorderings of these variables; in preliminary experiments we found that the order presented here resulted in fastest parsing.<footnote anchor="2"/></p><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">31</doubt><p>Dunlop et al. (2010) point out that the algorithm in Figure 1 requires a grammar rule retrieval for each mid-point <i>j</i><i> </i>of each <i>(i,</i><i> </i>k) span (as well as each com­bination of nonterminals <i>a, b </i>and c), and show how to reduce this by factoring the algorithm as shown in Figure 2. This changes the "grammar constant" as mentioned above. They point out that this also improves the cache efficiency on modern CPUs. As we experimentally confirm below, the improvement that factoring brings can be dramatic.</p></section><section number="4" title="Multi-core SMP parallelism using OpenMP"><p>It is straight-forward to parallelise both the base­line and factored algorithms for multi-core SMP machines using OpenMP (Chapman et al., 2007). OpenMP programs are C++ programs with pragmas that indicate which loops should be parallelised. We experimented with several alternative reorderings of the loops and using an optimised matrix-algebra package (Guennebaud et al., 2010), but these did not improve parsing speed.</p><p>Developing OpenMP versions of the baseline and factored CKY algorithms is relatively straightfor­ward. The main technical challenges in parallelising the CKY algorithm are synchronising the parallel threads and ensuring that different parallel threads do not interfere with each other. This is achieved by using synchronisation constructs with implicit bar­riers, <i>thread-private </i>temporary variables and con­structs that ensure that updates to shared variables occur as <i>atomic </i>operations.</p><p>For the baseline CKY algorithm we constructed three parallel variants by parallelising (i) the outer­most two for loops (over the <i>i </i>and <i>a </i>variables) us­ing the OpenMP <i>parallel for </i>construct, (ii) the inner three loops (over <i>j, b </i>and <i>c)</i><i> </i>using a <i>parallel for re­duction </i>into a temporary variable, and (iii) a variant in which all loops (except the one involving the <i>gap </i>variable) are parallelised.</p><footnote label="2">These reorderings do not affect the theoretical complexity of the CFG parsing algorithm; it is still O(n 3 ), where n is the length of the sentence being parsed. However, the loop ordering may affect the opportunities for SIMD optimisation and mem­ory cache efficiency, since reordering the loops affects locality of memory access.</footnote><p>For the factored CKY algorithm we constructed three parallel variants by parallelising (i) the outer­most for loop (involving the <i>i </i>variable), (ii) the in­nermost variables (involving the <i>j, b, c </i>and <i>a </i>vari­ables), and (iii) a variant in which all loops (except the one involving the <i>gap </i>variable) are parallelised. Multiple <i>thread-private </i>instances of the <i>BC </i>variable are required when the outermost loops are paral­lelised, and we used the OpenMP <i>atomic </i>construct to synchronise updates to <i>BC </i>when the innermost loops were parallelised.</p></section><section number="5" title="A CUDA GPU kernel for PCFG parsing"><p>We experimented with several approaches to GPU parsing based on standard GPU matrix algebra pack­ages (NVIDIA Corporation, 2010) but results were extremely disappointing; the resulting code ran or­ders of magnitude slower than the baseline CPU-based parser above. In order to obtain results com­petitive with the multi-core SMP algorithms de­scribed above we developed custom GPU programs. Our GPU subroutines or <i>kernels </i>were written in CUDA, which is a C++ dialect for specifying pro­grams consisting of CPU code and GPU kernels (Sanders and Kandrot, 2011).</p><p>We focused on developing a CUDA implementa­tion of the factored CKY algorithm here. CUDA programming is considerably more complicated than OpenMP programming, and we don't claim to have produced an optimal program here; additional experimentation could yield further speed improve-ments.<footnote anchor="3"/></p><p>A straight-forward translation into CUDA kernels of the baseline and factored algorithms above pro­duced disappointing results: it ran approximately <i>200 times slower </i>than the factored CKY parser de­scribed above. A quick survey of the CUDA developer message boards showed that direct translations of CPU-based programs often perform poorly, and for good performance one needs to redesign the al­gorithms to take advantage of the specialised GPU hardware.<page local="5" global="33"/></p><footnote label="3">We also experimented with CUBLAS, a CUDA implemen­tation of BLAS (Basic Linear Algebra Subprograms), which we found yielded performance one to two orders of magnitude slower than our custom CUDA kernels. However, a new ver­sion of CUBLAS was released after this paper was submitted; this new version has several technical improvements that may enable it to be effective for PCFG parsing.</footnote><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">32</doubt><p>Computation on NVIDIA GPUs is organised into <i>blocks </i>of up to 1,024 parallel threads. A single CUDA <i>launch </i>starts up to hundreds of thousands of blocks; modern GPUs can execute several hundred thread blocks in parallel (the remainder are queued). Just as with SMP programming, the chief techni­cal challenges in CUDA programming are synchro­nising the parallel threads and ensuring that differ­ent parallel threads do not interfer with each other. CUDA programming is more difficult than SMP programming because each individual GPU proces­sor is much less capable than a CPU (CUDA pro­gramming is done using a restricted subset of C++), and data access must follow a very tightly prescribed set of rules if it is to perform reasonably efficiently.</p><p>Unlike on a regular CPU, the memory on a GPU has a complex organisation which the CUDA pro­grammer must be aware of; the following sketch omits many details. <i>Global memory </i>is compara­tively slow but accessible to all threads of all blocks; it is used to store persistent information and commu­nicate between threads in different blocks; we store the inside chart C in global memory. <i>Texture mem­ory </i>is a kind of global memory that permits more ef­ficient cached read-only access; we stored the gram­mar rules R and the terminal probabilities T in tex­ture memory. <i>Shared memory </i>is local to and acces­sible to all threads in the same block and is much faster than global memory; we stored (a local copy of) the <i>BC </i>array in shared memory. In addition, we also use <i>thread-local variables </i>to maintain lo­cal state and accumulate intermediate results within a single thread.</p><p>Our CUDA kernels consist of over 500 lines of code, so we only sketch them here. Our central data structures are the chart <i>C,</i><i> </i>the rule probabilities <i>R</i><i> </i>and the lexical probabilities <i>T.</i><i> </i>Our CUDA imple­mentation starts by launching a kernel that copies the terminal probabilities <i>T </i>for each of the words W into the chart C; this is easily and completely parallelised, and takes very little time.</p><p>Then it computes the chart one diagonal at a time in parallel.   It launches one or more kernels for each value of <i>gap </i>in 2,..., n. If <i>n — gap </i>is small enough then all of the chart entries C<i>[i,k,</i><i> </i><b>•</b>] (where <i>k = i + gap) </i>can be computed by a single thread block, and only one kernel launch is required. But for larger values of <i>gap </i>we decompose the compu­tation into multiple thread blocks based on the mid string position j and store intermediate results in global memory; a second kernel launch is used to reduce these into the chart entries <i>C[i,</i><i> </i><i>k,</i><i> </i><b>•</b>].</p><p>A major goal in designing the CUDA kernel was to perform the sum in the factored CKY algorithm as efficiently as pos­sible. In order to achieve this we first copy all of the relevant chart entries C[i, j, <b>•</b>] and C[j, k, <b>•</b>] from global memory into the faster shared memory (this can be done in parallel), and then accumulate the re­sults into <i>BC</i>, which is also stored in shared mem­ory. This step can also be done completely in paral­lel.</p><doubt alpha="46.2" length="26" tooSmall="False" monospace="0.0">BC[b,c] +=C[ij,b]*C{j,k,c]</doubt><p>Finally, we compute the chart entries C [i, k, <i>a]</i><i> </i>for all k = i + <i>gap </i>and all <i>a </i>in parallel. If n <b>— </b><i>gap </i>is small enough that the computation can be done in one thread group then this is done only using shared memory, otherwise temporary results are stored in global memory so they are visible to other thread blocks. The reduction in the factored CKY algorithm is also tricky, as it requires a double sum over b and c. In order to do this we generalised the parallel tree-based reduction algorithm presented in Harris (2010) to compute all of the chart entries C[i, k, <b>•</b>] as a parallel reduction.</p><doubt alpha="44.4" length="27" tooSmall="False" monospace="0.0">C[i,k,a]+= R[a,b,c]*BC[b,c]</doubt></section><section number="6" title="Evaluation on a dense PCFG"><p>We experimented with a number of different gram­mars, but because the results were generally similiar, we only describe one experiment here. The strings we parsed consist of the yields of the 1,345 trees in section 24 of the Penn WSJ treebank. Any word that did not appear 5 or more times in sections 2-21 was replaced with *<b>unc</b>*. We constructed a dense PCFG with 32 non-terminals (i.e., 32,768 binary rules) and random rule probabilities, which might be typical of the initial grammar in an unsupervised<page local="6" global="34"/></p><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">33</doubt><doubt alpha="33.3" length="3" tooSmall="False" monospace="0.0">C/3</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">id</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">ca</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">ï</doubt><p>Sa pl.</p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">0.1-</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0.01-</doubt><doubt alpha="0.0" length="6" tooSmall="False" monospace="0.0">0.001-</doubt><doubt alpha="16.7" length="6" tooSmall="False" monospace="0.0">1e-04-</doubt><doubt alpha="16.7" length="6" tooSmall="False" monospace="0.0">1e-05-</doubt><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">20</doubt><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">40</doubt><p>Sentence length</p><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">60</doubt><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">80</doubt><doubt alpha="100.0" length="6" tooSmall="False" monospace="0.0">Parser</doubt><p>• baseline</p><p>• factored</p><p>• baseline+outer SMP</p><p>• factored+outer SMP</p><doubt alpha="66.7" length="6" tooSmall="False" monospace="0.0">• CUDA</doubt><p>Figure 3: Parsing times as a function of sentence length on 1,345 sentences from section 24 of the Penn WSJ treebank for the baseline CYK parser, the baseline parser with SMP parallelism (outer loops parallelised), the factored CKY parser, the factored CKY parser (outer loops parallelised), and the CUDA implementation of the factored CKY parser.</p><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">1</doubt><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">100-</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">c</doubt><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">.&gt;</doubt><p><b>id i) i) </b>p.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">10-</doubt><p>Figure 4: Speed-up relative to the baseline CKY parser as a function of sentence length on 1,345 sentences from section 24 of the Penn WSJ treebank for the baseline parser with SMP parallelism (outer loops parallelised), the factored CKY parser, the factored CKY parser (outer loops parallelised), and the CUDA implementation of the factored CKY parser.</p><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">34</doubt><p>i­Sa p.</p><doubt alpha="50.0" length="6" tooSmall="False" monospace="0.0">"S i­I</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>.............</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>• •••</p></td><td class="cell"><p>.........</p><p>:::::::••........;;</p></td><td class="cell"><p>........</p><p>. :...... "</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>I*</b></p></td><td class="cell"><p>........*</p><p>..........</p></td><td class="cell"><p>........:</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>■</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><b>■ til</b></p></td><td class="cell"><p><b>.........</b><b><i>\</i></b><b>.........</b></p></td><td class="cell"><p><b>........ • ■</b></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>.1</b></p><p><b>1</b></p><p><b>I...J</b></p></td><td class="cell"><p><b>.............!'</b></p></td><td class="cell"><p><b>..........</b><b>'•"</b></p><p><b>.........</b></p><p><b>............••.....</b></p></td><td class="cell"><p><b>........ '</b></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>■</p><p>•••</p><p><b>■ •</b></p><p><b>!•••!•••</b></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>.... ..</b></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>•</b></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="7" global="35"/><p>Table 1: Parsing speeds of the various algorithms on 1,345 sentences from section 24 of the Penn WSJ tree-bank. Speed-up is relative to the baseline parser.</p><p>PCFG induction system using the Inside-Outside al­gorithm or a Metropolis-Hastings sampler. We used single-precision floating-point arithmetic in all ex­periments, and multiplied the terminal rule proba­bilities A — w by 10<footnote anchor="4"/> to avoid underflow.</p><p>We ran our experiments on a single node of an SGI Altix XE 320 cluster with two quad-core 3.0GHz Intel Harpertown CPUs, a 1600MHz front side bus, 16GB DDR2-800 memory and two NVIDIA Fermi s2050 GPUs, each with 448 CUDA cores running at 1.15GHz (we only used one GPU here). We used the CUDA 3.2 toolkit and gcc 4.4.4. We selected compiler flags that enabled full opti­misation, including enabling SSE3 SIMD floating­point vector subsystem, as prior experiments showed that this significantly speeds all calculations.</p><p>Table 1 presents the results of our experiments. We repeated all of our experiments twice in succes­sion and report the time of the second run here; how­ever, run-times varied by less than 1% between the two runs. Figures 3 and 4 depict parsing time and speed-up as a function of sentence length respec­tively (in order to avoid overloading the graphs, they only show a subset of the results). It's important to recognise that even our baseline parser is very fast (averaging 11 sentences/second), and both our SMP and GPU implementations were significantly faster.</p></section><section number="7" title="Conclusions"><p>We obtained large speedups over an already very fast baseline parser using both multi-core SMP and CUDA parallelism.   Parallelising the outer loops in the multi-core SMP algorithms seems to be ex­tremely effective; we see speed-ups close to the theoretical maximum of 8 times for both the base­line and factored algorithms. Parallelising the in­ner loops is devastating to performance, perhaps be­cause it interfers with cache optimisation and SSE3 SIMD vectorisation (turning off the SSE SIMD vec-torisation in these cases did not improve perfor-mace). The factored algorithm with parallelised outer loops performed fastest in our experiments, but the CUDA implementation was next best, parsing faster than all of the parallelised baseline algorithms.</p><p>As Figure 4 makes clear, the speed-up obtained by both the CUDA and factored algorithm with par­allelised outer loops relative to the baseline increases with sentence length (with the CUDA speed-up in-creasining fastest), which suggests that parallelisa-tion helps most where it is most needed, i.e., on longer sentences.</p><p>It is surprising that the CUDA implementation did not outperform the best SMP implementation. Per­haps this is because our SMP implementation uses highly-optimised, OpenMP/SSE3-parallelised code and can exploit the powerful Xeon CPUs. It is also possible that our dense PCFG parsing task is "too easy" to take full advantage of the power of the GPU; the entire corpus of 1,345 sentences took just a few seconds to parse, and it's possible that initialisa­tion and data-transfer from the host machine to the GPU imposed a significant overhead. It would be interesting to repeat the experiments described here with a grammar that is one or two orders of magni­tude larger.</p><p>In fact, as Figures 3 and 4 make clear, the CUDA implementation is comparatively slow on short sen­tences; for sentences of length 5 or less, the CUDA implementation is slower than even the baseline parser, which is consistent with the hypothesis that initialisation and data-transfer are imposing signif­icant performance costs. It would be interesting to repeat these experiments on a larger corpus with larger and perhaps sparser grammars. It also might be more efficient to parse more than one sentence in parallel on a single GPU, which might keep more of the CUDA cores busy more of the time, although we did not try this here.</p><p>There are several lessons to draw from these re­sults. First, parallelisation does not always produce speed-ups; indeed parallelising the inner loops did not improve performance on either the baseline or factored algorithms.<page local="8" global="36"/> Second, parsing algorithms that perform well on conventional CPUs may need con­siderable redesign in order to produce good results on GPUs. Third, as the impressive performance of the factored algorithm shows, good algorithm design is of crucial importance.</p><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">35</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Parser</b></p></td><td class="cell"><p><b>Sentences/sec</b></p></td><td class="cell"><p><b>Speed-up</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Baseline</b></p></td><td class="cell"><p>11</p></td><td class="cell"><p>1.0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(i) outer parallel</p></td><td class="cell"><p>84</p></td><td class="cell"><p>7.5</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(ii) inner parallel</p></td><td class="cell"><p>11</p></td><td class="cell"><p>1.0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(iii) both parallel</p></td><td class="cell"><p>29</p></td><td class="cell"><p>2.6</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Factored</b></p></td><td class="cell"><p>122</p></td><td class="cell"><p>11.0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(i) outer parallel</p></td><td class="cell"><p><b>649</b></p></td><td class="cell"><p><b>60.0</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(ii) inner parallel</p></td><td class="cell"><p>27</p></td><td class="cell"><p>2.4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(iii) both parallel</p></td><td class="cell"><p>64</p></td><td class="cell"><p>5.7</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>CUDA</b></p></td><td class="cell"><p>206</p></td><td class="cell"><p>18.4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Finally, this is an area where both the hardware and software are still rapidly improving. The num­ber of cores in a single multi-core processor is likely to increase rapidly; already it is possible to obtain commodity machines with 24 cores. The improve­ment in GPU technology is if anything even more dramatic: as well as increasing the number of pro­cessors, new GPUs are equipped with more flexi­ble buses that permit more complex kinds of data parallelism and ease programming. On the software side, up-coming versions of OpenMP will permit a greater range of efficient reduction constructs, which may permit us to avoid using the relatively expen­sive <i>atomic </i>synchronisation primitive. For GPUs, upcoming versions of CUDA will provide a vari­ety of parallel programming libraries (including for sparse matrix algebra), which may make it easier to write considerably more efficient parallel parsing al­gorithms. Thus it is reasonable to expect a dramatic improvement in parallel parsing in the near future.</p></section><section title="Acknowledgments"><p>I'd like to thank the reviewers for their thought­ful comments and suggestions. This work was supported was supported under the Australian Re­search Councils Discovery Projects funding scheme (project number DP110102593).</p></section><references><p>Alfred V. Aho and Jeffery D. Ullman. 1972. <i>The The­ory of Parsing, Translation and Compiling; Volume 1: Parsing. </i>Prentice-Hall, Englewood Cliffs, New Jersey.</p><p>Jacir Bordim, Yasuaki Ito, and Koji Nakano. 2002. Ac­celerating the CKY parsing algorithm using FPGAs. In Sartaj Sahni, Viktor Prasanna, and Uday Shukla, editors, <i>High Performance Computing HiPC 2002, </i>volume 2552 of <i>Lecture Notes in Computer Science, </i>pages 41-51. Springer Berlin / Heidelberg.</p><p>Barbara Chapman, Gabriele Jost, and Ruud van der Pas. 2007. <i>Using OpenMP: Portable Shared Memory Par­allel Programming. </i>The MIT Press, Cambridge, Mas­sachusetts.</p><p>Eugene Charniak. 1993. <i>Statistical Language Learning. </i>The MIT Press, Cambridge, Massachusetts.</p><p>Aaron Dunlop, Nathan Bodenstab, and Brian Roark. 2010. Reducing the grammar constant: an analysis of CKY parsing efficiency. Technical Report CSLU-2010-02, Oregon Health and Science University.</p><p>William Gropp, Ewing Lusk, and Anthony Skjellum. 1999. <i>Using MPI: Portable Parallel Programming with the Message Passing Interface. </i>The MIT Press, Cambridge, Massachusetts.</p><p>Gael Guennebaud, Benoît Jacob, et al. 2010. Eigen v3.0beta2. http://eigen.tuxfamily.org.</p><p>Mark Harris. 2010. Optimizing parallel reduction in CUDA. Technical report, NVIDIA Corporation.</p><p>Jane C. Hill and Andrew Wayne. 1991. A CYK approach to parsing in parallel: a case study. In <i>Proceedings ofthe twenty-second SIGCSE technical symposium on Computer science education, </i>SIGCSE '91, pages 240­245, New York, NY, USA. ACM.</p><p>Mark Johnson, Thomas Griffiths, and Sharon Goldwa-ter. 2007. Bayesian inference for PCFGs via Markov chain Monte Carlo. In <i>Human Language Technologies 2007: The Conference of the North American Chap­ter ofthe Association for Computational Linguistics; Proceedings of the Main Conference, </i>pages 139-146, Rochester, New York, April. Association for Compu­tational Linguistics.</p><p>Jimmy Lin and Chris Dyer. 2010. <i>Data-Intensive Text Processing with MapReduce. </i>Morgan and Claypool.</p><p>Anton Nijholt. 1994. Parallel approaches to context-free language parsing. In Geert Adriaens and Udo Hahn, editors, <i>Parallel Natural Language Processing, </i>pages 135-167. Ablex Publishing Corporation.</p><p>Takashi Ninomiya, Kentaro Torisawa, Kenjiro Taura, and Jun'ichi Tsujii. 1997. A parallel CKY parsing algo­rithm on large-scale distributed-memory parallel ma­chines. In <i>The Proceedings of the Pacific Association for Computational Linguistics (PACLING 97), </i>pages 223-231, Tokyo.<page local="9" global="37"/> Department of Informatics, Meisei University.</p><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">36</doubt><p>NVIDIA Corporation, 2010. <i>CUDA CUBLAS Library,</i> <i>Example:</i><i> An Introduction to General-Purpose GPU</i> <i>Programming.</i><i> </i>Addison-Wesley, Upper Saddle River,</p><doubt alpha="60.3" length="73" tooSmall="False" monospace="0.0">PG-05326-032 v02 edition. Jason Sanders and Edward Kandrot.  2011.CUDA by</doubt><p>New Jersey.</p><p>Greg Sandstrom. 2004. A parallel extension of Earleys parsing algorithm. Technical report, Earlham College.</p><p>Henry Thompson. 1994. Parallel parsers for context-free grammars: Two actual implementations compared. In Geert Adriaens and Udo Hahn, editors, <i>Parallel Natu­ral Language Processing, </i>pages 168-187. Ablex Pub­lishing Corporation.</p><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">37</doubt></references></body></article>