<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="63"/><title>Constructing Linguistically Motivated Structures from Statistical Grammars</title><pubinfo>Proceedings of Recent Advances in Natural Language Processing,pages 63-69, Hissar, Bulgaria, 12-14 September 2011.</pubinfo><author surname="Basirat" givenname="Ali"><org  name="University of Tehran" country="Iran"/></author><author surname="Faili" givenname="Heshaam"><org  name="University of Tehran" country="Iran"/></author></firstpageheader><frontmatter><p><b>Constructing Linguistically Motivated Structures from Statistical Grammars</b></p><p><b>Ali Basirat Heshaam Faili</b></p><p>NLP Lab, School of ECE, NLP Lab, School of ECE,</p><p>College of Engineering, College of Engineering,</p><p>University of Tehran, Tehran, Iran University of Tehran, Tehran, Iran a.basirat@srbiau.ac.ir hfaili@ut.ac.ir</p></frontmatter><abstract>This paper discusses two Hidden Markov Models (HMM) for linking linguistically motivated XTAG grammar and the auto­matically extracted LTAG used by MICA parser. The former grammar is a detailed LTAG enriched with feature structures. And the latter one is a huge size LTAG that due to its statistical nature is well suited to be used in statistical approaches. Lack of an efficient parser and sparseness in the supertags set are the main obstacles in us­ing XTAG and MICA grammars respec­tively. The models were trained by the standard HMM training algorithm, Baum-Welch. To converge the training algo­rithm to a better local optimum, the ini­tial state of the models also were estimated using two semi-supervised EM-based al­gorithms. The resulting accuracy of the model (about 91%) shows that the models can provide a satisfactory way for linking these grammars to share their capabilities together. </abstract></header><body><section number="1" title="Introduction"><p>Tree Adjoining-Grammar (TAG) is a tree generat­ing system that forms the object language by the set of derived trees (Joshi and Schabez, 1991). This formalism as a Mildly Context Sensitive Grammar is supposed to be powerful enough to model the natural languages (Joshi, 1985).</p><p>In the lexicalized case (LTAG), each lexical item of the object language is associated with at least one elementary structure of the grammar called <i>elementary tree. </i>Each elementary tree in LTAGs can be considered as a complex descrip­tion of its anchor that provides a domain of lo­cality over which the anchor can specify syntac­tic and semantic constraints(Bangalore and Joshi, 1999). Extended domain of locality and factoring of recursion from the domain of dependency are the main key properties of using these grammars (Bangalore and Joshi, 1999).</p><p>There are two ways for creating the set of ele­mentary trees (Faili and Basirat, 2010). The first method is the manual crafting of the elementary trees as it was done in the XTAG project (XTAG-Group, 2001). And the alternate one is the au­tomatically extraction of them from some anno­tated treebanks as it was done in (Xia, 2001; Chen, 2001). The result of the former method is a de­tailed LTAG that is enriched with semantic rep­resentation but suffers from the lack of statisti­cal information. The output of the latter one on the other hand, is a huge size LTAG that suffers from the sparseness problem in the elementary trees set but contains enough statistical informa­tion that make it suitable to be used in statistical approaches. The relatively huge size of the auto­matically extracted elementary trees set is an ob­stacle in annotating these structures with semantic representation (Chen, 2001).</p><p>One of the negative aspects of using LTAGs is their high computational complexity of pars­ing algorithm, (O(n<footnote anchor="6"/>)) (Kallmeyer, 2010). Re­garding the work presented in (Sarkar, 2007), the factors that affect the parsing complexity of such lexicalized grammars are the number of trees se­lected by the words in the input sentence and the clausal complexity of the sentence to be parsed. The first factor, named <i>Syntactic Lexical Ambigu­ity, </i>directly addresses <i>Supertagging, </i>proposed by Bangalore and Joshi (1999).</p><p>Supertagging is a robust partial parsing ap­proach that can be applied for increasing up the speed of LTAG parsing algorithm (Bangalore and Joshi, 1999). In supertagging the flexibility of linguistically motivated lexical descriptions are integrated with the robustness of statistical ap­proaches. The idea is based on extending the notion of 'tag' from the standard <i>Part Of Speech </i>to a tag that represents a rich and complex syntac­tic structure, called <i>Supertag.</i><page local="2" global="64"/><i> </i>In the lexicalized grammars like LTAGs each elementary structure of the grammar can be considered as a supertag. Supertagging itself is the task of assigning the su­pertags to each word of the processing sentence. <i>After </i>supertagging the only thing that the LTAG parser should do is to attach these selected su­pertags for creating a forest of derived/derivation trees.</p><p>Supertagging as a search problem can be mod­eled by two major methods, <i>generative model </i>and <i>classification approach </i>(Bangalore et al., 2005). In the former method the problem is modeled by a <i>Hidden Markov Model </i>and in the latter one it is modeled by the discriminant approaches like <i>SVM </i>and <i>Maximum Entropy Estimation. </i>Apply­ing each of these methods in supertagging is sub­ject to the availability of enough statistical infor­mation about the problem. Hence, due to their statistical nature, the automatically extracted LT-AGs are more suitable to be used by supertag-ging algorithm than the manually crafted LTAGs. This characteristic of automatically extracted LT-AGs caused the emergence of some powerful sta­tistical parsers like MICA (Bangalore et al., 2009) that works based on the supertagging approach.</p><p>The lack of an efficient LTAG parser for man­ually crafted LTAGs beside the weakness of the automatically extracted LTAGs in representing se­mantic representation, encouraged us to rectify these deficiencies by making an interface between these grammars. The interface was established be­tween individual elementary trees of each gram­mars such that any elementary tree of the source LTAG could be mapped onto an elementary tree of the target LTAG. The idea is similar to the <i>Hidden TAG Model </i>(Chiang and Rambow, 2006) that links many spoken dialects of a language to benefit from sharing rich resources. Here by relating two differ­ent perspectives of a natural language presented in the form of two LTAGs, we are going to share their capabilities together.</p><p>The interface was modeled as a sequence tag­ger that deals with the problem of how to map each supertag sequence of the source LTAG onto a supertag sequence of the target LTAG given the local and non-local information of the source se­quence. An unsupervised sequence tagger based on <i>Hidden Markov Model </i>(HMM) was proposed that produces a target supertag sequence given a source supertag sequence. The sequence tagger was trained using the standard HMM training al­gorithm called <i>Baum-Welch. </i>Due to this fact that the algorithm convergence is tightly depending on the HMM initial state, the initial state of the HMM also was trained intellectually using an EM-Based semi-supervised bootstrapping algorithm. The so­lution was applied on the manually crafted English XTAG grammar (XTAG-Group, 2001) as target LTAG and the automatically extracted LTAG used by MICA parser (Bangalore et al., 2009) as source</p><p>LTAG.</p><p>The significance of this work is as follow. First, as a solution for enhancing the parsing efficiency of the XTAG grammar, as it was done by Faili (2009). Second, as a fully automated method for bridging between grammars in order to share their capabilities together.</p></section><section number="2" title="Related Work"><p>Bridging between grammars in order to share their capabilities is considered by some researchers. Improving the parsing quality in the resource-poor languages (Chiang and Rambow, 2006), enriching automatically extracted LTAGs with semantic rep­resentation (Chen, 2001; Faili and Basirat, 2010; Faili and Basirat, 2011), increasing the syntac­tic coverage of lexicalized resources (Dang et al., 2000; Kipper et al., 2000), and finding the overlap between two grammars (Xia and Palmer, 2000) are considered as the most important reasons for per­forming this task.</p><p>In general, the proposed methods for perform­ing such a task could be classified into two ma­jor categories. The first category consists of the methods that try to link the grammars using the structural similarities of the grammar's elements regardless of the syntactic environments that the elements may be placed. The approaches pro­posed in (Chen, 2001), (Xia and Palmer, 2000), and (Ryant and Kipper, 2004) are classified in this category.</p><p>The second one consists of the methods that try to make the connection regarding the statis­tical information of the syntactic environments where the grammar's elements appear on. Chi­ang and Rambow (2006) by introducing a novel concept, namely <i>hidden TAG model, </i>proposed a model analogous to a HMM for linking a resource-rich language to a resource-poor language. In (Faili and Basirat, 2010; Faili and Basirat, 2011) also a statistical approach based on HMM for link­ing the automatically extracted LTAG from Penn Treebank (Chen, 2001) and English XTAG gram­mar (XTAG-Group, 2001) was proposed.<page local="3" global="65"/> Here by introducing two statistical models, we have closely followed the approach presented in (Faili and Basirat, 2011).</p></section><section number="3" title="HMM-based LTAG mapping"><p>The task of mapping a MICA elementary tree se­quence onto an appropriate XTAG elementary tree sequence could be formulated as below:</p><p><i>Given a sequence </i><i>ofMICA</i><i> elementary trees T = </i>(t1,..., <i>tn</i>) <i>assigned to sen­tence S = </i>(w1,..., wn) <i>by MICA, tag each element of T with an elementary tree t' e XTAG Grammar such that the likelihood of T ' = (t\</i><i>t'n</i><i> </i>) <i>given T and S be maximized.</i></p><p>This problem directly addresses a Hidden Markov Model (HMM) that relates a MICA elementary tree sequence as an observation sequence to the most probable XTAG elementary tree sequence as a hidden state path. Given such a model, the <i>Viterbi </i>algorithm can be used for finding the most probable hidden state path that generates the ob­servation sequence. The rest of this part deals with the problem modeling using HMM.</p><subsection number="3.1" title="Problem Modeling Using HMM"><p>Regarding the existence gap between XTAG and MICA grammars (Chen, 2001), two possible map­ping models were proposed. The M-1 model sim­ply ignores this gap. It assumes every syntac­tic structure in the MICA grammar has at least one corresponding element in the XTAG grammar. In this case, each hidden state is exactly corre­sponded to a XTAG elementary tree. The MICA supertags also are considered as the observation symbols. Given any XTAG elementary tree t' and t';, the state transition matrix (A = [ai ,]) contains the probability of seeing <i>t'j </i>after <i>ti'</i><i> </i>in a sequence of</p><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">ji</doubt><p>XTAG elementary trees. For each MICA elemen­tary tree <i>t</i><i> </i><i>j </i>and XTAG elementary tree <i>ti'</i><i> </i>the obser­vation probability matrix <i>(B</i><i> </i>= <i>[b</i><i>i j]) </i>also contains the probability P(tj\t').</p><p>On the other hand, the alternate model, M-2, tries to model the relation between the grammars with respect to the existence gap between them. In this model it is assumed that there are some syn­tactic structures in the MICA grammar that are not supported by the XTAG grammar. The main dif­ference between M-1 and M-2 is in their hidden states. In addition to the hidden states used in M-1, a new symbolic state, namely <i>UNKNOWN, </i>is added to the M-2 hidden states set. This new state is the representative of all syntactic structures that are modeled by MICA grammar but not by XTAG grammar.</p></subsection><subsection number="3.2" title="Training"><p>Both of the M-1 and M-2 models were trained by the Baum-Welch algorithm. As the other HMM training algorithm, Baum-Welch algorithm also cannot find the global optimum of the search space. This weakness is inherited from the HMM in which does not provide any clear solution to use any extra information of the problem. In this case, the initial state of the training algorithm pro­vides a way to use a part of environment's knowl­edge that can largely cover the mentioned weak­ness (Rabiner, 1989).</p><p>To lead the training algorithm to a better solu­tion two methods was peoposed for estimating the initial state of the models. Next part, introduces these algorithms.</p></subsection><subsection number="3.3" title="Initialization"><p>The initial state of the models has been trained using two novel semi-supervised EM-based train­ing algorithms. The algorithms work based on the available set of MICA and XTAG elementary tree sequences achieved from parsing a set of English sentences namely <i>Initialization Data Base (IDB).</i></p><p>In the M-1 model, IDB must be selected so that all of its sentences can be modeled in both of XTAG and MICA grammars. This constraint is due to the M-1 assumption about the problem.</p><p>In M-2 the only constraint over the IDB sen­tences is that the sentences must be modeled in the MICA's grammar. In this case, IDB can be parti­tioned into two parts. The sentences that can be modeled by XTAG grammar, <i>Parsable Initializa­tion dataset (PI), </i>and the sentences that cannot be modeled by the XTAG grammar, <i>NotParsable Ini­tialization dataset (NPI). </i>The partitioning enables the model to consider the existence gap between the grammars.</p><page local="4" global="66"/><subsubsection number="3.3.1" title="Initializing M-1"><p>Let C and C' be two sets of elementary tree se­quences achieved from parsing IDB using MICA and XTAG parsers, respectively. Due to the sta­tistical nature of MICA parser, for any sentence <i>Si</i><i> e </i>IDB, C contains a set of scored elementary tree sequences. Nevertheless, C' contains an am­biguity set of elementary tree sequences without any clear way to disambiguate it.</p><p>Given C and C', the simplest and most intuitive way for estimating the initial values of the HMM is MLE. Nevertheless, performing this application is subject to disambiguating the output of the XTAG parser stored in C'. This problem addresses a func­tion that assigns a real value to each member ofC' as shown in eq. 1.</p><doubt alpha="30.8" length="13" tooSmall="False" monospace="0.0">cd:C' — R (1)</doubt><p>Given such a weighting function co, the proba­bility of transition <i>(Si</i><i> - </i><i>S</i><i> </i><i>j)</i><i> </i>in hidden states can be estimated by taking weighted count from all bi-grams <i>(S</i><i> </i><i>i</i><i> </i><i>S</i><i> </i><i>j)</i><i> </i>in C' and normalizing by the sum of all bigrams (Si, <i>Sk</i>) that share the same first el­ements. A similar method also can be used for computing the probabilities presented in the ob­servation matrix (B) and n.</p><p>Given C" = cd(C') and C, we define function A for generating the HMM A using the aforemen­tioned MLE (eq. 2).</p><doubt alpha="33.3" length="15" tooSmall="False" monospace="0.0">A: C" xC - A(2)</doubt><p>The main problem here is to find an appropri­ate function co. Function co was estimated using a semi-supervised EM-based method. The algo­rithm takes the C and C' as input and attempts to estimate some values for function co such that the objective function 3 presented in eq. 3 is being maximized. Function 3 shows the likelihood of observing C given the HMM A achieved by A.</p><doubt alpha="24.0" length="25" tooSmall="False" monospace="0.0">3 = P(C\A = A(C", C)) (3)</doubt><p>In the EM formulation, the E-step was defined as the computing the value of A using A. In M-step the algorithm attempts to update the co regard­ing the earlier model resulted from E-step. Eq. 4 shows how to estimate the value of co for a XTAG elementary tree sequences <i>T' </i>e <i>C'. </i>In this equa­tion, £ shows the set of XTAG elementary tree se­quences in C' that are generated from the sentence <i>S,</i><i> </i>the generator of <i>T'.</i><i> Ti </i>e <i>C </i>also represents the <i>ith </i>MICA elementary tree sequence in £. The in­dex <i>n </i>shows the total number of sequences in C generated from S (\ £ \).</p><doubt alpha="36.6" length="41" tooSmall="False" monospace="0.0">,     Zn=iP(Ti)P(Ti,T'\A)co(T') =i=1——(4)</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">\£\</doubt></subsubsection><subsubsection number="3.3.2" title="Initializing M-2"><p>In this part also for the sake of simplicity, <i>C</i><i>MP, </i><i>C</i><i>XP </i>and <i>C</i><i>MNP </i>are used as the supertagging result of <i>PI </i>in MICA grammar, <i>PI </i>in XTAG grammar, and <i>NPI </i>in MICA grammar, respec­tively. Unlike the M-1 that uses all of the MICA elementary tree sequences resulted from parsing a sentence in IDB, here only the most proba­ble MICA elementary tree sequence was used. So, related to each sentence in PI and NPI, we have a single MICA elementary tree sequence in <i>CMP</i><i> </i>and <i>CMNP</i><i> </i>respectively.</p><p>In this model, in addition to computing co, ap­plying MLE is subject to generating the set of elementary tree sequences for the sentences in dataset <i>NPI. </i>We name this set of elementary tree sequences <i>C</i><i>XNP. </i>Each sequence in <i>C</i><i>XNP </i>consists of XTAG elementary trees and have to contain at least one <i>UNKNOWN</i><i> </i>symbol regarding this fact that NPI contains the sentences that couldn't be modeled in XTAG grammars. Given the paired sets <i>(CMNP,</i><i> </i><i>CXNP)</i><i> </i>and <i>(CMP,</i><i> </i><i>CXP)</i><i> </i>and an appro­priate weighting function co as shown in eq. 5, the initial values of HMM can be estimated using the mentioned MLE method.</p><doubt alpha="64.7" length="17" tooSmall="False" monospace="0.0">co:CxnpUCxp- R(5)</doubt><p>The co was estimated using a <i>semi-supervised boot strapping EM-based </i>algorithm. Like the ini­tialization algorithm proposed in sec. 3.3.1, this algorithm also has an iterative nature that tries to estimate some values for cd (hence for HMM parameters) in a greedy manner. The objective function in this phase is to maximize the likeli­hood of observing MICA supertag sequences in <i>C</i><i>MNP </i>U <i>C</i><i>MP </i>(eq. 6). In the heart of the algo­rithm, the <i>C</i><i>XNP </i>is bootstrapped by applying a cus­tomized version of Viterbi algorithm on the <i>CMNP</i><i></i>using the earlier value of HMM.</p><doubt alpha="50.0" length="20" tooSmall="False" monospace="0.0">3= P(CmpUCmnp\A) (6)</doubt><p>The algorithm consists of four main stages as below:</p><p>1. <i>Pre Initializing: </i>Initializing the HMM parameters with out considering UNKNOWN hidden state.</p><page local="5" global="67"/><p>2. <i>Bootstrapping: </i>Bootstrapping <i>CXNP </i>by annotating <i>CMNP </i>with hidden states labels.</p><p>3. <i>Updating: </i>Estimating the new value of HMM using Maximum Likelihood Estimation (MLE) on the paired sequences <i>(CMNP, CXNP) </i>and (CMP, CXP)</p><p>4. <i>Termination: </i>Until the termination criterion is not sat­isfied go to step 2.</p><p>In the rest of this part we will express each phase in detail.</p><p><b><i>Pre Initializing: </i></b>In this step, it tries to esti­mate the HMM parameters from the related se­quences in <i>(C</i><i>MP, </i><i>C</i><i>XP) </i>using the MLE. Apply­ing the MLE over these sets gives some approx­imations about the probabilities presented in the HMM parameters except the probabilities related to UNKNOWN hidden state. The weighting func­tion used in this phase gives a uniform distribution of probability to each member of <i>CMP</i><i> </i>that are gen­erated from same sentence.</p><p>The probabilities related to UNKNOWN hidden states also could be estimated using some heuris­tics over the existence gap between the grammars. For instance, the amount of uncertainty involved in the HMM parameters resulted by the MLE is a criterion for estimating the probabilities related to the UNKNOWN.</p><p><b><i>Bootstrapping: </i></b>In this phase it tries to anno­tate each MICA supertag sequence in <i>C</i><i>MNP </i>with a set of hidden state paths given the earlier value of HMM. To do this, a modified version of Viterbi algorithm, namely <i>Forced Viterbi, </i>was used. The algorithm looks for the hidden state paths that have the highest consistency with the earlier HMM and pass through UNKNOWN hidden state.</p><p>Before applying Forced Viterbi over <i>C</i><i>MNP, </i>we need some assumptions about the source elemen­tary trees that are more likely to be corresponded to UNKNOWN. A simple solution for making such assumption is feasible via taking a differen­tial between <i>C</i><i>MNP </i>and <i>C</i><i>MP, </i>and looking for the n-grams in the former that are not presented in the latter. The result of this process is a set of n-grams of MICA elementary trees, namely <i>Gap-set, </i>that their related n-gram in the original sentence couldn't be modeled in the XTAG grammar. For any n-gram in Gap-Set that is observed in a MICA elementary tree sequence member of <i>C</i><i>MNP, </i>by considering all conditions that the UNKNOWN can be assigned to the elementary trees of the ob­served n-gram, the Forced Viterbi algorithm will generate 2n XTAG elementary tree sequences.</p><figure caption="Figure 1: The HMM Initialization algorithm used"></figure><doubt alpha="50.0" length="6" tooSmall="False" monospace="0.0">in M-2</doubt><p><b><i>Updating: </i></b>In this step, the HMM parame­ters will be updated regarding the paired sets</p><p><i>(CMNP, CXNP) </i>and <i>(CMP, CXP). </i>Having thesepaired sets and a scoring function cd, the HMM pa­rameters can be updated using the mentioned MLE method.</p><p>For each XTAG elementary tree sequence <i>T' </i>e <i>C</i><i>XP </i>U <i>C</i><i>XNP </i>and its related MICA elementary tree sequence <i>Ti e </i><i>CMP</i><i> U</i><i>CMNP,</i><i> </i>the scoring function cd can be defined as shown in eq. 7. £ in this equation refers to the set of XTAG elementary trees that are generated from the same sentence and <i>T' </i>e £.</p><doubt alpha="40.0" length="10" tooSmall="False" monospace="0.0">P(T',T\ A)</doubt><doubt alpha="33.3" length="9" tooSmall="False" monospace="0.0">cd(T') =-</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">\ £ \</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(7)</doubt><p>Fig. 1 gives an outline over the HMM initial­ization algorithm. Observing same values for the probability presented in eq. 6 or exceeding the predefined maximum number of iterations are two candidates to be used as termination criteria.</p></subsubsection></subsection></section><section number="4" title="Numerical Results"><subsection number="4.1" title="Experiments Description"><p>To evaluate the accuracy of the proposed mod­els, the models have been initialized and trained with three real world data sets including ATIS , IBM Manual and Wall Street Journal (WSJ) cor­pora. Some parts of these datasets were randomly selected and divided into three distinct sections as initialization dataset (IDB), training dataset (TRDB) and testing dataset (TSDB). Table 4.1<page local="6" global="68"/></p><p>Table 1: Statistical information about initializa­tion, training and testing datasets used in M-1 and shows some statistics about the datasets used in initialization, training and testing the models.</p><doubt alpha="33.3" length="3" tooSmall="False" monospace="0.0">M-2</doubt><doubt alpha="16.7" length="6" tooSmall="True" monospace="0.0">-5 5J-</doubt><doubt alpha="100.0" length="4" tooSmall="True" monospace="0.0">Iter</doubt><figure caption="Figure 2: The values of the objective function pre­sented in eq. 8 while initializing the M-1"></figure></subsection><subsection number="4.2" title="Initializing"><p>The results of applying each of the initializing methods M-1 and M-2 over the IDBs are presented in figure 2 and 3 respectively. These figures show the value of 0 presented in eq. 8. 'O' in this equa­tion refers to all MICA elementary tree sequences used in the algorithms. The observed progress in the likelihood of observing the MICA elementary tree sequences is an evidence on the successful of the algorithms.</p><doubt alpha="63.2" length="19" tooSmall="False" monospace="0.0">0ZTieOlogP(Ti\A)(8)</doubt><footnote label="0">=  i ö i (8)</footnote><p>As these show, while the values resulted from M-2 are strictly ascending in a logarithmic man­ner, increasing in the values resulted from M-1 has no specific, predictable manner. It is due to the ob­jective function shown in eq. 3 in which doesn't consider the score values of each MICA elemen­tary tree sequences in C. In fact, related to any sen­tence in each IDB, C contains many scored MICA elementary tree sequences used in initializing al­gorithm but in the value of the objective function.</p></subsection><subsection number="4.3" title="Models Evaluation"><p>The models were evaluated in two ways, <i>tagging accuracy </i>and <i>parsed sequences. </i>The first criterion originally introduced in (Faili and Basirat, 2011), enables us to evaluate the models as XTAG su­pertaggers. The latter one also, provide a way to evaluate them when combining with a LTAG parser. In <i>parsed sequences </i>the main focus is on the number of resulted XTAG sequences that their constituents elementary trees can be attached to each other regarding the standard operations de­fined in TAG formalism, <i>Substitution </i>and <i>Adjunc­tion.</i></p><p>Due to the lack of a gold annotated corpus, the tagging accuracy has been done manually. Table 4.3 shows the result of the tagging accuracy over the mentioned test sets (TSDBs). The base line here is the result of tagging accuracy reported in (Faili and Basirat, 2011). As it can be seen, M-2 gives the best accuracy in comparison to the M-1 and the base line.</p><p>The result of the alternate criterion, parsed sen­tences, is given in table 4.3. As it shows, here also the M-2 gives a better response in compare to the M-1. An important point that should be noted is that, not all of the sentences in the test sets are covered by the XTAG grammar. In fact, our ex­periments showed that of all sentences in each of the ATIS-TSDB, IBM-TSDB, and WSJ-TSDB, all but 6%, 13% and 24% of them could be parsed by XTAG parser respectively.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>No. Sentences</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><i>IDBm-1</i></p></td><td class="cell"><p><i>IDBm-2</i></p></td><td class="cell"><p>TRDB</p></td><td class="cell"><p>TSDB</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ATIS</p></td><td class="cell"><p>904</p></td><td class="cell"><p>991</p></td><td class="cell"><p>1280</p></td><td class="cell"><p>18</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>IBM</p></td><td class="cell"><p>3463</p></td><td class="cell"><p>4473</p></td><td class="cell"><p>9742</p></td><td class="cell"><p>102</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>WSJ</p></td><td class="cell"><p>11913</p></td><td class="cell"><p>16871</p></td><td class="cell"><p>21709</p></td><td class="cell"><p>197</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>No. words</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ATIS</p></td><td class="cell"><p>7726</p></td><td class="cell"><p>9734</p></td><td class="cell"><p>16917</p></td><td class="cell"><p>209</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>IBM</p></td><td class="cell"><p>32840</p></td><td class="cell"><p>46833</p></td><td class="cell"><p>154668</p></td><td class="cell"><p>1547</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>WSJ</p></td><td class="cell"><p>102355</p></td><td class="cell"><p>155879</p></td><td class="cell"><p>221337</p></td><td class="cell"><p>2029</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 2: The result of the tagging accuracy on the test sets" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>M-1</p></td><td class="cell"><p>M-2</p></td><td class="cell"><p>Base Line</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ATIS</p></td><td class="cell"><p>59.83%</p></td><td class="cell"><p>80.00%</p></td><td class="cell"><p>78.30%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>IBM</p></td><td class="cell"><p>79.55%</p></td><td class="cell"><p>88.30%</p></td><td class="cell"><p>88.70%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>WSJ</p></td><td class="cell"><p>87.75%</p></td><td class="cell"><p>91.50%</p></td><td class="cell"><p>88.96%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="7" global="69"/><p>Two Hidden Markov Models (HMM) were pro­posed to make a bridge between the linguistic view of the English XTAG grammar and the statistical nature of the LTAG used by MICA parser (Banga­lore et al., 2009). The models were trained by the standard HMM training algorithm, Baum-Welch. The initial state of the models also were estimated using two semi-supervised EM-based algorithms. The models can be used to combine the statistical approaches with the grammar engineering.</p></subsection><table caption="Table 3: Number ofthe parsed sentences5 Conclusion" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>M-1</p></td><td class="cell"><p>M-2</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ATIS</p></td><td class="cell"><p>5%</p></td><td class="cell"><p>33%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>IBM</p></td><td class="cell"><p>12.74%</p></td><td class="cell"><p>43.10%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>WSJ</p></td><td class="cell"><p>50.25%</p></td><td class="cell"><p>57%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><references><p>Srinivas Bangalore and Arvanid K. Joshi. 1999. Su­pertagging: An approach to almost parsing. <i>Com­putational Linguistics, </i>25(2):237-266.</p><p>S. Bangalore, P. Haffner, and G. Emami. 2005. Fac­toring global inference by enriching local represen­tations. Technical report, AT&amp;T Labs - Reserach.</p><p>Srinivas Bangalore, P. Boulllier, A. Nasr, O. Rambow, and B. Sagot. 2009. Mica: A probabilistic depen­dency parser based on tree insertion grammar. <i>North American Chapter of the Association for Computa­tional Linguistics (NAACL).</i></p><p>John Chen. 2001. <i>Toward Efficient Statistical Parsing Using Lexicalized Grammatical Information. </i>Ph.D. thesis, University of Delaware.</p><p>David Chiang and Owen Rambow. 2006. The hid­den tag model: Synchronous grammars for parsing resource-poor languages. Proceedings of the 8th In­ternational Workshop on Tree Adjoining Grammar and Related Formalisms, July.</p><p>H. Dang, K. Kipper, and Martha Palmer. 2000. Inte­grating compositional semantic into a verb lexicon.</p><p>In <i>In Proceedings </i><i>ofthe</i><i> Eighteenth International Conference on Computational Linguistic (COLING-2000).</i></p><p>Heshaam Faili and Ali Basirat. 2010. Augmenting the automated extracted tree adjoining grammars by se­mantic representation. In <i>6th IEEE International Conference on Natural Language Processing and Knowledge Engineering (IEEE NLP-KE'10), </i>Bei­jing.</p><p>Heshaam Faili and Ali Basirat. 2011. An unsupervised approach for linking automatically extracted and manually crafted ltags. In <i>12th International con­ference on Intelligent Text Processing and Compu­tational Linguistics (CICLing-2011), </i>Tokyo, Febru­ary.</p><p>Heshaam Faili. 2009. From partial toward full parsing. In <i>Recent Advances In Natural Language Process­ing (RANLP).</i></p><p>Arvanid K. Joshi and Yves Schabez. 1991. Tree ad­joining grammars and lexicalized grammars. Tech­nical Report MS-CIS 91-22, Department of Com­puter &amp; Information Science, University of Pennsyl­vania.</p><p>Arvanid K. Joshi. 1985. How much context-sensitivity is necessary for characterizing structural descrip­tions? <i>Natural Language Processing: Theoret­ical, Computational, and Psychological Perspec­tives, </i>pages 206-250. New York, NY: Cambridge University Press.</p><p>Laura Kallmeyer. 2010. <i>Parsing Beyond Context-Free Grammars, </i>volume 0 of <i>Cognitive Technolo­gies. </i>Springer.</p><p>K. Kipper, H. Dang, and Martha Palmer. 2000. Class-based construction of verb lexicon. In <i>In Proceed­ings of Seventh nation Conference on Artificial In­telligence (AAAI-2000).</i></p><p>Lawrence R. Rabiner. 1989. A tutorial on hidden markov models and selected applications in speech recognition. <i>Proceedings </i><i>ofthe</i><i> IEEE, </i>77(2):257-286.</p><p>Neville Ryant and Karin Kipper. 2004. Assigning xtag trees to verbnet. TAG+7: Seventh International Workshop on Tree Adjoining Grammar and Related Formalisms, May 20-22.</p><p>Anoop Sarkar. 2007. Combining supertagging and lexicalized tree-adjoining grammar parsing. In Srinivas Bangalore and Aravind Joshi, editors, Com­plexity of Lexical Descriptions and its Relevance to Natural Language Processing: A Supertagging Ap­proach.</p><p>Fia Xia and Martha Palmer. 2000. Evaluating the cov­erage of ltags on annotated corpora. the Workshop on Using Evaluation within HLT Programs: Results and Trends, May 30.</p><p>Fia Xia. 2001. <i>Automatic grammar generation from two different perspectives. </i>Ph.D. thesis, University of Pennsylvania.</p><p>XTAG-Group. 2001. A lexicalized tree adjoining grammar for english. Technical Report IRCS 01­03, Institute for Research in Cognitive Science, Uni­versity of Pennsylvania.</p></references></body></article>