<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="284"/><title>Modeling Perspective Using Adaptor Grammars</title><pubinfo>Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing,pages 284—292, MIT, Massachusetts, USA, 9-11 October 2010. ©2010 Association for Computational Linguistics</pubinfo><author surname="Hardisty" givenname="Eric"><org  name="University of Maryland" country="USA" city="College Park"/></author><author surname="Boyd-Graber" givenname="Jordan"><org  name="University of Maryland" country="USA" city="College Park"/></author><author surname="Resnik" givenname="Philip"><org  name="University of Maryland" country="USA" city="College Park"/></author></firstpageheader><frontmatter><p><b>Modeling Perspective using Adaptor Grammars</b></p><p><b>Eric A. Hardisty</b></p><p>Department of Computer Science and UMIACS University of Maryland College Park, MD</p><p>hardisty@cs.umd.edu</p><p><b>Jordan Boyd-Graber</b></p><p>UMD iSchool and UMIACS</p><p>University of Maryland</p><p>College Park, MD</p><p>jbg@umiacs.umd.edu</p><p><b>Philip Resnik</b></p><p>Department of Linguistics</p><p>and UMIACS</p><p>resnik@umd.edu</p></frontmatter><abstract>Strong indications of perspective can often come from collocations of arbitrary length; for example, someone writing <i>get the government out of my X </i>is typically expressing a conserva­tive rather than progressive viewpoint. How­ever, going beyond unigram or bigram features in perspective classification gives rise to prob­lems of data sparsity. We address this prob­lem using nonparametric Bayesian modeling, specifically adaptor grammars (Johnson et al., 2006). We demonstrate that an <i>adaptive naïve Bayes </i>model captures multiword lexical usages associated with perspective, and establishes a new state-of-the-art for perspective classifica­tion results using the Bitter Lemons corpus, a collection of essays about mid-east issues from Israeli and Palestinian points of view. </abstract></header><body><section number="1" title="Introduction"><p>Most work on the computational analysis of senti­ment and perspective relies on lexical features. This makes sense, since an author's choice of words is often used to express overt opinions (e.g. describing healthcare reform as <i>idiotic </i>or <i>wonderful) </i>or to frame a discussion in order to convey a perspective more implicitly (e.g. using the term <i>death tax </i>instead of <i>estate tax). </i>Moreover, it is easy and efficient to rep­resent texts as collections of the words they contain, in order to apply a well known arsenal of supervised techniques (Laver et al., 2003; Mullen and Malouf, 2006; Yu et al., 2008).</p><p>At the same time, standard lexical features have their limitations for this kind of analysis. Such fea­tures are usually created by selecting some small n-gram size in advance. Indeed, it is not uncommon to see the feature space for sentiment analysis limited to unigrams. However, important indicators of per­spective can also be longer (get <i>the government out of my). </i>Trying to capture these using standard ma­chine learning approaches creates a problem, since allowing n-grams as features for larger n gives rise to problems of data sparsity.</p><p>In this paper, we employ nonparametric Bayesian models (Orbanz and Teh, 2010) in order to address this limitation. In contrast to parametric models, for which a fixed number of parameters are specified in advance, nonparametric models can "grow" to the size best suited to the observed data. In text analysis, models of this type have been employed primarily for unsupervised discovery of latent structure — for example, in topic modeling, when the true number of topics is not known (Teh et al., 2006); in grammatical inference, when the appropriate number of nontermi­nal symbols is not known (Liang et al., 2007); and in coreference resolution, when the number of enti­ties in a given document is not specified in advance (Haghighi and Klein, 2007). Here we use them for supervised text classification.</p><p>Specifically, we use <i>adaptor grammars </i>(Johnson et al., 2006), a formalism for nonparametric Bayesian modeling that has recently proven useful in unsuper-vised modeling of phonemes (Johnson, 2008), gram­mar induction (Cohen et al., 2010), and named entity structure learning (Johnson, 2010), to make super­vised naive Bayes classification nonparametric in order to improve perspective modeling. Intuitively, naive Bayes associates each class or label with a probability distribution over a fixed vocabulary. We introduce <i>adaptive naïve Bayes </i>(ANB), for which in principle the vocabulary can grow as needed to in­clude collocations of arbitrary length, as determined by the properties of the dataset.<page local="2" global="285"/> We show that using adaptive naive Bayes improves on state of the art classification using the Bitter Lemons corpus (Lin et al., 2006), a document collection that has been used by a variety of authors to evaluate perspective classification.</p><p>In Section 2, we review adaptor grammars, show how naive Bayes can be expressed within the for­malism, and describe how — and how easily — an adaptive naïve Bayes model can be created. Section 3 validates the approach via experimentation on the Bit­ter Lemons corpus. In Section 4, we summarize the contributions of the paper and discuss directions for future work.</p></section><section number="2" title="Adapting Naive Bayes to be Less Naive"><p>In this work we apply the <i>adaptor grammar </i>formal­ism introduced by Johnson, Griffiths, and Goldwa-ter (Johnson et al., 2006). Adaptor grammars are a generalization of probabilistic context free grammars (PCFGs) that make it particularly easy to express non-parametric Bayesian models of language simply and readably using context free rules. Moreover, John­son <i>et al. </i>provide an inference procedure based on Markov Chain Monte Carlo techniques that makes parameter estimation straightforward for all models that can be expressed using adaptor grammars.<footnote anchor="1"/> Vari-ational inference for adaptor grammars has also been recently introduced (Cohen et al., 2010).</p><p>Briefly, adaptor grammars allow nonterminals to be rewritten to entire subtrees. In contrast, a non­terminal in a PCFG rewrites only to a collection of grammar symbols; their subsequent productions are independent of each other. For instance, a tradi­tional PCFG might learn probabilities for the rewrite rule PP — P NP. In contrast, an adaptor gram­mar can learn (or "cache") the production PP — (P up)(NP(DET a)(N tree)). It does this by posit­ing that the distribution over children for an adapted non-terminal comes from a Pitman-Yor distribution.</p><p>A Pitman-Yor distribution (Pitman and Yor, 1997) is a distribution over distributions. It has three pa­rameters: the discount, a, such that 0 &lt; <i>a &lt; </i>1, the strength, b, a real number such that <i>—a&lt;b,</i> and a probability distribution Go known as the base distribution. Adaptor grammars allow distributions over subtrees to come from a Pitman-Yor distribu­tion with the PCFG's original distribution over trees as the base distribution. The generative process for obtaining draws from a distribution drawn from a Pitman-Yor distribution can be described by the "Chi­nese restaurant process" (CRP). We will use the CRP to describe how to obtain a distribution over obser­vations composed of sequences of <i>n</i>-grams, the key to our model's ability to capture perspective-bearing <i>n</i>-grams.</p><footnote label="1">And, better still, they provide code that implements the inference algorithm; see http://www.cog.brown.edu/ mj/Software.htm.</footnote><p>Suppose that we have a base distribution Q that is some distribution over all sequences of words (the exact structure of such a distribution is unimportant; such a distribution will be defined later in Table 1). Suppose further we have a distribution 0 drawn from <i>PY</i><i>(a,</i><i> b, </i>Q), and we wish to draw a series of obser­vations w from 0. The CRP gives us a generative process for doing those draws from 0, marginaliz­ing out 0. Following the restaurant metaphor, we imagine the ith customer in the series entering the restaurant to take a seat at a table. The customer sits by making a choice that determines the value of the n-gram wi for that customer: she can either sit at an existing table or start a new table of her own.<footnote anchor="2"/></p><p>If she sits at a new table j, that table is assigned a draw yj from the base distribution, Q; note that, since Q is a distribution over n-grams, y is an n-gram. The value of wi is therefore assigned to be y, and <i>y</i><i>j </i>becomes the sequence of words assigned to that new table. On the other hand, if she sits at an existing table, then <i>w</i><i>i </i>simply takes the sequence of words already associated with that table (assigned as above when it was first occupied).</p><p>The probability of joining an existing table j, with Cj patrons already seated at table j, is <i>cf--f</i>, where c. is the number of patrons seated at all tables: c. = <i>j' </i><i>C</i><i>j'</i>. The probability of starting a new table is ir+f, where t is the number of tables presently occupied.</p><p>Notice that 0 is a distribution over the same space as Q, but it can drastically shift the mass of the dis­tribution, compared with Q, as more and more patrons are seated at tables.<page local="3" global="286"/> However, there is always a chance of drawing from the base distribution, and therefore every word sequence can also always be drawn from 0.</p><footnote label="2">Note that we are abusing notation by allowing w i  to cor­respond to a word sequence of length &gt; 1 rather than a single word.</footnote><p>In the next section we will write a naive Bayes-like generative process using PCFGs. We will then use the PCFG distribution as the base distribution for a Pitman-Yor distribution, adapting the naïive Bayes process to give us a distribution over n-grams, thus learning new language substructures that are useful for modeling the differences in perspective.</p><subsection number="2.1" title="Classification Models as PCFGs"><p>Naïve Bayes is a venerable and popular mechanism for text classification (Lewis, 1998). It posits that there are K distinct categories of text — each with a distinct distribution over words — and that every doc­ument, represented as an exchangeable bag of words, is drawn from one (and only one) of these distribu­tions. Learning the per-category word distributions and global prevalence of the classes is a problem of posterior inference which can be approached using a variety of inference techniques (Lowd and Domingos, 2005).</p><p>More formally, naïive Bayes models can be ex­pressed via the following generative process:<footnote anchor="3"/></p></subsection></section><section number="1." title="Draw a global distribution over classes 6 ~ Dir (a)"><doubt alpha="61.2" length="67" tooSmall="False" monospace="0.0">2. For each class iG{1,..., K}, draw a word distribution 0i~Dir (A)</doubt><doubt alpha="56.2" length="32" tooSmall="False" monospace="0.0">3. For each documentdG{1,...,M}:</doubt><p>(a) Draw a class assignment zd ~ Mult (6)</p><doubt alpha="64.9" length="57" tooSmall="False" monospace="0.0">(b) For each word position nG{1,...,Nd,drawwd,n~Mult(0Zd)</doubt><p>A variant of the naïve Bayes generative process can be expressed using the adaptor grammar formalism (Table 1). The left hand side of each rule represents a nonterminal which can be expanded, and the right hand side represents the rewrite rule. The rightmost indices show replication; for instance, there are | V | rules that allow WO R D<i>i </i>to rewrite to each word in the vocabulary. One can assume a symmetric Dirichlet prior of Dir (1) over the production choices unless otherwise specified — as with the DOCd production rule above, where a sparse prior is used.</p><footnote label="3">Here a and A are hyperparameters used to specify priors for the class distribution and classes' word distributions, respec­tively; a is a symmetric K -dimensional vector where each ele­ment is n. N d  is the length of document d. Resnik and Hardisty (2010) provide a tutorial introduction to the naive Bayes genera­tive process and underlying concepts.</footnote><p>Notice that the distribution over expansions for WORD<i>i </i>corresponds directly to 0<i>i </i>in Figure 1(a). There are, however, some differences between the model that we have described above and the standard naïive Bayes model depicted in Figure 1(a). In par­ticular, there is no longer a single choice of class per document; each <i>sentence </i>is assigned a class. If the distribution over per-sentence labels is sparse (as it is above for DOCd), this will closely approximate naïive Bayes, since it will be very unlikely for the sentences in a document to have different labels. A non-sparse prior leads to behavior more like models that allow parts of a document to express sentiment or perspective differently.</p><subsection number="2.2" title="Moving Beyond the Bag of Words"><p>The naïive Bayes generative distribution posits that when writing a document, the author selects a distri­bution of categories <i>zd </i>for the document from 6. The author then generates words one at a time: each word is selected independently from a flat multinomial distribution 0<i>Zd </i>over the vocabulary.</p><p>However, this is a very limited picture of how text is related to underlying perspectives. Clearly words are often connected with each other as collocations, and, just as clearly, extending a flat vocabulary to include bigram collocations does not suffice, since sometimes relevant perspective-bearing phrases are longer than two words. Consider phrases like <i>health care for all </i>or <i>government takeover of health care, </i>connected with progressive and conservative posi­tions, respectively, during the national debate on healthcare reform. Simply applying naïive Bayes, or any other model, to a bag of n-grams for high n is<page local="4" global="287"/></p><table caption="Table 1: A naïve Bayes-inspired model expressed as a PCFG." class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Sent</p></td><td class="cell"><p>i—&gt;</p></td><td class="cell"><p>DOCd</p></td><td class="cell"><p>d = 1,... , m</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>DoCd<footnote anchor="0"/>-<footnote anchor="001"/></p></td><td class="cell"><p>i—&gt;</p></td><td class="cell"><p>IDd WORDSi</p></td><td class="cell"><p><i>d </i>= 1,... , m;</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p><i>i</i><i> </i><i>e </i><i>{1,K}</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>WORDSi</p></td><td class="cell"><p>—</p></td><td class="cell"><p>WORDSi WORDi</p></td><td class="cell"><p><i>i </i><i>e </i>{1,K}</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>WORDSi</p></td><td class="cell"><p>—</p></td><td class="cell"><p>WORDi</p></td><td class="cell"><p>i e {1,K}</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>WORDi</p></td><td class="cell"><p>—</p></td><td class="cell"><p>v</p></td><td class="cell"><p><i>v </i><i>e </i><i>V </i>; i e {1,K}</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>(a) Naive Bayes (b) Adaptive Naive Bayes</p><p>Figure 1: A plate diagram for naïve Bayes and adaptive naïve Bayes. Nodes represent random variables and parameters; shaded nodes represent observations; lines represent probabilistic dependencies; and the rectangular plates denote replication.</p><p>going to lead to unworkable levels of data sparsity; a model should be flexible enough to support both unigrams and longer phrases as needed.</p><p>Following Johnson (2010), however, we can use adaptor grammars to extend naïve Bayes <i>flexibly </i>to include richer structure like collocations when they improve the model, and not including them when they do not. This can be accomplished by introduc­ing <i>adapted </i>nonterminal rules: in a revised genera­tive process, the author can draw from Pitman-Yor distribution whose base distribution is over word <i>se­quences </i>of arbitrary length.<footnote anchor="4"/> Thus in a setting where, say, K = 2, and our two classes are progressive and conservative, the sequence <i>health care for all </i>might be generated as a single unit for the progressive perspective, but in the conservative perspective the same sequence might be generated as three separate draws: <i>health care, for, all. </i>Such a model is pre­sented in Figure 1(b). Note the following differences between Figures 1(a) and 1(b):</p><p>• zd selects which Pitman-Yor distribution to draw from for document d.</p><p>• 0j is the distribution over n-grams that comes from the Pitman-Yor distribution.</p><p>• Wd n represents an n-gram draw from 0j</p><p>• a, b are the Pitman-Yor strength and discount parameters.</p><p>• Q is the Pitman-Yor base distribution with t as its uniform hyperparameter.</p><footnote label="4">As defined above, the base distribution is that of the PCFG production rule W ords i . Although it has non-zero probability of producing any sequence of words, it is biased toward shorter word sequences.</footnote><p>Returning to the CRP metaphor discussed when we introduced the Pitman-Yor distribution, there are two restaurants, one for the PROGRESSIVE distribution and one for the conservative distribution. <i>Health care for all </i>has its own table in the progressive restaurant, and enough people are sitting at it to make it popular. There is no such table in the CONSERVA­TIvE restaurant, so in order to generate those words, the phrase <i>health care for all </i>would need to come from a new table; however, it is more easily explained by three customers sitting at three existing, popular tables: <i>health care, for, </i>and all.</p><p>We follow the convention of Johnson (2010) by writing adapted nonterminals as underlined. The grammar for adaptive naïve Bayes is shown in Ta­ble 2. The adapted <u>Cqllqq</u> rule means that every time we need to generate that nonterminal, we are actually drawing from a distribution drawn from a Pitman-Yor distribution. The distribution over the possible yields of the WORDS^ rule serves as the base distribution.</p><p>Given this generative process for documents, we can now use statistical inference to uncover the pos­terior distribution over the latent variables, thus dis­covering the tables and seating assignments of our metaphorical restaurants that each cater to a specific perspective filled with tables populated by words and n-grams.</p><p>The model presented in Table 2 is the most straight­forward way of extending naïve Bayes to collocations. For completeness, we also consider the alternative of using a shared base distribution rather than dis­tinguishing the base distributions of the two classes.</p><page local="5" global="288"/><p>from a Pitman-Yor distribution rather than a Dirichlet over production rules.</p><p>Table 3: An adaptive naïve Bayes grammar with a com­mon base distribution for collocations. Note that, in con­trast to Table 2, there are no subscripts on WO RDS or WORD.</p><p>Briefly, using a shared base distribution posits that the two classes use similar word distributions, but generate collocations unique to each class, whereas using separate base distributions assumes that the distribution of words is unique to each class.</p></subsection></section><section number="3" title="Experiments"><subsection number="3.1" title="Corpus Description"><p>We conducted our classification experiments on the Bitter Lemons (BL) corpus, which is a collection of 297 essays averaging 700-800 words in length, on various Middle East issues, written from both the Israeli and Palestinian perspectives. The BL corpus was compiled by Lin <i>et al. </i>(2006) and is derived from a website that invites weekly discussions on a topic and publishes essays from two sets of authors each week.<footnote anchor="5"/> Two of the authors are guests, one from each perspective, and two essays are from the site's regular contributors, also one from each perspective, for a total of four essays on each topic per week. We chose this corpus to allow us to directly compare our results with Greene and Resnik's (2009) Observable Proxies for Underlying Semantics (OPUS) features and Lin <i>et al. </i>'s Latent Sentence Perspective Model (LSPM). The classification goal for this corpus is to label each document with the perspective of its author, either Israeli or Palestinian.</p><footnote label="5">http://www.bitterlemons.org</footnote><figure caption="Figure 2: An alternative adaptive naïve Bayes with a com­mon base distribution for both classes."></figure><doubt alpha="92.3" length="13" tooSmall="True" monospace="0.0">AG Classifier</doubt><figure caption="Figure 3: Corpus preparation and experimental setup."></figure><p>Consistent with prior work, we prepared the corpus by dividing it into two groups, one group containing all of the essays written by the regular site contrib­utors, which we call the Editor set, and one group comprised of all the essays written by the guest con­tributors, which we call the Guest set. Similar to the above mentioned prior work, we perform classifica­tion using one group as training data and the other as test data and perform two folds of classification. The overall experimental setup and corpus preparation process is presented in Figure 3.</p><table caption="Table 2: An adaptive naïve Bayes grammar. TheCOLLOCnonterminal's distribution over yields is drawn" class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Sent</p></td><td class="cell"><p>i—&gt;</p></td><td class="cell"><p>Docd</p></td><td class="cell"><p><i>d = </i>1,... , m</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Docd<footnote anchor="0"/>-<footnote anchor="001"/></p></td><td class="cell"><p>i—&gt;</p></td><td class="cell"><p>IDd Spani</p></td><td class="cell"><p><i>d </i>= 1,... , m; <i>i</i><i> </i>e <i>{1,K}</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Spani</p></td><td class="cell"><p>—</p></td><td class="cell"><p>Spani Colloci</p></td><td class="cell"><p><i>i </i><i>e</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Spani</p></td><td class="cell"><p>—</p></td><td class="cell"><p>Colloci</p></td><td class="cell"><p>i e</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Colloci</p></td><td class="cell"><p>—</p></td><td class="cell"><p>Wordsi</p></td><td class="cell"><p>i e</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Wordsi</p></td><td class="cell"><p>—</p></td><td class="cell"><p>Wordsi Wordi</p></td><td class="cell"><p>i e</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Wordsi</p></td><td class="cell"><p>—</p></td><td class="cell"><p>Wordi</p></td><td class="cell"><p>i e</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Wordi</p></td><td class="cell"><p>—</p></td><td class="cell"><p>v</p></td><td class="cell"><p><i>v </i>e V ; i e }</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Sent</p></td><td class="cell"><p>—</p></td><td class="cell"><p>Docd</p></td><td class="cell"><p>d = 1,... , m</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Docd<footnote anchor="0"/>-<footnote anchor="001"/></p></td><td class="cell"><p>—</p></td><td class="cell"><p>IDd Spani</p></td><td class="cell"><p>d = 1,... , m;</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>i e</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Spani</p></td><td class="cell"><p>—</p></td><td class="cell"><p>Spani Colloci</p></td><td class="cell"><p>i e</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Spani</p></td><td class="cell"><p>—</p></td><td class="cell"><p>Colloci</p></td><td class="cell"><p>i e</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Colloci</p></td><td class="cell"><p>—</p></td><td class="cell"><p>Words</p></td><td class="cell"><p>i e</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Words</p></td><td class="cell"><p>—</p></td><td class="cell"><p>Words Word</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Words</p></td><td class="cell"><p>—</p></td><td class="cell"><p>Word</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Word</p></td><td class="cell"><p>—</p></td><td class="cell"><p>v</p></td><td class="cell"><p>veV</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="6" global="289"/></subsection><subsection number="3.2" title="Experimental Setup"><p>The vocabulary generator determines the vocabulary used by a given experiment by converting the training set to lower case, stemming with the Porter stemmer, and filtering punctuation. We remove from the vocab­ulary any words that appeared in only one document regardless of frequency within that document, words with frequencies lower than a threshold, and stop words.<footnote anchor="6"/> The vocabulary is then passed to a grammar generator and a corpus filter.</p><p>The grammar generator uses the vocabulary to gen­erate the terminating rules of the grammar from the ANB grammar presented in Tables 2 and 3. The cor­pus filter takes in a set of documents and replaces all words not in the vocabulary with "out of vocabulary" markers. This process ensures that in all experiments the vocabulary is composed entirely of words from the training set. After the groups have been filtered, the group used as the test set has its labels removed. The test and training set are then sent, along with the grammar, into the adaptor grammar inference engine.</p><p>Each experiment ran for 3000 iterations. For the runs where adaptation was used we set the initial Pitman-Yor <i>a </i>and b parameters to 0.01 and 10 respec­tively, then slice sample (Johnson and Goldwater, 2009).</p><p>We use the resulting sentence parses for classifi­cation. By design of the grammar, each sentence's words will belong to one and only one distribution. We identify that distribution from each of the test set sentence parses and use it as the sentence level classification for that particular sentence. We then use majority rule on the individual sentence classifi­cations in a document to obtain the document classifi­cation. (In most cases the sentence-level assignments are overwhelmingly dominated by one class.)</p></subsection><subsection number="3.3" title="Results and Analysis"><p>Table 4 gives the results and compares to prior work. The support vector machine (SVM), NB-B and LSPM results are taken directly from Lin <i>et al. </i>(2006). NB-B indicates naïve Bayes with full Bayesian inference. LSPM is the Latent Sentence Perspective Model, also from Lin <i>et al. </i>(2006). OPUS results are taken from Greene</p><footnote label="6">In these experiments, a frequency threshold of 4 was se­lected prior to testing.</footnote><p>Table 4: Classification results. ANB* indicates the same grammar as Adapted Naïve Bayes, but with adaptation dis­abled. Com and Sep refer to whether the base distribution was common to both classes or separate.</p><p>and Resnik (2009). Briefly, OPUS features are gener­ated from observable grammatical relations that come from dependency parses of the corpus. Use of these features provided the best classification accuracy for this task prior to this work. ANB* refers to the gram­mar from Table 2, but with adaptation disabled. The reported accuracy values for ANB*, ANB with a common base distribution (see Table 3), and ANB with separate base distributions (see Table 2) are the mean values from five separate sampling chains. Bold face indicates statistical signficance (p &lt; 0.05) by unpaired t-test between the reported value and</p><p>ANB*.</p><p>Consistent with all prior work on this corpus we found that the classification accuracy for training on editors and testing on guests was lower than the other direction since the larger number of editors in the guest set allows for greater generalization. The dif­ference between ANB* and ANB with a common base distribution is not statistically significant. Also of note is that the classification accuracy improves for testing on Guests when the ANB grammar is al­lowed to adapt and a separate base distribution is used for the two classes (88.28% versus 84.98% without adaptation).</p><table caption="Table 5 presents some data on adapted rules"></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Training Set</p></td><td class="cell"><p>Test Set</p></td><td class="cell"><p>Classifier</p></td><td class="cell"><p>Accuracy</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Guests</p></td><td class="cell"><p>Editors</p></td><td class="cell"><p>SVM</p></td><td class="cell"><p>88.22</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Guests</p></td><td class="cell"><p>Editors</p></td><td class="cell"><p>NB-B</p></td><td class="cell"><p>93.46</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Guests</p></td><td class="cell"><p>Editors</p></td><td class="cell"><p>LSPM</p></td><td class="cell"><p>94.93</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Guests</p></td><td class="cell"><p>Editors</p></td><td class="cell"><p>OPUS</p></td><td class="cell"><p>97.64</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Guests</p></td><td class="cell"><p>Editors</p></td><td class="cell"><p>ANB*</p></td><td class="cell"><p>99.32</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Guests</p></td><td class="cell"><p>Editors</p></td><td class="cell"><p>ANB Com</p></td><td class="cell"><p><b>99.93</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Guests</p></td><td class="cell"><p>Editors</p></td><td class="cell"><p>ANB Sep</p></td><td class="cell"><p><b>99.87</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Editors</p></td><td class="cell"><p>Guests</p></td><td class="cell"><p>SVM</p></td><td class="cell"><p>81.48</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Editors</p></td><td class="cell"><p>Guests</p></td><td class="cell"><p>NB-B</p></td><td class="cell"><p>85.85</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Editors</p></td><td class="cell"><p>Guests</p></td><td class="cell"><p>LSPM</p></td><td class="cell"><p>86.99</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Editors</p></td><td class="cell"><p>Guests</p></td><td class="cell"><p>OPUS</p></td><td class="cell"><p>85.86</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Editors</p></td><td class="cell"><p>Guests</p></td><td class="cell"><p>ANB*</p></td><td class="cell"><p>84.98</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Editors</p></td><td class="cell"><p>Guests</p></td><td class="cell"><p>ANB Com</p></td><td class="cell"><p>82.76</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Editors</p></td><td class="cell"><p>Guests</p></td><td class="cell"><p>ANB Sep</p></td><td class="cell"><p><b>88.28</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="7" global="290"/><p>the same number of times but appears only 18 times in the corpus. In other words, the sequence <i>people go </i>is more easily explained by two sequential unigrams, not a bigram. The ratio of cache use counts to raw bigrams gives a measure of strength of collocation between the terms of the n-gram. We conjecture that the rareness of caching for n &gt; 2 is a function of the small corpus size. Also of note is the improvement in performance of ANB* over NB-B when training on guests, which we suspect is due to our use of sampled versus fixed hyperparameters.</p><p>learned once inference is complete. The column labeled <i>unique unigrams cached </i>indicates the num­ber of unique unigrams that appear on the right hand side of the adapted rules. Similarly, <i>unique n-grams cached </i>indicates the number of unique n-grams that appear on the right hand side of the adapted rules. The rightmost column indicates the percentage of terms from the group vocabulary that appear on the right hand side of adapted rules as unigrams. Values less than 100% indicate that the remaining vocabu­lary terms are cached in n-grams. As the table shows, a significant number of the rules learned during infer­ence are n-grams of various sizes.</p><p>Inspection of the captured bigrams showed that it captured sequences that a human might associate with one perspective over the other. Table 6 lists just a few of the more charged bigrams that were captured in the adapted rules.</p><p>More specific caching information on the individ­ual groups and classes is provided in Table 7. This data clearly demonstrates that raw n-gram frequency alone is not indicative of how many times an n-gram is used as a cached rule. For example, consider the bigram <i>people go, </i>which is used as a cached bigram only three times, yet appears in the corpus 407 times. Compare that with <i>isra palestinian, </i>which is cached</p></subsection></section><section number="4" title="Conclusions"><p>In this paper, we have applied adaptor grammars in a supervised setting to model lexical properties of text and improve document classification according to perspective, by allowing nonparametric discovery of collocations that aid in perspective classification. The adaptive naïve Bayes model improves on state of the art supervised classification performance in head-to-head comparisons with previous approaches.</p><p>Although there have been many investigations on the efficacy of using multiword collocations in text classification (Bekkerman and Allan, 2004), usually such approaches depend on a preprocessing step such as computing <i>tf-idf </i>or other measures of frequency based on either word bigrams (Tan et al., 2002) or character n-grams (Raskutti et al., 2001). In con­trast, our approach combines phrase discovery with the probabilistic model of the text. This allows for the collocation selection and classification to be ex­pressed in a single model, which can then be extended later; it also is truly generative, as compared with fea­ture induction and selection algorithms that either under- or over-generate data.</p><p>There are a number of interesting directions in which to take this research. As Johnson <i>et al. </i>(2006) argue, and as we have confirmed here, the adaptor<page local="8" global="291"/></p><table caption="Table 5: Counts of cached unigrams andn-grams for the two classes compared to the vocabulary sizes." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Class</p></td><td class="cell"><p>Group</p></td><td class="cell"><p>Unique</p><p>Unigrams Cached</p></td><td class="cell"><p>Unique</p><p>n-grams Cached</p></td><td class="cell"><p>Percent of Group Vocabulary Cached</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Israeli</p></td><td class="cell"><p>Editors</p></td><td class="cell"><p>2,292</p></td><td class="cell"><p>19,614</p></td><td class="cell"><p>77.62</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Palestinian</p></td><td class="cell"><p>Editors</p></td><td class="cell"><p>2,180</p></td><td class="cell"><p>17,314</p></td><td class="cell"><p>86.54</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Israeli</p></td><td class="cell"><p>Guests</p></td><td class="cell"><p>2,262</p></td><td class="cell"><p>19,398</p></td><td class="cell"><p>79.91</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Palestinian</p></td><td class="cell"><p>Guests</p></td><td class="cell"><p>2,005</p></td><td class="cell"><p>16,946</p></td><td class="cell"><p>74.94</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 6: Charged bigrams captured by the framework." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Israeli</p></td><td class="cell"><p>Palestinian</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>zionist dream zionist state zionist movement american leadership american victory abandon violence freedom (of the) press palestinian violence</p></td><td class="cell"><p>american jew achieve freedom palestinian freedom support palestinian palestinian suffer palestinian territory palestinian statehood palestinian refugee</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Table 7: Most frequently used cached bigrams. The first colum in each section is the number of times that bigram was used as a cached rule. The second column indicates the raw count of that bigram in the Guests or Editors group.</p><p>grammar formalism makes it quite easy to work with latent variable models, in order to automatically dis­cover structures in the data that have predictive value. For example, it is easy to imagine a model where in addition to a word distribution for each class, there is also an additional shared "neutral" distribution: for each sentence, the words in that sentence can ei­ther come from the class-specific content distribution or the shared neutral distribution. This turns out to be the Latent Sentence Perspective Model of Lin <i>et al. </i>(2006), which is straightforward to encode using the adaptor grammar formalism simply by introduc­ing two new nonterminals to represent the neutral</p><p>Running this grammar did not produce improvements consistent with those reported by Lin <i>et al. </i>We plan to investigate this further, and a natural follow-on would be to experiment with adaptation for this variety of latent structure, to produce an adapted LSPM-like model analogous to adaptive naïve Bayes. Viewed in a larger context, computational classification of perspective is closely connected to social scientists' study of <i>framing, </i>which Entman (1993) characterizes as follows: "To frame is to select some aspects of a perceived reality and make them more salient in a communicating text, in such a way as to promote a particular problem definition, causal interpretation, moral evaluation, and/or treatment rec­ommendation for the item described." Here and in other work (e.g. (Laver et al., 2003; Mullen and Mal-ouf, 2006; Yu et al., 2008; Monroe et al., 2008)), it is clear that lexical evidence is one key to under­standing how language is used to frame discussion from one perspective or another; Resnik and Greene (2009) have shown that syntactic choices can pro­vide important evidence, as well. Another promising direction for this work is the application of adaptor grammar models as a way to capture both lexical and grammatical aspects of framing in a unified model.</p></section><section title="Acknowledgments"><p>This research was funded in part by the Army Re­search Laboratory through ARL Cooperative Agree­ment W911NF-09-2-0072 and by the Office of the Director of National Intelligence (ODNI), Intelli­gence Advanced Research Projects Activity (IARPA), through the Army Research Laboratory. All state­ments of fact, opinion or conclusions contained herein are those of the authors and should not be con­strued as representing the official views or policies of ARL, IARPA, the ODNI, or the U.<page local="9" global="292"/>S. Government. The authors thank Mark Johnson and the anonymous reviewers for their helpful comments and discussions. We are particularly grateful to Mark Johnson for mak­ing his adaptor grammar code available.</p><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Guest</p></td><td class="cell"><p>Editor</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Israeli</p></td><td class="cell"><p>Palestinian</p></td><td class="cell"><p>Israeli</p></td><td class="cell"><p>Palestinian</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>palestinian OOV</p></td><td class="cell"><p>11</p></td><td class="cell"><p>299</p></td><td class="cell"><p>palestinian isra</p></td><td class="cell"><p>6</p></td><td class="cell"><p>178</p></td><td class="cell"><p>palestinian OOV</p></td><td class="cell"><p>8</p></td><td class="cell"><p>254</p></td><td class="cell"><p>OOV israel</p></td><td class="cell"><p>7</p></td><td class="cell"><p>198</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>OOV palestinian</p></td><td class="cell"><p>7</p></td><td class="cell"><p>405</p></td><td class="cell"><p>OOV palestinian</p></td><td class="cell"><p>6</p></td><td class="cell"><p>405</p></td><td class="cell"><p>OOV palestinian</p></td><td class="cell"><p>7</p></td><td class="cell"><p>319</p></td><td class="cell"><p>OOV palestinian</p></td><td class="cell"><p>6</p></td><td class="cell"><p>319</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>isra OOV</p></td><td class="cell"><p>6</p></td><td class="cell"><p>178</p></td><td class="cell"><p>palestinian OOV</p></td><td class="cell"><p>5</p></td><td class="cell"><p>29</p></td><td class="cell"><p>OOV israel</p></td><td class="cell"><p>7</p></td><td class="cell"><p>123</p></td><td class="cell"><p>OOV work</p></td><td class="cell"><p>5</p></td><td class="cell"><p>254</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>israel OOV</p></td><td class="cell"><p>6</p></td><td class="cell"><p>94</p></td><td class="cell"><p>one OOV</p></td><td class="cell"><p>4</p></td><td class="cell"><p>25</p></td><td class="cell"><p>OOV us</p></td><td class="cell"><p>6</p></td><td class="cell"><p>115</p></td><td class="cell"><p>OOV agreement</p></td><td class="cell"><p>5</p></td><td class="cell"><p>75</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>sharon OOV</p></td><td class="cell"><p>4</p></td><td class="cell"><p>74</p></td><td class="cell"><p>side OOV</p></td><td class="cell"><p>3</p></td><td class="cell"><p>21</p></td><td class="cell"><p>OOV part</p></td><td class="cell"><p>5</p></td><td class="cell"><p>56</p></td><td class="cell"><p>palestinian reform</p></td><td class="cell"><p>4</p></td><td class="cell"><p>49</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>polit OOV</p></td><td class="cell"><p>4</p></td><td class="cell"><p>143</p></td><td class="cell"><p>polit OOV</p></td><td class="cell"><p>3</p></td><td class="cell"><p>299</p></td><td class="cell"><p>israel OOV</p></td><td class="cell"><p>5</p></td><td class="cell"><p>81</p></td><td class="cell"><p>palestinian OOV</p></td><td class="cell"><p>4</p></td><td class="cell"><p>81</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>OOV us</p></td><td class="cell"><p>4</p></td><td class="cell"><p>29</p></td><td class="cell"><p>peoplgo</p></td><td class="cell"><p>3</p></td><td class="cell"><p>407</p></td><td class="cell"><p>attempt OOV</p></td><td class="cell"><p>5</p></td><td class="cell"><p>91</p></td><td class="cell"><p>OOV isra</p></td><td class="cell"><p>4</p></td><td class="cell"><p>15</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>OOV state</p></td><td class="cell"><p>4</p></td><td class="cell"><p>37</p></td><td class="cell"><p>palestinian govern</p></td><td class="cell"><p>3</p></td><td class="cell"><p>94</p></td><td class="cell"><p>time OOV</p></td><td class="cell"><p>4</p></td><td class="cell"><p>63</p></td><td class="cell"><p>one OOV</p></td><td class="cell"><p>4</p></td><td class="cell"><p>27</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>israel palestinian</p></td><td class="cell"><p>4</p></td><td class="cell"><p>52</p></td><td class="cell"><p>palestinian accept</p></td><td class="cell"><p>3</p></td><td class="cell"><p>220</p></td><td class="cell"><p>remain OOV</p></td><td class="cell"><p>4</p></td><td class="cell"><p>85</p></td><td class="cell"><p>isra palestinian</p></td><td class="cell"><p>4</p></td><td class="cell"><p>17</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>even OOV</p></td><td class="cell"><p>4</p></td><td class="cell"><p>43</p></td><td class="cell"><p>OOV state</p></td><td class="cell"><p>3</p></td><td class="cell"><p>150</p></td><td class="cell"><p>OOV time</p></td><td class="cell"><p>4</p></td><td class="cell"><p>70</p></td><td class="cell"><p>isra OOV</p></td><td class="cell"><p>4</p></td><td class="cell"><p>63</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>arafat OOV</p></td><td class="cell"><p>4</p></td><td class="cell"><p>41</p></td><td class="cell"><p>OOV israel</p></td><td class="cell"><p>3</p></td><td class="cell"><p>18</p></td><td class="cell"><p>OOV area</p></td><td class="cell"><p>4</p></td><td class="cell"><p>49</p></td><td class="cell"><p>howev OOV</p></td><td class="cell"><p>4</p></td><td class="cell"><p>149</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>appear OOV</p></td><td class="cell"><p>4</p></td><td class="cell"><p>53</p></td><td class="cell"><p>OOV end</p></td><td class="cell"><p>3</p></td><td class="cell"><p>20</p></td><td class="cell"><p>OOV arafat</p></td><td class="cell"><p>4</p></td><td class="cell"><p>28</p></td><td class="cell"><p>want OOV</p></td><td class="cell"><p>3</p></td><td class="cell"><p>36</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>total OOV</p></td><td class="cell"><p>3</p></td><td class="cell"><p>150</p></td><td class="cell"><p>OOV act</p></td><td class="cell"><p>3</p></td><td class="cell"><p>105</p></td><td class="cell"><p>isra OOV</p></td><td class="cell"><p>4</p></td><td class="cell"><p>8</p></td><td class="cell"><p>us OOV</p></td><td class="cell"><p>3</p></td><td class="cell"><p>35</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>palestinian would</p></td><td class="cell"><p>3</p></td><td class="cell"><p>65</p></td><td class="cell"><p>isra palestinian</p></td><td class="cell"><p>3</p></td><td class="cell"><p>18</p></td><td class="cell"><p>would OOV</p></td><td class="cell"><p>3</p></td><td class="cell"><p>28</p></td><td class="cell"><p>recent OOV</p></td><td class="cell"><p>3</p></td><td class="cell"><p>220</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>palestinian isra</p></td><td class="cell"><p>3</p></td><td class="cell"><p>35</p></td><td class="cell"><p>israel OOV</p></td><td class="cell"><p>3</p></td><td class="cell"><p>198</p></td><td class="cell"><p>use OOV</p></td><td class="cell"><p>3</p></td><td class="cell"><p>198</p></td><td class="cell"><p>palestinian isra</p></td><td class="cell"><p>3</p></td><td class="cell"><p>115</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>distribution:</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Sent</p></td><td class="cell"><p>i—&gt;</p></td><td class="cell"><p>Docd</p></td><td class="cell"><p><i>d</i><i> </i><i>=</i><i> </i><i>1,</i>. .. , m</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Docd</p></td><td class="cell"><p>i—&gt;</p></td><td class="cell"><p>IDd Wordsi</p></td><td class="cell"><p><i>d = </i>1,... , m;</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p><i>i</i><i> </i><i>e </i><i>{1,K}</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Docd</p></td><td class="cell"><p>—</p></td><td class="cell"><p>IDd Neuts</p></td><td class="cell"><p><i>d = </i>1,... , m;</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Wordsi</p></td><td class="cell"><p>—</p></td><td class="cell"><p>Wordsi Wordi</p></td><td class="cell"><p>i e {1,K}</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Wordsi</p></td><td class="cell"><p>—</p></td><td class="cell"><p>Wordi</p></td><td class="cell"><p>i e {1,K}</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Wordi</p></td><td class="cell"><p>—</p></td><td class="cell"><p>v</p></td><td class="cell"><p><i>v </i><i>e </i><i>V</i>; i e {1,k}</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Neut</p></td><td class="cell"><p>—</p></td><td class="cell"><p>Neutsi Neuti</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Neut</p></td><td class="cell"><p>—</p></td><td class="cell"><p>Neut</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Neut</p></td><td class="cell"><p>—</p></td><td class="cell"><p>v</p></td><td class="cell"><p><i>v </i><i>e </i><i>V</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><references><p>R. Bekkerman and J. Allan. 2004. Using bigrams in text categorization. Technical Report IR-408, Center of Intelligent Information Retrieval, UMass Amherst.</p><p>S. B. Cohen, D. M. Blei, and N. A. Smith. 2010. Varia­tional inference for adaptor grammars. In <i>Conference of the North American Chapter of the Association for Computational Linguistics.</i></p><p>R.M. Entman. 1993. Framing: Toward Clarification of a Fractured Paradigm. <i>The Journal of Communication, </i>43(4):51-58.</p><p>Stephan Greene and Philip Resnik. 2009. More than words: Syntactic packaging and implicit sentiment. In <i>Conference of the North American Chapter of the Asso­ciation for Computational Linguistics, </i>pages 503-511.</p><p>Aria Haghighi and Dan Klein. 2007. Unsupervised coref-erence resolution in a nonparametric bayesian model. In <i>Proceedings of the 45th Annual Meeting of the Asso­ciation of Computational Linguistics, </i>pages 848-855, Prague, Czech Republic, June.</p><p>Mark Johnson and Sharon Goldwater. 2009. Improving nonparameteric bayesian inference: experiments on unsupervised word segmentation with adaptor gram­mars. In <i>Conference of the North American Chapter of the Association for Computational Linguistics, </i>pages 317-325, Boulder, Colorado, June.</p><p>Mark Johnson, Thomas L. Griffiths, and Sharon Gold-water. 2006. Adaptor grammars: A framework for specifying compositional nonparametric Bayesian mod­els. In <i>Proceedings of Advances in Neural Information Processing Systems.</i></p><p>Mark Johnson. 2008. Using adaptor grammars to identify synergies in the unsupervised acquisition of linguistic structure. In <i>Proceedings of ACL-08: HLT, </i>pages 398­406, Columbus, Ohio, June.</p><p>Mark Johnson. 2010. PCFGs, topic models, adaptor gram­mars and learning topical collocations and the structure of proper names. In <i>Proceedings of the Association for Computational Linguistics, </i>pages 1148-1157, Uppsala, Sweden, July.</p><p>Michael Laver, Kenneth Benoit, and John Garry. 2003. Extracting policy positions from political texts using words as data. <i>American Political Science Review, </i>pages 311-331.</p><p>David D. Lewis. 1998. Naive (bayes) at forty: The inde­pendence assumption in information retrieval. In Claire</p><p>Nedellec and Celine Rouveirol, editors, <i>Proceedings ofECML-98, 10th European Conference on Machine Learning, </i>number 1398, pages 4-15, Chemnitz, DE. Springer Verlag, Heidelberg, DE. Percy Liang, Slav Petrov, Michael Jordan, and Dan Klein.</p><p>2007. The infinite PCFG using hierarchical Dirichlet processes. In <i>Proceedings ofEmperical Methods in Natural Language Processing, </i>pages 688-697.</p><p>Wei-Hao Lin, Theresa Wilson, Janyce Wiebe, and Alexan­der Hauptmann. 2006. Which side are you on? Identi­fying perspectives at the document and sentence levels. In <i>Proceedings ofthe Conference on Natural Language Learning (CoNLL).</i></p><p>Daniel Lowd and Pedro Domingos. 2005. Naive bayes models for probability estimation. In <i>ICML '05: Pro­ceedings of the 22nd international conference on Ma­chine learning, </i>pages 529-536, New York, NY, USA. ACM.</p><p>Burt L. Monroe, Michael P. Colaresi, and Kevin M. Quinn.</p><p>2008. Fightin' Words: Lexical Feature Selection and Evaluation for Identifying the Content of Political Con­flict. <i>Political Analysis, Vol. 16, Issue 4, pp. 372-403, 2008.</i></p><p>Tony Mullen and Robert Malouf. 2006. A preliminary in­vestigation into sentiment analysis of informal political discourse. In <i>AAAAI Symposium on Computational Ap­proaches to Analysing Weblogs (AAAI-CAAW), </i>pages 159-162.</p><p>P. Orbanz and Y. W. Teh. 2010. Bayesian nonparamet-ric models. In <i>Encyclopedia ofMachine Learning. </i>Springer.</p><p>J. Pitman and M. Yor. 1997. The two-parameter Poisson-Dirichlet distribution derived from a stable subordinator.</p><p><i>Annals of Probability, </i>25(2):855-900.</p><p>Bhavani Raskutti, Herman L. Ferra, and Adam Kowal-czyk. 2001. Second order features for maximising text classification performance. In <i>EMCL '01: Proceedings ofthe 12th European Conference on Machine Learning, </i>pages 419-430, London, UK. Springer-Verlag.</p><p>Philip Resnik and Eric Hardisty. 2010. Gibbs sampling for the uninitiated. Technical Re­port UMIACS-TR-2010-04, University of Maryland. http://www.lib.umd.edu/drum/handle/1903/10058.</p><p>Chade-Meng Tan, Yuan-Fang Wang, and Chan-Do Lee. 2002. The use of bigrams to enhance text categoriza­tion. <i>Inf. Process. Manage., </i>38(4):529-546.</p><p>Yee Whye Teh, Michael I. Jordan, Matthew J. Beal, and David M. Blei. 2006. Hierarchical Dirichlet pro­cesses. <i>Journal ofthe American Statistical Association, </i>101(476):1566-1581.</p><p>B. Yu, S. Kaufmann, and D. Diermeier. 2008. Classify­ing party affiliation from political speech. <i>Journal of Information Technology and Politics, </i>5(1):33-48.</p></references></body></article>