<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="1546"/><title>Active Learning with Amazon Mechanical Turk</title><pubinfo>Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing,pages 1546-1556, Edinburgh, Scotland, UK, July 27-31, 2011. ©2011 Association for Computational Linguistics</pubinfo><author surname="Laws" givenname="Florian"><org  name="Institute for Natural Language Processing Universität Stuttgart"/></author><author surname="Scheible" givenname="Christian"><org  name="Institute for Natural Language Processing Universität Stuttgart"/></author><author surname="Schütze" givenname="Hinrich"><org  name="Institute for Natural Language Processing Universität Stuttgart"/></author></firstpageheader><frontmatter><p><b>Active Learning with Amazon Mechanical Turk</b></p><p><b>Florian Laws    Christian Scheible    Hinrich Schütze</b></p><p>Institute for Natural Language Processing</p><p>Universität Stuttgart {lawsfn, scheibcn}@ims.uni-stuttgart.de</p></frontmatter><abstract>Supervised classification needs large amounts of annotated training data that is expensive to create. Two approaches that reduce the cost of annotation are <i>active learning </i>and <i>crowd­sourcing. </i>However, these two approaches have not been combined successfully to date. We evaluate the utility of active learning in crowdsourcing on two tasks, named entity recognition and sentiment detection, and show that active learning outperforms random selec­tion of annotation examples in a noisy crowd­sourcing scenario. </abstract></header><body><section number="1" title="Introduction"><p>Supervised classification is the predominant tech­nique for a large number of natural language pro­cessing (NLP) tasks. The large amount of labeled training data that supervised classification relies on is time-consuming and expensive to create, espe­cially when experts perform the data annotation. Recently, crowdsourcing services like Amazon Me­chanical Turk (MTurk) have become available as an alternative that offers acquisition of non-expert an­notations at low cost. MTurk is a software service that outsources small annotation tasks - called <i>HITs </i>- to a large group of freelance workers. The cost of MTurk annotation is low, but a consequence of us­ing non-expert annotators is much lower annotation quality. This requires strategies for quality control of the annotations.</p><p>Another promising approach to the data acqui­sition bottleneck for supervised learning is active learning (AL). AL reduces annotation effort by set­ting up an annotation loop where, starting from a small seed set, only the maximally informative ex­amples are chosen for annotation. With these an­notated examples, the classifier is then retrained to again select more informative examples for further annotation. In general, AL needs a lot fewer anno­tations to achieve a desired performance level than random sampling.</p><p>AL has been successfully applied to a number of NLP tasks such as part-of-speech tagging (Ringger et al., 2007), parsing (Osborne and Baldridge, 2004), text classification (Tong and Koller, 2002), senti­ment detection (Brew et al, 2010), and named entity recognition (NER) (Tomanek et al., 2007). Until recently, most AL studies focused on simulating the annotation process by using already available gold standard data. In reality, however, human annota­tors make mistakes, leading to noise in the annota­tions. For this reason, some authors have questioned the applicability of AL to noisy annotation scenarios such as MTurk (Baldridge and Palmer, 2009; Re-hbeinetal.,2010).</p><p>AL and crowdsourcing are complementary ap­proaches: AL reduces the number of annotations used while crowdsourcing reduces the cost per an­notation. Combined, the two approaches could sub­stantially lower the cost of creating training sets.</p><p>Our main contribution in this paper is that we show for the first time that AL is significantly bet­ter than randomly selected annotation examples in a real crowdsourcing annotation scenario. Our experiments directly address two tasks, named en­tity recognition and sentiment detection, but our evidence suggests that AL is of general benefit in crowdsourcing.<page local="2" global="1547"/> We also show that the effectiveness of MTurk annotation with AL can be further en­hanced by using two techniques that increase label quality: <i>adaptive voting </i>and <i>fragment recovery.</i></p></section><section number="2" title="Related Work"><subsection number="2.1" title="Crowdsourcing"><p>Pioneered by Snow et al. (2008), Crowdsourcing, especially using MTurk, has become a widely used service in the NLP community. A number of stud­ies have looked at crowdsourcing for NER. Voyer et al. (2010) use a combination of expert and crowd-sourced annotations. Finin et al. (2010) annotate Twitter messages - short sequences of words - and this is reflected in their vertically oriented user in­terface. Lawson et al. (2010) choose an annotation interface where annotators have to drag the mouse to select entities. Carpenter and Poesio (2010) ar­gue that dragging is less convenient for workers than marking tokens.</p><p>These papers do not address AL in crowdsourc-ing. Another important difference is that previous studies on NER have used data sets for which no "linguistic" gold annotation is available. In con­trast, we reannotate the CoNLL-2003 English NER dataset. This allows us to conduct a detailed com­parison of MTurk AL to conventional expert anno­tation.</p></subsection><subsection number="2.2" title="Active Learning with Noisy Labels"><p>Hachey et al. (2005) were among the first to in­vestigate the effect of actively sampled instances on agreement of labels and annotation time. They demonstrate applicability of AL when annotators are trained experts. This is an important result. How­ever, AL depends on accurate assessments of uncer­tainty and informativeness and such an accurate as­sessment is made more dificult if labels are noisy as is the case in crowdsourcing. For this reason, the problem of AL performance with noisy labels has become a topic of interest in the AL community. Re­hbein et al. (2010) investigate AL with human expert annotators for word sense disambiguation, but do not ind convincing evidence that AL reduces anno­tation cost in a realistic (non-simulated) annotation scenario. Brew et al. (2010) carried out experiments on sentiment active learning through crowdsourcing. However, they use a small set of volunteer labelers instead ofanonymous paid workers.</p><p>Donmez and Carbonell (2008) propose a method to choose annotators from a set of noisy annotators. However, in a crowdsourcing scenario, it is not pos­sible to ask speciic annotators for a label, as crowd-sourcing workers join and leave the site. Further­more, they only evaluate their approach in simula­tions. We use the actual labels of human annotators to avoid the risk of unrealistic assumptions when modeling annotators.</p><p>We are not aware of any study that shows that AL is signiicantly better than a simple baseline of hav­ing annotators annotate randomly selected examples in a highly noisy annotation setting like crowdsourc-ing. While AL generally is superior to this base­line in simulated experiments, it is not clear that this result carries over to crowdsourcing annotation. Crowdsourcing differs in a number of ways from simulated experiments: the dificulty and annotation consistency of examples drawn by AL differs from that drawn by random sampling; crowdsourcing la­bels are noisy; and because of the noisiness of labels statistical classiiers behave differently in simulated and real annotation experiments.</p></subsection></section><section number="3" title="Annotation System"><p>One fundamental design criterion for our annotation system was the ability to select examples <i>in real time </i>to support, e.g., the interactive annotation experi­ments presented in this paper. Thus, we could not use the standard MTurk workflow or services like CrowdFlower.<footnote anchor="1"/></p><p>We therefore designed our own system for anno­tation experiments. It consists of a two-tiered ap­plication architecture. The frontend tier is a web application that serves two purposes. First, the ad­ministrator can manage annotation experiments us­ing a web interface and publish annotation tasks as­sociated with an experiment on MTurk. The front­end also provides tools for eficient review of the received answers. Second, the frontend web appli­cation presents annotation tasks to MTurk workers. Because we wanted to implement interactive anno­tation experiments, we used the "external question"</p><p>'http ://crowdflower.com/<page local="3" global="1548"/></p><p>feature of MTurk. An external question contains an URL to our frontend web application, which is queried when a worker views an annotation task. Our frontend then in turn queries our backend com­ponent for an example to be annotated and renders it in HTML.</p><p>The backend component is responsible for selec­tion of an example to be annotated in response to a worker's request for an annotation task. The back-end implements a diverse choice of random and ac­tive selection strategies as well as the multilabel-ing strategies described in section 3.2. The backend component runs as a standalone server and is queried by the frontend via REST-like HTTP calls.</p><p>For the NER task, we present one sentence per HIT, segmented into tokens, with a select box under­neath each token containing the classes. The defini­tion of the classes is based on the CoNLL-2003 an­notation guidelines (Tjong Kim Sang and De Meul-der, 2003). Examples were given for every class. Annotators are forced to make a selection for upper­case tokens. Lowercase tokens are prelabeled with "O" (no named entity), but annotators are encour­aged to change this label if the token is in fact part of an entity phrase.</p><p>For sentiment annotation, we found in prelim­inary experiments that using simple radio button selection for the choice of the document label (positive or negative) leads to a very high amount of spam submissions, taking the overall clas­sification accuracy down to around 55%. We then designed a template that forced annotators to type the label as well as a randomly chosen word from the text. Individual label accuracy was around 75% in this scheme.</p><subsection number="3.1" title="Concurrent example selection"><p>AL works by setting up an interactive annotation loop where at each iteration, the most informative example is selected for annotation. We use a pool-based AL setup where the most informative exam­ple is selected from a pool of unlabeled examples. Informativeness is calculated as uncertainty (Lewis and Gale, 1994) using the margin metric (Schein and Ungar, 2007). This metric chooses examples for which the margin of probabilities from the classifier between the two most probable classes is the small­est:</p><doubt alpha="45.8" length="24" tooSmall="False" monospace="0.0">Mn=\P{ci\xn) - P{c2\xn)\</doubt><p>Here, <i>xn </i>is the instance to be classified, <i>c\ </i>and C2 are the two most likely classes, and <i>P </i>the classifier's estimate of probability.</p><p>For NER, the margins of the tokens are averaged to get an uncertainty assessment of the sentence. For sentiment, whole documents are classified, thus un­certainties can be used directly.</p><p>After annotation, the selected example is removed from the unlabeled pool and, together with its la­bels), added to the set of labeled examples. The classifier is then retrained on the labeled examples and the informativeness of the remaining examples in the pool is re-evaluated.</p><p>Depending on the classifier and the sizes of pool and labeled set, retraining and réévaluation can take some time. To minimize wait times, traditional AL implementations select examples in batches of the <i>n </i>most informative examples. However, batch se­lection might not give the optimum selection (exam­ples in a batch are likely to be redundant, see Brinker (2003)) and wait times can still occur between one batch and the next.</p><p>When performing annotation with MTurk, wait times are unacceptable. Thus, we perform the re­training and uncertainty rescoring concurrently with the annotation user interface. The unlabeled pool is stored in a priority queue that is ordered according to the examples' informativeness. The annotation user interface takes the most informative example from the pool and presents it to the annotator. The la­beled example is then inserted into a second queue that feeds and updates retraining and rescoring pro­cesses. The pool queue then is resorted according to the new informativeness. In this way, annotation and example selection can run in parallel. This is similar toHaertel etal. (2010).</p></subsection><subsection number="3.2" title="Adaptive voting and fragment recovery"><p>MTurk labels often have a high error rate. A com­mon strategy for improving label quality is to ac­quire <i>multiple labels </i>by different workers for each example and then consolidate the annotations into a single label of higher quality. To trade off num­ber of annotated examples against quality of anno­tations, we adopt <i>adaptive voting. </i>It uses majority<page local="4" global="1549"/></p><p>Table 1: For NER, active learning consistently beats random sampling on MTurk. NER <i>F\ </i>evaluated on CoNLL test set A. #train = number of sentences in training set, S = single, 3-v = 3-voting, 5/4-voting = 5-and 4-voting for NER and sentiment resp., +f = using fragments; sentiment budget 1130 for run 1, sentiment budget 1756 averaged over 2 runs.</p><p>voting and is adaptive in the number of repeated an­notations. For NER, a sentence is irst annotated by two workers. Then majority voting is performed for each token individually. If there is a majority for ev­ery token that is greater than an agreement threshold a, the sentence is accepted with each token labeled with the majority label. Otherwise additional anno­tations are requested. A sentence is discarded if the number of repeated annotations exceeds a discard threshold d (d-voting).<footnote anchor="2"/> We use the same scheme for sentiment; note that there is just one decision per HIT in this case, not several as in NER.</p><p>For NER, we also use <i>fragment recovery: </i>we sal­vage tokens with agreeing labels from discarded sen­tences. We cut the token sequence of a discarded sentence into several fragments that have agreeing tokens and discard only those parts that disagree. We then include these recovered fragments in the train­ing data just like complete sentences.</p><p><b>Software release. </b>Our active learning framework used can be downloaded at http:// www. ims . uni-Stuttgart. de/~lawsfn/active/.</p></subsection></section><section number="4" title="Experiments, Results and Analysis 4.1 Experiments"><p>In our NER experiments, we have workers reannotate the English corpus of the CoNLL-2003 NER shared task. We chose this corpus to be able to com­pare crowdsourced annotations with gold standard annotations. A HIT is one sentence and is offered for a base payment of $0.01. We filtered out answers that contained unannotated tokens or were obvious spam (e.g., all tokens labeled as MISC). For test­ing NER performance, we used a system based on conditional random ields with standard named en­tity features including the token itself, orthographic features like the occurrence of capitalization or spe­cial characters and context information about the to­kens to the left/right of the current token.</p><footnote label="2">It can take a while in this scheme for annotators to agree on a final annotation for a sentence. We make tentative labels of a sentence available to the classifier immediately and replace them with the inal labels once voting is completed.</footnote><p>The sentiment detection task was modeled after a well-known document analysis setup for sentiment classiication, introduced by Pang et al. (2002). We use their corpus of 1000 positive and 1000 negative movie reviews and the Stanford maximum entropy classifier (Manning and Klein, 2003) to predict the sentiment label of each document <i>d </i>from a unigram representation of <i>d.</i><i> </i>We randomly split this corpus into a test set of 500 reviews and an active learn­ing pool of 1500 reviews. Each HIT consists of one document, valued at $0.01.</p><p>We compare random sampling (RS) and AL in combination with the proposed voting and fragment strategies with different parameters. We want to avoid rerunning experiments on MTurk over and over again, but on the other hand, we believe that us­ing synthetic data for simulations is problematic be­cause it is dificult to generate synthetic data with a realistic model of annotator errors. Thus, we logged a play-by-play record of the annotator interactions and labels. With this recording, we can then rerun strategies with different parameters.</p><p>We chose voting with at most d = 5 repetitions as our main reannotation strategy for both random and active sampling for NER annotation.<page local="5" global="1550"/> We use simple majority voting <i>(a</i><i> </i>= .5) for NER.</p><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>NER</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>Sentiment</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Budget</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>5820</p></td><td class="cell"><p></p></td><td class="cell"><p>6931</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>1130</p></td><td class="cell"><p></p></td><td class="cell"><p>1756</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>#train</p></td><td class="cell"><p><i>F</i><i>i</i></p></td><td class="cell"><p>cost/sent</p></td><td class="cell"><p>w.-accuracy</p></td><td class="cell"><p>#train</p></td><td class="cell"><p>F<i>i</i></p></td><td class="cell"><p>#train</p></td><td class="cell"><p>Acc</p></td><td class="cell"><p>cost/doc</p></td><td class="cell"><p>w.-accuracy</p></td><td class="cell"><p>#train Acc</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>RS 1</p></td><td class="cell"><p>S</p></td><td class="cell"><p>5820</p></td><td class="cell"><p>59.6</p></td><td class="cell"><p>1.00</p></td><td class="cell"><p>51.6</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>1130</p></td><td class="cell"><p>70.4</p></td><td class="cell"><p>1</p></td><td class="cell"><p>74.8</p></td><td class="cell"><p>- -</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>2</p></td><td class="cell"><p>3-v</p></td><td class="cell"><p>1624</p></td><td class="cell"><p>61.4+</p></td><td class="cell"><p>3.58</p></td><td class="cell"><p>70.1</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>- -</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>3</p></td><td class="cell"><p>5/4-v</p></td><td class="cell"><p>1488</p></td><td class="cell"><p>63.0+</p></td><td class="cell"><p>3.91</p></td><td class="cell"><p>71.6</p></td><td class="cell"><p>1774</p></td><td class="cell"><p>63.5</p></td><td class="cell"><p>450</p></td><td class="cell"><p>71.2</p></td><td class="cell"><p>2.51</p></td><td class="cell"><p>89.6</p></td><td class="cell"><p>735 79.2</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>4</p></td><td class="cell"><p>5-v+f</p></td><td class="cell"><p>1996</p></td><td class="cell"><p>63.6+</p></td><td class="cell"><p>2.91</p></td><td class="cell"><p>71.8</p></td><td class="cell"><p>2385</p></td><td class="cell"><p>64.9+</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>- -</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>AL 5</p></td><td class="cell"><p>S</p></td><td class="cell"><p>5820</p></td><td class="cell"><p>67.0</p></td><td class="cell"><p>1.00</p></td><td class="cell"><p>66.5</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>1130</p></td><td class="cell"><p>74.8</p></td><td class="cell"><p>1</p></td><td class="cell"><p>76.0</p></td><td class="cell"><p>- -</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>6</p></td><td class="cell"><p>3-v</p></td><td class="cell"><p>1808</p></td><td class="cell"><p>70.0+</p></td><td class="cell"><p>3.21</p></td><td class="cell"><p>78.8</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>- -</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>7</p></td><td class="cell"><p>5/4-v</p></td><td class="cell"><p>1679</p></td><td class="cell"><p>70.4+</p></td><td class="cell"><p>3.46</p></td><td class="cell"><p>79.6</p></td><td class="cell"><p>1966</p></td><td class="cell"><p>70.6</p></td><td class="cell"><p>455</p></td><td class="cell"><p>77.4</p></td><td class="cell"><p>2.48</p></td><td class="cell"><p>89.0</p></td><td class="cell"><p>715 81.8</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>8</p></td><td class="cell"><p>5-v+f</p></td><td class="cell"><p>2165</p></td><td class="cell"><p>70.5</p></td><td class="cell"><p>2.68</p></td><td class="cell"><p>79.3</p></td><td class="cell"><p>2691</p></td><td class="cell"><p>71.2</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>- -</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>For sentiment, we set <i>d </i>= 4 and minimum agree­ment <i>a = </i>.75 because the number of labels is smaller (2 vs. 5) and so random agreement is more likely for sentiment.</p><p>To get results for 3-voting NER, we take the recording and discard 5-voting votes not needed in 3-voting. This will result in roughly the same num­ber of annotated sentences, but at a lower cost. This simulation of 3-voting is not exactly what would have happened on MTurk (e.g., the final vote on a sentence might be different, which then influences AL example selection), but we will assume that dif­ferences are rare and simulated and actual results are similar. The same considerations apply to sin­gle votes and to the sentiment experiments.</p><p>We always compare two strategies for the same annotation budget. For example, the <i>number of training sentences </i>in Table 1 differ in the two rel­evant columns, but all strategies compared use ex­actly <i>the same annotation budget </i>(5820, 6931, 1130, and 1756, respectively).</p><p>For the single annotation strategy, each interac­tion record contained only about 40% usable anno­tations, the rest were repeats. A comparison with the single annotation strategy over approx. 2000 sen­tences or 450 documents would not have been mean­ingful; therefore we chose to run an extra experiment with the single annotation strategy to match this up with the budgets of the voting strategies. The re­sults are presented in two separate columns of Ta­ble 1 (budgets 6931 and 1756).</p><subsection number="4.2" title="Results"><p>For sentiment detection, <i>worker accuracy </i>or <i>label quality - </i>the percentage of correctly annotated doc­uments - is 74.8. In contrast, for NER, worker accu­racy - the percentage of non-0 tokens annotated cor­rectly - is only 51.6 (Table 1, line 1). This demon­strates the challenge of using MTurk for NLP an­notation tasks. When we use single annotations of each sentence, NER performance is 59.6 <i>F\ </i>for ran­dom sampling (line 1). When training with gold la­bels on the same sentences, the performance is 80.0 (not shown). This means we lose more than 20% due to poor worker accuracy. Adaptive voting and fragment recovery manage to recover a small part of the lost performance (lines 2-4); each of the three Fi scores is significantly better than the one above it as indicated by f (Approximate Randomization Test (Noreen, 1989; Chinchor et al, 1993) as im­plemented by Padö (2006)).</p><p>Using AL turns out to be quite successful for NER performance. For single annotations, NER perfor­mance is 67.0 (line 5), an improvement of 7.4% compared to random sampling. Adaptive voting and fragment recovery again increase worker accu­racy (lines 6-8) although total improvement of 3.5% (lines 8 vs. 5) is smaller than 4% for random (lines 4 vs. 1). The learning curves of AL vs. random in Figure 1 (top left) confirm this good result for AL. These learning curves are for tokens - not for sen­tences - to show that the reason for AL's better per­formance is not that it selects slightly longer sen­tences than random. In addition, the relative advan­tage of AL vs random decreases over time, which is typical of pool-based AL experiments.</p><p>We carried out two runs of the same experiment for sentiment to validate our first positive result since the difference between the two conditions is not as large as in NER (Figure 1, top right). After about 300 documents, active learning consistently outper­forms random sampling. The first AL run performs better because of higher label quality in the begin­ning. The overall advantage of AL over random is lower than for NER because the set of labels is smaller in sentiment, making the classification task easier. Second, there is a large amount of simple lex­ical clues for detecting sentiment (cf. Wilson et al. (2005)). It is likely that some of them can be learned well through random sampling at first; however, ac­tive learning can gain accuracy over time because it selects examples with more difficult clues.</p><p>In Figure 1 (bottom), we compare single annota­tion with adaptive voting. The graphs show <i>F\ </i>as a function of cost. Adaptive voting trades quantity of sampled sentences for quality of labels and thus incurs higher net costs per sentence. This results in a smaller dataset for a given budget, but this dataset is still more useful for classifier training. For NER (Figure 1, bottom left), the single annotation strat­egy has a faster start; so for small budgets, cover­ing a somewhat larger portion of the sample space is beneficial. For larger budgets, however, quality of<page local="6" global="1551"/></p><doubt alpha="100.0" length="2" tooSmall="True" monospace="0.0">CO</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">ci</doubt><doubt alpha="60.0" length="5" tooSmall="False" monospace="0.0">r~ ci</doubt><doubt alpha="33.3" length="3" tooSmall="False" monospace="0.0">(s3</doubt><p><b>ci ci ci</b></p><doubt alpha="100.0" length="4" tooSmall="False" monospace="0.0">coci</doubt><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">5000</doubt><doubt alpha="50.0" length="12" tooSmall="False" monospace="0.0">10000 Tokens</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">15000</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">20000</doubt><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">cra</doubt><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">c c</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">&lt;</doubt><doubt alpha="8.7" length="46" tooSmall="False" monospace="0.0">1000    2000    3000    4000    5000 6000 Cost</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">en</doubt><p>0.</p><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">co</doubt><p>0. 0.</p><p>.6 0.</p><p>.5 0.</p><p>.4 0.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">500</doubt><doubt alpha="44.4" length="9" tooSmall="False" monospace="0.0">1000 Cost</doubt><p>active 1 active 2 random 1 random 2</p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1500</doubt><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">2000</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">0</doubt><doubt alpha="62.5" length="8" tooSmall="False" monospace="0.0">o u co I</doubt><p>Figure 1: Top: Active learning vs. Random sampling for NER (left) and sentiment (right). Bottom: Active learning: adaptive voting vs. single annotation for NER (left) and sentiment (right).</p><p>the voted labels trumps quantity.</p><p>For sentiment (Figure 1, bottom right), results are similar: voting has no beneit initially, but as ind­ing maximally informative examples to annotate be­comes harder in later stages of learning, adaptive voting gains an advantage over single annotations.</p><p>The main result of the experiment is that active learning is better by about 7% Fi than random sam­pling for NER and by 2.6% accuracy for sentiment (averaged over two runs at budget 1756). Adaptive voting further improves AL performance for both NER and sentiment.</p></subsection><subsection number="4.3" title="Annotation time per token"><p>Most AL work assumes constant cost per annotation unit. This assumption has been questioned because AL often selects hard examples that take longer to annotate (Hachey et al., 2005; Settles et al., 2008). In annotation with MTurk, cost is not a function of annotation time because workers are paid a ixed amount per HIT. Nevertheless, annotation time plays a part in whether workers are willing to work on a given task for the offered reward. This is particularly problematic for NER since workers have to examine each token individually. We therefore investigate for NER whether the time MTurk workers spend on annotating sentences differs for random vs. AL.</p><p>We irst compute median and mean annotation times and number of tokens per sentence:</p><p>We see that most sentences are annotated in a very short time; but the mean is much larger than the me­dian because there are outliers of up to eight min­utes. AL tends to select slightly longer sentences as<page local="7" global="1552"/></p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>if"</i></p></td><td class="cell"><p>........ active, 5-voting</p><p>- random, 5-voting</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>1           1 1 0        200 400</p></td><td class="cell"><p>1</p><p>600</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Documents</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>- single ann.</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>...... 4-voting</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>$</i></p><p><b>.1'</b></p></td><td class="cell"><p>- single ann.</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>k</i></p></td><td class="cell"><p>........ 3-voting</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>■7</p></td><td class="cell"><p>---- 5-voting</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>i </b>)</p></td><td class="cell"><p>........ 5-voting +frags</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>sec/sentence</p></td><td class="cell"><p>tokens/sentence</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>strategy</p></td><td class="cell"><p>median mean</p></td><td class="cell"><p>all required</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>random</p></td><td class="cell"><p>17.2 33.1</p></td><td class="cell"><p>15.0 3.4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>AL</p></td><td class="cell"><p>17.8 33.0</p></td><td class="cell"><p>17.7 4.0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><doubt alpha="100.0" length="2" tooSmall="True" monospace="0.0">CO</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">&lt;</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">co</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">600</doubt><p>Documents</p><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">0</doubt><doubt alpha="100.0" length="2" tooSmall="True" monospace="0.0">CD</doubt><doubt alpha="62.5" length="8" tooSmall="False" monospace="0.0">o o ot i</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">LL</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">ö</doubt><doubt alpha="25.0" length="4" tooSmall="False" monospace="0.0">00 O</doubt><doubt alpha="75.0" length="4" tooSmall="False" monospace="0.0">co ö</doubt><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">loö</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">o</doubt><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">o o</doubt><doubt alpha="75.0" length="4" tooSmall="False" monospace="0.0">ii r</doubt><doubt alpha="23.1" length="39" tooSmall="False" monospace="0.0">500       1000      1500 2000 Sentences</doubt><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">00</doubt><doubt alpha="0.0" length="7" tooSmall="False" monospace="0.0">200 400</doubt><figure caption="Figure 3: Performance on gold labels. Left: NER. Right: sentiment (run 1)."></figure><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">wo</doubt><doubt alpha="50.0" length="4" tooSmall="False" monospace="0.0">■a o</doubt><doubt alpha="66.7" length="9" tooSmall="False" monospace="0.0">c co oüa)</doubt><doubt alpha="33.3" length="3" tooSmall="False" monospace="0.0">» o</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">Eo</doubt><doubt alpha="28.6" length="7" tooSmall="False" monospace="0.0">■•;= eg</doubt><p>random active</p><doubt alpha="28.7" length="80" tooSmall="False" monospace="0.0">0   2   4   6   8   10     13     16     19     23 29 Number of uppercase tokens</doubt><figure caption="Figure 2: Annotation time vs. # uppercase tokens"></figure><p>well as sentences with slightly more uppercase to­kens that require annotation.</p><p>In a more detailed analysis, we attempt to distin­guish between (i) the effect of more uppercase ("an­notation required") tokens vs. (ii) the effect of ex­ample dificulty. We it a linear regression model to annotation time vs. the number of uppercase to­kens. For the regression it, we removed all annota­tion times &gt; 60 seconds. Such long times indicate distraction of the worker and are not a reliable mea­sure of difficulty.</p><p>Figure 2 shows the distribution of annotation times for both cases combined and the itted models for each. The model estimated an annotation time of 2.3 secs for each required token for random vs. 2.7 secs for AL. We conclude that the difference in dif-iculty between sentences selected by random sam­pling vs. AL is small, but noticeable.</p></subsection><subsection number="4.4" title="Influence of noise on the selection process"><p>While NER performance for AL is much higher than for random sampling, it is still quite a bit lower than what is possible on gold labels. In the case of AL, there are two reasons why this happens: (i) The noisy labels negatively affect the classifier's ability to learn a good model that is used for classifying the test set. (ii) The noisy labels result in bad interme­diate models that then select suboptimal examples to be annotated next. The AL selection process is "misled" by the noisy examples.</p><p>We conduct an experiment to determine the con­tribution of factors (i) and (ii) to the performance loss. First, we preserve the sequence of sentences chosen by our AL experiments on MTurk, with 5-voting for NER and 4-voting for sentiment but re­place the noisy worker-provided labels by gold la­bels. The performance of classiiers trained on this sequence is the dashed line "MTurk selection, gold labels" in Figure 3 for NER (left) and sentiment (right).</p><p>Second, we compare with a traditional simulated AL experiment with gold labels. Here, the selection too is controlled by gold labels, so the selection has a noiseless classiier available for scoring and can perform optimal uncertainty selection. These are the<page local="8" global="1553"/></p><doubt alpha="100.0" length="1" tooSmall="True" monospace="0.0">O</doubt><doubt alpha="62.5" length="8" tooSmall="False" monospace="0.0">as o c c</doubt><doubt alpha="0.0" length="3" tooSmall="True" monospace="0.0">100</doubt><doubt alpha="89.5" length="19" tooSmall="True" monospace="0.0">Number of sentences</doubt><doubt alpha="0.0" length="2" tooSmall="True" monospace="0.0">10</doubt><doubt alpha="0.0" length="2" tooSmall="True" monospace="0.0">20</doubt><doubt alpha="0.0" length="2" tooSmall="True" monospace="0.0">50</doubt><doubt alpha="89.5" length="19" tooSmall="True" monospace="0.0">Number of documents</doubt><p>Figure 4: Worker accuracy vs. number of HITs. Each point corresponds to one worker (o =random sampling; black and grey for different runs). Left: NER. Right: Sentiment.</p><doubt alpha="66.7" length="9" tooSmall="False" monospace="0.0">active, +</doubt><p>dotted lines "gold selection, gold labels" in Figure 3.</p><p>We used a batch-mode AL setup for this compari­son experiment. For a fair comparison, we adjust the batchsize to be equal to the average <i>staleness </i>of a se­lected example in concurrent MTurk active learning. The staleness of an example is defined as the num­ber of annotations the system has received, but not yet incorporated in the computation of an example's uncertainty score (Haertel et al., 2010).</p><p>For our concurrent NER system, the average stal­eness of an example was about 12 (min: 1, max: 40), for sentiment it was about 2. The figure for NER is higher than the number cited by Haertel et al. (2010) because there are more annotators accessing our sys­tem at the same time via MTurk but not as high for sentiment since documents are longer and retraining the sentiment classifier is faster. The average stale­ness of an example in a batch-mode system is half the batch size. Thus, we set the batch size of our comparison system to 25 for NER and to 4 for sen­timent.</p><p>Returning to the two factors introduced above (i) final effect of noise on test set performance vs.</p><p>(ii) intermediate effect of noise on example selec­tion - we see in Figure 3 that (i) has a large effect on NER whereas (ii) has a noticeable, but small ef­fect.<footnote anchor="3"/> For example, at 1966 sentences, <i>F\ </i>scores are 70.6 (MTurk-MTurk), 81.4 (MTurk-gold) and 84.9 (gold-gold). This means that a performance differ­ence of 10 points Fi has to be attributed to noisy labels resulting in a worse final classifier (effect i), and another 3.5 points are lost due to sub-optimal example selection (effect ii).</p><footnote label="3">Our comparison unit for NER is the sentence. We can­not compare on cost here since we do not know what the per-sentence cost of a "gold" expert annotation is.</footnote><p>For sentiment, the results are different. There is no clear difference between the three runs. We at­tribute this to the fact that the quality of the labels is higher in sentiment than in NER. Our initial ex­periments on sentiment were all negative (showing no improvement of AL compared to random) be­cause label quality was too low. Only after we intro­duced the template described in Section 3 and used 4-voting with <i>a </i>= .75 did we get positive results for AL. This leads to an overall label quality of about 90% (over all runs) which is so high that the differ­ence to using gold labels is small if present at all.</p></subsection></section><section number="5" title="Worker Quality"><p>So far we have assumed that all workers provide annotations of the same quality. However, this is not the case. Figure 4 shows plots of worker accu­racy as a function of worker productivity (number of annotated examples). Some workers submit only one or two HITs just to try out the task. For NER, the majority of workers submit between 5 and 10 sentences, with label qualities between 0.5 and 0.8. The chance level for correctness is around 0.25 (four different named entity categories for uppercase to­kens).<page local="9" global="1554"/> For sentiment, most workers submit 1 to 5 documents, with label qualities between 0.5 and 1. Chance level lies at around 0.5 (for two equally dis­tributed labels).</p><p>While quality for highly productive workers is mediocre in our experiments, other researchers have found extremely bad quality for their most prolific workers (Callison-Burch, 2009). Some of these workers might be spammers who try to submit an­swers with automatic scripts. We encountered some spammers that our heuristics did not detect (shown in the bottom-right areas of Figure 4, left), but the voting mechanism was able to mitigate their nega­tive influence.</p><p>Given the large variation in Figure 4, using worker quality in crowdsourcing for improved training set creation seems promising. We now test two such strategies for NER in an oracle setup.</p><subsection number="5.1" title="Blocking low-quality workers"><p>A simple approach is to refuse annotations from workers that have been determined to provide low quality answers. We simulated this strategy on NER data using oracle quality ratings. We chose NER be­cause of its lower overall label quality. The re­sults are presented in Figure 5 for random (a) and AL (b). For random, quality filtering with low cut­offs helps by removing bad annotations that likely come from spammers. While the voting strategy prevented a performance decrease with bad anno­tations, it needed to expend many extra annotations for correction. With filtering, these extra annotations become unnecessary and the system can learn faster. When low-quality workers are less active, as in the AL dataset, we find no meaningful performance in­crease for low cutoffs up to 0.4. For very high cut­offs (0.7), the beginning of the performance curve shows that further cost reductions can be achieved. However, we did not have enough recorded human annotations available to perform a simulation for the full budget.</p></subsection><subsection number="5.2" title="Trusting high-quality workers"><p>The complementary approach is to take annotations from highly rated workers at face value and imme­diately accept them as the correct label, <i>bypassing </i>the voting procedure. Bypassing saves the cost of repeated annotation of the same sentence. Figure 5 shows learning curves for two bypass thresholds on worker quality (measured as proportion of correct non-O tokens) for random (c) and AL (d). Bypass­ing performs surprisingly well. We find a steeper rise of the learning curve, meaning less cost for the same performance. Not only do we find substantial cost reductions, but also higher overall performance. We believe this is because high-quality annotations can sometimes be voted down by other annotations. If we can identify high-quality workers and directly use their annotations, this can be avoided.</p><p>These experiments are oracle experiments using gold data that is normally not available. In future work, we would like to repeat the experiments using methods for worker quality estimation (Ipeirotis et al., 2010; Donmez et al., 2009). For AL, the choice as to which labels are used (as a result ofvoting, by­passing or other) also has an influence on the selec­tion. However, we had to keep the sequence of the selected sentences fixed in the simulations reported above. While our method of sample selection for AL proved to be quite robust even in the presence of noise, higher quality labels do have an influence on the sample selection (see section 4.4), so the im­provement could be even better than indicated here.</p></subsection><subsection number="5.3" title="Differences in quality between AL and random"><p>The essence of AL is to select examples that are dif­ficult to classify. As observed in our experiments on annotation time, this difficulty is reflected in the amount of time a human needs to work on examples selected through AL. Another effect to expect from difficulty could be lower annotation accuracy. We therefore examined the accuracies for each worker who contributed to both the AL and the random ex­periment. We found that in the NER task, the 20 workers in this group had a slightly higher (0.07) av­erage quality for randomly selected examples. This difference is low and does not suggest a significant drop in accuracy for examples selected in AL.</p></subsection></section><section number="6" title="Conclusion"><p>We have investigated the use of AL in a real-life annotation experiment with human annotators in­stead of traditional simulations with gold labels for named entity recognition and sentiment classifica­tion.<page local="10" global="1555"/> The annotation was performed using MTurk in an AL framework that features concurrent example selection without wait times. We also evaluated two strategies, adaptive voting and fragment recovery, to improve label quality at low additional cost. We find that even for the relatively high noise levels of anno­tations gathered with MTurk, AL is successful, im­proving performance by +6.9 points <i>F\ </i>compared to random sampling for NER and by +2.6% accuracy for sentiment. Furthermore, this performance level is reached at a smaller MTurk cost compared to ran­dom sampling. Thus AL not only reduces annotation costs, but also offers an improvement in absolute performance for these tasks. This is clear evidence that active learning and crowdsourcing are comple­mentary methods for lowering annotation cost and should be used together in training set creation for natural language processing tasks.</p><doubt alpha="100.0" length="2" tooSmall="True" monospace="0.0">CO</doubt><doubt alpha="0.0" length="1" tooSmall="True" monospace="0.0">0</doubt><doubt alpha="100.0" length="2" tooSmall="True" monospace="0.0">LL</doubt><doubt alpha="66.7" length="3" tooSmall="True" monospace="0.0">o o</doubt><doubt alpha="33.3" length="3" tooSmall="False" monospace="0.0">(a)</doubt><doubt alpha="33.3" length="3" tooSmall="False" monospace="0.0">(b)</doubt><doubt alpha="33.3" length="3" tooSmall="False" monospace="0.0">(c)</doubt><doubt alpha="33.3" length="3" tooSmall="False" monospace="0.0">(d)</doubt><doubt alpha="64.7" length="68" tooSmall="True" monospace="0.0">baseline 5-voting min. quality 0.1 min. quality 0.4 min. quality 0.7</doubt><doubt alpha="12.5" length="32" tooSmall="True" monospace="0.0">0     1000  2000  3000 4000 Cost</doubt><doubt alpha="16.7" length="24" tooSmall="True" monospace="0.0">1000 2000 3000 4000 Cost</doubt><doubt alpha="66.7" length="39" tooSmall="True" monospace="0.0">baseline 5-voting bypass 0.9 bypass 0.7</doubt><doubt alpha="12.5" length="8" tooSmall="False" monospace="0.0">^-,-,-r-</doubt><doubt alpha="0.0" length="23" tooSmall="True" monospace="0.0">0   1000 2000 3000 4000</doubt><doubt alpha="100.0" length="4" tooSmall="True" monospace="0.0">Cost</doubt><figure caption="Figure 5: Blocking low-quality workers: (a) random, (b) AL. Bypass voting: (c) random, (d) AL."></figure><doubt alpha="0.0" length="2" tooSmall="True" monospace="0.0">0)</doubt><doubt alpha="44.4" length="9" tooSmall="True" monospace="0.0">0) O o CO</doubt><doubt alpha="44.4" length="9" tooSmall="True" monospace="0.0">0) O O CO</doubt><p>We have also conducted oracle experiments that show that further performance gains and cost sav­ings can be achieved by using information about worker quality. We plan to confirm these results by using estimates of quality in the future.</p></section><section number="7" title="Acknowledgments"><p>Florian Laws is a recipient of the Google Europe Fellowship in Natural Language Processing, and this research is supported in part by his fellowship. Christian Scheible is supported by the Deutsche Forschungsgemeinschaft project Sonderforschungs­bereich 732.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>iff</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>I</i></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>!</p></td><td class="cell"><p>- baseline 5-voting</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>; f</i></p></td><td class="cell"><p>---bypass 0.9</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>—  bypass 0.7</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><references><p>Jason Baldridge and Alexis Palmer. 2009. How well does active learning <i>actually </i>work? Time-based eval­uation of cost-reduction strategies for language docu­mentation. In <i>Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, </i>pages 296-305.</p><p>Anthony Brew, Derek Greene, and Padraig Cunningham. 2010. Using crowdsourcing and active learning to track sentiment in online media. In <i>Proceeding ofthe 2010 conference on ECAI2010: 19th European Con­ference on Artificial Intelligence</i>, pages 145-150.</p><p>Klaus Brinker. 2003. Incorporating diversity in active learning with support vector machines. In <i>Proceed­ings ofthe Twentieth International Conference on Ma­chine Learning (ICML 2003), </i>pages 59-66.</p><p>Chris Callison-Burch. 2009. Fast, cheap, and cre­ative: evaluating translation quality using Amazon's Mechanical Turk. In <i>Proceedings of the 2009 Confer­ence on Empirical Methods in Natural Language Pro­cessing, </i>pages 286-295.</p><p>Bob Carpenter and Massimo Poesio. 2010. Models of data annotation. Tutorial at the seventh interna­tional conference on Language Resources and Eval­uation (LREC 2010).</p><p>Nancy Chinchor, David D. Lewis, and Lynette Hirschman. 1993. Evaluating message understanding systems: an analysis of the third message understand­ing conference (muc-3).  <i>Computational Linguistics,</i> 19(3):409-449.</p><p>Pinar Donmez and Jaime G. Carbonell. 2008. Proactive learning: cost-sensitive active learning with multiple imperfect oracles. In <i>Proceeding of the 17thACMcon-ference on Information and knowledge management, </i>pages 619-628.</p><p>Pinar Donmez, Jaime G. Carbonell, and Jeff Schnei­der.  2009.  Efficiently learning the accuracy of labeling sources for selective sampling.<page local="11" global="1556"/> In <i>Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, </i>pages 259­268.</p><p>Tim Finin, William Murnane, Anand Karandikar, Nicholas Keller, Justin Martineau, and Mark Dredze. 2010. Annotating named entities in twitter data with crowdsourcing. In <i>Proceedings ofthe NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, </i>pages 80-88. Ben Hachey, Beatrice Alex, and Markus Becker. 2005. Investigating the effects of selective sampling on the annotation task. In <i>CoNLL '05: Proceedings ofthe 9th Conference on Computational Natural Language Learning, </i>pages 144-151. Robbie Haertel, Paul Felt, Eric K. Ringger, and Kevin Seppi. 2010. Parallel active learning: Eliminating wait time with minimal staleness. In <i>Proceedings of the NAACL HLT 2010 Workshop on Active Learning for Natural Language Processing, </i>pages 33-41. Panagiotis G. Ipeirotis, Foster Provost, and Jing Wang. 2010. Quality management on amazon mechanical turk. In <i>Proceedings ofthe ACMSIGKDD Workshop</i></p><p><i>on Human Computation (HCOMP '10). </i>Nolan Lawson, Kevin Eustice, Mike Perkowitz, and Meliha Yetisgen-Yildiz. 2010. Annotating large email datasets for named entity recognition with mechanical turk. In <i>Proceedings ofthe NAACL HLT 2010 Work­shop on Creating Speech and Language Data with Amazon's Mechanical Turk, </i>pages 71-79. David D. Lewis and William A. Gale. 1994. A sequential algorithm for training text classifiers. In <i>Proceedings of the 17th Annual International ACM SIGIR Confer­ence on Research and Development in Information Re­trieval, </i>pages 3-12. Christopher Manning and Dan Klein. 2003. Optimiza­tion, maxent models, and conditional estimation with­out magic.   In <i>Proceedings ofthe 2003 Conference ofthe North American Chapter ofthe Association for Computational Linguistics on Human Language Tech­nology: Tutorials - Volume 5, </i>pages 8-8. Eric W. Noreen. 1989. <i>Computer-intensive methods for</i></p><p><i>testing hypotheses: an introduction. </i>Wiley. Miles Osborne and Jason Baldridge. 2004. Ensemble-based active learning for parse selection. In Daniel Marcu Susan Dumais and Salim Roukos, edi­tors, <i>HLT-NAACL 2004: Main Proceedings, </i>pages 89­96.</p><p>Sebastian Pado, 2006.  <i>User's guide to sigf: Signifi­cance testing by approximate randomisation. </i>Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002.   Thumbs up?   Sentiment classification using machine learning techniques. In <i>Proceedings ofthe</i> <i>2002 Conference on Empirical Methods in Natural Language Processing (EMNLP), </i>pages 79-86.</p><p>Ines Rehbein, Josef Ruppenhofer, and Alexis Palmer. 2010. Bringing active learning to life. In <i>Proceed­ings ofthe 23rd International Conference on Compu­tational Linguistics (Coling 2010), </i>pages 949-957.</p><p>Eric Ringger, Peter McClanahan, Robbie Haertel, George Busby, Marc Carmen, James Carroll, Kevin Seppi, and Deryle Lonsdale. 2007. Active learning for part-of-speech tagging: Accelerating corpus annotation. In <i>Proceedings ofthe Linguistic Annotation Workshop at ACL-2007, </i>pages 101-108.</p><p>Andrew Schein and Lyle Ungar. 2007. Active learn­ing for logistic regression: An evaluation. <i>Machine Learning, </i>68(3):235-265.</p><p>Burr Settles, Mark Craven, and Lewis Friedland. 2008. Active learning with real annotation costs. In <i>Proceed­ings ofthe NIPS Workshop on Cost-Sensitive Learn­ing, </i>pages 1069-1078.</p><p>Rion Snow, Brendan O'Connor, Daniel Jurafsky, and An­drew Ng. 2008. Cheap and fast - but is it good? evaluating non-expert annotations for natural language tasks. In <i>Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing,</i> pages 254-263.</p><p>Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: language-independent named entity recognition. In <i>Proceedings ofthe seventh conference on Natural language learn­ing at HLT-NAACL (CoNLL 2003), </i>pages 142-147.</p><p>Katrin Tomanek, Joachim Wermter, and Udo Hahn. 2007. An approach to text corpus construction which cuts annotation costs and maintains reusability of an­notated data. In <i>Proceedings of the 2007 Joint Confer­ence on Empirical Methods in Natural Language Pro­cessing and Computational Natural Language Learn­ing, </i>pages 486-495.</p><p>Simon Tong and Daphne Koller. 2002. Support vec­tor machine active learning with applications to text classification. <i>The Journal of Machine Learning Re­search, </i>2:45-66.</p><p>Robert Voyer, Valerie Nygaard, Will Fitzgerald, and Han­nah Copperman. 2010. A hybrid model for anno­tating named entity training corpora. In <i>Proceedings of the Fourth Linguistic Annotation Workshop, </i>pages 243-246.</p><p>Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In <i>Proceedings ofthe conference on Human Language Technologyand Empirical Meth­ods in Natural Language Processing, </i>pages 347-354.</p></references></body></article>