The first step in the swat-E algorithm is to create a ranked list of English substitutes. For each En­glish test sentence Hw containing the target word w, we use the syntagmatic coherence criterion de­scribed above to create E, a ranking of the syn­onyms of w taken from Roget's thesaurus. We use the Lancaster stemmer to ensure that we count all morphologically similar lexical substitutes.

Next, we use a bilingual dictionary to convert our candidate English substitutes into candidate Spanish substitutes, forming a new ranked list S'. For each item (e, n, /) in E', and for each Spanish translation s of e, we add the triple (s, n, /) to S'. Since different English words can have the same Spanish translation s, we can end up with multiple triples in S' that have the same values for s and n. For example, if s\ is a translation of both e\ and e2, and the triples (ei, 4,87) and (e2,4,61) appear in E', then S' will contain the triples (si, 4, 87) and (si, 4,61). We merge all such "duplicates" by summing their frequencies. In this example, we would replace the two triples containing s\ with a new triple, (si, 4,148). After merging all dupli­cates, we re-sort S' by n, breaking ties by /. No­tice that since triples are merged only when both s and n are the same, Spanish substitutes can appear multiple times in S'\ once for each value of n.

At this point, we have a ranked list of candi­date Spanish substitutes, S'. From this list S', we keep only those Spanish substitutes that are direct translations of our original word w. The reason for doing this is that some of the transla­tions of the synonyms of w have no overlapping meaning with w. For example, the polysemous English noun "bug" can mean a flaw in a com­puter program (cf. test item 572). Our thesaurus lists "hitch" as a synonym for this sense of "bug". Of course, "hitch" is also polysemous, and not ev­ery translation of "hitch" into Spanish will have a meaning that overlaps with the original "bug" sense. Translations such as "enganche", having the "trailer hitch" sense, are certainly not appro­priate substitutes for this, or any, sense of the word "bug". By keeping only those substitutes that are also translations of the original word w, we main­tain a cleaner list of candidate substitutes. We call this filtered list of candidates S.

For each English sentence in the test set, we now have a ranked list of cross-lingual lexical substi-

6: end while

tutes, S. In the oot scoring method, we selected the top 10 substitutes in the ranked list S. If there were less than 10 items (but at least one item) in S, we duplicated answers from our ranked list un­til we had made 10 guesses. (See Section 3.4 for further details on this process.) If there were no items in our ranked list, we returned the most fre­quent translations of w as determined by the un-igram counts of these translations in the Spanish Web IT corpus.

For our best answer, we returned multiple re­sponses when the highest ranked substitutes had similar frequencies. Since S was formed by trans­ferring the frequency of each English substitute e onto all of its Spanish translations, a single English substitute that had appeared with high frequency would lead to many Spanish substitutes, each with high frequencies. (The frequencies need not be ex­actly the same due to the merging step described above.) In these cases, we hedged our bet by re­turning each of these translations.

Representing the i-th item in S as (si,rii, fa), our procedure for creating the best answer can be found in Figure 1. We allow all items from S that have the same value of n as the top ranked item and have a frequency at least 75% that of the most frequent item to be included in the best answer.

Of the 1000 test instances, we provided a single "best" candidate 630 times, two candidates 253 times, three candidates 70 times, four candidates 30 times, and six candidates 17 times. (We never returned five candidates).

The Swat-S system used both Google's and Ya­hoo's online translation tools, the Spanish section of the WeblT European 5-gram corpus, Roget's online thesaurus, TreeTagger (Schmid, 1994) for morphological analysis and both Google's and Ya­hoo's English-Spanish dictionaries. We formed a single Spanish-English dictionary by combining the translations found in both dictionaries.

To find the cross-lingual lexical substitutes for a target word in an English sentence, we first trans­late the sentence into Spanish and then use the syntagmatic coherence criterion on the translated Spanish sentence.

In order to perform this monolingual Spanish lexical substitution, we need to be able to iden­tify the target word we are attempting to substitute in the translated sentence. We experimented with using Moses (Koehn et al., 2007) to perform the machine translation and produce a word alignment but we found that Google's online translation tool produced better translations than Moses did when trained on the Europarl data we had available.

In the original English sentence, the target word is marked with an XML tag. We had hoped that Google's translation tool would preserve the XML tag around the translated target word, but that was not the case. We also experimented with using quotation marks around the target word instead of the XML tag. The translation tool often preserved quotation marks around the target word, but also yielded a different, and anecdotally worse, transla­tion than the same sentence without the quotation marks. (We will, however, return to this strategy as a backoff method.) Although we did not exper­iment with using a stand-alone word alignment al­gorithm to find the target word in the Spanish sen­tence, Section 4.3 provides insights into the possi­ble performance gains possible by doing so.

Without a word alignment, we were left with the following problem: Given a translated Span­ish sentence H, how could we identify the word w that is the translation of the original English target word, vl Our search strategy proceeded as fol­lows.

1. We looked up v in our English-Spanish dictio­nary and searched H for one of these trans­lations (or a morphological variant), choosing the matching translation as the Spanish target word. If the search yielded multiple matches, we chose the match that was in the most similar position in the sentence to the position of v in the English sentence. This method identified a match in 801 of the 1000 test sentences.

2. If we had not found a match, we translated each word in H back into English, one word at a time. If one of the re-translated words was a synonym of v, we chose that word as the target word. If there were multiple matches, we again used position to choose the target.

3. If we still had no match, we used Yahoo's trans­lation tool instead of Google's, and repeated steps 1. and 2. above.

4. If we still had no match, we reverted to using Google's translation tool, this time explicitly offsetting the English target word with quota­tion marks.

In 992 of the 1000 test sentences, this four-step procedure produced a Spanish sentence Hw with a target w. For each of these sentences, we pro­duced E', the list of ranked Spanish substitutes using the syntagmatic selection coherence crite­rion described in Section 3.1. We used the Span­ish Web IT corpus as a source of n-gram counts, and we used the Spanish translations of v as the candidate substitution set E. For the remaining 8 test sentences where we could not identify the target word, we set E' equal to the top 10 most frequently occurring Spanish translations of v as determined by the unigram counts of these trans­lations in the Spanish Web IT corpus.

For each English sentence in the test set, we se­lected the single best item in E as our answer for the best scoring method.

For the oot scoring method, we wanted to en­sure that the translated target word w, identified in Section 3.3.2, was represented in our output, even if this substitute was poorly ranked in E'. Ifw ap­peared in E', then our oot answer was simply the first 10 entries in E'. \îw was not in E', then our answer was the top 9 entries in E' followed by w.

As we had done with our swat-E system, if the oot answer contained less than 10 items, we repeated answers until we had made 10 guesses. See the following section for more information.