Translation retrieval is defined as the task of, for a given source language (LI) input, retrieving the target language (L2) string which best translates it. Retrieval is carried out over a translation memory made up of translation records, that is LI strings coupled with an L2 translation. A single transla­tion retrieval task was offered in Senseval-2, from Japanese into English, and it is this task that we target in this paper.

Conventionally, translation retrieval is carried out by way of determining the LI string in the trans­lation memory most similar to the input, and re­turning the L2 string paired with that string as a translation for the input. It is important to realise that at no point is the output compared back to the input to determine its "translation adequacy", a job which is left up to the system user.

Determination of the degree of similarity between the input and LI component of each translation record can take a range of factors into consideration, including lexical (character or word) content, word order, parse tree topology and conceptual similarity. In this paper, we focus on a simple character-based (lexical) method and more sophisticated parse tree comparison (structural) method.

Both methods discussed herein are fully unsuper­vised. The lexical method makes use of no exter­nal resources or linguistic knowledge whatsoever. It treats each string as a "bag of character bigrams" and calculates similarity according to Dice's Coef­ficient. The structural method, on the other hand, relies on both morphological and syntactic analysis, in the form of the publicly-available JUMAN (Kurohashi and Nagao, 1998b) and KNP (Kurohashi and Nagao, 1998a) systems, respectively, and also the Japanese Goi-Taikei thesaurus (Ikehara et al., 1997) to measure conceptual distance. A parse tree is generated for the LI component of each translation record, and also each input, and similarity gauged by both topological resemblance between parse trees and conceptual similarity between nodes of the parse tree.

Translation records used by the two systems were taken exclusively from the translation memory pro­vided for the task.

In the proceeding sections, we briefly review the Japanese translation task (§ 2) and detail our par­ticular use of the data provided for the task (§ 3). Next, we outline the lexical method (§ 4) and struc­tural method (§ 5), and compare and discuss the performance of the two methods (§ 6).

The Japanese translation task data was made up of a translation memory and test set. The translation memory was dissected into 320 disjoint segments according to headwords, with an average of 21.6 translation records per headword (i.e. 6920 transla­tion records overall). The purpose of the task was to select for a given headword which (if any) of the translation records gave a suitable translation for that word. The task stipulated that a maximum of one translation record could be selected for each in­put (allowing for the possibility of an unassignable output, indicating that no appropriate translation could be found). Translations were selected by way of a translation record ID, and systems were not re­quired to actually identify what part of the L2 string in the selected translation record was the translation for the headword.

Translation records took the form of Japanese-English pairings of word clusters, isolated phrases, clauses or sentences containing the headword, at an average of 8.0 Japanese characters and 4.0 English words per translation record. In some instances, multiple semantically-equivalent translations were given for a single expression, such as "corporation which is in danger of bankruptcy" and "unsound cor­poration" for abunai kigyö; all such occurrences were marked by the annotator. For some other transla­tion records, the annotator had provided a list of lex­ical variants or a paraphrase of the LI expression to elucidate its meaning (not necessarily involving the headword), or made a note as to typical arguments taken by that expression (e.g. "refers to a person").

In the test data, inputs took the form of para­graphs taken from newspaper articles, within which a single headword had been identified for transla­tion. The average input length was 697.9 characters, nearly 90 times the LI component of each translation record. In its raw form, therefore, the translation task differs from a conventional translation retrieval task in that translation records and inputs are not directly comparable, in the sense that translation records are never going to provide a full translation approximation for the overall input.

In adapting the task data to our purposes, we first carried out limited normalisation of both the trans­lation memory and test data by: (a) replacing all numerical expressions with a common NUM marker, and (b) normalising punctuation.

In order to maximise the disambiguating poten­tial of the translation memory, we next set about automatically deriving as many discrete translation records as possible from the original translation memory. Multiple lexical variants of the same basic translation record (indexed identically) were gener­ated in the case that: (a) a lexical alternate was provided (in which case all variants were listed in parallel); (b) a paraphrase was provided by the an­notator (irrespective of whether the paraphrase in­cluded the headword or not); (c) syntactic or seman­tic preferences were listed for particular arguments in the basic translation record (in which case lexical variants took the form of strings expanded by adding in each preference as a string). At the same time, for each headword, any repetitions of the same LI string were completely removed from the translation record data. This equates to the assumption that the translation listed first in the translation memory is the most salient or commonplace.

This method of translation record derivation re­sulted in a total of 152 new translation records, whereas the removal of duplicate LI strings for a given headword resulted in the deletion of 670 translation records; the total number of translation records was thus 6402, at an average of 20.0 trans­lation records per headword.

We experimented with a number of methods for abbreviating the inputs, so as to achieve direct com­parability between inputs and translation records. First, we extracted the clause containing the head­word instance to be translated. This was achieved through a number of ad hoc heuristics driven by the analysis of punctuation. These clause-level instances served as the inputs for the structural method. We then further "windowed" the inputs for the lexical method, by allowing a maximum of 10 characters to either side of the headword. No attempt was made to identify or enforce the observation of word bound­aries in this process.

As stated above, the lexical method is based on character-based indexing, meaning that each string is naively treated as a sequence of characters. Rather than treat each individual character as a single seg­ment, however, we chunk adjacent characters into bigrams in order to capture local character contigu­ity. String similarity is then determined by way of Dice's Coefficient, calculated according to:

2 x EeeiNt^TRj min {freQiN:7l(e)JreqTR.(e)) len(IN^) + len{TRi) where IN^ is the abbreviated version of the in­put string INm (see above) and TRi is a transla­tion record; each e is a character bigram occurring in either IN^ or TRi, freqIN* (e) is defined as the weighted frequency of bigram type e in liV^, and /en(i7V^) is the character bigram length of IN^.Bigram frequency is weighted according to character type: a bigram made up entirely of hiragana charac­ters (generally used in functional words/particles) is given a weight of 0.2 and all other bigrams a weight of 1. Note that Dice's Coefficient ignores segment order, and that each string is thus treated as a "bag of character bigrams".

Our choice of the combination of Dice's Coef­ficient, character-based indexing and character bi­grams (rather than any other n-gram order or mixed n-gram model) is based on the findings of Baldwin (2001b; 2001a), who compared character- and word-based indexing in combination with both segment order-sensitive and bag-of-words similarity measures and with various n-gram models. As a result of extensive evaluation, Baldwin found the combina­tion of character bigram-based indexing and a bag-of-words method (in the form of either the vector space model or Dice's Coefficient) to be optimal. Our choice of Dice's Coefficient over the vector space model is due to the vector space model tending to blithely prefer shorter strings in cases of low-level character overlap, and the ability of Dice's Coeffi­cient to pick up on subtle string similarities under such high-noise conditions.

Given the limited lexical context in translation records (8.0 Japanese characters on average), our method is highly susceptible to the effects of data sparseness. While we have no immediate way of rec­onciling this shortcoming, it is possible to make use of the rich lexical context of the full inputs (i.e. in original paragraph form rather than clause or win­dowed clause form). Direct comparison of the full inputs with translation records is undesirable as high levels of spurious matches can be expected outside the scope of the original translation record expres­sion. Inter-comparison of full inputs, on the other hand, provides a primitive model of domain similar­ity. Assuming that high similarity correlates with a high level of domain correspondence, we can apply a cross-lingual corollary of the "one sense per dis­course" observation (Gale et al., 1992) in stipulat­ing that a given word will be translated consistently within a given domain. By ascertaining that a given input closely resembles a second input, we can use the combined translation retrieval results for the two inputs to hone in on the optimal translation for the two. We term this procedure domain-based sim­ilarity consolidation.

The overall retrieval process thus involves: (1) carrying out standard translation retrieval based on the abbreviated input, (2) using the original test set to determine the full input string most similar to the current input, and (3) performing translation re­trieval independently using the abbreviated form of the maximally similar alternate input. Numerically, the combined similarity is calculated as:

+ maxsimi(INmJNn) simi(IN*,TRi)) where INm is the current input (full form), INmis the abbreviated form of INm, sim\ is as defined above, and INn is any input string other than the current input. Note that the multiplication by 0.5 simply normalises the output of szra-2 to the range [0,1]. For each input INm, the ID for that transla­tion record which is deemed most similar to INm is returned, with translation records occurring earlier in the translation memory selected in the case of a tie.

The structural method- contrasts starkly with the lexical method in that it is heavily resource-dependent, requiring a morphological analyser, parser and thesaurus. It operates over the same translation memory data as the lexical method, but uses only the abbreviated forms of the inputs (to the clause level) and does not consider inter-input similarity.

JUMAN (Kurohashi and Nagao, 1998b) is first used to segment each string (translation records and inputs), based on the output of which, the KNP parser (Kurohashi and Nagao, 1998a) is used to de­rive a parse tree for the string. The reason for ab­breviating inputs only as far as the clause level for the structural method, is to enhance parseability.

Further pruning takes place implicitly further down­stream as part of the parse tree matching process.

KNP returns a binary parse tree, with leaves cor­responding to optionally case-marked phrases. Each leaf node is simplified to the phrase head and the (optional) case marker normalised (according to the KNP output).

As for the lexical method, all translation records corresponding to the current headword are matched against the parse tree for the input, and the ID of the closest-matching tree returned. In comparing a given pair of parse trees T and T, we proceed as follows in direction d G {up, down}:

1. Set p to the leaf node containing the headword in T, and similarly initialise p in T; initialise n to 0

4. Increment n by 1, set p and p to their respec­tive adjacent leaf nodes in direction d within the parse tree; goto step 2.

Here, plm is the case marker associated with node p, plf is the filler associated with node p%, and the ^ operator represents lexical inequality; conceptsim calculates the conceptual similarity of the two fillers in question according to the Goi-Taikei thesaurus (Ikehara et al., 1997). We do this by, for each sense pairing of the fillers, determining the least common hypernym and the number of edges separating each sense node from the least common hypernym. The conceptual distance of the given senses is then de­termined according to the inverse of the greater of the two edge distances to the hypernym node, and the overall conceptual distance for the two fillers as the minimum such sense-wise conceptual distance.

We match both up and down the tree structure from the headword node, and evaluate the com­bined similarity as the sum of the individual ele­ments of the returned tuples. That is, if an up­ward match returned (i,m) and a downward match (j,n), the overall similarity would be (i -f j,m + n). The translation output is the ID of the translation record producing the greatest such similarity, where (w,x) > (y,z) iff w > y or (w = y A x > z). As a result, conceptual similarity is essentially a tie-breaking mechanism, and the principal determining factor is the number of phrase levels over which the parse trees match. In the case that there is a tie for best translation, the translation record with the longest LI string is (arbitrarily) chosen, and in the case that this doesn't resolve the stalemate, a trans­lation record is chosen randomly. In the case that all translation records score (0,0), we deem there to be no suitable translation in the translation memory, and return unassignable.

As mentioned in Section 2, crude selectional preferences (of the form Person or Building) were provided on certain argument slots in translation records. These were supported by semi-automatically mapping the preference type onto the Goi-Taikei thesaurus structure, and modifying the 7^ operator to non-sense subsumption of the trans­lation record filler by the input selectional prefer­ence, in step 3 of the parse tree match algorithm. Selectional preferences were automatically mapped onto nodes of the same name if they existed, and manually linked to the thesaurus otherwise.

The translation retrieval accuracy for the two meth­ods is given in Table 1, along with a baseline accu­racy arrived at through random translation record selection for the given headword. Note that as we attempt to translate all inputs, the presented accu­racy figures correspond to both recall and precision.

The most striking feature of the results is that the lexical method has a clear advantage over the structural method, while both methods outperform the baseline. Obviously, it would be going too far to discount structural methods outright based on this limited evaluation, particularly as the lexical method has undergone extensive testing and tuning over other datasets, whereas the structural method is novel to this task. It is surprising, however, that a technique as simple as the lexical method, requir­ing no external resources and ignoring even word boundaries and word order, should perform so well.

The main area in which the structural method fell short was unassignable inputs where no transla­tion record displayed even the same case marking on the headword. Indeed 130 or 10.8% of inputs were tagged unassignable, despite them compris­ing only 0.3% of the solution set. Note, however, that even for only those inputs where the struc­tural method was able to produce a match, the lexi­cal method significantly outperformed the structural method (50.2% vs. 45.4%, respectively).

Conversely for the lexical method, at present, a translation record is selected irrespective of the mag­nitude of the similarity value, and it would be a trivial process to implement a similarity cutoff, be­low which an unassignable result would be re­turned. Preliminary analysis of the correlation be­tween the lowest similarity values and inputs anno­tated as unassignable indicates that this method could be moderately successful (see Baldwin et al. (to appear)).

The translation task was designed such that par­ticipants didn't get access to annotated inputs until after the submission of final results, meaning that parameter settings and fine-tuning of techniques had to be carried out according to intuition only. Post hoc evaluation of methods such as domain-based similarity consolidation suggests that it does have a significant impact on system performance (Baldwin et al., to appear), although even in its basic config­uration (using clause inputs and no domain-based similarity consolidation), the lexical method is su­perior to the structural method as presented herein.

In conclusion, this paper has served to describe each of a lexical and structural translation retrieval method, as applied to the Senseval-2 Japanese translation task. The lexical method modelled strings as a bag of character bigrams, but incor­porated a number of novel techniques including domain-based similarity consolidation in reaching a final decision as to the translation record most sim­ilar to the input. The structural method, on the other hand, compared parse trees and had recourse to conceptual similarity, but in a relatively rudimen­tary form. Of the two proposed methods, the lexical method proved to be clearly superior, although both methods were well above the baseline performance.

This paper was supported in part by the Research Collaboration between the Nippon Telegraph and Telephone Company (NTT) Communication Science Laboratories and CSLI, Stanford University.