Scoring Penalty Characteristics

The scoring function s generates a non-negative value for each example word wh penalty characteristic Cj (Table 3), distance dt of wt from the example anchor. In IIT1, dt is not considered, so a penalty calculation is independent of the word position in the example. In IIT2, di reduces penalties for wt further away from the example anchor.

If an example anchor alignment with the context word is the only open-class match for an example, the example receives a zero score.

Haynes (2001) describes these calculations in more detail.

A sense of a target word receives the maximum score of the examples related to that sense. The systems suggest the sense(s) with the highest score, with multiple senses in the response in the event of ties. (If a tie occurs because the same example was included for two senses, the other senses are eliminated, the common example is dropped from the example set of the remaining senses, and the sense scores are recomputed.) If no sense receives a score greater than zero, the first sense is chosen.

IIT1 and IIT2 match a context word independent of other sense assignment decisions. The IIT3 system (English All Word

SENSEVAL-2 English All Word Results

task only) uses the IIT1 scoring algorithm for target words, but limits the senses of preceding context words to the sense tags already assigned.

The implementation and results presented here do not seem to justify this optimism. There are several reasons, though, why the method should not be dismissed without further investigation:

• The example sets were empty for a number of the candidate word senses. When this occurred, the system constructed a pseudo example by appending the WordNet gloss to the target word. This was sufficient for most collocation senses and some non-collocation senses such as call as in calling a square dance (where the gloss includes square and dance, one of which is highly likely to occur in any use of the sense). Others such as day as in sidereal day or turn off (gloss cause to feel intense dislike or distaste) competed at a disadvantage.

• The pattern matching and scoring methods were never tuned against any corpus data. This allowed the algorithm to have few competitors in the class of untrained systems, but scoring methods relied on intuition-founded heuristics. Such tuning should improve precision and recall.

• The approach was developed to be used in tandem with statistical approaches. Further research is required before its additive value can be fully assessed. IIT3 would have done better to be based on IIT2 and an approach maximizing the scores for a sentence should do even better.

• The best-matching example was chosen regardless of how bad a match was involved. The system also defaulted to the first sense encountered when all examples had a zero score. Using threshold score values may well provide substantial precision improvements (at the expense of recall).

• Semantic annotation of the WordNet examples should improve the results.

In addition, the following programming errors affected the precision and recall results:

• The generated answers for many adjective senses (those with similarity relations) were incorrectly formatted and were therefore always scored as incorrect. For example, in the IIT1 entry for the English Lexical Sample, 7.1% of all annotations were incorrectly formatted. Scoring only the answers that were correctly formatted raises the course-grained precision for IIT1

to 36.7% and fine-grained precision to 26.1%, competitive with the course-grained performance of the best non-corpus system.

• No annotations were generated for target words preceded by the word to. This results in recall ^ precision as seen in Table 4 and Table 5.

• In a few rare cases, the system identified the incorrect example word as the example anchor. One such occurrence was the synset art, fine art and the example a fine collection of art The system considered it an example of the fine art collocation and chose fine as the anchor.