Relaxation is a well-known technique used to solve consistent labelling problems. Actually, relax­ation is a family of energy-function-minimizing al­gorithms closely related to Boltzmann machines, gradient step, and Hopfield nets.

À consistent labelling problem consists of, given a set of variables, assigning to each variable a la­bel compatible with the labels of tin; other ones, according to a set of compatibility constraints.

Many problems can be stated as a labelling problem: the travelling salesman problem, n-queens, corner and edge recognition, image; smoothing, etc.

In this paper we will try to make a first insight into applying relaxation labelling to natural lan­guage processing. The main idea of the work is that NLP problems such as POS tagging or WSD can be stated as constraint satisfaction problems, thus, they could be addressed with the usual tech­niques of that field, such as relaxation labelling.

It seems reasonable to consider POS tagging or WSD as combinatorial problems in which we have a set of variables (words in a sentence) a set of possible labels for each one (POS tags or senses), and a set of constraints for these labels. We might also combine both problems in only one, and ex­press constraints between the two types of tags, using semantic information to disambiguate POS tags and visccversa. This is not the point in this paper, but it will be addressed in further work.

Relaxation labelling is a generic name for a family of iterative algorithms which perform function op­timization, based on local information. See (Tor­ras 89) for a clear exposition.

Let ti = {t[, tl,..., t*n.} be the set of possible labels for variable Wj.

Let GS be a set of constraints between the la­bels of the variables. Each constraint C € GS states a "compatibility value" Cr for a combina­tion of pairs variable-label. Constraints can be of any order (that is, any number of variables may be involved in a constraint).

The aim of the algorithm is to find a weighted labelling such that "global consistency" is maxi­mized. A weighted labelling is a weight assigna­tion for each possible label of each variable;. Max­imizing "Global consistency" is defined as maxi­mizing YlijP) x Sij > V'ui- Where p*- is the weight for label j in variable w4 and 5V, the support re­ceived by the same combination. The support for a pair variable-label expresses how compatible is that pair with the labels of neighbouring variables, according to the constraint set.

The relaxation algorithm consists of:

• start in a random weighted labelling.

• for each variable, compute the "support" that each label receives from the current, •weights for the labels of the other variables.

• Update the weight of each variable label ac­cording to the support obtained.

• iterate the process until a convergence crite­rion is met.

The support computing and label weight chang­ing must be performed in parallel, to avoid that changing the a variable weights would affect the support computation of the others.

The algorithm requires a way to compute which is the support for a variable label given the others and the constraints. This is called the "support function".

Several support functions are used in the liter­ature to define the support received by label j of variable i (SVj).

Rii = {r\r- [(«,, , fk\),(vu *}),..., (vrd, trk"d)}} the set of constraints on label j for variable i, i.e. the constraints formed by any combination of pairs variable-label that includes the pair (vi,tj). P'k\ (m) the weight assigned to label for variable vri at time m.

Viy) the set of all possible subsets of variables in V.

Rfj (for G € Viy)) the set of constraints on tag i for word j in which the involved variables are exactly those of G.

Usual support functions arc based on comput­ing, for each constraint r involving (t>i,ij), the "constraint influence", Inf(r) = Cr x pk\(rn) x ... x p^d (m), which is the product of the current weights for the labels appearing the constraint except (vi,tj) (representing how applicable is the constraint in the current context) multiplied by Crwhich is the constraint compatibility value (stat­ing how compatible is the pair with the context).

The first formula combines influences just adding them:

The next formula adds the constraint influences grouped according to the variables they involve, then multiplies the results of each group to get the final value:

Gev(v) reiî«

The last formula is the same than the previous one, but instead of adding the constraint influ­ences in the same group, just picks the maximum.

Gev\v)reR$

The algorithm also needs an "updating func­tion" to compute at each iteration which is the new weight for a variable label, and this compu­tation must be done in such a way that it can be proven to meet a certain convergence criterion, at least under appropriate conditions

Several formulas have been proposed and some of them have been proven to be approximations of a gradient step algorithm.

Usual updating functions are the following.

The first formula increases weights for labels with support greater than 1, and decreases those with support smaller than 1. The denominator expression is a normalization factor.

The second formula increases weight for labels with support greater than 0 and decreases weight for those with support smaller than 0.

where — 1 < SVj < +1 Advantages of the algorithm are:

• Its highly local character (only the state at previous time step is needed to compute each new weight). This makes the algorithm highly parallelizable.

• Its expressivity, since we state the problem in terms of constraints between labels.

• Its flexibility, we don't have to check absolute coherence of constraints.

• Its robustness, since it can give an answer to problems without an exact solution (incom­patible constraints, insufficient data...)

• Its ability to find local-optima solutions to NP problems in a non-exponential time. (Only if wc have an upper bound for the num­ber of iterations, i.e. convergence is fast or the algorithm is stopped after a fixed number of iterations. See section 4 for further details)

Drawbacks of the algorithm are:

• Its cost. Being n the number of variables, v the average number of possible labels per variable, c the average number of constraints per label, and I the average number of iter­ations until convergence, the average cost is n x v x c x 7 , an expression in which the multi­plying terms might be much bigger than n if we deal with problems with many values and constraints, or if convergence is not quickly achieved.

• Since it acts as an approximation of gradi­ent step algorithms, it has similar weakness: Found optima are local, and convergence is not always guaranteed.

• In general, constraints must be written manu­ally, since they are the modelling of the prob­lem. This is good for easily modelablc or reduced constraint-set problems, but in the case of POS tagging or WSD constraints are too many and too complicated to be written by hand.

In this section we expose our application of relax­ation labelling to assign part of speech tags to the words in a sentence.

Addressing tagging problems through optimiza­tion methods has been done in (Schmid 94) (POS tagging using neural networks) and in (Cowie et al. 92) (WSD using simulated annealing). (Pelillo & Rcficc 94) use a toy POS tagging problem to ex­periment their methods to improve the quality of compatibility coefficients for the constraints used by a relaxation labelling algorithm.

The model used is the following: each word in the text is a variable and may take several labels, which are its POS tags.

Since the number of variables and word po­sition will vary from one sentence to another, constraints are expressed in relative terms (e.g. [('fj, Determiner)(vu \, Adjective)(vi i-2, Noun)]).

The Constraint Set

Relaxation labelling is able; to deal with con­straints between any subset of variable».

Any relationship between any subset of words and tags may be expressed as constraint and used to feed the algorithm. So, linguists are free to ex­press any kind of constraint and are not restricted to previously decided patterns like in (Brill 92).

Constraints for subsets of two and three vari­ables are automatically acquired, and any other subsets are left to the linguists' criterion. That is, we are establishing two classes of constraints: the automatically acquired, and the manually writ­ten. This means that we have a great model flex­ibility: we can choose among a completely hand written model, where a linguist has written all the constraints, a completely automatically de­rived model, or any intermediate! combination of constraints from each type.

Wc can use the same: information than HMM taggers to obtain automatic constraints: the probability, of transition from one; tag to an­other- (bigrarn -or binary constraint- probability) will give; us an ielea of hew compatible they are in the positions i and i +1, and the same for trigram -or ternary constraint- probabilities. Extending this to higher order constraints is possible;, but would result in prohibitive computational costs.

Dealing with hand-written constraints will not be so easy, since it is not obvious how to com­pute "transition probabilities" for a e:oinple;x ein­strahlt.

Although accurate! -but costly- methods to esti­mate e;e>mpatibility valuers have' beeui proposeiel in (Pclillo & Refice 94), we; will choose a simpler and rnue:h e:heaper computationally solution: Comput­ing the compatibility degree; for the manually writ­ten constraints using the number of occurrences of the constraint pattern in the training corpus to compute the probability of the restricted word-tag pair given the context deline;d by the constraint .

Relaxation doesn't nee;d -as HMMs do- the prior probability of a e:ertain tag for a word, since it is not a constraint, but it can be; used to set the; initial state; to a not completely random one. Ini­tially we; will assign to each word its most probable tag, so we start optimization in a biassed point.

Alternative. Support Functions

The; support functions described in section 2 arc; traditionally used in relaxation algorithms, ft seems better for our purpose; to choose an addi­tive one, since the; multiplicative functions might yie;ld ze;ro or tiny value;s when -as in our case;- for a certain variable; or tag no constraints are available; for a given subset of variables.

Since that functions are gemeral, we may try to find a support function more specific: for our prob­lem. Sinex; HMMs find the; maximum seque!ne:e; probability and relaxation is a maximizing algo­rithm, we; e;an make; relaxation maximize the; se~ epience probability and we should get the same results. To achieve this we define a new support function, which is the sequence probability:

if* the tag for variable; Vk with highest weight value at the current time; step.

-/(•(vi,/;) the probability for the seeiuence to start in tag t.

P(vt) the le;xie:al probability for the; word repre­sented by v to have tag /;.

Y'(tj,/,2) the probability of tag i2 givem that the previous one; is t\.

Rfj the set of all ternary constraints on tag j for word i.

Iljj the set of all hand-written constraints on tag j tor word i.

To this extent, we have presented the relaxation labelling algorithm family, and stated some con­siderations to apply them to POS tagging.

In this section we will describe the experiments performed on applying this technique to our par­ticular problem.

Our experiments will consist of tagging a corpus with all logical combinations of the following pa­rameters: Support function, Updating function, Compatibility values, Normalization function and Constraints degree, which can be binary, ternary, or hand-written constraints, we will experiment with any combination of them, as well as with a particular combination consisting of a back-off technique described below.

In order to have a comparison reference we will evaluate the performance of two taggers: A blind most-likely-tag tagger and a HMM tagger (Elwor-thy 93) performing Viterbi algorithm . The train­ing and test corpora will be the same for all tag­gers.

All results are given as precision percentages over ambiguous words.

We can apply the same; algorithm to the task of disambiguating the sense of a word in a certain context. All we need is to state the constraints between senses of neighbour words. We can com­bine this task with POS tagging, since there are also constraints between the POS tag of a word and its sense;, or the sense of a neighbour word.

We have applied relaxation labelling algorithm to the task of POS tagging. Results obtained show that the algorithm not only can equal markovian taggers, but also outperform them when given enough constraints or a good enough model.

The main advantages of relaxation over Marko­vian taggers are the following: First of all, relax­ation can deal with more information (constraints of any degree), secondly, we can decide whether we want to use only automatically acquired con­straints, only linguist-written constraints, or any combination of both, and third, we can tune the model (adding or changing constraints or compat­ibility coefficients).

We can state that in all experiments, the re­finement of the model with hand written con­straints led to an improvement in performance. We improved performance adding few constraints which were not linguistically motivated. Probably adding more "linguistic" constraints would yield more significant improvements.

Several parametrizations for relaxation have been tested, and results seem to indicate that:

• support function (1.2) produces clearly worse results than the others. Support function (1.1) is slightly ahead (1.3).

• using mutual information as compatibility values gives better results.

• waiting for convergence is not a good policy, and so alternative stopping criterions must be studied.

• the back-off tedhnique, as well as the trigram model, requires a really big training corpus.

The experiments reported and the conclusions stated in this paper seem to provide a solid back­ground for further work. We intend to follow sev­eral lines of research:

• Applying relaxation to WSD and to WSD plus POS-tagging.

• Experiment with different stopping criteri­ons.

• Consider automatically extracted constraints (Màrqucz & Rodriguez 95).

• Investigate alternative ways to compute compatibility degrees for hand-written con­straints.

• Study back-off techniques that take into ac­count all classes and degrees of constraints.

• Experiment stochastic relaxation (Simulated annealing).

• Compare with other optimization or con­straint satisfaction techniques applied to NLP tasks.

I thank Horacio Rodfguez for his help, support and valuable comments on this paper. I also thank Kiku Ribas, German Rigau and Pedro Meseguer for their interesting suggestions.