We use the existing TRIPS parser (Allen et al, 2008) to produce deep logical forms from text. The system is generic and no grammatical rules or lexical entries were added specifically for this task. The TRIPS grammar is lexicalized context-free grammar, augmented with feature structures and feature unification. The grammar is moti­vated from X-bar theory, and draws on principles from GPSG (e.g., head and foot features) and HPSG. The parser uses a packed-forest chart rep­resentation and builds constituents bottom-up using a best-first search strategy similar to A*, based on rule and lexical weights and the influ­ences of statistical preprocessing. The search terminates when a pre-specified number of span­ning constituents have been found or a pre-specified maximum chart size is reached. The chart is then searched using a dynamic pro­gramming algorithm to find the least cost se­quence of logical forms according to a cost table that can be varied by genre.

The TRIPS system here uses a wide range of statistically driven preprocessing, including part of speech tagging, constituent bracketing, inter­pretation of unknown words using WordNet, and named-entity recognition (Allen et al, 2008). All these are generic off-the-shelf resources that ex­tend and help guide the deep parsing process.

The TRIPS LF (logical form) ontology is de­signed to be linguistically motivated and domain independent. The semantic types and selectional restrictions are driven by linguistic considera­tions rather than requirements from reasoning components in the system (Dzikovska et al. 2003). As much as possible the semantic types in the LF ontology are compatible with types found in FrameNet (Johnson & Fillmore 2000). Fra­meNet generally provides a good level of ab­straction for applications since the frames are derived from corpus examples and can be relia­bly distinguished by human annotators. However TRIPS parser uses a smaller, more general set of semantic roles for linking the syntactic and se­mantic arguments rather than FrameNet's exten­sive set of specialized frame elements. The LF ontology defines approximately 2500 semantic types and 30 semantic roles. The TRIPS parser will produce LF representations in terms of this linguistically motivated ontology.

As an example, the result of parsing the sen­tence, He fought in the war, is expressed as set of expressions in an unscoped logical formalism with reified events and semantic roles.

The main event (V2) is of ontology type fight­ing, which is a subclass of intentional-action, and which corresponds to the first WordNet sense of fight, and includes verbs such as fight, defend, contend and struggle. The agent role of

research/trips/lexicon/browse-ont-lex.html

this event is the referent of the pronoun he, and the event is situated in an event described by the word war. For words not in the TRIPS core lexi­con, the system looks up the WordNet senses and maps them to the TRIPS ontology. The word war is not in the core lexicon, and via WordNet is classified into the TRIPS ontology as the abstract type action.

One of the statistical relational learning (SRL) frameworks that has recently gained momentum as a platform for global learning and inference in AI is Markov Logic (Richardson and Domingos, 2006). Markov logic is a combination of first order logic and Markov networks. It can be seen as a formalism that extends first-order logic to allow formulae to be violated with some penalty.

For our different classification tasks, we used different classifiers based on MLNs. We used an off-the-shelf MLN classifier Markov thebeast, using Cutting Plane Inference (Riedel, 2008) with an Integer Linear Programming (ILP) solver for inference.

To use thebeast or any other MLN framework, at first we have to write the formulas, which cor­responds to defining features for other machine learning approaches. The Markov network then learns the weights for these formulas from the training corpus and uses these weights for infer­ence in testing phase.

One easy example will give a brief idea about these weights. To classify the event feature class, we have a formula that captures influence of both tense and aspect together. Here are two examples that show the learned weights for the formula from training data.

The MLN then uses these weights for reason­ing about the class. Generally, larger the weights are, the more likely the formula holds. These weights could be negative as well, i.e. the formu­las are most likely not to hold.

Finding useful features for MLNs is the same as any other ML algorithm. However, the MLN framework gives the opportunity to capture the relations between different features in first order logic, which can lead to better inference. For ex­ample, when filtering events, we have formula combining word and pos, or word and previous word, or pos and next pos, where we can capture relationship of two predicates together. Many of these predicates (features) could be encoded in other classifiers concatenating the features. But when the size of relations between features in­creases it complicates matters and we have to regenerate the whole classifier data, every time we introduce a new relationship.

For event extraction, we parse the raw text with the TRIPS parser. Then we take the resulting Logical Form (LF) and apply around hundred of hand-coded extraction patterns to extract events and features, by matching semantic patterns of phrases. These hand-coded rules are devised by checking the parse output in our development set. It was 2-3 weeks of work to come up with most of the extraction rules that extract the events. There were minor incremental improve­ments in rules afterwards. It is worth mentioning, these rules are very generic and can be used in a new domain without any extra work, because, the TRIPS parser and ontology are domain inde­pendent, and use mappings from WordNet to interpret unknown words. Hence, the extraction rules will apply (and can be tested) for any natu­ral language text without any extra work.

Because of the ontology, we can usually express general rules that capture a wide range of phenomena. For instance, all noun-phrases describing objects that fall under the TRIPS Ontology's top-level type situation-root are extracted as described events. This situation is captured by the extraction rule:

Since war has the type action, which falls under situation-root in TRIPS ontology, this extraction rule will match the LF (THE V5 (:* action war)) and will extract war as event. Beside matching war under situation-root in ontology, it also matches the specifier the, which indicates that it is a definite noun phrase.

The result of matching around hundred of such rules to the sentence above is:

<EVENT eid=V2 word=FIGHT pos=VERBAL ont-type=FIGHTING class=OCCURRENCE tense=PAST voice=ACTIVE aspect=NONE polarity=POSITIVE

nf-morph=NONE> <RLINK eventInstanceID=V2 ref-word=HE ref-ont-type=PERSON

relType=AGENT> <SLINK signal=IN eventInstanceID=V2 subordinatedEventInstance=V5 relType=SITUATED-IN> <EVENT eid=V5 word=WAR pos=NOUN ont-type=ACTION class=OCCURRENCE voice=ACTIVE polarity=POSITIVE aspect=NONE tense=NONE>

In this way, we extract events and TimeML-suggested event features (class, tense, aspect, pos, polarity, modality). We also extract a few additional features such as ontology type (ont-type). TimeML tries to capture event information by very coarse-grained event features class or pos. The ontology type feature captures more fine-grained information about the event, but still coarse-grained than the words. The extraction rules also map our fine-grained types to the coarse-grained TimeML event class. We also extract relations between events (SLINK), when­ever one event syntactically dominates the other, so it extracts more than TimeML's SLINKs and another new relation, relation between event and its arguments (RLINK). Details about these new additions can be found in UzZaman and Allen (2010).

This system extracts events from the Tem-pEval-2 corpus with high recall. However, this high performance comes with the expense of precision. The reasons for lower precision in­clude, (i) the fact that generic events are not coded as events in TempEval, (ii) errors in pars­ing and, (iii) legitimate events found by the parser but missed by TempEval annotators. To remedy this problem, we introduced a MLN based filtering classifier, using the event features extracted from the TRIPS parser. The formulas in MLN for filtering were derived by linguistic intuition and by checking the errors in our devel­opment set. We devised around 30 formulas.

There were two goals for this filtering step: (1) Eliminating events that result from errors in the parse, and (2) Removing event-classes, such as generics, that were not coded in TempEval.

The second goal is needed to perform a meaningful evaluation on the TempEval challenge. For our long-term goal of using the

For our long-term goal of using the temporal summary in natural language understanding task, however, we would retain these other events. The resulting system, including parsing, extrac­tion, and post-filtering, is named as TRIOS sys­tem.

The TRIPS parser and extraction rules already give us event features along with events, which is reported in the results as the TRIPS system. To improve performance, we implemented MLN classifiers (TRIOS system) for the class, tense, aspect and pos features, using the features gener­ated from the TRIPS parser plus lexical and syn­tactical features generated from the text using the Stanford POS tagger. The TRIOS system re­ports the polarity and modality performance of TRIPS system, i.e. we don't have any extra clas­sifiers to classify those features in TRIOS sys­tem. The Table 1 gives a summary of features used to classify these event features.

The TRIPS parser extracts temporal expres­sions the same way as we extract events. The

One Penn tag derived features is verb word sequence, which captures all previous verbs, or TO (infinitive), or modal verbs, of the event word. That is, it will capture all consecutive verbs before the event until we get non­verbal word. Similarly verb pos sequence is the penn tag sequence of these verbs.

performance of TRIPS parser's temporal extrac­tion doesn't outperform state-of-the-art tech­niques on the evaluation measures. To improve on this, we also use a traditional machine learn­ing classifier straight from the text. We used a token-by-token classification for temporal ex­pressions represented by B-I-O encoding with a set of lexical and syntactic features, using Condi­tional Random Field (CRF) classifier. CRF is widely used for labeling and segmenting se­quence data. Unlike Hidden Markov Models, CRFs are based on exponential models in which probabilities are computed based on the values of a set of features induced from both the observa­tion and label sequences. They have been used in POS tagging, shallow parsing, named entity rec­ognition and also for temporal expression extrac­tion in TERN dataset [Ahn et al. (2005), Hacio-glu et al. (2005) and Poveda et al. (2007)].

We used lexical features like word, shape, is year, is date of week, is month, is number, is time, is day, is quarter, is punctuation, if belong to word-list like init-list, follow-list, etc. We then use CRF++ templates to capture relation between different features to extract the se­quence. For example, we will write a template to capture the current word is in init-list and the next word is in follow-list, this rule will train the model to capture sequences like this weekend, earlier morning, several years, etc.

On the other hand, the TRIPS parser does ex­tract temporal relations between events and tem­poral expressions, which helps us in the temporal relation identification tasks. So we take the tem­poral expressions from the CRF based extractor and for the cases where TRIPS parser extracts the temporal expression, we use TRIPS parser id, so that we can relate to relations generated by the parser.

The temporal expressions that are suggested by CRF based system, are passed to a filtering step that tries to extract a normalized value and type of the temporal expression. If we can find a normalized value and type, we accept the temporal expressions. We reported this CRF based sys­tem with filtering as both TRIPS and TRIOS systems.

3.2 Determining The Normalized Value and Type of temporal expression

Temporal expressions are most useful for later processing when a normalized value and type is determined. We implemented a rule-based tech­nique to determine the types and values. We match regular expressions to identify the type of temporal expressions. Type could be either of time, date, duration and set.

Then in next step we extract the normalized value of temporal expression, as suggested by TimeML scheme. We take the Document Crea­tion Time (DCT) and then calculate the values for different dates in terms of document creation date, e.g. last month, Sunday, today. We will make our type and value extractor and temporal expression extractor modules available for pub­lic use.

On event extraction, the TRIPS system has the highest recall, while the TRIOS system is second best in precision with the highest scoring system, TIPSem. But overall they do very well compared to our system on event extraction. Performance of our both systems and the best performing TIPSem system is reported in Table 4.

On event feature extraction, our TRIOS sys­tem, which is based on MLN based classifiers, has the best performance on aspect and polarity; we also do very well (second-best performances mostly) on tense, class, pos and modality.

Feature TRIPS

Both the TRIPS and TRIOS systems use the same CRF-based approach for temporal expres­sion extraction. Our system has the second best performance on combined temporal expression extraction and normalization task (identifying type and value). It is worth mentioning that the average of identifying value performance is 0.61 and if we remove our systems and the best sys­tem, HeidelTime-1, the average is only 0.56. Hence, our freely distributed normalization tool could be beneficiary to many people. Perform­ance of our system and the best system on task A is reported in Table 5.

For temporal relation identification (Task C - F), other teams used events, temporal expressions and their features from human-annotated corpus, whereas, we used our extracted entities and their features that we extracted in Task A and B. So our performances represent the capability of identifying these relationships from raw text and it is also harder classification task, since we are starting with imperfect features.

Even though we are using our own generated features, we outperformed other groups in task C (temporal relation between event and temporal expression) and task E (temporal relation be­tween main events of consecutive sentences). We also have second-best/equivalent performance for other two tasks (temporal relation between event and DCT; and temporal relation between events, where one syntactically dominates other).

Many teams chose just to attempt one task be­tween task A and B, or both, or only attempt some of tasks C to F. Only three teams attempted all tasks, our team, TIPSem and JU_CSE_TEMP. For tasks C - F, we used our generated features, whereas all other teams used the features provided in the corpus.

In this section, we will show head-to-head comparison of the performance of these three systems to see which team handles the overall challenge of TempEval-2 better. Table 6 sum­marizes our analysis.

Table 6: Head-to-head comparison of TRIOS, TIP­Sem and JU_CSE_TEMP (teams that approached all tasks) in TempEval-2 challenge

Note that JU_CSE_TEMP didn't perform best in any particular task. However, they do a little temporal expression type and value.

Both TRIOS and TIPSem teams submitted two systems. For this comparison, we pick the best system of each team then compare between them. On temporal expression extraction, we have very close extraction scores (TRIOS fscore 0.85 and TIPSem fscore 0.855). However on temporal expression attributes, we are far supe­rior to TIP-Sem. So overall in Task A, we claim we did better. TIPSem clearly did better on the event extraction task.

On the other hand, given that task A and task B has many subtasks, if we split them into entity extraction and attribute extraction, then we have four tasks of extraction and four tasks on relation identification. In that case, TIPSem does better than us on event extraction, but on event feature extraction we have a tie; for temporal expression extraction, we have another tie, but we outper­form in temporal expression attribute extraction.