<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="973"/><title>Cross Lingual Information Access System for Indian Languages</title><author surname="Consortium" givenname="CLIA"><org  name="Ministry of Communications &amp; Information Technology" country="India" city="Government of"/></author></firstpageheader><frontmatter><p><b>Cross Lingual Information Access System for Indian Languages</b></p><p><b>CLIA Consortium</b></p><p>A Consortium of 11 Institutions as the Implementing Agency for the Project "Development of Cross Lingual Information Access (CLIA) System" funded by Gov­ernment of India, Ministry of Communications &amp; Information Technology, Department of In­formation Technology (No. 14(5)/2006 - HCC (TDIL) Dated 29-08-2006)</p><p><b>1 Introduction</b></p><p>The <b>CLIA (Cross Lingual Information Access) Project </b>is a mission mode project funded by Gov­ernment of India, Ministry of Communications &amp; Information Technology, Department of Informa­tion Technology vide its approval No. 14(5)/2006 - HCC (TDIL), Dated 29-08-2006. It is being exe­cuted by a consortium of 11 academic and research institutions and industry partners, IIT Bombay, IIT Kharagpur, IIIT Hyderabad, AU-KBC Chennai, AU-CEG Chennai, ISI Kolkata, Jadavpur Univer­sity Kolkata, C-DAC Pune, C-DAC Noida, Utkal University Bhubaneswar and STDC, DIT New Delhi. The final deliverables of the project at the end of two years will be a portal where:</p><p>• A user will be able to give a query in one Indian language and</p><p>• S/he will be able to access documents available in</p><p>(a) . the language of the query,</p><p>(b) . Hindi (if the query language is not Hindi),</p><p>and</p><p>(c) . English</p></frontmatter><abstract>• Results will be presented to the user in the lan­guage of the query. The results can also be pre­sented in the language in which the information originally resided. The languages involved are <i>Bengali, Hindi, Marathi, Punjabi, Tamil </i>and <i>Te-lugu.</i> </abstract></header><body><section number="2" title="Motivation"><p>With the tremendous growth of digital and online information repositories new opportunities and new problems are created for achieving information retrieval across different languages. Online documents are available internationally in many different languages. Cross Lingual Information Access (CLIA) systems makes it possible for users to directly access sources of information which may be available in languages other than the lan­guage of query. However in conventional informa­tion retrieval systems the user must enter a search query in the language of the documents in order to retrieve it. This requires that the user can formulate his/her queries in all possible languages and can decipher documents returned by the retrieval proc­ess. This restriction clearly limits the amount and type of information, which an individual user really has access to.</p><p>Cross-language information retrieval enables users to enter queries in languages they are familiar to, and uses language translation methods to re­trieve documents originally created in other lan­guages. Cross-Language Information Access is an extension of the Cross-Language Information Re­trieval paradigm. Users who are unfamiliar with the language of documents retrieved are often un­able to obtain relevant information from these documents. The objective of Cross-Language In­formation Access is to introduce additional post retrieval processing to enable users make sense of these retrieved documents. This additional process­ing may take the form of machine translation of snippets, summarization and subsequent translation of summaries and/or information extraction.</p><p>There have been efforts globally towards devel­opment of such systems. Cross-Language Evalua­tion Forum (CLEF), NTCIR Asian Language Re­trieval, Question-answering Workshop and others have been working towards achieving the similar goals. In Indian context the need of such system becomes more evident that being multi-lingual country, the people here are familiar with more than one language. The availability of such system helps in reaching the information if it is available in language other than the language of query.<page local="2" global="974"/> In order to meet this requirement, the CLIA (Cross Lingual Information Access) project has been initi­ated.</p></section><section number="3" title="System Features"><p>The system is intended to search different docu­ments in Indian languages. Once the user starts the system, an initial screen with logo is displayed. By default, the screen is displayed in Hindi or English depending on the default language selected on the browser of the user. If the user wants to display this initial screen in any other language, he/she can select the language from the bottom of the screen. The screen is then displayed in the selected lan­guage. At present, the screen is available in seven languages: Hindi, English, Marathi, Punjabi, Ben­gali, Tamil and Telugu. To search a document, the first activity the user performs is the selection of the source language. Selection of source language allows the user to enter the text in the selected lan­guage.</p><p>•Selection of the Source Language: The user can select the source language by clicking a drop-down box. The system displays the languages available to select the source language.</p><p>•Entering String for Search: The user enters the query string on which the search is to be made in the appropriate place. The system allows the user to enter the string in the source language selected by the user using a soft keyboard for the language.</p><p>•Search the Web or the Site: Once the string is en­tered, the user should select whether to search the local site or the World Wide Web. The user can then click the search option to search the site for the string entered.</p><p>• Displaying the Results: Once the query is prop­erly expanded and translated, it is used to search the web or the local site and the documents are re­trieved according to the query. The snippets of the retrieved documents are displayed in the original language of the document as well as in the source language selected by the user. Thus, if the source language selected is Bengali, the user can enter query string in Bengali, the CLIA system searches</p><p>for documents in English, Hindi and Bengali either from the web or the local site. The snippets of the retrieved documents are displayed in English/ Hindi and Bengali.</p><p>•Advanced Search: The user can also select the advanced search option and the CLIA system dis­plays all the options accordingly. The user can se­lect here the domain for which he/she wants to search the documents. At present, the tourism and health domains are available. The user can also select the number of items to be displayed on a single page. By default, the system displays 10 items on a single page. Once the selection is made, the user can click the 'search' option to start the search. In the advanced search option, the CLIA system provides summary as well as extracted in­formation in the form of predefined information extraction templates, of the retrieved documents along with the generated snippet. The summary and the extracted information templates can be displayed in the original language of the document as well as in the source language selected by the user.</p></section><section number="4" title="Technical Details"><p>The CLIA system achieves its performance by means of the following five subsystems:</p><p><b>• </b>Input processing</p><doubt alpha="85.7" length="7" tooSmall="False" monospace="0.0">•Search</doubt><p><b>• </b>Processing of Retrieved Documents</p><p><b>• </b>Output Generation</p><p><b>• </b>UNL Based Search</p><p>The main purpose of each of these subsystems is described below:</p><p><b>• Input Processing Subsystem</b></p><p>Input processing analyses the query entered by the user using language processing tools, expands the query to add more relevant terms and based on its analyses, either translates or transliterates all the query terms to the target language and then pro­vides this as input to the search modules. The CLIA Input Processing subsystem consists of Lan­guage Analyzer (Tokenization, Named Entity Rec­ognition, Multiword Expression, Stop word identi­fication, Stemmer), Query Expansion (Pre- and Post-Translation Query Expansion), Query Translation and Query Transliteration.<page local="3" global="975"/> The CLIA Focus Crawler subsystem consists of Classifier, Lan­guage identifier and the Parallel crawler.</p><p><b>• Search Subsystem</b></p><p>The search subsystem lies at the heart of the CLIA-search engine. The main purpose of this module is (a) . Crawl the web and download files for a spe cific language and domain.</p><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">to:</doubt><p>(b) . Extract the text part in these documents and perform certain processing on those texts and con­vert them into indices.</p><p>(c) . Extract results for a particular query by look­ing up the indices built so far.</p><p>(d) . Arrange the document references returned by the search subsystem according to some order de­pending upon page ranking.</p><p><b>• Document Processing Subsystem</b></p><p>The document-processing module facilitates the access of documents written in English, Hindi and in the other languages. The documents crawled from the web are preprocessed using language processing tools for extracting information and translating the extracted information into target languages. This module consists of many lan­guage-processing tools such as Document Con­verter, Language Pre-processors, POS taggers, Text Chunker, Named Entity Recognizer, Domain Dictionaries, Information Extraction Engine and Translation engine. These modules are used in processing the documents.</p><p><b>• Output Generation Subsystem</b></p><p>This subsystem consists of the snippet generation, summary generation and the snippet translation modules. Brief details of the modules are described</p><doubt alpha="83.3" length="6" tooSmall="False" monospace="0.0">below:</doubt><p>(a). Snippet Generation: The Snippet Generation Module generates the snippet corresponding to the retrieved document. This module gets the parse text of the retrieved documents and the query from the search engine and generates the Snippet of each document and returns the generated snippet on the output screen.</p><p>(b)<b>. </b>Summary Generation: The Summary Genera­tion module generates the summary corresponding to the retrieved document. This module gets the parsed text of the retrieved documents and the query from the search engine and generates the summary of the documents.</p><p>(c) <b>. </b>Snippet and Summary Translation: Generated snippets for English and Hindi documents are translated to the query language. If the query lan­guage is Hindi, then English documents are trans­lated to Hindi. Translated snippet in the query lan­guage is displayed on the output screen along with the original snippet.</p><p><b>• UNL-Based Search Subsystem</b></p><p>The advanced search system uses UNL as a lan­guage independent intermediate representation to enable translation between the languages. The ad­vanced search using UNL is based on concepts, and relations between concepts rather than bag of words. Hence it enables semantic search. Although the current system is designed for Tamil, it can be extended to other languages.</p></section><section number="5" title="Future Roadmap"><p>The functionalities of the CLIA system have been currently developed for Bengali, Marathi and Te-lugu. The search option has been limited to the crawled documents that are stored and indexed in the CLIA server. The crawled documents are in the tourism domain. At present, the user can provide 3­4 word queries to the CLIA system using soft key­boards for the respective language. The output of the system shows only the snippets in the original language of the document.</p><p>The CLIA system is being enhanced to provide full functionalities in the other Indian languages, i.e., Hindi, Tamil and Punjabi. The search option is expanded to provide search facility on the web also. Work is also going on for providing CLIA functionalities in the health domain. In future, snippet translation, summary generation and trans­lation as well as information extraction templates generation and translation are going to be included in the CLIA system. The evaluation engine will judge the CLIA system based on the ranks of the relevant documents in the list of documents re­trieved by the system.</p></section></body></article>