<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="49"/><title>The Link Structure of Language Communities and its Implication for Language-specific Crawling</title><pubinfo>The 6th Workshop on Asian Languae Resources, 2008</pubinfo><author surname="Mikami" givenname="Yoshiki"><org  name="Nagaoka University of Technology" country="Japan" city="Nagaoka City"/></author><author surname="Caminero" givenname="Rizza"><org  name="Nagaoka University of Technology" country="Japan" city="Nagaoka City"/></author></firstpageheader><frontmatter><p><b>The Link Structure of Language Communities and its Implication for</b></p><p><b>Language-specific Crawling</b></p><p><b>Rizza Camus Caminero</b></p><p>Language Observatory Nagaoka University of Technology Nagaoka, Niigata, Japan</p><p>rhyze.caminero@gmail.com</p><p><b>Yoshiki Mikami</b></p><p>mikami@kj s.nagaokaut.ac.jp</p></frontmatter><abstract>Since its inception, the World Wide Web (WWW) has exponentially grown to shelter billions of monolingual and multilingual web pages that can be navigated through hyperlinks. Its structural properties provide useful information in presenting the socio-linguistic properties of the web. In this study, about 26 million web pages under the South East Asian country code top-level-domains (ccTLDs) are analyzed, sev­eral language communities are identified, and the graph structure of these communi­ties are analyzed. The distance between language communities are calculated by a distance metrics based on the number of outgoing links between web pages. Inter­mediary languages are identified by graph analysis. By creating a language subgraph, the size and diameter of its strongly-connected components are derived, as these values are useful parameters for language-specific crawling. Performing a link struc­ture analysis of the web pages can be a use­ful tool for socio-linguistic and technical research purposes. </abstract></header><body><section number="1" title="Introduction"><p>The World Wide Web contains interlinked docu­ments, called web pages, navigable by hyperlinks. Since its creation, it has grown to contain billions of web pages and several billion hyperlinks. These web pages are created by millions of people from all parts of the world. Each web page contains a large amount of information that can be shared and disseminated to people with Internet access. Au­thors and creators of a web page come from differ­ent backgrounds, different cultures, and different languages. Thus, a web page is a resource of multi­lingual content, and a fertile source for socio-linguistic analysis.</p><subsection number="1.1" title="Web Crawling"><p>While search-engines are important means of ac­cessing the web for most users, automated systems of retrieving information from the web have been developed. These systems are called web crawlers, where a software agent is given a list of pages to visit. As the crawler visits these pages, it follows their outgoing links and adds these to the list of pages to visit. Each page is visited recursively ac­cording to some sets of policies (e.g. the type of pages to retrieve, direction, the maximum depth from the URL (uniform resource locator), etc.). The result of this crawl is a vast amount of data, most of which may be irrelevant to certain indi­viduals. Thus, a focused-crawling approach was implemented by some systems to limit the search to only a subset of the web.</p><p>Focused crawlers rely on classifiers to work ef­fectively. Language-specific crawlers, for example, need a very good language identification module that properly identifies the language of the web pages. General crawlers can be extended to include focused-crawling capabilities by incorporating the classifiers. Another important requirement to effi­ciently crawl the desired domain is the list of initial web pages, called seed URLs. Each of these URLs will be enqueued into a list. The agent visits each URL in the list. Since the crawler recursively visits<page local="2" global="50"/></p><p>The 6th Workshop on Asian Languae Resources, 2008</p><p>the outgoing links of each URL, it is possible that the seed URL is an outgoing link of another seed URL, or an outgoing link of an outgoing link of another seed URL. Listing these URLs as seed URLs will just waste the crawler's time in visiting them, when they have already been visited. If sev­eral URLs can be reached from just one URL, the seed URL list size will decrease and the crawler will be more efficient. However, the maximum distance between these web pages must also be considered, since the crawler can have a policy to stop the crawling after reaching a certain depth.</p></subsection><subsection number="1.2" title="The Web as Graph"><p>A graph consists of a set of nodes, denoted by <i>V </i>and a set of edges, denoted by E. Each edge is a pair of nodes <i>(u, v) </i>representing a connection be­tween <i>u </i>and v. A path between two nodes is a se­quence of edges that is passed through from one node to reach the other node.</p><p>In a directed graph, each edge is an ordered pair of nodes. A path from <i>u </i>to <i>v </i>does not imply a path from <i>v </i>to <i>u. </i>The distance between any two nodes is the number of edges in a shortest path connecting them. The diameter of the graph is the maximum distance between any two nodes.</p><p>In an undirected graph, each edge is an unor­dered pair of nodes. There is an edge between <i>u </i>and <i>v </i>if there is a link between <i>u </i>and <i>v, </i>regardless of which node is the source of the link.</p><p>A strongly-connected component <i>(SCC) </i>of a di­rected graph is a set of nodes such that for any pair of nodes <i>u </i>and <i>v </i>in the set, there is a path from <i>u </i>to <i>v. </i>The strongly-connected components of a graph consist of disjoint sets of nodes. A <i>subgraph </i>is a graph whose nodes and edges are subsets of an­other graph.</p><p>With the interlinked nature of the web, it can be represented as a graph, with the web pages as the nodes and the edges as the hyperlinks between the pages.</p></subsection><subsection number="1.3" title="Languages"><p>Languages are expressions of individuals and groups of individuals, essential to all form of communication. It is a fundamental medium of expressing one's self whether in spoken or written form. Ehtnologue (2005) lists 6,912 known living languages in the world. Only a small portion of these languages can be found in the web today.</p><p>A language community in the web is the group of web pages written in a language. Major lan­guage communities discovered in each country indicates the dominant language of the country's web space. How one language community is re­lated to another language community can be shown by analyzing the hyperlinks between them. Thus, a language graph can be created with the language communities as nodes, and the links between the language communities as edges.</p></subsection></section><section number="2" title="Previous Studies"><p>One of the earliest web survey in Asia (Ciolek, 1998) presented statistical data of the Asian web space by using the Altavista WWW search engine in gathering its data. In 2001, he wrote a paper pre­senting the trends in the volume of hyperlinks con­necting websites in 10 Eash Asian countries.</p><p>Several studies have also been done regarding the representation of the web as a graph. Kumar et al. (2000) showed that a graph can be induced by the hyperlinks between pages. Measures on the connected component sizes and diameter were pre­sented to show the high-level structure of the web. Broder et al. (2000) did experiments on the web on a larger scale and showed the web's macroscopic structure consisting of the SCC, IN, OUT, and TENDRILS. Balakrishnan and Deo (2006) ob­served that the number of edges grow superlinearly with the number of nodes, showing the degree dis­tributions and diameter. Petricek et al. (2006) used web graph structural metrics to measure properties such as the distance between two random pages and interconnectedness of e-government websites. Bharat et al. (2001) studied the macro-structure of the web, showing the linkage between web sites by creating the "hostgraph", with the nodes represent­ing the hosts and the edges as the count of hyper­links between pages on the corresponding hosts.</p><p>Chakrabarti et al. (1999) proposed a new ap­proach to topic-specific web resource discovery by creating a focused crawler that selectively retrieves web pages relevant to a pre-defined set of topics. Stamatakis et al. (2003) created CROSSMARC, a focused web crawler that collects domain-specific web pages. Deligenti et al. (2000) presented a fo­cused crawling algorithm that builds a model for the context within which relevant pages to a topic occur on the web. Pingali et al. (2006) created an Indian search engine with a language identification<page local="3" global="51"/></p><p>The 6th Workshop on Asian Languae Resources, 2008</p><p>module that returns a language only if the number of words in a web page are above a given threshold value. The web pages were transliterated first into UTF-8 encoding. Tamura et al. (2007) presented a simulation study of language specific crawling and proposed a method for selectively collecting web pages written in a specific language by doing a linguistic graph analysis of real web data, and then transforming them into variation of link selection strategies.</p><p>Despite several studies on web graph and lan­guage-specific crawling, no study has been done showing the "language graph". Herring et al. (2007) showed a study on language networks of a selected web community, LiveJournal, but not on the web as a whole.</p></section><section number="3" title="Scope and Objectives of the Study"><p>This research was conducted on the 10 ccTLDs of the South East Asian countries. This paper, how­ever, will only show the results for the Indonesian domain (.id).</p><p>This research aims to show the socio-linguistic properties of the language communities in each country at the macroscopic level. The web page distribution for each language community in a given ccTLD and its most frequently linked to lan­guages are shown. The distance is also computed and the language graph is illustrated.</p><p>This research also aims to show the graph prop­erties of some Filipino language communities and its implication for crawling. Graph properties like the SCC size and the diameter will be presented to show these characteristics in a subset of the web.</p><p>Finally, this research demonstrates the useful­ness of graph analysis approaches.</p></section><section number="4" title="Methodology"><p>This study was conducted by performing a series of steps from the collection of data to the presenta­tion of the results through images.</p><subsection number="4.1" title="UbiCrawler"><p>UbiCrawler (Boldi et al., 2004) was used to download the web pages under the Asian ccTLDs. These pages were downloaded primarily for the purpose of assessing the usage level of each language in cyberspace, one of the objectives of the Language Observatory Project (LOP)<footnote anchor="1"/>. The crawl was started on July 5, 2006, running for 14 days. 107,168,733 web pages were collected from 43 ccTLDs in Asia. Each page contains several information such as the character set and outgoing links. For this study, the URL of a web page and the URL of its outgoing links were used. Although there are many web pages of Asia that can be found in generic domains, they were not included in this survey to limit the volume of the crawl data.</p></subsection><subsection number="4.2" title="Language Identification"><p>A language identification module (LIM) developed for LOP based on the n-gram method (Suzuki et al., 2002) was used to identify on what language a page is written in. This method first creates byte sequences for each training text of a language. It then checks the byte sequences of the web pages that match the byte sequences of the training texts. The language having the highest matching rate is considered as the language of the web page. The language identification module used the parallel corpus of the Universal Declaration of Human Rights (UDHR), translated into several languages. After crawling, LIM was executed to identify the languages of each downloaded web page. The identification result was stored in a LIM result file that contains the URL, the language, and matching rate, among others. In this study, the issues regarding the accuracy of LIM will not be discussed.</p></subsection><subsection number="4.3" title="Web Page Analysis"><p>For this study, the web pages of the 10 South East Asian ccTLDs were selected for analysis. There were 26,196,823 web pages downloaded under these ccTLDs. The web pages for each country were grouped by languages. The list of languages was narrowed down to 20 based on the number of pages, arranged from highest to lowest.</p><p>The link structure can be analyzed by traversing the outgoing links of each web page. For each web page, its outgoing links are retrieved. For each out­going link, the LIM result file is checked for its language. The number of outgoing links in each language is counted. If the URL of the outgoing link is not on the file, it wasn't downloaded. There­fore, the language of the ougoing link is unidenti­fied. This is usually the case of outgoing links un-<page local="4" global="52"/></p><footnote label="1"> http://www.language-observatory.org</footnote><p>The 6th Workshop on Asian Languae Resources, 2008</p><p>der the generic TLDs (e.g., .com, .org, .gov, etc.) and non-Asian ccTLDs.</p></subsection><subsection number="4.4" title="Language Graph"><p>There is a link between two languages if there is at least one outgoing link from a web page in one language to the other language. The language graph is created through contraction procedure, where all edges linking the same language page are contracted.</p></subsection><subsection number="4.5" title="Language Adjacency Matrix"><p>Based on the number of web pages in a language and the number of outgoing links from one lan­guage to another, the language adjacency table <i>N </i>for each country is created. The row and column headers are the same - the top 20 languages based on the number of web pages. The value <i>N</i><i>j</i><i> </i>is the number of outgoing links from language <i>i </i>to lan­guage <i>j.</i></p><p>The language adjacency matrix <i>P </i>contains the ratio of the number of outgoing links and the total number of outgoing links as can be found in the language adjacency table. Each cell value, Pj is the probability that a web page in a language <i>i </i>has an outgoing link to language <i>j.</i></p><p><i>Pj </i><i>= </i><i>Nj </i>/ <i>^ </i><i>Nik</i></p><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">k</doubt><p>A link from language <i>i </i>to language <i>j </i>is not nec­essarily accompanied by a link from language <i>j </i>to i. Even if there is a link, the number of outgoing links is not equal. To show the relationship be­tween two languages based on the link structure, the language distance is computed.</p></subsection><subsection number="4.6" title="Distance between Languages"><p>The distance between two languages measures their level of connectedness. It is the relationship between the number of outgoing links from lan­guage <i>i </i>to language <i>j </i>and vice versa. The distance is computed as the ratio of the number of outgoing links between two languages and the total number of outgoing links of the two languages.</p><p>The distance between language <i>i </i>and language <i>j,</i> <i>D</i><i>j </i>is, Dij <i>= </i><i>J</i>l/(Rij <i>+a) </i><i>- </i><i>J3</i><i> </i>where <i>R</i><i>j</i><i> </i>is the lan­guage link ratio is defined as,</p><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">Vk</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">J</doubt><doubt alpha="60.0" length="5" tooSmall="False" monospace="0.0">Rj =l</doubt><doubt alpha="50.0" length="10" tooSmall="False" monospace="0.0">for(i = j)</doubt><p>where <i>a </i>is an adjusting parameter introduced to avoid division-by-zero, which may happen when <i>R</i><i>j</i><i>=0, </i>i.e. no links between two languages. We set a=0.0001. Thus, the maximum distance between languages becomes 99. <i>p </i>is another adjusting pa­rameter to make <i>D</i><i>j </i>as a distance metrics, and we set p=1. Assumption behind this definition is based on commonly-observed rules in our world. It is widely observed that interaction between two ob­jects is proportional to the inverse square of dis­tance between two objects. The number of web links between two language communities is con­sidered as a kind of interaction. Languages with no links between them have a distance of 99, and the distance of a language to itself is 0.</p><p>Based on this distance metrics, the macroscopic language graph is created. A distance limit of 15 is used to clearly show which languages are closely-related by their link structure.</p></subsection><subsection number="4.7" title="Intermediary Languages"><p>Considering the direction of the outgoing links, the possibility that language <i>i </i>will link to language <i>j </i>may be lesser than the possibility that language <i>i </i>will link to language <i>j </i>by passing through an in­termediary language k, such that <i>PikP</i><i>kj </i><i>&gt; P</i><i>j,</i><i> </i>The intermediary language is identified, as this would mean that there are better ways to reach another language from one language.</p></subsection><subsection number="4.8" title="Graph Analysis using JGraphT"><p>JGraphT is a free Java graph library that provides graph objects and algorithms. The library provides classes that calculates and returns the strongly-connected components subgraphs. To compute the distance between nodes, several graph searching algorithms are available, one of which is the Dijik-stra algorithm that computes for the shortest path. A utility to export the graph into a format readable by most graph visualization tools is also available. A graph file, written in the DOT language (a plain text graph description language) was created, con­taining the nodes and edges of language pages. From this, the strongly-connected components and its properties (i.e. size, diameter) were determined.</p></subsection><subsection number="4.9" title="GraphVis: Visualization Tool"><p>GraphViz is open-source graph visualization software that takes descriptions of graphs in a simple text language and makes diagrams in<page local="5" global="53"/></p><doubt alpha="57.9" length="19" tooSmall="False" monospace="0.0">Rj=(Nj+Np)/£Nik+^Nk</doubt><p>The 6th Workshop on Asian Languae Resources, 2008</p><p>several formats, including images. The neato layout, which makes "spring model" layouts, was used to visualize the distance between languages. However, the calculated distance cannot be drawn exactly, and this visualization is only two-dimensional. So, the images are distorted and do not illustrate the exact distance, only an approximation.</p></subsection></section><section number="5" title="Results"><p>This section shows some results on the Indonesian domain.</p><subsection number="5.1" title="Link Structure"><p>Indonesia is a country with one of the biggest lan­guage diversity in the world. According to Eth-nologue<footnote anchor="2"/>, 742 languages are spoken in the country. But, the LIM results show that only five of these languages are listed in the top 10 languages in the country, i.e., Javanese, Indonesian, Malay, Sun-danese, and Madurese.</p><p>Javanese is the most popular language in the web space of Indonesia, and constitutes 28% of the total number of web pages. It is followed by Eng­lish and Indonesian. Although Indonesian is an official language of the country, it is ranked third.</p><p>English, a major business language of Indonesia, has the second largest number of web pages in the domain. But, Indonesian occupies the second rank in the number of outgoing links.</p><p>The table above only shows the top 10 lan­guages. Among these, 8 languages are most fre­quently linked to the same language. The two other languages are Sundanese and Tatar, both mostly linked to Javanese.</p><p>The language graph below shows the languages as the nodes and the edges representing the dis­tance between languages. In the figure above, the six languages of Indonesia are found to be closely-connected to each other.</p><footnote label="2">Gordon, Raymond G., Jr. (ed.), 2005. Ethnologue: Lan­guages of the World, Fifteenth edition. Dallas, Tex.: SIL In­ternational.</footnote><footnote label="3">Luxembourgish</footnote><footnote label="4">Occitan Languidocien</footnote><table caption="Table 2. Language Link for Indonesian Domain" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>No.</p></td><td class="cell"><p>Language</p></td><td class="cell"><p>Languages Linked to (# of outgoing links)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>1</p></td><td class="cell"><p>Javanese</p></td><td class="cell"><p>Javanese (22,641,844)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(jav)</p></td><td class="cell"><p>Indonesian (3,761,205)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>2</p></td><td class="cell"><p>English</p></td><td class="cell"><p>English (11,036,726)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(eng)</p></td><td class="cell"><p>Javanese (1,443,054)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>3</p></td><td class="cell"><p>Indonesian</p></td><td class="cell"><p>Indonesian (12,530,636)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(ind)</p></td><td class="cell"><p>Javanese (4,168,734)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>4</p></td><td class="cell"><p>Thai</p></td><td class="cell"><p>Thai (4,367,895)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(tha)</p></td><td class="cell"><p>English (1,775,132)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>5</p></td><td class="cell"><p>Malay</p></td><td class="cell"><p>Malay (1,944,430)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(mly)</p></td><td class="cell"><p>Indonesian (1,516,778)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>6</p></td><td class="cell"><p>Sundanese</p></td><td class="cell"><p>Javanese (2,004,316)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(sun)</p></td><td class="cell"><p>Sundanese (1,641,623)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>7</p></td><td class="cell"><p>Luxemburg</p></td><td class="cell"><p>Luxemburg (1,128,342)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(ltz)</p></td><td class="cell"><p>Javanese (393,524)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>8</p></td><td class="cell"><p>Occitan</p></td><td class="cell"><p>Occitan (119,734)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(lnc)</p></td><td class="cell"><p>English (83,380)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>9</p></td><td class="cell"><p>Madurese</p></td><td class="cell"><p>Madurese (387,585)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(mad)</p></td><td class="cell"><p>Javanese (93,837)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>10</p></td><td class="cell"><p>Tatar</p></td><td class="cell"><p>Javanese (1,802,601)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(tat)</p></td><td class="cell"><p>Thai (397,988)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 1. LIM result for Indonesian Domain" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>No.</p></td><td class="cell"><p>Language</p></td><td class="cell"><p># of Pages</p></td><td class="cell"><p># of Outgo­ing Links</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>1</p></td><td class="cell"><p>Javanese (jav)</p></td><td class="cell"><p>797,300 (28.01%)</p></td><td class="cell"><p>33,411,032 (27.20%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>2</p></td><td class="cell"><p>English (eng)</p></td><td class="cell"><p>743,457 (26.11%)</p></td><td class="cell"><p>16,645,014 (13.55%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>3</p></td><td class="cell"><p>Indonesian (ind)</p></td><td class="cell"><p>516,528 (18.14%)</p></td><td class="cell"><p>20,783,793 (16.92%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>4</p></td><td class="cell"><p>Thai (tha)</p></td><td class="cell"><p>218,453 (7.67%)</p></td><td class="cell"><p>8,952,101 (7.29%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>S</p></td><td class="cell"><p>Malay (mly)</p></td><td class="cell"><p>197,535 (3.47%)</p></td><td class="cell"><p>4,990,402 (4.06%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>6</p></td><td class="cell"><p>Sundanese (sun)</p></td><td class="cell"><p>98,835 (3.47%)</p></td><td class="cell"><p>5,349,194 (4.35%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>7</p></td><td class="cell"><p>Luxemburg<footnote anchor="3"/>(ltz)</p></td><td class="cell"><p>43,376 (1.52%)</p></td><td class="cell"><p>2,307,602 (1.88%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>B</p></td><td class="cell"><p>Occitan<footnote anchor="4"/>(lnc)</p></td><td class="cell"><p>27,663 (0.97%)</p></td><td class="cell"><p>351,318 (0.29%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>9</p></td><td class="cell"><p>Madurese (mad)</p></td><td class="cell"><p>22,121 (0.78%)</p></td><td class="cell"><p>777,903 (0.63%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>10</p></td><td class="cell"><p>Tatar (tat)</p></td><td class="cell"><p>20,709 (0.73%)</p></td><td class="cell"><p>3,334,651 (2.71%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Others</p></td><td class="cell"><p>160,917 (5.65%)</p></td><td class="cell"><p>25,930,885 (21.11%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Total</p></td><td class="cell"><p>2,846,894</p></td><td class="cell"><p>122,833,898</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="6" global="54"/><p>The 6th Workshop on Asian Languae Resources, 2008</p><figure caption="Figure 1. Language Graph of Indonesia"></figure></subsection><subsection number="5.2" title="Intermediary Languages"><p>For languages with only a few outgoing links be­tween them, there exists a language that acts as its intermediary, that makes access between the two languages more convenient if passed through.</p><p>The above table shows the number of language pairs having the given language as its intermediary language. English has the highest frequency as an intermediary language. However, it is likely that several pages were misidentified by LIM. The sec­ond, Javanese is not surprising since it is a major language of Indonesia. The table below shows se­lected language pairs, where one of its intermedi­ary languages is Javanese.</p></subsection><subsection number="5.3" title="Graph Properties"><p>This section discusses the size distribution of the SCCs and the diameter of the Filipino language community in Indonesia.</p><p><b>SCC size</b></p><p>The SCCs of a graph are those sets of nodes such that for every node, there is a path to all the other nodes. The size of each SCC refers to the number of nodes it contains. Distribution of the sizes of SCCs gives a good understanding of the graph structure of the web, and has important implica­tions for crawling. If most components have large sizes, only a few nodes are needed as seed URLs (a list of starting pages for a crawler) to be able to reach all the other nodes. If all nodes are members of a single SCC, one URL is enough to crawl all pages.</p><p><b>SCC diameter</b></p><p>The maximum distance between any two nodes is the diameter. For each node size, the diameter is<page local="7" global="55"/></p><footnote label="5">k = Javanese</footnote><footnote label="6">Total</footnote><table caption="Table 4. Selected Languages of Indonesia in which it's Intermediary Language is Javanese" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Language <i>i</i></p></td><td class="cell"><p>Language <i>j</i></p></td><td class="cell"><p><i>Pj</i></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Tatar</p></td><td class="cell"><p>Indonesian</p></td><td class="cell"><p>0.01753</p></td><td class="cell"><p>0.06085</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Tatar</p></td><td class="cell"><p>Luxemburg</p></td><td class="cell"><p>0.00210</p></td><td class="cell"><p>0.01193</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Samoan</p></td><td class="cell"><p>Balinese</p></td><td class="cell"><p>0.00000</p></td><td class="cell"><p>0.00052</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 3. Intermediary Languages of Indonesia" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>No.</p></td><td class="cell"><p>Intermediary Language</p></td><td class="cell"><p>Frequency</p></td><td class="cell"><p>Percentage</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>1</p></td><td class="cell"><p>English</p></td><td class="cell"><p>89</p></td><td class="cell"><p>17.73%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>2</p></td><td class="cell"><p>Javanese</p></td><td class="cell"><p>43</p></td><td class="cell"><p>8.57%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>3</p></td><td class="cell"><p>Tatar</p></td><td class="cell"><p>42</p></td><td class="cell"><p>8.37%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>4</p></td><td class="cell"><p>Thai</p></td><td class="cell"><p>40</p></td><td class="cell"><p>7.97%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>5</p></td><td class="cell"><p>Madurese</p></td><td class="cell"><p>38</p></td><td class="cell"><p>7.57%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>6</p></td><td class="cell"><p>Indonesian</p></td><td class="cell"><p>34</p></td><td class="cell"><p>6.77%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>7</p></td><td class="cell"><p>Latin</p></td><td class="cell"><p>34</p></td><td class="cell"><p>6.77%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>8</p></td><td class="cell"><p>Sundanese</p></td><td class="cell"><p>29</p></td><td class="cell"><p>5.78%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>9</p></td><td class="cell"><p>Malay</p></td><td class="cell"><p>28</p></td><td class="cell"><p>5.58%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>10</p></td><td class="cell"><p>Luxemburg</p></td><td class="cell"><p>24</p></td><td class="cell"><p>4.78%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Others (top11-20)</p></td><td class="cell"><p>122</p></td><td class="cell"><p>24.30%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Total</p></td><td class="cell"><p>502</p></td><td class="cell"><p>100.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 5. SCC size distribution of the Filipino language community in Indonesia" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>SCC size</p></td><td class="cell"><p>1</p></td><td class="cell"><p>2</p></td><td class="cell"><p>4</p></td><td class="cell"><p>16</p></td><td class="cell"><p>19</p></td><td class="cell"><p>20</p></td><td class="cell"><p>26</p></td><td class="cell"><p>45</p></td><td class="cell"><p>T6</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p># of SCCs</p></td><td class="cell"><p>9</p></td><td class="cell"><p>1</p></td><td class="cell"><p>3</p></td><td class="cell"><p>1</p></td><td class="cell"><p>1</p></td><td class="cell"><p>1</p></td><td class="cell"><p>1</p></td><td class="cell"><p>1</p></td><td class="cell"><p>18</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p># of nodes</p></td><td class="cell"><p>9</p></td><td class="cell"><p>2</p></td><td class="cell"><p>12</p></td><td class="cell"><p>16</p></td><td class="cell"><p>19</p></td><td class="cell"><p>20</p></td><td class="cell"><p>26</p></td><td class="cell"><p>45</p></td><td class="cell"><p>149</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>The 6th Workshop on Asian Languae Resources, 2008</p><doubt alpha="87.5" length="8" tooSmall="True" monospace="0.0">SCC size</doubt><p>calculated and plotted in the chart. Their corre­sponding SCC graph is also shown.</p><doubt alpha="4.3" length="23" tooSmall="False" monospace="0.0">o '-1-1-1-1-'-'-'-'-'-'</doubt><doubt alpha="0.0" length="80" tooSmall="True" monospace="0.0">0 5         10       15       20       25       30       35       40       45 50</doubt><figure caption="Figure 2. Diameter distribution of the Filipino language community in Indonesia"></figure><p>For the Filipino language subgraph of Indonesia, the component with the largest node size also has the largest diameter. However, the largest diameter size is only 3, which is a very small number. Most of the components have a diameter of 4.</p></subsection></section><section number="6" title="Implications"><p>For the Filipino language community, many SCCs can be found in the Philippines, since Filipino is one of its national languages. However, for some countries, there are not many SCCs. For example, Indonesia only has 18 SCCs, half of which consist only of one node. However, the largest component size is 45, and there are 4 more large components. By picking out just one node from each SCC with a large size and using it as a seed URL, many web pages can already be downloaded. Add to it the depth parameter of 3, which is the largest diameter, these web pages can be downloaded within a short period of time.</p><p>The choice of seed URL and the crawly depth are useful parameters for crawling. The analysis is done for each language community to get these parameters for language-specific crawling pur­poses. These parameters are different for each lan­guage community. This paper shows only shows the case of the Filipino language community of Indonesia as a sample illustration of the diameter metric.</p></section><section number="7" title="Conclusion"><p>The vastness and multilingual content of the web makes it a rich source of culturally-diversified in­formation. Since web pages are connected by hyperlinks, information can be readily accesses by jumping from one page to another via the hyper­links.</p><p>For each country domain, the web pages written in the same language form a language community. The link structure between language communities shows how connected a language community is with another language community. It can be as­sumed that the close links between two language communities on the web imply the existence of multilingual speakers of the two languages. Oth­erwise linked pages will not be visited. In this con­text, the language graph analysis demonstrated in this study gives an effective tool to understand the linguistic scenes of the country. If the same analy­sis is performed for the secondary level domain data, further insight into the socio-linguistic status of each language can be drawn. Secondary domain corresponds to different social area of language activities, such as "ac" or "edu" for academic and education arena, "go" or "gov" for government or public arena, and "co" or "com" for commercial business and occupational arena. Although this study does not extend its scope to the secondary level domain analysis, the effectiveness of the ap­proach was demonstrated.</p><p>Another implication drawn from this study is that the language graph analysis can identify in­termediary languages in the multilingual communi­ties. In the real world, some languages are acting as a medium of communications among the differ­ent language speakers. In most cases, such lingua franca are international languages such as English, French, Arabic, etc. But it's difficult to identify which language is acting as such in detail. But on the web link structure among languages, the lan­guage graph can give us a clue to identify this. As shown in this paper, there are a number of lan­guages acting as intermediary between two lan­guages having only a few hyperlinks between them. Although the result of this category is doubtful because of misidentification of language, some cases show the expected result.</p><p>The second objective of the study is to give a microscopic level structure of the web communi­ties for much more practical and technical reasons, such as how to design more effective crawling strategy, and how to prepare starting URLs with minimal efforts. The key issue in this context is to reveal the connectedness of the web. To show the connectedness of language communities, several<page local="8" global="56"/></p><p>The 6th Workshop on Asian Languae Resources, 2008</p><p>graph theory metrics, the size and numbers of strongly-connected components and the diameters are calculated and visual presentations of language communities are also given. This information can aid in defining parameters used for crawling, par­ticularly language-specific crawling.</p><p>As a summary, the link structure analysis of lan­guage graphs can be a useful tool for various spec­trums of socio-linguistic and technical research purposes.</p></section><section number="8" title="Limitations and Future Work"><p>The results of this research are highly dependent on the language identification module (LIM). With a more improved LIM, more accurate results can be presented. Currently, there is an ongoing ex­periment that uses a new LIM.</p><p>This analysis will also be done to the secondary-level-domains to show the language distribution for different social areas.</p><p>Future work also includes the creation of a lan­guage-specific crawler that will incorporate the results derived from the analysis of the SCC size and diameter of the language subgraphs.</p><p><b>Acknowledgment</b></p><p>The study was made possible by the financial sup­port of the Japan Science and Technology Agency (JST) under the RISTEX program and the Asian Language Resource Network Project. We also thank UNESCO for giving official support to the project since its inception.</p></section><references><p>Balakrishnan, Hemant and Narsingh Deo. 2006. Evolu­tion in Web graphs. <i>Proceedings of the 37th South­eastern International Conference on Combinatorics, Graph Theory, and Computing. </i>Boca Raton, FL.</p><p>Bharat Krishna, Bay-Wei Chang, Monika Henzinger, and Matthias Ruhl. 2001. Who links to whom: min­ing linkage between Web sites. In <i>Proceedings of the 2001 IEEE International Conference on Data Mining, </i>pages 51-58, San Jose, California.</p><p>Boldi, Paolo, Bruno Codenotti, Massimo Santini, and Sebastiano Vigna. 2004. UbiCrawler: A scalable fully distributed web crawler. <i>Software: Practice &amp; Experience, 34(8):711-726.</i></p><p>Broder, Andrei, Ravi Kumar, Farzin Maghoul, Prab-hakar Raghavan, Sridhar Rajagopalan, Raymie Stata,</p><p>Andrew Tomkins, and Janet Wiener. 2000. Graph structure in the web. In <i>Proceedings of the 9th World Wide Web Conference, </i>pages 309-320, Amsterdam, Netherlands.</p><p>Chakrabarti, Soumen, Martin van den Berg, and By­ronDom. 1999. Focused Crawling: a new approach to topic-specific Web resource discovery. In <i>Pro­ceedings of the 8th International World Wide Web Conference</i>, pages 1623-1640, Toronto, Canada.</p><p>Herring, Susan C., John C. Paolillo, Irene Ramos-Vielba, Inna Kouper, Elijah Wright, Sharon Stoerger, Lois Ann Scheidt, and Benjamin Clark. 2007. Language Networks on LiveJournal. In <i>Proceedings of the 40th Annual Hawaii International Conference on System Sciences, </i>Hawaii.</p><p>Kumar, Ravi, Prabhakar Raghavan, Sridhar Rajagopalan, D. Sivakumar, Andrew S. Tompkins, and Eli Upfal. 2000. The Web as a graph. In <i>Proceedings of the Nineteenth ACM SIGMOD-SIGACT-SIGART Sympo­sium on Principles of Database Systems, </i>pages 1-10, Dallas, Texas, United States.</p><p>Petricek, Vaclav, Tobias Escher, Ingemar J. Cox, and Helen Margetts. 2006. The web structure of e-government: developing a methodology for quantita­tive evaluation. In <i>Proceedings of the 15th Interna­tional World Wide Web Conference, </i>pages 669-678, Edinburgh, Scotland.</p><p>Pingali, Prasad, Jagadeesh Jagarlamudi, and Vasudeva Varma. 2006. WebKhoj: Indian language IR from Multiple Character Encodings. In <i>Proceedings of the 15th International World Wide Web Conference, </i>pages 801-809, Edinburgh, Scotland.</p><p>Stamatakis, Konstantinos, Vangelis Karkaletsis, Geor-gios Paliouras, James Horlock, Claire Grover, James R.  Curran, and Shipra Dingare. 2003. Domain-</p><p>Specific Web Site Identification: The CROSSMARC</p><p>Focused Web Crawler. In <i>Proceedings of the 2nd In­ternational Workshop on Web Document Analysis (WDA 2003), </i>pages 75-78. Edinburgh, UK.</p><p>Suzuki, Izumi, Yoshiki Mikami, Ario Ohsato, and Yo-shihide Chubachi. 2002. A language and character set determination method based on N-gram statistics. <i>ACM Transactions on Asian Language Information Processing, </i>1(3): 269-278.</p><p>Tamura, Takayuki, Kulwadee Somboonviwat, and Ma-saru Kitsuregawa. 2007. A method for language-specific Web crawling and its evaluation. <i>Systems and Computers in Japan, </i>38(2):10-20.</p></references></body></article>