Breadth-first BASED WEB Crawling Application

May Phyu Htun
Computer University (Mandalay)
[email protected]

Abstract

The large size and the dynamic nature of the Web highlight the need for continuous support and updating of Web-based information retrieval systems. Crawlers facilitate the process by following the hyperlinks in Web pages to automatically download a partial snapshot of the Web. Traversing the web graph in breadth-first search order is a good crawling. This system is intended to study a crawling infrastructure and basic concepts in Web crawling. Then, web crawler application is implemented by using breadth-first search technique. Breadth-First Crawling checks each link on a page before proceeding to the next page. Thus, it crawls each link on the first page and then crawls each link on the first page’s first’ link, and so on, until each level of link has been exhausted. While Crawling the links of a URL address, the local HTML web pages are saved in a folder as MHTML format: (Single File Web Page).

Introduction

The Web is a very large collection of pages and search engines serve as the primary discovery mechanism to the content. To be able to provide the search functionality, search engines use crawlers that automatically follow links to web pages and extract. Web crawlers are programs that exploit the graph structure of the Web to move from page to page. In their infancy such programs were also called wanderers, robots, spiders, fish, and worms, words that are quite evocative of Web imagery. Crawler can be viewed as a graph search problem. The Web is seen as a large graph with pages at its nodes and hyperlinks as its edges. Web Crawler moves from node to node by means of the hyperlinks that each node contains and that define the edges of the web graph. Therefore, many algorithms used in graph searching can be frequently observed in web crawling of transformed versions. Traversing the web graph in breadth-first search order is a good crawling strategy, as it tends to discover high-quality pages early on in the crawl. In its simplest form, a crawler starts from a seed page and then uses the external links within it to attend to other pages. The process repeats with the new pages offering more external links to follow, until a sufficient number of pages are identified or some higher level objective is reached. There is a continual need for crawlers to help applications stay current as new pages are added and old ones are deleted or moved. When a web crawler is given a set of starting URL the web crawler downloads the corresponding documents. If save as a Web page  need to creates a folder that contains an .htm file and all supporting files, such as images, sound files, cascading style sheets, scripts, and more. Save your presentation as a Web page when you want to edit it with FrontPage or another HTML editor, and then post it to an...

-11 1 10 10th 1994 1998 1stinternational 2 2001 2003 2padrmini 3 3.1 3.2 3.3 3.4 3.start 4 47408 5 52242 6 7 abl abstract access acm across ad address advanc aggreg al algorithm alreadi also anoth anywher applet applic architectur archiv attempt attend attract auto automat averag avoid b base baselin basic becom best best-first big bind blind bloomington bottom bound box breadth breadth-first browser build built button call cannot cascad case certain chang check child cho choos chosen citi clear click client code collect combin come compani compar complet comput concept conclus confer connect construct contain content continu conveni convert copi correspond cost crawer crawl crawler crawler.web creat current data dead dead-end decreas defin delet demonstr depart depict depth depth-first describ design develop devic diagram differ discov discoveri discuss disk display document doesn download drive duplic durat dynam earli easili edg edit editor empti encapsul end engin entir environ et evalu even everyth evoc examin exhaust exist experi explain exploit explor extens extern extract facilit fast februari feed fetch fifo figur file filipomencz filippo final find first fish five flash flow folder follow form format frequent frist frontier frontpag function fundament futur gautam gautamp general geneva given goal good graph graph-travers graphic guid halt head help henc high high-qual higher highlight histori hot htm html http htun hyperlink ia identifi illustr imag imageri implement import improv in-memori includ indiana individu industri infanc infinit inform informat infospid infrastructur inlin integr intellig intend intern internet introduct involv iowa item j knowledg l larg learn left level librari like linear link list local long lookup loop lower m made maintain make manag mandalay mani manual mathemat may mean mechan memori menczer menez mht mhtml minutes.while modul moreov most move [email protected] must n najork natur necessarili need neither new newli next node not.futher number object observ occasion offer offlin old one onlin open order organ outlin outward overload overview padmini page pagerank pan pant paper pars parser partial path peopl perform period person phyu pick pinkerton polici portion possibl post pp present previous primari problem proc procedur proceed process program provid put qualiti queue queue.breadth quit reach read readi receiv recent refer referenc releat remov reopen repeat repositori repres request requir research resourc respect respons rest result retriev revisit right robot run save school scienc science.the score script search section see seed seem seen select send separ serv server set shark shark-search sheet short signal simpl simplest simplic sinc singl site situat size snapshot sophist sound sourc specif spider srinivasan srinvasan stack standard start stay step stinivasan stop storag store strategi structur studi style sub suffici support system system.section systemat tail take taken techniqu technolog tend termin terminolog text theori therefor thus tie time.this to-do togeth topic total transact transform travers treat tri two type typic unexpand univers unvisit unvist updat url urls.crawled usa use user v version view vigor visit vol wander want watch way web web-bas webcrawl well whether whole wide wiener within without word work world worm would yes yield