2 Open source distribution, installation

The focused crawler has been restructured and packaged as a Debian package in order to ease distribution and installation. The package contains dependency information to make sure that all software that is needed to run the crawler is installed at the same time. In connection with this we have also packaged a number of necessary Perl-modules as Debian packages.

All software and packages are available from a number of places:

In addition to the distribution sites there is a public discussion list at SourceForge7.

2.1 Installation

This distribution is developed and tested on Linux systems. It is implemented entirely in Perl and uses the MySQL8 database system, both of which are supported on many other operating systems. Porting to other UNIX dialects should be easy.

The system is distributed either as source or as a Debian package.

2.1.1 Installation from source for the impatient

Unless you are on a system supporting Debian packages (in which case look at Automated installation (section 2.1.3)), you should download and unpack the source. The following command sequence will then install Combine:

perl Makefile.PL  
make test  
make install  
mkdir /etc/combine  
cp conf/* /etc/combine/  
mkdir /var/run/combine

Test that it all works (run as root)

2.1.2 Porting to not supported operating systems - dependencies

In order to port the system to another platform, you have to verify the availability, for this platform, of the two main systems:

If they are supported you stand a good chance to port the system.

Furthermore, the external Perl modules (listed in 10.3) should be verified to work on the new platform.

Perl modules are most easily installed using the Perl CPAN automated system
(perl -MCPAN -e shell).

Optionally the following external programs will be used if they are installed on your system:

2.1.3 Automated Debian/Ubuntu installation

This also installs all dependencies such as MySQL and a lot of necessary Perl modules.

2.1.4 Manual installation

Download the latest distribution11.

Install all software that Combine depends on (see above).

Unpack the archive with tar zxf
This will create a directory named combine-XX with a number of subdirectories including bin, Combine, doc, and conf.

bin’ contains the executable programs.

Combine’ contains needed Perl modules. They should be copied to where Perl will find them, typically /usr/share/perl5/Combine/.

conf’ contains the default configuration files. Combine looks for them in /etc/combine/ so they need to be copied there.

doc’ contains documentation.

The following command sequence will install Combine:

perl Makefile.PL  
make test  
make install  
mkdir /etc/combine  
cp conf/* /etc/combine/  
mkdir /var/run/combine

2.1.5 Out-of-the-box installation test

A simple way to test your newly installed Combine system is to crawl just one Web-page and export it as an XML-document. This will exercise much of the code and guarantee that basic focused crawling will work.

sudo combineINIT --jobname aatest --topic /etc/combine/Topic_carnivor.txt

combine --jobname aatest  
        --harvest http://combine.it.lth.se/CombineTests/InstallationTest.html

combineExport --jobname aatest --profile dc

<?xml version="1.0" encoding="UTF-8"?>  
<documentCollection version="1.1" xmlns:dc="http://purl.org/dc/elements/1.1/">  
<metadata xmlns:dc="http://purl.org/dc/elements/1.1/">  
<dc:format>text/html; charset=iso-8859-1</dc:format>  
<dc:subject>Carnivorous plants</dc:subject>  
<dc:title transl="yes">Installation test for Combine</dc:title>  
<dc:date>2006-05-19 9:57:03</dc:date>  

Or run – as root – the script ./doc/InstallationTest.pl (see A.1 in the Appendix) which essentially does the same thing.

2.2 Getting started

A simple example work-flow for a trivial crawl job name ’aatest’ might look like:

  1. Initialize database and configuration (needs root privileges)
    sudo combineINIT --jobname aatest
  2. Load some seed URLs like (you can repeat this command with different URLs as many times as you wish)
    echo ’http://combine.it.lth.se/’ | combineCtrl load --jobname aatest
  3. Start 2 harvesting processes
    combineCtrl start --jobname aatest --harvesters 2
  4. Let it run for some time. Status and progress can be checked using the program ’combineCtrl --jobname aatest’ with various parameters.
  5. When satisfied kill the crawlers
    combineCtrl kill --jobname aatest
  6. Export data records in the ALVIS XML format
    combineExport --jobname aatest --profile alvis
  7. If you want to schedule a recheck for all the crawled pages stored in the database do
    combineCtrl reharvest --jobname aatest
  8. Go back to 3 for continuous operation.

Once a job is initialized it is controlled using combineCtrl. Crawled data is exported using combineExport.

2.3 Online documentation

The latest, updated, detailed documentation is always available online12.

2.4 Use scenarios

2.4.1 General crawling without restrictions

Use the same procedure as in section 2.2. This way of crawling is not recommended for the Combine system since it will generate really huge databases without any focus.

2.4.2 Focused crawling – domain restrictions

Create a focused database with all pages from a Web-site. In this use scenario we will crawl the Combine site and the ALVIS site. The database is to be continuously updated, i.e. all pages have to be regularly tested for changes, deleted pages should be removed from the database, and newly created pages added.

  1. Initialize database and configuration
    sudo combineINIT --jobname focustest
  2. Edit the configuration to provide the desired focus
    Change the <allow> part in /etc/combine/focustest/combine.cfg from
    #use either URL or HOST: (obs ’:’) to match regular expressions to either the  
    #full URL or the HOST part of a URL.  
    #Allow crawl of URLs or hostnames that matches these regular expressions  
    HOST: .*$  


    #use either URL or HOST: (obs ’:’) to match regular expressions to either the  
    #full URL or the HOST part of a URL.  
    #Allow crawl of URLs or hostnames that matches these regular expressions  
    HOST: www\.alvis\.info$  
    HOST: combine\.it\.lth\.se$  

    The escaping of ’.’ by writing ’\.’ is necessary since the patterns actually are Perl regular expressions. Similarly the ending ’$’ indicates that the host string should end here, so for example a Web server on www.alvis.info.com (if such exists) will not be crawled.

  3. Load seed URLs
    echo ’http://combine.it.lth.se/’ | combineCtrl load --jobname focustest
    echo ’http://www.alvis.info/’ | combineCtrl load --jobname focustest
  4. Start 1 harvesting process
    combineCtrl start --jobname focustest
  5. Daily export all data records in the ALVIS XML format
    combineExport --jobname focustest --profile alvis
    and schedule all pages for re-harvesting
    combineCtrl reharvest --jobname focustest

2.4.3 Focused crawling – topic specific

Create and maintain a topic specific crawled database for the topic ’Carnivorous plants’.

  1. Create a topic definition (see section 4.5.1) in a local file named cpTopic.txt. (Can be done by copying /etc/combine/Topic_carnivor.txt since it happens to be just that.)
  2. Create a file named cpSeedURLs.txt with seed URLs for this topic, containing the URLs:

  3. Initialization
    sudo combineINIT --jobname cptest --topic cpTopic.txt

    This enables topic checking and focused crawl mode by setting configuration variable doCheckRecord = 1 and copying a topic definition file (cpTopic.txt) to

  4. Load seed URLs
    combineCtrl load --jobname cptest < cpSeedURLs.txt
  5. Start 3 harvesting process
    combineCtrl start --jobname cptest --harvesters 3
  6. Regularly export all data records in the ALVIS XML format
    combineExport --jobname cptest --profile alvis

Running this crawler for an extended period will result in more than 200 000 records.

2.4.4 Focused crawling in an Alvis system

Use the same procedure as in section 2.4.3 (Focused crawling – topic specific) except for the last point. Exporting should be done incrementally into an Alvis pipeline (in this example listening at port 3333 on the machine nlp.alvis.info):
combineExport --jobname cptest --pipehost nlp.alvis.info --pipeport 3333 --incremental

2.4.5 Crawl one entire site and it’s outlinks

This scenario requires the crawler to:

I.e. all of http://my.targetsite.com/*, plus any other URL that is linked to from a page in http://my.targetsite.com/*.

  1. Configure Combine to crawl this one site only. Change the <allow> part in
    /etc/combine/XXX/combine.cfg to
    #use either URL or HOST: (obs ’:’) to match regular expressions to either the  
    #full URL or the HOST part of a URL.  
    #Allow crawl of URLs or hostnames that matches these regular expressions  
    HOST: my\.targetsite\.com$  

  2. Crawl until you have the entire site (if it’s a big site you might want to do the changes suggested in FAQ no 7).
  3. Stop crawling.
  4. Change configuration <allow> back to allow crawling of any domain (which is the default).
    #Allow crawl of URLs or hostnames that matches these regular expressions  
    HOST: .*$  

  5. Schedule all links in the database for crawling, something like (change XXX to your jobname)
    echo ’select urlstr from urls;’ | mysql -u combine XXX
              | combineCtrl load --jobname XXX
  6. Change configuration to disable automatic recycling of links:
    #Enable(1)/disable(0) automatic recycling of new links
    AutoRecycleLinks = 0

    and maybe (depending or your other requirements) change:
    #User agent handles redirects (1) or treat redirects as new links (0)
    UserAgentFollowRedirects = 0

  7. Start crawling and run until no more in queue.