#!/usr/local/bin/perl -w

=head1 NAME 

setup.pl - Preprocess Senseval-2 data for sample experiments

=head1 SYNOPSIS 

A Perl script that preprocesses and prepares DATA for experiments with SenseClusters. 

=head1 USAGE 

setup.pl [Options] DATA 

Type 'setup.pl --help' for quick summary of options

=head1 INPUT

=head2 Required Arguments:

=head3 DATA

SenseClusters requires input DATA in Senseval-2 format. DATA in any other 
format has to be first converted to this format. We provide with this 
distribution a pre-processing program Toolkit/preprocess/plain/text2sval.pl 
that converts  data in plain text format (with the context of single 
instance on each line) into Senseval-2 format.

SenseClusters uses an unsupervised clustering approach and hence doesn't 
require DATA to be sense tagged at all. However, if the true sense classes
of the DATA instances are available, those could be used for evaluation.

=head2 Optional Arguments:

=head3 Data Options (--training | --split)? 

Data Options specify the input DATA to setup.pl. There are three different 
possibilities, which can be denoted via the following regex:

C<setup.pl DATA (--training TRAIN or --split P)?>

1. If neither --training nor --split options are used, DATA will be clustered 
using the features extracted from the same DATA file.

2. If a separate Training file is provided via --training TRAIN option, 
DATA will be clustered using features extracted from the given TRAIN file.
TRAIN file is expected to be in the Senseval-2 format.

3. If DATA file is provided with the --split P option, (100-P)% of DATA will be 
clustered using features extracted from the rest P% DATA.

Thus, DATA can be provided in a single DATA file, or with the --split P option
or with a separate training file via --training TRAIN option. Options --split 
and --training can't be both used together.

=head3 Sense Tag Options :

If the correct answer tags of the DATA instances are known, these can be used 
for performing some special tasks like evaluating results or filtering low 
frequency senses.

=head4 --key KEY

Specifies the true sense tags of the DATA instances.

Each line in the KEY file should show 

   <instance id=\"I\"\/>  [<sense id=\"S\"\/>]+

where an Instance-Id is followed by its true sense tag/s. 

KEY file in any other format has to be first converted to this format. We 
provide with this distribution a script called keyconvert.pl (in
Toolkit/preprocess/sval2) to convert a KEY file in Senseval-2 format (such as
fine.key) to SenseClusters' KEY format.

If the KEY file is not specified and if any sense-tag options like evaluation
or sense-filter are used in wrapper discriminate.pl, SenseClusters assumes 
that the sense ids are embedded in the same DATA file and these will be 
searched by matching an expression

                /sense\s*id=\"SID\"/

assuming that SID shows a true sense id of an immediately preceding 
instance ID matched by /instance id=\"ID\"/ expression.

=head3 Tokenization Options :

=head4 --token TOKENFILE

TOKENFILE is a file of Perl regexes that define tokenization scheme in DATA. 
A sample regex file, token.regex is provided with this distribution and if the
user doesn't specify the TOKENFILE via --token option, token.regex will be 
searched in the current directory. 

=head4 --nontoken NONTOKENFILE

NONTOKENFILE is a file of Perl regexes that define strings that will be removed
prior to tokenization. If NONTOKENFILE is not specified, only those string 
sequences that are not tokens will be removed. 

=head3 Other Options :

=head4 --verbose

Displays to STDERR the current program status. Silent by default.

=head4 --showargs

Displays to STDOUT values of required and option arguments.

=head4 --help

Displays this message.

=head4 --version

Displays the version information.
	
=head1 OUTPUT

setup.pl preprocesses given DATA in following ways -

=over 4

=item 1.

Creates a LexSample directory that contains a separate directory for each 
<lexelt> found in the DATA file. Each LEXELT directory (found within LexSample) 
will have following files -

=over 8

=item * LEXELT-test.xml 

A test XML file containing instances of single 'LEXELT' from a given DATA file.
If --split P is specified, LEXELT-test.xml will have (1-P)% of the instances 
in DATA with the value of <lexelt> tag as LEXELT.
Otherwise, will have all DATA instances that come under the LEXELT item.

=item * LEXELT-test.count

A count file containing instance data within <context> and </context> tags on 
a single line for each instance appearing in corresponding LEXELT-test.xml.


=item * [LEXELT-training.xml]

This file is created only if --training or --split options are used. 
If --split P is used, this will contain P% of the DATA instances having 
<lexelt> tag value = LEXELT. Otherwise, this file will have all instances 
from TRAINING that come under the LEXELT tag.

=item *	[LEXELT-training.count]

This is created only if its equivalent LEXELT-training.xml is created and  
contains instance data within <context> </context> tags on a single line for 
each instance appearing in LEXELT-training.xml.

=back

Additionally, LexSample directory will have files token.regex and 
nontoken.regex if --nontoken option is used.

=item 2. 

Converts data within <context> and </context> tags to lowercase. 

=item 3.

If KEY file is specified via --key option, answer tags are put along with the 
instances in the corresponding LEXELT files. These will be accessed only 
during the sense tag options like evaluation or filtering low frequency senses.
Sense tags are ignored during clustering and feature selection. The inclusion
of sense tags in the same XML files is only meant for sake of convenience for
programming.

=back

=head1 SYSTEM REQUIREMENTS

setup.pl uses a preprocessing program L<preprocess.pl> which is included 
in SenseClusters

=head1 SEE ALSO

L<discriminate.pl> uses the training and test files created by setup.pl

=head1 AUTHORS

 Amruta Purandare, University of Pittsburgh

 Ted Pedersen, University of Minnesota, Duluth
 tpederse at d.umn.edu

=head1 COPYRIGHT

Copyright (c) 2002-2008, Amruta Purandare and Ted Pedersen

This program is free software; you can redistribute it and/or modify it under 
the terms of the GNU General Public License as published by the Free Software 
Foundation; either version 2 of the License, or (at your option) any later 
version.

This program is distributed in the hope that it will be useful, but WITHOUT 
ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS 
FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with 
this program; if not, write to

 The Free Software Foundation, Inc.,
 59 Temple Place - Suite 330,
 Boston, MA  02111-1307, USA.

=cut
###############################################################################

#                           ================================
#                            COMMAND LINE OPTIONS AND USAGE
#                           ================================

# command line options
use Getopt::Long;
GetOptions ("help","version","showargs","verbose","training=s","split=f","token=s","nontoken=s","key=s");

# show help option
if(defined $opt_help)
{
        $opt_help=1;
        &showhelp();
        exit;
}

# show version information
if(defined $opt_version)
{
        $opt_version=1;
        &showversion();
        exit;
}
# show minimal usage message if required arguments not specified
if($#ARGV<0)
{
        &showminimal();
        exit;
}

###############################################################################

#			============================
#			    Initialization Section
#			============================

# $0 contains the program name along with the complete path. 
# Extract just the program name and use in error messages

$0=~s/.*\/(.+)/$1/;

if(!defined $ARGV[0])
{
	print STDERR "ERROR($0): 
	Specify a Senseval-2 formatted DATA file.\n";
	exit;
}
$datafile=$ARGV[0];

if(defined $opt_training && defined $opt_split)
{
	print STDERR "ERROR($0):
	--split and --training can't be used together. Can't split DATA into 
	Training and Test parts when a training file is already given.\n";
	exit;
}

if(defined $opt_training)
{
	$trainfile=$opt_training;
}

if(defined $trainfile && $datafile eq $trainfile)
{
	print STDERR "ERROR($0):
	Training file <$trainfile> and DATA file <$datafile> are same.\n";
	exit;
}
# if token file not specified, use our token.regex 
if(!defined $opt_token)
{
	$opt_token="token.regex";
}

# if --showargs is selected show command line argument values 
if(defined $opt_showargs)
{
	print "DATA: $datafile\n";
	if(defined $opt_training)
	{
               print "--training=$trainfile\n";
	}
	print "--token=$opt_token\n";
        if(defined $opt_nontoken)
        {
                print "--nontoken=$opt_nontoken\n";
        }
	if(defined $opt_split)
	{
		print "--split=$opt_split\n";
	}
	if(defined $opt_key)
	{
		print "--key=$opt_key\n";
	}
	if(defined $opt_showargs)
	{
		print "--showargs=ON\n";
	}
}
###############################################################################

#			==========================
# 			   Preprocessing Section
#			==========================

if(! -e $datafile)
{
	print STDERR "ERROR($0):
	Specified DATA file <$datafile> doesn't exist.\n";
	exit;
}
if(defined $trainfile && !-e $trainfile)
{
	print STDERR "ERROR($0):
	Specified Training file <$trainfile> doesn't exist.\n";
	exit;
}
if(! -e $opt_token)
{
	print STDERR "ERROR($0):
	Token file <$opt_token> doesn't exist.\n";
	exit;
}
if(defined $opt_nontoken && !-e $opt_nontoken)
{
	print STDERR "ERROR($0):
	Specified NonToken file <$opt_nontoken> doesn't exist.\n";
	exit;
}
# -------------
# prepare_sval2 
# -------------
	if(defined $opt_verbose)
	{
		print STDERR "Preprocessing $datafile ...\n";
	}
	if(defined $opt_key)
	{
		system("prepare_sval2.pl --key $opt_key --modifysat $datafile > $datafile.proc");
	}
	else
	{
		system("prepare_sval2.pl --modifysat $datafile > $datafile.proc");

	}

	if(defined $trainfile)
	{
		if(defined $opt_key)
		{
			system("prepare_sval2.pl --key $opt_key --modifysat $trainfile > $trainfile.proc");
		}
		else
		{
			system("prepare_sval2.pl --modifysat $trainfile > $trainfile.proc");
		}
	}
	
# --------------
# preprocess.pl
# --------------
		$lexdir="LexSample";
                if(-e $lexdir)
                {
                        print STDERR "ERROR($0):
		        Directory $lexdir already exists. Aborting ...\n";
                        exit;
                }
                mkdir $lexdir;
		system("cp $opt_token $lexdir/token.regex");
		if(defined $opt_nontoken)
		{
			system("cp $opt_nontoken $lexdir/nontoken.regex");
		}
	# preprocessing with --split ON
	if(defined $opt_split)
	{
		system("mv $datafile.proc $lexdir/$datafile");
		chdir $lexdir;
		if(-e "nontoken.regex")
		{
			if(defined $opt_verbose)
			{
				system("preprocess.pl --verbose --split $opt_split --token token.regex --removeNotToken --nontoken nontoken.regex $datafile");
			}
			else
			{
				system("preprocess.pl --split $opt_split --token token.regex --removeNotToken --nontoken nontoken.regex $datafile");
			}
		}
		else
		{
			if(defined $opt_verbose)
			{
				system("preprocess.pl --verbose --split $opt_split --removeNotToken --token token.regex $datafile");
			}
			else
			{
				system("preprocess.pl --split $opt_split --removeNotToken --token token.regex $datafile");
			}
		}
		system("mkdir train-lexelts");
		system("mkdir test-lexelts");

		system("/bin/rm -f $datafile");
		system("mv *-training.* train-lexelts/");
		system("mv *-test.* test-lexelts/");
		chdir "../";
	}
	else
	{	
		# preprocessing training file
		if(defined $trainfile && -e $trainfile)
		{
			system("mv $trainfile.proc $lexdir/$trainfile");
			chdir $lexdir;
			if(-e "nontoken.regex")
	                {
				if(defined $opt_verbose)
				{
        	                	system("preprocess.pl --verbose --token token.regex --removeNotToken --nontoken nontoken.regex $trainfile");
				}
				else
				{
					system("preprocess.pl --token token.regex --removeNotToken --nontoken nontoken.regex $trainfile");
				}
                	}
	                else
        	        {
				if(defined $opt_verbose)
				{
                	        	system("preprocess.pl --verbose --removeNotToken --token token.regex $trainfile");
				}
				else
				{
					system("preprocess.pl --removeNotToken --token token.regex $trainfile");
				}
	                }

        	        # gathering separate lexelt files created by 
			# preprocess.pl in dir train-lexelts

			system("/bin/rm -f $trainfile");
	                system("mkdir train-lexelts");
        	        system("mv *.count train-lexelts/");
                	system("mv *.xml train-lexelts/");
			chdir "../";
		}
		system("mv $datafile.proc $lexdir/$datafile");
		chdir $lexdir;
		# preprocessing DATA file
		if(-e "nontoken.regex")
                {
			if(defined $opt_verbose)
			{
	                       	system("preprocess.pl --verbose --token token.regex --removeNotToken --nontoken nontoken.regex $datafile");
			}
			else
			{
				system("preprocess.pl --token token.regex --removeNotToken --nontoken nontoken.regex $datafile");
			}
                }
                else
                {
			if(defined $opt_verbose)
			{
                       		system("preprocess.pl --verbose --removeNotToken --token token.regex $datafile");
			}
			else	
			{
				system("preprocess.pl --removeNotToken --token token.regex $datafile");
			}
                }
                # gathering separate lexelt files created by preprocess.pl
                # in dir test-lexelts

                system("mkdir test-lexelts");
		system("/bin/rm -f $datafile");
                system("mv *.count test-lexelts/");
	        system("mv *.xml test-lexelts/");

		chdir "../";
	}
	# -----------------------------------------------------------
	# After preprocessing, there will be a test-lexelt dir 
	# and an optional train-lexelt dir containing separate
	# XML/count files of each lexelt element
	# -----------------------------------------------------------	

if(defined $opt_verbose)
{
	print STDERR "Preprocessing complete ...\nCreating LexSample...\n";
}
# --------------------------------------
# Setting up directories for experiment
# --------------------------------------
system("setdirs.sh");

# after this there will be a LexSample dir containing a separate dir for each
# lexelt and each lexelt containing a separate test dir and optional train
# dir

if(defined $opt_verbose)
{
	print STDERR "Created LexSample Directory !!!\nWe are now ready to start experiments!\n";
	print STDERR "Run sc-toolkit.sh, target-wrapper.sh, or word-wrapper.sh\n";    
        print STDERR "Remember to rename the LexSample file after each experiment\n";
	print STDERR "and run makedata.sh again.\n";

}

##############################################################################

#                      ==========================
#                          SUBROUTINE SECTION
#                      ==========================

#-----------------------------------------------------------------------------
#show minimal usage message
sub showminimal()
{
	print "Usage: setup.pl [Options] DATA\n";
	print "Type 'setup.pl --help' for quick summary of the Options.\n";
	print "Type 'perldoc setup.pl' for detailed description of setup.pl.\n'";
}

#-----------------------------------------------------------------------------
#show help
sub showhelp()
{
	print "Usage: setup.pl [Options] DATA";
	print "Required Arguments:

DATA	a Senseval-2 formatted file

Optional Arguments:

Data Options :
--------------

--training TRAINFILE
	DATA will be clustered using the features extracted from the given
	TRAINFILE file. By default, features are extracted from the same DATA
	file.

--split P
	Given DATA is randomly split into P:1-P TRAIN::TEST parts such that,
	(1-P)% TEST portion is clustered using the features extracted from the
	rest P% TRAIN.

If neither --training nor --split options are used, DATA will be clustered
using the features extracted from the same DATA file.

Sense Tag Options :
-------------------

--key KEY
	If the sense ids of the DATA instances are available in a separate KEY
	file, this information can be used for evaluation of results. The 
	KEY file specified via --key option should show the correct sense tags
	of the DATA instances.
 
Tokenization Options :
----------------------
--token TOKENFILE

	TOKENFILE is a file of Perl regexes that define tokenization scheme in 
	DATA. If not specified, a default file token.regex will be searched 
	in current directory.

--nontoken NONTOKENFILE

	NONTOKENFILE is a file of Perl regexes that define strings that  
	will be removed prior to tokenization.

Other Options :
---------------

--verbose
	Displays to STDERR the current program status. Silent by default.

--showargs
	Displays to STDOUT values of required and option arguments.

--help
        Displays this message.

--version
        Displays the version information.

Documentation :
----------------
See perldoc page of setup.pl for detailed help.\n";

}
#------------------------------------------------------------------------------

#version information
sub showversion()
{
#        print "setup.pl      -       Version 0.03\n";
	print '$Id: setup.pl,v 1.3 2008/03/29 23:43:35 tpederse Exp $';
        print "\nPrepares samples DATA for running experiments with SenseClusters\n";
#        print "\nCopyright (c) 2002-2006, Ted Pedersen, Amruta Purandare.\n";
#        print "Date of Last Update:     07/28/2006\n";
}

#############################################################################