NAME
ALBD README
SYNOPSIS
This package consists of Perl modules along with supporting Perl
programs that perform Literature Based Discovery (LBD). The core
data from which LBD is performed are co-occurrences matrices
generated from UMLS::Association. ALBD is based on the ABC
co-occurrence model. Many options can be specified, and many
ranking methods are available. The novel ranking methods that use
association measure are available as well as frequency based
ranking methods. See samples/lbd for more info. Can perform open and
closed LBD as well as time slicing evaluation.
ALBD requires UMLS::Association both to compute the co-occurrence
database that the co-occurrence matrix is derived from, but also for
ranking the generated C terms.
UMLS::Association requires the UMLS::Interface module to access
the Unified Medical Language System (UMLS) for semantic type filtering
and to determine if CUIs are valid.
The following sections describe the organization of this software
package and how to use it. A few typical examples are given to help
clearly understand the usage of the modules and the supporting
utilities.
INSTALL
To install the module, run the following magic commands:
perl Makefile.PL
make
make test
make install
This will install the module in the standard location. You will, most
probably, require root privileges to install in standard system
directories. To install in a non-standard directory, specify a prefix
during the 'perl Makefile.PL' stage as:
perl Makefile.PL PREFIX=/home/programs
It is possible to modify other parameters during installation. The
details of these can be found in the ExtUtils::MakeMaker documentation.
However, it is highly recommended not messing around with other
parameters, unless you know what you're doing.
CO-OCCURRENCE MATRIX SETUP
ALBD requires that a co-occurrence matrix of CUIs has been created. This
matrix is stored as a flat file, in a sparse matrix format such that
each line contains three tab seperated values, cui_1, cui_2, n_11 = the
count of their co-occurrences. Any matrix with that format is
acceptable, however the intended method of matrix generation is to
convert a UMLS::Association database into a flat matrix file. These
databases are created using the CUICollector tool of UMLS::Association,
and are run over the MetaMapped Medline baseline. With that file, run
utils/datasetCreator/fromMySQL/dbToTab.pl to convert the desired
database into a matrix file. Notice that code in dbToTab.pl is just a
sample mysql command. If the input database is created in another
method, a different command may be needed. As long as the resulting
co-occurrence matrix is in the correct format LBD may be run on it. This
allows flexibility in where co-occurrence information comes from.
Note: utils/datasetCreator/fromMySQL/removeQuotes.pl may need to be run
on the resulting tab seperated file, if quotes are inlcuded in the
resulting co-ocurrence matrix file.
Set Up Dummy UMLS::Association Database
UMLS::Association requires that a database can be connected to that is
in the correct format. Although this database is not required for ALBD
(since co-occurrence data is loaded from a co-occurrence matrix), it is
required to run UMLS:Association. If you ran UMLS::Association to
generate a co-occurrence matrix, you should be fine. Otherwise you will
need to create a dummy database that it can connect to. This can be done
in a few steps:
1) open mysql type mysql at the terminal
2) create the default database in the correct format, type: CREATE
DATABASE cuicounts; use cuicounts; CREATE TABLE N_11(cui_1 CHAR(10),
cui_2 CHAR(10), n_11 BIGINT(20));
INITIALIZING THE MODULE
To create an instance of the ALBD object, using default values for all
configuration options: %options = (); $options{'lbdConfig'} =
'configFile'; my $lbd = LiteratureBasedDiscovery->new(\%options);
$lbd->performLBD();
The following configuration options are also provided though:
'assocConfig' path to a UMLS::Association configuration file. Default
location is 'config/association'. Replace this file for your computer to
avoid having to specify each time
'interfaceConfig' path to a UMLS::Interface configuration file. Default
location is '../config/interface'. Replace this file for your computer
to avoid having to specify each time.
These are passed through a hash. For example:
my %options = ();
$options{'assocConfig'} = '/home/share/ALBD/config/association';
$options{'interfaceConfig'} = '/home/shar/ALBD/config/interface';
$options{'lbdConfig'} = 'configFile'
my $lbd = LiteratureBasedDiscovery->new(\%options);
$lbd->performLBD();
CONTENTS
All the modules that will be installed in the Perl system directory are
present in the '/lib' directory tree of the package.
The package contains a utils/ directory that contain Perl utility
programs. These utilities use the modules or provide some supporting
functionality.
runDiscovery.pl -- runs LBD using the parameters specified in the input
file, and outputs to an output file.
The package contains a large selection of functions to manipulate CUI
Co-occurrence matrices in the utils/datasetCreator/ directory. These are
short scripts and generally require modifying the code at the top with
user input paramaters specific for each run. These scripts include:
applyMaxThreshold.pl -- applies a maximum co-occurrence threshold to the
co-occurrence matrix
applyMinThreshold.pl -- applies a minimum co-occurrence threshold to the
co-occurrence matrix
applySemanticFilter.pl -- applies a semantic type and/or group filter to
the co-occurrence matrix.
combineCooccurrenceMatrices.pl -- combines the co-occurrence counts of
multiple co-occurrence matrices
makeOrderNotMatter.pl -- makes the order of CUI co-occurrences not
matter by updating the co-occurrence matrix file. (UMLS::Association
generates co-occurrence files where order does matter, so the sentence
'cui1 cui2' will only mark a co-occurrence between cui1 and cui2, but
not between cui2 and cui1).
removeCUIPair.pl -- removes all occurrences of the specified CUI pair
from the co-occurrence matrix
removeExplicit.pl -- removes any keys that occur in an explicit
co-occurrence matrix from another co-occurrence matrix (typically the
squared explicit co-occurrence matrix itself, which generates a
prediction matrix, or the post cutoff matrix used in time slicing to
generate a gold standard file)
testMatrixEquality.pl -- checks to see if two co-occurrence matrix files
contain the same data
Also included are several subfolders with more specific purposes. Within
the dataStats subfolder are scripts to collect various statistics about
the co-occurrence matrices used in LBD. These scriptsinclude:
getCUICooccurrences.pl -- a data statistics file that gets the number of
co-occurrences, and number of unique co-occurrences for every CUI in the
dataset
getMatrixStats.pl -- determines the number of rows, columns, and entries
of a co-occurrence matrix
metaAnalysis.pl -- determines the number of rows, columns, vocabulary
size, and total number of co-occurrences of a co-occurrence file, or set
of co-occurrence files
There is another folder containing scripts to square co-occurrence
matrices. Squaring an explicit (A to B) co-occurrence matrix results in
a co-occurrence matrix containing all implicit (A to C) connections.
This is useful for time slicing and other analysis. Removal of the
original explicit matrix is an additional step that is required if you
wish to create a predictions matrix file for every CUI. This can be done
with the removeExplicit.pl script. Squaring a co-occurrence matrix can
be very computationally expensive, both in terms of ram and cpu. For
this reason MATLAB scripts are preferred over perl scripts. Even using
MATLAB ram can become an issue, and squaring sections of a matrix and
combining them into a single output matrix may be necassary, but takes
much longer. Scripts in the squaring folder include:
convertForSquaring_MATLAB.pl -- functions to convert to and from ALBD
and MATLAB sparse matrix formats
squareMatrix.m -- MATLAB script to square a matrix while holding
everything in ram. Faster, but requires more ram.
squareMatrix_partial.m -- MATLAB script to square a matrix in chunks.
Only loads parts of the matrix into ram at a time which makes squaring
any size matrix possible, but potentially take impracticle amounts of
time.
squareMatrix_perl.pl -- squares a matrix in perl, but requires the most
ram of any squaring method. The easiest method to use, but only
practical for small datasets.
The fromMySQL folder contains scripts that convery UMLS::Association
databases to ALBD co-occurrence matrices. The files contained are:
dbToTab.pl -- converts a UMLS::Association co-occurrence database to a
sparse format co-occurrence matrix used for ALBD
removeQuotes.pl -- removes quotes from lines in the co-occurrence matrix
file after converting from a database (sometimes needed)
REFERENCING
If you write a paper that has used UMLS-Association in some way, we'd
certainly be grateful if you sent us a copy.
CONTACT US
If you have any trouble installing and using ALBD, please contact us
directly if you prefer :
Sam Henry: henryst at vcu.edu
Bridget McInnes: btmcinnes at vcu.edu
SOFTWARE COPYRIGHT AND LICENSE
Copyright (C) 2017 Sam Henry & Bridget McInnes
This suite of programs is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License as published
by the Free Software Foundation; either version 2 of the License, or (at
your option) any later version.
This program is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General
Public License for more details.
You should have received a copy of the GNU General Public License along
with this program; if not, write to the Free Software Foundation, Inc.,
59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
Note: The text of the GNU General Public License is provided in the file
'GPL.txt' that you should have received with this distribution.