=head1 NAME

makepp_scanning -- How makepp finds include files and other hidden dependencies

=for vc $Id: makepp_scanning.pod,v 1.11 2006/10/16 21:55:37 pfeiffer Exp $

=head1 DESCRIPTION

Makepp can guess additional dependencies or targets for certain commands
that it knows something about.  This is especially important for C/C++
compilation, where it is too error-prone to list manually all of the
include files that a given source file depends on.  By looking at the
compilation command and the source files themselves, makepp is able to
determine accurately which object files need to be rebuilt when some
include file changes.

Makepp looks at the first word of your command line.
If it recognizes it, it uses the scanner corresponding to
that first word.  Specifically, it isolates the first word and looks it
up in its table; if nothing is found, it strips off the directory
information and looks it up again.  (This means that you can specify the
path to your compiler and makepp will still recognize it.)  Currently,
makepp recognizes most C/C++ compiler names.  It also recognizes
commands invoking C<sh> and C<libtool>; for these commands, it skips to
the first word in the command that's not an option and looks it up in
the table again.

If makepp thinks it's compiling a C/C++ program but can't find a
scanner, it will give a warning message to let you know.  This usually
means that you buried your compiler command too deeply in the action for
makepp to find it.  For example, I have seen rules like this:

    %.o: %.c
 	@echo Compiling $< now
 	@gcc -c $< $(CFLAGS) -o $@

The first word of the action here is C<echo>, for which therer is no
scanner, so makepp will not scan for include files in this case.

=head2 C/C++ compilation

The C/C++ scanner is activated by a command beginning with a compile
program that makepp knows about.  (I've included every compiler I've
ever heard of, but if I missed your compiler, you can tell makepp about
it by adding an entry to the C<%Makesubs::scanners> array (see
F<Makesubs.pm> in the distribution).  Also send me email at
holt-makepp@gholt.net so I can include it in subsequent releases.

It looks at the command for C<-Idir> options specifying the include path
or C<-Ldir> options specifying the link path.  It then scans any source
files for C<#include> directives, and also looks at the command line to
see if there are any source files or libraries mentioned which are not
listed as dependencies.  It recognizes these by their extension.

This scanner gives a warning message if files included with S<C<#include
"file.h">> are not found in the include path, or in the directory
containing the file which is C<#includ>ing, or in F</usr/include>.  No
warning is given if a file included with S<C<#include E<lt>file.hE<gt>>>
is not found.  makepp assumes it is in some system include directory
that the compiler knows about, and that files in system include
directories won't change.

In addition, files in F</usr/include>, F</usr/local/include>,
F</usr/X11R6/include>, and any other directory which is not writable are
not scanned to see what they include.  Makepp assumes that these files
won't change.  (If you're running as root, the writability test is
performed with the UID and GID of the directory you ran makepp from.
This is so compiling a program as an ordinary user and then doing
S<C<makepp install>> as root won't cause extra directories to be scanned.)

This is a fairly simple-minded scanner.  It will get confused if you do
things like this:

    #ifdef INCLUDE_THIS
    #include "this.h"
    #endif

because it doesn't know about preprocessor conditionals.  This is
usually harmless; it might cause additional extra files to be labelled
as dependencies (occasionally causing unnecessary rebuilds), or else it
might cause makepp to warn that the include file was not found.  You can
either ignore the warning messages, or put an empty file C<this.h> out
there to shut makepp up.

=head2 Libtool

Libtool is a very clever compilation system that greatly simplifies
making shared libraries by hiding all the system-dependent details away
in a shell script.  The only difficulty is that the library binary files
are not actually stored in the same directory as the output
file--libtool actuall creates a subdirectory, C<.libs>, which contains
the real files.  This is ordinarily not a problem, but makepp has to
know where the real binaries are if it is to link them in from a
repository.  At the moment, libtool libraries (C<.la> files) are not
linked in from repositories; they are always rebuilt if needed.  Also,
makepp at the moment is not able to use the dependency information that
is stored inside the C<.la> file itself.  This will hopefully change
soon.

=head2 Swig

Swig (Simplified Wrapper and Interface Generator, http://www.swig.org)
is a program that converts a C/C++ header file into the wrapper
functions needed to make your code callable from a variety of other
languages, such as perl, python, tcl, C#, ruby, ocaml, and probably some
others that I don't know about.

Makepp understands and parses the swig command line, looking for C<-I> and
C<-l> options.  It also knows how to scan swig's interface definition files
(F<.i> files) looking for C<%include>, C<%import>, and also C<#include>
if C<-includeall> is in effect.

=head2 Other special command words

Makepp recognizes the following command words and skips over them
appropriately in in its search for the correct scanner:
C<condor_compile>, C<echo> C<ignore_error> C<libtool>, C<noecho>
C<purify>, C<sh>.

=head2 Quickscan and smartscan

The :quickscan and :smartscan rule options, if applicable, affect the way
that files are scanned.

In :quickscan mode (the default), all include directives are assumed
active. This allows for very efficient scanning.

In :smartscan mode, an attempt is made to interpret macros and expressions
so that inactive include directives are ignored.
For example, the executable produced by compiling the following C program
ought I<not> to depend on F<foo.h>:

    #if 0
    # include "foo.h"
    #endif
    int main() { return 0; }

In the future, :smartscan might become the default.

=head2 Custom scanners

Writing your own scanner is somewhat involved, but it is possible.
There are 3 ways to hook into this:

=over 3

=item 1.

If scanner foo is specified in a rule option, and there is a subroutine
called parser_foo defined in the makefile's package, then that subroutine
must return an object of type C<ActionParser>, which will be used to
determine a command parser for every command.

=item 2.

If scanner foo is specified in a rule option, and there is a subroutine
called scanner_foo defined in the makefile's package, then that subroutine
must either do all of the scanning work (which is deprecated), or return
an object of type C<CommandParser> which will be used to parse every command.

=item 3.

If no scanner is specified in a rule option, then an object of the
C<ActionParser> base class is used.
That object determines the scanner based on the first word of each command.
The base name is sought as a key in %scanners hash in the makefile's package,
which is seeded with all of the default scanners.
This hash can be extended using the register_scanner and
register_command_parser statements.
The value from this hash lookup is a coderef that either does the scanning
work (which is deprecated), or returns an object of type C<CommandParser>.

=back

In most cases, objects of type C<CommandParser> should instantiate at least
one object of type C<Scanner>.
The C<Scanner> base class takes care of the distinction between quickscan
and smartscan.
Note that the behavior of C<Scanner> can be markedly affected by this
distinction, but that should be transparent to the derived class if it
is well-formed.
New derived C<Scanner> classes ought to be tested in both modes.

If you want to specify a particular C<CommandParser> class for a rule, then
you can specify a scanner with a rule option, and arrange for the parser_*
subroutine to return an object of type C<ActionParser::Specific>.

For more details, refer to the respective class documentation.
For examples, see C<CommandParser::Gcc> and C<CommandParser::Vcs>.

=head2 Caching scanner info

If the all of the scanner's important side effects are effected through calls
to methods of the C<CommandParser> base class, then those side effects can
be cached in the build info file, so that they can be played back by a
subsequent invocation of makepp without doing all of the costly scanning work.
This can save quite a bit of time, especially in smartscan mode.

If the scanner has other important side effects, then it should call the
C<Rule> object's mark_scaninfo_uncacheable method.
Otherwise, the scanner info retrieved from the build info may be inaccurate,
causing the build result possibly to be incorrect.
This method is called automatically when a value from the %scanners hash does
not return an object of type C<CommandParser>, or when the scanner is specified
with a rule option, no parser_* subroutine is defined and the scanner_*
routine does not return an object of type C<CommandParser>.

Cached scan info is invalidated using criteria similar to those used for
determining when the target is out of date.
Similarly, it can be retrieved from a repository using criteria similar
to those used for determining when a target can be linked in from a
repository.

You can force makepp to ignore the cached scanner info with the
C<--force-rescan> option.
This is useful when a broken scanner may have caused incorrect scanner info
to be cached.

=head2 Ad Hoc Scanner

Often you will have just one or few files which contain dependency
information.  You don't want to write this into a makefile reduntantly (since
redundancy later often leads to inconsistencies when one update gets
forgotten).  But you also don't want to write a Scanner?  As a workaround you
can generate an include file on the fly.  For example Qt has F<.qrc> files
which can look like:

    <RCC>
      <qresource prefix="...">
 	<file>abc</file>
 	<file>xyz</file>
 	...

If you adhere to the above layout, you can transform the relevant lines into a
makepp include file, which gets automatically created by being included.

    %.qrc.makepp: %.qrc
 	&grep 's!<RCC>\n!$(stem).cc:! || s! *<file>! ! && s!</file>\n!!' $(input) -o $(output)

    include $(wildcard *.qrc)	# .makepp is appended automatically