Tapper User Manual


Next: , Previous: (dir), Up: (dir)

Table of Contents


Next: , Previous: Top, Up: Top

1 Synopsis


Next: , Previous: Synopsis, Up: Synopsis

1.1 Tapper infrastructure

Tapper is an infrastructure.

It consists of applications, tools and protocols for testing software and evaluating the results. One initial main focus was on testing Operating Systems in virtualization environments. It is now a modular infrastructure for lots of other, related scenarios, like benchmarking or build systems.

There are 3 important layers:

The layers can work completely autonomously, though can also be connected together and are targeted to be stacked in this order:

To fully exploit the system you need to learn:


Previous: Tapper infrastructure, Up: Synopsis

1.2 Vocabulary


Next: , Previous: Vocabulary, Up: Vocabulary

1.2.1 Master Control Program (MCP)

There is a central server controlling the automation by running the Master Control Program, aka. MCP. Usually it also centralizes several other services: it is the TFTP server for network booting, runs the daemons of the reports framework (reports receiver, remote api) and the web application, including the mysql databases, and also serves the file repository via NFS.


Next: , Previous: Vocabulary - Master Control Program (MCP), Up: Vocabulary

1.2.2 Program Run Control (PRC)

When machines run automated tests, these test program runs are controlled by a Program Run Control, aka. PRC. In virtualization scenarios, each host and guest has its own PRC, numbered PRC0 (for the host), PRC1 (1st guest), PRC2 (2nd guest), etc.


Next: , Previous: Vocabulary - Program Run Control (PRC), Up: Vocabulary

1.2.3 Reports Receiver

The Reports Receiver means the daemons that accept reports. We often run them on the same machine as the MCP and the Web framework, but that's not neccessary.


Next: , Previous: Vocabulary - Reports Receiver, Up: Vocabulary

1.2.4 Reports API

Similar to the reports receiver is the Reports API which is the daemon for all more complex interfaces, like uploading files, downloading files, querying the reports. Similar to reports API we often run them on the same machine as the MCP and the Web application, but that's not neccessary.


Next: , Previous: Vocabulary - Reports API, Up: Vocabulary

1.2.5 Web User Interface

The Web User Interface is an independent web application. Similar to the reports receiver and the reports API it can run anywhere, either standalone or in Apache, via mod_perl, FCGI, etc.. The only common thing for all those central applications (MCP, reports receiver, reports api, web application) is the config to use the same databases.


Next: , Previous: Vocabulary - Web User Interface, Up: Vocabulary

1.2.6 Reports DB

The Reports DB contains all data that are reported. It's the base for the reports receiver, the reports API, the web application.


Next: , Previous: Vocabulary - Reports DB, Up: Vocabulary

1.2.7 Testrun DB

The Testrun DB is the DB for the automation layer. It contains hosts, testrun specifications and scheduling information.


Next: , Previous: Vocabulary - Testrun DB, Up: Vocabulary

1.2.8 Testrun

A Testrun is a request to the automation layer to set up a host machine and run a workload on it. It consists of “preconditions” and scheduling information (host name, wanted host features, scheduling queue).


Next: , Previous: Vocabulary - Testrun, Up: Vocabulary

1.2.9 Preconditions

Preconditions are specifications that describe how to set up a host. They are the essential part of a testrun.


Next: , Previous: Vocabulary - Preconditions, Up: Vocabulary

1.2.10 Report

A Report is the reported result of any workload, regardless of how it was produced (automatically, by a tes suite, manually via echo and netcat). Its only requirement is to be formatted in TAP (the Test Anything Protocol), or as TAP archive.


Next: , Previous: Vocabulary - Report, Up: Vocabulary

1.2.11 Test Anything Protocol (TAP)

The Test Anything Protocol aka. TAP is the syntax to describe test results.


Previous: Vocabulary - Test Anything Protocol (TAP), Up: Vocabulary

1.2.12 TAP archives

A TAP archive is a .tar.gz file that contains files of TAP. It's the result of a test suite that can consist of many parts compressed into a single file.


Next: , Previous: Synopsis, Up: Top

2 Technical Infrastructure

See also the “Getting Started Guide” for more complete step-by-step instructions how to install the infrastructure from scratch up to a first example test run.


Previous: Technical Infrastructure, Up: Technical Infrastructure

2.1 Adding a new host into automation

This chapter describes what you need to do in order to get a new machine into the Tapper test scheduling rotation.


Next: , Previous: Adding a new host into automation, Up: Adding a new host into automation

2.1.1 Make machine remote hard resetable

Connect the machine physically to some facility to programmatically switch it completely off.

This can be the Reset cable wires connected to a dedicated reset box which can be programmed usually with an external tool. It can also be a TCP/IP controllable Power Control.

As an example Tapper comes with a plugin for the “Infratec PM211 MIP” ethernet controllable multi socket outlet. To use it write this in the configuration file:

reset_plugin: PM211MIP
reset_plugin_options:
  ip: 192.168.1.39
  user: admin
  passwd: secret
  outletnr:
    johnconnor: 1
    sarahconnor: 2

This configures to use the PM211MIP plugin for reset and gives it the configuration that the host “johnconnor” is connected on port 0 and the host “sarahconnor” on port 1, together with IP address, username and password of the multi-socket outlet.

If you have other hardware then write your own reset plugin FooBar in a Perl module Tapper::MCP::Net::Reset::FooBar. Look into the code of Tapper::MCP::Net::Reset::PM211MIP to get inspiration.


Next: , Previous: Make machine remote hard resetable, Up: Adding a new host into automation

2.1.2 Machine specific configuration

Tapper configuration happens in /etc/tapper.cfg which contains everything that needs to be overwritten from the default values that come with the library Tapper::Config.

Config entries for grub can be configured for defaults like this:

mcp:
  installer:
    default_grub: |
      serial --unit=0 --speed=115200
      terminal serial
      default 0
      timeout 2
      title Test run (Install)
        tftpserver $TAPPER_TFTPSERVER
        kernel $TAPPER_KERNEL root=/dev/nfs reboot=force ro ip=dhcp \
               nfsroot=$TAPPER_NFSROOT $TAPPER_OPTIONS $HOSTOPTIONS
  test:
    default_grub: |
      default 0
      timeout 2
      title Test run (Execute)
        root $grubroot
        kernel /boot/vmlinuz root=$root reboot=force ip=dhcp noapic \
               $TAPPER_OPTIONS $HOSTOPTIONS
        $initrd_options

You can see two entries, one for the installer to boot from TFTP, and one for the actual test run after the machine was installed. Usually the second part (for test run) is defined in a precondition because it needs special needs.

In these grub entries the MCP substitutes several variables before the actual grub entry is written:


Next: , Previous: Machine specific configuration, Up: Adding a new host into automation

2.1.3 Make machine PXE boot aware


Next: , Previous: Make machine PXE boot aware, Up: Adding a new host into automation

2.1.4 Configure TFTP on central MCP machine

The MCP server is also acting as a TFTP server, so it has to be configured as such:


Next: , Previous: Configure TFTP on central MCP machine, Up: Adding a new host into automation

2.1.5 Make the hosts known in the TestrunDB

 $ tapper-testrun newhost --name=sarahconnor --active=1
 $ tapper-testrun newhost --name=johnconnor  --active=1

This makes the hosts generally available (active) for scheduling testruns by machine name. For scheduling hosts by more detailed machine features (cpu, memory, family, etc.) you need to add according key/value pairs in the ‘HostFeature’ table.


Previous: Make the hosts known in the TestrunDB, Up: Adding a new host into automation

2.1.6 Optionally: enable ‘temare’ to generate tests for this host

Temare’ is an utility that generates preconditions according to a test matrix of host/guest virtualization scenarios (but not yet shipped publicly).

For generating preconditions for a host, you can register the host in ‘temare’.

If you want tests scheduled for the new machine then follow these steps:


Next: , Previous: Technical Infrastructure, Up: Top

3 Test Protocol

In order to write test suites you need to understand the output protocol, which is ‘TAP’, the ‘Test Anything Protocol’.

The protocol is trivially to produce, you can do it with simple Shell ‘echo’s or you can use TAP emitting toolchains, like practically all ‘Test::*’ modules from the Perl world.

This chapter explains the protocol and the Tapper specific extensions, which are usually headers that can be transported inside TAP comments.


Next: , Up: Test Protocol

3.1 Test Anything Protocol (TAP)


Next: , Previous: Test Anything Protocol (TAP), Up: Test Protocol

3.2 Tutorial


Next: , Previous: Tutorial, Up: Tutorial

3.2.1 Just plan and success

Example:

 1..3
 ok
 ok
 not ok

Remarks:


Next: , Previous: Just plan and success, Up: Tutorial

3.2.2 Succession numbers

Example:

 1..3
 ok 1
 ok 2
 not ok 3

Remarks:

Example with missing test:

 1..3
 ok 1
 not ok 3

Remarks:


Next: , Previous: Succession numbers, Up: Tutorial

3.2.3 Test descriptions

Example:

 1..3
 ok 1 - input file opened
 ok 2 - file content
 not ok 3 - last line

Remarks:


Next: , Previous: Test descriptions, Up: Tutorial

3.2.4 Mark tests as TODO

Example:

 1..3
 ok 1 - input file opened
 ok 2 - file content
 not ok 3 - last line # TODO

Remarks:


Next: , Previous: Mark tests as TODO, Up: Tutorial

3.2.5 Comment TODO tests with reason

Example:

 1..3
 ok 1 - input file opened
 ok 2 - file content
 not ok 3 - last line # TODO just specced

Remarks:


Next: , Previous: Comment TODO tests with reason, Up: Tutorial

3.2.6 Mark tests as SKIP (with reason)

Example:

 1..3
 ok 1 - input file opened
 ok 2 - file content
 ok 3 - last line # SKIP missing prerequisites

Remarks:


Next: , Previous: Mark tests as SKIP (with reason), Up: Tutorial

3.2.7 Diagnostics

Example:

 1..3
 ok 1 - input file opened
 ok 2 - file content
 not ok 3 - last line # TODO just specced
 # Failed test 'last line'
 # at t/data_dpath.t line 410.
 # got: 'foo'
 # expected: 'bar'

Remarks:


Next: , Previous: Diagnostics, Up: Tutorial

3.2.8 YAML Diagnostics

Example:

 1..3
 ok 1 - input file opened
 ok 2 - file content
 not ok 3 - last line # TODO just specced
   ---
   message: Failed test 'last line' at t/data_dpath.t line 410.
   severity: fail
   data:
     got: 'foo'
     expect: 'bar'
   ...

Remarks:


Next: , Previous: YAML Diagnostics, Up: Tutorial

3.2.9 Meta information headers for reports

TAP allows comment lines, starting with ‘#’. We allow meta information transported inside those comment lines when declared with Tapper specific headers.

Example:

 1..3
 # Tapper-Suite-Name: Foo-Bar
 # Tapper-Suite-Version: 2.010013
 ok 1 - input file opened
 ok 2 - file content
 not ok 3 - last line # TODO just specced

Remarks:

These are the headers that apply to the whole report:

 # Tapper-suite-name:                 -- suite name
 # Tapper-suite-version:              -- suite version
 # Tapper-machine-name:               -- machine/host name
 # Tapper-machine-description:        -- more details to machine
 # Tapper-reportername:               -- user name of the reporter
 # Tapper-starttime-test-program:     -- start time for complete test
                                          (including guests)
 # Tapper-endtime-test-program:       -- end time for complete test
                                          (including guests)
 # Tapper-reportgroup-testrun:        -- associate this report with other 
                                          reports of same testrun_id
 # Tapper-reportgroup-arbitrary:      -- associate this report with other 
                                          reports of same arbitrary id
                                          (can be any string, but should be
                                          unique between all groups of the db,
                                          eg., an md5-hash of common 
                                          characteristics of all test of one 
                                          group)

There are more headers that apply to single sections of a report.


Next: , Previous: Meta information headers for reports, Up: Tutorial

3.2.10 Report sections

Standard TAP contains of exactly one block with one plan (eg., 1..5) and some TAP lines. In Tapper you can concatenate several such blocks at once. They are interpreted like different files, and are named sections in Tapper jargon.

The delimiter between such sections is the plan line. This requires the plan to come first for each section. See chapters “Explicit section markers with lazy plans” and “TAP archives” below for explicitely providing other TAP section delimiters.

Please remember: Concatenating several sections into one big block of TAP is a Tapper extension. To interact with other TAP toolchains you should try to use “TAP archives” when submitting sections into Tapper.

Example:

 1..2
 # Tapper-section: arithmetics
 ok 1 add
 ok 2 multiply
 1..1
 # Tapper-section: string handling
 ok 1 concat
 1..3
 # Tapper-section: benchmarks
 ok 1
 ok 2
 ok 3

Remarks:


Next: , Previous: Report sections, Up: Tutorial

3.2.11 Meta information headers for report sections

These are the headers that apply to single sections:

 # Tapper-explicit-section-start:   -- explicitely start a section now
                                        instead of autorecognition
 # Tapper-ram:                      -- memory
 # Tapper-cpuinfo:                  -- what CPU
 # Tapper-uname:                    -- kernel information
 # Tapper-osname:                   -- OS information
 # Tapper-bios:                     -- BIOS information
 # Tapper-flags:                    -- flags, usually linux kernel
 # Tapper-changeset:                -- exact changeset of the currently 
                                        tested software or kernel
 # Tapper-description:              -- more description of the currently
                                        tested software or kernel, 
                                        e.g., if changeset is not enough
 # Tapper-uptime:                   -- uptime, maybe the test run time
 # Tapper-language-description:     -- for Software tests, 
                                        like "Perl 5.10", "Python 2.5"
 # Tapper-reportcomment:            -- Freestyle comment

 # Tapper-xen-version:              -- Xen version
 # Tapper-xen-changeset:            -- particular Xen changeset
 # Tapper-xen-dom0-kernel:          -- the kernel version of the dom0
 # Tapper-xen-base-os-description:  -- more verbose OS information
 # Tapper-xen-guest-description:    -- description of a guest
 # Tapper-xen-guest-test:           -- the started test program
 # Tapper-xen-guest-start:          -- start time of test
 # Tapper-xen-guest-flags:          -- flags used for starting the guest

 # Tapper-kvm-module-version:       -- version of KVM kernel module
 # Tapper-kvm-userspace-version:    -- version of KVM userland tools
 # Tapper-kvm-kernel:               -- version of kernel
 # Tapper-kvm-base-os-description:  -- more verbose OS information
 # Tapper-kvm-guest-description:    -- description of a guest
 # Tapper-kvm-guest-test:           -- the started test program
 # Tapper-kvm-guest-start:          -- start time of test
 # Tapper-kvm-guest-flags:          -- flags used for starting the guest

 # Tapper-simnow-version:           -- version of simnow
 # Tapper-simnow-svn-version:       -- svn commit id of simnow
 # Tapper-simnow-svn-repository:    -- used svn repository
 # Tapper-simnow-device-interface-version: -- internal simnow device 
                                               interface version
 # Tapper-simnow-bsd-file:          -- used BSD file (machine model)
 # Tapper-simnow-image-file:        -- used OS image botted in simnow
                                        (usually similar to
                                        Tapper-osname or
                                        Tapper-xen-base-os-description or
                                        Tapper-kvm-base-os-description)


Next: , Previous: Meta information headers for report sections, Up: Tutorial

3.2.12 Meta information structure summary

There are groups of reports (e.g. for virtualization scenarios), optionally identified by a testrun ID or by an arbitrary ID. Every report has an ID and a set of meta information. A report consists of sections, which can each have section specific set of meta information.

The resulting meta information hierarchy looks like this.


Next: , Previous: Meta information structure summary, Up: Tutorial

3.2.13 Explicit section markers with lazy plans

In TAP it is allowed to print the plan (1..n) after the test lines (a “lazy plan”). In our Tapper environment with concatenated sections this would break the default section splitting which uses the plan to recognize a section start.

If you want to use such a “lazy plan” in your report you can print an Tapper header Tapper-explicit-section-start to explictely start a section. Everything until the next header Tapper-explicit-section-start is building one section. This also means that if you used this header once in a report you need to use it for all sections in this report.

The Tapper-explicit-section-start typically ignores its value but it is designed anyway to allow any garbage after the value that can help you visually structure your reports because explicit sections with “lazy plans” make a report hard to read.

Example:

 # Tapper-explicit-section-start: 1 ------ arithmetics -------
 # Tapper-section: arithmetics
 ok 1 add
 ok 2 multiply
 1..2
 # Tapper-explicit-section-start: 1 ------ string handling -------
 # Tapper-section: string handling
 ok 1 concat
 1..1
 # Tapper-explicit-section-start: 1 ------ benchmarks -------
 # Tapper-section: benchmarks
 ok 1
 ok 2
 ok 3
 1..3

Please note again: The sectioning in general and this auxiliary header for marking sections is a Tapper extension, not standard TAP. An alternative way better than fiddling with this sectioning is to produce TAP archives and submit them instead. See chapter “TAP Archives”.


Next: , Previous: Explicit section markers with lazy plans, Up: Tutorial

3.2.14 Developing with TAP

TAP consuming is provided via the Test::Harness aka. TAP::Parser Perl toolchain. The frontend utility to execute TAP emitting tests and evaluate statistics is prove.

 $ prove t/*.t
 t/00-load.........ok
 t/boilerplate.....ok
 t/pod-coverage....ok
 All tests successful.
 Files=4, Tests=6, 0 wallclock secs
 ( 0.05 usr 0.00 sys + 0.28 cusr 0.05 csys = 0.38 CPU)
 Result: PASS

Remarks:

It helps to not rely on Tapper extensions (like report sections) when using the prove command.


Previous: Developing with TAP, Up: Tutorial

3.2.15 TAP tips

These tips keep later TAP evaluation consistent.


Next: , Previous: Tutorial, Up: Test Protocol

3.3 Particular use-cases


Previous: Particular use-cases, Up: Particular use-cases

3.3.1 Report Groups


Next: , Previous: Report Groups, Up: Report Groups
3.3.1.1 Report grouping by same testrun

If we have a Xen environment then there are many guests each running some test suites but they don't know of each other.

The only thing that combines them is a common testrun-id. If each suite just reports this testrun-id as the group id, then the receiving side can combine all those autonomously reporting suites back together by that id.

So simply each suite should output

 # Tapper-reportgroup-testrun: 1234

with 1234 being a testrun ID that is available via the environment variable $TAPPER_TESTRUN. This variable is provided by the automation layer.


Previous: Report grouping by same testrun, Up: Report Groups
3.3.1.2 Report grouping by arbitrary idenitifier

If the grouping id is not a testrun id, e.g., because you have set up a Xen environment without the Tapper automation layer, then generate one random value once in dom0 by yourself and use that same value inside all guests with the following header:

How that value gets from dom0 into the guests is left as an exercise, e.g. via preparing the init scripts in the guest images before starting them. That's not the problem of the test suite wrappers, they should only evaluate the environment variable TAPPER_REPORT_GROUP.


Next: , Previous: Particular use-cases, Up: Test Protocol

3.4 TAP Archives

Some TAP emitting toolchains allow the generation of .tar.gz files containing TAP, so called TAP archives. E.g., via ‘prove’:

 $ prove -a /tmp/myresults.tgz t/

You can later submit such TAP archive files to the Tapper reports receiver tha same way as you report raw TAP.


Previous: TAP Archives, Up: Test Protocol

3.5 Reporting TAP to the reports receiver

The Tapper reports receiver is a daemon that listens on a port and slurps in everything between the open and close of a connection to it. Therefore you can use ‘netcat’ to report TAP.

Remember that using ‘netcat’ in turn can be a mess, the are several flavours with different options which are also changing their behaviour over time. So to be sure, you better do your own socket communication with Perl or Python: open socket, print to socket, close socket, done. We just keep with ‘netcat’ for illustrating the examples.


Next: , Previous: Reporting TAP to the reports receiver, Up: Reporting TAP to the reports receiver

3.5.1 Submitting raw TAP

Simply submit all TAP directly into the socket of the reports receiver:

 $ ./my_tap_emitting_test_suite | netcat tapper_server 7357


Previous: Submitting raw TAP, Up: Reporting TAP to the reports receiver

3.5.2 Submitting TAP Archives

You submit the content of a .tar.gz file in the same way you submit raw TAP, via the same API. The receiver recognizes the .tar.gz contenttype by itself.

 $ prove -a /tmp/myresults.tgz t/
 $ cat /tmp/myresults.tgz | netcat tapper_server 7357


Next: , Previous: Test Protocol, Up: Top

4 Test Suite Wrappers

This section is about the test suites and wrappers around existing suites. These wrappers are part of our overall test infrastructure.

It's basically about the middle part in the following picture:

tapper_architecture_overview.png

We have wrappers for existing test and benchmark suites.

Wrappers just run the suites as a user would manually run them but additionally extract results and produce TAP (Test Anything Protocol).

We have some specialized, small test suites that complement the general suites, e.g. for extracting meta information or parsing logs for common problems.

If the environment variables

TAPPER_REPORT_SERVER
TAPPER_REPORT_PORT

are set the wrappers report their results by piping their TAP output there, else they print to STDOUT.


Next: , Previous: Test Suite Wrappers, Up: Test Suite Wrappers

4.1 Available test suite wrappers

Originally we have a lot of direct wrappers available but haven't them all published as open source. For OS testing the most important wrapper which is also publicly available is tapper-testsuite-autotest aka. Tapper-Testsuite-AutoTest. You should look at that.

4.1.1 tapper-testsuite-autotest

A suite that wraps the autotest client with the export of TAP and sends the resulting TAP archives to Tapper server.

That is the primary testsuite wrapper for OS testing.


Previous: Available test suite wrappers, Up: Test Suite Wrappers

4.2 Environment variables

The TAPPER automation layer provides some environment variables that the wrappers can use:

TAPPER_TESTRUN
Currently active Testrun ID.
TAPPER_SERVER
The controlling automation Server that initiated this testrun.
TAPPER_REPORT_SERVER
The target server to which the tests should report their results in TAP.
TAPPER_REPORT_PORT
The target port to which the tests should report their results in TAP. Complements TAPPER_REPORT_SERVER.
TAPPER_REPORT_API_PORT
The port on which the more sophisticated Remote Reports API is available. It's running on the same host as TAPPER_REPORT_SERVER.
TAPPER_TS_RUNTIME
Maximum runtime after which the testprogram will not be restarted when it runs in a loop. (This is a more passive variant than a timeout.)
TAPPER_GUEST_NUMBER
Virtualisation guests are ordered, this is the guest number or 0 if not a guest.
TAPPER_NTP_SERVER
The server where to request NTP dates from.

These variables should be used in the TAP of the suite as Tapper headers. Important use-case is "report groups", see next chapter.


Next: , Previous: Test Suite Wrappers, Up: Top

5 Preconditions

The central thing that is needed before a test is run is a so called precondition. Creating those preconditions is the main task needed to do when using the automation framework.

Most of the preconditions describe packages that need to be installed. Other preconditions describe how subdirs should be copied or scripts be executed.

A precondition can depend on other preconditions, leading to a tree of preconditions that will be installed from the leaves to the top.


Next: , Previous: Preconditions, Up: Preconditions

5.1 SYNOPSIS

5.2 Layers of preconditions

5.3 Precondition repository

5.4 Overview Precondition Types

5.5 Details Precondition Types

5.6 Macro Preconditions

5.7 Testplans

5.8 Precondition Producers


Next: , Previous: SYNOPSIS, Up: Preconditions

5.9 Layers of preconditions

There are “normal preconditions” and “macro preconditions”.

5.9.1 Normal preconditions

We store preconditions in the database and assign testruns to them (also in the database).

Usually the preconditions were developed in a (temporary) file and then entered into the database with a tool. After that the temporary file can be deleted. Note that such a precondition file can contain multiple precondition as long as they are formated as valid YAML.

Preconditions can be kept in files to re-use them when creating testruns but that's not needed for archiving purposes, only for creation purposes.

Please note: Normal preconditions are usually not what you want. It's the low level mechanism. Its advantage is in reusing the preconditions by referring to IDs and creating trees of preconditions. This reuse is usually too complex. What you typically want are Macro Preconditions.

5.9.2 Macro preconditions

There is another mechanism on top of normal preconditions: Macro Preconditions. These allow to bundle multiple preconditions at once into a common use-case.

A macro precondition is evaluated when the testrun is added via the cmdline utils (or the web app, both use the same underlying layer). The result are “normal preconditions” which are inserted into the DB everytime together with the testrun, so there is no reuse of preconditions and preconditions are always a list, no tree. Anyhow, they are much easier to handle.

Macro preconditions are template files which should be archived in the precondition repository, as only the finally resulting preconditions are stored in the database.

5.9.3 Testplans

Testplans are a variant of macro preconditions to control multiple testruns and queue them to particular hosts and queues. They are mostly useful to provide end-users with complex but polished frontend use-cases where they can define values to prepared parameters.


Next: , Previous: Layers of preconditions, Up: Preconditions

5.10 Precondition repository

Macro preconditions can be stored in

/data/tapper/live/repository/macropreconditions/


Next: , Previous: Precondition repository, Up: Preconditions

5.11 Overview Precondition Types

There are two variants of preconditions: Action preconditions and Highlevel preconditions. Action preconditions describe single actions, like “copy a file” or “execute a program”. Highlevel preconditions can contain other (action) preconditions and are used for instance for virtualization install scenarios where hosts and guests are described.

Please note the wording: A precondition is the particular YAML block with all the details (think of an object instance). Such a block is of a “precondition type” which defines its allowed structure (think of a class).

5.11.1 Overview: Action precondition types

The following action precondition types are allowed:

package
A package (kernel, library, etc.), of type .tar, .tar.gz or .tar.bz2
image
A complete OS image of type .iso, .tar.gz, .tgz, .tar, .tar.bz2
prc
Create a config for the PRC module of the automation layer.
copyfile
One file that can just be copied/rsync'd
installer_stop
Don't reboot machine after system installer finished
grub
Overwrite automatically generated grub config with one provided by the tester
fstab
Append a line to /etc/fstab
repository
Fetch data from a git, hg or svn repository
exec
Execute a script during installation phase
reboot
Requests a reboot test and states how often to reboot.

5.11.2 Overview: Highlevel precondition types

Currently only the following high level precondition type is allowed:

virt
Generic description for Xen or KVM

High level preconditions both define stuff and can also contain other preconditions.

They are handled with some effort to Do The Right Thing, i.e., a defined root image in the high level precondition is always installed first. All other preconditions are installed in the order defined by its tree structure (depth-first).


Next: , Previous: Overview Precondition Types, Up: Preconditions

5.12 Details Precondition Types

We describe preconditions in YAML files (http://www.yaml.org/).

All preconditions have at least a key

     precondition_type: TYPE

and optionally

     name: VERBOSE DESCRIPTION
     shortname: SHORT DESCRIPTION

then the remaining keys depend on the TYPE.

5.12.1 installer_stop

stop run after system installer

     ---
     precondition_type: installer_stop

5.12.2 grub

overwrite automatically generated grub config

     ---
     precondition_type: grub
     config: |
       title Linux
         root $grubroot
         kernel /boot/vmlinuz root=$root"

5.12.3 package

     ---
     precondition_type: package
     filename: /data/tapper/live/repository/packages/linux/linux-2.6.27.7.tar.bz2

5.12.4 copyfile

a file that just needs to be scp or copied:

     ---
     precondition_type: copyfile
     protocol: nfs
     source: osko:/export/image_files/official_testing/README
     dest: /usr/local/share/tapper/

5.12.5 fstab

a line to add to /etc/fstab, e.g., to enable mounts once the system boots

     ---
     precondition_type: fstab
     line: "165.204.85.14:/vol/osrc_vol0 /home nfs auto,defaults 0 0"

5.12.6 image

usually the root image that is unpacked to a partition (this is in contrast to a guest file that's just there)

     ---
     precondition_type: image
     mount: /
     partition: testing
     image: /data/tapper/live/repository/images/rhel-5.2-rc2-32bit.tgz

5.12.7 repository

     ---
     precondition_type: repository
     type: git
     url: git://git.kernel.org/pub/scm/linux/kernel/git/avi/kvm.git
     target: kvm
     revision: c192a1e274b71daea4e6dd327d8a33e8539ed937

5.12.8 type: prc

Is typically contained implicitely with the abstract precondition virt. But can also be defined explicitely, e.g., for kernel tests.

Creates config for PRC. This config controls what is to be run and started when the machine boots.

     precondition_type: prc
     config:
          runtime: 30
          test_program: /bin/uname_tap.sh
          timeout_after_testprogram: 90
          guests:
           - svm:  /xen/images/..../foo.svm
           - svm:  /xen/images/..../bar.svm
           - exec: /xen/images/..../start_a_kvm_guest.sh

5.12.16 General precondition keys “mountfile”

These 2 options are possible in each precondition. With that you can execute the precondition inside guest images:

     mountfile: ...
     mountpartition: ...
     mounttype: @TODO{is this the same as mountfile, mountpartition?}


Next: , Previous: Details Precondition Types, Up: Preconditions

5.13 Macro Preconditions

This section describes macro precondition files as they are stored in /data/tapper/live/repository/macropreconditions/.

A macro precondition denotes a file containing one or multiple preconditions and additional TemplateToolkit code.

In most cases “normal preconditions” for similar tests will only differ in one or very few keys. Thus precondition files could easily be reused by only changing these few keys. This is made easier with using “macro preconditions”.

The macro precondition file should contain all “normal preconditions” to be reused. All variable keys should be substituted by appropriate TemplateToolkit variables. When creating the new testrun actual values for these TemplateToolkit variables have to provided.

Macro preconditions are not stored in the database. They are only a tool to ease the creation of preconditions. Only the resulting preconditions are stored in database.

To make parsing macro preconditions easier required and optional fields can be named after a comment field in the first lines of the file after the keys tapper-mandatory-fields and tapper-optional-fields respectively as in the following example:

     # tapper-mandatory-fields: id
     # tapper-optional-fields: kernel

These # tapper-* headers are also used in web frontend to render forms out of it and submit testruns from there.

The values for the placeholders can be filled via such a command line:

     $ tapper-testrun new [all usual options] \
          --macroprecond=FILENAME \
           -Did=value1 \
           -Dkernel=2.6.37

The FILENAME is a complete filename with absolute path.

There is no restriction on TemplateToolkit code for variable substitution. The following example could be used to generate a default value for the precondition key id.

     [%id = BLOCK%][%IF id%][%id%][%ELSE%]2009-06-29-perfmon[%END%][%END%]


Previous: Macro Preconditions, Up: Macro Preconditions

5.13.1 A real live example - kernel boot test


Next: , Previous: Macro Preconditions, Up: Preconditions

5.14 Testplans

This section describes testplan files as they are stored in /data/tapper/live/testplans/.

A testplan is very similar to a macro precondition but can additionally bundle several testruns and specify required hostname and used queue inside. This and the possibility of using Template-Toolkit (see Macro Preconditions) inside makes it most useful for frontend usage: You can take all parameters, especially queue+hosts from commandline via -Dhosts=affe,zomtec,birne, split them via Temaplte Toolkit and loop to create multiple testruns.

To tell it's one test out of a set, we use the type multitest, like this:

     ---
     type: multitest
     description:
       shortname: my-testplan
       topic: my-testplan
       queue: AdHoc
       requested_hosts_all:
       - affe
       - zomtec
       preconditions:
       - ...
       - ...

The preconditions are then as already described in earlier chapters.

It additionally allows to embed self-documentation which can be viewd with the command tapper-testrun newtestplan --guide --file my-testplan-file.

To take parameters and create multiple testruns, you could use it like this:

     ### This is an example testplan
     ### Allowed params:
     ###  - -Dqueue=QUEUE
     ###  - -Dtests=TEST1,TEST2,TEST3
     ###  - -Dmachines=HOST1,HOST2
     [%# define default values %]
     [%- IF queue == '' %][% queue = 'AdHoc' %][% END -%]
     [%- IF tests == '' %][% tests = 'hackbench' %][% END -%]
     [%- IF machines == '' %][% machines = 'arges' %][% END -%]
     [% AllTests = tests.split(',') %]
     [% AllDistros = distros.split(',') %]
     [% AllMachines = machines.split(',') %]
     [%- FOREACH machine = AllMachines %]
     [%- FOREACH test = AllTests %]
     ---
     type: multitest
     description:
       shortname: [% test %]
       topic: Topic-[% AllTests.join('-') %]
       queue: [% queue %]
       requested_hosts_all:
       - [% machine %]
       preconditions:
       - ...
       -
         precondition_type: testprogram
         program: /opt/tapper/bin/tapper-testsuite-autotest
         parameters:
           - --test
           - [% test %]
       - ...

From a command line this can be used like below.

Show self-doc:

      $ tapper-testrun newtestplan --guide --file my-testplan
     
        This is an example testplan
        Allowed params:
         - -Dqueue=QUEUE
         - -Dtests=TEST1,TEST2,TEST3
         - -Dmachines=HOST1,HOST2

Start a new testplan consisting of potentially many testruns:

      $ tapper-testrun newtestplan --verbose \
                       --file my-testplan \
                       -Dqueue=TrackWorkload \
                       -Dtests=hackbench,dbench
     
        Plan created
          id:
          241
          url: http://tapper/tapper/testplan/id/241
          path: topic/kernel/misc/my-testplan
          file: /data/tapper/topic/kernel/misc/my-testplan

Whenever you want to provide a polished use-case to an end-user consider writing a testplan file. You add all the complexity, details and documentation into the file and provide your user with just this file. He/she can then specify the prepared parameters for his/her particular needs, like the tests to run.

We additionally use Testplans to bi-directional map from such testplans to a task in our planning software where we simply assume a mapping of the task hierarchy to a subdirectory hierarchy.

Based on this mapping the planning software can specify when to execute such a testplan according to the project plan.

A testplan then typically contains several testruns: some for the known-good sanity tests without a changed feature and some for the actual tested feature.


Previous: Testplans, Up: Preconditions

5.15 Precondition Producers

Sometimes, parameters for preconditions shall be defined when the testrun, this precondition is assigned to, is choosen for execution. This might apply for example when you want to test the newest build of a certain package. Also in combination with autorerun testruns dynamic assignment of preconditions is useful. These testruns are reinserted into the database automatically as soon as the scheduler chooses them for execution. In this case dynamic precondition assignment allows these rerun tests to differ slightly. Preconditions with dynamically assigned parameters are called Lazy Precondition.

Dynamic precondition assignment is implemented using Precondition Producers. A producer is a modul that is called by the scheduler for handling of lazy preconditions. To use a lazy preconditio the user has to assign a precondition of type ‘producer’ to the testrun. This precondition has to contain the basename of an existing producer module and may contain additional parameters. The producer will substitute the ‘producer’ precondition with a normal precondition that has values assigned to all parameters.

5.15.1 Lazy precondition

Lets assume for example that you want to include the newest kernel package into your test. This can be achieved with the existing “Kernel” producer. Instead of a precondition of type ‘package’ with a certain filename you should assign the following precondition to your testrun.

     precontition_type: producer
     producer: Kernel

This precondition will be substituted with a package precondition that has the latest Sysint kernel build set as filename.

5.15.2 Producer API

Producer are modules loaded into the scheduler. Thus they need to be written in Perl and reside inside the Tapper::MCP::Scheduler::PreconditionProducer:: namespace. A producer has to implement a method ‘produce’. This function gets a job object as first parameter and a hash containing all additional options from the precondition as second parameter. It suggested that each producer inherits from Tapper::MCP::Scheduler::PreconditionProducer. Producers shall return a hash that has the produced preconditions as YAML text assigned to the hash key precondition_yaml. An optional key topic allows the producer to set the topic for the test. If the hash key error is set, the associated error string is reported and the testrun is cancled. In this case the other hash keys are not evaluated.

5.15.3 Existing producers

Currently the following producers exist:

5.15.4 Example: “Kernel” precondition producer

The kernel producer returns a package precondition that contains the latest kernel package from the kernel package path. Furthermore, it returns an exec precondition that triggers the creation of an initrd file for the kernel.

Valid options:

5.15.4.1 Lazy precondition

The lazy precondition, pointing to the “Kernel” precondition producer:

     precondition_type: produce
     producer: Kernel
     arch: i686
     version: 2.6.32
     stable: 1
5.15.4.2 Resulting preconditions

The resulting preconditions may look like this:

     ---
     precondition_type: package
     filename: kernel/stable/i686/kernel-2.6.32-rc1.tar.gz
     ---
     precondition_type: exec
     filename: /bin/gen_initrd.sh
     options:
       - 2.6.32-rc1


Next: , Previous: Preconditions, Up: Top

6 Command line interface


Next: , Previous: Command line interface, Up: Command line interface

6.1 Commandline Synopsis


Previous: Commandline Synopsis, Up: Command line interface

6.2 Scheduling


Next: , Previous: Scheduling, Up: Scheduling

6.2.1 Create new queue and new host and bind both together

Done.


Next: , Previous: Create new queue and new host and bind both together, Up: Scheduling

6.2.2 Change queue priority


Next: , Previous: Change queue priority, Up: Scheduling

6.2.3 requested features

Hosts for testruns can be choosen based on requested features. Supported features are:


Previous: requested features, Up: Scheduling

6.2.4 Cancel current testrun on host

Freeing a host need the config for the currently running testrun. Thus, the command is only tested on bancroft and may not work on other machines.

     $ tapper-testrun freehost \
                      --name=bullock\
                      --desc='I need this host right now'


Next: , Previous: Command line interface, Up: Top

7 Web User Interface

The Web User Interface is a frontend to the Reports database. It allows to overview reports that came in from several machines, in several test suites.

It can filter the results by dates, machines or test suite, gives colorful (RED/YELLOW/GREEN) overview about success/failure ratios, allows to zoom into details of single reports.

To evaluate reported test results in a more programmatic way, have a look into the DPath Query Language that is part of the Reports API.


Next: , Previous: Web User Interface, Up: Web User Interface

7.1 Usage

The main URL is typically somerthing like

     http://tapper/tapper


Previous: Usage, Up: Web User Interface

7.2 Understanding Tapper Details

7.2.1 Part 1 Overview

7.2.2 Part 2 Details

7.2.3 Part 3 Testrun


Next: , Previous: Web User Interface, Up: Top

8 Reports API


Next: , Previous: Reports API, Up: Reports API

8.1 Overview

There runs yet another daemon, the so called Tapper::Reports::API, on the same host where already the TAP Receiver runs. This ‘Reports API’ is meant for everything that needs more than just dropping TAP reports to a port, e.g., some interactive dialog or parameters.

This Tapper::Reports::API listens on Port 7358. Its API is modeled after classic unix script look&feel with a first line containing a description how to interpret the rest of the lines.

The first line consists of a shebang (#!), a api command and command parameters. The rest of the file is the payload for the api command.

The syntax of the ‘command params’ varies depending on the ‘api command’ to make each command intuitively useable. Sometimes they are just positional parameters, sometimes they look like the start of a HERE document (i.e., they are prefixed with << as you can see below).


Next: , Previous: Overview, Up: Reports API

8.2 Raw API Commands

In this section the raw API is described. That's the way you can use without any dependencies except for the minimum ability to talk to a port, e.g., via netcat.

See section tapper-api for how to use a dedicated command line utility that makes talking to the reports API easier, but is a dependency that might not be available in your personal test environment.


Next: , Previous: Raw API Commands, Up: Raw API Commands

8.2.1 upload aka. attach a file to a report

This api command lets you upload files, aka. attachments, to reports. These files are available later through the web interface. Use this to attach log files, config files or console output.


Next: , Previous: upload aka. attach a file to a report, Up: upload aka. attach a file to a report
8.2.1.1 upload Synopsis
#! upload REPORTID FILENAME [ CONTENTTYPE ]
payload


Next: , Previous: upload Synopsis, Up: upload aka. attach a file to a report
8.2.1.2 Parameters


Next: , Previous: Parameters, Up: upload aka. attach a file to a report
8.2.1.3 upload Payload

The raw content of the file to upload.


Previous: upload Payload, Up: upload aka. attach a file to a report
8.2.1.4 upload Example usage

Just echo the first api-command line and then immediately cat the file content:

$ ( echo "#! upload 552 xyz.tmp" ; cat xyz.tmp ) | netcat -w1 bascha 7358


Next: , Previous: upload aka. attach a file to a report, Up: Raw API Commands

8.2.2 download - download a file which is attached to a report

This api command lets you download files, aka. attachments, from reports.


Next: , Previous: download - download a file which is attached to a report, Up: download - download a file which is attached to a report
8.2.2.1 download Synopsis
#! upload REPORTID FILENAME

There is no other payload neccessary here, just this single line.


Next: , Previous: download Synopsis, Up: download - download a file which is attached to a report
8.2.2.2 download Parameters


Previous: download Parameters, Up: download - download a file which is attached to a report
8.2.2.3 download Example usage

Just echo the first api-command line and redirect the answer into a file.

$ ( echo "#! download 552 xyz.tmp" ) | netcat -w1 bascha 7358 > xyz.tmp


Previous: download - download a file which is attached to a report, Up: Raw API Commands

8.2.3 mason - Render templates with embedded query language

To query report results we provide sending templates to the API in which you can use a query language to get report details: This api-command is called like the template engine so that we can provide other template engines as well.


Next: , Previous: mason - Render templates with embedded query language, Up: mason - Render templates with embedded query language
8.2.3.1 mason Synopsis
#! mason debug=0 <<ENDMARKER
payload
ENDMARKER


Next: , Previous: mason Synopsis, Up: mason - Render templates with embedded query language
8.2.3.2 mason Parameters


Next: , Previous: mason Parameters, Up: mason - Render templates with embedded query language
8.2.3.3 mason Payload

A mason template.

Mason is a template language, see http://masonhq.com. Inside the template we provide a function reportdata to access report data via a query language. See section Query language for details about this.


Previous: Payload, Up: mason - Render templates with embedded query language
8.2.3.4 Example usage

This is a raw Mason template:

 % my $world = "Mason World";
 Hello <% $world %>!
 % my @res = reportdata '{ "suite.name" => "perfmon" } :: //tap/tests_planned';
 Planned perfmon tests:
 % foreach (@res) {
    <% $_ %>
 % }

If you want to submit such a Mason template you can add the api-command line and the EOF marker like this:

$ EOFMARKER="MASONTEMPLATE".$$
$ payload_file="perfmon_tests_planned.mas"
$ ( echo "#! mason <<$EOFMARKER" ; cat $payload_file ; echo "$EOFMARKER" ) \
    | netcat -w1 bascha 7358

The output of this is the rendered template. You can extend the line to save the rendered result into a file:

$ ( echo "#! mason <<$EOFMARKER" ; cat $payload_file ; echo "$EOFMARKER" ) \
    | netcat -w1 bascha 7358 > result.txt

The answer for this looks like this:

Hello Mason World!
Planned perfmon tests:
   3
   4
   17


Next: , Previous: Raw API Commands, Up: Reports API

8.3 Query language DPath

The query language, which is the argument to the reportdata as used embedded in the ‘mason’ examples above:

 reportdata '{ "suite.name" => "perfmon" } :: //tap/tests_planned'

consists of 2 parts, divided by the ‘::’.

We call the first part in braces reports filter and the second part data filter.


Next: , Previous: Query language DPath, Up: Query language DPath

8.3.1 Reports Filter (SQL-Abstract)

The reports filter selects which reports to look at. The expression inside the braces is actually a complete SQL::Abstract expression (http://search.cpan.org/~mstrout/SQL-Abstract/) working internally as a select in the context of the object relational mapper, which targets the table Report with an active JOIN to the table Suite.

All the matching reports are then taken to build a data structure for each one, consisting of the table data and the parsed TAP part which is turned into a data structure via TAP::DOM (http://search.cpan.org/~schwigon/TAP-DOM/).

The data filter works then on that data structure for each report.


Next: , Previous: Reports Filter (SQL-Abstract), Up: Reports Filter (SQL-Abstract)
8.3.1.1 SQL::Abstract expressions

The filter expressions are best described by example:


Previous: SQL-Abstract expressions, Up: Reports Filter (SQL-Abstract)
8.3.1.2 The data structure


Next: , Previous: Reports Filter (SQL-Abstract), Up: Query language DPath

8.3.2 Data Filter

The data structure that is created for each report can be evaluated using the data filter part of the query language, i.e., everything after the ::. This part is passed through to Data::DPath (http://search.cpan.org/~schwigon/Data-DPath/).


Previous: Data Filter, Up: Data Filter
8.3.2.1 Data-DPath expressions


Previous: Data Filter, Up: Query language DPath

8.3.3 Optimizations

Using the query language can be slow. The biggest slowdown occurs with the ‘ANYWHERE’ element //, again with several of them, because they span up a big search tree.

Therefore, if you know the depth of your path, try to replace the // with some * because that only spans up on the current step not every possible step, like this:

  { ... } :: //section/stats-proc-interrupts-before//tap//data/TLB";
  { ... } :: /results/*/section/stats-proc-interrupts-before/tap/lines/*/_children/*/data/TLB";


Previous: Query language DPath, Up: Reports API

8.4 Client Utility tapper-api

There is a command line utility tapper-api that helps with using the API without the need to talk the protocol and fiddle with netcat by yourself.


Next: , Previous: Client Utility tapper-api, Up: Client Utility tapper-api

8.4.1 help

You can aquire a help page to each sub command:

$ /opt/tapper/perl/bin/tapper-api help upload

prints

tapper-api upload --reportid=s --file=s [ --contenttype=s ]
   --verbose          some more informational output
   --reportid         INT; the testrun id to change
   --file             STRING; the file to upload, use '-' for STDIN
   --contenttype      STRING; content-type, default 'plain', 
                      use 'application/octed-stream' for binaries


Next: , Previous: help, Up: Client Utility tapper-api

8.4.2 upload

Use it from the Tapper path, like:

$ /opt/tapper/perl/bin/tapper-api upload \
  --file /var/log/messages \
  --reportid=301

You can also use the special filename - to read from STDIN, e.g., if you need to pipe the output of tools like dmesg:

$ dmesg | /opt/tapper/perl/bin/tapper-api upload \
  --file=- \
  --filename dmesg \
  --reportid=301


Previous: upload, Up: Client Utility tapper-api

8.4.3 mason

TODO


Previous: Reports API, Up: Top

9 Complete Use Cases

In this chapter we describe how the single features are put together into whole use-cases.


Previous: Complete Use Cases, Up: Complete Use Cases

9.1 Automatic Xen testing

This is a description on how to run Xen tests with Tapper using SLES10 with one RHEL5.2 guest (64 bit) as an example.

The following mainly applies to manually assigning Xen tests. In the OSRC we use temare (not yet published) to automatically create the here described steps.

9.1.1 Paths

9.1.2 Choose an image for Dom0 and images for each guest

We use suse/suse_sles10_64b_smp_raw.tar.gz as Dom0 and

 osko:/export/images/testing/raw/redhat_rhel5u2_64b_smp_up_small_raw.img

as the only guest.

The SuSE image is of precondition type image. Thus its path is relative to /mnt/images which has tapper:/data/tapper/live/repository/images/ mounted.

The root partition is named in the section ‘root’ of the Xen precondition. Furthermore, you need to define the destination partition to be Dom0 root. We use /dev/sda2 as an example. The partition could also be named using its UUID or partition label. Thus you need to add the following to the dom0 part of the Xen precondition:

  root: 
    precondition_type: image
    mount: / 
    image: suse/suse_sles10_64b_smp_raw.tar.gz 
    partition: /dev/sda2

The RedHat image is of type ‘copyfile’.

It is copied from osko:/export/image_files/official_testing/raw_img/ which is mounted to /mnt/nfs before.

This mounting is done automatically because the protocol type nfs is given. The image file is copied to the destination named as dest in the ‘copyfile’ precondition. We use /xen/images/ as an example. To allow the System Installer to install preconditions into the guest image, the file to mount and the partition to mount need to be named. Note that even though in some cases, the mountfile can be determined automatically, in other cases this is not possible (e.g. when you get it from a tar.gz package). The resulting root secition for this guest is:

 
   root:
     precondition_type: copyfile
     name: osko:/export/images/testing/raw/redhat_rhel5u2_64b_smp_up_small_raw.img
     protocol: nfs
     dest: /xen/images/
     mountfile: /xen/images/redhat_rhel5u2_64b_smp_up_small_raw.img
     mountpartition: p1

9.1.3 PRC configuration

PRC (Program Run Control) is responsible for starting guests and test suites.

9.1.3.1 Guest Start Configuration

Making PRC able to start Xen guests is very simple. Every guest entry needs to have a section named "config". In this section, a precondition describing how the config file is installed and a filename have to be given. As for guest images the file name is needed because it can't be determined in some cases. We use 001.svm installed via copyfile to /xen/images/001.svm. The resulting config section is:

 
     config:
       precondition_type: copyfile
       name: /usr/share/tapper/packages/mhentsc3/001.svm
       protocol: local
       dest: /xen/images/
       filename: /xen/images/001.svm

9.1.3.2 Testsuite Configuration

You need to define, where you want which test suite to run. This can be done in every guest and the Dom0. In this example, the Dom0 and the single guest will run different testsuites. this chapter only describes the Dom0 test program. See the summary at the end for details on the guest test program.

The section testprogram consists of a precondition definition describing how the test suite is installed. In our example we use a precondition type package with a relative path name. This path is relative to ”'/data/tapper/live/repository/packages/”'. Since ”'tapper:/data/tapper/”' is mounted to ”'/data/tapper/”' in the install system, this directory can be accessed at ”'tapper:/data/tapper/live/repository/packages/”'.

Beside the precondition you need to define an execname which is the full path name of the file to be executed (remember, it can't be determined). This file is called in the root directory (”'/”') in the test system thus in case you need to use relative paths inside your test suite they need to be relative to this. The program may take parameters which are named in the optional array ”'parameters”' and taken as is. The parameter is ”'timeout_after_testprogram”' which allows you to define that your test suite shall be killed (and an error shall be reported) after that many seconds. Even though this parameter is optional, leaving it out will result in Tapper waiting forever if your test doesn't send finish messages. The resulting testprogram section looks like this:

 
   testprogram:
     precondition_type: package
     filename: tapper-testsuite-system.tar.gz
     path: mhentsc3/
     timeout_after_testprogram: ~
     execname: /opt/system/bin/tapper_testsuite_system.sh 
     parameters: 
       - --report

9.1.4 Preconditions

Usually your images will not have every software needed for your tests installed. In fact the example images now do but for the purpose of better explanation we assume that we need to install dhcp, python-xml and bridge-utils in Dom0. Furthermore we need a script to enable network and console. At last we install the Xen package and a Xen installer package. These two are still needed on our test images. Package preconditions may have a ”'scripts”' array attached that name a number of programs to be executed after the package was installed. This is used in our example to call the Xen installer script after the Xen package and the Xen installer package were installed. See the summary at the end for the resulting precondition section. The guest image only needs a DHCP client. Since this precondition is appended to the precondition list of the appropriate guest entry, the System Installer will automatically know that the guest image has to be mounted and the precondition needs to be installed inside relative to this mount.

9.1.5 Resulting YAML config

After all these informations are gathered, put the following YAML text into a file. We use /tmp/xen.yml as an example.

 
   precondition_type: xen
   name: SLES 10 Xen with RHEL5.2 guest (64 bit)
   dom0:
     root:
       precondition_type: image
       mount: /
       image: suse/suse_sles10_64b_smp_raw.tar.gz
       partition: /dev/sda2
     testprogram:
       precondition_type: package
       filename: tapper-testsuite-system.tar.gz
       path: mhentsc3/
       timeout_after_testprogram: 3600
       execname: /home/tapper/x86_64/bin/tapper_testsuite_ctcs.sh
       parameters: 
         - --report
     preconditions:
       - precondition_type: package
         filename: dhcp-3.0.3-23.33.x86_64.rpm
         path: mhentsc3/sles10/
       - precondition_type: package
         filename: dhcp-client-3.0.3-23.33.x86_64.rpm
         path: mhentsc3/sles10/
       - precondition_type: package
         filename: python-xml-2.4.2-18.7.x86_64.rpm
         path: mhentsc3/sles10/
       - precondition_type: package
         filename: bridge-utils-1.0.6-14.3.1.x86_64.rpm
         path: mhentsc3/sles10/
   # has to come BEFORE xen because config done in here is needed for xens initrd
       - precondition_type: package
         filename: network_enable_sles10.tar.gz
         path: mhentsc3/sles10/
         scripts:
           - /bin/network_enable_sles10.sh
       - precondition_type: package
         filename: xen-3.2_20080116_1546_16718_f4a57e0474af__64bit.tar.gz
         path: mhentsc3/
         scripts: ~
       - precondition_type: package
         filename: xen_installer_suse.tar.gz
         path: mhentsc3/sles10/
         scripts:
           - /bin/xen_installer_suse.pl
   # only needed for debug purpose
       - precondition_type: package
         filename: console_enable.tar.gz
         path: mhentsc3/
         scripts:
           - /bin/console_enable.sh
   guests:
     - root:
         precondition_type: copyfile
         name: osko:/export/images/testing/raw/redhat_rhel5u2_64b_smp_up_small_raw.img
         protocol: nfs
         dest: /xen/images/
         mountfile: /xen/images/redhat_rhel5u2_64b_smp_up_small_raw.img
         mountpartition: p1
         #       mountpartition: /dev/sda3 # or label or uuid
       config:
         precondition_type: copyfile
         name: /usr/share/tapper/packages/mhentsc3/001.svm
         protocol: local
         dest: /xen/images/
         filename: /xen/images/001.svm
       testprogram:
         precondition_type: copyfile
         name: /usr/share/tapper/packages/mhentsc3/testscript.pl
         protocol: local
         dest: /bin/
         timeout_after_testprogram: 100
         execname: /bin/testscript.pl
       preconditions:
         - precondition_type: package
           filename: dhclient-4.0.0-6.fc9.x86_64.rpm
           path: mhentsc3/fedora9/

9.1.6 Grub

For Xen to run correctly, the defaults grub configuration is not sufficient. You need to add another precondition to your test. System Installer will replace $root with the /dev/* notation of the root partition and $grubroot with the grub notation of the root partition (including parenthesis). Put the resulting precondition into a file. We use /tmp/grub.yml as an example. This file may read like this:

 
  precondition_type: grub
  config: |
   serial --unit=0 --speed=115200
   terminal serial
   timeout 3
   default 0
   title XEN-test
     root $grubroot
     kernel /boot/xen.gz com1=115200,8n1 console=com1
     module /boot/vmlinuz-2.6.18.8-xen root=$root showopts console=ttyS0,115200
     module /boot/initrd-2.6.18.8-xen 

9.1.7 Order Testrun

To order your test run with the previously defined preconditions you need to stuff them into the database. Fortunatelly there are commandline tools to help you with this job. They can be found at /opt/tapper/perl/perls/current/bin. In our production environment the server for Tapper is tapper. Log in to this server (as user tapper). Make sure that /opt/tapper/perl/perls/current/bin/ is at the beginning of your $PATH (so the correct perl will always be found). For each precondition you want to put into the database you need to define a short name. Call tapper-testrun newprecondition with the appropriate options, e.g. in our example:

 
 $ tapper-testrun newprecondition --shortname=grub \
                                  --condition_file=/tmp/grub.yml

 $ tapper-testrun newprecondition --shortname=xen \
                                  --condition_file=/tmp/xen.yml

tapper-testrun will return a precondition ID in each case. You will need those soon so please keep them in mind. In the example the precondition id for grub is 4 and for Xen its 5.

You can now put your test run into the database using tapper-testrun new. This expects a hostname, a test program and all preconditions. The test program is never evaluated and only there for historical reasons. Put in anything you like. Add --owner with an appropriate user if you don't want the default tapper. The resulting call looks like this:

  tapper-testrun new --hostname=bullock \
                     --precondition=4 \
                     --precondition=5 \
                     --test_program=whatever \
                     --owner=mhentsc3
 

Please note: There is a more central approach to describe all needed preconditions at once, see Macro Preconditions and A real live example - kernel boot test.

When the requested testrun is scheduled Tapper will setup the system you requested and execute your defined testrun. When everything went well, you'll see test output soon after. For more information on what is going on with Tapper, see /var/log/tapper-debug.