Release Management System: Development Manager Component

1. Overview

The scope of this document is limited to those tools that make up the developer side of the Release Management System. By definition, these are the tools that are used directly by the developers on a project (be they applications engineers or content developers). These tools are installed on each of the machines designated as development hosts. The primary goal of the suite of developer tools is to provide enhanced CVS functionality, tools (commands) to perform management of files for release, and lastly, a "wrapper" utility activated directly by the CVS tools that performs a series of pathname and URI translations on files during check-out and commit operations.

Throughout this document, the following names are used to refer to the three distinct sub-parts of the Development Manager:

The encapsulator for the CVS commands
release tool
The tool responsible for managing the larger aspects of file management and releases
The utility invoked directly by CVS to manage path and URI translations.
This is not intended to be a line-by-line dissection of the tools. Where it will help, specific variables, parameters or subroutines will be referenced. But overall, the primary intent is to focus more on why the tools do what they do, without as much emphasis on how it is done (except where it contributes to the former).

2. General Design Principle

The elements of the Development Manager package follow the same design philosophy as the Release Management System as a whole. The basic premise is that each of the individual components should be maintainable and upgradable independent of the others. Provided a common interface layer is maintained, the internal implementation of that interface should not be of concern to other tools, or to other top-level components. This will be a recurring theme throughout the explanation of the individual scripts and their methods of interaction.

3. The cvs_front_end script

The first of the tools to be covered is the script that encapsulates the various CVS actions, allowing for site-specific configuration to be added to all such invocations. This script is called cvs_front_end, though it is very rarely invoked under that name. The script is designed to be the target of several symbolic links, and the behavior is dictated by which name is used to call the tool itself. For many of the CVS operations, the use of the encapsulating tool merely provides a sort of "shorthand" calling convention, freeing the end-users the need to start each of the commands with the "cvs" prefix.

3.1. Pass-through invocations

The following invocations of the front-end tool are simply convenience functions for the sake of the end-user. Each of these passes control directly to the underlying CVS binary, without any changes to the parameters:

3.2. The checkout invocation

The only special operation performed by this invocation is changing into the user's public_html directory (in their home directory) prior to executing the cvs checkout command. It currently has as an unfortunate (and undesirable) side-effect in that the automatic changing of directory prevents the user from checking out project sub-directories from within the project directory hierarchy.

Once the checkout is complete (assuming there were no errors), the tool checks for the presence of a file named "Makefile" in the root directory of the project. If such a file exists, then the tool changes to the root directory of the project and executes the command, "make checkout". If the make utility reports that no such target exists, that error is silently ignored. Any other errors are reported. For more on this functionality, see the section titled Make Hooks below.

Note: Upon examination of the make-hook code in this case, I think that it may not currently work correctly. This needs closer scrutiny and some degree of unit testing.

3.3. The finish invocation

This calling method is provided to perform the tasks of the CVS command, "cvs release -d". When invoked on a specified project, it will first warn the user that a check for modified files is being performed in the workspace. The results of the search are presented to the user, who is then asked to confirm their desire to cease work in the given workspace and authorize its removal. Note that this does not remove files from the CVS repository itself, nor even mark files to be moved to the "attic" (a concept used by CVS to denote files that are considered deleted, but retained for possible legacy needs).

Answering in the affirmative will completely remove the specified workspace (project) from the user's working area, even if there are uncommitted changes pending.

3.4. The import and new invocations

These two are structured in very similar terms, differing only in the underlying CVS command that control is passed to. Each checks the repository to ensure that the project (internal or third-party) name is not currently in use. If not, then control is passed to the relevant CVS command. If the name is in use, an error is reported to the user.

3.5. The xnew invocation

This calling context is an extension to the existing CVS suite for the sake of specific needs within the development environment. The specific problem that this invocation solves is the case in which a third-party source tree is maintained using the internal development and release process, but is subject to update from the originating party. In simpler terms, the project specified when this tool is called is treated as a third-part project that is assumed to already exist in the repository. The contents of the repository are updated to reflect the user's workspace, and the new set of revisions tagged with a specified symbol. This greatly simplifies the process of integrating third-party updates.

3.6. The commit invocation

Lastly, there is the encapsulation of the CVS "commit" operation, which is the basic means by which changes are made to the repository itself. While the basic commit operation is not altered, it is at this point in the process flow that functionality was inserted to ensure that the permissions on the user's workspace are kept consistent. Because of the "helpful" nature of CVS and RCS in terms of setting file access permissions, the tool performs a quick check on the permissions of the files within the user's workspace prior to performing the actual commit. Since discrepancies in access permissions are most likely to affect an attempted commit operation, this seemed the logical place to insert these tests.

3.7. General to most invocations

Several of the invocations (specifically, those that govern the extraction of versioned files from the repository) pass information along to the wrapper component of the development system by means of an environment variable called WM_CONTROL (the "WM" is a historical reference to the team's original charter as simply a web-master team). The new generation of the front-end component currently retains this interface until such time as the wrapper component is re-engineered.

3.8. Calling the script by the base name

As was said at the start of this section, the cvs_front_end script is rarely called by that name. The calling of the script under the base name is part of the installation process. When called in this fashion, the script creates symbolic links in the same directory as that which the script resides in. Each of the commands listed above is created as a link in this fashion. At present, there is a boolean-style variable declared early in the application called $TESTING. If this variable has any non-null value, then all commands are also given the numeral "2" at the end of the names. This is to allow for the original tools to co-exist with the new tools until such time as the new tools can completely replace the older set of tools.

Performing this task should be considered a part of setting up any host as a development platform. The current architecture of these tools on hosts such as webdev also involves a setuid wrapper, a tool written in a lower-level language (in this case, the C programming language) that performs certain security-related checks and then executes the intended program under a different user ID than that of the person running the command. This is necessary for the sake of permissions on the CVS repository. The management of this security application and the means by which it maps to the Perl scripts is not directly part of the Release Manager, and thus not discussed here.

4. The dev_rls_tool script

This is a much more complex tool than the wrapper just described. Though there are few means of invocation, the actions performed and the depth of detail demand greater attention.

The functions performed by this tool are those vital to the core design of the Release Manager: this tool keeps the development HTTP document area up to date, creates and maintains the staging areas in preparation for release to external servers, and most importantly of all, bundles and transmits the data that comprises a new release of a given project. As with the front-end tool, this is accomplished by one Perl application that uses symbolic links in order to be called under different names. This tool is invoked as:

As with the previous tool, this application uses an invocation by the native name to perform various installation and maintenance actions.
A populate operation is one in which the system's HTTP document tree is brought up-to-date with regards to the CVS repository. It is during this stage that certain control files (defined and documented below) are also brought up-to-date.
This action is used to create or update the staging area for a particular project being released to a particular host. It is during this step that file paths and URI references are translated based on a set of rules that define the target host's server environment. This step utilizes the control files maintained by the populate step, and updates some of its own in preparation for the actual release of the project.
This last invocation is the form of the tool that actually analyzes a project area for changed content, creates a bundle of those files that have changed, and transmits the changes to the remote server host.

4.1. The maintenance invocation: dev_rls_tool

As with the above tool, the maintenance function of this tool is to create the symbolic links by which the tool is also known. The names for the populate, stage and release invocations are declared towards the top of the script, where they are easily identified. They are expressed this way so as to simplify switching between the "normal" names, and the currently-used practice of appending the numeral "2" to the ends, so as to allow co-existence with the legacy tools. When the time comes for these tools to completely supersede the older tools, the numbering can be removed from the names.

4.2. The populate invocation

4.3. The stage invocation

4.4. The release invocation

5. Some thoughts on the wrapper utility

Though a new-generation CVS wrapper has not yet been designed or developed, there are some matters that may help to consider:

6. The dev_release.cfg file

There are a variety of parameters that govern parts of the system's overall functionality. These lend themselves to centralization in one file both for the sake of sharing between applications and so that these values can change independent of the actual program files. The applications look for the configuration file by examining their own fully-qualified path within the file system, dropping the last element of the path and then appending "local". This is based on an assumption that the trailing element of executables used in this system is "ahp-bin". Additionally, to accommodate the suid-wrapper/symlink issue described earlier, the path component "suid_scripts" is also checked for. If present, it is simply discarded. In this directory, the file named dev_release.cfg is expected to exist and contain the settings.

The structure and format of the file is a simple sequence of lines of the form:

This should be obvious enough in style and intent. The amount of empty space surrounding the equal sign is flexible. From the first non-whitespace character to the right of the equal sign to the end of the current line is all encompassed at the value of the parameter. This allows for parameters with spaces within them. However, extra space after the last non-whitespace character is removed. As an additional feature, lines may optionally start with either of the special words, setenv or export. These keywords are used by the C-shell and Korn shell, respectively, to set environment variables. (The Bourne shell and "Bourne-Again" shell bash also use export for environment variables.) To date, the flexibility has not been needed, but it adds negligible overhead or complexity to the tool, and leaves the option of future application open. Lastly, blank lines are permitted for the sake of readability, and lines whose first character is "#" are considered commentary and ignored.

The values currently stored in the file, and their definitions, are:

Default: etc/acl

The directory in which the access-control lists (ACL) are stored on a per-host basis. These files are described in more detail elsewhere, but the short definition is that they govern which projects can release to a specific target host, and which users are entitled to release a given project. If the value is not an absolute path, it is considered relative to the home directory of the USER defined below.
Default: /opt/ims/bin/cvs

This is the fully-qualified path to the executable cvs program. Specifying it here accomplishes two important goals: (1) the programs are not reliant on the running user to have the correct directories in their execution path, and (2) the danger of the tools being compromised by placement of a dangerous program named cvs in an alternate element of the execution path is eliminated. It also allows for the explicit selection of a binary to use, a feature that was utilized when evaluating version 1.10 of CVS.
Default: /opt/ims/repository

The cvs command relies on either a command-line specification of a repository root, or for the value to be defined in an environment variable named CVSROOT. Defining it here not only guarantees consistency, but also prevents errors or data loss due to an incorrectly-set value on the user end.

The fully-qualified-domain name of the development host the tools are running on. This is not currently used to any real extent, but was planned for use by the wrapper component.
Default: /opt/ims/htdocs/development-log.txt

Many of the actions performed by the tools are check-pointed by an entry written to this log as a single line. Not all tools are making maximum use of this at present.
Default: /opt/ims/bin/ftp

This is the full path to the ftp binary, defined for the same reasons as the cvs binary. Additionally, this serves the purpose of ensuring that the program used has been built with correct network library code to allow it to pass through the firewall.
Default: idsweb

When files are created or updated in the project root or staging areas, this value (which may also be expressed numerically) is used to set the group ownership on such files and the directories that contain them.
Default: etc/halt-

This prefix is used when checking for "halt files" pertaining to the command about to be run. Halt files are explained earlier. If the value is not an absolute path, it is considered relative to the home directory of the USER defined below.
Default: http://web-proxy:8088

When the new-generation tools were being designed, HTTP file upload was favored over FTP-based transmission, since the password and user name were of less consequence if compromised. Due to limitations related to the HTTP-based solution, this is gradually being phased out and the original FTP-based approach is now preferred. This parameter is used to define a proxy for the HTTP approach, if one is needed. Without it, transfers beyond a firewall would not be possible.
Default: idsweb

This user name (or numeric ID if expressed as such) is used for several purposes: it is the ownership assigned to files both in the CVS repository and the staging areas, and it is the source for a home directory to use as a base directory for the values HALTFILE_PREFIX and ACL_DIR. It is also this user that is permitted to override certain safeguards such as access-control lists and halt files.
Default: /opt/ims/htdocs

This is the area in which the developmental versions of project hierarchies are kept. This is the designated document root for the server instance that runs on the development host, and each project is kept in its own unique directory under this root.
Default: .release

The file which the release-specific tool updates to mark a successful release. It is expected to be a simple file, not a path, and will be created/updated in the project's directory under PROJECT_ROOT.
Default: /opt/ims/staging

When a project is being staged in preparation for release to a different host, the project directory is under this directory as its root. As explained earlier, this directory fans out into a set of directories named for the target external hosts, and the projects are staged under these host-specific directories on a per-request basis. This allows for projects to be staged to multiple hosts (i.e. a testing and a production machine) without concern for conflicts in the staging area.
Default: /opt/ims/bin/tar

See the entries for CVS and FTP. The added benefit for this specification is in guaranteeing that the selected binary supports on-the-fly compression and decompression of archive files (as the GNU tar implementation does).
Default: .topiclist

This is one of the files updated by the populate-specific tool and referenced by the stage tool. The information in it is not generally used in guiding the software, but rather for noting informational aspects of the project.
Default: .weblist

Lastly, this file is the basis for the management of the files that comprise a project. It is updated by the populate tool and referenced by the stage tool. It is explained in greater detail elsewhere. As with RELEASE and TOPICLIST, it should have no path component, and will be created and updated in the project's area under PROJECT_ROOT.

7. Inter-tool Communication

As was referred to in the section on the CVS front-end, there are times when tools have to pass information between them that can't be passed on the command line. In general, this is limited to passing information from the front-end component to the wrapper component. This is done by means of an environment variable called WM_CONTROL. The front-end function sets this variable with one or more words that describe the action currently in progress. The wrapper then knows whether the operation is going into the repository, coming out from the repository, etc. It can also then tell which specific host is the target of actions like stage. Since the wrapper is not called directly by the tools (but rather by the actual cvs binary), this is the only feasible way of passing this information.

Since the necessary context is apparent from the command-line of the front-end tool, the tool that handles population, staging, etc. does not use this. By calling the front-end versions of CVS commands, the information is already present.

8. Conclusion