QEF home page

Chapter 11: Advanced Tools

The Guide's Table of Contents Previous Chapter Bottom Of Page Next Chapter

The QEF product's bin directory contains 140+ different programs (165 counting links) -- see Appendix E: Q-Tree Command Summaries. This chapter presents some of the more useful tools that are not directly associated with software construction, but are very useful in controlling and monitoring builds across the network.


The QEF subset is actually only about a third of the full Q-Tree package. For the most part the QEF tools are associated with software construction or the software process. Some tools have been added in response to user requests to serve various esoteric purposes in a way that behaves consistently across all platforms. The interested reader should review the complete list of QEF tools given in Appendix E.

This chapter discusses the use of programs often used to launch and monitor build processes. It also discusses two very useful programs ct and pa, and the oldest Q-Tree program of all, the program that some users refer to as "Our friend rpl".

11.1) Qremote, Cush, Lash, Josh, & Envset

In 1975, Tom Duff, Academy Award winner and inventor of Duff's device, said that the most interesting programs were those that wrote or ran other programs. He should find this section interesting indeed as qremote, cush, lash, and josh are all programs that run other programs, often each other, to facilitate launching programs with the correct environment and controls. envset does not fit into this class directly, except that its semantics are duplicated internally by the other programs.

The reader should read the brief descriptions in Appendix E for these programs. Briefly,
cushcould (could not) use shell interface
cush runs its argument command in the specified directory with I/O redirection, environmental settings (possibly using a envset environment set) specified via options. Note that cush is invoked by qremote to actually run the latter program's argument command.
envsetoutput environment setting commands
envset is the direct interface to its database specified in the user's ~/.qtree/envset and the system <qtree>/lib/envset which specifies sets of the setting and unsetting of environment variables. envset sets can be used by cush and qremote. A tree's root.vrs file's QremoteEnv setting is actually the name of a envset set to be used when doing a qremote in the tree.
josha job shell
josh interprets command sets selected by argument names from the user's ~/.qtree/josh database which becomes a convenient place to store commonly used commands.
lashlong argument list shell interface
lash runs its argument command with substitutions in that command from either lash's argument list or the standard input.
qremoterun a command on a remote host (if necessary)
qremote runs rsh or some equivalent to run a command on a remote or the local system. qremote can accept a path number argument that specifies in what directory the command is to be run. Furthermore, by default it uses the BuildHost setting in the tree's root.vrs file as the name of the host on which to run.
Note:qremote needs to know the location of the Q-Tree on the remote host. This is done by retrieving the value of the qdsrv variable host.qtree. See Chapter 6: Variable Database Server, Appendix A: qdsrv host.qtree variables, and Appendix C: The *.qtree Variables.
These programs are presented here as the build masters at some organizations use them in combination to manage the builds of many products across many platforms using simple one line commands. The Cook's Tour: Envset, Cush, ... page illustrates their use to accomplish this in some detail.

Briefly, the user adds commands to his/her ~/.qtree/josh file similar to the following:

    paths = 585 588 613
    goproj6: ! run detached builds for proj6 
	lash -o 'qremote -g -%1 qef -d @D @*' @(paths)
When the user invokes the command:
    % josh -Ddir goproj6 - other args
josh will select from its database the goproj6 set, replace the "@D" by the -D argument, the "@*" by other args, and the "@(paths)" by "585 588 613", and then run the resulting command:
    lash -o 'qremote -g -%1 qef -d dir other args' 585 588 613
lash will then run its argument command once for each argument, replacing the "%1" by the argument, thus lash executes:
    % qremote -g -585 qef -d dir other args
    % qremote -g -588 qef -d dir other args
    % qremote -g -613 qef -d dir other args
Considering the first of the above, qremote will chdir to the specified path (i.e., -585), extract that directory's settings for BuildHost and QremoteEnv. It then retrieves the Q-Tree for the specified host from qdsrv and given the host is not the current host executes:
    % rsh host host-qtree/bin/cush -Q -E qremote-env
	-r pathdir go -d qef -d dir other args
If the command is to be run on the local host, the rsh host is not included.
Note:The rsh used is what ever the command should be for the current host or a shell command called host in the user's ~/.qtree/qremote or the system's <qtree>/data/qremote directories.
Note:The -Q flag specifies that cush should set $QTREE to host-qtree.
Note:qremote translates the pathdir argument to the form used on the target host as might be necessary if the current host is a windows system whereas the target host is unix.
Note:The go command will detach its argument command after redirecting the standard outputs to a diagnostic file as described in the next section.
So, josh begats lash, and lash begats qremotes, and qremotes begat rshs, and rshs begat cushs, and cushs begat gos, and gos begat qefs and qefs begat a multitude of other processes (often in parallel) and you can watch it all happening with the various go-file monitors.

I wonder if Tom finds that interesting.
Note:The other users on your network might find it offensive and you might consider added -m1 to the qef arguments to suppress parallel execution.

11.2) Go, G, Goes, Gomonitor, and the Build Monitor

The program go simply runs its argument command, after redirecting the new process' standard output and standard error to a go-file, described later. The original go process exits, thereby returning control to the executing program (e.g., the shell). A forked child go process actually executes the command and waits for its completion. To track go'd commands, lines are added to the user's ~/.qtree/goes/host/gomsgs file. The lines contain:
    pid    command ~ host key directory date&time
for example:
    00662     ls ~ matt - /m/guide/o/guide 2000/05/04 14:31:53
    00723     josh -d testing - these arguments are ignored
	~ matt a /usr/lib/gcc-lib/i386-redhat-linux/egcs-2.90.29
	/include 2000/05/04 14:42:56
Once the command has terminated, another line is added to the same file with the exit status and time as in:
00723:0   josh -d testin... ~ matt a /u*/l*/g*/i*/e*/include
	2000/05/04 14:43:01
00662:0   ls ~ matt - /m/guide/o/guide 2000/05/04 14:31:53
Note:The command and directory elements may be abbreviated, as illustrated in the above example, to make the line fit into various displays.
The go files By default go redirects the child's merged standard output and standard diagnostic output to a file called ,.g. Any previously existing ,.Og files, where O is zero or more `o's, are moved to a new name by effectively inserting an `o' immediately after the `.'. Thus ,.*g files are moved to ,.o*g.
Note:This is an adoption of yet another Duff device as Tom was the author of the original upd program which implements a similar scheme.

If the go -d flag is specified, the outputs are redirected to ~/.qtree/goes/host/99999key,g, where the 99999 is replaced by the pid of the go process and the key is replaced by a lower case alphabetic to ensure that the file does not have the same name as a previously created go-file.
Note:The key appears in the gomsgs line immediately after the host name. A minus sign (i.e., `-') signifies that the output was redirected to a ,.g file.

josh supports similar I/O redirection. Its -g flag simulates go and its -d flag simulates go's -d flag. qremote with a -g flag also duplicates the basic functionality of go -d as it builds the latter command into the command that will be executed on the remote host. The QEF GUI qefgui uses qremote -g to dispatch all its builds and a variety of other commands. As such it is not uncommon for a user to use or simulate go directly or indirectly scores of times a day. To deal with the scores of go-files and the gomsgs audit trails, the Q-Tree provides the following facilities:

g display, list, and/or delete go-files
Flags are provided to list (-l), head (-h), tail (-t), tail continuously (-f), cat and remove (-P), remove and cat next (-r), and remove (-R) go-files. The go-file to which the operation is to be applied can be specified by one or more -os (e.g., -oo shows 3rd youngest) or using the age index. The -d flag is used to specify that the .qtree/goes/*/*,g files are to be processed. The default is to process the ,.g files in the current directory. The -q and -Q flags show the statuses of qef processes in the go-files.
goes display and purge gomsgs files
goes by default displays the active and completed job lines from the current host's gomsgs file. The -hhost flag can be used to specify the host. The -h flag will process all the existing gomsgs files. Flags are provided to remove lines for completed or day old jobs.
gomonitor display list of active and inactive go'd jobs
gomonitor is usually run continuously in its own window. It displays the N (default 20) most recent gomsgs lines for completed or active jobs. Completed job lines appear in reverse video.
qefgui the build monitor window
See Chapter 5.5: The Build Monitor. Facilities are provided to view, tail, and purge the gomsgs files and any of the go-files in the user's ~/.qtree/goes/* directories.
GODIR & HOME_QTREE Sometimes it is impossible to have a user's ~/.qtree/goes/host directory in the same directory as those of other hosts. For example, a user's home directory on VMS cannot usually contain a directory called .qtree. Similarly a user's home directory on windows cannot be easily shared by other hosts. To arrange for the gomsgs and go-files directory to be accessible on other systems so that the go family of tools, $HOME_QTREE can be set to the name of the directory to be used as the user's ~/.qtree directory. If that directory is a container file system that is accessible to a unix system, a symbolic link on the latter system can be made to link the VMS system's $HOME_QTREE/goes/host into the unix side's ~/.qtree/goes/ directory. On a windows system, $GODIR can be set to the name of the user's unix-side ~/.qtree/goes/windows-host directory. Thus the go-files tools on a unix system can process go-files created on the VMS and windows system, thereby circumventing those systems' limitations.

11.3) Mkvernum and Vcc

Chapter 6 mentioned mkvernum and its link vcc as qdsrv client programs and illustrated their use. Another illustration of their use is given at the bottom of Page 12 of the Cook's Tour. mkvernum produces a version information file by translating a template with embedded `^' sequences such as `^T' for the time, `^r' for revision, and `^O' for operating system. A listing of all the special `^' strings is output by:
    % mkvernum -X
    Format special characters:

    ^(<string>)     evaluate <string> as qvrs expression
    ^C              Century (19-20)
    ^c              unique count -- retrieved from qdsrv
    ...
The template can be specified in one of four ways:
-bbuilt-inuse named built-in format -- listed by the -X flag as in:
    % mkvernum -b defl example
    char version_string[]={"V@(#)Example 8.4(42)
	2000/05/07 15:09:55 Linux-2.0.34-i686"};
-ffmtfileuse named file
-llibfileuse named library file found in the directories named by @MkvernumPath or <qtree>/lib/verlib, as in:
    % mkvernum -l backup.vf example
    Example 8.4(43) 2000/05/07 15:03:37
-sstringuse string as format as in:
    % mkvernum -s '^N (^c) ^P(^r) ^C^E ^T ^Z' example
    Example (50) Guide(8.4) 2000/05/07 15:19:01 EDT

vcc is a link to mkvernum that is used to create a version file and link it into a program. For example, for the following command:

    % vcc -mtest cc test.c +VFILE+
vcc will create a temporary C version file _vtest.c that contains something similar to:
    char version_string[]={"V@(#)Test 8.4(2) 2000/05/08
	03:49:38 Linux-2.0.34-i686"};
The "+VFILE+" is replaced by "_vtest.c" and the cc command is run, creating an executable with the embedded version string. The qsg program, library, and commands scripts described in Chapter 7.5: The Construction Script provide -v flag to specify mkvernum/vcc is to be used for the argument programs or libraries.

A number of qvrs variables are used within mkvernum:

Project
name of the project -- incorporated using ^P
Revision
Project revision -- incorporated using ^r or ^v for modules for which a matched @Revision[pat] does not exist
Revision[pat]
Revision for modules matched by pat, incorporated using ^R or ^v.
RevisionString[pat]
The ^X control is replaced by the format specified by @RevisionString[pat] for modules matched by pat or @RevisionString -- defaults to "^N^t^v(^c)", that is the capitalized module name, a tab, @Revision[pat] or @Revision and the module build count.
_F_mkvernum_cc[pat]
Specifies cc flags to be used compiling the version file for modules matched by pat (@_F_mkvernum_cc used if none).
MkvernumPath
Names of directories to be searched for -f files.
MkvernumFormat[pat]
The -bfls flag to be used to create version file for modules matched by pat -- @MkvernumFormat used if none.

11.4) mnfdb: The Source Manifest Database Package

Originally the mnfdb package was not part of the QEF product but was in the full Q-Tree system, as it was primarily intended for use with SCCS maintained systems. However, some clients found it useful, even when using other version systems that supposedly provided similar mechanisms.

Briefly, mnfdb and its associated tools are used to maintain and query source manifest databases. A manifest, as used by this tool set, is a set of file and version tuples as in:

    README		8.2
    Sums.QEF		0.0
    arlo/README		8.1
A manifest should represent the source files and versions of those files that are required to produce a release of software. The mnfdb package provides tools manage, modify, and query a database that is a collection of releases.
Note:The qsg script newmnfdb can be used to add manifests to a database, provided SCCS is being used and sidsnap is available. However, creation of a command to produce a manifest for other versioning systems is usually fairly simple.
A manifest database consists of release information lines consisting of a release name, date and time and description, and file-version entries which bind versions of the file to releases. A file version entry consist of three tab separated fields, which are the file name, the file version number, and space separated release names. Subsequent lines that have an empty first field are continuations for the file. For example:
    # This is a comment
    $V6.1   1995/01/18 01:43:41     this is an example
            release entry
    $V6.2   1995/01/20 12:43:42     this is a correction to V6.1
    $V7.1   1996/07/01 01:23:45     New Version (about time)
    file1   1.1     V6.1 V6.2
	    1.22    V7.1
    file2   1.1     V6.1
            1.2     V6.2
    file3   2.1     V7.1
The mnfdb package consists of:
mnfaddadd a new manifest to manifest database
mnfchkcheck manifest database for syntax and consistency
mnfcmpcompare two manifests, as in:
% mnfcmp -r V6.2 -R V7.1
file1   1.1     1.22
file2   1.1     ^
file3   ^       2.1
mnfdboutput manifest database release information
mnfdeldelete releases from manifest database
mnfputoutput a manifest or manifest database, as in:
% mnfput -m -rV7.1
file1	1.22
file3	2.1
To use the manifest database system one does need tools to extract the version identifiers from the version system administration files and a tool that can extract the files named in a manifest, but both are relatively easy commands to implement. A very useful facility is a modification (if necessary) to the version system's check-out facility is to use the version number as specified by a manifest -- kdbm can help.

11.5) Ct and Pa

ct and its link and partner pa are not directly part of the construction system, although they are sometimes used in lieu of temporary files. They are described in this chapter as they are exceedingly useful and sometimes special setups are required.

Briefly, ct cats its argument files or input into a file in ~/.qtree/ctdir. The actual file name used is ,NN, where NN is the lowest two digit number not already in use.

pa displays, removes, displays and removes, or lists files from the ~/.qtree/ctdir directory. The default is to process the youngest such file, but options are provided to specify explicit files, or files by age index.

The truly useful aspects of ct and pa are:

  1. one doesn't need to name the temporary file
  2. the files are accessible from anywhere on the network from any terminal and/or window
  3. one can write to other user's cut directories to quickly and efficiently send data without fear of destroying that user's data.
  4. one can direct input to a file and then use that file's name in commands that cannot manage two input streams as in:
        % ... | ct
        % ... | diff - `pa -n` # -n outputs cutfile name
        % pa -R # remove temporary file
    Note:The above depends on there being no intervening ct between the ct and pa -n. To be safe one can use the ct -n or -e flags to retrieve the name or index of the created file.
Once again the limitations of VMS or windows can get in the way. However, one can set $CTDIR to name the directory to be used. Furthermore one can symbolically link another host's ~/.qtree/ctdir as a sub-directory of the local host's ctdir directory and then use the -s subdir option to specify the sub-directory of the user's ctdir/ to be used.
Note:The -u option can be used to ct or pa files to or from another user's cut directory, provided permissions permit. Sometimes one sets the mode of their ctdir to 777 to permit other uses create cut files. Alternatively, one can send the cut file's name to the other user which does not require write permission.

A final note. One use of ct/pa is so common and useful it's almost an idiom and that is:

    % pwd | ct # in some window
    % cd `pa -r` # in another window
One uses ct to save the current directory's pathname in a cut file and then "pastes" the contents of that file using pa into a chdir command in another window, possibly on another host and monitor, and then removes the cut file.

11.6) Rpl -- An Oldie But Goldie

rpl reads lines of the form:
    <file>:<address><action><text>
The named file is opened and the actions are performed. If the -r (for replace) flag is specified, the modified file is written back into the original file. Otherwise the modified file is written to the standard output. The actions include line insertion, deletion, replacement, and output. The addresses can incorporate line numbers followed by plus or minus signs, and/or `/' enclosed regular expressions. The text can be extended over multiple lines by prefixing the continuation lines with a `\' -- unconventional but useful as so many `\'s at the end of lines must be treated literally.

rpl is described here for a variety of reasons:

  • a variety of programs (e.g., rep) provide a -r flag that generates rpl input.
  • it's very useful, in fact it is referred to as "our friend rpl" by some users
  • it's the oldest Q-Tree program being the first significant dt program on Unix, written in Feb., 1975.

11.7) The -f Flag

Of the 1500 or so different flags in the complete Q-Tree, the ubiquitous -x is the most common as it is provided by every program. The second most common flag is -f, which usually means one of following:
  1. name the file to be processed (e.g., qef -fscript)
  2. specify that the files to be processed are named in the standard input or the argument files
  3. output the file name (e.g., rep -f)
  4. override the permissions (e.g., rm -f)
The second use in the above list merits further elucidation as it is useful when dealing with large software projects.

For over 15% of the Q-Tree programs, the -f flag means that the files or arguments to be processed are named by the argument files or by the standard input. For example:

    % detox -r -f srclist._ # undosify files named by srclist._
    % cat srclist._ | detox -r -f # the same using stdin
processes the files named by srclist._ (i.e., the file produced by sls).
Note:Some programs use -F (e.g., rep) or -l instead of -f, as the latter was already being used.
Unfortunately the semantics of the -f flag is not consistent, even when it does imply process argument files or its input as file lists due to historic uses of the -f, however, the reader is encouraged to bear with these inconsistencies as the normal usefulness of the -f flag compensates for the occasional irritation.

The following programs of the QEF subset use -f to indicate files to be processed are named in the standard input or the argument files: bfsic, chstat, detox, dmpdeps, filelist, fnd, fndstr, gotstr (by default), incls, lash (use -l), libs, lines, linked, lls, mkquete, rep (use -F), rotf, sfsic, strfix, transl, vci, wmode, and wot.


c130.qh - 8.19 - 00/05/30 QEF Home The Guide's Table of Contents Previous Chapter Top Of Page Next Chapter