(To check for possible updates to this document, please see http://www.spec.org/accel2023/Docs/ )
Contents
1. Introduction
a. What are the tools?
b. Pre-compiled binaries
c. When to build the tools yourself
2. How to build the tools
a. Notes specific to Unix systems
b. Notes specific to Windows systems
3. How to verify that your build succeeded
4. Describing your newly-built tools
a. Toolset presentation
1. Excluding otherwise valid candidates
2. Affecting presentation order
5. Packagetools
a. Example: use packagetools
b. Example: test the newly packaged tools
6. Troubleshooting: What to do if something goes wrong
a. Try it by hand
b. Known problems
c. Executing just part of buildtools
7.Support
SPEC supplies various tools that are used to ensure consistent operation of benchmarks across a variety of platforms. In order to generate a valid result file which can be submitted to SPEC, you must use the SPEC supplied tools.
The tools include (but are not limited to):
| specdiff | examines results to see if the correct answer was obtained |
|---|---|
| specinvoke | invokes benchmarks for SPECaccel |
| specmake | GNU make (Calling it "specmake" avoids possible conflicts with versions of make that may already be on your system. SPEC requires that published results use the versions of the tools that it supplies, so that if SPEC applies patches or extensions from time to time, all users run with a consistent tool set. Similar considerations apply to other tools in this list.) |
| specxz | The xz compression program |
| specmd5sum | md5sum from GNU coreutils, with enhancements |
| spectar | GNU tar |
| specperl | Perl |
| Various Perl modules such as Algorithm::Diff, GD, HTML::Parser, PDF::API2, URI, XML::SAX, etc. | |
Many of these tools are based on the freely available programs of the same name.
Note: In this document, "Unix" is used to describe Unix and all Unix-like systems such as Linux.
You will find pre-compiled binaries for the tools in the directories:
$SPEC/tools/bin/<archname>
The term $SPEC designates the top directory where you installed the benchmark suite. (On a Windows system, it would be called %SPEC%.)
The precompiled binaries will be automatically installed when you run install.sh (Unix) or install.bat (Windows).
Sometimes it may be necessary to rebuild the tools, for example if a change in an operating system renders a precompiled binary inoperable, or if you are the first person to add support for a new architecture.
It is generally not possible to do cross-compilation of Perl, as Perl's build process involves running binaries generated during the build. Therefore, getting a toolset onto an embedded platform without a native compiler is most likely impossible.
NOTICE: If you are adding support for a new architecture, and intend to submit results to SPEC, you must ask SPEC to review your tool build. Please turn on your operating system's session recorder (e.g. in Unix, typically the script command) prior to doing buildtools. In Windows, you might have to do repetitious tedious repeated cut/paste clicking and clicking (so make that command window as big as possible). Also, you will be expected to provide the output from runspec -V and runspec --test on a system OTHER than the build system, where you have installed your new tools build.
If the directory $SPEC/tools/src (Unix) or %SPEC%\tools\src (Windows) does not yet exist, use xz and tar to expand install_archives/accel.tar.xz into a writable directly local to the build system. (Building tools on networked filesystems can work, but it's more likely to make some of GNU tar's tests fail, so it's safer to build on a local filesystem.)
xz -dc install_archives/accel.tar.xz | tar -xf - tools/src
The scripts $SPEC/tools/src/buildtools (Unix) or %SPEC%\tools\src\buildtools.bat (Windows) will build the tools. But you may need to invoke the buildtools script with appropriate environment variables set first - see the sections immediately following that provide notes about specific platforms.
You will need a C99 compiler. Most of the tools are C++ or C89, but XZ is C99.
If everything goes right, you won't have to do very much at all, other than watch build commands fly by.
When building the tools under Unix, the following warnings appear to be harmless as of May 2012:
You may find the following flags useful on the listed systems:
AIX (32-bit):
CC=xlc CFLAGS='-O -qarch=ppc -D_LARGE_FILES -D_ILS_MACROS' PERLFLAGS="-Doptimize='-O' -Accflags='-D_LARGE_FILES'" OBJECT_MODE=32 ./buildtools
AIX (64-bit):
CC=xlc CFLAGS='-O3 -qarch=ppc64 -q64 -D_ILS_MACROS' PERLFLAGS="-Doptimize='-O' -Aoptimize='-qarch=ppc64 -q64'" OBJECT_MODE=64 ./buildtools
HPUX:
CC="/opt/ansic/bin/cc" CFLAGS="-Ae +O2 +z -D_FILE_OFFSET_BITS=64 -AC99" XZCFLAGS=-AC99 PERLFLAGS="-Doptimize=+O1 -Dlibpth=/usr/lib/hpux32 -Dldflags=-lm -Dcppstdin=cc -Acppstdin=-E -Acppstdin=-Aa -Acppstdin=-D__STDC_EXT__ -Acppstdin=-AC89" ./buildtools
IRIX:
ulimit -s 128000; CFLAGS="-c99 -D_FILE_OFFSET_BITS=64" MD5CFLAGS="-O2" PERLFLAGS="-Doptimize=-O3 -Uloclib -Dlibpth=/usr/lib32" XZFLAGS="--disable-assembler --disable-shared" XZCFLAGS="-O1" ./buildtools
Check out those who have gone before. If the system for which you're attempting to build tools is similar to one for which a tools build exists, you may gain insight by looking at the top of the build logs in tools/src/buildtools.log. Even if your system is very unusual and very different than previous systems, it is probably still worth your while to look over the previous logs. You don't have to read all 20,000 lines; just check out the environment variables and path settings and so forth from the top of previous builds.
Because XZ is written in C99, no version of Visual Studio (as of May 2012) can be used to build the tools. Instead, the Windows version of buildtools is designed to use the MinGW GCC compiler. It has been tested successfully with both 4.3 and 4.5 versions of GCC; the latest version available should be used. The tools are not yet ported to be built with a 64-bit compiler; use the 32-bit tools instead.
The builds use makefiles to be processed with either mingw-make.exe (which is GNU make) or dmake.exe (which is built in the
process of building tools). The full MSYS environment is not used, so all of the makefiles are pre-generated; the GNU autoconf
system is not used. Compiler flags are defined in those pre-generated makefiles.
Building tools under Cygwin is not supported and probably won't work.
If it is necessary to unzip and untar the sources of the tools in Windows, open a Command-Prompt window, cd to %SPEC% (i.e. the top directory of SPEC). Then execute:
cd tools bin\windows-i386\specxz.exe -d install_archives/accel.tar.xz ooo\windows-i386\spectar.exe -xvf install_archives/accel.tar
Then go ahead and execute
%SPEC%\tools\src\buildtools.bat
You can ignore warnings about objects that are not found, especially at the beginning of each tool build, when cleanup is attempted from previous builds. Some tools may not know how to make 'clean', 'distclean', or 'realclean'; don't worry about it.
In Windows XP with Service Pack 2 or later, there may be a Security Alert, depending on the Firewall settings. "Windows Firewall has blocked this program from accepting connections from the Internet or a network." You may ignore this warning.
All compiler warnings seen as of May 2012 appear to be harmless.
After a tool build, you should:
cd $SPEC (Unix) or %SPEC% (Windows)
shrc.bat (Windows)
. ./shrc (Unix, if you are in an sh-compatible shell.
If not, start one!)
See if you can at least get as far as asking the major tools
to identify themselves:
runspec -V
Or you can ask individual tools about themselves:
specmake -v
specxz -h
specperl -v
specdiff -h
runspec -h
specinvoke -h
Running runspec --test will run more comprehensive tests on specperl.
As a courtesy to others who might like to know which systems you intended your new tools to work on, and to describe the environment in which they were built, you may write a short description file. The contents of this file will be displayed when install.sh is run. Windows users only have one choice, and thus no description is printed. The file is $SPEC/tools/bin/<archname>/description. If that directory does not exist, you should create it. (If the directory does not exist, packagetools will create it, but in order for the description to be packaged, it must be present before packagetools is run.) See the "Packagetools" section below for guidance about picking "<archname>".
The format is very simple; in order to avoid wrapping on an 80-column screen, the first line must be no longer than 50 characters. In order to line up properly, subsequent lines should begin with 30 spaces and be no longer than 80 columns total.
For example, given $SPEC/tools/bin/turboblaster-m68k/description with the following contents
For TurboBlaster v2.1+ systems running on M68010
hardware.
Built on TurboBlaster v2.0 with GCC 1.81.
A user using a TurboBlaster system when running install.sh would see in the list of toolset choices, something that looks approximately like
turboblaster-m68k For TurboBlaster v2.1+ systems running on M68010
hardware.
Built on TurboBlaster v2.0 with GCC 1.81.
There are plenty of examples on the install media in tools/bin/*/description.
If the toolset you're building isn't the only one that will work on a particular system, you may want to adjust where in the list of toolsets it appears, or even whether it appears at all. By default toolsets are presented in lexical ASCII order. However, it is possible to both adjust the position of a toolset in the list of candidates, as well as keeping it from appearing at all.
Let's assume that the TurboBlaster systems mentioned in the last section have migrated to PowerPC and have an emulation layer so that they can run old M68K binaries. Given two toolsets called turboblaster-m68k and turboblaster-ppc, normally the turboblaster-m68k would be presented and attempted first. And the emulation layer is so good that it would work, but it'd be slower than it needs to be.
There are several files that can be used to prevent install.sh from even attempting to use a toolset. The exclusion is silent, and for all intents and purposes makes the affected toolset behave just as one from another OS or architecture — it doesn't even show up on the candidate list.
To effect this exclusion, install.sh looks for signatures of the current system in files in each toolset's directory. A toolset is excluded from consideration if any of the following are true:
| Output from... | Appears in... |
|---|---|
| uname -m | tools/bin/<archname>/excludearch |
| uname -s | tools/bin/<archname>/excludearch |
| uname -r | tools/bin/<archname>/excluderev |
| uname -p | tools/bin/<archname>/excludeproc |
On a PPC-based TurboBlaster system, uname -m returns "PowerPC". Putting "PowerPC" into tools/bin/turboblaster-m68k/excludearch would ensure that TurboBlaster/PPC users never even saw the TurboBlaster/M68K tools.
This is most useful for preventing cross-OS confusion. FreeBSD, for example, used to try to execute Solaris binaries. Putting "FreeBSD" (the output from uname -s on a FreeBSD system) into tools/bin/solaris-x86/excludearch fixed that problem and kept the unmaskable and alarming failure messages from appearing during installation.
In the case where there may be two equally valid toolsets (as might be the case when there's a 32-bit and 64-bit build for the same OS), it's possible to change the order in which they're presented. Each toolset has a file named order in its directory. If it doesn't exist, packagetools will create this file with contents that are simply:
1:<archname>
(See the packagetools section for guidelines on picking <archname>.) The "priority" of a toolset decreases as that number increases. Continuing with the TurboBlaster example, if we wanted the PowerPC toolset to be preferred, we'd put
2:turboblaster-m68k
into tools/bin/turboblaster-m68k/order. That would ensure that turboblaster-ppc would appear first in the menu, as the number in its order file is "1". Since it appears first in the menu, it's installed before turboblaster-m68k.
If everything has succeeded, and you intend to submit results using your new tools, you should submit the tools to SPEC. To do so:
cd $SPEC (Unix) or cd %SPEC% (Windows)
packagetools <archname>
Pick an architecture name that other users will recognize. Check on the install media in tools/bin for some examples.
The packagetools script will create:
$SPEC/tools/bin/<archname>/specxz
$SPEC/tools/bin/<archname>/spectar
$SPEC/tools/bin/<archname>/specmd5sum
$SPEC/tools/bin/<archname>/tools-<archname>.tar.xz
Having created a large tarfile with everything in it, packagetools will then proceed to create an even larger tarfile with specxz and spectar it in too. This even bigger file is known as:
$SPEC/<archname>-<version>.tar
and is to be submitted to SPEC. (See also the example below.)
You can optionally add components to your platform's toolset. For example, if you would like $SPEC/config/default.cfg to be set in an appropriate way, you can add the relative path to default.cfg as a parameter to packagetools:
(on non-Windows systems)
cd $SPEC
packagetools <archname> config/default.cfg
More than one file may be specified in this way.
Operation on Windows is substantially similar; just provide the relative paths with backslashes instead of forward slashes.
Please submit the resulting tarfile to SPEC for review, along with the recording of your tool build session. SPEC will review your tools, and assuming that they pass review, will add the tools you have built to its patch library, for possible distribution to future users of your interesting new architecture.
NOTE 1: If your operating system is unable to execute the packagetools script, please have a look at what the script does and enter the corresponding commands by hand. Again, you will need to submit the results to SPEC.
NOTE 2: Be sure to test your packaged tools on a different system, preferably one with a different disk layout. If the destination system is unable to invoke libperl.so, check that libperl.so exists in one of the locations where shrc expects to find it.
Here is an example use of packagetools. In the example below, notice that:
1 $ . ./shrc 2 $ which runspec 3 /spec/accel/kit117/bin/runspec 4 $ 5 $ export MYTOOLS=myOS-myHW 6 $ mkdir tools/bin/$MYTOOLS 7 $ echo \ 8 "For systems running myOS. Note: works only with V12 or later" \ 9 tools/bin/$MYTOOLS/description 10 $ 11 $ cp typescript $MYTOOLS.buildlog.txt 12 $ runspec -V > $MYTOOLS.runspec-V.txt 2>&1 13 $ runspec --test > $MYTOOLS.runspec--test.txt 2>&1 14 $ 15 $ packagetools $MYTOOLS $MYTOOLS.buildlog.txt $MYTOOLS.runsp*txt 16 Preparing /spec/accel/kit117/tools/bin/myOS-myHW 17 Generating MD5 checksums for the tools 18 Generating the tools installation tarball 19 100 % 12.0 MiB / 80.1 MiB = 0.149 1.0 MiB/s 1:17 20 Making the distribution tarball 21 tools/bin/myOS-myHW/ 22 tools/bin/myOS-myHW/order 23 tools/bin/myOS-myHW/tools-myOS-myHW.tar.xz 24 tools/bin/myOS-myHW/description 25 tools/bin/myOS-myHW/unbundled 26 tools/bin/myOS-myHW/specxz 27 tools/bin/myOS-myHW/specmd5sum 28 tools/bin/myOS-myHW/spectar 29 30 ******************************************************************* 31 32 The tarball to submit for inclusion in the distribution is 33 34 myOS-myHW-117.tar 35 36 ABSOLUTELY DO NOT submit the tarball in tools/bin/myOS-myHW 37 as it is not complete. 38 39 ******************************************************************* 40 41 $ specmd5sum myOS-myHW-117.tar > myOS-myHW-117.tar.md5
Testing a newly-built toolset on the system where it was built is not enough to ensure basic sanity of the tools. Test for unintended dependencies by installing on an entirely different system. For example:
1 $ cd /spec/accel 2 $ cp /net/(buildsys)/spec/accel/kit117/myOS-myHW-117.tar* . 3 $ cp /net/(buildsys)/spec/accel/kit117/bin/spectar . 4 $ cp /net/(buildsys)/spec/accel/kit117/bin/specmd5sum . 5 $ ls 6 myOS-myHW-117.tar myOS-myHW-117.tar.md5 specmd5sum spectar 7 $ ./specmd5sum -c myOS-myHW-117.tar.md5 8 myOS-myHW-117.tar: OK 9 $ ./spectar -xvf myOS-myHW-117.tar 10 tools/bin/myOS-myHW/ 11 tools/bin/myOS-myHW/spectar 12 tools/bin/myOS-myHW/tools-myOS-myHW.tar.xz 13 tools/bin/myOS-myHW/unbundled 14 tools/bin/myOS-myHW/description 15 tools/bin/myOS-myHW/specxz 16 tools/bin/myOS-myHW/order 17 tools/bin/myOS-myHW/specmd5sum 18 $
19 $ install.sh -u myOS-myHW 20 . 21 . 22 . 23 Installation successful. Source the shrc or cshrc in 24 /spec/accel/kit117 25 to set up your environment for the benchmark. 26 27 28 $ . ./shrc 29 $ runspec -V > newsys.runspec-V.txt 2>&1 30 $ runspec --test > newsys.runspec--test.txt 2>&1 31 $ 32 $ grep -c "...ok" *runspec--test* 33 myOS-myHW.runspec--test.txt:215 34 newsys.runspec--test.txt:215 35 $ 36 $ spectar cvf - newsys.runspec* | specxz > newsys.tests.tar.xz
If something goes wrong, unfortunately, you're probably just going to have to take it apart and figure out what. Here are some hints on how to go about doing that.
If something goes wrong, you probably do NOT want to make some random adjustment (like: reinstall a compiler, fix an environment variable, or adjust your path) and start all over again. That's going to be painful and take a lot of your time. Instead, you should temporarily abandon the buildtools script at that point and just try to build the offending tool, until you understand exactly why that particular tool is failing.
Consider turning on verbose diagnostics if your system has a way to do that. Make a huge terminal window (e.g. 200 columns wide by 84 lines tall, with 9999 lines recorded off the top), so you can see what is going on.
Read what buildtools (or buildtools.bat) does for you, then cd to tools/src/<tool> and try the commands by hand. For example, you might do something like this:
cd $SPEC/tools/src/<toolname>
./configure
make (or build.sh or whatever you note buildtools would have done)
Now, try fixing that environment variable or reinstalling that compiler, and rebuild the single tool. Does it look better?
If not, have a close look at the error messages and the Makefile. Does the Makefile use a feature that is not present in your version of make? If so, can you get it to work with GNU make?
Note that for GNU configure based tools (everything except Perl and its modules) you may specify your compiler by setting the CC environment variable. For compiler flags, set CFLAGS.
When building perl, note that:
-Doptimize=-option1 -Aoptimize=-option2 -Aoptimize=option3 ...
Many of the build logs in tools/src/buildtools.log have examples of settings used for PERLFLAGS.
If you want to see more about what buildtools is doing for you, turn on your shell's verbose mode. For example:
sh -x ./buildtools
Try doing a web search to see if there are known problems with the tool on your architecture.
If SPEC supplies Version X.Y of a tool and it just won't build on your operating system, you might check whether there is a new Version X.Y+1 available. If so, download the new version to a scratch directory outside of the SPEC tree and try building it there. If that version succeeds, try to deduce why. Narrow it down to a one-line fix, won't you please? Then tell SPEC that you'd like the same one-line fix applied to its variant of the tool. Or, if you just can't narrow the fix down, ask SPEC whether it will accept use of Version X.Y+1 instead of X.Y on your system.
Once you believe that you understand how to fix the problem tool, and can build it by hand, see whether the buildtools script can build it. You can execute just a portion of buildtools by defining environment variables. Please read the script itself to see what variables are allowed; the following are just some examples:
SKIPALL - turns off everything. If you like, set this, then
turn individual phases on by setting them.
DOTOOLSRM - Remove previously installed tools
DOCLEAN - Run 'make clean', 'make distclean', and
'make realclean' in all source directories.
DOMAKE - build make
DOXZ - build xz
DOTAR - build tar
DOMD5 - build specmd5sum
DOSPECINVOKE - build specinvoke
DORXP - build rxp
DOEXPAT - build the expat XML parser, used by XML::SAX::ExpatXS
DOPERL - build perl
DOPERL2 - build perl modules
DOCOPY - copy the results to $SPEC/bin and fix shbangs
It doesn't matter what you set the environment variables to -- any non-zero-length string will do. In all cases, "DO" can be replaced with "SKIP" to reverse the sense of the setting.
If you are using Windows, be sure to note the WARNING in buildtools.bat about how the variables work.
If you can now build the (formerly misbehaving) tool using just part of buildtools, then you can try building the whole set (i.e. a build of all tools from start to finish). It should work now, so be sure to turn on your logging facility.
You'll find information about technical support in techsupport.html, either in the same directory where you are found the document you are reading now, or at www.spec.org/accel.
Copyright 2014-2023 Standard Performance Evaluation Corporation All Rights Reserved