PerfKitBenchmarker Installation and Run

(Got Errors highlighted below)

[ec2-user@CDH ~]$ cd PerfKitBenchmarker-1.0.1/
[ec2-user@CDH PerfKitBenchmarker-1.0.1]$ ll
total 128
-rw-rw-r-- 1 ec2-user ec2-user
168 Dec 10 18:28 AUTHORS
-rw-rw-r-- 1 ec2-user ec2-user 25520 Dec 10 18:28 CHANGES.md
-rw-rw-r-- 1 ec2-user ec2-user 3744 Dec 10 18:28 CONTRIBUTING.md
drwxrwxr-x 3 ec2-user ec2-user 4096 Dec 10 18:28 hooks
-rw-rw-r-- 1 ec2-user ec2-user 11357 Dec 10 18:28 LICENSE
-rw-rw-r-- 1 ec2-user ec2-user
154 Dec 10 18:28 MANIFEST.in
drwxrwxr-x 11 ec2-user ec2-user 4096 Dec 11 09:18 perfkitbenchmarker
-rwxrwxr-x 1 ec2-user ec2-user
704 Dec 10 18:28 pkb.py
-rw-rw-r-- 1 ec2-user ec2-user 35340 Dec 10 18:28 README.md
-rw-rw-r-- 1 ec2-user ec2-user
740 Dec 10 18:28 requirements.txt
-rw-rw-r-- 1 ec2-user ec2-user
796 Dec 10 18:28 setup.cfg
-rw-rw-r-- 1 ec2-user ec2-user
978 Dec 10 18:28 setup.py
-rw-rw-r-- 1 ec2-user ec2-user
675 Dec 10 18:28 test-requirements.txt
drwxrwxr-x 8 ec2-user ec2-user 4096 Dec 10 18:28 tests
drwxrwxr-x 4 ec2-user ec2-user 4096 Dec 10 18:28 tools
-rw-rw-r-- 1 ec2-user ec2-user
658 Dec 10 18:28 tox.ini
[ec2-user@CDH PerfKitBenchmarker-1.0.1]$ sudo pip install -r requirements.txt
Downloading/unpacking python-gflags==2.0 (from -r requirements.txt (line 14))
Downloading python-gflags-2.0.tar.gz (65kB): 65kB downloaded
Running setup.py egg_info for package python-gflags
Downloading/unpacking jinja2>=2.7 (from -r requirements.txt (line 15))
Downloading Jinja2-2.8.tar.gz (357kB): 357kB downloaded
Running setup.py egg_info for package jinja2
warning: no files found matching 'run-tests.py'
warning: no files found matching '*' under directory 'custom_fixers'
warning: no files found matching '*' under directory 'jinja2/testsuite/res'
warning: no previously-included files matching '*' found under directory 'docs/_build'
warning: no previously-included files matching '*.pyc' found under directory 'jinja2'
warning: no previously-included files matching '*.pyc' found under directory 'docs'
warning: no previously-included files matching '*.pyo' found under directory 'jinja2'

warning: no previously-included files matching '*.pyo' found under directory 'docs'

Requirement already satisfied (use --upgrade to upgrade): distribute in /usr/lib/python2.6/site-packages (from -r
requirements.txt (line 16))
Downloading/unpacking colorlog[windows]==2.6.0 (from -r requirements.txt (line 17))
Downloading colorlog-2.6.0.tar.gz
Running setup.py egg_info for package colorlog
Installing extra requirements: 'windows'
Downloading/unpacking blinker>=1.3 (from -r requirements.txt (line 18))
Downloading blinker-1.4.tar.gz (111kB): 111kB downloaded
Running setup.py egg_info for package blinker
Downloading/unpacking futures>=3.0.3 (from -r requirements.txt (line 19))
Downloading futures-3.0.3.tar.gz
Running setup.py egg_info for package futures
Downloading/unpacking PyYAML==3.11 (from -r requirements.txt (line 20))
Downloading PyYAML-3.11.tar.gz (248kB): 248kB downloaded
Running setup.py egg_info for package PyYAML
skipping 'ext/_yaml.c' Cython extension (up-to-date)
Downloading/unpacking csapi>=0.0.7 (from -r requirements.txt (line 21))
Downloading csapi-0.0.7.tar.gz
Running setup.py egg_info for package csapi
Requirement already satisfied (use --upgrade to upgrade): paramiko in /usr/lib/python2.6/site-packages (from -r
requirements.txt (line 22))
Downloading/unpacking MarkupSafe (from jinja2>=2.7->-r requirements.txt (line 15))
Downloading MarkupSafe-0.23.tar.gz
Running setup.py egg_info for package MarkupSafe
Requirement already satisfied (use --upgrade to upgrade): colorama in /usr/lib/python2.6/site-packages (from
colorlog[windows]==2.6.0->-r requirements.txt (line 17))
Requirement already satisfied (use --upgrade to upgrade): requests in /usr/lib/python2.6/site-packages/requests1.2.3-py2.6.egg (from csapi>=0.0.7->-r requirements.txt (line 21))
Downloading/unpacking docopt (from csapi>=0.0.7->-r requirements.txt (line 21))
Downloading docopt-0.6.2.tar.gz
Running setup.py egg_info for package docopt
Requirement already satisfied (use --upgrade to upgrade): pycrypto>=1.9 in /usr/lib/python2.6/sitepackages/pycrypto-2.6.1-py2.6-linux-x86_64.egg (from paramiko->-r requirements.txt (line 22))
Installing collected packages: python-gflags, jinja2, colorlog, blinker, futures, PyYAML, csapi, MarkupSafe, docopt
Running setup.py install for python-gflags
Running setup.py install for jinja2
warning: no files found matching 'run-tests.py'
warning: no files found matching '*' under directory 'custom_fixers'

warning: no files found matching '*' under directory 'jinja2/testsuite/res'

warning: no previously-included files matching '*' found under directory 'docs/_build'
warning: no previously-included files matching '*.pyc' found under directory 'jinja2'
warning: no previously-included files matching '*.pyc' found under directory 'docs'
warning: no previously-included files matching '*.pyo' found under directory 'jinja2'
warning: no previously-included files matching '*.pyo' found under directory 'docs'
Running setup.py install for colorlog
Running setup.py install for blinker
Running setup.py install for futures
Found existing installation: PyYAML 3.10
Uninstalling PyYAML:
Successfully uninstalled PyYAML
Running setup.py install for PyYAML
checking if libyaml is compilable
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC
-fwrapv -fPIC -I/usr/include/python2.6 -c build/temp.linux-x86_64-2.6/check_libyaml.c -o build/temp.linux-x86_642.6/check_libyaml.o
build/temp.linux-x86_64-2.6/check_libyaml.c:2:18: error: yaml.h: No such file or directory
build/temp.linux-x86_64-2.6/check_libyaml.c: In function main:
build/temp.linux-x86_64-2.6/check_libyaml.c:5: error: yaml_parser_t undeclared (first use in this function)
build/temp.linux-x86_64-2.6/check_libyaml.c:5: error: (Each undeclared identifier is reported only once
build/temp.linux-x86_64-2.6/check_libyaml.c:5: error: for each function it appears in.)
build/temp.linux-x86_64-2.6/check_libyaml.c:5: error: expected ; before parser
build/temp.linux-x86_64-2.6/check_libyaml.c:6: error: yaml_emitter_t undeclared (first use in this function)
build/temp.linux-x86_64-2.6/check_libyaml.c:6: error: expected ; before emitter
build/temp.linux-x86_64-2.6/check_libyaml.c:8: warning: implicit declaration of function
yaml_parser_initialize
build/temp.linux-x86_64-2.6/check_libyaml.c:8: error: parser undeclared (first use in this function)
build/temp.linux-x86_64-2.6/check_libyaml.c:9: warning: implicit declaration of function yaml_parser_delete
build/temp.linux-x86_64-2.6/check_libyaml.c:11: warning: implicit declaration of function
yaml_emitter_initialize
build/temp.linux-x86_64-2.6/check_libyaml.c:11: error: emitter undeclared (first use in this function)
build/temp.linux-x86_64-2.6/check_libyaml.c:12: warning: implicit declaration of function yaml_emitter_delete
libyaml is not found or a compiler error: forcing --without-libyaml
(if libyaml is installed correctly, you may need to
specify the option --include-dirs or uncomment and

modify the parameter include_dirs in setup.cfg)

skipping 'ext/_yaml.c' Cython extension (up-to-date)
Running setup.py install for csapi
Running setup.py install for MarkupSafe
building 'markupsafe._speedups' extension
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC
-fwrapv -fPIC -I/usr/include/python2.6 -c markupsafe/_speedups.c -o build/temp.linux-x86_642.6/markupsafe/_speedups.o
gcc -pthread -shared build/temp.linux-x86_64-2.6/markupsafe/_speedups.o -L/usr/lib64 -lpython2.6 -o
build/lib.linux-x86_64-2.6/markupsafe/_speedups.so
Running setup.py install for docopt
Successfully installed python-gflags jinja2 colorlog blinker futures PyYAML csapi MarkupSafe docopt
Cleaning up...
[ec2-user@CDH PerfKitBenchmarker-1.0.1]$ ./pkb.py --cloud=AWS --benchmarks=iperf -machine_type=t2.medium
Traceback (most recent call last):
File "./pkb.py", line 19, in <module>
from perfkitbenchmarker.pkb import Main
File "/home/ec2-user/PerfKitBenchmarker-1.0.1/perfkitbenchmarker/pkb.py", line 60, in <module>
from perfkitbenchmarker import archive
File "/home/ec2-user/PerfKitBenchmarker-1.0.1/perfkitbenchmarker/archive.py", line 22, in <module>
from perfkitbenchmarker.providers.aws.util import AWS_PATH
File "/home/ec2-user/PerfKitBenchmarker-1.0.1/perfkitbenchmarker/providers/__init__.py", line 18, in <module>
from perfkitbenchmarker import import_util
File "/home/ec2-user/PerfKitBenchmarker-1.0.1/perfkitbenchmarker/import_util.py", line 17, in <module>
import importlib
ImportError: No module named importlib
[ec2-user@CDH PerfKitBenchmarker-1.0.1]$ ./pkb.py --cloud=AWS --benchmarks=iperf -machine_type=t2.medium
Traceback (most recent call last):
File "./pkb.py", line 19, in <module>
from perfkitbenchmarker.pkb import Main
File "/home/ec2-user/PerfKitBenchmarker-1.0.1/perfkitbenchmarker/pkb.py", line 61, in <module>
from perfkitbenchmarker import linux_benchmarks
File "/home/ec2-user/PerfKitBenchmarker-1.0.1/perfkitbenchmarker/linux_benchmarks/__init__.py", line 29
for module in BENCHMARKS}
^
SyntaxError: invalid syntax

[ec2-user@CDH PerfKitBenchmarker-1.0.1]

Hi-Bench Benchmarking Runs on Hadoop

[ec2-user@CDH ~]$ cd hibench/HiBench/src
[ec2-user@CDH src]$ pwd
/home/ec2-user/hibench/HiBench/src
[ec2-user@CDH src]$ mvn clean package -D spark1.5 -D MR2
...
Downloading: https://oss.sonatype.org/content/groups/scala-tools/org/apache/maven/maven-error-diagnostics/2.0/mavenerror-diagnostics-2.0.pom
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/maven-error-diagnostics/2.0/maven-errordiagnostics-2.0.pom
Downloaded: http://repo.maven.apache.org/maven2/org/apache/maven/maven-error-diagnostics/2.0/maven-errordiagnostics-2.0.pom (812 B at 27.3 KB/sec)
Downloading: https://oss.sonatype.org/content/groups/scala-tools/org/apache/maven/maven-plugin-registry/2.0/mavenplugin-registry-2.0.pom
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-registry/2.0/maven-plugin-registry2.0.pom
Downloaded: http://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-registry/2.0/maven-plugin-registry2.0.pom (2 KB at 41.8 KB/sec)
Downloading: https://oss.sonatype.org/content/groups/scala-tools/org/apache/maven/maven-plugin-descriptor/2.0/mavenplugin-descriptor-2.0.pom
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-descriptor/2.0/maven-plugindescriptor-2.0.pom
Downloaded: http://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-descriptor/2.0/maven-plugindescriptor-2.0.pom (2 KB at 47.6 KB/sec)
Downloading: https://oss.sonatype.org/content/groups/scala-tools/org/apache/maven/maven-monitor/2.0/maven-monitor2.0.pom
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/maven-monitor/2.0/maven-monitor-2.0.pom
Downloaded: http://repo.maven.apache.org/maven2/org/apache/maven/maven-monitor/2.0/maven-monitor-2.0.pom (400 B at
13.5 KB/sec)
Downloading: https://oss.sonatype.org/content/groups/scala-tools/org/codehaus/plexus/plexus-utils/1.4.2/plexusutils-1.4.2.pom
Downloading: http://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/1.4.2/plexus-utils-1.4.2.pom
Downloaded: http://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/1.4.2/plexus-utils-1.4.2.pom (2 KB
at 63.8 KB/sec)
Downloading: https://oss.sonatype.org/content/groups/scala-tools/org/codehaus/plexus/plexus-archiver/1.0-alpha7/plexus-archiver-1.0-alpha-7.pom

Downloading: http://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-archiver/1.0-alpha-7/plexus-archiver1.0-alpha-7.pom
Downloaded: http://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-archiver/1.0-alpha-7/plexus-archiver-1.0alpha-7.pom (2 KB at 34.0 KB/sec)
Downloading: https://oss.sonatype.org/content/groups/scala-tools/org/codehaus/plexus/plexus-components/1.1.6/plexuscomponents-1.1.6.pom
Downloading: http://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-components/1.1.6/plexus-components1.1.6.pom
Downloaded: http://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-components/1.1.6/plexus-components1.1.6.pom (2 KB at 61.5 KB/sec)
Downloading: https://oss.sonatype.org/content/groups/scala-tools/org/codehaus/plexus/plexus/1.0.8/plexus-1.0.8.pom
Downloading: http://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/1.0.8/plexus-1.0.8.pom
Downloaded: http://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/1.0.8/plexus-1.0.8.pom (8 KB at 235.4
KB/sec)
Downloading: https://oss.sonatype.org/content/groups/scala-tools/org/apache/maven/reporting/maven-reportingimpl/2.0.4/maven-reporting-impl-2.0.4.pom
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting-impl/2.0.4/mavenreporting-impl-2.0.4.pom
Downloaded: http://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting-impl/2.0.4/mavenreporting-impl-2.0.4.pom (2 KB at 50.1 KB/sec)
Downloading: https://oss.sonatype.org/content/groups/scala-tools/org/apache/maven/reporting/mavenreporting/2.0.4/maven-reporting-2.0.4.pom
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting/2.0.4/maven-reporting2.0.4.pom
Downloaded: http://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting/2.0.4/maven-reporting2.0.4.pom (700 B at 23.6 KB/sec)
...
[INFO]
[INFO]
[INFO]
[INFO]
[INFO]
[INFO]
[INFO]
[INFO]
[INFO]
[INFO]
[INFO]

-----------------------------------------------------------------------Reactor Summary:
hibench ...........................................
nutchindexing .....................................
mahout ............................................
hivebench .........................................
Sparkbench ........................................
PEGASUS: A Peta-Scale Graph Mining System .........
HiBench data generation tools .....................
Streamingbench ....................................

SUCCESS
SUCCESS
SUCCESS
SUCCESS
SUCCESS
SUCCESS
SUCCESS
SUCCESS

[0.287s]
[1:06.917s]
[39.758s]
[18.672s]
[2:00.383s]
[2.814s]
[6.413s]
[0.004s]

[INFO] Streamingbench data generation tools .............. SUCCESS [6.504s]

[INFO] Streaming Benchmark Zookeeper Helper Utils ........ SUCCESS [20.996s]
[INFO] Streaming Benchmark Spark ......................... SUCCESS [1:25.198s]
[INFO] Streaming Benchmark Storm ......................... FAILURE [45.001s]
[INFO] Streaming Benchmark Samza ......................... SKIPPED
[INFO] -----------------------------------------------------------------------[INFO] BUILD FAILURE
[INFO] -----------------------------------------------------------------------[INFO] Total time: 6:53.871s
[INFO] Finished at: Fri Dec 11 10:38:27 UTC 2015
[INFO] Final Memory: 74M/326M
[INFO] -----------------------------------------------------------------------[ERROR] Failed to execute goal org.apache.thrift.tools:maven-thrift-plugin:0.1.11:compile (thrift-sources) on
project streaming-bench-storm: thrift did not exit cleanly. Review output for more information. -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]
mvn <goals> -rf :streaming-bench-storm

Since only Storm Build failed, proceeding with

wordcount
[ec2-user@CDH prepare]$ pwd
/home/ec2-user/hibench/HiBench/workloads/wordcount/prepare
[ec2-user@CDH prepare]$ ./prepare.sh
patching args=
Parsing conf: /home/ec2-user/hibench/HiBench/conf/00-default-properties.conf
Parsing conf: /home/ec2-user/hibench/HiBench/conf/01-default-streamingbench.conf

Parsing conf: /home/ec2-user/hibench/HiBench/conf/10-data-scale-profile.conf

Parsing conf: /home/ec2-user/hibench/HiBench/conf/20-samza-common.conf
Parsing conf: /home/ec2-user/hibench/HiBench/conf/30-samza-workloads.conf
Parsing conf: /home/ec2-user/hibench/HiBench/conf/99-user_defined_properties.conf
Parsing conf: /home/ec2-user/hibench/HiBench/workloads/wordcount/conf/00-wordcount-default.conf
Parsing conf: /home/ec2-user/hibench/HiBench/workloads/wordcount/conf/10-wordcount-userdefine.conf
Traceback (most recent call last):
File "/home/ec2-user/hibench/HiBench/bin/functions/load-config.py", line 556, in <module>
load_config(conf_root, workload_root, workload_folder, patching_config)
File "/home/ec2-user/hibench/HiBench/bin/functions/load-config.py", line 161, in load_config
generate_optional_value()
File "/home/ec2-user/hibench/HiBench/bin/functions/load-config.py", line 297, in generate_optional_value
assert hadoop_version, "ERROR, execute '%s' with no return, please confirm hadoop environment is configured
properly." % cmd
AssertionError: ERROR, execute '/usr/bin/hadoop/bin/hadoop version | head -1 | cut -d \
-f 2' with no return,
please confirm hadoop environment is configured properly.
/home/ec2-user/hibench/HiBench/bin/functions/workload-functions.sh: line 39: .: filename argument required
.: usage: . filename [arguments]
start HadoopPrepareWordcount bench
./prepare.sh: line 25: INPUT_HDFS: unbound variable