wiki:ccafebgpOld
Last modified 9 years ago Last modified on 10/06/10 07:43:10

Installing ccaffeine on BG/P (Surveyor at Argonne)

Current build instructions (Notes by Stephen Tramer tramer@…)

You will need both the latest versions of cca-spec-babel and ccaffeine from the repository for this. ccaffeine is not configured for spec-classic or spec-neo; if you want either of those features you're on your own (but they should compile out of the box with a little love to the config/config.sub file to add the -bgp OS type; see the config/config.sub file for cca-spec-babel).

See here for instructions on checking out the cca-tools sources. You will need to build the latest version of babel as described on poring babel to BG/P before attempting any of this. You will also need the latest development libtool installed. See instructions on how to use this (or where the prebuilt one is) on the babel porting instruction page.

Make sure that your ccaffeine trunk checkout is up to date as well.

Building cca-spec-babel

  1. Check cca-spec-babel out from the repository, and configure:
./configure CPP="gcc -E" CC=mpixlc_r CXX=mpixlcxx_r FC=mpixlf2003_r F77=mpixlf77_r \
  --with-babel-config=$BABEL_INSTALL/bin/powerpc64-ibm-bgp-babel-config \
  --with-libxml2=$LIBXML2_LOC --with-babel-python \
  --host=powerpc-bgp-linux-gnu --target=powerpc64-ibm-bgp \
  target_alias=powerpc64-ibm-bgp --prefix=$CCA_INSTALL_DIR
  1. Replace the LIBTOOL_LINKFLAGS variable in makefiles:
find . -name "Makefile" -exec sed -il -e 's|LIBTOOL_LINKFLAGS=.*|LIBTOOL_LINKFLAGS=-qnostaticlink|g' {} \; -print
  1. Add -qnostaticlink to some more builds that don't check LIBTOOL_LINKFLAGS:
find . -name Makefile -exec sed -il -e 's|LOCALFLAGS=\(.*\)|LOCALFLAGS=\1 -qnostaticlink|g' {} \; -print
  1. 'make clean all install'
  1. Edit *Vars* scripts to point to the right genSCLCCA.sh:
cd $CCA_INSTALL_DIR/share
find . -name '*Vars*' -exec sed -i1 \
  -e "s|^CCASPEC_GENSCLCCA_SH=.*|CCASPEC_GENSCLCCA_SH='$CCA_INSTALL_DIR/bin/genSCLCCA.sh'|g" {} \; -print

Building ccaffeine

Building ccaffeine will not work out of the box. You need to edit some parts of the cca-spec-babel install and also some of the makefiles for ccaffeine itself. NOTE: You will be modifying files which are tracked by svn, and should NOT do a commit until you revert them.

Cut & Paste instructions

Make sure that CCA_INSTALL_DIR and LIBTOOL_LOC variables are set before cutting and pasting. bgpfix.sh handles all of the makefile modifications that are mentioned below.

chmod +x bgpfix.sh
./bgpfix.sh
./configure CPP="gcc -E" CC=mpixlc_r CXX=mpixlcxx_r FC=mpixlf2003_r F77=mpixlf77_r \
  --with-csb-config=$CCA_INSTALL_DIR/bin/cca-spec-babel-config \
  --with-babel-python LDFLAGS="-qnostaticlink" --host=powerpc-bgp-linux-gnu \
  --target=powerpc64-ibm-bgp target_alias=powerpc64-ibm-bgp --prefix=$CCA_INSTALL_DIR \
  --with-boost=/home/norris/soft/boost-1_38_0/include/boost-1_38 \
  --with-xml2-config=/gpfs/home/projects/ccatools/libxml2/bin/xml2-config \
  --with-xml2-includes=-I/gpfs/home/projects/ccatools/libxml2/include/libxml2 \
  --with-xml2-libs=-L/gpfs/home/projects/ccatools/libxml2/lib \
  --with-babel-libtool=$LIBTOOL_LOC/bin/libtool
make clean all install
cd cxx/drivers
make clean
touch .preload-lo
make all
rm .preload-lo
make .preload-lo
cd ../..
make install

Manual instructions

Note that you may skip steps 1-3 and part of step 7 (the Makefile modification) by running the bgpfix.sh script. These steps are documented here for posterity and also for anyone who wants to manually check the installation in case it breaks at some future point.

  1. Remove all instances of -Wall flags. '-Wall' is not compatible with the IBM C++ compilers. You will need to edit $CCAFE_SRC/cxx/Makefile.Rules.in and $CCAFE_SRC/cxx/gnu/getopt/Makefile.in. Or you can run the following:
find . -name "Make*.in" -exec sed -il -e 's|-Wall||g' {} \; -print
  1. Edit $CCAFE_SRC/config/shlib.m4:
===================================================================
--- shlib.m4	(revision 196)
+++ shlib.m4	(working copy)
@@ -1133,7 +1133,7 @@
 		CXX_SHLIB_LD="${CXX} -shared"
 		DL_OBJS="tclLoadDl.o"
 		DL_LIBS="-ldl"
-		LDFLAGS="-rdynamic"
+		LDFLAGS="-Wl,-dy"
 		CC_SEARCH_FLAGS='-Wl,-rpath,${LIB_RUNTIME_DIR}'
 		LD_SEARCH_FLAGS=${CC_SEARCH_FLAGS}
 	    else

This signals the linker to perform dynamic linking.

  1. Fix setup.py calls. Replace any call to python which runs a 'setup.py' script with a reference to Boyana's python for the login nodes. This version has a 'correct' distutils, which links against the libraries and include files for the BG/P nodes instead of the login nodes. At the same time, you will need to add the --compiler=mpixlc option to each invoke of setup.py. The following will handle it for you:
find . -name "Make*.in" -exec \
    sed -il -e 's|python setup.py|/home/norris/soft/bin/python setup.py|g'\
            -e 's|build_ext|build_ext --compiler=mpixlc|g' {} \; -print
  1. Run 'aclocal && autoconf'.
  1. Configure:
./configure CPP="gcc -E" CC=mpixlc_r CXX=mpixlcxx_r FC=mpixlf2003_r F77=mpixlf77_r \
  --with-csb-config=$CCA_INSTALL_DIR/bin/cca-spec-babel-config \
  --with-babel-python LDFLAGS="-qnostaticlink" --host=powerpc-bgp-linux-gnu \
  --target=powerpc64-ibm-bgp target_alias=powerpc64-ibm-bgp --prefix=$CCA_INSTALL_DIR \
  --with-boost=/home/norris/soft/boost-1_38_0/include/boost-1_38 \
  --with-xml2-config=/gpfs/home/projects/ccatools/libxml2/bin/xml2-config \
  --with-xml2-includes=-I/gpfs/home/projects/ccatools/libxml2/include/libxml2 \
  --with-xml2-libs=-L/gpfs/home/projects/ccatools/libxml2/lib \
  --with-babel-libtool=$LIBTOOL_LOC/bin/libtool
  1. Run 'make clean all install'
  1. Rebuild the drivers. Even though everything builds from steps 1-9, it is not guaranteed to link correctly. In order to guarantee the right libraries are linked, cd to 'trunk/cxx/drivers' and edit trunk/cxx/drivers/Makefile so that the FWK_LIB variable is replaced with the following:
FWK_LIB= -L$(libdir) -L$(libdir)/ccafe-0.8.8 -lccafePreload -lccafeCore -rpath $(libdir) \
-R$(libdir) $(CSNRUNTIMEFLAGS)

Then do the following:

make clean
touch .preload-lo
make all
rm .preload-lo
make .preload-lo

Then head back into trunk and run 'make install'. You (should) have a working ccaffeine build!