wiki:ccafeBgpNew

Version 30 (modified by muszala, 9 years ago) (diff)

--

<< back >>

Building the CCA Tool Chain on Surveyor

The build below works. This is still a work in progress however as we try to further automate the entire build process.

The shell prompt is >>. Valid instructions as of Oct. 13 2010--SPM. The order of the build is summarized here.

  1. Libtool-2.4
  2. zlib-1.2.5
  3. libxml-2.7.3
    • run select regression tests
  4. babel out of dev repo
    • run select regression tests
  5. cca-spec-babel
  6. ccaffeine
  7. bocca
  8. Test the tool chain.

1. Build Libtool 2.4

Libtool may be built using patches or may be built by downloading an already patched tar-ball and building with that.

a. Build Libtool with provided patches

  • Download the source
    >>wget http://ftp.gnu.org/gnu/libtool/libtool-2.4.tar.gz
    >>tar xvfz libtool-2.4.tar.gz
    >>cd libtool-2.4/
    
  • Use wget or download the following two patches into the libtool-2.4/ directory.
    >>wget http://nucleus.txcorp.com/~muszala/ccaEcloud/bgpPorting/libtool.m4.patch
    >>wget http://nucleus.txcorp.com/~muszala/ccaEcloud/bgpPorting/shl_load.c.patch
    
  • Patch the Libtool-2.4 source and run 'bootstrap', which rebuilds autotools:
    >>pwd
    /home/muszala/ccaToolChain/libtool-2.4
    >> patch -p0 -i shl_load.c.patch
    patching file libltdl/loaders/shl_load.c
    >> patch -p0 -i libtool.m4.patch
    patching file libltdl/m4/libtool.m4
    >>./bootstrap
    
  • Run configure. I set my install location in my .bashrc file. I'm building out of my home directory and not scratch since you cannot run regression tests out of /scratch and we will need to test both libxml and babel later on.
    >>env | grep LIBTOOL_LOCATION
    LIBTOOL_LOCATION=/gpfs/home/muszala/ccaToolBuild/install/libtool-2.4
    >>./configure CC=bgxlc_r CXX=bgxlC_r F77=bgxlf_r FC=bgxlf90_r \
    LDFLAGS="-qnostaticlink -r" --enable-shared \
    --disable-static --enable-ltdl-install --prefix=$LIBTOOL_LOCATION \
    --host=powerpc-bgp-linux-gnu CFLAGS=-qpic
    >>make ; make install
    
  • All should complete without errors. The last few lines of the entire build process should be:
     /usr/bin/install -c -m 644 ./doc/libtool.1 ./doc/libtoolize.1 '/gpfs/home/muszala/ccaToolBuild/install/libtool-2.4/share/man/man1'
    make[3]: Leaving directory `/gpfs/home/muszala/ccaToolChain/libtool-2.4'
    make[2]: Leaving directory `/gpfs/home/muszala/ccaToolChain/libtool-2.4'
    make[1]: Leaving directory `/gpfs/home/muszala/ccaToolChain/libtool-2.4'
    

b. Build Libtool from patched tar ball

  • You do not need to had patch this tar ball.
    >>wget http://nucleus.txcorp.com/~muszala/ccaEcloud/bgpPorting/libtool-2.4-bgpPatch.tgz
    >>tar xvfz libtool-2.4-bgpPatch.tgz
    >>cd libtool-2.4-bgpPatch/
    >>./configure CC=bgxlc_r CXX=bgxlC_r F77=bgxlf_r FC=bgxlf90_r \
    LDFLAGS="-qnostaticlink -r" --enable-shared \
    --disable-static --enable-ltdl-install --prefix=$LIBTOOL_LOCATION \
    --host=powerpc-bgp-linux-gnu CFLAGS=-qpic
    >>make ; make install
    

2. Build zlib-1.2.5

  • Download source and extract:
    >>wget http://zlib.net/zlib-1.2.5.tar.gz
    >>tar xvfz zlib-1.2.5.tar.gz
    >>cd zlib-1.2.5
    
  • Configure and build. I use bash so can set environment variables this way:
    >>CC=mpixlc CFLAGS="-qpic" ./configure  --static --prefix=/gpfs/home/muszala/ccaToolBuild/install/zlib-1.2.5
    >>make
    
  • make install will end with an error, but this error occurs after moving libz.a so it should be OK.
    >> make install
    cp libz.a /gpfs/home/muszala/ccaToolBuild/install/zlib-1.2.5/lib
    cp  /gpfs/home/muszala/ccaToolBuild/install/zlib-1.2.5/lib
    cp: missing destination file operand after `/gpfs/home/muszala/ccaToolBuild/install/zlib-1.2.5/lib'
    Try `cp --help' for more information.
    make: *** [install-libs] Error 1
    >>ls ../../ccaToolBuild/install/zlib-1.2.5/lib/
    libz.a  pkgconfig
    

3. Build libxml-2.7.3

Sub-sections a and b allow you to build from original source or patched source that we provide. Sub-section c pertains to both cases and is used to run two select regression test on Surveyor compute nodes. These tests are also used to show that our libtool build is correct.

a. Build and patch yourself

  • Download and extract source:
    >>wget http://xmlsoft.org/sources/libxml2-2.7.3.tar.gz
    >>tar xvfz libxml2-2.7.3.tar.gz
    >>cd libxml2-2.7.3/
    
  • Rerun aclocal, autoconf and automake and make sure you point aclocal to the libtool installation directory you used to build libtool-2.4.
    >>env | grep LIBTOOL_LOCATION
    LIBTOOL_LOCATION=/gpfs/home/muszala/ccaToolBuild/install/libtool-2.4
    >>aclocal --verbose -I /gpfs/home/muszala/ccaToolBuild/install/libtool-2.4/share/aclocal
    >>autoconf
    >>automake
    

  • Important step. Since libxml was released with an older version of libtool, you also need to update ltmain.sh.
    >>cp /gpfs/home/muszala/ccaToolBuild/install/libtool-2.4/share/libtool/config/ltmain.sh .
    
  • configure, then build and install.
    >>env | grep LIBXML2_LOC
    LIBXML2_LOC=/gpfs/home/muszala/ccaToolBuild/install/libxml-2.7.3
    >>./configure --prefix=$LIBXML2_LOC CC=mpixlc_r CPP="gcc -E" \
    --with-python=/bgsys/drivers/ppcfloor/gnu-linux/bin/python \
    --with-zlib=/gpfs/home/muszala/ccaToolBuild/install/zlib-1.2.5 --enable-shared \
    --disable-static LDFLAGS=-qnostaticlink
    >>make ; make install
    

b. Use a pre-patched version

  • I use the same location for LIBXML2_LOC as in the above example.
    >>wget http://nucleus.txcorp.com/~muszala/ccaEcloud/bgpPorting/
    >>tar xvfz libxml2-2.7.3-bgpPatch.tgz
    >>cd libxml2-2.7.3-bgpPatch/ 
    >>./configure --prefix=$LIBXML2_LOC CC=mpixlc_r CPP="gcc -E" \
    --with-python=/bgsys/drivers/ppcfloor/gnu-linux/bin/python \
    --with-zlib=/gpfs/home/muszala/ccaToolBuild/install/zlib-1.2.5 --enable-shared \
    --disable-static LDFLAGS=-qnostaticlink
    >>make ; make install
    

c. Run two regression tests on back-end nodes.

>>pwd
>>cd .libs
>>qsub --mode script -A cca-tools -t 60 -n 1 -O testapi --env LD_LIBRARY_PATH=$LIBXML2_LOC/lib ./testapi
>>qsub --mode script -A cca-tools -t 60 -n 1 -O runtest --env LD_LIBRARY_PATH=$LIBXML2_LOC/lib ./runtest
  • When the runs complete, you should see 6 new files:
    >>ls -ltr testapi.* runtest.*
    -rw-r--r-- 1 muszala users     0 2010-10-06 05:14 testapi.error
    -rw-r--r-- 1 muszala users  1390 2010-10-06 05:15 runtest.output
    -rw-r--r-- 1 muszala users   105 2010-10-06 05:15 runtest.error
    -rw-r--r-- 1 muszala users  1673 2010-10-06 05:15 testapi.output
    -rw-r--r-- 1 muszala cobalt 3229 2010-10-06 05:15 runtest.cobaltlog
    -rw-r--r-- 1 muszala cobalt 3227 2010-10-06 05:16 testapi.cobaltlog
    
  • The results of testapi.output look like:
    >>tail testapi.output 
    Testing xmlsave : 4 of 10 functions ...
    Testing xmlschemas : 15 of 25 functions ...
    Testing xmlschemastypes : 26 of 34 functions ...
    Testing xmlstring : 26 of 30 functions ...
    Testing xmlunicode : 166 of 166 functions ...
    Testing xmlwriter : 51 of 79 functions ...
    Testing xpath : 30 of 38 functions ...
    Testing xpathInternals : 106 of 117 functions ...
    Testing xpointer : 17 of 21 functions ...
    Total: 1161 functions, 291335 tests, 0 errors
    
  • While the results of runtest.output and runtest.error look like:
    >>tail runtest.output 
    ## Path URI conversion tests
    ## Schemas regression tests
    ## Relax-NG regression tests
    ## Relax-NG streaming regression tests
    ## Pattern regression tests
    ## C14N with comments regression tests
    ## C14N without comments regression tests
    ## C14N exclusive without comments regression tests
    ## Catalog and Threads regression tests
    Total 8 tests, 2 errors, 0 leaks
    >>more runtest.error 
    failed the URL passing test for urip://example.com/a b.htmlSpecific platform thread support not detected
    

4. Build Babel out of the dev. repo

  • The babel directions have their own page since babel is often used without the entire CCA tool chain.
  • If you're still reading this...awesome. The next step is building babel. I prefer to build out of the development repository since I can then commit changes back quicker if I find something that needs fixing. I will assume you have an account. If you do not have one contact Tom Epperly at llnl.
  • For now build without Java and without any Fortran variant. If we bring Impact (a Fortran code) back into the mix, we will have to address this.
    >>svn co svn+ssh://svn.cca-forum.org/svn/babel/trunk babelDev
    >>cd babelDev
    >>./autotool_rebuild.sh
    >>./configure CC=mpixlc CXX=mpixlcxx FC=mpixlf90 F77=mpixlf77 --prefix=$BABEL_INSTALL --host=powerpc-bgp-linux-gnu \
    --without-sidlx \
    --with-gcc=no --enable-python=/bgsys/drivers/ppcfloor/gnu-linux/bin/python --with-libxml2=$LIBXML2_LOC \
    LDFLAGS=-qnostaticlink --enable-shared --disable-static host_alias=powerpc-bgp-linux-gnu --target=powerpc64-ibm-bgp \
    --with-ltdl-lib=$LIBTOOL_LOCATION/lib --with-ltdl-include=$LIBTOOL_LOCATION/include --without-libparsifal \
    PYTHONPATH=/soft/apps/python/python-2.6-cnk-gcc/numpy-1.3.0/lib/python2.6/site-packages CPP="gcc -E" \
    target_alias=powerpc64-ibm-bgp --enable-fortran03=yes --enable-fortran90=yes --enable-fortran77=yes --enable-java=no \
    --with-arch=IBMXL --with-F90-vendor=IBMXL
    >>./contrib/bgpSurveyorLibtoolFix.sh
    >>make ; make install
    
  • The build and install should have proceeded without error.
  • Since I was having problems with getting regression tests to run, I first tried to get something in examples/hello working:
    >>cd examples/hello
    >>make check
    >>cd ../..
    >>./contrib/bgpSurveyorExampleFix.sh
    >>cd examples/hello/runPython/
    >>qsub --mode script -A cca-tools -t 5 -n 1 ./runPy2C
    
  • If it works you should see the following in an output file:
    >>more 198866.output
    Hello world!
    
  • Now run:
    >>qsub --mode script -A cca-tools -t 5 -n 1 ./runPy2Cxx
    >>more 198867.output
    Hello world2!
    process group 974 has completed 
    
  • Try running something from runCxx:
    >>cd ../runCxx
    >>qsub --mode script -A cca-tools -t 15 -n 1 runAll.sh
    >>more 198868.output
    Hello from C...
    Hello from C++...
    f77 support not enabled
    

And the output is as we expect it. Now I will move on to regression tests (regression/args) to make sure Cxx->Python and Python->Python work.

  • All regression tests must be run individually. Note, when you run a test with 1 node we're still charged for using all 64 processors. For this reason, I run one or two tests to make sure babel was compiled correctly but nothing more. If we run into problems down the road we will test that individual babel functionality or if necessary, build an MPI test harness which would allow us to use all 64 processors.
    >>cd regression/output
    >>make check
    >>cd ../args ; make check
    >>cd ../.. ; ./contrib/bgpSurveyorRegressionFix.sh ; cd -
    
  • Python -> {C, Cxx, Python}. For this I change directories to runPython and use the runPy2* scripts. I also added the line below to runPy2C.
    PYTHONPATH=/soft/apps/python/python-2.6-cnk-gcc/numpy-1.3.0/lib/python2.6/site-packages:${PYTHONPATH}
    
    • Python -> C
      >>cd ../runPython
      >>qsub --mode script -A cca-tools -t 5 -n 1 ./runPy2C
      198879
      >>tail 198879.output
      COMMENT: (1, 3.14 + 3.14i) == obj.passoutdcomplex()
      RESULT 46 PASS
      PART 47
      COMMENT: (1, 3.14 - 3.14i) == obj.passinoutdcomplex(3.14 + 3.14i)
      RESULT 47 PASS
      PART 48
      COMMENT: (3.14 + 3.14i, 3.14 + 3.14i, 3.14 + 3.14i) = obj.passeverywheredcomplex(3.14 + 3.14i, 3.14 - 3.14i)
      RESULT 48 PASS
      TEST_RESULT PASS
      process group 991 has completed
      
    • Python -> Cxx
      >>cd ../runPython
      >>qsub --mode script -A cca-tools -t 5 -n 1 ./runPy2Cxx
      198880
      >>tail 198880.output
      COMMENT: (1, 3.14 + 3.14i) == obj.passoutdcomplex()
      RESULT 46 PASS
      PART 47
      COMMENT: (1, 3.14 - 3.14i) == obj.passinoutdcomplex(3.14 + 3.14i)
      RESULT 47 PASS
      PART 48
      COMMENT: (3.14 + 3.14i, 3.14 + 3.14i, 3.14 + 3.14i) = obj.passeverywheredcomplex(3.14 + 3.14i, 3.14 - 3.14i)
      RESULT 48 PASS
      TEST_RESULT PASS
      process group 993 has completed
      
    • Python -> Python
      >>cd Py2Py
      >>qsub --mode script -A cca-tools -t 5 -n 1 ./runPy2Py.sh
      198882
      >>tail 198882.output
      PART 46
      COMMENT: (1, 3.14 + 3.14i) == obj.passoutdcomplex()
      RESULT 46 PASS
      PART 47
      COMMENT: (1, 3.14 - 3.14i) == obj.passinoutdcomplex(3.14 + 3.14i)
      RESULT 47 PASS
      PART 48
      COMMENT: (3.14 + 3.14i, 3.14 + 3.14i, 3.14 + 3.14i) = obj.passeverywheredcomplex(3.14 + 3.14i, 3.14 - 3.14i)
      RESULT 48 PASS
      TEST_RESULT PASS
      
  • Cxx -> {Cxx, C, Python}. I run the binaries directly out of the .libs directory. The SIDL_DLL_PATH to the library as well as the argument to lt-runAll is changing.
    • Cxx -> Cxx
      >>qsub --mode vn -n 1 -A cca-tools -t 15 \
      --env SIDL_DEBUG_OPEN=1:\
      LD_LIBRARY_PATH="../../../lib/sidlstubs/Cxx/.libs:/bgsys/drivers/ppcfloor/lib64:/dbhome/db2cat/sqllib/lib64:/dbhome/db2cat/sqllib/lib32":\
      SIDL_DLL_PATH="../libCxx/libImpl.scl;../../output/libC/libOutput.scl;../../../runtime/sidlx/libsidlx.scl" \
      /gpfs/home/muszala/ccaToolChain/babelDev/regression/args/runCxx/.libs/lt-runAll C++
      >>tail 198872.output
      COMMENT: obj.passoutdcomplex( out ) == true && out.real() == 3.14 && out.imag() == 3.14
      RESULT 38 PASS
      PART 39
      COMMENT: obj.passinoutdcomplex( inout ) == true && inout.real() == 3.14 && inout.imag() == -3.14
      RESULT 39 PASS
      COMMENT: retval = obj.passeverywheredcomplex( in, out, inout );
      PART 40
      COMMENT: retval.real() == 3.14 && retval.imag() == 3.14 && out.real() == 3.14 && out.imag() == 3.14 && inout.real() == 3.14 && inout.imag() == 3.14
      RESULT 40 PASS
      TEST_RESULT PASS
      
    • Cxx -> C
      >>qsub --mode vn -n 1 -A cca-tools -t 15 \
      --env SIDL_DEBUG_OPEN=1:\
      LD_LIBRARY_PATH="../../../lib/sidlstubs/Cxx/.libs:/bgsys/drivers/ppcfloor/lib64:/dbhome/db2cat/sqllib/lib64:/dbhome/db2cat/sqllib/lib32":\
      SIDL_DLL_PATH="../libC/libImpl.scl;../../output/libC/libOutput.scl;../../../runtime/sidlx/libsidlx.scl" \
      /gpfs/home/muszala/ccaToolChain/babelDev/regression/args/runCxx/.libs/lt-runAll C
      >>q
      198875  muszala  00:15:00  1      queued  None  
      >>tail 198875.output
      COMMENT: obj.passoutdcomplex( out ) == true && out.real() == 3.14 && out.imag() == 3.14
      RESULT 38 PASS
      PART 39
      COMMENT: obj.passinoutdcomplex( inout ) == true && inout.real() == 3.14 && inout.imag() == -3.14
      RESULT 39 PASS
      COMMENT: retval = obj.passeverywheredcomplex( in, out, inout );
      PART 40
      COMMENT: retval.real() == 3.14 && retval.imag() == 3.14 && out.real() == 3.14 && out.imag() == 3.14 && inout.real() == 3.14 && inout.imag() == 3.14
      RESULT 40 PASS
      TEST_RESULT PASS
      
  • If you do run 'make check' you will build the tests, attempt to run them on the login nodes and all will be reported as broken. If you do run 'make check' to completion you will see the following:
    >>make check
    ...
    ...
    305 B struct/runPython/Py2Py/runPy2Py.sh              0   0% [parse error] 
    306 B sort/runPython/Py2Py/runPy2Py.sh                0   0% [parse error] 
    307 B strings/runPython/Py2Py/runPy2Py.sh             0   0% [parse error] 
    308 B wrapper/runPython/Py2Py/runPy2Py.sh             0   0% [parse error] 
    309 B wrapper/runPython/runPy2C                       0   0% [parse error] 
    310 B wrapper/runPython/runPy2Cxx                     0   0% [parse error] 
    Error: gantlet: 3 failed tests and 221 broken tests.
    make[4]: *** [run-gantlet] Error 224
    make[4]: Leaving directory `/gpfs/home/muszala/ccaToolChain/babelDev/regression'
    make[3]: *** [check-local] Error 2
    make[3]: Leaving directory `/gpfs/home/muszala/ccaToolChain/babelDev/regression'
    make[2]: *** [check-am] Error 2
    make[2]: Leaving directory `/gpfs/home/muszala/ccaToolChain/babelDev/regression'
    make[1]: *** [check-recursive] Error 1
    make[1]: Leaving directory `/gpfs/home/muszala/ccaToolChain/babelDev/regression'
    make: *** [check-recursive] Error 1
    

5. Build cca-babel-spec

  • Check out source:
    >>svn co https://cca-forum.svn.sourceforge.net/svnroot/cca-forum/cca-spec/trunk cca-spec
    >>cd cca-spec/cca-spec-babel
    
  • The rest of these instructions are very similar to those found here. I've put them here so the entire tool-chain build is on one page.
    >>env | grep CCA
    CCA_INSTALL_DIR=/gpfs/home/muszala/ccaToolBuild/install/ccaTools
    >> env | grep BABEL_INSTALL
    BABEL_INSTALL=/gpfs/home/muszala/ccaToolBuild/install/babel
    >>env | grep LIBXML2_LOC
    LIBXML2_LOC=/gpfs/home/muszala/ccaToolBuild/install/libxml-2.7.3 
    >>./configure CPP="gcc -E" CC=mpixlc_r CXX=mpixlcxx_r FC=mpixlf2003_r F77=mpixlf77_r \
      --with-babel-config=$BABEL_INSTALL/bin/powerpc64-ibm-bgp-babel-config \
      --with-libxml2=$LIBXML2_LOC --with-babel-python \
      --host=powerpc-bgp-linux-gnu --target=powerpc64-ibm-bgp \
      target_alias=powerpc64-ibm-bgp --prefix=$CCA_INSTALL_DIR
    
  • Replace the LIBTOOL_LINKFLAGS variable in makefiles:
    >>find . -name "Makefile" -exec sed -il -e 's|LIBTOOL_LINKFLAGS=.*|LIBTOOL_LINKFLAGS=-qnostaticlink|g' {} \; -print
    
  • Add -qnostaticlink to some more builds that don't check LIBTOOL_LINKFLAGS:
    >>find . -name Makefile -exec sed -il -e 's|LOCALFLAGS=\(.*\)|LOCALFLAGS=\1 -qnostaticlink|g' {} \; -print
    
  • 'make clean all install'
  • Edit *Vars* scripts to point to the right genSCLCCA.sh:
    >>cd $CCA_INSTALL_DIR/share
    >>find . -name '*Vars*' -exec sed -i1 \
      -e "s|^CCASPEC_GENSCLCCA_SH=.*|CCASPEC_GENSCLCCA_SH='$CCA_INSTALL_DIR/bin/genSCLCCA.sh'|g" {} \; -print
    

6. Build ccaffeine

  • As in step 5, these instructions are similar to those found here. I have only kept the 'cut and paste' instructions and have modified path names in configure to match steps 1-5.
  • Check out the code:
    >>svn co https://cca-forum.svn.sourceforge.net/svnroot/cca-forum/ccafe/trunk/ccaffeine
    >>cd ccaffeine
    
  • Configure and build:
    >>chmod +x bgpfix.sh
    >>./bgpfix.sh
    >>./configure CPP="gcc -E" CC=mpixlc_r CXX=mpixlcxx_r FC=mpixlf2003_r F77=mpixlf77_r \
      --with-csb-config=$CCA_INSTALL_DIR/bin/cca-spec-babel-config \
      --with-babel-python LDFLAGS="-qnostaticlink" --host=powerpc-bgp-linux-gnu \
      --target=powerpc64-ibm-bgp target_alias=powerpc64-ibm-bgp --prefix=$CCA_INSTALL_DIR \
      --with-boost=/home/norris/soft/boost-1_38_0/include/boost-1_38 \
      --with-xml2-config=$LIBXML2_LOC/bin/xml2-config \
      --with-xml2-includes=-I/$LIBXML2_LOC/include/libxml2 \
      --with-xml2-libs=-L$LIBXML2_LOC/lib \
      --with-babel-libtool=$LIBTOOL_LOC/bin/libtool
    >>make clean all install
    >>cd cxx/drivers ; make clean ; touch .preload-lo ; make all ; rm .preload-lo ; make .preload-lo 
    >>cd ../..
    >>make install
    

7. Build bocca

  • Older instructions are found here while repository info can be found here.
  • Start by checking out the bocca source:
    >>svn co https://cca-forum.svn.sourceforge.net/svnroot/bocca/trunk bocca
    >>cd bocca
    
  • Configure and build:
    >>./configure CPP="gcc -E" CC=mpixlc CXX=mpixlcxx FC=mpixlf2003 F77=mpixlf77 LDFLAGS="-G -qnostaticlink" \
      --host=powerpc-bgp-linux-gnu --target=powerpc64-ibm-bgp target_alias=powerpc64-ibm-bgp \
      --prefix=$CCA_INSTALL_DIR --with-python=/bgsys/drivers/ppcfloor/gnu-linux/bin/python \
      --with-ccafe-config=$CCA_INSTALL_DIR/bin/ccafe-config \
      --with-cca-spec-babel-config=$CCA_INSTALL_DIR/bin/cca-spec-babel-config \
      --with-babel-config=$BABEL_INSTALL/bin/powerpc64-ibm-bgp-babel-config
    >>make ; make install
    

8. Testing the build

Please see boccaTestProj, which is also accessible from the previous page.