fortran test MPI_Errhandler_fatal fails with SMPD on the nightlies

[jayesh]

fortran/env_manager/functional/MPI_Errhandler_fatal

Make step:
make[1]: Entering directory `/sandbox/mpich2/nightly/output/comb15.ubuntu-hardy-amd64/test_1/IntelMPITEST/Test/fortran/env_manager/functional/MPI_Errhandler_fatal'
/sandbox/mpich2/nightly/output/comb15.ubuntu-hardy-amd64/test_1/mpich2-installed/bin/mpif77 -I/sandbox/mpich2/nightly/output/comb15.ubuntu-hardy-amd64/test_1/IntelMPITEST/include -I/sandbox/mpich2/nightly/output/comb15.ubuntu-hardy-amd64/test_1/IntelMPITEST/include -I/sandbox/mpich2/nightly/output/comb15.ubuntu-hardy-amd64/test_1/IntelMPITEST/Test/fortran/env_manager/functional/MPI_Errhandler_fatal   -fno-globals -Wno-globals -o node /sandbox/mpich2/nightly/output/comb15.ubuntu-hardy-amd64/test_1/IntelMPITEST/Test/fortran/env_manager/functional/MPI_Errhandler_fatal/node.F  /sandbox/mpich2/nightly/output/comb15.ubuntu-hardy-amd64/test_1/IntelMPITEST/lib/libmpitestf_mpich2.a -lm
make[1]: Leaving directory `/sandbox/mpich2/nightly/output/comb15.ubuntu-hardy-amd64/test_1/IntelMPITEST/Test/fortran/env_manager/functional/MPI_Errhandler_fatal'
Run step:
Fatal error in MPI_Send: Invalid rank, error stack:
MPI_Send(174): MPI_Send(buf=0x7fffc2e7a554, count=1, MPI_INTEGER, dest=2, tag=0, comm=0x84000000) failed
MPI_Send(99).: Invalid rank has value 2 but must be nonnegative and less than 2

job aborted:
rank: node: exit code[: error message]
0: bb33: -2
1: bb33: 1: process 1 exited without calling finalize

[jayesh]

The test passes when run locally on crush (unable to reproduce locally on crush). This looks to me like one of those output buffering problems that we have seen with SMPD before. The test program prints a (atleast) "PASSED" string in case of success and performs a barrier before doing an invalid MPI_Send() to abort the program. This output in stdout of the process (although the output is flushed on an abort in the client side it looks like the SMPD manager does not process this output) is not forwarded correctly to the stdout for the job. The mtest script looks for "PASSED" string to determine whether the test passed. Since we are too close to the next release (and since it appears to be a minor bug) I am pushing this ticket to the next release.

-Jayesh