Resolve mpiexec Conflict – EyeVision

Remove system mpiexec

which mpiexec
# /usr/bin/mpiexec
sudo apt-get remove --purge openmpi-bin libopenmpi-dev

# /home/kevin/.pyenv/shims/mpiexec

Install artifact mpiexec

julia --project=. -e 'using Pkg; Pkg.add("MPI")'
julia --project=. -e 'using Pkg; ENV["JULIA_MPI_BINARY"]="artifact"; Pkg.build("MPI")'
julia --project=. -e 'using MPI; println(MPI.mpiexec())'

Set Environment Variables

julia --project=. -e 'using Pkg; ENV["JULIA_MPI_BINARY"]="artifact"; Pkg.build("MPI")'

export PATH="$HOME/.julia/artifacts/xxxxxxx/bin:$PATH"
export MPIROOT=$HOME/.julia/artifacts/xxxxxxx
export PATH=$MPIROOT/bin:$PATH
export LD_LIBRARY_PATH=$MPIROOT/lib:$HOME/julia-1.11.6/lib/julia:$LD_LIBRARY_PATH

unset UCX_ERROR_SIGNALS UCX_MEMTYPE_CACHE OMPI_MCA_* OMPI_*

export MPIEXEC=$HOME/.julia/artifacts/xxxxxxx/bin/mpiexec
$MPIEXEC --version
export PATH=$HOME/.julia/artifacts/xxxxxxx/bin:$PATH
export LD_LIBRARY_PATH=$HOME/.julia/artifacts/xxxxxxx/lib:$HOME/julia-1.11.6/lib/julia:$LD_LIBRARY_PATH

Check SetUp

mpiexec -n 1 julia --project=. -e 'using MPI, Libdl; MPI.Init(); foreach(println, filter(x->occursin("libmpi", x), Libdl.dllist())); println(MPI.Get_library_version()); MPI.Finalize()'

Make sure that it leaves error log

nohup stdbuf -oL -eL mpiexec -l -n 6 julia --project=. scripts.jl > result.log 2>&1 &

Trace Error

nohup mpiexec -n 6 \
  -outfile-pattern logs/r%r.out \
  -errfile-pattern logs/r%r.err \
  bash -lc 'ulimit -c unlimited; exec stdbuf -o0 -e0 julia --project=. script.jl \
    -on_error_abort -error_output_stderr' \
  > run.log 2>&1 &

Install mpiexecjl

I realized that there is similar library, which is Julia-specific wrapper launcher (provided by MPI.jl).

julia -e 'using MPI; MPI.install_mpiexecjl(force=true)'

It puts mpiexecjl on ~/.julia/bin/

Trace error

nohup $HOME/.julia/bin/mpiexecjl -n 6 julia --project=. scripts/Examples/Unfitted/Wheel3D_CutFEM.jl \
    -on_error_abort -error_output_stderr -ksp_converged_reason \
    > run.log 2>&1 &