autogen.sh missing compopnents

[bbrout]

Using Ubuntu 20.04 on HP Z-600

used git clone etc to clone the repository and ran ./autogen.sh. Received the following errors:

mpiuser@BD-Main:~/mpich$ ./autogen.sh
Verifying the location of autogen.sh... done
Checking if autotools are in the same location... yes, all in /usr
Checking for autoconf version... >= 2.67
Checking for automake version... >= 1.15
Checking for libtool version... >= 2.4.4
Checking for bash... done
Checking for UNIX find... done
Checking for UNIX patch... done
Checking if xargs rm -rf works... yes
Checking for Python 3... ./autogen.sh: 1: python: not found
./autogen.sh: 247: test: =: argument expected
python3
===> ERROR: Submodule modules/hwloc is not checked out

I have installed python 3.6 and receive same error. I have hunted for hwloc and installed sudo apt-get install hwloc with success however this does not correct the problem. In trying to install from the tarball mpich-4.0.1.tar.gz I have success with config, make and make install, however running with different nodes hangs with:

mpiuser@BD-Main:~$ mpiexec -f machinefile -np 3 ./examples/cpi
/home/mpiuser/mpich-install/bin/hydra_pmi_proxy: error while loading shared libraries: libhwloc.so.15: cannot open shared object file: No such file or directory

I have hunted and cannot find a way to install the missing library to make the tarball install work.

It seems there is a problem with the hwloc libraries and if anyone has a solution to this, it would be appreciated. Thank you.

[hzhou]

Run git submodule update --init after fresh clone to get the submodules.

[hzhou]

@bbrout Let's back up and start fresh --

• Stick to the distribution until you are sure you need custom package and until you understand why
• Distribution installs to /usr. Default custom package installs to /usr/local. Unless you know what you have, you may clean out /usr/local
• Install mpich using your distro's package manager. Distro will take care of the dependency.
• You may check with which mpicc, which mpirun to verify what exactly what your are running.
• You may verify in /usr/bin, look for mpicc, mpirun, hydra_pmi_proxy
• You may run mpicc -show and mpirun -help to verify details, to make sure you are not running another implementation. You also can verify the library path and include path.
• MPI require all executables and library have the exact same path. So if on one node you are running /usr/local/bin/mpiexec and on another node you are running /usr/bin/mpiexec, that is not going to work.
• Your data and your application should be on a network file system mounted on the same path for each node.
• Then mpirun your_application should work.
• If not, go back and double-check.

[bbrout]

Good plan. I take it by mpirun you mean mpiexec? Also, ensuring all is ok on the nodes re path variable is a good idea too. I'm going to clean out /use/local/bin since mpich is all I have tried to install from source.

[bbrout]

Thanks

…

On Sat., Mar. 26, 2022, 10:28 p.m. Hui Zhou, ***@***.***> wrote: Run git submodule update --init after fresh clone to get the submodules. — Reply to this email directly, view it on GitHub <#5910 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHRCUH3DV77YQX7ZERW72V3VB7PXZANCNFSM5RYD63VA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[bbrout]

I did a fresh clone then entered: git submodule update --init and received following error: ***@***.***:~$ git submodule update --init fatal: not a git repository (or any parent up to mount point /home) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). Do I have to enter the http site in this command? Silly question. I'll try it. Yours, Bruce

…

On Sat, Mar 26, 2022 at 10:45 PM Bruce Rout ***@***.***> wrote: Thanks On Sat., Mar. 26, 2022, 10:28 p.m. Hui Zhou, ***@***.***> wrote: > Run git submodule update --init after fresh clone to get the submodules. > > — > Reply to this email directly, view it on GitHub > <#5910 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AHRCUH3DV77YQX7ZERW72V3VB7PXZANCNFSM5RYD63VA> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> >

-- GreenNABR.com

[hzhou]

Make sure you are in the mpich directory. For example:

git clone https://github.com/pmodels/mpich
**cd mpich**
git submodule update --init

[bbrout]

Sorry, did not work. This is the output: ***@***.***:~$ git submodule update --init https://github.com/pmodels/mpich.git fatal: not a git repository (or any parent up to mount point /home) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). ***@***.***:~$ Thanks for your help and time. Maybe I'm not entering the command correctly. Here is the entire output from new git clone command: ***@***.***:~$* git clone https://github.com/pmodels/mpich.git Cloning into 'mpich'... remote: Enumerating objects: 211393, done. remote: Counting objects: 100% (180/180), done. remote: Compressing objects: 100% (131/131), done. remote: Total 211393 (delta 72), reused 91 (delta 47), pack-reused 211213 Receiving objects: 100% (211393/211393), 75.14 MiB | 4.17 MiB/s, done. Resolving deltas: 100% (171431/171431), done. ***@***.***:~$* git submodule update --init fatal: not a git repository (or any parent up to mount point /home) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). ***@***.***:~$* git clone https://github.com/pmodels/mpich.git^C ***@***.***:~$* git submodule update --init https://github.com/pmodels/mpich.git fatal: not a git repository (or any parent up to mount point /home) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). ***@***.***:~$ * I made the prompt in bold for easier reading. Thank you for your time and help. Yours, Bruce

…

On Sun, Mar 27, 2022 at 12:21 PM Bruce Rout ***@***.***> wrote: I did a fresh clone then entered: git submodule update --init and received following error: ***@***.***:~$ git submodule update --init fatal: not a git repository (or any parent up to mount point /home) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). Do I have to enter the http site in this command? Silly question. I'll try it. Yours, Bruce On Sat, Mar 26, 2022 at 10:45 PM Bruce Rout ***@***.***> wrote: > Thanks > > On Sat., Mar. 26, 2022, 10:28 p.m. Hui Zhou, ***@***.***> > wrote: > >> Run git submodule update --init after fresh clone to get the submodules. >> >> — >> Reply to this email directly, view it on GitHub >> <#5910 (comment)>, >> or unsubscribe >> <https://github.com/notifications/unsubscribe-auth/AHRCUH3DV77YQX7ZERW72V3VB7PXZANCNFSM5RYD63VA> >> . >> You are receiving this because you authored the thread.Message ID: >> ***@***.***> >> > -- GreenNABR.com

-- GreenNABR.com

[hzhou]

After git clone, cd into the directory you just cloned before running the submodule command

[bbrout]

AHA! You first have to go into the cloned directory. cd mpich then run the git command to get modules. So sorry for all the emails. I got this to get the modules. OK, I'll try the autogen.sh again. Thank you for your patience. Yours, Bruce

…

On Sun, Mar 27, 2022 at 12:26 PM Bruce Rout ***@***.***> wrote: Sorry, did not work. This is the output: ***@***.***:~$ git submodule update --init https://github.com/pmodels/mpich.git fatal: not a git repository (or any parent up to mount point /home) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). ***@***.***:~$ Thanks for your help and time. Maybe I'm not entering the command correctly. Here is the entire output from new git clone command: ***@***.***:~$* git clone https://github.com/pmodels/mpich.git Cloning into 'mpich'... remote: Enumerating objects: 211393, done. remote: Counting objects: 100% (180/180), done. remote: Compressing objects: 100% (131/131), done. remote: Total 211393 (delta 72), reused 91 (delta 47), pack-reused 211213 Receiving objects: 100% (211393/211393), 75.14 MiB | 4.17 MiB/s, done. Resolving deltas: 100% (171431/171431), done. ***@***.***:~$* git submodule update --init fatal: not a git repository (or any parent up to mount point /home) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). ***@***.***:~$* git clone https://github.com/pmodels/mpich.git^C ***@***.***:~$* git submodule update --init https://github.com/pmodels/mpich.git fatal: not a git repository (or any parent up to mount point /home) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). ***@***.***:~$ * I made the prompt in bold for easier reading. Thank you for your time and help. Yours, Bruce On Sun, Mar 27, 2022 at 12:21 PM Bruce Rout ***@***.***> wrote: > I did a fresh clone then entered: > > git submodule update --init > > and received following error: > > ***@***.***:~$ git submodule update --init > fatal: not a git repository (or any parent up to mount point /home) > Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). > > Do I have to enter the http site in this command? Silly question. I'll > try it. > > Yours, > > Bruce > > On Sat, Mar 26, 2022 at 10:45 PM Bruce Rout ***@***.***> wrote: > >> Thanks >> >> On Sat., Mar. 26, 2022, 10:28 p.m. Hui Zhou, ***@***.***> >> wrote: >> >>> Run git submodule update --init after fresh clone to get the >>> submodules. >>> >>> — >>> Reply to this email directly, view it on GitHub >>> <#5910 (comment)>, >>> or unsubscribe >>> <https://github.com/notifications/unsubscribe-auth/AHRCUH3DV77YQX7ZERW72V3VB7PXZANCNFSM5RYD63VA> >>> . >>> You are receiving this because you authored the thread.Message ID: >>> ***@***.***> >>> >> > > -- > > > GreenNABR.com > -- GreenNABR.com

-- GreenNABR.com

[bbrout]

Sorry, I did get the repositories and submodules and ran autogen.sh successfully. However, I get the same error that I get when I compile from thetarball with including hwloc in the compile which you suggested in the other thread. This is the output: ***@***.***:~$ mpiexec -f machinefile -n 3 ./examples/cpi /home/mpiuser/mpich-install/bin/hydra_pmi_proxy: error while loading shared libraries: libhwloc.so.15: cannot open shared object file: No such file or directory ***@***.*** Sending Ctrl-C to processes as requested ***@***.*** Press Ctrl-C again to force abort This seems to be a problem for a number of people. I have a rar file with the missing library but I doin't know how to install it. I have attached the rar file for you to examine if this would help. Thank you for your time, Bruce

…

On Sun, Mar 27, 2022 at 12:33 PM Bruce Rout ***@***.***> wrote: AHA! You first have to go into the cloned directory. cd mpich then run the git command to get modules. So sorry for all the emails. I got this to get the modules. OK, I'll try the autogen.sh again. Thank you for your patience. Yours, Bruce On Sun, Mar 27, 2022 at 12:26 PM Bruce Rout ***@***.***> wrote: > Sorry, did not work. This is the output: > > ***@***.***:~$ git submodule update --init > https://github.com/pmodels/mpich.git > fatal: not a git repository (or any parent up to mount point /home) > Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). > ***@***.***:~$ > > Thanks for your help and time. Maybe I'm not entering the command > correctly. Here is the entire output from new git clone command: > > ***@***.***:~$* git clone https://github.com/pmodels/mpich.git > Cloning into 'mpich'... > remote: Enumerating objects: 211393, done. > remote: Counting objects: 100% (180/180), done. > remote: Compressing objects: 100% (131/131), done. > remote: Total 211393 (delta 72), reused 91 (delta 47), pack-reused 211213 > Receiving objects: 100% (211393/211393), 75.14 MiB | 4.17 MiB/s, done. > Resolving deltas: 100% (171431/171431), done. > ***@***.***:~$* git submodule update --init > fatal: not a git repository (or any parent up to mount point /home) > Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). > ***@***.***:~$* git clone https://github.com/pmodels/mpich.git^C > ***@***.***:~$* git submodule update --init > https://github.com/pmodels/mpich.git > fatal: not a git repository (or any parent up to mount point /home) > Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). > ***@***.***:~$ * > > I made the prompt in bold for easier reading. Thank you for your time and > help. > > Yours, > > Bruce > > > On Sun, Mar 27, 2022 at 12:21 PM Bruce Rout ***@***.***> wrote: > >> I did a fresh clone then entered: >> >> git submodule update --init >> >> and received following error: >> >> ***@***.***:~$ git submodule update --init >> fatal: not a git repository (or any parent up to mount point /home) >> Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not >> set). >> >> Do I have to enter the http site in this command? Silly question. I'll >> try it. >> >> Yours, >> >> Bruce >> >> On Sat, Mar 26, 2022 at 10:45 PM Bruce Rout ***@***.***> wrote: >> >>> Thanks >>> >>> On Sat., Mar. 26, 2022, 10:28 p.m. Hui Zhou, ***@***.***> >>> wrote: >>> >>>> Run git submodule update --init after fresh clone to get the >>>> submodules. >>>> >>>> — >>>> Reply to this email directly, view it on GitHub >>>> <#5910 (comment)>, >>>> or unsubscribe >>>> <https://github.com/notifications/unsubscribe-auth/AHRCUH3DV77YQX7ZERW72V3VB7PXZANCNFSM5RYD63VA> >>>> . >>>> You are receiving this because you authored the thread.Message ID: >>>> ***@***.***> >>>> >>> >> >> -- >> >> >> GreenNABR.com >> > > > -- > > > GreenNABR.com > -- GreenNABR.com

-- GreenNABR.com

[hzhou]

What is your configure line? Did you add the --with-hwloc=embedded option?

[bbrout]

I am doing that now with a downloaded tarball. I'll report back when done.

[bbrout]

Yes, I did. It compiled fine and installed in both the tarball version and the git clone version. However there were execution errors. ***@***.***:~$* which mpicc /home/mpiuser/mpich-install/bin/mpicc ***@***.***:~$* which mpiexec /home/mpiuser/mpich-install/bin/mpiexec ***@***.***:~$* mpiexec -n 3 ./examples/cpi Process 0 of 3 is on BD-Main Process 1 of 3 is on BD-Main Process 2 of 3 is on BD-Main pi is approximately 3.1415926544231318, Error is 0.0000000008333387 wall clock time = 0.002373 ***@***.***:~$* mpiexec -f machinefile -n 3 ./examples/cpi ./examples/cpi: error while loading shared libraries: libefa.so.1: cannot open shared object file: No such file or directory ./examples/cpi: error while loading shared libraries: libefa.so.1: cannot open shared object file: No such file or directory ./examples/cpi: error while loading shared libraries: libefa.so.1: cannot open shared object file: No such file or directory ***@***.***:~$* This is the machinefile contents: ***@***.***:~$* more machinefile node1 node2 node3 ***@***.***:~$* Thank you for your help.

…

On Sun, Mar 27, 2022 at 3:18 PM Hui Zhou ***@***.***> wrote: What is your configure line? Did you add the --with-hwloc=embedded option? — Reply to this email directly, view it on GitHub <#5910 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHRCUH3UV7BNGKNBOSZU2OTVCDGCRANCNFSM5RYD63VA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- GreenNABR.com

[hzhou]

What is your ldd ./examples/cpi?

You have libefa installed on the login node BD-Main. The same library is not available on node1 to node3.

[bbrout]

Sorry. lld of ./examples/cpi?? the directory ./examples/cpi is installed on the master node which is BD-Main. The nodes are named BD-1, BD-2 and BD-3. They are set at node1, node2 and nod3 in /etc/hosts on the master node, ie, BD-Main and that is the node running the mpiexec command. This is the /etc/hosts file 127.0.0.1 localhost 127.0.1.1 BD-Main # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters # The following sets up the local network for cluster 10.0.0.1 master 10.0.0.2 node1 10.0.0.3 node2 10.0.0.4 node3 I will try to hunt down and find libefa and install it on the slave nodes. I'll get back to you. Yours, Bruce

…

On Mon, Mar 28, 2022 at 7:14 PM Hui Zhou ***@***.***> wrote: What is your ldd ./examples/cpi? You have libefa installed on the login node BD-Main. The same library is not available on node1 to node3. — Reply to this email directly, view it on GitHub <#5910 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHRCUH6XHDVA462I36CV5CLVCJKPDANCNFSM5RYD63VA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- GreenNABR.com

[hzhou]

ldd - https://www.man7.org/linux/man-pages/man1/ldd.1.html

[bbrout]

I don't know what the lld of ./examples.cpi means. Nevertheless, I have found the library at this site: https://github.com/linux-rdma/rdma-core Should I download and install or just use this: $ apt-get install build-essential cmake gcc libudev-dev libnl-3-dev libnl-route-3-dev ninja-build pkg-config valgrind python3-dev cython3 python3-docutils pandoc Thanks again for your help. The nodes were installed with Ubuntu 20.0.4 with updates and upgrade. I don't know why the nodes do not have the same libraries as the master. Yours, Bruce

…

On Mon, Mar 28, 2022 at 8:32 PM Bruce Rout ***@***.***> wrote: Sorry. lld of ./examples/cpi?? the directory ./examples/cpi is installed on the master node which is BD-Main. The nodes are named BD-1, BD-2 and BD-3. They are set at node1, node2 and nod3 in /etc/hosts on the master node, ie, BD-Main and that is the node running the mpiexec command. This is the /etc/hosts file 127.0.0.1 localhost 127.0.1.1 BD-Main # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters # The following sets up the local network for cluster 10.0.0.1 master 10.0.0.2 node1 10.0.0.3 node2 10.0.0.4 node3 I will try to hunt down and find libefa and install it on the slave nodes. I'll get back to you. Yours, Bruce On Mon, Mar 28, 2022 at 7:14 PM Hui Zhou ***@***.***> wrote: > What is your ldd ./examples/cpi? > > You have libefa installed on the login node BD-Main. The same library is > not available on node1 to node3. > > — > Reply to this email directly, view it on GitHub > <#5910 (reply in thread)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AHRCUH6XHDVA462I36CV5CLVCJKPDANCNFSM5RYD63VA> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> > -- GreenNABR.com

-- GreenNABR.com

[hzhou]

Did you compile your own or used apt install on BD-Main?

[bbrout]

Tried running

sudo apt-get install libibverbs-dev

on all nodes but did not solve problem. Reboot everything?

[bbrout]

I compiled my own on BD-Main. Easy to install with sudo apt-get which is what I did on all the nodes.

[bbrout]

I can remove path in .bashrc and remove mpich-install then reinstall from apt-get. Should I do that?

[hzhou]

Try apt install mpich on BD-Main as well. You were running you compiled copy on nodes, that is why they don't work.

[bbrout]

slightly different error. had to remove path change in .bashrc and log out and back in to get correct mpiexec in /user. Howewver, slightly different error:

mpiuser@BD-Main:~$ mpiexec -f machinefile -n 3 ./examples/cpi
./examples/cpi: error while loading shared libraries: libmpi.so.0: cannot open shared object file: No such file or directory
./examples/cpi: error while loading shared libraries: libmpi.so.0: cannot open shared object file: No such file or directory
./examples/cpi: error while loading shared libraries: libmpi.so.0: cannot open shared object file: No such file or directory

[hzhou]

Recompile cpi with mpicc -o cpi examples/cpi.c. Then run mpiexec -f machinefile -n 3 ./cpi

[bbrout]

[bbrout]

mpiuser@BD-Main:~$ mpiexec -f machinefile -n 3 ./cpi
bash: /usr/local/bin/hydra_pmi_proxy: No such file or directory
bash: /usr/local/bin/hydra_pmi_proxy: No such file or directory
bash: /usr/local/bin/hydra_pmi_proxy: No such file or directory

[bbrout]

mpiuser@BD-Main:$ more machinefile
node1
node2
node3
mpiuser@BD-Main:$

[hzhou]

Remove /usr/local/bin/mpiexec

[bbrout]

mpiuser@BD-Main:$ sudo /usr/local/bin/mpiexec
[sudo] password for mpiuser:
[mpiexec@BD-Main] HYDU_create_proxy_list (utils/alloc/alloc.c:436): Missing executable.
[mpiexec@BD-Main] main (ui/mpich/mpiexec.c:263): unable to create proxy list
mpiuser@BD-Main:$

[bbrout]

mpiuser@BD-Main:~$ mpiexec -f machinefile -n 3 ./cpi
bash: /usr/local/bin/hydra_pmi_proxy: No such file or directory
bash: /usr/local/bin/hydra_pmi_proxy: No such file or directory
bash: /usr/local/bin/hydra_pmi_proxy: No such file or directory

[bbrout]

mpiuser@BD-Main:$ which mpiexec
/usr/bin/mpiexec
mpiuser@BD-Main:$ which mpicc
/usr/bin/mpicc
mpiuser@BD-Main:~$

[bbrout]

Unicode????

[bbrout]

nope. retyped it in not copied.

mpiuser@BD-Main:$ mpiexec -n 3 ./cpi
Process 0 of 3 is on BD-Main
Process 1 of 3 is on BD-Main
Process 2 of 3 is on BD-Main
pi is approximately 3.1415926544231318, Error is 0.0000000008333387
wall clock time = 0.000244
mpiuser@BD-Main:$ mpiexec -f machinefile -n 3 ./cpi
mpiexec: Error: unknown option "-f"
Type 'mpiexec --help' for usage.
mpiuser@BD-Main:~$

[bbrout]

mpiuser@BD-Main:~$ ./pingit.sh
PING master (10.0.0.1) 56(84) bytes of data.
64 bytes from master (10.0.0.1): icmp_seq=1 ttl=64 time=0.056 ms
64 bytes from master (10.0.0.1): icmp_seq=2 ttl=64 time=0.043 ms
64 bytes from master (10.0.0.1): icmp_seq=3 ttl=64 time=0.048 ms

--- master ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2051ms
rtt min/avg/max/mdev = 0.043/0.049/0.056/0.005 ms
PING node1 (10.0.0.2) 56(84) bytes of data.
64 bytes from node1 (10.0.0.2): icmp_seq=1 ttl=64 time=0.385 ms
64 bytes from node1 (10.0.0.2): icmp_seq=2 ttl=64 time=0.413 ms
64 bytes from node1 (10.0.0.2): icmp_seq=3 ttl=64 time=0.415 ms

--- node1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2045ms
rtt min/avg/max/mdev = 0.385/0.404/0.415/0.013 ms
PING node2 (10.0.0.3) 56(84) bytes of data.
64 bytes from node2 (10.0.0.3): icmp_seq=1 ttl=64 time=0.192 ms
64 bytes from node2 (10.0.0.3): icmp_seq=2 ttl=64 time=0.190 ms
64 bytes from node2 (10.0.0.3): icmp_seq=3 ttl=64 time=0.186 ms

--- node2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2045ms
rtt min/avg/max/mdev = 0.186/0.189/0.192/0.002 ms
PING node3 (10.0.0.4) 56(84) bytes of data.
64 bytes from node3 (10.0.0.4): icmp_seq=1 ttl=64 time=0.228 ms
64 bytes from node3 (10.0.0.4): icmp_seq=2 ttl=64 time=0.216 ms
64 bytes from node3 (10.0.0.4): icmp_seq=3 ttl=64 time=0.228 ms

--- node3 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2045ms
rtt min/avg/max/mdev = 0.216/0.224/0.228/0.005 ms
mpiuser@BD-Main:~$

[bbrout]

mpiuser@BD-Main:~$ ssh node1
Welcome to Ubuntu 20.04.4 LTS (GNU/Linux 5.13.0-39-generic x86_64)

• Documentation: https://help.ubuntu.com
• Management: https://landscape.canonical.com
• Support: https://ubuntu.com/advantage

0 updates can be applied immediately.

Failed to connect to https://changelogs.ubuntu.com/meta-release-lts. Check your Internet connection or proxy settings

Your Hardware Enablement Stack (HWE) is supported until April 2025.
Last login: Thu Mar 31 22:04:40 2022 from 10.0.0.1
mpiuser@node1:~$ ssh node2
Welcome to Ubuntu 20.04.4 LTS (GNU/Linux 5.13.0-39-generic x86_64)

• Documentation: https://help.ubuntu.com
• Management: https://landscape.canonical.com
• Support: https://ubuntu.com/advantage

0 updates can be applied immediately.

Failed to connect to https://changelogs.ubuntu.com/meta-release-lts. Check your Internet connection or proxy settings

Your Hardware Enablement Stack (HWE) is supported until April 2025.
Last login: Thu Mar 31 22:16:10 2022 from 10.0.0.1
mpiuser@node2:~$ ssh node3
Welcome to Ubuntu 20.04.4 LTS (GNU/Linux 5.13.0-39-generic x86_64)

• Documentation: https://help.ubuntu.com
• Management: https://landscape.canonical.com
• Support: https://ubuntu.com/advantage

0 updates can be applied immediately.

Failed to connect to https://changelogs.ubuntu.com/meta-release-lts. Check your Internet connection or proxy settings

Your Hardware Enablement Stack (HWE) is supported until April 2025.
Last login: Thu Mar 31 22:16:36 2022 from 10.0.0.3
mpiuser@node3:~$

[bbrout]

mpiuser@BD-Main:~$ /usr/bin/mpiexec -f machinefile -n 3 ./cpi
/usr/bin/mpiexec: Error: unknown option "-f"
Type '/usr/bin/mpiexec --help' for usage.

[bbrout]

looks like lots of stuff in /usr/local/bin that shouldn't be there.

[bbrout]

rebooting to see what happens

[bbrout]

same result

mpiuser@BD-Main:$ which mpicc
/usr/bin/mpicc
mpiuser@BD-Main:$ which mpiexec
/usr/bin/mpiexec
mpiuser@BD-Main:~$ mpiexec --help
mpiexec (OpenRTE) 4.0.3

Usage: mpiexec [OPTION]... [PROGRAM]...
Start the given program using Open RTE

-c|-np|--np Number of processes to run
-h|--help This help message
-n|--n Number of processes to run
-q|--quiet Suppress helpful messages
-v|--verbose Be verbose
-V|--version Print version and exit

For additional mpirun arguments, run 'mpirun --help '

The following categories exist: general (Defaults to this option), debug,
output, input, mapping, ranking, binding, devel (arguments useful to OMPI
Developers), compatibility (arguments supported for backwards compatibility),
launch (arguments to modify launch options), and dvm (Distributed Virtual
Machine arguments).

Report bugs to http://www.open-mpi.org/community/help/
mpiuser@BD-Main:~$

[bbrout]

interesting. The help file goes to an openmpi version,

[bbrout]

tried this

sudo apt-get purge --auto-remove openmpi-bin

[bbrout]

Now get the correct help file.

[bbrout]

also mpirun is still installed.

How do I remove and get rid of openmpi?? it is still on my computer even though i have tried to remove it.

[bbrout]

It's late. Thanks for your help. I'll try again tomorrow. All I can think of doing is to unmount my hard drive to save my local cata and nuke and pave my home machine. I don't want to do that. There may be a way to manually remove everything that is mpi related from /usr/local/bin. Good night and thanks again.

[bbrout]

this is the result of mpiexec --help

mpiuser@BD-Main:~$ mpiexec --help

Usage: ./mpiexec [global opts] [local opts for exec1] [exec1] [exec1 args] : [local opts for exec2] [exec2] [exec2 args] : ...

Global options (passed to all executables):

Global environment options:
-genv {name} {value} environment variable name and value
-genvlist {env1,env2,...} environment variable list to pass
-genvnone do not pass any environment variables
-genvall pass all environment variables not managed
by the launcher (default)

Other global options:
-f {name} file containing the host names
-hosts {host list} comma separated host list
-wdir {dirname} working directory to use
-configfile {name} config file containing MPMD launch options

Local options (passed to individual executables):

Local environment options:
-env {name} {value} environment variable name and value
-envlist {env1,env2,...} environment variable list to pass
-envnone do not pass any environment variables
-envall pass all environment variables (default)

Other local options:
-n/-np {value} number of processes
{exec_name} {args} executable name and arguments

Hydra specific options (treated as global):

Launch options:
-launcher launcher to use (ssh rsh fork slurm ll lsf sge manual persist)
-launcher-exec executable to use to launch processes
-enable-x/-disable-x enable or disable X forwarding

Resource management kernel options:
-rmk resource management kernel to use (user slurm ll lsf sge pbs cobalt)

Processor topology options:
-topolib processor topology library (hwloc)
-bind-to process binding
-map-by process mapping
-membind memory binding policy

Checkpoint/Restart options:
-ckpoint-interval checkpoint interval
-ckpoint-prefix checkpoint file prefix
-ckpoint-num checkpoint number to restart
-ckpointlib checkpointing library (none)

Demux engine options:
-demux demux engine (poll select)

Other Hydra options:
-verbose verbose mode
-info build information
-print-all-exitcodes print exit codes of all processes
-iface network interface to use
-ppn processes per node
-profile turn on internal profiling
-prepend-rank prepend rank to output
-prepend-pattern prepend pattern to output
-outfile-pattern direct stdout to file
-errfile-pattern direct stderr to file
-nameserver name server information (host:port format)
-disable-auto-cleanup don't cleanup processes on error
-disable-hostname-propagation let MPICH auto-detect the hostname
-order-nodes order nodes as ascending/descending cores
-localhost local hostname for the launching node
-usize universe size (SYSTEM, INFINITE, )

Please see the intructions provided at
http://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager
for further details

[hzhou]

I just want to make sure it is hydra -- that it is. :)

[bbrout]

Also messed up more by reading the help file and "using the hydra process manager.

I had this going:

mpiuser@BD-Main:$ mpiexec -n 10 ./cpi
Process 2 of 10 is on BD-Main
Process 8 of 10 is on BD-Main
Process 9 of 10 is on BD-Main
Process 0 of 10 is on BD-Main
Process 1 of 10 is on BD-Main
Process 3 of 10 is on BD-Main
Process 4 of 10 is on BD-Main
Process 5 of 10 is on BD-Main
Process 7 of 10 is on BD-Main
Process 6 of 10 is on BD-Main
pi is approximately 3.1415926544231256, Error is 0.0000000008333325
wall clock time = 0.026859
mpiuser@BD-Main:$ mpiexec -f ./nodes -n 10 ./cpi
bash: /usr/bin/hydra_pmi_proxy: No such file or directory

in the help file saw this:
Please see the intructions provided at
http://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager for further details

so I read that. Then tried this:

mpiuser@BD-Main:$ export HYDRA_HOST_FILE=/home/mpiuser/hosts
mpiuser@BD-Main:$ mv nodes hosts
mpiuser@BD-Main:~$ more hosts
node1
node2
node3

I think that was a boo boo.

mpiuser@BD-Main:$ mpiexec ./cpi
bash: /usr/bin/hydra_pmi_proxy: No such file or directory
mpiuser@BD-Main:$ mpiexec -n 3 ./cpi
bash: /usr/bin/hydra_pmi_proxy: No such file or directory

mpiuser@BD-Main:~$ unset HYDRA_HOST_FILE=/home/mpiuser/hosts
bash: unset: `HYDRA_HOST_FILE=/home/mpiuser/hosts': not a valid identifier

I purged and auto-removed mpich
I still had a copy of mpiexec on /usr/bin
So I ran it and got some interesting output on the hosts file. here is the copy of the commands and output

mpiuser@BD-Main:$ sudo apt-get purge --auto-remove mpich
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
libmpich-dev* libmpich12* mpich*
0 upgraded, 0 newly installed, 3 to remove and 6 not upgraded.
After this operation, 18.0 MB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 282839 files and directories currently installed.)
Removing libmpich-dev:amd64 (3.3.2-2build1) ...
Removing mpich (3.3.2-2build1) ...
update-alternatives: using /usr/bin/mpirun.lam to provide /usr/bin/mpirun (mpirun) in auto mode
Removing libmpich12:amd64 (3.3.2-2build1) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.7) ...
(Reading database ... 282744 files and directories currently installed.)
Purging configuration files for libmpich-dev:amd64 (3.3.2-2build1) ...
mpiuser@BD-Main:$ which mpiexec
/usr/bin/mpiexec
mpiuser@BD-Main:$ mpiexec -n 3 ./cpi
Lamnodes Failed!
Check if you had booted lam before calling mpiexec else use -machinefile to pass host file to mpiexec
mpiuser@BD-Main:$ mpiexec -machinefile hosts -n 3 ./cpi

The boot SSI rsh module found that your local host is not in the
hostfile "hosts".

The local host name must be in the list of hosts in the hostfile.
In other words, you must boot LAM from a node that will be part of the
universe.

    - If you simply forgot to put the local host in the boot
      schema file, add it and re-run The boot SSI rsh module
    - If you are trying to boot LAM from a node that will not be
      part of the universe, you must login to on of the nodes that
      will be part of the universe (i.e., one of the nodes in the
      hostfiles), and re-run The boot SSI rsh module

Although the local host name is usually the first in the list to avoid
I/O ambiguities, it can actually appear anywhere in the list.

---

It seems that there is no lamd running on the host BD-Main.

This indicates that the LAM/MPI runtime environment is not operating.
The LAM/MPI runtime environment is necessary for the "lamhalt" command.

Please run the "lamboot" command the start the LAM/MPI runtime
environment. See the LAM/MPI documentation for how to invoke
"lamboot" across multiple machines.

mpiuser@BD-Main:$ nano hosts
mpiuser@BD-Main:$ mpiexec -machinefile hosts -n 3 ./cpi
ERROR: LAM/MPI unexpectedly received the following on stderr:
bash: hboot: command not found

LAM failed to execute a LAM binary on the remote node "node1".
Since LAM was already able to determine your remote shell as "hboot",
it is probable that this is not an authentication problem.

*** PLEASE READ THIS ENTIRE MESSAGE, FOLLOW ITS SUGGESTIONS, AND
*** CONSULT THE "BOOTING LAM" SECTION OF THE LAM/MPI FAQ
*** (http://www.lam-mpi.org/faq/) BEFORE POSTING TO THE LAM/MPI USER'S
*** MAILING LIST.

LAM tried to use the remote agent command "/usr/bin/rsh"
to invoke the following command:

    /usr/bin/rsh node1 -n hboot -t -c lam-conf.lamd -s -I '"-H 10.0.0.1 -P 32985 -n 1 -o 0"'

This can indicate several things. You should check the following:

    - The LAM binaries are in your $PATH
    - You can run the LAM binaries
    - The $PATH variable is set properly before your 
      .cshrc/.profile exits

Try to invoke the command listed above manually at a Unix prompt.

You will need to configure your local setup such that you will not
be prompted for a password to invoke this command on the remote node.
No output should be printed from the remote node before the output of
the command is displayed.

When you can get this command to execute successfully by hand, LAM
will probably be able to function properly.

ERROR: LAM/MPI unexpectedly received the following on stderr:
bash: tkill: command not found

LAM failed to execute a LAM binary on the remote node "node1".
Since LAM was already able to determine your remote shell as "tkill",
it is probable that this is not an authentication problem.

*** PLEASE READ THIS ENTIRE MESSAGE, FOLLOW ITS SUGGESTIONS, AND
*** CONSULT THE "BOOTING LAM" SECTION OF THE LAM/MPI FAQ
*** (http://www.lam-mpi.org/faq/) BEFORE POSTING TO THE LAM/MPI USER'S
*** MAILING LIST.

LAM tried to use the remote agent command "/usr/bin/rsh"
to invoke the following command:

    /usr/bin/rsh node1 -n tkill

This can indicate several things. You should check the following:

    - The LAM binaries are in your $PATH
    - You can run the LAM binaries
    - The $PATH variable is set properly before your 
      .cshrc/.profile exits

Try to invoke the command listed above manually at a Unix prompt.

You will need to configure your local setup such that you will not
be prompted for a password to invoke this command on the remote node.
No output should be printed from the remote node before the output of
the command is displayed.

When you can get this command to execute successfully by hand, LAM
will probably be able to function properly.

---

It seems that there is no lamd running on the host BD-Main.

This indicates that the LAM/MPI runtime environment is not operating.
The LAM/MPI runtime environment is necessary for the "lamhalt" command.

Please run the "lamboot" command the start the LAM/MPI runtime
environment. See the LAM/MPI documentation for how to invoke
"lamboot" across multiple machines.

mpiuser@BD-Main:$ which mpiexec
/usr/bin/mpiexec
mpiuser@BD-Main:$ sudo rm /usr/bin/mpiexec
mpiuser@BD-Main:$ which mpicc
mpiuser@BD-Main:$ which mpiexec
mpiuser@BD-Main:$ mpiexec
bash: /usr/bin/mpiexec: No such file or directory
mpiuser@BD-Main:$ sudo apt-get install mpich
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
libmpich-dev libmpich12

etcetera

then
mpiuser@BD-Main:$ mpiexec -n 4 ./cpi
bash: /usr/bin/hydra_pmi_proxy: No such file or directory
mpiuser@BD-Main:$ mpiexec -n 4 ./cpi
Process 0 of 4 is on BD-Main
pi is approximately 3.1415926544231239, Error is 0.0000000008333307
wall clock time = 0.000111
Process 1 of 4 is on BD-Main
Process 2 of 4 is on BD-Main
Process 3 of 4 is on BD-Main

So far so good. Lots of work to get back to where I started.

mpiuser@BD-Main:$ mpiexec -n 4 ./cpi
Process 0 of 4 is on BD-Main
pi is approximately 3.1415926544231239, Error is 0.0000000008333307
wall clock time = 0.000111
Process 1 of 4 is on BD-Main
Process 2 of 4 is on BD-Main
Process 3 of 4 is on BD-Main
mpiuser@BD-Main:$ more hosts
master
node1
node2
node3
mpiuser@BD-Main:~$ mpiexec -f hosts -n 4 ./cpi
bash: /usr/bin/hydra_pmi_proxy: No such file or directory

[bbrout]

Sorry for all the posts and replies. I have to work now and will try later. The environment variable and how it is handled may be corrupted somehow. There was a post somewhere on the internet about how to completely eradicate all instances of mpi, openmpi and mpich. I have to hunt that down again. It was an operation by hand and I can repeat that in /user/local/bin. There were some hidden files in there that are probably being called upon somehow. Thanks for your help and patience.

[bbrout]

mpiuser@BD-Main:~$ ssh node2
Welcome to Ubuntu 20.04.4 LTS (GNU/Linux 5.13.0-39-generic x86_64)

• Documentation: https://help.ubuntu.com
• Management: https://landscape.canonical.com
• Support: https://ubuntu.com/advantage

0 updates can be applied immediately.

Failed to connect to https://changelogs.ubuntu.com/meta-release-lts. Check your Internet connection or proxy settings

Your Hardware Enablement Stack (HWE) is supported until April 2025.
Last login: Fri Apr 1 15:25:12 2022 from 10.0.0.1
mpiuser@node2:~$ ping -c 3 node1
PING node1 (10.0.0.2) 56(84) bytes of data.
64 bytes from node1 (10.0.0.2): icmp_seq=1 ttl=64 time=0.370 ms
64 bytes from node1 (10.0.0.2): icmp_seq=2 ttl=64 time=0.332 ms
64 bytes from node1 (10.0.0.2): icmp_seq=3 ttl=64 time=0.344 ms

--- node1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2026ms
rtt min/avg/max/mdev = 0.332/0.348/0.370/0.015 ms
mpiuser@node2:~$

[hzhou]

The error says the process on one node failed to receive message from another node. I currently don't have any idea what may be wrong.

[bbrout]

I'd have to check the software. I can see the error pop up in the middle of the run and have success after the posting of the error, Anyway, hellow works fine and I have other software that I can use to try to trrace it. But for now, the nodes are mostly talking to each other. Thanks for all the help. When in doubt, nuke and pave. :-)

[bbrout]

Fixed it. It is in the /etc/hosts file on each node. The node was defined twice once as localhost and then as the name of the computer. On the master node, BD-Main is defined after localhost but is also defined as an alias on the nodes /etc/hosts file. Commenting out the original node address in /etc/hosts on each node did the trick. If BD-Main is not defined in the local static network, mpich cannot "find" it.

This is now /etc/hosts on master node which is BD-Main

mpiuser@BD-Main:~$ cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 BD-Main

The following lines are desirable for IPv6 capable hosts

::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

The following sets up the local network for cluster

10.0.0.1 master BD-Main
10.0.0.2 node1
10.0.0.3 node2
10.0.0.4 node3

and this is /etc/hosts on node1, which is the same as other nodes.

mpiuser@BD-Main:~$ ssh node1
Welcome to Ubuntu 20.04.4 LTS (GNU/Linux 5.13.0-39-generic x86_64)

• Documentation: https://help.ubuntu.com
• Management: https://landscape.canonical.com
• Support: https://ubuntu.com/advantage

0 updates can be applied immediately.

Failed to connect to https://changelogs.ubuntu.com/meta-release-lts. Check your Internet connection or proxy settings

Your Hardware Enablement Stack (HWE) is supported until April 2025.
Last login: Fri Apr 1 17:40:17 2022 from 10.0.0.1
mpiuser@node1:~$ cat /etc/hosts
127.0.0.1 localhost
#127.0.1.1 node1

The following lines are desirable for IPv6 capable hosts

::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

for cluster

10.0.0.1 master BD-Main
10.0.0.2 node1
10.0.0.3 node2
10.0.0.4 node3

And this is the result of cpi run on master

mpiuser@BD-Main:$ mpiexec -hosts node1,node2,node3 -n 12 ./cpi
Process 8 of 12 is on node3
Process 11 of 12 is on node3
Process 2 of 12 is on node3
Process 5 of 12 is on node3
Process 0 of 12 is on node1
Process 3 of 12 is on node1
Process 6 of 12 is on node1
Process 9 of 12 is on node1
Process 1 of 12 is on node2
Process 4 of 12 is on node2
Process 10 of 12 is on node2
Process 7 of 12 is on node2
pi is approximately 3.1415926544231256, Error is 0.0000000008333325
wall clock time = 0.053315
mpiuser@BD-Main:$

and with nodes file as this

mpiuser@BD-Main:$ cat nodes
node1:4
node2:2
node3:3
mpiuser@BD-Main:$

we get
mpiuser@BD-Main:$ mpiexec -f nodes -n 12 ./cpi
Process 8 of 12 is on node3
Process 6 of 12 is on node3
Process 7 of 12 is on node3
Process 4 of 12 is on node2
Process 5 of 12 is on node2
Process 0 of 12 is on node1
Process 1 of 12 is on node1
Process 3 of 12 is on node1
Process 9 of 12 is on node1
Process 10 of 12 is on node1
Process 11 of 12 is on node1
Process 2 of 12 is on node1
pi is approximately 3.1415926544231256, Error is 0.0000000008333325
wall clock time = 0.033806
mpiuser@BD-Main:$

Faster with the nodes file declaring the number of processors to use. So, looks good. You are amazing sir. Thanik you.

[hzhou]

Congratulations on getting it to work!