Add new PMIX_GROUP_FINAL_MEMBERSHIP_ORDER attribute#3716
Add new PMIX_GROUP_FINAL_MEMBERSHIP_ORDER attribute#3716rhc54 merged 1 commit intoopenpmix:masterfrom
Conversation
The PMIx_Group_construct API itself is not order sensitive in the provided proc array - i.e., we ignore that order when executing the operation. This allows each caller to provide the proc array in arbitrary order - which is advantageous for most libraries. However, some users may need us to return a specific order of the procs in the final membership. Allow them to specify the order in a new attribute, This can be specified by individual processes or by namespace (with a wildcard rank). If multiple participants provide this attribute, then the values must match - i.e., the desired final membership order must be identical. Signed-off-by: Ralph Castain <rhc@pmix.org>
|
@sonjahapp I understand that MPI may have a requirement to control the ordering of procs in the final membership. I added this to help provide that control - see the accompanying PRRTE change openpmix/prrte#2303. Occurred to me that if you specify an ordering using an nspace/wildcard combination, then you might want any specific ranks to appear in rank order - true? If so, that is easily provided over in PRRTE. Let me know what you think. |
|
Hi Ralph!
I would say yes. For any other rank order, wildcards/nspace cannot be used but rather the explicit order must be provided. A few (perhaps stupid) questions: Does the process order in the membership array have any influence on how OpenPMIx or PRRTE treat the PMIx Group? Are there any benefits or downsides for PMIx or the host server implementation that could originate from a specific proc order in the membership array? Or is this feature "just" for convenience on PMIx client side - so that the clients would not have to sort the processes by themselves? I would assume that for MPI this feature is useful for optimization, but not really strictly required, or is it? As long as the group membership array is identical in all procs of the group, MPI should be able to deal with it and sort the procs in whatever order is required. |
None whatsoever.
Not that I am aware of - we don't care about the order
The question arose in terms of efficiency. According to @abouteiller, MPI requires that the order of the procs in the eventual communicator be set based on how connect/accept is called. If we want to use The PMIx API treats the provided membership array as order insensitive. This was done because the OMPI folks hit a problem where the participating procs were calling the API with different orders - basically, they would have had to inject a collective to agree upon the order prior to calling the API, but the procs don't have each others endpoint info until after the PMIx operation completes. Bit of a chicken/egg problem. So we made the membership input order insensitive and then insure that the final membership array returned to each participant is identical. Note that other programming models don't care about order, and so this is strictly an MPI problem - and may be even more strictly an OMPI problem. Not entirely sure. With this new feature, at least the OMPI connect/accept can simply call the group construct API and pass the new attribute to indicate that members from nspace A should come first, followed by members from nspace B. Other programming models might get more atomistic and specify individual ordering. However, I do admit that one could instead sort the final membership array to get whatever you wanted. Ultimately a question for you folks: is this useful? 🤷♂️ Easy enough to remove if not. |
|
Oh, just FWIW: note that the host (and not the PMIx library) assumes the responsibility for performing the sort as it is the only one that knows the final membership. |
|
MPI connect/accept is order agnostic. Moreover, all MPI dynamic processes functionality is order agnostic., as none of it create an intra communicator (where all processes are part of the same group and the order is important), but an intracomm (where each side of the connect/accept is on their own group with their own original ranking). The only MPI function that need to know the ranking is |
|
Interesting - @abouteiller could you please reiterate your concerns about ordering of the final membership? Perhaps I'm not stating them correctly. |
The PMIx_Group_construct API itself is not order sensitive in the provided proc array - i.e., we ignore that order when executing the operation. This allows each caller to provide the proc array in arbitrary order - which is advantageous for most libraries.
However, some users may need us to return a specific order of the procs in the final membership. Allow them to specify the order in a new attribute, This can be specified by individual processes or by namespace (with a wildcard rank). If multiple participants provide this attribute, then the values must match - i.e., the desired final membership order must be identical.