standardize opaque handle types

[jeffhammond]

Problem

The type of the opaque handles -- Comm, Datatype, File, Group, Info, Message, Op, Request, Window -- must be standardized to have an ABI.

MPICH currently uses int. Open-MPI uses pointers. There are advantages to both.

Unfortunately, because we are required to support equality comparison in C, which lacks operator overloading, we cannot do something elegant like struct or union (which would allow compilers to do type-checking).

Proposal

Option 1

Standardizing intptr_t is compatible with both existing designs, although it is an ABI change for MPICH (on 64-bit platforms, at least).

intptr_t would allow Open-MPI to continue to use pointers internally since, by definition, intptr_t can be cast to/from valid pointers. Since it is also an integer, MPICH can continue to use the tricks it has for encoding special values, and just ignore the additional zeroes.

Option 2

Handles that are incomplete struct pointers (ISPs) allow for type checking. Encoding compile-time constants for predefined handles relies on implementation-defined behavior (IB) but should be viable.

The translation from ISP to MPICH handles not as simple as with Option 1. If one implements a compatibility layer, it could maintain a vector of integers and the pointers could pointer to these, assuming casting an ISP to int* is always legal. Encoding the int handles directly in the ISP as a value relies on IB.

Changes to the Text

Impact on Implementations

Implementations will have to change, particularly macros that manipulate handles.

The biggest impact is that this breaks the MPICH ABI, but that is a necessary evil in this process.

Impact on Users

Breaking existing MPICH ABI but have standard ABI.

References and Pull Requests

https://github.com/jeffhammond/blog/blob/main/MPI_Needs_ABI_Part_4.md (although this proposes the invalid union/struct design)

[jeffhammond]

Unfortunately, the above proposal appears to involve UB in C if we use magic values for built-ins, because casting (u)intptr_t to and from pointers is only defined for valid pointers (see below).

As best I can tell, this means that we must cannot have compile-time constants or magic integer values for built-in handles if they also behave like C pointers.

Implementations can still choose between a fully integer handle design like MPICH and a pointer design like Open-MPI, but there is no "best of both" situation.

7.20.1.4 Integer types capable of holding object pointers

The following type designates a signed integer type with the property that any valid pointer to void can be converted to this type, then converted back to pointer to void, and the result will compare equal to the original pointer:

intptr_t

The following type designates an unsigned integer type with the property that any valid pointer to void can be converted to this type, then converted back to pointer to void, and the result will compare equal to the original pointer:

uintptr_t

These types are optional.

[jedbrown]

It's sad we can't get compile-time type checking without making MPICH-using code hold invalid pointers. If we had overcommit and 64-bit addresses, then we could just allocate/map 4 GB of memory and never access it (basically free on 64-bit systems) to make all MPICH-style handles valid pointers. Are there architectures of interest for which (a) holding an invalid pointer is actually problematic and (b) the mapping strategy to make the pointers valid is problematic? A desperate pitch, but I have to ask.

[jeffhammond]

To get compile time constants, we'd need to mmap FIXED, which isn't particularly portable.

The proposed design is problematic on 32b platforms and likely has issues with an OS like Blue Gene CNK.

I haven't figured out the major downside of link-time constants...

[devreal]

I wonder: why can't we encode compile-time constant values in pointers and check for them (or the known range of them) inside the MPI library? They will never be dereferenced but replaced with the actual pointer upon entry into the library.

[jeffhammond]

I wonder: why can't we encode compile-time constant values in pointers and check for them (or the known range of them) inside the MPI library? They will never be dereferenced but replaced with the actual pointer upon entry into the library.

Because it's UB in C and C++, as best I can tell.

In practice, there are unlikely to be any values that are universal anyways. People who try such things have problems, e.g.

• https://stackoverflow.com/questions/54967429/what-is-a-valid-pointer-in-gcc-linux-x86-64-c
• https://stackoverflow.com/questions/47712502/why-cant-i-mmapmap-fixed-the-highest-virtual-page-in-a-32-bit-linux-process-o

[jeffhammond]

Here's what I'm reading in C11 draft N1570 §6.6 right now:

More latitude is permitted for constant expressions in initializers. Such a constant expression shall be, or evaluate to, one of the following:
— an arithmetic constant expression,
— a null pointer constant,
— an address constant, or
— an address constant for a complete object type plus or minus an integer constant expression.

...

An address constant is a null pointer, a pointer to an lvalue designating an object of static storage duration, or a pointer to a function designator; it shall be created explicitly using the unary & operator or an integer constant cast to pointer type, or implicitly by the use of an expression of array or function type. The array-subscript [] and member-access . and -> operators, the address & and indirection * unary operators, and pointer casts may be used in the creation of an address constant, but the value of an object shall not be accessed by use of these operators.

If we want an address constant, it has to be a pointer to something. It cannot be any random integer.

[jedbrown]

You can have a normal mapping and define pointer_handle = base + integer_handle. Where base can be a private pointer to struct { char c; } for internal arithmetic in char-sized units.

I suspect this mapping approach is DoA due to 32-bit address spaces and environments without overcommit, though maybe creating the out of bounds pointer value is not a real problem there.

There might be other ways to encode with modest memory requirements.

[besnardjb]

I'm not sure I understand the problem. Does it means MAP_FAILED is wrong in mmap ?

On success, mmap() returns a pointer to the mapped area.   On  error,  the  value
       MAP_FAILED  (that  is, (void *) -1) is returned, and errno is set to indicate the
       cause of the error.

[jedbrown]

Yeah, same issue with MPI_IN_PLACE, at least as defined by Open MPI and MPICH. Note that mmap is POSIX, not ISO C, and this is another interaction:

       Existing  implementations of mmap() return the value -1 when unsuccessful. Since the
       casting of this value to type void * cannot be guaranteed by the ISO C  standard  to
       be  distinct from a successful value, this volume of POSIX.1‐2017 defines the symbol
       MAP_FAILED, which a conforming implementation does not return as  the  result  of  a
       successful call.

It's also common for malloc(0) to return a unique non-NULL pointer that will SEGV if you dereference it.

I recall Bill commenting (in the context of PETSc multi-dim array pointer tricks) that the ISO C restriction only matters for systems with segmented memory. The standard doesn't have advice for implementers regarding MPI_IN_PLACE. An implementation could stay ISO C compliant by by making it a link-time constant pointing to a constant dummy object, but it seems none do.

[besnardjb]

Ok today I learn something new thanks! 🤯

So, as you pointed out, and if I’m not mistaking: it means it is already impossible to implement both MPI_BOTTOM and MPI_IN_PLACE.

If MPI already ignores, and if POSIX also deemed it ‘ok’… I’m not sure there are many handles needing multiple non standard values (mostly null handles which can be NULL) because otherwise it could be pointer to something at runtime as given by constructors.

Non exhaustively out of my head:

Maybe the MPI_STATUS_* but they are all roughly synonyms to NULL (if you ignore error checking).

And the comm : NULL, WORLD and SELF. Sessions already solved the construction of these. But yes, can we leave with 0, 1, 2 (or whatever) in these pointer values.

The group: GROUP_NULL and GROUP_EMPTY (no constructor for empty).

Other enums are passed as integers otherwise. (Plans to typedef ?)

[besnardjb]

And… all the datatypes which could be an array unfolded by base indexing with a macro as stated earlier in the issue. Sorry missed the largest part !

[jeffhammond]

MPI_BOTTOM and MPI_IN_PLACE are not specified to be compile-time constants. They are sentinels and explicitly prohibited as initializers.

Offsets relative to bottom are probably the most sketchy aspect of MPI from a C language compliance perspective, although we added AINT arithmetic macros in MPI-3 to address this.

[jeffhammond]

I recommend everybody read 2.5.4 slowly and carefully a few times, since it addresses this stuff. Learning from implementations is not a great way to understand this topic.