ISO/IEC JTC1/SC22/WG5 N1989

             Result of the WG5 letter ballot on N1967

                         John Reid

N1986 asked this question

Please answer the following question "Is N1983 ready for forwarding to 
SC22 as the DTS?" in one of these ways. 

1) Yes.
2) Yes, but I recommend the following changes. 
3) No, for the following reasons.
4) Abstain.

The numbers of answers in each category were:
0 for 1) Yes. 
3 for 2) Yes, but I recommend the following changes (Burnus, Lionel, Nagle)
8 for 3) No, for the following reasons (Bader, Chen, Cohen, Long, Maclaren, 
         Muxworthy, Reid, Snyder)
1 for 4) Abstain (Corbett)

The ballot has failed. J3 is requested to prepare a revised version that
takes the comments into account. 

Here are the responses in detail

Reinhold Bader

3) No, for the following reasons:

* the editor himself in 13-296 has pointed out a number of technical 
  issues that needs fixing. 
* there are still some items from the first comment round N1971 that
  have not yet been discussed; in particular the issue of how the
  atomic functions behave has not yet been satisfactorily resolved.
* while removing the "big barrier" from CHANGE TEAM was in my 
  opinion the right thing to do, not all possible interactions 
  with other features have been dealt with. 

Here some more detailed comments on N1983:

(A) Teams:
~~~~~~~~~~

(A.1) Missing support for computational domains

This is item (B.2) from my comments in N1971 which has not 
received any discussion in the last meeting. 
On the J3 list, there was some mention of using a THIS_NODE() 
intrinsic to achieve the desired effect. However, I believe that
this is not quite sufficient for what is in general required, and
therefore have submitted 13-300 with a suggested specification
for a DOMAIN argument to the FORM SUBTEAM statement.

(A.2) Issues with team coindexing

N1983 has introduced the concept of specifying a <team-variable>
in the image selector in order to be able to revert to ancestor 
team coindexing. There are some open questions here:

(A.2.1) Is it intended that cosubscripts can be specified that
address an image outside the presently executing team? If not
(which is the answer I favor, in order to keep the scheme as
simple as possible, while retaining the possibility of future
extension), a restriction that states this must be added.

(A.2.2) A restriction is missing that the coarray on which the
new addressing scheme is used must have been established by
the referenced team or an ancestor of it.

(A.2.3) Because coarrays can change their corank and cobounds 
through procedure invocations during execution of the program, 
there must be a rule (or a set of rules) that states which 
corank and cobounds the team coindexing scheme refers to.

Consider the following example:

real, allocatable :: a[:]
type(team_type) :: t, u

allocate(a[4:*])

form subteam(..., u) ! original team
id = ...
form subteam(id, t)

change team(t)
  call foo(a, u)
end team

call bar(a, t, u)

with foo and bar as follows:

subroutine foo(a, u)
  real :: a[6:*]
  type(team_type) :: u
  real :: x
  x = a[u :: 4] + ...
  ...
end subroutine
subroutine bar(a, t, u)
  real :: a[6:*]
  type(team_type) :: t, u
  real :: x
  change team (t) 
    x = a[u :: 4] + ...
  end team
  ...
end subroutine

The question is whether the coindexed access in bar is to image
1 of the original team (because that is the way the ALLOCATE statement
established the coarray), or whether it is invalid because the last
known lower cobound when executing with the specified team is relevant, 
and has the value 6.
None of the possible answers appears entirely satisfying. Perhaps it 
is indeed preferable to oblige the programmer to specify a remapping 
scheme, or to simply use lists of image numbers, or (as suggested by 
me in N1971) to prohibit coindexing of ancestor team inherited 
coarrays altogether. 

(A.2.4) DISTANCE optional argument

In my contribution to N1971, I asked "Is it allowed to use the 
DISTANCE argument in THIS_IMAGE if a coarray argument is also 
specified?". This question was never answered; if the answer was
intended to be "yes", a similar issue as for (A.2.3) above must
be addressed, and IMAGE_INDEX should be analogously extended.

(A.3) Form subteam properties

Given the synchronization properties of CHANGE TEAM, it appears 
to be unnecessary for FORM SUBTEAM to perform synchronization 
of all images, or for it to be an image control statement.

Furthermore, allowing a team variable to have the POINTER attribute
was considered useful. I agree with this, but would suggest to prohibit
a POINTER team variable in an execution of FORM SUBTEAM (it would
however be allowed in execution of a CHANGE TEAM).


(B) Events:
~~~~~~~~~~~

(B.1) Split phase barrier

This is item (C.2) from my comments in N1971 which has not 
received any discussion in the last meeting.
I have prepared (but not yet uploaded) a J3 paper with a 
suggested specification; it also includes an example that 
gives an indication how this feature might be fruitfully used. 
However, the concept presupposes that events are not reduced 
to only binary semaphores as suggested by Nick MacLaren.


(C) Atomic functions:
~~~~~~~~~~~~~~~~~~~~~

I agree with Malcolm and Nick that specification of a memory
model for the atomic functions is desirable. Depending on the
outcome, it should then also be possible to decide whether or
not it is necessary to make further changes to the event 
semantics.

_______________________________________________________________________

Tobias Burnus


N1983 permits at some places the use of a team variable, which refers to the 
ancestor team. In particular in image selectors: 
"[team-variable :: cosubscript-list]".

The PDTS requires that the team-variable was assigned a value by a 
FORM SUBTEAM statement. However, that makes it rather complicated to access 
the initial team (which encompasses all images). [Or the current subteam, in 
case that the caller has already done some partition before calling a 
procedure (in a library), which creates further subteams.] Hence, I think it 
would be useful to have an intrinsic procedure, which sets the value of the 
current team to a team-variable.
assuming
For instance, "subroutine this_team(team-variable)". One could add an optional 
"distance" argument but as the caller has to be aware (for practical use) of 
the image indexes, it is probably not needed.

One usage case would be a partition in a climate model into air, land and sea 
(cf. Annex A.1) where one exchanges every few iterations information for the 
boundaries between different subteams. This could be done in the "air" subteam 
via "boundary[parent_team :: sea_neighbor] = values; 
event post(data_is_there[parent_team :: sea_neighbor])".
Without the new subroutine, one had either leave the subteam, exchange the 
boundaries, and re-enter. (Implies two global synchronizations for end change 
team/change team for each exchange.) Or one had to form an artificial subteam, 
which encompasses all of the images of the (current, initial) team. The effect 
of the latter would be similar to the proposed "this_team()" except that it is 
uglier and requires three pointless synchronization (form subteam, change team, 
end change team) - but at least not during the iteration.


_______________________________________________________________________

Daniel Chen

My vote is "NO" because

1. It is not clear how the team variable that identifies all image is 
   initialized.
2. It is not clear how the state of the failed image is handled 
   (i.e. Should it undo everything up to the failing point?).
3. There are also issues raised by others that need to be addressed.

_______________________________________________________________________

Malcolm Cohen

No.

Reason 1: With the introduction of cross-team access and synchronisation, the 
TEAM concept has become too complicated and unwieldy.  Those who need cross-team 
access and synchronisation issues are, IMO, better served by using simple arrays 
of image numbers - lots of discussion at the meeting was about manually aligning 
the teams and coarrays and image numbers, and all that just goes away if you use 
image number lists (and it is far from clear that the new facilities invented 
out of whole cloth at the meeting satisfy the concerns expressed - my impression 
is certainly that they do not).

Reason 2: There needs to be an explicit memory model for atomics so that users 
are clearly told if and when causality may be violated and vendors are clearly 
told what the requirements are on their implementation.  Handwaving it all away 
as "processor dependent" is a disservice to everyone - some of the users will 
think that some rational rules apply, and some of the vendors will provide 
facilities that don't obey those apparently-rational rules (reasonable people 
might differ on whether particular rules are "reasonable"!).

_______________________________________________________________________

Steve Lionel

2) Yes, but I recommend the following changes. 

On page 11, R507 "form-team-spec" should be "form-subteam-spec" (probably 
left over from an older draft)

7.4.1 (ATOMIC_ADD) - change "ATOM becomes defined with the value of 
ATOM + VALUE." To "ATOM becomes defined with the value of 
ATOM + INT(VALUE,ATOMIC_INT_KIND)."   Or is the intent that the normal 
mixed-mode arithmetic and assignment rules apply? This seems rather ugly 
to me, and "becomes defined with the value" is not intrinsic assignment.

_______________________________________________________________________

Bill Long

No. The issues detailed in J3/13-296 need to be addressed. 

_______________________________________________________________________

Nick Maclaren

    3) No, for the following reasons.

I have included a few points from other people's comments on N1967, but
much of this is from my comments on that.  I have not repeated the
commentary, but most of it remains relevant, and N1971 should be
referred to for that.


Teams
-----

5.1, p9:14-15.  This states "Within the body of a CHANGE TEAM construct
the current team is the subteam specified by the CHANGE TEAM
statement.", which implies lexical scope only.  This should say "Within
the body of a CHANGE TEAM construct and in procedures called from it the
current team ...."

5.2 and 5.3, p9:16-37.  It is still not clear whether TEAM_TYPE has
value or location semantics.  C502 is not enough, because of the
implicit copying implied by passing assumed-shape arrays to
explicit-shape or assumed-size ones, and R502 says 'variable'.  This
must be clarified and is linked to some of the next points [A].

5.3, p10:22-24.  Executing a common CHANGE TEAM statement the same
number of times is not enough, because the variable could be a dummy
argument associated with a different team on different images.  There
needs to be an explicit restriction (probably in lines 14-16) that all
variables must have been created by the same execution of the same FORM
SUBTEAM statement.  See [A].

5.5 p11:4-16 (especially 4, 11-12 and 14).  This is seriously
inconsistent.  FORM SUBTEAM cannot be an image control statement if
it occurs within a segment, ordered or not (Fortran 2008 8.5.2) and,
if it does, it has explicit synchronisation.  I think that the
sentence on line 4 "It is an image control statement." is erroneous
and should be removed.

5.5 p11:12-13.  I have no idea why the restriction on the subscripts
is needed, and cannot see that it does anything useful.  A single
dummy argument can be associated with different variables on different
images and, if it has value semantics, there is no reason to restrict
it in this way.  Indeed, different images can even pass the same
variable in different elements of the same dummy array on different
images.  See [A].

5.5 p11:14-16.  FORM SUBTEAM does not say what happens if an error
occurs, unlike the collectives.

5.6 p11:17+.  There is nothing said about when resources may be
released, and no mechanism for the user to free them.  This is not
reasonable, and there needs to be some defined way for a programmer to
avoid memory leaks when using FORM SUBTEAM heavily.

7.4.13 p22:16-18.  I can find no guarantee that the subteam id. is
assigned in a defined order, and hope that is not the case.  The example
comments should say "Code for half of the images in the current team"
and "Code for the other half of the images in the current team".


Events
------

6.2 p13:15-18.  To avoid the array copying problem, this also needs
to exclude explicit-shape and assumed-size arrays containing event
variables.

6.3 p13:29 and 6.4 p14:2.  The sentences referring to unsuccessful waits
should be removed, as there are no defined conditions that can cause
them to fail and still given them access to the event variable.

6.4 p14:3-7.  At least two responses to N1967 pointed out that the
definition of sequencing makes no sense, on the grounds that ordering
between images is defined only by image control statements, and the
order of these image control statements depends on the ordering between
images!  This is not a simple matter, and simple wording changes will
not resolve it.  As I commented there, the easiest solution to this is
to change events from general semaphores into binary ones, and define an
error return from EVENT POST if the event is already posted.  If this is
not done, some mathematically consistent definition of the ordering must
be provided.  As this appears to be beyond the state of the art in
computer science, it may be hard to achieve.

7.4.11 p21  It needs to say that EVENT_QUERY may be used in segments
that are unordered with respect to EVENT POST on the same variable.


Collectives
-----------

7.3 p15:24+.  Since we have not agreed whether and how collectives
should be consistent with atomics, a Note along the following lines
should be provided:

    Collective subroutines are ordered only by the usual execution
    sequence (2.3.5) and segment ordering (8.5.2) rules and behave
    as if they were implemented using private coarrays to transfer
    the data.  In particular, using them together with atomic
    subroutines in unordered segments is likely to produce effects
    that appear to be inconsistent.

7.3 p15:32-39.  This should state that all INTENT(INOUT) and INTENT(OUT)
arguments become undefined if an error occurs.

7.3 p15:38.  "If an image had failed" should be changed to "If a
processor detects that an image has failed".

7.4.6 p18:2.  "shall be a coarray" is almost certainly erroneous and
should be removed.  If it is a coarray, then all images can access it
directly.  And the usual requirement is to broadcast a local array,
anyway.

Passim.  All collectives except CO_BROADCAST specify that, if SOURCE is
an array, all images must have the same type parameters and shape, but
there is no such wording for scalars.  This is a problem for character
lengths, but may also be one for polymorphic procedures.  Either all
images need to specify that SOURCE is the same, or it needs to state
what happens if that is not the case; the former would be a lot easier
for implementors and to specify.  Specific issues that must be fixed are
described below [B].

Passim.  The specification is messy and restrictive, and should be
changed.  For example, it is not possible to reduce INTENT(IN)
variables, which is a fairly common requirement.  While copying them is
possible, it involved extra data accesses and is potentially
inefficient.  A far better solution would be either two procedures or a
generic interface, of the following forms (CO_REDUCE is used as an
example):

    CO_REDUCE (OBJECT, OPERATOR [, STAT, ERRMSG])
    ! OBJECT would be INTENT(INOUT)

    CO_REDUCE (RESULT, SOURCE, OPERATOR [, RESULT_IMAGE, STAT, ERRMSG])
    ! RESULT would be INTENT(INOUT) and SOURCE would be INTENT(IN)

7.4.6 p18:2-4.  CO_BROADCAST needs to include the same restrictions on
the consistency of type parameters and shape as for the other
collectives.  See [B].

7.4.6 p18:2-4, 7.4.7 p18:16-20 and 7.4.8 p1-5.  If polymorphic
procedures allow the same dummy argument to be different types on
different images, CO_BROADCAST, CO_MIN and CO_MAX need to forbid mixing
numeric and character.  See [B].

7.4.9 p19:37-39.  The last two sentences should be replaced by "OPERATOR
shall implement a mathematically associative and commutative
operation.", as I remember we agreed.

7.4.9 p19:35-39.  This must specify that OPERATOR must implement the
same mathematical function on all images, but I have no idea how to say
that.  Indeed, I am not completely sure that there is no way of
providing different procedures for OPERATOR, because it involves areas
of Fortran I am not familiar with.

7.4.9 p20:8-11.  This must state that the procedure may be called on any
image an indeterminate number of times.


Atomic Intrinsics
-----------------

7.2 p15:13-15.  This specification will cause massive confusion, and it
was clear from WG5 in Delft that there was no agreement on even the
minimal semantics specified by Fortran.  In particular, several people
were assuming levels of consistency that are not always available in
existing hardware, and would need extra work in the compiler to provide.
At the very least, there needs to be a Note saying clearly and
explicitly that currently their behaviour is deliberately left entirely
processor-dependent, and WG5 intends to provide a proper semantic
specification in due course.

Passim.  These must specify that their argument is a coarray, in
the same way as ATOMIC_DEFINE and ATOMIC_REF.

_______________________________________________________________________


David Muxworthy

I acknowledge the work done by the HPC subgroup but my vote is 'no' on
a matter of principle.  The TR/TS system in Fortran was intended to
try out new facilities in parallel with the main language, analogous
to beta-testing, and to incorporate them into the standard when they
had proved stable across platforms.  The proposals for the current TS
do not remotely meet these criteria.  Although the material has been
around for several years, the requirements were being revised only
last month and the specification is still being developed.  Edits to
the specification from no fewer than 18 papers (15 of them 'r' papers)
were approved in June.

Van has already raised one technical point and over thirty editorial
glitches.  Clearly the document is not ready to forward to SC22.  I
very much agree with his comment:

   It looks like much of this stuff has had only a few moments'
   reflection before being proposed to be cast in concrete.  Sure,
   Cray might have done some of it already, and their customers might
   use it successfully, but that doesn't mean it's the best solution,
   or even a good one.

   I'd rather see more discussion, introspection, consensus, and
   experience, and less controversy, before we rush to publish a
   definitive solution as a Technical Specification that is almost
   promised to end up as part of the standard.

Well said, Van!

______________________________________________________________________

Dan Nagle

I vote YES with Comments

Comments on 13-293

My vote is YES with Comments,
comments follow

[9:3
Do teams synchronize with other teams,
or do images within a team synchronize?
It might be a stretched reading, but the sentence could be clarified.

[9:9]
Change "...from values stored in its team variable"
to "...from values obtained via its team variable"
Don't constrain the implementation.  What if the application-visible
team variable is a handle?

[11:27-12:1-4]
Are the STAT_* errors (>0) or information (<0)?
"Failed" sounds like an error but its intended usage
sounds like information to be detected, a response made,
and execution continued.

[13:9]
Change "includes" to "references"
Don't constrain the implementation.

[13:22+]
Clarify that the order of receipt of event postings
is processor dependent.
Add a new paragraph
"The order of arrival of the effects of event post statements
is processor dependent."

[16:3-5]
Change the paragraph to give an explanatory message
when an informative value is passed to the stat= specifier.

Question about atomic_and, atomic_or, atomic_xor:
Why not allow type logical?  And and or are well defined,
xor is neqv, and there is an atomic logical kind.

[21:10-12]
Replace status= being zero or nonzero to distinguish
ok =0, errors >0, and information <0 values.

Somewhere in the discussion, clarify that the order of arrival
of the effects of event posts and event waits is processor dependent.

Question:
Are there directions for the Editor to place the TS Annex A words
into the 007 annex C?  The Editor can probably wrangle something,
but it's better to give some hints.

______________________________________________________________________

John Reid

3) No, for the following reasons.

1. Reasons for no vote

1.1 Continued execution in the presence of failed images

It is not clear how execution is intended to continue in the presence of 
failed images. 

For most calculations, the failure of one image leads to the failure of the 
whole calculation. To recover from this, the program probably needs to revert 
to a previous "check point" and continue the execution from there using
images that have not failed. One possibility is to not to use all the
images for the calculation but keep a few "spares". Execution is within a
CHANGE TEAM construct, with spares in a separate team and idle. When an image 
fails, the CHANGE TEAM construct is left, a new team is formed by
substituting a spare for the failed image, the check point data are recovered, 
and the CHANGE TEAM construct is re-entered. This avoids any need in the main 
code for remapping of data - it only has to detect failed images and
exit the construct if there are any. 

Some calculations are "massively parallel". Most of the work is done 
completely independently on separate images. Perhaps one image acts as 
"master" handing out tasks and collecting results. As long as the master does 
not fail, the calculation can continue happily with failed images. The master 
sends the work that it gave to a failed image to the next image that is free. 

I will assume that we want to cater for both situations. Even in the first 
case, the parent team needs to execute in a team that has failed images while 
it forms the new team and recovers the check-point data. 

The collective procedures are not massively parallel. They should surely fail 
if any of the images of the team have failed. However, the last paragraph of 
page 15 says "If an image has failed, but no other error condition occurred, 
the argument is assigned the value STAT_FAILED_IMAGE.". If this behaviour is 
retained, the effect of failed images on the result needs to be described.

The effect of SYNC ALL and SYNC TEAM in the presence of failed images should 
be to synchronize the images that have not failed. This should be stated. 
I am not sure about SYNC IMAGES. 

The FORM SUBTEAM statement should work in the presence of failed images. I am
inclined to think that if a subteam has a nonfailed image, all its images 
should be nonfailed.  

The CHANGE TEAM statement should work in the presence of failed images.

The ALLOCATE and DEALLOCATE statements should work in the presence of 
failed images. 

For locks and events, at most one other image is involved and its failure
must be regarded as an error. 

1.2 Cosubscripts of arrays declared in ancestor teams

New syntax (R624) was added during the Delft meeting to allow a coarray
to be addressed by the cosubscipts of a team other than the current team. 
It is not restricted to be the current team or an ancestor, but I think
that was the intention. Because there is no means of specifying the 
mapping between cosubscripts when teams change, the new syntax should be  
restricted to refer to the team in which the coarray was declared.
Alternatively, a mechanism for specifying the mapping should be added.
I suggest that it should be as for the association of a dummy coarray with
the corresponding actual coarray. A possibility is the statement
   new cosubscripts (<cosubscript-decl-list>) 
where
   <cosubscript-decl> is <coarray-name>    []
                         <lbracket> <explicit-coshape-spec> <rbracket>     


2. Other comments

2.1 FORM SUBTEAM
What happens if NEW_INDEX is absent? Is the mapping from parent image index 
to child image index processor dependent? Or is THIS_IMAGE(DISTANCE=1) 
monotic increasing? 

2.2 REDUCE
For CO_MAX, CO_MIN, and CO_SUM, there is a corresponding transformational
function so it is easy to write code for the common case where the max, min, 
or sum of all the elements of the arrays on all the images is wanted. We
need to add REDUCE to play the same role for CO_REDUCE. 

2.3. Error condition for a collective
The last para. of 15, says that if an error condition occurs and STAT is 
present, the effect is as if SYNC MEMORY were executed. This seems wrong 
because RESULT has intent out so one expects it to become undefined. 
Do we expect SOURCE to be used by the implementation for workspace
when RESULT is absent? If so, an error condition should cause SOURCE to 
become undefined. 

2.4 Examples
More examples are needed, particularly of continued execution in the presence 
of failed images. 

3. Edits
[15:22] After "the beginning" add "to the end".

_______________________________________________________________________


Van Snyder

3) No, for the following reasons.

I have identified at least 31 places where change appears to be
necessary.  Most of these are niggling details that would reasonably
fall under the rubric "Yes, but I recommend the following changes."

The problem that causes me to vote "No" is that CHANGE TEAM doesn't
accomplish the necessary rearrangement of coindexing for coarrays having
corank > 1.  Adding "<team-variable> ::" to <image-selector> doesn't
solve the problem, and is a fundamentally bad idea.

Two files are attached.  One is a list of the mostly-niggling changes,
and the other is a proposal, stimulated by the ideas immanent in the
ASSOCIATE construct, to deal with the rearrangement problem for coarrays
having corank > 1.

.......................................................................

File 1:

[9:18] Insert "variable" after "scalar".

[10:13-14] Values of variables are not "formed," they are defined.

[10:27-32]
Image selectors don't solve the problem that motivated the discussion
that led to them.  In any case, they seem to be a fundamentally bad
idea, somewhat akin to the idea that a subroutine can access a
sequence-associated dummy argument array using subscripts relative to
the actual argument.  A proposal to solve the original problem is
attached.

[11:1] "form-team-spec" -> "form-subteam-spec"

[11:4-6]
One could calculate a team variable describing the initial team by
executing a FORM SUBTEAM statement in which <subteam-id> has the same
value on all images.

If this would be a common thing to do, it would be useful to have a team
variable describing the initial team, maybe in ISO_FORTRAN_ENV.  

[11:6] Restricting <subteam-id> to be positive seems pointless.  Can it
at least be non-negative?  One might reasonably want to use
mod(this_image(),2), instead of 1+mod(this_image(),2) or
2-mod(this_image(),2).

[11:9] "a different value" -> "different values"

[11:13] "on" -> "in" or "these statements" -> "images of the current
team"

[11:17]
A prohibition against changing the value of a team variable during
execution of a CHANGE TEAM construct appears to be necessary. 
Otherwise, how does synchronization at the END TEAM statement work
reliably?  Does the processor keep a cached copy of the team variable,
maybe on something like a "team stack," just in case it's changed?  The
restriction at [11:17] doesn't cover this problem.  Maybe it's enough to
insert "or the current team".

[11:21] "FORM SUBTEAM" -> "a FORM SUBTEAM statement"

[9:17] and [13:7] use inconsistent style.

[13:15] "in a" -> "in an"

[13:24] Is the EVENT POST statement the only way to post an event?  It
would be neutral to say "Execution of an EVENT POST statement posts an
event."

[13:31] Is the EVENT WAIT statement the only way to wait for an event? 
It would be neutral to say "Execution of an EVENT WAIT statement waits
until an event is posted."

[14:1-2] What is "success" for a wait?  Is it waiting, or continuing
without needing to wait, or what?

[15:3-4] lines are wider than the margins.  Look for "overfull hbox"
after running LaTeX.

[15:11-13] First two sentences are redundant.  Delete the first one.

[16:15] "scalar of the same type as ATOM" -> "scalar and of type
integer" or "scalar and of type integer with the same kind as ATOM"
(whichever is really intended).  Also at [16:29], [17:21], [17:34].

[16:18] Delete "have its"

[16:35] It's hard to justify "swap" in the description of a subroutine
in which only one argument has INTENT(INOUT).  Swap what with what?

[17:4] We don't have a concept of equality for objects of logical type.
Appears to require a messy sentence using "equivalent," or maybe two
sentences.

[18:5] Is the image index an index in the current team or the initial
team?  Also [18:25], [19:10], [19:44], [20:31]

[19:29] If SOURCE is scalar, is it required to have the same type
parameter values on all images?

[19:32] "parameters" -> "parameter values"

[21:7] Why default integer?  Why not a kind for which HUGE(0_thatkind)
>= HUGE(0)?

......................................................................

File 2

                                                               WG5/N19xx
To:      WG5
Subject: Accessing team coarrays in subteams
From:    Van Snyder
Date:    27 June 2013

1. Problem description
----------------------

It is desired to access a portion of a coarray created in a team, either
by static declaration or allocation, within a subteam created by a
CHANGE TEAM construct, using cosubscripts such that the range of
accessible coelements, taken in coarray coelement order, is bounded by
the size of the subteam.  Further it is desirable that the portion of
the coarray that is accessible within a subteam depends upon the
membership of the subteam.  It is undesirable for the subteam to be
required to be aware of the mapping from coelements of the entire
coarray to coelements germane to the subteam.

The correspondence between image indices and coelements applies only to
the indexing of coelements, taken in coarray coelement order.  If one
needs to revise the leading codimension extent(s), CHANGE TEAM offers no
help.

2. Example
----------

As a simple example, assume a corank-2 coarray:

  real :: A[10,*]
 
Assume the number of images has been verified to be 100, and four teams
are formed:

  integer :: ME, S, S1, S2
  type(team_type) :: T(2,2)
  me = this_image()
  s2 = merge(1,2,me>49)
  s1 = merge(1,2,mod(me-1,10)>4)
  s = s1 + 2*s2
  form subteam ( s, t(s1,s2) )

Assume it is desired that subteam 1 accesses the A[1:5,1:5]
coelements of A, subteam 2 accesses the A[6:10,1:5] coelements of A,
subteam 3 accesses the A[1:5,6:10] coelelements of A, and subteam 4
accesses the A[6:10,6:10] coelements of A.

Assume subteams are entered thus:

  change team ( t(s) )
    ! At this point, valid image indices are 1..25, in each
    ! subteam.  Which coelements of A[] are accessible in subteam S?
    ! Presumably, coelements 1..25, taken in coarray coelement order
    ! are accessible in subteam 1, 26:50 in subteam 2, 51:75 in subteam
    ! 3, and 76:100 in subteam 4.  But subteam 1 wants 1:5, 11:15,
    ! 21:25, 31:35, and 41:45, while subteam 2 wants 6:10, 16:20, 26:30,
    ! 36:40, and 46:50, subteam 3 wants 51:55, 61:65, 71:75, 81:85 and
    ! 91:95, and subteam 4 wants 56:60, 66:70, 76:80, 86:90, and 96:100.
    ..
  end team

3. Proposal
-----------

A scheme to re-map cosubscripting is needed.

There is presently no concept of a cosection, and no concept of
cosequence coassocation.  Therefore, for example, assuming the
subroutine executes a CHANGE TEAM construct, the following is not even
valid syntax:

  call do_teams ( s, t(s), a[1+5*(s1-1):5*s1,1+5*(s2-1):5*s2] )

This would be nontrivial to implement, and very difficult or impossible
if vector cosubscripted cosections are allowed, at least if it is
necessary to avoid copying.  Vector cosubscripted cosections might be
indispensible in adaptive grid applications.

An alternative that would solve the same problem, but also is not (yet)
valid syntax, might be something like

  change team ( t(s)), a1 => a[1+5*(s1-1):5*s1,1+5*(s2-1):5*s2] )
    ! Herein, A1 is a coarray that is coassociated with the specified
    ! coelements of A on each subteam.
    ...
  end team

Vector cosubscripted cosections would not be an insurmountable problem
within the CHANGE TEAM construct, because the processor clearly could
see the vector.  If the subteam coarray is an actual argument, however,
the vector needs to go along with it.  Even if the team coarray does not
have a vector cosubscript, a descriptor needs to accompany a coarray
argument.  This might be included in the team variable, which would then
need to accompany coarray arguments.  Even without this proposal, it
might be necessary for team variables to accompany coarray arguments.

4. Draft proposed edits to N1983
--------------------------------

[9:36] Replace "<sync-stat-list>" with "<change-list>"

[9:36+] Insert syntax rules and constraints

R502a <change> <<is>> <sync-stat>
               <<or>> <coassociation>

R502b <coassociation> <<is>> <subteam-coarray> => <team-cosection>

R502c <subteam-coarray> <<is>> <name>

R502d <team-cosection> <<is>> <coarray-name> <lbracket>
                              [ <coselector> ] ... <final-coselector>
                              <rbracket>

R502e <coselector> <<is>> [ <lower-cobound> : ] <upper-cobound>
                          [ : <costride> ]
                   <<or>> <vector-subscript>

R502f <final-coselector> <<is>> [ <lower-cobound> : ] *
                                [ : <costride> ]

R502g <lower-cobound> <<is>> <scalar-int-expr>

R502h <upper-cobound> <<is>> <scalar-int-expr>

R502h <costride> <<is>> <scalar-int-expr>

R502g (R503d) The <coarray-name> shall be the name of a coarray.

C502a (R503d) The number of <coselector>s shall be equal to the corank
      of <coarray-name>.

C502b (R503d) There shall be at least one <coselector> that includes a
      colon, is an asterisk, or is a <vector-subscript>.

C502c (R503e) If <coselector> is <vector-subscript>, <subteam-coarray>
      shall not appear as an actual argument.

[10:16+] Insert a paragraph:

"If <coassociation> appears, the <subteam-coarray> is the name of a
coarray that is coassociated with the specified <team-cosection>, in
coarray coelement order.  The corank of the <subteam-coarray> is the
number of <coselector>s that include a colon, is an asterisk, or are
<vector-subscript>s.  Within the CHANGE TEAM construct, the number of
coelements of <subteam-coarray> is equal to the number of images in the
subteam."

Definitions of terms "coassociated", "coelement", and "coelement order"
are needed as well.