Define function composition for :string values

[eemeli]

Closes #797

The intent here is to say that if a :string annotated value is used anywhere, you always get a string out of it, but that we keep track of its locale and directionality.

[aphillips]

I think this could be improved. The first part of the sentence is really hard to read. Also, I think that we should use normative language and we should permit some implementation-specific things to occur.

Suggested change

	When an _operand_ or an _option_ value uses a _variable_ with a _declaration_
	using an _expression_ with a `:string` _annotation_,
	its resolved value contains only the string value of the _operand_ of the annotated _expression_,
	together with its resolved locale and directionality,
	and no _option_ values.
	The _resolved value_ of an _expression_ with a `:string` _annotation_ consists of
	the string value of the _operand_ of the annotated _expression_.
	It MUST also include the `locale`, `direction`, and `id`, if they exist, from the formatting context.
	It MAY include implementation-defined attributes or option values.

[macchiati]

Suggested change

	When an _operand_ or an _option_ value uses a _variable_ with a _declaration_
	using an _expression_ with a `:string` _annotation_,
	its resolved value contains only the string value of the _operand_ of the annotated _expression_,
	together with its resolved locale and directionality,
	and no _option_ values.
	The _resolved value_ of an _expression_ with a `:string` _annotation_ consists of:
	- the string value of the _operand_ of the annotated _expression_
	- if they exist, the `locale`, `direction`, and `id` from the formatting context
	- optionally, implementation-defined attributes or option values

Reason: it is strange to say that "X consists of Y", which implies that X consists of exactly that, no more, no less. So reworded to list all the things it could consist of, marking the ones that are optional.

[eemeli]

I am in general fine with this sort of an approach to defining resolved value, but it'll mean that we need to do so more deeply than this small change I'm trying to propose, because we need to account for what happens here:

.input {$x :string}
.match {$x}
foo {{{$x} is foo}}
* {{{$x} isn't foo, but as a number it's {$x :number}}}

Where do we place the matching and formatting behaviour for $x if its resolved value only consists of data fields? Even though those are relatively simple for :string, our general solution will need to work for all sorts of functions, and provide a way for defining how their formatting, matching, and composition works when all you have is a resolved value.

As I mentioned most recently in #515 (comment), such a resolved value is almost certainly most easily defined as an object with different methods for its different use cases.

However, defining the spec in such a manner has received a significant amount of pushback, and so it's currently done by only defining the observable parts, i.e. in this case the resolved value of a variable referring to a :string annotated value when that variable is used as an operand or an option value.

If we do want to define resolved values more explicitly, then we need to go all in on that approach, and start that definition from the base class definition of something like a ResolvedValue with abstract methods, which would allow for a ResolvedString child class to be defined as the resolved value of a :string expression, together with the behaviour of at least its getValue, getOptions, formatToString, and matchKeys methods.

Or we can stick to our current approach of not explaining the internals, and directly describing the observable parts of that behaviour.

[eemeli]

To get an idea for what such a spec could look like, please take a look at #198, my first attempt at doing so. Our language has changed in the interim, and that exact "Formattable" API doesn't match our current needs (e.g. it's missing getOptions, and we dropped formatted parts), but something like that could be a way for us to define a proper resolved value.

[aphillips]

You're not wrong, but what I would say is... the "resolved value" is the value of the operand (the input), not the value of the output (the formatted string or Formattable object in the case of a formatter or the sorted options list in the case of a selector).

Our existing functions do not alter the operand, so their "resolved value" is whatever the function decides the operand value is. Here's an example that might demonstrate a function with a 'resolved value' side-effect:

.input {$x :uppercase @locale=fr}
.match {$x :string}
foo {{Never matches}}
FOO {{{$x} matches if $x is any permutation of foo/Foo/FOO/etc}}
* {{}}

[stasm]

I think the wording here is akward because we're trying (rightly so!) to avoid defining any concrete interfaces. However, I note that we do have some requirements about what these interfaces should look like. How about we document these requirements instead? Something like:

Suggested change

	When an _operand_ or an _option_ value uses a _variable_ with a _declaration_
	using an _expression_ with a `:string` _annotation_,
	its resolved value contains only the string value of the _operand_ of the annotated _expression_,
	together with its resolved locale and directionality,
	and no _option_ values.
	The return type of the `:string` annotation is implementation-defined.
	It MUST meet the following requirements:
	- Expose the string value of the operand.
	- Expose the locale of the operand.
	- Expose the directionality of the operand.

[eemeli]

The proposed text does not sufficiently restrict an implementation when considering the corner cases. For example, in

.local $x = {foo :string bar=42}
{{x is {$x :custom}}}

it should be clear that :custom does not have access to the bar=42 option value.

Rather than iterating on the exact language within this PR, I'd be much happier if we could find agreement in the proposed behaviour of :string values. The main point of this exercise is to define those, rather than lock down absolutely the language describing them.

[aphillips]

What about implementation defined namespaced options?

.input {wildebeest :string addison:normalize=nfc}

[eemeli]

They'd be lost, unless you're using a different :string than the one we define here. If you need options, then you could/should be using :addison:string.

[aphillips]

That doesn't make sense to me?

Suppose I'm @mihnita and suppose I'm writing the ICU4J implementation. Now suppose I want to support skeletons for dates. I want that to look like: {$d :datetime icu:skeleton=yMMMdjm}, not {$d icu:datetime icu:skeleton=yMMMdjm}. Make sense so far?

I don't want to namespace the core function, because I want other options to work unchanged. Obviously, my namespaced option won't work in implementations where it isn't recognized. But dropping it on the floor doesn't make sense.

[eemeli]

That's a fair point; I filed #829 as a consequence of thinking through some of the implications here.

Technically, we do pass through all :datetime options so your example would be fine. For :string, I'd rather hoped that we could avoid having to deal with any options in composition.

That means that something like your addison:normalize could still work, as long as it changed the resolved string value of the expression, rather than getting passed along separately to the next consumer of the value.

Is that enough?

[aphillips]

I think :string (or whatever :x we define) only controls the options it owns. It's okay to say that :string passes none of its built-in options.

I think we should say that implementation-defined or user-installed installed options MAY be passed in the resolved value or swallowed by the implementation, while unrecognized (but otherwise valid) options MUST be passed in the resolved value.

This would permit an implementation to "eat" an option so that it has no downstream effect. You'd want this on destructive options:

.local $a = {McGowan :string x:transform=uppercase}
.local $b = {$a :append string=|_foo|}
.match {$b}
MCGOWAN_foo {{Should match because x:transform is not transitive}}
* {{...}}

[eemeli]

What your example shows is possible with the currently proposed text, including an alternate version where you'd replace the second declaration with

.local $b = {$a :string x:append=|_foo|}

The specific question that I think we should ask here is, "Is there any reasonable custom option for :string that would not modify the input string and get swallowed up?"

If we can think of at least one, then we should change the text here.

[aphillips]

I can't think of one for :string that would behave in that way.

I think we should adopt a formula for dealing with options. I would reword this as:

Suggested change

	and no _option_ values.
	The _resolved value_ of a `:string` _annotation_ contains
	the resolved string value of the _operand_,
	the resolved locale of the _operand_,
	and the resolved direction of the _operand_.
	None of the _options_ defined by `:string` are part of the _resolved value_.

We might want a note to go with this to signal our intention:

Note

Custom or implementation-defined options that modify
the operand result in a modification of the resolved string value
of the operand and aren't included in the resolved value.

[macchiati]

I also thought that was very clear, that you could both extend other functions with a namespaced option, and add new namespaced functions.

[catamorphism]

LGTM (I agree with Eemeli about dropping namespaced options).