Changes to mid-agreement payments to minimize missed payments

[jalas167]

Intention:

Initial specification of mid agreement payment model may cause a situation when premature termination of a requestor agent (Crl+C) may result in a lack of payment for providers (they would not have enough time to send debit notes or invoices).
Note: currently observed situation on mainnet where providers complain about not receiving payments might not be caused by mid agreement payments as at this moment only JS SDK is using this model as a default one.

Changes and defaults for implementation:

1. New payment scheme will replace the current mid-agreement spec.

2. Each debit note will be payable (within payment-timout set).
   2.1 “golem.com.scheme.payu.payment-timeout-sec?” is mandatory.
   2.2 default limit for payment-timeout-sec = 24h, implemented on provider.

3. Debit notes will be sent according to debit-note.interval. Payment-timeout is no longer related to frequency of sending debit notes.
   3.1"golem.com.scheme.payu.debit-note.interval-sec?” - is mandatory,
   3.2 It can be set within the range of 120 s (default) to 900 s. (default =120),
   4 "Golem.com.payment.debit-notes.accept-timeout?" is mandatory
   4.1. Should be 2 x time debit note interval value.
   4.2 Provider / (yagna) will resend the debit note in case of an error during sending a debit note for acceptance. (max 5 x, not after sending interval)
   4.3 Requestor / (yagna) will resend debit note acceptance in case of a network error. (max 5 x, not after acceptance timeout)
   4.4 Provider will not terminate the agreement if their unaccepted debit note would be followed by an accepted one. (Acceptance of a debit note means acceptance of all previously issued for the same activity). - no change

Justification:

Why should all debit notes be payable?
Current payment solution leads to unpaid work in case mid-agreement engagements are interrupted before sending and accepting invoices. Golem believes there is no need to postpone the payments for more than 1/days (Golem goal is not to pay more often then 1 / day )

Why do we keep using debit notes as a heartbeat mechanism?
Debit note is still a heartbeat mechanism for both providers and requestor to monitor if the other side of agreement is still active. There is no need to implement an additional heartbeat mechanism, as there is no use for providers to get a heartbeat more often if the subsequent debit notes would not be accepted and thus not paid.

What should be interval values?
For Jobs where multiple providers are engaged, the requestor might need to decrease the frequency of debit-notes, however the practical upper limit is 15 minutes. The higher the interval the bigger the risk for providers.

Intervals of 2 minutes are too short!
Requestors might argue that with short debit note intervals they need to accept costs for partial tasks, without a guarantee the task would be completed to the state its result would be useful. Golem believes the requestor hires resources not a task output, and is primarily responsible for demand specification and requestor code, thus to a higher degree in control of potential failure. Therefore in most situations the requestor is responsible for a failure. Requestor should organise its code in a wait that could handle provider failures. Their risk’s EMV for a single debit note period is relatively small and does not justify implementation of any complex mechanism to withdraw already accepted debit notes in order to penalise providers. Agreement cancelation and impact of provider reputation should be a sufficient encouragement for providers to deliver the quality service.

How to address that some debit note or its acceptance might not be delivered?
Debit note acceptance is part of the “heartbeat” mechanism. Configurable timeout is designed to allow some tolerance in case the debit note acceptance would not be processed in time. As the debit-note interval is variable, we decided not to relate it to the number of “potentially” missed debit notes, but to define it as a time period. It should be a double value of the debit note interval to allow at least for a single miss.
Note: this time is not intended for the requestor to perform detailed output verification, rather then a quick verification if the usage increase is justified and price aligned with the agreement.

As the recommended tolerance of a single debit-might be considered too small, provider and requestor will implement additional measures to ensure a debit-note is sent, in case a network issues.

Providers might abuse the start price to get paid even when deployment fails:

Requestors should not accept providers with excessive start price, especially if the provider has low reputation, for unique resources paying for start might make sense.

Note: Point 3 has an impact on yapapi that checks and validates frequency Debit Notes are sent by Providers.

Changes considered but not proposed:

Separate heartbeat mechanism. There is no need as the debit note interval and debit note timeout actually work as one.

Automatic allocation expiration update (when the requestor agent accepts a debit note:
This is a scope of another change related to allocation logic.

Additional flag to indicate spot Providers. There is no actual penalty for not respecting such an obligation.

Additional negotiable parameter defining a provider close notice: There is no actual penalty for not respecting such an obligation. We might consider adding a warning for a provider that there is activity running, to prevent stopping yagna. Even on cloud systems most of the crashes are unexpected. Rather than a requestor app should be resilient to sudden provider failure.
[Such param could define the closing delay, so would give the provider a reference delay value].

Debit note on demand. Requestors can utilise getCurrentUsage() to align task completion with the resource usage.

Agreement expiration as optional param. This parameter is actually optional (but SDKs use defaults) These defaults make a potential issue for requestors of early agreement expiration before all tasks are completed.

Demand subscription expiration. Currently yagna uses its default of 1h. It was added for symmetry to offer expiration. Subscribed demands are used to filter offers that providers are refreshing every 1h.