Task queue webhook

[irevoire]

Original PRD: https://www.notion.so/meilisearch/Task-Queue-Webhook-2c9ff1c6b820489399194539d37c4d11

Summary

Introduce a configuration for a webhook url that will be called whenever a task finishes so a third party can be notified. It will be used to compute some of the required metrics for the Monitoring Metrics v1 project.

---

Changes

• The webhook URL is configured either via the env var MEILI_TASK_WEBHOOK_URL or via the CLI option flag --task-webhook-url
  • By default, the webhook URL value is empty.
  • If the given value does not match a valid URL format, a human-readable error is returned to the user and the engine does not start.
  • A valid user-provided URL is kept as is.
  • You can optionally set an Authorization header by using the --task-webhook-authorization-header CLI option flag or via the MEILI_TASK_WEBHOOK_AUTHORIZATION_HEADER env var.
• When a valid webhook URL is configured and when a batch of tasks reaches a finished status (status: succeeded/failed), the engine sends a POST request to the user-configured webhook URL. The HTTP request contains the finished tasks in its payload.
  • The sent payload is compressed with gzip and represented in the JSON Lines text format.
• A task sent over the webhook URL matches the task object definition.
• Payload example
  • In that case, the user configured https://myproject.com/mywebhook?common=people

//POST HTTP request to https://myproject.com/mywebhook?common=people

{"uid":4,"indexUid":"movie","status":"failed","type":"indexDeletion","canceledBy":null,"details.deletedDocuments":0,"error.message":"Index `movie` not found.","error.code":"index_not_found","error.type":"invalid_request","error.link":"https://docs.meilisearch.com/errors#index_not_found","duration":"PT0.001192S","enqueuedAt":"2022-08-04T12:28:15.159167Z","startedAt":"2022-08-04T12:28:15.161996Z","finishedAt":"2022-08-04T12:28:15.163188Z"}
{"uid":5,"indexUid":"movie","status":"failed","type":"indexDeletion","canceledBy":null,"details.deletedDocuments":0,"error.message":"Index `movie` not found.","error.code":"index_not_found","error.type":"invalid_request","error.link":"https://docs.meilisearch.com/errors#index_not_found","duration":"PT0.001192S","enqueuedAt":"2022-08-04T12:28:15.159167Z","startedAt":"2022-08-04T12:28:15.161996Z","finishedAt":"2022-08-04T12:28:15.163188Z"}
{"uid":6,"indexUid":"movie","status":"failed","type":"indexDeletion","canceledBy":null,"details.deletedDocuments":0,"error.message":"Index `movie` not found.","error.code":"index_not_found","error.type":"invalid_request","error.link":"https://docs.meilisearch.com/errors#index_not_found","duration":"PT0.001192S","enqueuedAt":"2022-08-04T12:28:15.159167Z","startedAt":"2022-08-04T12:28:15.161996Z","finishedAt":"2022-08-04T12:28:15.163188Z"}

Misc

• Update telemetry datapoints (if needed; Apply the Telemetry label)

[macraig]

LGTM!