Instrumentation with influxdb backend #368
Conversation
…skHeartbeat and RecordActivityTaskHeartbeat endpoints
define hostname attribute for metrology purpose fix hostname parameter string
| @@ -127,6 +127,9 @@ def heartbeat(self, task_token, details=None): | |||
| task_token, | |||
| format.heartbeat_details(details), | |||
| ) | |||
| # instrumentation | |||
| self.send_endpoint_usage("RecordActivityTaskHeartbeat") | |||
There was a problem hiding this comment.
All the instrumentation lines should be put before the actual call to the endpoint. Otherwise if the call makes a 429, an exception is raised and you'll never hit influxdb for this call. Which means in practice that graphs won't ever reach the rate limiting thresholds, whereas we want them to do so. I'm actually unsure if when SWF answers a 429 it's counted in our rate limit, but I think we're safer if we have an exaggerated view than the other way around. Last but not least the line above is never called (it's after a "return").
Also: I'd be happy that we integrate instrumentation in a generic way that doesn't pollute API calls. Basically connection.call_endpoint() should result in the endpoint instrumentation being triggered AND the endpoint being called. But that can come in a second phase I think.
|
|
||
| self.hostname = socket.gethostname() | ||
|
|
||
| def send_endpoint_usage(endpoint, host="grafana.botify.com", port=8089): |
There was a problem hiding this comment.
Shouldn't be hardcoded, please push force with no botify value asap, since we're disclosing a private info on a public repository here ;-) You have examples in the repo of using simpleflow settings. You can grep for "SIMPLEFLOW_ENABLE_DISK_CACHE" for instance to get a simple enough example.
Description
See comments from this issue for details about the implementation choice of InfluxDB.
I looked for the 2 endpoints referred in the issue 298
RecordActivityTaskHeartbeatandPollForActivityTask, and implement a method insideswf/core.pyto send statistic to InfluxDB.I define the tags
regionendpointandhostname.Considerations for Reviewers
This library is used to talk to InfluxDB backend, I assume this part of the code would be non blocking and should not impact the simpleflow behavior, because UDP do not except any answer from the server.
Maybe we should use some AWS tag instead of the worker hostname (not human friendly)
Maybe we should pay attention if the worker would not be able to resolve the grafana fqdn.
Authentication and encryption :
The UDP listener on InfluxDB server doesn't seems to support this, a solutions could consist in writing to a Telegraf proxy agent wich would send the data to Influx over HTTPS.
Or use a local database in the same network.
Test environment/Test case
Considerations for Deployment
In order to implement this solution, the InfluxDB server would have to be setup and the environment secured.