telemetry: Adding metrics for amazonq unit test generation #915
Conversation
| "description": "Number of characters of code generated" | ||
| }, | ||
| { | ||
| "name": "generatedCount", |
There was a problem hiding this comment.
this name is unhelpful. why not generatedTestCasesCount?
There was a problem hiding this comment.
Please refer this PR
Long story short Toolkit want to reuse the fields so the ask is to keep this as generic as possible.
There was a problem hiding this comment.
Each field is "namespaced" in a metric, so there is no need to name each field in a globally-unique way, and it's actually counterproductive (because it results in 10x or 100x the number of field names, which reduces discoverability, re-use, and leads to less-helpful dashboards and queries).
So for example, this field would live on a metric named amazon_foo and so amazonq_foo.generatedCount would clearly be understood based on the metric it is part of.
There was a problem hiding this comment.
You probably don't need this field, just use the count field:
| { | ||
| "name": "buildSystemVersion", | ||
| "type": "string", | ||
| "description": "The build system version on the user's machine" | ||
| }, | ||
| { | ||
| "name": "buildZipFileBytes", |
There was a problem hiding this comment.
We does not have a field for Number of files and I do not think it adds a value as file can be of any size and number does not define our latency or test generation experience.
Correct me If I am missing something.
| "required": false | ||
| }, | ||
| { | ||
| "type": "cwsprChatProgrammingLanguage" |
There was a problem hiding this comment.
why is this a chat programming language?
There was a problem hiding this comment.
We planned to reuse existing fields as much as possible. Please refer this PR
| "required": false | ||
| }, | ||
| { | ||
| "type": "generatedCount", |
There was a problem hiding this comment.
This is on a metric named amazonq_unitTestGeneration. Can just use the count field:
each field does not need its own namespace; it's already namespaced in a metric.
There was a problem hiding this comment.
Follow this suggestion.
There was a problem hiding this comment.
Sure, but count already exists... when a metric's main topic needs a count, just use the existing count field.
| "required": false | ||
| }, | ||
| { | ||
| "type": "userEnteredPromptMessage" |
There was a problem hiding this comment.
you cannot save this as telemetry. This is customer data.
There was a problem hiding this comment.
As declared this is boolean, so there is no harm for customer data.
| "required": false | ||
| }, | ||
| { | ||
| "type": "userEnteredPromptMessage" |
| "required": false | ||
| }, | ||
| { | ||
| "type": "generatedCount", |
There was a problem hiding this comment.
This is on a metric named amazonq_unitTestGeneration. Can just use the count field:
each field does not need its own namespace; it's already namespaced in a metric.
| "name": "amazonq_utg_generateTests", | ||
| "description": "Client side invocation of the AmazonQ Unit Test Generation", |
There was a problem hiding this comment.
how is this different than amazonq_unitTestGeneration ?
"name": "amazonq_unitTestGeneration", "description": "Client side metrics of AmazonQ Unit Test Generation",
There was a problem hiding this comment.
This will be the E2E metric after Build execution loop
There was a problem hiding this comment.
That isn't indicated in the description, and still doesn't explain how it's different
There was a problem hiding this comment.
This is not being added as part of RIV.
Problem
Solution
Planning to add 4 metrics for Unit test generation.
License
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.