Annotation on Trace Level

[hristogg]

Hi,
I am looking for a way to implement capturing end user feedback, however the only possible way I see in the documentation currently is a annotating programmaticaly the span_id through an annotation payload.
However I am ussing llamaindex workflow which generates multiple spans.
I can manage to get the trace_id, however I see no option to programmaticaly update the anontation on the trace_id. Is this possible? The documentation does not seem to mention anything.

Thanks in advance!

[dosubot[bot]]

Yes, it is possible to programmatically update the annotation on the trace_id when using the llamaindex workflow. The OpenInferenceTraceCallbackHandler class allows you to handle events and spans, enabling you to update span attributes. You can modify the attributes of a span by updating the attributes dictionary in the _EventData class, which is then used to set attributes on the span in the _finish_tracing function.

Additionally, with Arize Phoenix (version 4.20.0 and above), you can attach user feedback to spans and traces in the form of annotations. This involves creating an annotation with fields such as span_id, name, annotator_kind, and result, which includes a label, score, and explanation. You can send this annotation to Phoenix via its REST API. Here's an example of how you can send an annotation using Python:

from opentelemetry import trace
import httpx

span = trace.get_current_span()
span_id = span.get_span_context().span_id.to_bytes(8, "big").hex()

client = httpx.Client()

annotation_payload = {
    "data": [
        {
            "span_id": span_id,
            "name": "user feedback",
            "annotator_kind": "HUMAN",
            "result": {"label": "thumbs-up", "score": 1},
            "metadata": {},
        }
    ]
}

client.post(
    "http://PHOENIX_HOST:PHOENIX_PORT/v1/span_annotations?sync=false",
    json=annotation_payload,
)

This approach allows you to capture and annotate user feedback on spans, providing valuable insights into application performance and user interactions ^[1][2].

To continue talking to Dosu, mention @dosu.

[dosubot[bot]]

To annotate at the trace level in Arize Phoenix, you can use the annotate_traces API endpoint. This allows you to create or update trace annotations directly. You can send annotations to Phoenix via its REST API by posting to the /v1/span_annotations endpoint, which supports attaching user feedback to both spans and traces. The fields required for trace-level annotations include trace_id, name, annotator_kind, label, score, and explanation ^[1][2].

To continue talking to Dosu, mention @dosu.

[hristogg]

@dosu, can you give me example for utilizing the API given a TRACE_ID?

[dosubot[bot]]

To utilize the API to annotate a trace given a TRACE_ID, you can use the following example in Python:

import httpx

client = httpx.Client()

annotation_payload = {
    "data": [
        {
            "span_id": "TRACE_ID",  # Replace TRACE_ID with your actual trace ID
            "name": "user feedback",
            "annotator_kind": "HUMAN",
            "result": {"label": "thumbs-up", "score": 1},
            "metadata": {},
        }
    ]
}

client.post(
    "http://PHOENIX_HOST:PHOENIX_PORT/v1/span_annotations?sync=false",
    json=annotation_payload,
)

Replace "TRACE_ID" with the actual trace ID you want to annotate. This code sends a POST request to the Phoenix API to attach an annotation to the specified trace. This allows you to collect feedback in the form of annotations, which can be from human users or evaluations from LLMs ^[1][2].

To continue talking to Dosu, mention @dosu.

[hristogg]

@dosu, these does not seem to work. I receive success from the post request, but the feedback does not appear on the CHAIN - Trace level.

[dosubot[bot]]

If the feedback is not appearing at the trace level despite a successful POST request, ensure the following:

1. Correct Endpoint: Verify that you are using the correct API endpoint for trace-level annotations. The endpoint should be specifically for trace annotations, not span annotations.

2. Payload Structure: Double-check that the payload structure matches the required fields for trace-level annotations. The fields should include trace_id, name, annotator_kind, label, score, and explanation ^[1].

3. Version Compatibility: Ensure that you are using a compatible version of Arize Phoenix. Feedback and annotations are available for arize-phoenix>=4.20.0, with improvements in later versions like 4.29.0 ^[2].

4. Authentication: Make sure that the request includes the necessary authentication headers if authentication is enabled. This typically involves setting the Authorization header with a bearer token ^[3].

5. Configuration: Confirm that your environment variables, such as PHOENIX_CLIENT_HEADERS and PHOENIX_COLLECTOR_ENDPOINT, are correctly set.

If all these conditions are met and the issue persists, there might be a need to check the server logs for any errors or consult the documentation for any additional configuration requirements.

To continue talking to Dosu, mention @dosu.

[RogerHYang]

@hristogg Annotating by trace ID is not currently supported in the UI. Alternatively, would it be helpful if you could obtain the span_id of the root span in your workflow? If so, we could add that functionality to the llama-index instrumentor.

[hristogg]

@RogerHYang thanks! I ended up annotating all the span_ids under the trace. Generally from my perspective I think it makes sense to annottate on trace level as that user has no idea in some cases of the underlying spans and what they do. Furthermore if I annotate a span on the UI it is not shown that there is annotation unless I expand the trace. Thanks for the support!