What to do about DagsterUserCodeUnreachableError?

[sryza]

These errors can arise while launching a job:

dagster._core.errors.DagsterUserCodeUnreachableError: Timed out waiting for call to user code GET_SUBSET_EXTERNAL_PIPELINE_RESULT [9f172afa-3dde-4cff-a4d9-a7526fbfd6cc]

or while evaluating sensors:

dagster._core.errors.DagsterUserCodeUnreachableError: dagster._core.errors.DagsterUserCodeUnreachableError: The sensor tick timed out due to taking longer than 60 seconds to execute the sensor function. One way to avoid this error is to break up the sensor work into chunks, using cursors to let subsequent sensor calls pick up where the previous call left off.

Stack Trace:
  File "/dagster-cloud/dagster_cloud/agent/dagster_cloud_agent.py", line 802, in _process_api_request
    api_result = self._handle_api_request(
  File "/dagster-cloud/dagster_cloud/agent/dagster_cloud_agent.py", line 662, in _handle_api_request
    serialized_sensor_data_or_error = client.external_sensor_execution(
  File "/dagster/dagster/_grpc/client.py", line 406, in external_sensor_execution
    chunks = list(
  File "/dagster/dagster/_grpc/client.py", line 194, in _streaming_query
    self._raise_grpc_exception(
  File "/dagster/dagster/_grpc/client.py", line 145, in _raise_grpc_exception
    raise DagsterUserCodeUnreachableError(

The above exception was caused by the following exception:
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Deadline Exceeded {created_time:"2023-11-04T20:49:29.061099977+00:00", grpc_status:4}"
>

[garethbrickman]

Setting these timeout values is deployment-specific. If using a Kubernetes agent configured with Helm chart, ­in the chart template there is:

​serverProcessStartupTimeout​​ - Specifies the timeout in seconds the agent should use when waiting for a code server to be ready after the Kubernetes deployment is created. This timeout should be increased if your code locations can take more than 3 minutes to load your Dagster definitions.

This is the time required for the code location container to start and the grpc server being ready and serving requests. If you are using many libraries, or are creating a dbt manifest dynamically, you may need to increase this.

​deploymentStartupTimeout​​ - Specifies the timeout in seconds when creating a new Kubernetes deployment for a code server. If not set, defaults to 300 seconds.

This is the time the agent will wait from the deployment initialization to the pod being running. If your cluster depends on slow node provisioning when starting new pods, you might need to increase that.

There is an extensive guide for debugging and managing sensor timeouts specifically in #20021

[babaMar]

@garethbrickman that chart template is for the dagster-cloud agents right? Are there similar parameters for dagster-user-deployments? I'm getting a similar error:

dagster._core.errors.DagsterUserCodeUnreachableError: User code server request timed out due to taking longer than 180 seconds to complete.

  File "/usr/local/lib/python3.10/site-packages/dagster/_core/workspace/context.py", line 633, in _load_location
    origin.reload_location(self.instance)
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/remote_representation/origin.py", line 347, in reload_location
    self.create_client().reload_code(timeout=instance.code_server_reload_timeout)
  File "/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py", line 323, in reload_code
    return self._query("ReloadCode", api_pb2.ReloadCodeRequest, timeout=timeout)
  File "/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py", line 173, in _query
    self._raise_grpc_exception(
  File "/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py", line 151, in _raise_grpc_exception
    raise DagsterUserCodeUnreachableError(

The above exception was caused by the following exception:
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2025-01-14T18:35:57.316206811+00:00"}"
>

  File "/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py", line 171, in _query
    return self._get_response(method, request=request_type(**kwargs), timeout=timeout)
  File "/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py", line 141, in _get_response
    return getattr(stub, method)(request, metadata=self._metadata, timeout=timeout)
  File "/usr/local/lib/python3.10/site-packages/grpc/_channel.py", line 1181, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.10/site-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
    ```