MQTT connection via WebSocket with aws-iot-device-sdk-cpp-v2 (v1.31.0)

[omri-s-electreon]

Hi,

We are developing an IoT solution with the aws-iot-device-sdk-cpp-v2 library. Our devices run an Embedded Linux OS with a user-space application written in C++ (C runtime is uclibc, ARM CPU). This application links with the AWS SDKs (*.so libraris):

aws-iot-device-sdk-cpp-v2 (v1.31.0)
aws-crt-cpp (v0.25.0)
aws-sdk-cpp (v1.11.230)
aws-c-auth (v0.7.8)
aws-c-cal (v0.6.9)
aws-c-common (v0.9.10)
aws-c-compression (v0.2.17)
aws-c-event-stream (v0.3.2)
aws-c-http (v0.7.15)
aws-c-io (v0.13.36)
aws-c-iot (v0.1.18)
aws-c-mqtt (v0.10.0)
aws-c-s3 (v0.4.6)
aws-c-sdk-utils (v0.1.13)
aws-checksums (v0.1.17)
aws-lc (v1.19.0)

We have been trying to implement a MQTT connection based on Websocket with no luck. Normal connection (TCP socket, port 8883) works with no issues but when trying to run our implementation of a Websocket connection (port 443) - We get an error.

Our devices hold 3 "secret files" to allow the application to connect to IoT Core (created using IoT Core APIs):

1. Device certificate
2. Private key
3. Root CA

These secrets allow the device to:
a. Connect to the MQTT enpoint (port 8883) and publish/subscribe messages (telemetry, reported properties etc.).
b. Connect to the authentication endpoint (HTTPS) to get temporary AWS credentials (secret, key & token). Fetching these credentials is currently done by our own custom Credentials Provider (which refreshes them when expired). The credentials in this case are used by the application to write files to a S3 bucket (with a role defined on the IoT Core side).

All of the above are working fine. In addition to (a), we would also like to be able to connect to the WSS endpoint (port 443).

The follwing code is taken from the following examples: Websocket connection example and X509 auth provider example

To implement a WSS connection, we create a CredentialsProviderX509Config which points to the device certificate, private key & root CA files and the IoT Core endpoint. The endpoint is taken from the response of this API (AWS CLI): aws iot describe-endpoint --endpoint-type iot:Data-ATS. No manipulation is done on that string. Example: xyzabc123-ats.iot.eu-west-1.amazonaws.com:

Aws::Crt::Io::TlsContextOptions tlsCtxOptions = Aws::Crt::Io::TlsContextOptions::InitClientWithMtls(this->Cert_Path.c_str(), this->Private_Key_Path.c_str());
	
tlsCtxOptions.OverrideDefaultTrustStore(nullptr, this->CA_Path.c_str());
Aws::Crt::Io::TlsContext x509TlsCtx(tlsCtxOptions, Aws::Crt::Io::TlsMode::CLIENT);
Aws::Crt::Auth::CredentialsProviderX509Config x509Config;
x509Config.TlsOptions = x509TlsCtx.NewConnectionOptions();
x509Config.Endpoint = this->Endpoint.c_str();
x509Config.RoleAlias = this->AWS_Role_Alias.c_str();
x509Config.ThingName = this->Device_Id.c_str();

The CredentialsProviderX509Config object is then used to create a CreateCredentialsProviderX509:

provider = Aws::Crt::Auth::CredentialsProvider::CreateCredentialsProviderX509(x509Config);

The X509 credentials provider created is then used to configure the MQTT over WSS connection:

Aws::Iot::WebsocketConfig wsConfig(this->Region.c_str(), provider);
clientConfigBuilder = Aws::Iot::MqttClientConnectionConfigBuilder(wsConfig);
clientConfigBuilder.WithCertificateAuthority(this->CA_Path.c_str()).WithEndpoint(this->Endpoint.c_str());
auto clientConfig = clientConfigBuilder.Build();
auto connection = this->Mqtt_Client->NewConnection(clientConfig);
this->IoT_Shadow_Client = make_shared<Aws::Iotshadow::IotShadowClient>(connection);

My understanding is that the X509 credentials provider that was created will be using the secret files to fetch temporary AWS credentials when the connection is establshed (and will refresh these credentials when they expire). Please correct me if I'm wrong here.

The above code compiles but when executed - the MQTT over WSS connection fails, that is, the connection callback is called with an error:

ERROR: Connection failed with error: aws-c-auth: AWS_AUTH_SIGNING_NO_CREDENTIALS, Attempt to sign an http request without credentials

Please direct us to a proper solution to establish a MQTT over WSS connection to IoT Core.

[bretambrose]

There may be other issues, but the X509 credentials endpoint and the MQTT broker endpoint are not the same and your code appears to be using the same value for both.

Using the aws cli:

aws iot describe-endpoint --endpoint-type iot:Data-ATS

vs.

aws iot describe-endpoint --endpoint-type iot:CredentialProvider

[omri-s-electreon]

Hi Bret and thanks for spotting the usage of the incorrect enpoint when the provider is created. It definitly makes sense now with you comment.
We used the string outputed from the CLI : aws iot describe-endpoint --endpoint-type iot:CredentialProvider. Example: xyz123.credentials.iot.eu-west-1.amazonaws.com
We tried changing it as you suggested and still could not establish a valid MQTT over WSS connection, but this time with some differences in error messages.

Case I

Using the string as is (xyz123.credentials.iot.eu-west-1.amazonaws.com) or with a https:// prefix (https://xyz123.credentials.iot.eu-west-1.amazonaws.com) resulted with:

 ERROR: Connection failed with error: libaws-c-mqtt: AWS_ERROR_MQTT_UNEXPECTED_HANGUP, The connection was closed unexpectedly.(thread 3019097120)

Case II

Using this string: https://xyz123.credentials.iot.eu-west-1.amazonaws.com/role-aliases/<our_role_alias>/credentials as suggested from here. I understood tht the X509 provider builds this URL internally as it is also given the role alias in addition to the certificate files along with the Thing Name used for the x-amzn-iot-thingname: your thing name header.
Note that this is exactly the URL that our own custom credentials provider is using (to get credentials to upload files to S3) and for that use case it's working fine (this is a credentials provider writen for the aws-sdk-cpp, not the IoT device SDK, and used with the S3 & TransferManager APIs).
This resulted with the same error as with an invalid URL (same case with the incorrect endpoint from the start of this discussion).

ERROR: Connection failed with error: aws-c-auth: AWS_AUTH_SIGNING_NO_CREDENTIALS, Attempt to sign an http request without credentials(thread 3017221152)

I have a feeling we are very close and an additional push will be very helpful.

Thanks again,

Omri

[bretambrose]

Well, don't pass in a url to the x509 provider as it constructs the URL for you when building the http request.

At this point, you're likely running into an AWS resource configuration issue. You're sourcing credentials successfully, but they don't have the permission scope needed to do whatever you're trying to do (connect, subscribe, publish). I would carefully check all of the resource configuration related to the X509 provider (ie the config steps in https://docs.aws.amazon.com/iot/latest/developerguide/authorizing-direct-aws.html).

Consider opening up the associated IAM policy to * temporarily to see if that fixes things. If so, then you know it's just a mismatch between your policy and the operations you're performing with the client.

If you want to attach Debug SDK logs, we can take a look, but that is not likely to help much with a resource configuration/permission-scope issue.

[omri-s-electreon]

Thanks @bretambrose for clarifying how the URL is handled by the X509 provider.

All of our devices in IoT Core are assigned to a ThingGroup which has the folliwng policy:

{
  "Statement": [
    {
      "Action": [
        "iot:Publish",
        "iot:Receive"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:iot:eu-west-1:<account_id>:topic/*"
    },
    {
      "Action": "iot:Subscribe",
      "Effect": "Allow",
      "Resource": "arn:aws:iot:eu-west-1:<account_id>:topicfilter/*"
    },
    {
      "Action": "iot:Connect",
      "Effect": "Allow",
      "Resource": "arn:aws:iot:eu-west-1:<account_id>:client/*"
    },
    {
      "Action": "iot:AssumeRoleWithCertificate",
      "Effect": "Allow",
      "Resource": "arn:aws:iot:eu-west-1:<account_id>:rolealias/iot-s3-access"
    }
  ],
  "Version": "2012-10-17"
}

With the above policy, the regular MQTT connection (port 8883) is working seamlessly. Even more so - The process of getting credentials from the "IoT Core credentials endpoint" (which is done by our own custom provider implemented with libcurl) is also working allowing the devices to assume a role and write files to S3. The only issue is with the Websocket connection (port 443), which is configured with the X509 provider (built-in the SDK, not custom).
Is there any specific WSS configuration required on the policy side? Maybe on the IoT Core configurations side?

Thanks,

Omri

[omri-s-electreon]

Hi @bretambrose ,
We managed to establish a MQTT over WSS connection successfully.
It was indeed a policy configuration of the role alias our devices assume.
I belive it should be better expressed in the documentation.
Thanks again for directing us to a solution.

Omri