Publish via dashboard REST-API fails

[marvinkruse]

With the latest version (commit de344f1) of this plugin the from_client_id is now correctly set when receiving a publish message via the web hook (see #75). But after upgrading the web hook plugin to the latest version I wasn't able to publish via the dashboards HTTP API anymore. Publishes via web socket/regular clients work for me, but publishing over the dashboard HTTP API produces the following error:

Publish to 127.0.0.1:18083/api/v3/mqtt/publish with valid basic authentication (admin:public) sending JSON: "{"topic": "testtopic", "payload": "hello", "qos": 1, "retain": true, "client_id":"mqttjs_ab9069449e"} returns an Internal Error

The log of the broker shows:

2018-11-30 13:05:59.556 [error] POST /api/v3/mqtt/publish error: function_clause, stacktrace:
[{emqx_web_hook,format_from,
                [{message,<<0,5,123,224,159,84,118,22,244,66,0,0,8,121,0,0>>,
                          1,<<"mqttjs_ab9069449e">>,
                          #{retain => true},
                          #{},<<"testtopic">>,<<"hello">>,
                          {1543,579559,556628}}],
                [{file,"src/emqx_web_hook.erl"},{line,278}]},
 {emqx_web_hook,'-on_message_publish/2-fun-0-',2,
                [{file,"src/emqx_web_hook.erl"},{line,175}]},
 {emqx_hooks,run_,3,[{file,"src/emqx_hooks.erl"},{line,105}]},
 {emqx_broker,publish,1,[{file,"src/emqx_broker.erl"},{line,155}]},
 {lists,foreach,2,[{file,"lists.erl"},{line,1338}]},
 {emqx_mgmt_api_pubsub,publish,2,
                       [{file,"src/emqx_mgmt_api_pubsub.erl"},{line,68}]},
 {minirest_handler,dispatch,3,[{file,"src/minirest_handler.erl"},{line,55}]},
 {minirest,handle_request,2,[{file,"src/minirest.erl"},{line,66}]}]

Any idea why this doesn't work anymore?

[gilbertwong96]

Thanks for your feedback, we'll check it.

[gilbertwong96]

Hi, @marvinkruse , it is not the bug of emqx rest api. We have adopted new strategy for rest api usage. You could refer this document. First, you have to create new app by using dashboard, then use the : as the basic authentication fields.

[marvinkruse]

Thanks @gilbert-wong , I created an app in the dashboard and I am able to use the API with these credentials (like before, nothing changed). But publishes are still producing the same error.

Publish to 127.0.0.1:8080/api/v3/mqtt/publish with valid basic authentication (ab14b3ccdb135:Mjg0ODE4MDk1NjA4NjI5ODQ3NDI1NjAzMjA2NjI3MzI4MDA) sending JSON: "{"topic": "testtopic", "payload": "hello", "qos": 1, "retain": true, "client_id":"mqttjs_ab9069449e"}" returns an Internal Error

The log of the broker shows:

2018-12-05 10:25:03.806 [error] POST /api/v3/mqtt/publish error: function_clause, stacktrace:
[{emqx_web_hook,format_from,
                [{message,<<0,5,124,66,245,2,70,197,244,66,0,0,8,111,0,0>>,
                          1,<<"mqttjs_ab9069449e">>,
                          #{retain => false},
                          #{},<<"test_topic">>,<<"hello">>,
                          {1544,1903,806116}}],
                [{file,"src/emqx_web_hook.erl"},{line,278}]},
 {emqx_web_hook,'-on_message_publish/2-fun-0-',2,
                [{file,"src/emqx_web_hook.erl"},{line,175}]},
 {emqx_hooks,run_,3,[{file,"src/emqx_hooks.erl"},{line,105}]},
 {emqx_broker,publish,1,[{file,"src/emqx_broker.erl"},{line,155}]},
 {lists,foreach,2,[{file,"lists.erl"},{line,1338}]},
 {emqx_mgmt_api_pubsub,publish,2,
                       [{file,"src/emqx_mgmt_api_pubsub.erl"},{line,68}]},
 {minirest_handler,dispatch,3,[{file,"src/minirest_handler.erl"},{line,55}]},
 {minirest,handle_request,2,[{file,"src/minirest.erl"},{line,66}]}]

[gilbertwong96]

Hi, @marvinkruse , I cannot reproduce your problem, Which version of emqx-rel did you use ? I tested with emqx3.0-rc.5,there is no such problems. Please update the whole project to emqx3.0-rc.5 and try again.

[marvinkruse]

Thanks @gilbert-wong , I was using the latest commit, now I tried to build emqx3.0-rc.5 but it always fails. The error below occurs all the time.

 DEP    emqx_lua_hook
 DEP    jsx
{"init terminating in do_boot",{{badmatch,{ok,{{"HTTP/1.1",403,"Forbidden"},[{"date","Mon, 10 Dec 2018 09:11:13 GMT"},{"server","AmazonS3"},{"content-length","243"},{"content-type","application/xml"},{"x-amz-request-id","60E653041CFD4401"},{"x-amz-id-2","UhPsdBcWrMo0Xp1c2Ek4eIXgNW9k38lcBEy0MRK9RVjSloTUISap+sgdLbrQrEVcqw2k1/QDMXg="}],<<60,63,120,109,108,32,118,101,114,115,105,111,110,61,34,49,46,48,34,32,101,110,99,111,100,105,110,103,61,34,85,84,70,45,56,34,63,62,10,60,69,114,114,111,114,62,60,67,111,100,101,62,65,99,99,101,115,115,68,101,110,105,101,100,60,47,67,111,100,101,62,60,77,101,115,115,97,103,101,62,65,99,99,101,115,115,32,68,101,110,105,101,100,60,47,77,101,115,115,97,103,101,62,60,82,101,113,117,101,115,116,73,100,62,54,48,69,54,53,51,48,52,49,67,70,68,52,52,48,49,60,47,82,101,113,117,101,115,116,73,100,62,60,72,111,115,116,73,100,62,85,104,80,115,100,66,99,87,114,77,111,48,88,112,49,99,50,69,107,52,101,73,88,103,78,87,57,107,51,56,108,99,66,69,121,48,77,82,75,57,82,86,106,83,108,111,84,85,73,83,97,112,43,115,103,100,76,98,114,81,114,69,86,99,113,119,50,107,49,47,81,68,77,88,103,61,60,47,72,111,115,116,73,100,62,60,47,69,114,114,111,114,62>>}}},[{erl_eval,expr,5,[{file,"erl_eval.erl"},{line,450}]},{erl_eval,exprs,5,[{file,"erl_eval.erl"},{line,126}]},{init,start_it,1,[]},{init,start_em,1,[]},{init,do_boot,3,[]}]}}
init terminating in do_boot ({{badmatch,{ok,{{_},[_],}}},[{erl_eval,expr,5,[{_},{_}]},{erl_eval,exprs,5,[{_},{_}]},{init,start_it,1,[]},{init,start_em,1,[]},{init,do_boot,3,[]}]})

Crash dump is being written to: erl_crash.dump...done

[marvinkruse]

When using the pre-built binaries from http://emqtt.io/downloads it doesn't work as well. I'm sending the following curl (which worked in previous versions of emqx):

curl --header "Content-Type: application/json" --request POST --data "{\"topic\": \"test_topic\",\"payload\": \"hello\",\"qos\": 1,\"retain\": false,\"client_id\": \"mqttjs_ab9069449e\"}" -v --basic -u testApp:Mjg0ODk3ODM5NDY4NzcxNzIxNDEwMjYwOTc3MjU3MDIxNDE -k http://localhost:8080/api/v3/mqtt/publish

The same error occurs:

(emqx@127.0.0.1)1> 2018-12-10 10:31:32.764 [debug] API publish Params:[{<<"topic">>,<<"test_topic">>},
                    {<<"payload">>,<<"hello">>},
                    {<<"qos">>,1},
                    {<<"retain">>,false},
                    {<<"client_id">>,<<"mqttjs_ab9069449e">>}]
2018-12-10 10:31:32.764 [info] PUBLISH to test_topic: <<"hello">>
2018-12-10 10:31:32.764 [error] POST /api/v3/mqtt/publish error: function_clause, stacktrace:
[{emqx_web_hook,format_from,
                [{message,<<0,5,124,167,161,102,48,75,244,66,0,0,6,86,0,0>>,
                          1,<<"mqttjs_ab9069449e">>,
                          #{retain => false},
                          #{},<<"test_topic">>,<<"hello">>,
                          {1544,434292,764763}}],
                [{file,"src/emqx_web_hook.erl"},{line,278}]},
 {emqx_web_hook,'-on_message_publish/2-fun-0-',2,
                [{file,"src/emqx_web_hook.erl"},{line,175}]},
 {emqx_hooks,run_,3,[{file,"src/emqx_hooks.erl"},{line,105}]},
 {emqx_broker,publish,1,[{file,"src/emqx_broker.erl"},{line,150}]},
 {lists,foreach,2,[{file,"lists.erl"},{line,1338}]},
 {emqx_mgmt_api_pubsub,publish,2,
                       [{file,"src/emqx_mgmt_api_pubsub.erl"},{line,68}]},
 {minirest_handler,dispatch,3,[{file,"src/minirest_handler.erl"},{line,55}]},
 {minirest,handle_request,2,[{file,"src/minirest.erl"},{line,66}]}]

[hibacky]

I also found a similar problem.
When not use emqx_web_hook (on_message_publish), the HTTP REST API (api/v3/mqtt/publish) is OK
When use emqx_web_hook (on_message_publish), the HTTP REST API (api/v3/mqtt/publish) return 'Internal Error'.

@gilbert-wong

[gilbertwong96]

Thanks for you feedback, I would check it.

[gilbertwong96]

Hi, @marvinkruse , Sorry for late response. I'm not familiar with curl, I try with insomnia, and there is no such problem. Maybe you misuse the curl. I could not reproduce your problem.

[deepakchhapru]

Hi @gilbert-wong,

I am also facing this problem when i try with fiddler.

When i disable emqx_web_hook (on_message_publish), the HTTP REST API (api/v3/mqtt/publish) is OK
When emqx_web_hook (on_message_publish) is enabled, the HTTP REST API (api/v3/mqtt/publish) return 'Internal Error'.

Kindly help.

Thanks

[marvinkruse]

Hi, @marvinkruse , Sorry for late response. I'm not familiar with curl, I try with insomnia, and there is no such problem. Maybe you misuse the curl. I could not reproduce your problem.

Hi @gilbert-wong, sorry for the late response. I didn't only try with curl, I also used other ways (other tools, various machines with a different OS, etc.) to verify this behavior. I will try with insomnia and report back, but I doubt that this will solve it. As you can see others are also facing the same issue.

[oOLucioleOo]

I also found a similar problem with version 3.0.0 of the docker pre-built binaries from http://emqtt.io/downloads.
For reproduce, run docker container with exposed 18083 port, then, on the dashboard, start web_hook plugins.
Finally, call the HTTP REST API (api/v3/mqtt/publish) with your preferred tools. This reproduce the same error.

[marvinkruse]

Sorry for the late response @gilbert-wong, I tried with Insomnia and the latest emqx release 3.0 now again. Same error when the web_hook plugin is enabled.

(emqx@127.0.0.1)1> 2019-01-16 14:43:20.189 [debug] API publish Params:[{<<"qos">>,1},
                    {<<"retain">>,false},
                    {<<"topic">>,<<"world">>},
                    {<<"payload">>,<<"test">>},
                    {<<"client_id">>,<<"C_1492145414740">>}]
2019-01-16 14:43:20.189 [info] PUBLISH to world: <<"test">>
2019-01-16 14:43:20.189 [error] POST /api/v3/mqtt/publish error: function_clause, stacktrace:
[{emqx_web_hook,format_from,
                [{message,<<0,5,127,147,118,0,19,3,244,67,0,0,6,152,0,0>>,
                          1,<<"C_1492145414740">>,
                          #{retain => false},
                          #{},<<"world">>,<<"test">>,
                          {1547,646200,189699}}],
                [{file,"src/emqx_web_hook.erl"},{line,278}]},
 {emqx_web_hook,'-on_message_publish/2-fun-0-',2,
                [{file,"src/emqx_web_hook.erl"},{line,175}]},
 {emqx_hooks,run_,3,[{file,"src/emqx_hooks.erl"},{line,105}]},
 {emqx_broker,publish,1,[{file,"src/emqx_broker.erl"},{line,168}]},
 {lists,foreach,2,[{file,"lists.erl"},{line,1338}]},
 {emqx_mgmt_api_pubsub,publish,2,
                       [{file,"src/emqx_mgmt_api_pubsub.erl"},{line,70}]},
 {minirest_handler,dispatch,3,[{file,"src/minirest_handler.erl"},{line,55}]},
 {minirest,handle_request,2,[{file,"src/minirest.erl"},{line,66}]}]

[marvinkruse]

Is there any update on this? It seems that this is still not fixed?