Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wrong type converter with Avro format #214

Open
onepas opened this issue Dec 12, 2023 · 1 comment
Open

Wrong type converter with Avro format #214

onepas opened this issue Dec 12, 2023 · 1 comment

Comments

@onepas
Copy link

onepas commented Dec 12, 2023

When used connector in ksqldb, some data types ex: DECIMAL, DATE (in ksqldb table) are not correct when convert value
ex:
-1,507,945.00 -> "9wMO/A=="
35,068.00 -> "NYJw"
01 DEC 2023 -> 19692

CREATE SINK CONNECTOR SINK_TRANSACTIONS_RULE01_HTTP WITH (
    'connector.class'                     = 'io.aiven.kafka.connect.http.HttpSinkConnector',
    'http.url'                            = 'https://webhook.site/6d74bf60-f754-4f57-b1ee-8eed9208b9ea',
    'http.authorization.type'             = 'none',
    'topics'                              = 'ksql_transactions_rule01',
    'key.converter'                       = 'io.confluent.connect.avro.AvroConverter',
    'key.converter.schema.registry.url'   = 'http://localhost:8081',
    'value.converter'                     = 'io.confluent.connect.avro.AvroConverter',
    'value.converter.schema.registry.url' = 'http://localhost:8081',
    'batching.enabled'                    = 'true',
    'batch.max.size'                      =  2,
    'tasks.max'                           = '1'
  );
@mgvinuesa
Copy link

Same here,

The received body:

{"ORDERID":"Mw==","ORDERDATE":1109721600000,"CUSTOMERID":"ZQ==","AMOUNT":3250.25}]

Both ORDERID and CUSTOMERID are org.apache.kafka.connect.data.Decimal in the avro schema

My configuration:

{
    "name": "http_sink_connector",
    "config": {
        "topics": "PULL_ORDER_DATA",
        "tasks.max": "1",
        "connector.class": "io.aiven.kafka.connect.http.HttpSinkConnector",
        "http.url": "http://api:8080/api/orderData",
        "key.converter": "org.apache.kafka.connect.storage.StringConverter",
        "value.converter": "io.confluent.connect.avro.AvroConverter",
        "value.converter.schema.registry.url": "http://schema-registry:8081",
        "schema.registry.url": "http://schema-registry:8081",
        "http.authorization.type": "none",
        "http.headers.content.type": "application/json"
    }
}

The schema

{
  "connect.name": "ORDER_DATA",
  "fields": [
    {
      "name": "ORDERID",
      "type": {
        "connect.name": "org.apache.kafka.connect.data.Decimal",
        "connect.parameters": {
          "connect.decimal.precision": "38",
          "scale": "0"
        },
        "connect.version": 1,
        "logicalType": "decimal",
        "precision": 38,
        "scale": 0,
        "type": "bytes"
      }
    },
    {
      "name": "ORDERDATE",
      "type": {
        "connect.name": "org.apache.kafka.connect.data.Timestamp",
        "connect.version": 1,
        "logicalType": "timestamp-millis",
        "type": "long"
      }
    },
    {
      "default": null,
      "name": "CUSTOMERID",
      "type": [
        "null",
        {
          "connect.name": "org.apache.kafka.connect.data.Decimal",
          "connect.parameters": {
            "connect.decimal.precision": "38",
            "scale": "0"
          },
          "connect.version": 1,
          "logicalType": "decimal",
          "precision": 38,
          "scale": 0,
          "type": "bytes"
        }
      ]
    },
    {
      "name": "AMOUNT",
      "type": "double"
    }
  ],
  "name": "ORDER_DATA",
  "type": "record"
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants