Poloniex sequences

[andre77]

Separated the issue, started here #83

[andre77]

Hi everybody,
i have some issues with the poloniex Push API, so decided to use the thread here and not to open a new one

Using the Poloniex Push API we face some weird issues with the sequence number, namely some messages are being dropped and do not arrive on our side.
The messages are received with a sequence number inside. We process them directly and change the internal state of the order book, if the seuence is exact one more than last processed one.
In case the messages are arriving not in the exact order of sequence numbers, the messages are buffered and processed later in the correct order. So we can handle messages very well independent of the order they arrive in.
But the problem, which we can not handle,occurs, if we simply miss a sequence number. It happens pretty often. Currently if we miss a sequence number the messages are buffered, and if the buffer grows over 100 messages, we stop waiting for that missing sequence number. The only thing, we can do, is to start the complete sync process again. We get the snapshot of the order book using the rest API, and apply the incoming messages using the sequence numbers.

The described problem occurs very often, for example if we look at the pair “BTC_ETH”, we see the problem almost twice a minute.

See attached the simple java program PoloniexWSJavampa.java, which can be used to reproduce the issue.
One example output is attached as well.

example.output.txt
PoloniexWSJavampa.java.txt

The big question here is, how is it possible, that we do not see some messages??? Can they be dropped somehow? By server or by jawampa?

[andre77]

@angiolep to excited to wait for your explanation of the problem, we face at Poloniex

[JeremieRodon]

Hi there,

here is the answer of the support team on the issue :
"
This happens sometimes with the WAMP API. It is due to inherent scalability problems with the routing software, and we are working on a pure WebSockets API (currently used by the web interface, but lacking documentation) to replace it. For now, part of the reason the sequence numbers are there is so that you can detect a missing message; if you don't see the message within a second or two, you can reset your order book using the regular polling API.

We hope the new websockets API will be ready soon, we are just working on the finishing touches and the documentation.
"
So, for now, @andre77 solution is the one we have to use :)

Best regards

[borjaj]

Hi there,
I am also working on a project which uses Poloniex orderbook. Multiple messages arrive in single packets so you have to dig down into the packet to get out all the messages. However, it is true that from time to time something gets missed out.
In case you miss the sequence, you have two options.. either you re-initialize the state of your orderbook from Poloniex REST service or, if suitable to your requirements, update the orderbook by last received message even if the sequence num is greater than expected. The first approach is more consistent but slower and there is a problem if an wamp message is received while the Rest init process is in progress so you need to run Rest and Wamp clients at the same time.

[rpike]

Hi there,
I am trying to connect to the API, I received the REST book and I believe I am now reading the push API messages correctly. The issue is that the sequence number I receive on the book from the REST API is substantially ahead of the push API modify and remove messages (sometimes by a couple of hundred thousand). Does anyone have a clue as to whats happening.

[nickbulmer]

rpike, I am also finding the WAMP API market updates are hours behind real time which would correlate to a couple of hundred thousand messages.

[ctubio]

donno why but push api data seems totally outdated

[pvk444]

It's not only outdated, but there is something massively wrong with the sequences. Whereas I can understand if messages come out of sequence at times, I get relatively often jumps of 100'000 or more in the seq numbers. This can't be the expected behavior.

[wranai]

@dhinesh-raja Where do the REST API have sequence numbers? (you do realize btw that only those 1+ billion people who live in India use "lakhs"? not that I'm not used to them by now...)

[avdva]

I'm having the same problem with WAMP, seq numbers are far behind of seqs from an order book received by rest api. Does anyone know rest api request limit for poloniex? How often can I call their api before they ban my IP? Thanks in advance.

[pvk444]

@avdva From the Poloniex API documentation:

Please note that making more than 6 calls per second to the public API, or repeatedly and needlessly
fetching excessive amounts of data, can result in your IP being banned.

[avdva]

@pvk444 Thanks a lot. I wonder, that didn't notice it before.

[wranai]

Random idea: reverse engineering the undocumented websocket API that @JeremieRodon's post mentioned? Poloniex can't afford to let it stop working or they'll stop making money.