Maintaining application message boundaries

[marygilliland]

Are there some configuration settings or ways to use the NORM API that make the API work the same as the linux recvfrom call for a connectionless socket? For instance, when recvfrom is called with a length of 512 and the datagram received has a payload with length of 640, the recvfrom call returns the first 512 bytes of payload from that datagram and the remaining 128 bytes of payload are discarded; the next call to recvfrom will return data from the next datagram received. Also, when the socket has received datagrams with payload lengths 128 bytes, 512 bytes, 128 bytes, 512 bytes, a call to recvfrom with a length value of 512 returns the 128 byte payload, and a second call to recvfrom with a length value of 512 returns the 512 byte payload from the next datagram, and so on. The datagram boundaries are maintained.

What I’m finding with the way I’m using the NORM API is that the payloads of sequential datagrams seem to have been combined together and the “boundaries” of the separate datagrams are lost. I am using NORM_OBJECT_STREAM mode. For the sender, I am calling NormStreamFlush with the EOM flag set to true after writing all of the data for one application message, and I have set the segment size large enough to contain the largest possible application message plus the NORM header. When my sending application sends an application message of 128 bytes followed by an application message of 512 bytes, and the receiving application makes a call to NormStreamRead with a length value of 512, rather than getting back only 128 bytes from the first message sent, I get back 512 bytes consisting of 128 bytes from the first message and 384 bytes from the second message. I would like to have the NORM API behave the same as calling recvfrom. Is it possible?

[bebopagogo]

It could be possible to do what you ask since the message boundary information as marked by the sender application is within the NORM stream state at the receiver. To be honest, I had not actually thought of this approach to using the NORM message stream as I typically embed a message length header into the stream data for my example applications and then read the message length header bytes to then determine how many bytes to read to fetch the message content. Your suggestion is probably a good one for some use cases where you don't want to incur the overhead of an application message header. One difference to consider with NORM and a datagram protocol like UDP is that a NORM receiver may receive a portion of message with the remainder to be received subsequently when application messages are segmented over multiple NORM_DATA packet payloads. This can occur when the application messages are larger than the NORM segment size or how the sender application writes to the NORM stream. A sender application can take care to make sure its message fit within single NORM_DATA payloads (less than or equal to the configured segment size) and what you request here would be easy to implement. It would take a little more work within a NORM API call to do what you ask to determine if an entire message is held within the receiver buffers for the cases when messages are segmented across multiple NORM_DATA payloads. The state is there to support that as the underlying code beneath the API call could "peek" ahead to see if contiguous portions of the stream are available up through the marked end-of-message. I will have to think about this a little bit and potentially implement this as an option, probably as something like a NormReceiveMessage() call or something like that. It would require some update to the underlying NormStreamObj::Read() call which is part of the code that is a little messier than I would like it to be. I think this is a good request and I will try to make some time to work on it.

…

On Tue, Mar 15, 2022 at 1:06 PM marygilliland ***@***.***> wrote: Are there some configuration settings or ways to use the NORM API that make the API work the same as the linux recvfrom call for a connectionless socket? For instance, when recvfrom is called with a length of 512 and the datagram received has a payload with length of 640, the recvfrom call returns the first 512 bytes of payload from that datagram and the remaining 128 bytes of payload are discarded; the next call to recvfrom will return data from the next datagram received. Also, when the socket has received datagrams with payload lengths 128 bytes, 512 bytes, 128 bytes, 512 bytes, a call to recvfrom with a length value of 512 returns the 128 byte payload, and a second call to recvfrom with a length value of 512 returns the 512 byte payload from the next datagram, and so on. The datagram boundaries are maintained. What I’m finding with the way I’m using the NORM API is that the payloads of sequential datagrams seem to have been combined together and the “boundaries” of the separate datagrams are lost. I am using NORM_OBJECT_STREAM mode. For the sender, I am calling NormStreamFlush with the EOM flag set to true after writing all of the data for one application message, and I have set the segment size large enough to contain the largest possible application message plus the NORM header. When my sending application sends an application message of 128 bytes followed by an application message of 512 bytes, and the receiving application makes a call to NormStreamRead with a length value of 512, rather than getting back only 128 bytes from the first message sent, I get back 512 bytes consisting of 128 bytes from the first message and 384 bytes from the second message. I would like to have the NORM API behave the same as calling recvfrom. Is it possible? — Reply to this email directly, view it on GitHub <#51>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABU46GN25BQBMGKGEAQAYJTVAC7Q7ANCNFSM5QZKPYZA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- Brian Adamson ***@***.***>

[marygilliland]

I was thinking that if I have to use the current NORM API as is, and try to emulate the behavior of recvfrom that my applications are built around, then I would need to use that message length header model from the NORM examples, and create a sort of layer between my apps and the NORM API to add that header on the send side, then use it on receive side and strip it off before returning the real message to the main part of the application.

But it would be great for our use case if the NORM API more directly provided a behavior equivalent to recvfrom. In general, I think an interface like that could make the NORM API more of a plug-and-play addition to existing applications. Thanks for considering the modification.