Generic receiver #554

John-Wiens · 2024-06-28T15:47:40Z

This PR adds a Generic UDP receiver that can receive the following message types: MAP, SPaT, BSM, TIM, SRM, SSM, TIM.

The generic receiver allows the ODE to receive multiple different message types using a single UDP port. A new UDP port allocation 44990 has been created to act as the generic receiver port.

Generic Message decoding is done by first identifying the message type, than using existing logic to decode the message and send it to the relevant decoder topic. This identification strategy has been in use by Maricopa County since November 2023. This PR formalizes the strategy developed for Maricopa County and integrates it with the new receiver model developed in March and May 2024.

Summary of Changes

The docker-compose file has been updated to open port 44990
The SVCS Ode Properties file has been updated to include buffer size and port information for the new receiver
A new Generic Receiver has been created similar to existing receivers for other message types
UperUtils has been updated to update the strategy for determine message types
Additional Message Types have been added to the available message types in UperUtils
Logic to build Json from a Hex packet has been removed from individual receiver classes and consolidated in a single UdpHexDecoder class. The purpose of this change is to allow the generic receiver and existing receivers to call the same code for decoding a given message type. This will remove the need to maintain the logic in multiple locations
A new test UDP sender has been added which sends all of the different message types to the generic receiver endpoint for testing.

Related Issue

Related Issue: #553

Motivation and Context

The purpose of this update is to allow support for RSU units that are able to forward messages via UDP, but unable to forward different messages to different ports.

How Has This Been Tested?

I have created a new python udp sender script (udpsender_generic.py) which sends the following message types to port 44990: MAP, SPaT, BSM, TIM, SRM, SSM, TIM. For each message type, I monitored the corresponding decoded message topic (OdeMapJson, OdeSpatJson, etc) to ensure messages were decoded properly on the output topic. The generic receiver test script also sends a bad invalid message to the receiver to ensure the system handles invalid messages safely.

As this change refactors the existing udp receivers, I have also validated that the existing udpsender scripts function as expected and create decoded output on the corresponding decoder topics.

Types of changes

Defect fix (non-breaking change that fixes an issue)
New feature (non-breaking change that adds functionality)
Breaking change (fix or feature that cause existing functionality to change)

Checklist:

I have added any new packages to the sonar-scanner.properties file
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
ODE Contributing Guide
I have added tests to cover my changes.
All new and existing tests passed.

jpo-ode-svcs/src/main/java/us/dot/its/jpo/ode/udp/UdpHexDecoder.java

mwodahl · 2024-07-24T18:07:22Z

docker-compose.yml

@@ -48,6 +48,7 @@ services:
      - "44920:44920/udp"
      - "44930:44930/udp"
      - "44940:44940/udp"
+      - "44990:44990/udp"


Does this port need to be exposed on any of the other docker-compose files?

It looks like this port should be opened in the confluent version of the docker-compose as well. Fixed in the latest update.

mwodahl

This looks good and worked well for me when run locally - I just had two small questions.

mwodahl

This looks good to me!

dmccoystephenson

This looks good to me!

dmccoystephenson · 2024-07-26T15:25:18Z

jpo-ode-svcs/src/main/java/us/dot/its/jpo/ode/udp/generic/GenericReceiver.java

+            // packet should be recreated on each loop to prevent latent data in buffer
+            DatagramPacket packet = new DatagramPacket(buffer, buffer.length);
+            try {
+                logger.debug("Waiting for Generic UDP packets...");


The UDP packets being waited for here are not generic themselves, so this debug message might be a little misleading.

dan-du-car · 2024-09-19T19:40:29Z

jpo-ode-svcs/src/main/java/us/dot/its/jpo/ode/coder/stream/LogFileToAsn1CodecPublisher.java

 				} else if (messageType == "TIM") {
 					publisher.publish(JsonUtils.toJson(odeData, false),
 						publisher.getOdeProperties().getKafkaTopicOdeRawEncodedTIMJson());
+				} else if (messageType == "BSM") {
+						publisher.publish(JsonUtils.toJson(odeData, false),
+							publisher.getOdeProperties().getKafkaTopicOdeBsmJson());


Do we use raw plain BSM json topic rather than raw encoded BSM JSON? I notice other message types are using encoded json topics?

Good Catch - Fixed!

dan-du-car · 2024-09-19T19:47:57Z

jpo-ode-svcs/src/main/java/us/dot/its/jpo/ode/udp/UdpHexDecoder.java

+
+public class UdpHexDecoder {
+
+    private static Logger logger = LoggerFactory.getLogger(AbstractUdpReceiverPublisher.class);


This logger class shall be UdpHexDecoder.class?

jpo-ode-svcs/src/main/java/us/dot/its/jpo/ode/udp/UdpHexDecoder.java

dan-du-car · 2024-09-19T20:20:10Z

jpo-ode-svcs/src/main/java/us/dot/its/jpo/ode/udp/UdpHexDecoder.java

+            return null;
+        OdePsmMetadata psmMetadata = new OdePsmMetadata(psmPayload);
+        // Add header data for the decoding process
+        ZonedDateTime utc = ZonedDateTime.now(ZoneOffset.UTC);


We could probably use a method for creating this timestamp if duplicated in multiple places.

Sounds good to me. I added a static method at the bottom that is shared between all the sources for this.

dan-du-car · 2024-09-19T20:22:12Z

jpo-ode-svcs/src/main/java/us/dot/its/jpo/ode/udp/bsm/BsmReceiver.java

+               String bsmJson = UdpHexDecoder.buildJsonBsmFromPacket(packet);
+
+               if(bsmJson != null){
+                  // Submit JSON to the OdeRawEncodedMessageJson Kafka Topic


The comment shall be OdeRawEncodedBSMJson?

jpo-ode-svcs/src/main/java/us/dot/its/jpo/ode/udp/map/MapReceiver.java

jpo-ode-svcs/src/main/java/us/dot/its/jpo/ode/udp/psm/PsmReceiver.java

jpo-ode-svcs/src/main/java/us/dot/its/jpo/ode/udp/spat/SpatReceiver.java

jpo-ode-svcs/src/main/java/us/dot/its/jpo/ode/udp/ssm/SsmReceiver.java

jpo-ode-svcs/src/main/java/us/dot/its/jpo/ode/udp/tim/TimReceiver.java

dan-du-car · 2024-09-24T15:28:51Z

@John-Wiens Is it possible to resolve the conflicts?

…ic-receiver

John-Wiens · 2024-09-24T22:33:29Z

Fixed Merge Conflicts

John-Wiens added 4 commits June 27, 2024 17:13

Adding Endpoint for Receiving All message Types

c415b8f

Refactored Static methods into a single static class

cba44e2

Fixed bug in generic receiver script

0ceaa6b

Removing Unused imports from BSM Receiver

478c836

mwodahl reviewed Jul 24, 2024

View reviewed changes

jpo-ode-svcs/src/main/java/us/dot/its/jpo/ode/udp/UdpHexDecoder.java Outdated Show resolved Hide resolved

mwodahl reviewed Jul 24, 2024

View reviewed changes

John-Wiens added 2 commits July 24, 2024 17:17

Changed Tim Variable Name

f6cf582

Added Port definition to confluent cloud docker-compose

aac46af

mwodahl approved these changes Jul 25, 2024

View reviewed changes

dmccoystephenson self-requested a review July 25, 2024 14:53

dmccoystephenson approved these changes Jul 26, 2024

View reviewed changes

Fixed bug with decoding message headers

7b76880

dan-du-car reviewed Sep 19, 2024

View reviewed changes

Updating to make receivers more conformal.

bc1397b

dan-du-car approved these changes Sep 24, 2024

View reviewed changes

John-Wiens added 2 commits September 24, 2024 15:56

Merge branch 'develop' of github.com:usdot-jpo-ode/jpo-ode into gener…

52b2f0a

…ic-receiver

Change allowed number of skip bits

89b819c

dan-du-car merged commit 2222b23 into usdot-jpo-ode:develop Sep 25, 2024
2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generic receiver #554

Generic receiver #554

John-Wiens commented Jun 28, 2024 •

edited

Loading

mwodahl Jul 24, 2024

John-Wiens Jul 24, 2024

mwodahl left a comment

mwodahl left a comment

dmccoystephenson left a comment

dmccoystephenson Jul 26, 2024

dan-du-car Sep 19, 2024

John-Wiens Sep 20, 2024

dan-du-car Sep 19, 2024

John-Wiens Sep 20, 2024

dan-du-car Sep 19, 2024

John-Wiens Sep 20, 2024

dan-du-car Sep 19, 2024

John-Wiens Sep 20, 2024

dan-du-car commented Sep 24, 2024

John-Wiens commented Sep 24, 2024


		public class UdpHexDecoder {

		private static Logger logger = LoggerFactory.getLogger(AbstractUdpReceiverPublisher.class);

Generic receiver #554

Generic receiver #554

Conversation

John-Wiens commented Jun 28, 2024 • edited Loading

Related Issue

Motivation and Context

How Has This Been Tested?

Types of changes

Checklist:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mwodahl left a comment

Choose a reason for hiding this comment

mwodahl left a comment

Choose a reason for hiding this comment

dmccoystephenson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dan-du-car commented Sep 24, 2024

John-Wiens commented Sep 24, 2024

John-Wiens commented Jun 28, 2024 •

edited

Loading