Interpreting ultrasonic readings (and data cleanup)

[XenoXilus]

The ultrasonic distance sensor used in the WaterElf project produces "dirty" readings - while it can produce usable readings, the readings are also interspersed with zeroes and nonsensical numbers.

I can look at cleaning up the readings that come from the ultrasonic sensor (discarding nonsense and zeroes). I can also produce some functions to interpret the data from other sensors (mean, standard deviation... trend? - "Oh no, water level changed more quickly than usual, send a text")

[layerzerolabs]

This is a good start! For the benefit of people who are reading this who don't know the background, can you add a sentence at the beginning describing the problem - i.e. why the need to clean up the ultrasonic readings? Also please can you add the label 'waterelf32' to this issue and assign the issue to yourself - ta!
Also you'll see that issue #13 is related - so keep an eye on that issue as well in case those developments affect your work.

[hamishcunningham]

I seem to remember that we had a similar issue (perhaps with the ultrasonics, or perhaps other sensors?) and added some hacky code to do averaging, or discard outliers, or somesuch. It could be good to find that code and review it?

[andrisole92]

Yeah, it would be good to see it.

[layerzerolabs]

The only code I can find in the waterelf device was a delay of 35ms between different ultrasonic sensors - I think there was some filtering implemented using Node-RED - which isn't easily ported! Maybe I missed it though... you could look through the various iterations of waterelf code?

[hamishcunningham]

Yes, there's the 35 mS delay in getLevel, and then a couple of tiny delays in there too that I think are the trigger/measure cycle? Getting those right, and not having interrupts go off during them, probably impacts accuracy?

The only other thing I can see is that we always yield between reads:
getLevel(LEVEL_ECHO_PIN1, &now->waterLevel1); yield();
getLevel(LEVEL_ECHO_PIN2, &now->waterLevel2); yield();
getLevel(LEVEL_ECHO_PIN3, &now->waterLevel3); yield();
I'm not sure of the semantics of this on ESP32 but it could be worth a look...

[XenoXilus]

Tinkered with it today, I didn't change much except for restructuring things into an array and increasing the number of readings it took (to get more reliable mean/outlier calculations), I managed to get accurate readings up to about 80-90cm as long as surfaces weren't moving around too much. The datasheet for the ultrasonic sensor does say that it prefers level surfaces, which I imagine might be tricky to get on the surface of the water... maybe some mesh baffles near the surface by the water inlet/outlet could help? Outside of the scope of software, though.

[hamishcunningham]

Cool. We generally site them looking down a 4 inch pipe onto the water surface, which is only filling fairly slowly, so not too turbulent. But generally adding some outlier detection and discarding is a good idea for all the sensors in the system methinks...

[layerzerolabs]

True - even the more stable sensors occasionally have a bad day and throw out a 0, a -127, 65536 or some other value that's clearly absurd. If you just take the raw data and make a graph, for example, you can get the very occasional value forcing the graph to scale to accommodate and making the data unreadable! So I agree with Hamish; the data cleanup routines developed for this particularly noisy sensor would also be useful in other places also.