Updated SensESP code for saving Magnetic Calibration

[BjarneBitscrambler]

@tedenda and @DanielG86 Just letting you know that I've updated the SensESP orientation code. It now includes the ability to save the current magnetic calibration using the web interface. After saving, you should no longer need to rotate the sensor manually after power-up to get it calibrated: it will automatically load the saved calibration. This is the same feature that was available through the NXP Sensor Fusion Toolbox app, except you won't need to use that app.

Saving the Magnetic Cal

First, get a valid magnetic calibration by manually rotating the sensor in all directions. Once you are satisfied (the sensor should be reporting the expected heading when you point it in a known direction), then access the sensor's web interface:

In this example I am using the Attitude and CompassHeading sensors - if you expand either one you should see a "Values Setting" entry. Click on that and a "Save Magnetic Calibration" button should appear:

Enter the numeral 1 in that value box and click That sends a command via the orientation interface to the fusion code to save the current magnetic calibration in EEPROM. Now whenever the sensor is powered up it will load this calibration.

If the magnetic environment of your sensor changes, you can use the web interface to save a new calibration (once it has been computed through motion of the sensor). The fusion algorithm collects magnetic readings over a 3 minute cycle and checks whether they are consistent with the current calibration. If they are not, then a new calibration is automatically calculated.

I've noticed that if a bad calibration gets saved, it can take several minutes for this re-calculation process to collect enough data and decide to replace the bad calibration. So if you are noticing erroneous headings, go through the random-movements of the sensor to give it more data points to work with.

Getting the Update

Let me know how this works for you. You can find the latest code on the SensESP github, pull request SignalK/SensESP#349 When you update your local code, be aware that I merged in all other SensESP changes from the past approx 1 month, so you may need to do a clean and rebuild to see all the changes. There's also an updated example main.cpp (found in examples/fx8700_heading_example.cpp).

Other Changes

I made two other changes you should know about:

• the Wifi parameters that the sensor connects with are now specified in the constructor sensesp_app = new SensESPApp(...) This saved 2496 bytes of heap (RAM). This is also more convenient than entering them in the web interface, though one can still see the settings in the web interface but they become read-only. To change them there, just revert back to using an empty constructor.
• Additional RAM (about 8k) is saved by adding -D DEBUG_DISABLED to the build-flags = ... in the platformio.ini file. This is not necessary for the ESP32 module, but when using an ESP8266 (e.g. d1_mini) then there isn't enough RAM to run the web interface unless you set this flag at compile time. The disadvantage of using this flag is that you won't see any messages pop up in the serial window, so it's best used once you have finished debugging and know that the executable is running OK. As I said, you don't have to worry about this on the ESP32 as it has more RAM.

I don't know if/when the PR will be accepted by Matti. He's reluctant to make any sensor changes until they figure out how best to isolate SensESP from changes to sensors. In the meantime, you should be able to access this updated code by linking though the PR itself.

[DanielG86]

Great improvement, Bjarne! Just installed in on an ESP32, and it works. Will do some testing to test its behavior.

[DanielG86]

With this progress, new ideas pop up!

1. It could be that a certain offset remains in place, i.e. that the sensor continues to give a 10 degrees offset, even when it is calibrated. To set a fixed offset through the web interface would then be a real bonus.
2. To let this work as a stand-alone unit, I was thinking that a small momentary switch (physical) could also do the trick of saving the calibration. I guess I'm a bit an old-fashioned guy, but having such a physical button on the small box containing the sensor and ESP would be easier (for me at least) to fix the calibration than going through the web interface.

These are just thoughts for further improvements. Already very impressed by the work done and very happy with it.

[DanielG86]

Small comment on the code: in line 113 of the example code (fx8700_heading_example.cpp), a ; is needed at the end of the line.
There was another error during compiling with the temperature code uncommented. Here is the message from the compiler:
Compiling .pio/build/esp-lolin/src/main.cpp.o
src/main.cpp: In lambda function:
src/main.cpp:232:27: error: expected type-specifier before 'Linear'
->connect_to(new Linear(1.0, 0.0, "/sensors/temperature/calibrate"))
^

I don't know how to fix this.

[DanielG86]

Found it: had to add the following line:
#include "transforms/linear.h"

[BjarneBitscrambler]

Sounds like you are zipping along well - great!

I've opened issues to remind me about your excellent suggestions (deviation correction, and switch). Don't feel shy about making more suggestions as you think of them, and if you wish you can open the issues yourself (I believe the Issues tab is usable by any Github user).

[danka74]

Having forked and cloned the SignalK-Orientation repo and successfully uploaded it to a ESP32dev board, I have a question:
Reading through the description of the magnetic calibration algorithm (understanding that I do not fully understand this...) https://www.nxp.com/docs/en/application-note/AN5019.pdf it seems the calibration adjusts for hard- and soft-iron deviation. Does this imply that the calibrated sensor provides "navigation.headingMagnetic" rather than "navigation.headingCompass"?

[BjarneBitscrambler]

Hi Daniel - excellent question. You are right about the magnetic calibration correcting for the hard- and soft-iron deviations. I haven't tested the sensor & algorithm enough yet to confidently say that it will correct for all the deviation under most conditions, but it certainly does have a good linear response through 360 degrees when calibrated (see for example the results on the wiki: https://github.com/BjarneBitscrambler/OrientationSensorFusion-ESP/wiki/Accuracy). That was only one test - I would like to perform more, under conditions with intentionally-introduced hard- and soft-iron. In that single test there seemed to be about a +/- 2 degree residual deviation with an approximately sinusoidal function over the 360 degree arc. If you end up doing any testing, I would love to hear about your results.

As to what the right SK path is: headingMagnetic might be more appropriate. Remember though that when the sensor is installed on a vessel, the north of the sensor likely won't be perfectly aligned with the Bow of the boat, leaving one with a small constant offset. I'm working on a method for manually entering deviation corrections using the sensor's web interface. That would allow one to hopefully adjust for small residual deviations such as due to installation. Once those corrections are applied, headingMagnetic would indeed be the right one to use. (You're probably already aware of this, but the SK path is adjustable by changing the const char* kSKPathHeading = "navigation.headingCompass"; line...)

Which particular ESP32 board are you using? I'll add it to the list of known-compatible hardware if you don't mind.

[danka74]

Thanks Bjarne,

the board I'm using is this https://www.amazon.se/dp/B08CCYWZN3?ref_=pe_20773151_507143461_E_301_dt_1 using esp32dev configuration. I will put this in a sail boat in the spring, and then I can test it live, so far I have only done some eyeballing confirmation indoors and it seems to work as expected (storing and loading calibration e.g.). I cannot run the NXP tool as I have no Windows machine. I changed to headingMagnetic mainly for testing the NMEA2000 output form PiCAN M and it does indeed show up on my Garmin plotter :).

/Daniel

[BjarneBitscrambler]

There are a few new additions to the OrientationSensorFusion-ESP and SignalK-Orientation libraries that should help with magnetic calibration. The Orientation interface now has Get___() methods for several internal variables of the calibration subsystem. These report the

• quality of fit for current and prospective calibrations. These are numbers ranging [0..100], lower is better fit, and a value less than 3.5 is considered a good calibration.
• order of calibration algorithm. This indicates complexity of the calibration algorithm and has possible values of 0 (no calibration), 4, 7, and 10 (most sophisticated).
• Geomagnetic field strength

These help decide whether calibration is good, but do not directly report magnetic interference (I'm still working on that).

If you have a look at the examples/example_main.cpp in the SignalK-Orientation library, it has three new paths reporting these values, and there are comments in that file about what they mean.

I would like eventually to wrap all three up into a single parameter (so we don't need 3 separate instruments to display the values) but haven't thought yet about how to make that work well.

One additional change: in the web interface, if you enter a -1 and Save, then the magnetic calibration is erased from EEPROM. Entering a 1 and Save will save the current calibration in EEPROM, just as before. I haven't updated the online prompt to reflect this new behaviour. Now you can experiment with creating/deleting magnetic calibrations, and how they affect the fit parameters. A reboot of the system is needed to load a saved calibration.

[DanielG86]

Great stuff, Bjarne! Just tried it out and it works really good. Met vriendelijke groet, Daniel Goedhuis 06-52 52 50 62

…

Op 23 jan. 2021, om 19:31 heeft Bjarne Hansen ***@***.***> het volgende geschreven: There are a few new additions to the OrientationSensorFusion-ESP and SignalK-Orientation libraries that should help with magnetic calibration. The Orientation interface now has Get___() methods for several internal variables of the calibration subsystem. These report the quality of fit for current and prospective calibrations. These are numbers ranging [0..100], lower is better fit, and a value less than 3.5 is considered a good calibration. order of calibration algorithm. This indicates complexity of the calibration algorithm and has possible values of 0 (no calibration), 4, 7, and 10 (most sophisticated). Geomagnetic field strength These help decide whether calibration is good, but do not directly report magnetic interference (I'm still working on that). If you have a look at the examples/example_main.cpp in the SignalK-Orientation library, it has three new paths reporting these values, and there are comments in that file about what they mean. <https://user-images.githubusercontent.com/68290965/105610703-dd407080-5d65-11eb-8d0f-07a8e2ef6d7f.png> I would like eventually to wrap all three up into a single parameter (so we don't need 3 separate instruments to display the values) but haven't thought yet about how to make that work well. One additional change: in the web interface, if you enter a -1 and Save, then the magnetic calibration is erased from EEPROM. Entering a 1 and Save will save the current calibration in EEPROM, just as before. I haven't updated the online prompt to reflect this new behaviour. Now you can experiment with creating/deleting magnetic calibrations, and how they affect the fit parameters. A reboot of the system is needed to load a saved calibration. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#22 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADRLPPXYHMIWE66X3PD3CITS3MIXFANCNFSM4WFVWBNA>.

[BjarneBitscrambler]

OK - I believe the last piece of the calibration puzzle is in place... I updated the SignalK-Orientation project examples/example_main.cpp with a short section of code to implement a physical switch to save magnetic calibration. That was the only file needing a change, which makes it pretty straightforward to incorporate (if you wish). Just drop that code section into your own main.cpp, add three SensESP *.h files (for the digital input, debounce, and consumer action functionality), adjust the pin number to whichever one you have a switch hooked up to, and recompile.

How it Works

A digital input pin is read periodically and if its state changes, the next consumer (debounce transform) is notified. I set the switch read interval and debounce period fairly long, as we don't need quick response for this function and this way an accidental quick push on the switch won't cause one to lose an existing calibration. With the settings I used, the switch needs to be held just under 500ms before it takes action.

The debounce transform sends its output to a lambda consumer function, which I defined as a call to the orientation library's InjectCommand() method. That causes the current magnetic calibration to be saved to EEPROM. There's also a debugI() print to the serial stream (unless DEBUG_DISABLED is defined) that indicates the save function has been called. It's a bit hard to pick out from all the other serial traffic, unless one turns the reporting rate for all the other sensors way down...

I chose to wire up a physical button that grounds a GPIO pin when pushed, so the GPIO has its pull-up resistor enabled. There's no reason one couldn't use a pull-down on the GPIO and tie the switch to logic high instead - just adjust the code that sets up the digital input.

Also (forgive me if I'm repeating something you already know) for good noise immunity and CPU protection in a device to be used in the field, I recommend putting a few passive components on the switch circuit. This is particularly a good idea if the wire length between the CPU and the switch is longer than, say, 10 cm or there's a chance of zapping the switch with static electricity. There's quite a few configurations commonly used for input protection - what I'll probably use is a series resistor between the switch and the GPIO pin of about 1000 ohms (the pull-up/pull-down resistance for ESP32 pins is 45 k, so 1 k will overcome that nicely), and then an ESD protection diode rated between 3V3 and 5V0 between the GPIO pin and board ground. An alternative is to use a small-value capacitor instead of the ESD diode - a 100 nF cap would be fine, as the RC time constant together with the 1k resistor will be plenty short. Anyway, just a suggestion...

[BjarneBitscrambler]

I've done some testing on Magnetic Interference, to find a good indicator of when a calibration might not be valid due to changes in the magnetic environment of the sensor. See the wiki page here: https://github.com/BjarneBitscrambler/OrientationSensorFusion-ESP/wiki/Magnetic-Interference for more details.

Consequently I've added a couple functions that report the level of magnetic interference - one of them directly, and the second one that can be used to infer interference when it changes. These are:

• GetMagneticNoiseCovariance() When this parameter rises too high, it suggests magnetic interference
• GetMagneticInclinationDeg() and GetMagneticInclinationRad() The magnetic inclination is usually quite constant for a given location on earth, so by flagging when it changes substantially (say by more than 10 degrees) one can deduce that something in the magnetic environment has changed.

These functions are available right now in the OrientationSensorFusion-ESP library. I'm going to experiment in the SignalK-Orientation library to see whether it makes sense to package these parameters up in a single JSON packet that can be sent to a custom SignalK widget for display. One can of course just display the parameters individually without any further changes.

[BjarneBitscrambler]

It's a bit complex to write a new instrument panel widget to display the combined calibration quality parameters, so I'm leaving them as separate values for now. As there are quite a few, and they won't need to be referred to unless one is calibrating or checking calibration, it probably makes sense to place their widgets on a secondary page of the instrument panel.

If anyone does want to experiment with the combined values, have a look at the examples/example_main.cpp for the MagCalValues producer. In the default Signal K instruments, this output displays as a long-ish string, but I'm sure a clever person could imagine how to package it up as a bar-graph or something...

[BjarneBitscrambler]

Folks - I've updated the example main.cpp demonstrating how to apply an offset correction to the compass reading. It uses the AngleCorrection transform (and optionally the CurveInterpolator transform).

The heading with the corrections applied is sent to the headingMagnetic path, since, as @danka74 pointed out, once the magnetic calibrations are applied and any other deviations corrected, then it truly is a Magnetic Heading.

I've tested it on my ESP8266 and ESP32 devices, and it has been functioning well. Next I'll probably box the PCBs up properly and start doing some more testing.

Enjoy!
Bjarne

[danka74]

Spring is here and it's time to solder and put the orientation sensor in a box. @BjarneBitscrambler do you have any recommendations about the positioning of the sensor board. Can I solder the board to my experiment board or should I keep some distance (cm:s) from the larger board?
Thanks,
Daniel

[BjarneBitscrambler]

Hi Daniel - yes it's nice to have spring arriving! We launched our boat yesterday, after a couple weeks of painting the bottom and doing a bit of other maintenance. I have my eCompass ready to mount onboard, and will probably do that this coming week.

The box I used for the circuitry is here:

The ESP (an 8266 LoLin Mini in this case) is at the top of the photo; power supply in the middle; and the orientation sensor at the bottom.

So you can see there's a few cm between the sensor and the rest of the components. I haven't experimented with closer placement - it might work just fine, but I have tested this present arrangement and it's given good results. I do know from earlier prototyping that proximity to some wires (a USB cable in particular) caused calibration problems, but I don't know whether that's due to steel content in the shielding braid, or current in the conductors. In any case, I fixed the wires inside the box so they would not shift around, and twisted the conductors together, and so far that has worked.

Good luck, and let me know how your installation goes! Cheers,
Bjarne

[danka74]

I tried to put the orientation sensor board on an experiment board in my box, but the calibration seems not good. Powered with 12 V, no USB. Will try some more tomorrow.