Sky Quality Meter using FDRS

[javrolijk]

Hi folks,

After brainstorming for a while, I finally managed to put together a project that I had in my head for some time. Since I'm not a programmer, it was challenging for me to reach a point where I could at least understand how to piece together bits of code and make it work to some extent.

The project involves a remote device that will transmit Sky Quality Meter (SQM) data as light pollution progresses in our local national park and, in general, on our island (Aruba). The data can eventually aid our local authorities statistically in taking proper action. As you can imagine, there is no internet connection in the local park, so the main idea is to use the FDRS platform and LoRa protocol to transmit data to a gateway located approximately 1-2 km away. The components used at the node side are a TSL2591 Light Sensor and a BME280. I've managed to merge codes from the following repositories:

• For energy conservation: https://github.com/nglessner/Auxin
• For the SQM data calculations: https://github.com/beckrd/DIY_SQM
• And, of course, the FDRS platform.

The main concept is illustrated below:

To test the code, I've made things a little simpler. The combined code is currently transmitting data from the sensor node to the first gateway via ESPNOW and then through UART to the front end with MQTT.

The main issue I'm facing is with the new SQM data type that is being loaded into the FDRS platform. I'm not sure, but the issue might be related to the JSON structure not being allowed through MQTT. Here's the string of data being received by the ESPNOW gateway from the sensor node:

16:27:05.658 -> Incoming ESP-NOW.
16:27:05.658 -> Sending Serial.
16:27:05.658 -> [{"id":1,"type":16,"data":0.284000009},{"id":1,"type":34,"data":7.646743774},{"id":1,"type":3,"data":40.99707031},{"id":1,"type":1,"data":25.34000015},{"id":1,"type":4,"data":1008.615479}]

Here's the error output from the MQTT gateway:

16:27:05.658 -> Incoming Serial.
16:27:05.658 -> Sending MQTT.
16:27:05.658 ->  Error on sending MQTT

I've tried to modify the file 'fdrs_datatypes.h' for the new SQM value, but not sure if that's the only file needing to be modified.
Any help is much appreciated.

As a reference, here's the combined code in the sensor node:

// setup for FDRS and BME
#include "fdrs_node_config.h"
#include <fdrs_node.h>
#include "esp_adc_cal.h"

// setup for BME280
#include <Adafruit_BME280.h>

// setup for TSL2591
#include <Wire.h>
#include <Adafruit_Sensor.h>
#include "SQM_TSL2591.h"

#define LOW_BATTERY_VOLTAGE 3.20
#define VERY_LOW_BATTERY_VOLTAGE 3.10
#define CRITICALLY_LOW_BATTERY_VOLTAGE 3.00

#define LUX_SENSITIVITY 10 //15     // the minimum LUX change we transmit, skip otherwise, 0 = always
#define POLLING_DURATION 60       // how often we check light intensity in seconds
#define USE_STATUS_LEDS false       // use LED to indicate awake processing
#define MINIMUM_TRANSMIT_CYCLES 2 //30  // transmit at least every x cycles

Adafruit_BME280 bme;
SQM_TSL2591 sqm = SQM_TSL2591(2591);
void readSQM(void);

// cached values - lifetime reset only by power cycle
RTC_NOINIT_ATTR struct {
  uint64_t NumberOfRestarts;     // number of restarts
  uint64_t LastTransmitRestart;  // the restart count during the last transmit
  int LastReportedLux;           // keep track of the last reported light value
} cache;

// Pinout setup
// v+ to TSL2591
uint16_t SENSOR_POWER_PIN = 12;
uint16_t STATUS_LED_PIN = 2;

void setup() {
  // stuff to do every run
  // setup pinouts and freq
  Serial.begin(115200);
  setCpuFrequencyMhz(80);
  pinMode(SENSOR_POWER_PIN, OUTPUT);
  if (USE_STATUS_LEDS) {
    pinMode(STATUS_LED_PIN, OUTPUT);
  }

  Serial.print("===================================================\r\n");
  
  if (esp_reset_reason() == ESP_RST_POWERON) {
    // stuff to do only once
    Serial.printf("ESP was just switched ON\r\n");
    cache.NumberOfRestarts = 0;
    cache.LastReportedLux = 0;
    cache.LastTransmitRestart = 0;
    sleepFDRS(1);
  } else {
    if (USE_STATUS_LEDS) {
      digitalWrite(STATUS_LED_PIN, HIGH);
    }
    cache.NumberOfRestarts++;
    Serial.printf("Number of Restarts: %d\r\n", cache.NumberOfRestarts);

    ReportLux();
  }

  if (sqm.begin()) {
    Serial.println("Found SQM (TSL) sensor");
    sqm.config.gain = TSL2591_GAIN_LOW;
    sqm.config.time = TSL2591_INTEGRATIONTIME_200MS;
    sqm.configSensor();
    sqm.showConfig();
    sqm.setCalibrationOffset(0.0);
  } else{
    Serial.println("SQM sensor not found");
 }

  // turn off activity LED and go to deep sleep
  if (USE_STATUS_LEDS) {
    digitalWrite(STATUS_LED_PIN, LOW);
  }
  Serial.printf("=== entering deep sleep after %d ms ===\r\n", millis());
  sleepFDRS(POLLING_DURATION);  // Goodnight

  Serial.println("This should never get printed");
}

void loop() {
  //sqm.takeReading(); 

/*
//write to serial monitor for debugging purposes
  Serial.print("full:   "); Serial.println(sqm.full);
  Serial.print("ir:     "); Serial.println(sqm.ir);
  Serial.print("vis:    "); Serial.println(sqm.vis);
  Serial.print("mpsas:  "); Serial.print(sqm.mpsas);
  Serial.print(" +/- "); Serial.println(sqm.dmpsas);
*/
  //Serial.println("======================================");

  //display.clearDisplay();
  //display.setTextColor(WHITE);
  //display.setTextSize(1);
  //display.setCursor(0,0);
  //display.print("   Mag/Arc-Sec");
  //display.setTextSize(3);
  //display.setCursor(0, 24);
  //display.print(" ");
  //display.print(sqm.mpsas);
  //display.setTextSize(1);
  //display.setCursor(0,48);
  //display.print("  ");
  //display.setCursor(0,56);
  //display.print("   +/- ");
  //display.print(sqm.dmpsas);
  //display.display();

//  delay(2000);
//}
  Serial.println("This loop message should never get printed");
}
/******************************************************************************
  Description.: reads the battery voltage through the voltage divider at GPIO34
                if the ESP32-E has calibration eFused those will be used.
                In comparison with a regular voltmeter, the values of ESP32 and
                multimeter differ only about 0.05V
  Return Value: battery voltage in volts
******************************************************************************/
float readBattery() {
  uint32_t value = 0;
  int rounds = 11;
  esp_adc_cal_characteristics_t adc_chars;

  // battery voltage divided by 2 can be measured at GPIO34, which equals ADC1_CHANNEL6
  adc1_config_width(ADC_WIDTH_BIT_12);
  adc1_config_channel_atten(ADC1_CHANNEL_6, ADC_ATTEN_DB_11);
  esp_adc_cal_characterize(ADC_UNIT_1, ADC_ATTEN_DB_11, ADC_WIDTH_BIT_12, 1100, &adc_chars);

  // to avoid noise, sample the pin several times and average the result
  for (int i = 1; i <= rounds; i++) {
    value += adc1_get_raw(ADC1_CHANNEL_6);
  }
  value /= (uint32_t)rounds;

  // due to the voltage divider (1M+1M), values must be multiplied by 2
  // and convert mV to V
  return esp_adc_cal_raw_to_voltage(value, &adc_chars) * 2.0 / 1000.0;
}

/******************************************************************************
  Description.: send message
******************************************************************************/
bool ReportLux() {
  // give the sensor 8ms to wake up
  digitalWrite(SENSOR_POWER_PIN, HIGH);
  delay(8);
  sqm.takeReading(); 
  //int sqmvis = sqm.vis;
  digitalWrite(SENSOR_POWER_PIN, LOW);

  int diff = abs(cache.LastReportedLux - sqm.vis);
  Serial.printf("Cached: %d ", cache.LastReportedLux);
  Serial.printf("Full: %d ", sqm.vis);
  Serial.printf("Diff: %d\r\n", diff);

  // transmit only if one of the following is true
  // 1) light intensity has changed more than the LUX_SENSITIVITY threshold
  // 2) light intensity is now zero and wasn't previously 0
  // 3) it's been X cycles since we last transmitted
  if (diff >= LUX_SENSITIVITY || (cache.LastReportedLux != 0 && sqm.vis == 0) || cache.LastTransmitRestart + MINIMUM_TRANSMIT_CYCLES <= cache.NumberOfRestarts) {
    Serial.println(F(" --- New Light Value ---  "));
    cache.LastReportedLux = sqm.vis;

    beginFDRS();
    bme.begin(0x76);  // BME280 address may vary, use the correct address if different

    float h = bme.readHumidity();
    float t = bme.readTemperature();
    float p = bme.readPressure() / 100.0;  // Convert pressure to hPa
    float b = readBattery();
    float s = sqm.mpsas;

    Serial.printf("Temp: (%f degC)\r\n", t);
    Serial.printf("Humidity: (%f RH)\r\n", h);
    Serial.printf("Pressure: (%f hPa)\r\n", p);
    Serial.printf("Battery Voltage: (%f V)\r\n", b);
    Serial.printf("Skyglow: (%f mag/arcsec^2)\r\n", s);
    Serial.print("vis:    "); Serial.println(sqm.vis);

    loadFDRS(b, VOLTAGE_T);
    //loadFDRS(sqm.vis, LIGHT_T);
    loadFDRS(s, SKYGLOW_T);

    if (isnan(h) || isnan(t) || isnan(p)) {
      DBG("Failed to read from BME280 sensor!");
    } else {
      loadFDRS(h, HUMIDITY_T);
      loadFDRS(t, TEMP_T);
      loadFDRS(p, PRESSURE_T);
    }
    sendFDRS();
    cache.LastTransmitRestart = cache.NumberOfRestarts;
  } else {
    Serial.println(F(" --- No Change ---  "));
  }

  return false;
}

[aviateur17]

The main issue I'm facing is with the new SQM data type that is being loaded into the FDRS platform. I'm not sure, but the issue might be related to the JSON structure not being allowed through MQTT. Here's the string of data being received by the ESPNOW gateway from the sensor node:

16:27:05.658 -> Incoming ESP-NOW.
16:27:05.658 -> Sending Serial.
16:27:05.658 -> [{"id":1,"type":16,"data":0.284000009},{"id":1,"type":34,"data":7.646743774},{"id":1,"type":3,"data":40.99707031},{"id":1,"type":1,"data":25.34000015},{"id":1,"type":4,"data":1008.615479}]

Here's the error output from the MQTT gateway:

16:27:05.658 -> Incoming Serial.
16:27:05.658 -> Sending MQTT.
16:27:05.658 ->  Error on sending MQTT

I've seen the same error. For me it was due to the default MQTT packet size limit. I believe 256 bytes by default. Can be changed by a #define or the setBufferSize() function.

Was having issues with MQTT messages not sending - just generating the error. Figured out that the Arduino PubSubClient default size is 256 bytes and need to increase the default buffer by using setBufferSize.

https://pubsubclient.knolleary.net/api#setBufferSize

#142 (comment)

By default, it is set to 256 bytes - as defined by the MQTT_MAX_MESSAGE_SIZE constant in PubSubClient.h.

Should be able to use client.setBufferSize(1024); and see if that works.

fdrs_gateway_mqtt.h in the begin_mqtt() function:

#define MQTT_MAX_BUFF_SIZE 1024

void begin_mqtt()
{
    client.setServer(mqtt_server, mqtt_port);
    client.setBufferSize(MQTT_MAX_BUFF_SIZE);
    if (!client.connected())
    {
        reconnect_mqtt(5);
    }
    client.setCallback(mqtt_callback);
}

[javrolijk]

Thanks for the prompt response.
Seems like the 'begin_mqtt()' function is different from what is posted. Not sure if there was some code change afterward in fdrs_gateway_mqtt.h. Here's what the 'begin_mqtt()' function looks like, followed by 'void mqtt_publish(const char *payload)' and 'void sendMQTT()'.

void begin_mqtt()
{
    client.setServer(mqtt_server, mqtt_port);
    if (!client.connected())
    {
        reconnect_mqtt(5);
    }
    client.setCallback(mqtt_callback);
}

void mqtt_publish(const char *payload)
{
    if (!client.publish(TOPIC_DATA, payload))
    {
        DBG(" Error on sending MQTT");
#if defined(USE_SD_LOG) || defined(USE_FS_LOG)
        sendLog();
#endif
    }
    else
    {
#if defined(USE_SD_LOG) || defined(USE_FS_LOG)
        if (last_log_write >= last_mqtt_success)
        {
            releaseLogBuffer();
            resendLog();
        }
        time(&last_mqtt_success);
#endif
    }
}

void sendMQTT()
{
    DBG("Sending MQTT.");
    DynamicJsonDocument doc(24576);
    for (int i = 0; i < ln; i++)
    {
        doc[i]["id"] = theData[i].id;
        doc[i]["type"] = theData[i].t;
        doc[i]["data"] = theData[i].d;
        doc[i]["time"] = time(nullptr);
    }
    String outgoingString;
    serializeJson(doc, outgoingString);
    mqtt_publish((char *)outgoingString.c_str());

[aviateur17]

In the code you posted, insert:
client.setBufferSize(1024);
On the next line, after the line that starts
client.setServer(mqtt_server, mqtt_port);

This will increase the MQTT buffer size from the default of 256 to 1024 bytes.

[javrolijk]

Got it...
It is working now. Awesome!
Much appreciated.

22:06:27.472 -> Incoming Serial.
22:06:27.472 -> Sending MQTT.

[timmbogner]

This is awesome and a great cause... thanks Jairo!
I like that that you built off of @nglessner's work. I hope you'll share your project around once it's up and running!

SKYGLOW_T will be coolest-sounding data type, although I understand it is not a good thing! You should be able to just add it to the fdrs_datatypes.h file and it will be good-to-go.

Stay in touch!