Failure to take-off bug (potentially related to no mavros status)

[PrimaryComet]

Hello!

I'm using this system to test a trajectory planner, and I currently restart the entire simulator after every test. However, after a few tests (usually 5-10), the mrs setup fails to start. My control script doesn't progress past "waiting for takeoff". The hardware api tab shows:

[ WARN] [1736177434.213920259, 3.320000000]: [MrsUavPx4Api]: Not recieving Mavros state message for '0.072 s'! Setup the PixHawk SD card!!

[ WARN] [1736177434.213967866, 3.320000000]: [MrsUavPx4Api]: This could be also caused by the not being PixHawk booted properly due to, e.g., antispark connector jerkyness.

[ WARN] [1736177434.213996403, 3.320000000]: [MrsUavPx4Api]: The Mavros state should be supplied at 100 Hz to provided fast refresh rate on the state of the OFFBOARD mode. 

[ WARN] [1736177434.214004597, 3.320000000]: [MrsUavPx4Api]: If missing, the UAV could be disarmed by safety routines while not knowing it has switched to the MANUAL mode.  

.. followed by repeating Not receiving Mavros state message until I kill it. I have checked, and the mavros state topic has no messages being published to it.

The 'core' tab shows this on a loop:

[ WARN] [1736177744.353512782, 313.268000000]: [EstimationManager/gps_garmin/alt_garmin/pos_garmin/excessive_tilt]: Did not receive any message from topic '/uav1/hw_api/orientation' for 304.000000s (1 publishers on this topic)

[ WARN] [1736177744.357060119, 313.272000000]: [EstimationManager/gps_garmin/hdg_hw_api]: has not received orientation yet

[ WARN] [1736177744.357122815, 313.272000000]: [EstimationManager/gps_garmin/hdg_hw_api]: has not received angular velocity yet

[ WARN] [1736177744.364938760, 313.280000000]: [EstimationManager/gps_garmin]: has not received orientation on topic /uav1/hw_api/orientation yet

[ INFO] [1736177744.365277865, 313.280000000]: [EstimationManager/gps_garmin]: WAITING FOR: msg on topic /uav1/hw_api/orientation

[ INFO] [1736177744.368633442, 313.284000000]: [EstimationManager/gps_garmin/lat_gps]: WAITING FOR: correction gps_garmin/lat_gps/pos_hw_api

[ INFO] [1736177744.372980944, 313.288000000]: [EstimationManager/gps_garmin/alt_garmin]: WAITING FOR: correction gps_garmin/alt_garmin/pos_garmin

[ WARN] [1736177744.409329863, 313.324000000]: [EstimationManager/garmin_agl/garmin_agl]: Estimator must be in READY_STATE to start it

[ WARN] [1736177744.421351085, 313.336000000]: [EstimationManager]: not publishing uav state in INITIALIZED_STATE

[ WARN] [1736177744.508951259, 313.424000000]: [EstimationManager]: not publishing uav state in INITIALIZED_STATE

[ERROR] [1736177744.652948634, 313.568000000]: [TransformManager/ground_truth]: Could not transform pose to uav1/local_origin. Not republishing odom in this frame.

[ WARN] [1736177744.756556351, 313.672000000]: [ControlManager]: Transformer: Exception caught while looking up transform from "uav1/world_origin" to "uav1/local_origin": "uav1/local_origin" passed to lookupTransform argument target_frame does not exist. 

Most other tabs are waiting for /uav1/gain_manager/diagnostics.

When I try running one of the default tmux sessions after this error has occurred using my custom session, I get the same error. I can see the drone has spawned in Gazebo, but it does not take off. The only solution I have found to fix this issue is to reboot the OS (Ubuntu 20.04), however it inevitably occurs again after a few restarts of the simulation. Could you offer some advice on tracking this down? I'm happy to upload any logs but I'm not sure where to start since there are so many things running.

While I'm here, is there a script or recommended way to completely reset the simulation to its default start-up position without killing everything and restarting it using start.sh?

Thank you very much!

[petrlmat]

Hi, how do you restart the simulation? Do you kill the whole tmux session? I suspect there is a PX4 process still running. Can you check in e.g. htop if everything was killed correctly?

[PrimaryComet]

Hi, I use a modified version of your start.sh script to restart everything. First all the windows of the tmux session are killed individually (in the session.yml), along with a rosnode kill -a and killgazebo. Then I call tmux detach -E "./restart.sh", where restart.sh is nearly identical to start.sh, with the addition of a tmux -L $TMUX_SOCKET_NAME kill-session -t $TMUX_SESSION_NAME before the tmuxinator start command.

I checked with htop and searches for ros and PX4 returned nothing related to the simulation.