Skip to content

Added RuntimeError handling to main loop, if caught i2c bus is reinitialized,...

Brady Conlon requested to merge ie_mux_crashfix into dev/inspired_energy_mux

Encountering a runtime error after querying the battery state for an intermittent amount of time(sometimes as little as 3 or up to 30 min). It appears to be a hardware issue (possibly due to the MCP2221) where part of the circuit suffers a Brownout, however further investigation is needed.

RuntimeError [INFO] [ 08/23/23 02:39:45 PM] [/InspiredEnergyMuxNode]: Inspired Energy Mux Exiting... Traceback (most recent call last): File "/home/b/ros_cpws/src/ros_circuitpython/src/inspired_energy_mux_driver.py", line 303, in node.run() File "/home/b/ros_cpws/src/ros_circuitpython/src/inspired_energy_mux_driver.py", line 256, in run bus_info_list = system.get_status() File "/home/b/ros_cpws/src/ros_circuitpython/src/inspired_energy_mux_driver.py", line 203, in get_status status.append(batt.status(update=update)) File "/home/b/ros_cpws/src/ros_circuitpython/src/inspired_energy_mux_driver.py", line 116, in status self.update_status() File "/home/b/ros_cpws/src/ros_circuitpython/src/inspired_energy_mux_driver.py", line 82, in update_status self.battery_status = self.read_register(address=InspiredEnergyBatteryNH2054HD34.BATTERY_STATUS_CMD, return_raw=True) File "/home/b/ros_cpws/src/ros_circuitpython/src/inspired_energy_mux_driver.py", line 149, in read_register device = I2CDevice(self.i2c, self._BATTERY_ADDRESS) File "/home/b/.local/lib/python3.8/site-packages/adafruit_bus_device/i2c_device.py", line 62, in __init__ self.__probe_for_device() File "/home/b/.local/lib/python3.8/site-packages/adafruit_bus_device/i2c_device.py", line 187, in __probe_for_device self.i2c.unlock() File "/home/b/.local/lib/python3.8/site-packages/adafruit_tca9548a.py", line 67, in unlock self.tca.i2c.writeto(self.tca.address, b"\x00") File "/home/b/.local/lib/python3.8/site-packages/busio.py", line 203, in writeto return self._i2c.writeto(address, buffer, stop=stop) File "/home/b/.local/lib/python3.8/site-packages/adafruit_blinka/microcontroller/mcp2221/i2c.py", line 22, in writeto self._mcp2221.i2c_writeto(address, buffer, start=start, end=end) File "/home/b/.local/lib/python3.8/site-packages/adafruit_blinka/microcontroller/mcp2221/mcp2221.py", line 333, in i2c_writeto self._i2c_write(0x90, address, buffer, start, end) File "/home/b/.local/lib/python3.8/site-packages/adafruit_blinka/microcontroller/mcp2221/mcp2221.py", line 264, in _i2c_write raise RuntimeError("Unrecoverable I2C state failure") RuntimeError: Unrecoverable I2C state failure [InspiredEnergyMuxNode-2] process has died [pid 70940, exit code 1, cmd /home/b/ros_cpws/src/ros_circuitpython/src/inspired_energy_mux_driver.py __name:=InspiredEnergyMuxNode __log:=/home/b/.ros/log/8bb7bda0-41e1-11ee-867d-e93a74f52a86/InspiredEnergyMuxNode-2.log]. log file: /home/b/.ros/log/8bb7bda0-41e1-11ee-867d-e93a74f52a86/InspiredEnergyMuxNode-2*.log

To prevent our driver from crashing, RuntimeError handling was added to the main loop, if caught i2c the bus is reinitialized, which circumvents the brownout at the cost of a few seconds of downtime.

Example of handled error [INFO] [ 08/25/23 10:15:55 AM] [/InspiredEnergyMuxNode]: About to publish latest update: header: seq: 0 stamp: secs: 1692972954 nsecs: 344049692 frame_id: '' total_connected_batteries: 3 bus: - header: seq: 0 stamp: secs: 0 nsecs: 0 frame_id: '' charge_remaining_percent: 72 capacity_remaining_mAh: 4590.0 time_to_empty_mins: 65535.0 battery_discharging: True battery_fully_charged: False voltage_V: 15.654 current_A: 0.0 temp_C: 23.850000000000023 serial_no: "42977" - header: seq: 0 stamp: secs: 0 nsecs: 0 frame_id: '' charge_remaining_percent: 72 capacity_remaining_mAh: 4761.0 time_to_empty_mins: 65535.0 battery_discharging: True battery_fully_charged: False voltage_V: 15.652 current_A: 0.0 temp_C: 23.750000000000057 serial_no: "42911" - header: seq: 0 stamp: secs: 0 nsecs: 0 frame_id: '' charge_remaining_percent: 71 capacity_remaining_mAh: 4684.0 time_to_empty_mins: 65535.0 battery_discharging: True battery_fully_charged: False voltage_V: 15.635 current_A: 0.0 temp_C: 23.450000000000045 serial_no: "43019" --

[WARN] [ 08/25/23 10:15:56 AM] [/InspiredEnergyMuxNode]: Caught an I2C state failure!

[INFO] [ 08/25/23 10:16:04 AM] [/InspiredEnergyMuxNode]: About to publish latest update: header: seq: 0 stamp: secs: 1692972963 nsecs: 45123100 frame_id: '' total_connected_batteries: 3 ...

Merge request reports