Hardware Tests Fail

[sanssecours]

Description

It looks like the hardware test fails currently, since the test sensor node (Test-STH) is not available. Unfortunately I do not know what the specific problem is.

[sanssecours]

@wementoat I just reran the hardware test on the main branch. It looks like the test sensor node is still missing. Do you maybe know why this is the case?

[wementoat]

Sorry for that.

The sensor node was still connected to my Dev machine from the last session.

[sanssecours]

The hardware test job works again. Thank you for the quick update 💖.

[sanssecours]

Reopening this, after the hardware test just failed again.

[wementoat]

After investigating, it appears that the runner was rebooted right at that time - the docker container must have crashed.

It is now back online; I will send you a private message on how to access the runner.

[sanssecours]

Looks like resetting the STU fails.

[sanssecours]

Restarting the host of the external runner did unfortunately not fix the problem. We now do have 3 runners (2 of them offline), which I also find kind of strange. As far as I can tell, there was only one the last time I looked. Maybe we create one each time we restart the computer of the external runner…

…And after another restart we now have four external runners 😅 (3 of them offline).

[sanssecours]

Current Status

Unfortunately resetting the STU:

uv run icon stu reset

still fails. It looks like the STU is not responding to the reset command:

Unable to reset node “STU 1”

The text below summarizes how I tried to get the system to work again.

Restart System

I restarted the system with:

sudo shutdown -r now

This did not fix the problem. The error message is still the same.

Check the CAN Adapter Identifier

Since the Revolution Pi contains two CAN interfaces, I was not sure if can0 is the correct one. That is why I stopped the runner:

systemctl stop icoapi-runner.service

and made sure that the container is not active anymore:

docker ps

After that I did a “reset” of the CAN interface (or what I assume is pretty much the same 😅):

sudo systemctl restart systemd-networkd

Note: I also tried this on my local machine, where this command was able to turn a red status LED into a green one, after I intentionally caused an error by

• disconnecting the STU from the CAN adapter and
• executing the STU reset command (uv run icon stu reset).

On my local machine everything also worked fine after I reconnected the STU again.

I then cloned the ICOtronic repository and executed the reset command:

uv run icon stu reset

Every time I executed the command it failed with the usual error message:

Unable to reset node “STU 1”

I then changed the default interface to can1 by creating ~/.config/ICOtronic/config.yaml with the following content:

# Enable merging (of maps and lists) globally
dynaconf_merge: true

can:
  linux:
    channel: can1

and tried to reset the STU again. The first time this command failed with the same error message as before. The second time it raised an CanOperationError:

…
can.exceptions.CanOperationError: Failed to transmit: No buffer space available [Error Code 105]

This is also the same error my local machine printed, after I disconnected the STU from the CAN adapter. So to me it looks like can1 is not the interface connected to the STU, which in turn would mean that can0 should be the correct one.

Conclusion

I still do not know how to get rid of the problem. @wementoat Do you maybe have an idea what I could try next?

[wementoat]

Thank you for your thorough investigation @sanssecours - as I currently do not have access to the machine, I would suggest that you coordinate with @LoydIrrving to see if he can aid in troubleshooting.

Sorry that I cannot be of further assistance.