Continued connectivity issues on pioreactor

Software
1

Hello!

We are still facing the same connectivity issues as before, and I’ve tried many troubleshooting steps in between that, so I think it is appropriate to start a new thread regarding this.

After a minute or so, the pioreactor loses connectivity and we have to unplug/replug power back into it for it to connect again (but alas, also only for a minute, repeating this vicious cycle). As mentioned before, we have two pioreactors that are encountering this issue. I have already tried the following steps to figure out where the connectivity issues could be coming from:

  1. Trying Wi-Fi that is not from my institution. The same issue happened when I used the Wi-Fi from a local coffee shop. My mobile phone’s hotspot did not work either.
  2. Using PuTTY to change the TCPKeepAlive setting and interval when I SSH into the pioreactor.
  3. Switching Raspberry Pis. Oddly enough, the issue still occurs even when we are running a Raspberry Pi by itself, only with the pioreactor software loaded and not physically attached to a pioreactor.
  4. Trying different power supplies, making sure that the supplies are giving 5V/3A to the Pi.
  5. Downgrading pioreactor versions
  6. Changing any relevant port/Windows firewall settings on my laptop related to network sharing

I think it could be beneficial to see if we can change the SSH settings stored on the Pi image itself, which is a bit hard to do given the connection doesn’t last very long. for this, we currently think the issue is more software-related, but are open to any ideas. I am growing more worried as my team would like to start collecting OD data soon. If anyone has any suggestions, please let me know!

Thank you!

1 Like
2

So I can help narrow down the issue:

  • it’s unlikely power supply issues
  • it’s unlikely Pioreactor software versions
  • It’s unlikely the hardware (Pis or HAT)
  • It’s unlikely TCPKeepAlive / Putty / SSH issues.

Can you tell me more about your set up? From your post, I’m guessing it’s:

[leader + worker] Pioreactor controls a few Pioreactors over a wifi connection. Or is it wired? You’ve tried a few wifi routers, and it sounds like the same problems?

For a wired connection between PC and Pis, I’ve seen this problem occur where Pi’s disconnect after a few minutes. This is because of some issue with Link-Local (which is different than Internet Sharing). Both look very similar: direct wired connection between PC and Pi. You can check if you’re indadvertedly using Link Local by checking if the PC and Pi have an ipv4 like 169.254.x.x.

For wifi problems, that’s harder to diagnose. If you can share more information about your set up, that would help!

3

We have two pioreactors, but both are being separately used as a standalone leader + worker (so each of our subteams has one to use and test). Our Pi has an ethernet port that we have been connecting to, with the other end being in my laptop. From there, I was giving the Pi internet via my institution’s Wi-Fi connection on my laptop through the internet sharing instructions outlined in the user guide.

I will check if there is some inadvertent Link Local happening. The ipv6 addresses have started with fe80:: at times too. I’ll look into it and let you know, thank you for the help!

4

Definitely test with a recent Pioreactor image, too. This link sounds like a similar issue too, https://forums.raspberrypi.com/viewtopic.php?t=377784, and has a work-around at the end.

Here’s the code with correct root privileges:

# Create a NetworkManager connection file that tries DHCP first
CONNFILE1=/etc/NetworkManager/system-connections/eth0-dhcp.nmconnection
UUID1=$(sudo uuid -v4)
sudo tee ${CONNFILE1} > /dev/null <<- EOF
[connection]
	id=eth0-dhcp
	uuid=${UUID1}
	type=ethernet
	interface-name=eth0
	autoconnect-priority=100
	autoconnect-retries=2
	[ethernet]
	[ipv4]
	dhcp-timeout=3
	method=auto
	[ipv6]
	addr-gen-mode=default
	method=auto
	[proxy]
EOF

# Create a NetworkManager connection file that assigns a Link-Local address if DHCP fails
CONNFILE2=/etc/NetworkManager/system-connections/eth0-ll.nmconnection
UUID2=$(sudo uuid -v4)
sudo tee ${CONNFILE2} > /dev/null <<- EOF
[connection]
	id=eth0-ll
	uuid=${UUID2}
	type=ethernet
	interface-name=eth0
	autoconnect-priority=50
	[ethernet]
	[ipv4]
	method=link-local
	[ipv6]
	addr-gen-mode=default
	method=auto
	[proxy]
EOF

# NetworkManager will ignore nmconnection files with incorrect permissions so change them here
sudo chmod 600 ${CONNFILE1}
sudo chmod 600 ${CONNFILE2}
5

Thanks for your help and patience with this. It has been a challenge pasting the file contents and executing the commands swiftly, as using SSH via the ethernet cable is currently the only way I’m able to access those files right now. I’m going to see if I can hook the Pis up to an external monitor. Hopefully I can provide a better update either later today or tomorrow.

(Interestingly, when I forced PuTTY to do an ipv4 connection it simply wouldn’t let me SSH into it altogether. I don’t know what I can conclude from that but it seemed something I should note here)

6

Okay, some good news. I was able to get both of the pioreactors running (and surviving for more than 1 minute) with the code you provided from that forum post. It took me a while because I had to be swift with the commands and I’m not too familiar with Linux. Nonetheless, that was exactly it, and both of them can maintain connectivity now.

In bad, perhaps weird, news though:

  1. Both pioreactors failed 9 out of the 10 self tests. The logs attributed them to old-style revision code errors. But the Pis had 10.29.24 freshly installed on them, so I don’t know what this is referring to exactly.
  2. The pioreactors can successfully install plugins, but nothing shows up in the “Installed plugins” section. Still, the logs say that the plugins were successfully installed. I am specifically trying to install Logs2Discord but it happens for other plugins too.
  3. Despite the interface showing that OD, stirring, and temperature control is off, I can’t turn them on because I get a warning message saying that it’s already on, so it skips the request.

Are there any more possibilities we could explore? I can send over some logs in a few hours. Do let me know. I really really appreciate the help!

7

hi @clamtime, yea please send the logs over to support@pioreactor.com. We can help there!