[EV3] Various issues to dig into

[laurensvalk]

It's hard to put a finger on, but the brick seems pretty unstable at the moment. This may be caused by a bunch of different things, so I'm just collecting some observations for now.

UART issues
Sensors seem to have various issues that appear related to the UART driver but this isn't as easy to test.

An easier test seems to be to enter paste mode on the REPL (CTRL+E) and paste something like the following. Anything of similar length will probably do.

try:
    from pybricks.pupdevices import Motor
except ImportError:
    from pybricks.ev3devices import Motor

It's expected that some characters may be missing since there is no pushback, but it shouldn't get stuck.

The pbio event loop keeps running since the run animation keeps going, but the REPL no longer accepts new characters. If the UART driver gets stuck then this could explain sensor issues too.

Exiting the program with the stop button also works, and new output is shown. It's just input that gets stuck.

I/O operations seem to affect loop time (or ADC)
When you plug in the NXT color sensor or NXT light sensor, it toggles through the colors (or on/off). This is smooth while idle, but you can see it stutter when I/O operations run. Or in this case any program while the run animation is running on the screen. Before we had the animation, it would flicker while typing or pasting text into the REPL.

I'd like to revisit some of the ev3_dcm code to fix some of its timing. It could well be down to this rather than being a symptom of another issue, but it's worth keeping an eye on. Nothing should visually blink once we get this right.

MicroPython crash

Looking into timing, I was dusting off some of our motor benchmarks in order to run them with pybricks/pybricks-micropython#361 (thanks @ArcaneNibble!). I didn't get much further than the import stage yet. Attempting to run the following makes the brick crash and reboot.

try:
    from pybricks.pupdevices import Motor
except ImportError:
    from pybricks.ev3devices import Motor

This doesn't seem to crash when typing it into the REPL directly (if slowly) on the current master branch.

[ArcaneNibble]

Interesting, I wasn't able to reproduce a crash using

try:
    from pybricks.pupdevices import Motor
except ImportError:
    from pybricks.ev3devices import Motor

but

try:
    from pybricks.pupdevices import Motor
except ImportError:
    from pybricks.ev3devices import Motor
print(Motor)

did reproduce the crash.

from pybricks.ev3devices import Motor
print(Motor)

also does not reproduce the crash, which is very mysterious.

EDIT: This is definitely a watchdog-related crash (which I tested by commenting out the watchdog code).

[ArcaneNibble]

I did notice UART REPL instability when I was testing things in the past, but I had always assumed it was because "debug REPL is buggy"

The REPL seems much more stable over USB (which I've written a custom hack to be able to use, since I can't figure out how to get pybricksdev to just dump me into a REPL)

[laurensvalk]

I did notice UART REPL instability when I was testing things in the past, but I had always assumed it was because "debug REPL is buggy"

You can change the debug port used to one of the PRU uarts and then it doesn't have this problem, so the hardware UART driver does have some problems (and maybe you've found them).

[ArcaneNibble]

I'm at a bit of a loss where to start trying to hunt down the issue with the watchdog reset. It doesn't seem to be related to the motor logic itself, since running a motor slowly via the REPL works fine.

[ArcaneNibble]

re: the light sensor flickering: I commented out https://github.com/pybricks/pybricks-micropython/blob/master/lib/pbio/sys/light_matrix.c#L45-L51 and the flickering seems to go away.

edit: this was a hunch, prompted by the fact that pressing the left/right d-pad buttons would cause a flicker except when the currently selected slot was already first/last

So it seems like we might need to optimize some of the display drawing code a bit (or see if enabling the dcache helps?)

[laurensvalk]

It isn't just the display, see also:

Before we had the animation, it would flicker while typing or pasting text into the REPL.

It could still be just the current timing logic of the EV3 dcm loop. I'll look into that at some point.

I'm at a bit of a loss where to start trying to hunt down the issue with the watchdog reset.

We still need to set some sensible defaults for heap and stack. They are still set at very small values. It's possible that MicroPython needs a bit more breathing room.

This reminds me of this hack I put in a while ago to get MicroPython to run at all on the EV3. We were using mp_stack_set_top everywhere but I could only get it to run with mp_stack_ctrl_init on EV3. Link. This was before we fixed various things in startup and the linker script, so maybe we can just use mp_stack_set_top everywhere now.

    char *stack_start;
    char *stack_end;
    pbdrv_stack_get_info(&stack_start, &stack_end);
    #if PYBRICKS_OPT_USE_STACK_END_AS_TOP
    mp_stack_set_top(stack_end);
    #else
    // Sets the top to current stack pointer.
    mp_stack_ctrl_init();
    #endif
    mp_stack_set_limit(MP_STATE_THREAD(stack_top) - stack_start - 1024);

[laurensvalk]

On the light flicker, I can confirm that it is steady if we use a simple wait with timer_extend between light toggles, instead of waiting for a new ADC sample. Even with the display animatio enabled. This is a good sign, as it doesn't point at a fundamental runloop issue. So we can wait on that in a different way to get consistency.

[ArcaneNibble]

I did some hacking to enable the MMU on the EV3, and the "MicroPython crash" issue finally has a useful lead. With the MMU properly configured, we get:

********************************************************************************
*                            Pybricks on EV3 Panic                             *
********************************************************************************
Exception type: Data Abort
R0:   0x155F1FD1
R1:   0xC006220B
R2:   0x00000059
R3:   0x155F1FD0
R4:   0x155F1FCD
R5:   0xC006220C
R6:   0xC0061FC0
R7:   0x155F1FD1
R8:   0xC3D2FEFC
R9:   0xC0050000
R10:  0x00000006
R11:  0xC3D2FF28
R12:  0x00000000
R13:  0xC3D2FDE4
R14:  0xC00299C4
R15:  0xC0041DE8
SPSR: 0xA000001F
DFSR: 0x00000005
IFSR: 0x00000008
FAR: 0x155F1FD1

which is a very-out-of-bounds read in... strcmp

At least that's a start?

[ArcaneNibble]

This is called from a symbol mpy_data_find.lto_priv.0

Still don't know why or what is so out-of-bounds though