The featured picture of this post has been created by DALL-E.

It often happens in embedded debugging that you suddenly end up in the interrupt dispatch table while single-stepping through your code. Another unrelated problem is that sometimes, single steps can take an eternity. In this blog post, I address both issues and describe how to circumvent them in a gdbserver implementation.

A few years ago, I wrote a blog post about interrupted single steps and presented an elegant solution for the hardware debugger dw-link. Three years later, I ended up looking at the same problem again because I decided to write a Python script that implements a gdbserver, which can talk to Microchip’s EDBG-based hardware debuggers.

Single-steps and interrupts

Let me recap the problem. What happens is that the gdbserver is instructed by GDB to execute a single instruction while the user single-steps line by line through the code. If an interrupt is raised in this situation, it will be serviced after executing the single instruction. Being in single-instruction mode, the program counter will be set to the appropriate location in the interrupt dispatch table. Then, execution is stopped and control is returned to the gdbserver. GDB does not know how to proceed single-stepping and stops. Trying to use the finish command to return to the point in the program where the interrupt was raised does not work, because GDB cannot backtrace after an interrupt has happened. So, one is basically “lost.”

One could argue that what is happening is entirely correct. An instruction was executed, the interrupt was raised, and program flow continued for this reason in the interrupt routine. However, such behavior does not make sense when single-stepping on the source-code level.

One way to find out where to continue is to inspect the stack, retrieve the return address of the interrupt service routine, and set a temporary breakpoint at this address. Doing that manually is a nuisance. However, the gdbserver AVaRICE does that for you if you use the command line option --ignore-intr. One potential problem might be that you ended up in the interrupt dispatch table by using an explicit jump to it, which, however, is very unlikely. Another issue is that it is not entirely well-defined when to exit the “repair” mode. So, a bit of guesswork is involved on the part of AVaRICE.

The advice one finds in different discussion forums for mitigating the problem is:

  1. Disable interrupts before doing a single step and enable them afterward.
  2. Simulate single-stepping by repeatedly setting a temporary breakpoint in front of the next line the program will reach and execute. The interrupts will then be serviced in the background.

A user could define GDB commands that implement these methods. Or even better, one could implement these strategies in the hardware debugger or gdbserver.

In my old blog post, I argued that both strategies are less than perfect. The first strategy might interfere with instructions that set or read the interrupt bit, e.g., SEI, CLI, BRIE, and BRID. Even if one would account for these instructions, all load and store operations could access the status register. This means one must compute the source and destination addresses for all different address modes to detect and mitigate such interference.

The second strategy has problems with conditional instructions. In this case, one would either need to set two breakpoints, precompute the destination of the conditional instruction, or simulate the execution altogether.

Interrupt-safe single-stepping

Three years ago, I could use the debugWIRE internal feature that allows the offline execution of a single instruction, which is protected against interrupts. Since this feature is apparently not exposed in the EDBG-protocol, we need a different solution this time.

While none of the two mitigation strategies mentioned above are particularly attractive, one could combine them as follows:

  • For all straight-line instructions (this includes SEI and CLI), we use the hardware breakpoint in order to stop the execution after the instruction has been executed.
  • For all unconditional branching instructions and all conditional instructions, except BRIE/BRID, we use the interrupt disable/enable pair around a single step.
  • For BRIE/BRID, we compute the branch destination address and set the hardware breakpoint at this location.

This will not interfere with the I-bit, does not require two breakpoints, and leads only to a minimal amount of interpreting instructions (the last case). In contrast to the AVaRICE strategy, one does not have to guess anything. Meanwhile, it has been implemented in the Python gdbserver and works quite well.

Note that there is still some potential interference. The instructions CALL and RCALL push the return address on the stack, and RET and RETI retrieve return addresses from the stack. Similarly, PUSH and POP manipulate the stack. If the stack pointer points to address 0x0060 or 0x005F before a return address is pushed to the stack or it points to 0x005D or 0x005E before a return address is popped from the stack, then the status register (located at 0x005F) will be affected. However, a lot of chaos is expected if the stack is that full anyway because important IO registers will be clobbered or have been clobbered already. For instance, 0x005D/0x005E contains the stack pointer! For these reasons, I will ignore this situation and leave it to the user to untangle such a situation.

Very long single steps

Another annoying single-stepping phenomenon is that single steps can take a very long time. In fact, so long that one will be forced to use Ctrl-C to interrupt the step. The reason for that is that when asking GDB to make a single step, it will single-step all MCU instructions that belong to the source line. This is not a problem if the line contains only straight-line instructions. However, if the line contains a loop, it can take a long time. In another context, I estimated the time necessary for the communication between GDB and gdbserver when executing a single instruction to be 100 milliseconds. An average instruction executes in 2 cycles, which is 0.125 microseconds on a 16 MHz MCU. In other words, single-stepping is almost a Million times slower. Now look at the line:

_delay_ms(100)

This is a macro that expands into the following loop:

start: ldi  r18, 0xFF  ; 255
       ldi  r24, 0xE1  ; 225
       ldi  r25, 0x04  ; 4
loop:  subi r18, 0x01  ; 1
       sbci r24, 0x00  ; 0
       sbci r25, 0x00  ; 0
       brne loop 
       rjmp end
end:   nop
exit:  ....

Being 1,000,000 times slower means you now have to wait 100,000 seconds or roughly one day instead of 100 ms.

Range-stepping

Since GDB version 7.7, there exists a range-stepping command in GDB’s remote serial protocol. Instead of requiring the execution of a single instruction, GDB can now require the execution of an entire range of instructions. Only if the execution leaves this range, the gdbserver needs to hand control to GDB. However, it may always return control to GDB before that.

So, how can we exploit this range-stepping command to speed up single-stepping? The first obvious optimization is to run over straight-line instructions and stop only on branching instructions. This can be accomplished using one temporary breakpoint (which could be a hardware breakpoint). The branching instructions can be executed in a single-stepping fashion, after which we continue as above. This would speed up the above loop by a factor of 7. So, we only have to wait a few hours instead of an entire day. Not good enough!

A control-flow analysis can help us to determine all potential exit points of the range. These are locations outside the range that are reached by branch instructions or locations inside the range with runtime-dependent branch destinations, such as RET or IJMP. Applying such an analysis to the above code reveals that the only exit point is the one labeled exit:. So, in the above case, it is enough to set a (hardware) breakpoint at exit:. And with that we are back at executing this line in approximately 100 milliseconds.

Is it always possible to reduce range-stepping to setting just one breakpoint? Obviously not, as the next example shows.

while (++i) { if (i < 0) return; }

This will be compiled into the following AVR code, assuming that i is a signed char.

start:  subi r24, 0xFF ; 255
        breq exit 
        brpl start 
return: ret
exit:   ...

As one can see, now we have two exit points, one at return: and one at exit:. If you only have one hardware breakpoint, you can use the fall-back strategy of breaking on all branch instructions, or you can employ software breakpoints. I don’t know what the best alternative is. And, honestly, how often does one put such crazy code into a single source line? The current implementation uses only hardware breakpoints and will fall back on breaking at every branch instruction if not enough hardware breakpoints are available.

Summary

All in all, interrupt-safe single-stepping and range-stepping should make debugging with dw-gdbserver more pleasant.

Views: 18