Avoiding Mysterious, Silent Failures of Your Arduino Sketch: Push PROGMEM to the far end

The featured picture of this post was created by DALL-E.

Have you ever used an Arduino Mega 2560 (or a similar board) and, at some point in the development process, experienced the LED mysteriously stopping to blink, garbled text being printed, or funny artifacts appearing in pictures? And all that without any apparent reason or any error or warning message? If you want to know what is behind it and how to solve this problem, read on.

The basics

It all starts with the observation that on processors using a modified Harvard architecture, such as the AVR MCUs, constant values should not occupy precious space in the data space. For example, string constants, such as “Hello world”, should not be stored in data space, but in program space. In fact, when requesting to store such a constant in data space—the usual way in C and C++—the string value will be stored in program space anyway, and when starting the program, it will be copied to data space. In order to minimize the use of data space, the PROGMEM macro was introduced, which instructs the compiler to store a data structure in program space, i.e., flash memory, in a way so that one can access it somehow, as shown in the following sketch.

#include <avr/pgmaspace.h>

const char PROGMEM hello[] = "Hello World";
const unsigned int PROGMEM table[3] = { 1 , 2, 3 };

void setup() {
  Serial.begin(19200);
  Serial.println((const __FlashStringHelper *)hello);
  for (int i=0; i < 3; i++)
    Serial.println(pgm_read_word(&table[i]));    
}

void loop() { }

The typecast in the first Serial.println call looks a bit ugly, but it is necessary. If you are using this string only once, you could use the more readable F-macro:

    Serial.println(F("Hello World"));

As demonstrated above, it is not only possible to store strings in program space, but all kinds of data. Access to such arrays in flash memory cannot be done using ordinary array operations, but you need to use special macros that read from flash, e.g. pgm_read_word(unsigned int addr), which retrieves one word from the flash address addr.

Accessing such data in flash memory is usually much slower than accessing data in SRAM. However, one saves on data space, which is generally quite restricted on AVR MCUs. For this reason, it is common wisdom among Arduinoists to store all constant strings, lookup tables, fonts, image data, and the like in flash memory using the PROGMEM macro.

So what does the PROGMEM macro do? It simply expands into __attribute((progmem))__, which tells the compiler that the decorated variable should be allocated in the progmem linker section. In the end, after all source files have been compiled, the linker will then allocate absolute addresses for all objects, including those in the progmem section, which will become part of the program space.

What is the problem?

As long as you work with MCUs that have at most 64 kB of flash memory, everything works without a hitch. However, once you go beyond this limit, you can no longer address all bytes in flash with 16-bit addresses. To mitigate this issue, the above-mentioned progmem linker section is allocated first. However, what will happen when more than 64 kB of PROGMEM data is allocated? The simple answer is: All kinds of crazy things can happen. All data allocated above the 64 kB limit is not accessible to the usual PROGMEM macros. This is illustrated in the following sketch.

#include <avr/pgmspace.h>

const byte tab0[5] PROGMEM = { 1, 2, 3};
const byte tab1[32767] PROGMEM = {  };
const byte tab2[32767] PROGMEM = {  };

void setup()
{
  Serial.begin(19200);
  pinMode(LED_BUILTIN, OUTPUT);
}

void loop()
{
  Serial.println();
  Serial.println(F("PROGMEM test"));
  Serial.println(pgm_read_byte(&tab0[0]));
  Serial.println((unsigned int)&tab0, HEX);
  Serial.println((unsigned int)&tab1, HEX);
  Serial.println((unsigned int)&tab2, HEX);
  digitalWrite(LED_BUILTIN, HIGH);
  delay(1000);
  digitalWrite(LED_BUILTIN, LOW);
  delay(1000);
}

This sketch will compile without an error or warning message. However, it will not blink the LED, it will not print “PROGMEM test”, and it will not print the content of tab0[0].

The conventional cure

The conventional cure, proposed also in the avr-libc manual, is to use macros that have a _far suffix, e.g., pgm_read_word_far(unsigned long addr), which expect a 32-bit address. You have to use pgm_get_far_address to get such a 32-bit address from a variable reference.

There are, however, at least two problems with this approach. First, nobody tells you when you cross the 64 kB limit. You do not get any error message, warning, or anything whatsoever. So, how should you know to switch to the more expensive form of accessing your PROGMEM data? Second, even if you do recognize that you are over the limit and have your program adapted to use 32-bit PROGMEM addresses, there is still the problem that the Arduino core and tons of libraries do not use 32-bit addresses to access PROGMEM variables. Further, all strings stored in the PROGMEM area using the F-macro have the same problem. Applied to our example sketch above, we will be able to print the content of tab0[0], but the F-macro string will not be shown, and the LED will not blink either.

Interestingly, all this has been known for more than 10 years. The proposed fix, however, has never been applied because it suffers from the problem that only the tables in the Arduino core are guaranteed to be located in the lower part of the flash space; the general issue, however, remains unsolved.

The real cure

A solution should not induce a performance penalty for sketches running on Arduinos with 64 kB or less, it should not break existing code, and it should require minimal changes. So, the idea of converting all access to flash to macros using 32-bit addresses is one of those ideas that do not sound very attractive. It would involve a lot of change, break existing code, and would lead to worse runtime performance.

A solution for the problem should at least recognize potentially problematic situations. And then one would like to have a means to add PROGMEM data to the sketch without interfering with string constants and the PROGMEM data already present in libraries and the Arduino core.

Identifying potentially dangerous situations

One part of the solution is to identify situations where more than 64 kB of PROGMEM data is used. This can be done by a linker script augmentation, perhaps named progmemcheck.ld, which looks as follows:

INSERT AFTER .text;

ASSERT (__ctors_start < 0x10000,  "PROGMEM capacity exceeded" );

One could then add a flag -T progmemcheck.ld to the call of the linker. Then, the execution of the augmented linker script will check whether the __ctors_start address is below the 64kB limit, which implies that PROGMEM is also below this limit. The relevant part of the default linker script looks as follows:

.text   :
  {
    *(.vectors)
    KEEP(*(.vectors))
    /* For data that needs to reside in the lower 64k of progmem.  */
    *(.progmem.gcc*)
    /* PR 13812: Placing the trampolines here gives a better chance
       that they will be in range of the code that uses them.  */
    . = ALIGN(2);
    __trampolines_start = . ;
    /* The jump trampolines for the 16-bit limited relocs will reside here.  */
    *(.trampolines)
    *(.trampolines*)
    __trampolines_end = . ;
    /* avr-libc expects these data to reside in lower 64K. */
    *libprintf_flt.a:*(.progmem.data)
    *libc.a:*(.progmem.data)
    *(.progmem.*)
    . = ALIGN(2);
    /* For code that needs to reside in the lower 128k progmem.  */
    *(.lowtext)
    *(.lowtext*)
     __ctors_start = . ;
     *(.ctors)
     __ctors_end = . ;
... }

This part of the linker script shows how the output linker section .text is filled with different parts of the program code. In particular, most of the .progmem parts come after the .trampolines section and before the .lowtext section. The interesting point is, I do not know of any parts of the Arduino code that need to be placed in the .lowtext section. And even if such parts exist, the size of such a code section is most likely only a few kilobytes. So, it is a good first approximation to check whether the __ctors_start value, which marks the end of the .lowtext section, is below 0x10000.

While this solution requires only a minor change, it may potentially break existing code. There might be Arduino code out in the wild that works perfectly well with more than 64 kB of PROGMEM data. So, instead of terminating a compilation with an error message, a warning might be more appropriate. For this reason, I wrote a shell script progmemcheck.sh:

#!/bin/bash
progmem_end=`$1 -t d $2 | grep __ctors_start | cut -d ' ' -f 1`
if [ $progmem_end -gt 65535 ]; then
    warning="| Severe Warning: PROGMEM section too large by $(expr $progmem_end - 65535) bytes.            "
    echo "_______________________________________________________________"
    echo "${warning:0:62}|"
    echo "| Your program will most probably be unstable!                |"
    echo "| Use the macro PROGMEM_FAR from <progmem_far.h> and          |"
    echo "| pgm_get_far_address/pgm_read_xxxx_far from <avr/pgmspace.h>.|"
    echo "|_____________________________________________________________|"
fi

The first parameter to the script should be the GNU avr-nm tool and the second should be the ELF file. When this script is called as part of the objcopy recipe like that,

## Check for progmem overflow
recipe.objcopy.progmem.pattern="/bin/bash" "{runtime.platform.path}/extras/progmemcheck.sh" "{compiler.path}/{compiler.symbol.cmd}" "{build.path}/{build.project_name}.elf"

it could then give you a warning as follows:

Of course, one also needs a Windows batch script. With a little help from VS Code’s Copilot, I was successful in writing one.

The downside of the warning approach is that by default, the Arduino IDE does not show compilation messages. You must enable this in the preference dialog explicitly. However, if someone experiences problems like the ones mentioned at the beginning, one will probably allow messages (and compiler warnings as well) to identify whether something is wrong.

Pushing PROGMEM to the far end

After receiving a warning as above, one needs to take action. The question is what one can do. The warning message already contains hints. You should use the macro PROGMEM_FAR instead of the ordinary PROGMEM macro. In case the Arduino code base would use a recent AVR-GCC toolchain, this macro would already be part of avr-libc. However, until the new toolchain is adapted, the tiny library progmem_far.h provides the missing PROGMEM_FAR macro. This library can be easily installed with the Arduino library manager. The code looks as follows.

#define PROGMEM_FAR __attribute__((section (".fini0")))

void __terminate_program__(void) __attribute__((naked, section(".fini1"), used, weak));
void __terminate_program__(void)
{
  while(1);
}

That means that the PROGMEM_FAR macro simply tells the compiler to use the .fini0 section to store the data. Where does this section come from? Here is another look into the .text section from the linker script:

   
.text   :
  { ...
    *(.text)
    . = ALIGN(2);
     *(.text.*)
    . = ALIGN(2);
    *(.fini9)  /* _exit() starts here.  */
    KEEP (*(.fini9))
    *(.fini8)
    KEEP (*(.fini8))
    *(.fini7)
    KEEP (*(.fini7))
    *(.fini6)  /* C++ destructors.  */
    KEEP (*(.fini6))
    *(.fini5)
    KEEP (*(.fini5))
    *(.fini4)
    KEEP (*(.fini4))
    *(.fini3)
    KEEP (*(.fini3))
    *(.fini2)
    KEEP (*(.fini2))
    *(.fini1)
    KEEP (*(.fini1))
    *(.fini0)  /* Infinite loop after program termination.  */
    KEEP (*(.fini0))
     _etext = . ;
  } 

So, after the input sections .text from every translation unit, there are 10 sections called .fini9 to .fini0, which deal with the termination of the program. .fini0 is the last one, which we will abuse to place the PROGMEM code that could be pushed to the far end. In order to guarantee nevertheless a proper termination, we add an infinite loop in the user-definable section .fini1. And with that, everything works perfectly. The following sketch is the “repaired” version of our above example. It demonstrates that things are now working flawlessly.

#include <pgmspace.h>
#include <progmem_far.h> 

const byte tab0[5] PROGMEM = { 1, 2, 3};
const byte tab1[32767] PROGMEM_FAR = {  };
const byte tab2[32767] PROGMEM = {  };

void setup()
{
  Serial.begin(19200);
  pinMode(LED_BUILTIN, OUTPUT);
}

void loop()
{
  Serial.println();
  Serial.println(F("PROGMEM test"));
  Serial.println(pgm_read_byte(&tab0[0]));
  Serial.println(pgm_get_far_address(tab0), HEX);
  Serial.println(pgm_get_far_address(tab1), HEX);
  Serial.println(pgm_get_far_address(tab2), HEX);
  digitalWrite(LED_BUILTIN, HIGH);
  delay(1000);
  digitalWrite(LED_BUILTIN, LOW);
  delay(1000);
}

This sketch differs from the original one in that we use the PROGMEM_FAR macro on tab1, and we now use the pgm_get_far_address macro to determine the actual addresses of our PROGMEM data structures. When we now compile and upload the sketch, we do not get a warning message, the LED will blink, the message is printed, and the content of tab0[0] is also correctly printed.

Outlook

I will propose a pull request to MCUdude’s MightyCore and MegaCore and to the Arduino AVR core. This PR will implement the PROGMEM size check as described above. The user then needs to deal with the rest.

UPDATE (01-Oct-2025): My PR has been accepted in MightyCore and MegaCore and will be deployed when the next version is generated.