Skip to content

[POC] stdio: avr pgmspace#12940

Open
fjmolinas wants to merge 3 commits intoRIOT-OS:masterfrom
fjmolinas:poc_avr_pgmspace
Open

[POC] stdio: avr pgmspace#12940
fjmolinas wants to merge 3 commits intoRIOT-OS:masterfrom
fjmolinas:poc_avr_pgmspace

Conversation

@fjmolinas
Copy link
Copy Markdown
Contributor

Contribution description

When testing #12461 I realized many tests where not compiling on arduino-% platforms because a lot of new strings were going into .bss. This pushed me to explore how to force storing them in flash.

When compiling for AVR because of harvard architecture constant data goes into RAM instead of flash. This can be quite a strain for prints loaded applications with small memory like arduino-uno.
pgmspace can be used to force strings and other variables to go intro flash, this POC overwrites
some of the stdio functions to use there pgmspace versions.

Maybe it could make sense to enable this by default for the smaller platforms. This can work as long as the stdio functions receive a string literal and not variables. This is because the variable needs to be stored in pgmspace by using the PROGMEM key word ass well as using the pgmspace version of the function.

  • works:

puts("toto")

  • does not work, since buffer would need to be declared with PROGMEM key word.
uint8_t buffer[8] = "toto";
puts(toto)

In the case of puts there are some uses that prevent this, but they could be migrated.

git grep '[^a-zA-Z]puts([^\"].*)'
examples/posix_sockets/udp.c:            puts(server_buffer);
pkg/lua/contrib/lua_run.c:            puts(lua_tostring(L, -1));
sys/log/log_printfnoformat/log_module.h:    puts(format);
sys/net/gnrc/network_layer/ipv6/blacklist/gnrc_ipv6_blacklist_print.c:            puts(ipv6_addr_to_str(addr_str, &gnrc_ipv6_blacklist[i], sizeof(addr_str)));
sys/net/gnrc/network_layer/ipv6/whitelist/gnrc_ipv6_whitelist_print.c:            puts(ipv6_addr_to_str(addr_str, &gnrc_ipv6_whitelist[i], sizeof(addr_str)));
sys/shell/commands/sc_gnrc_icmpv6_echo.c:        puts(dupmsg);
sys/shell/shell.c:                        puts(INCORRECT_QUOTING);
sys/shell/shell.c:                            puts(INCORRECT_QUOTING);
sys/shell/shell.c:                    puts(INCORRECT_QUOTING);
sys/shell/shell.c:                            puts(INCORRECT_QUOTING);
sys/shell/shell.c:                        puts(INCORRECT_QUOTING);

printf and others could not be enabled by default since too much code uses a variable instead of a string literal.

Maybe the best way of doing this is modifying the linker script itself? I tried doing it as well but it was quite dirty to be honest, maybe someone else can find a good solution? This would probably work better since all constant initialized data could be moved and not only strings.

Testing procedure

  • Does this make sense? Maybe it is too much of a hack and I know we don't like this kind of macro usage.

  • Run the test, enable globally and compare some compiled firmware .bss, .data, .txt usage.

  • Is this worth exploring further??

@fjmolinas fjmolinas added Discussion: RFC The issue/PR is used as a discussion starting point about the item of the issue/PR State: demonstrator State: This PR (loosely) demonstrates a concept and is not necessarily meant to be merged. labels Dec 13, 2019
@@ -0,0 +1,1006 @@
/* Copyright (c) 2002, 2005, 2007 Joerg Wunsch
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so this is not the default avr libs stdio.h?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it a verbatim copy? If so, note that avr-libc is used anyway for all AVR boards anyway.

include ../Makefile.tests_common

CFLAGS += -DAVR_STDIO_PGMSPACE
FEATURES_REQUIRES += arch_avr
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
FEATURES_REQUIRES += arch_avr
FEATURES_REQUIRED += arch_avr

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ups

@kaspar030
Copy link
Copy Markdown
Contributor

This should at least be optional and not default. Too bad the compiler cannot do this (use `puts_p() if used on a constant string literal)) automatically...

@maribu
Copy link
Copy Markdown
Member

maribu commented Dec 13, 2019

I was thinking about using PROGMEM as well. Note that PROGMEM is a macro defined by the AVR libc, which is defined as __attribute__((__progmem__)). Maybe it makes sense to use our own define instead, to have the possibility to use the same API in case other harvard architectures. E.g. the ESP8266 seems to also be a hardward architectured modified to be able to access program memory.

However, I'm not sure if overwriting (parts of) stdio is the best approach, as unprepared code will break by this change. What we could do instead is to provide special functions and qualifiers for working with flash memory. E.g. we could create our own define for PROGMEM that would just be defined as const on von-Neumann and as const __attribute__((__progmem__)) on AVR. And then we could provide puts_progmem() as well, which would be a static inline wrapper for puts() on von-Neumann and for puts_P() for AVR. That way code could be explicitly prepared to work with this without any disadvantages (except for additional clutter to the source code) for von-Neumann machines. And unprepared code that just keeps using const char *foo and regular puts() will not break.

One point that I was specifically looking at is the stringification in SAUL of both sensors/actuators and units. Those seem to be the largest chunk of data allocated constants on AVR and getting them to flash would allow examples/default to run again on the Arduino Uno. Given the popularity of that board, I really would like our default example to again work there. People having an Arduino Uno and trying to run examples/default on it will get a bad first impression of RIOT right now and I really would love to change this.

@fjmolinas
Copy link
Copy Markdown
Contributor Author

One point that I was specifically looking at is the stringification in SAUL of both sensors/actuators and units. Those seem to be the largest chunk of data allocated constants on AVR and getting them to flash would allow examples/default to run again on the Arduino Uno. Given the popularity of that board, I really would like our default example to again work there. People having an Arduino Uno and trying to run examples/default on it will get a bad first impression of RIOT right now and I really would love to change this.

Yes probably all the shell commands function could use this.

What we could do instead is to provide special functions and qualifiers for working with flash memory. E.g. we could create our own define for PROGMEM that would just be defined as const on von-Neumann and as const attribute((progmem)) on AVR.

Yes this would be better than refining puts, which would hide the actual behavior of the function. But how do you see this working with uses like puts("foo")? I guess having two versions of puts_pgmsapce where one is just puts_P(_s) and the other puts_P(PSTR(_s)) would work but at the price of some clutter. Maybe we don't mind this?

@benpicco
Copy link
Copy Markdown
Contributor

What I do not understand: If Cortex-M3 is also a Harvard machine, why can it be handled transparency by the compiler there?
Is this a shortcoming of avr-gcc or or is there something inherently different with the AVR architecture that makes this necessary?

@maribu
Copy link
Copy Markdown
Member

maribu commented Oct 30, 2020

Is this a shortcoming of avr-gcc or or is there something inherently different with the AVR architecture that makes this necessary?

The difference between the Harvard and the von-Neumann architecture in the "original" sense is that Harvard has two distinct address spaces for program and data, and von-Neumann only has one for both.

Many modern Harvard architectures however map program space also into the data address space. (Even the ATtiny of the AVR family do so.) If that is the case, you can safely access ROM data in the very same way as RAM data (except it is read-only).

In the classic Harvard machine, it is completely impossible to read data from program memory - you can only run it. The ATmega family is already an improvement upon this, as it has two load instructions - one working with data addresses, one with program addresses. However, the compiler must now which instruction it has to emit at compile time. And as data and program address can have the same value range, you couldn't even provide a generic helper like:

void memcpy_program_or_data(void *dest, void *src, size_t len)
    if (is_program_memory(src)) {
        memcpy_from_program_memory(dest, src, len);
    }
    else {
        memcpy(dest, src, len);
    }
}

So, yes: It is a limitation of the AVR (or specifically, the ATmega family - it would work fine for ATtinys).

fjmolinas added 3 commits June 1, 2022 09:46
- Add vendor files to define all stdio functions as their PGMSPACE
  equivalent. This can be enabled by setting the flag
  AVR_STDIO_PGMSPACE.
@github-actions github-actions bot added Area: cpu Area: CPU/MCU ports Area: tests Area: tests and testing framework Platform: AVR Platform: This PR/issue effects AVR-based platforms labels Jun 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area: cpu Area: CPU/MCU ports Area: tests Area: tests and testing framework Discussion: RFC The issue/PR is used as a discussion starting point about the item of the issue/PR Platform: AVR Platform: This PR/issue effects AVR-based platforms State: demonstrator State: This PR (loosely) demonstrates a concept and is not necessarily meant to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants