This is great, brilliant even !

Yes, having distinct implementations is not great (makes testing more difficult and so forth).

Also, I do not know how much changes are needed for the scalar implementation to be able to run at constexpr time (apart from the code organization).

Hmmm... what do you think can be a blocker ? They are non allocating in general. So they should be straight out immediately easy to turn into constexpr. There are a few instances of memcpy, but we know how to handle this (bit_cast?).

The scalar implementation may however not be ideal, I presume compile time performance may benefit from a simpler implementation.

Fair point. But maybe we can narrow it down to a few functions that are too messy in the scalar namespace.

So maybe we could do something like this...

1. Have scalar/constexpr in (public) header files, in some new namespace (like simdutf_constexpr or something).
2. Most of them would be taken out of the scalar namespace, so we would empty partially the src/scalar functions, moving them to new headers.
3. In specific cases, we would have new, simplier functions for the constexpr version. We would add few of those, so they would be easy to test.

I made partial progress:

• I moved the src/scalar directory to include/simdutf/scalar and that seems to keep existing things working with very minor changes
• I invoked the simdutf::scalar::utf8::validate() function from within the ìf consteval` block in simdutf.h. this is where the problems started.
• because simdutf::scalar::utf8::validate uses reinterpret_cast internally (fully legal in non-constexpr code), it is not a constant expression and fails to work during constexpr evaluation. see https://eel.is/c++draft/expr.const#10.15

I was able to work around this by making a std::vector with a copy of the data:

 const uint8_t *data{nullptr};
  std::vector<uint8_t> tmp(buf, buf + len);
  if consteval {
    data = tmp.data();
  } else {
    data = reinterpret_cast<const uint8_t *>(buf);
  }

this kind of works but I am going to do some more experimentation.

@pauldreik Are you sure ?

https://godbolt.org/z/EY45GWY6o

Gemini say:

Reference in N4950

Section 7.7, "Constant expressions," paragraph 5, item (5.15) states that an expression E is not a core constant expression if the evaluation of a potential result involves a:

"(5.15) a reinterpret_cast (7.6.1.10);"

@pauldreik Are you sure ?

https://godbolt.org/z/EY45GWY6o

yes, you have to also use the function to see it. https://godbolt.org/z/7GeKb7r76

@pauldreik Using AI and a bit C++ knowledge, I wrote a wrapper that seems to work:

https://godbolt.org/z/n86945sa4

The idea goes like this:

template <typename to, typename from>
    requires (sizeof(to) == sizeof(from))
struct constexpr_ptr {
    from* p;

    constexpr constexpr_ptr() noexcept = default;
    constexpr constexpr_ptr(from* ptr) noexcept : p(ptr) {}

    constexpr to operator*() const noexcept { return std::bit_cast<to>(*p); }

    constexpr constexpr_ptr& operator++() noexcept { ++p; return *this; }
    constexpr constexpr_ptr operator++(int) noexcept { auto old = *this; ++p; return old; }

    constexpr constexpr_ptr& operator--() noexcept { --p; return *this; }
    constexpr constexpr_ptr operator--(int) noexcept { auto old = *this; --p; return old; }

    constexpr constexpr_ptr& operator+=(std::ptrdiff_t n) noexcept { p += n; return *this; }
    constexpr constexpr_ptr& operator-=(std::ptrdiff_t n) noexcept { p -= n; return *this; }

    constexpr constexpr_ptr operator+(std::ptrdiff_t n) const noexcept { return p + n; }
    constexpr constexpr_ptr operator-(std::ptrdiff_t n) const noexcept { return p - n; }

    constexpr std::ptrdiff_t operator-(const constexpr_ptr& o) const noexcept { return p - o.p; }

    constexpr to operator[](std::ptrdiff_t n) const noexcept { return std::bit_cast<to>(*(p + n)); }

    constexpr auto operator<=>(const constexpr_ptr&) const noexcept = default;
};

template <typename to, typename from>
constexpr constexpr_ptr<to, from> constexpr_cast_ptr(from* p) noexcept {
    return p;
}

Then using it like so:

constexpr char g(const uint8_t* c) {
    return constexpr_cast_ptr<char>(c)[0];
}

This seems to work.

I wonder why this is not in the std library?

(When I indicate that it is made by AI, it is understood that it is not to be trusted. Just a demo.)

@pauldreik I think that such an abstraction should be zero (runtime) cost, so we could use it for all out casts.

@pauldreik Using AI and a bit C++ knowledge, I wrote a wrapper that seems to work:

https://godbolt.org/z/n86945sa4

The idea goes like this:

template <typename to, typename from>
    requires (sizeof(to) == sizeof(from))
struct constexpr_ptr {
    from* p;

    constexpr constexpr_ptr() noexcept = default;
    constexpr constexpr_ptr(from* ptr) noexcept : p(ptr) {}

    constexpr to operator*() const noexcept { return std::bit_cast<to>(*p); }

    constexpr constexpr_ptr& operator++() noexcept { ++p; return *this; }
    constexpr constexpr_ptr operator++(int) noexcept { auto old = *this; ++p; return old; }

    constexpr constexpr_ptr& operator--() noexcept { --p; return *this; }
    constexpr constexpr_ptr operator--(int) noexcept { auto old = *this; --p; return old; }

    constexpr constexpr_ptr& operator+=(std::ptrdiff_t n) noexcept { p += n; return *this; }
    constexpr constexpr_ptr& operator-=(std::ptrdiff_t n) noexcept { p -= n; return *this; }

    constexpr constexpr_ptr operator+(std::ptrdiff_t n) const noexcept { return p + n; }
    constexpr constexpr_ptr operator-(std::ptrdiff_t n) const noexcept { return p - n; }

    constexpr std::ptrdiff_t operator-(const constexpr_ptr& o) const noexcept { return p - o.p; }

    constexpr to operator[](std::ptrdiff_t n) const noexcept { return std::bit_cast<to>(*(p + n)); }

    constexpr auto operator<=>(const constexpr_ptr&) const noexcept = default;
};

template <typename to, typename from>
constexpr constexpr_ptr<to, from> constexpr_cast_ptr(from* p) noexcept {
    return p;
}

Then using it like so:

constexpr char g(const uint8_t* c) {
    return constexpr_cast_ptr<char>(c)[0];
}

This seems to work.

I wonder why this is not in the std library?

(When I indicate that it is made by AI, it is understood that it is not to be trusted. Just a demo.)

interesting! I will give it a try.

I have now implemented constexpr support for a few randomly selected functions and I think it definitely is doable. I also think the changes needed are quite reasonable, and the runtime performance should not be affected at all.

It would be nice to hear from someone who would actually benefit from this feature!

I am working on this (slowly). It goes in the right direction!