This is not just a code comment, it's a specification for the syntax expected by the program you end up compiling. Someone who wants to run the program with a particular complicated argv list will need to understand these rules in order to unparse their intended argument list so that it will be parsed into the right thing. So it should live in the documentation, not just here.

Also, I think it's worth considering these rules at least a little bit, because changing them later will break existing command lines. Why these rules? They don't exactly match any existing scheme:

• POSIX sh uses both single and double quotes, but treats them differently (e.g. \ inside single quotes isn't special, whereas inside double quotes it is)
• Windows (really MSVC crt0) command line splitting doesn't treat ' as a quote character at all, and treats \\ as an escaped literal \ only in some circumstances

Good point, thanks! I will document in llvmlibc.md when we conclude the logic.

I thought about shell-like behavior however I had really difficult time trying to pass anything like that through QEMU - most of the special symbols are handled by the shell that is invoking QEMU before it reaches QEMU or the emulated program.

So I decided that a better reference would be existing embedded libraries:

• picolibc seems to only split by space https://github.com/picolibc/picolibc/blob/15f0b685ee1effe218528e994ab60b4cf600fa8a/picocrt/crt0.h#L141
• Arm libc, however, handles quotations, so that it is at least possible to pass an argument with a space in it.

So Arm libc was the bar for the proposed implementation.

Happy to consider if there is any other good reference or self-consistent model of what it makes sense to handle.

I documented the behavior which is a simple three statements: args are split by space, if you need a space use quotations, if you need a quotation mark escape it.

I think if we're going to support two quote characters, they ought to do something different, to justify having both of them at all. Each one imposes inconvenience on the user, by requiring them to quote it if they want to use it literally, so each one ought to provide some benefit in return for that inconvenience.

How about taking one step further towards POSIX sh syntax, by saying that \ is not special inside single quotes, although it is inside double quotes? That means you can use single quotes for a situation where you have a lot of backslashes and don't want to have to escape them all:

myprogram "c:\\foo\\bar\\baz\\quux"      # ugh, what a pain
myprogram 'c:\foo\bar\baz\quux'          # ah, much nicer

This means that if you want a literal single quote inside your single-quoted word it's a bit awkward: you have to close the single quotes, then write a \', then reopen them:

myprogram 'here'\''s a single quote'

But POSIX sh users know this already and it will be familiar to them. And Windows users will probably prefer to use " in any case, and then ' inside that doesn't need escaping at all.

Thank you for the idea - I removed the estimation function and simplified the escaping logic: now single quotes do not treat \ as a special escape symbol and in general they support escaping usual shell symbols like $ and ?, however this is not very useful since semihosting or embedded targets are not supposed to interpret them anyway.

ACfE's handling of the string returned from SYS_GET_CMDLINE considers it to include the program name, but here you're considering it not to. So running semihosting binaries compiled by both toolchains, you'd have to pass them different command line strings.

The specification of SYS_GET_CMDLINE is vague about the meaning of the command line string. Nonetheless, given that there's an existing convention, wouldn't it be better to stick to it?

(Also, argv is supposed to contain a list of writable strings, so it's not advisable to put a string literal directly into it.)

Hm, looks like there is a difference in behavior between ACfE and picolibc.

I would prefer not to have the dummy name, indeed, however that by default (without any args passed to QEMU) argc will always be 0, which is different from hosted behavior where the program name is always (?) available.

If this seems a better choice, happy to remove the dummy name.

I removed the dummy program name.

If we're about to call main, then that presumably means the rest of libc has already been set up, including the heap? So we could malloc the space for the command line, and then malloc the space for argv too – perhaps by walking along the command line string twice, first deciding how many argument pointers you need, then generating them on the second pass.

That would avoid at least one of these arbitrary limits (I think 7 arguments other than the command name is too few even for a 256-byte limit on the command line itself – I was dealing with a benchmark program just this week which has a lot of short command line options). It also shifts large blocks of memory from the stack to the heap, which I think is preferable, since stacks are often small.

Good point, let me see if we always have the heap, if yes, then it may be a better strategy, agreed.

Same point, but for helper functions like is_space - could we use the llvmlibc ones at this point? And maybe strtok?

I agree with using malloc for the arguments array - we've been bitten by a fixed length buffer used for this exact purpose.
We should probably consider error handling - exit with a nice error message if the semihosting or malloc fails? Otherwise the program runs not as the user intended...

I added use of malloc() for memory. Unfortunately, I cannot find any API to get the actual size of the command line from semihosting, so kept one hardcoded buffer size.

I replaced local is_space() with C standard one, however strtok() seems to be less compelling because of the need to handle quotations and escape sequences.

I think if we're going to support two quote characters, they ought to do something different, to justify having both of them at all. Each one imposes inconvenience on the user, by requiring them to quote it if they want to use it literally, so each one ought to provide some benefit in return for that inconvenience.

How about taking one step further towards POSIX sh syntax, by saying that \ is not special inside single quotes, although it is inside double quotes? That means you can use single quotes for a situation where you have a lot of backslashes and don't want to have to escape them all:

myprogram "c:\\foo\\bar\\baz\\quux"      # ugh, what a pain
myprogram 'c:\foo\bar\baz\quux'          # ah, much nicer

This means that if you want a literal single quote inside your single-quoted word it's a bit awkward: you have to close the single quotes, then write a \', then reopen them:

myprogram 'here'\''s a single quote'

But POSIX sh users know this already and it will be familiar to them. And Windows users will probably prefer to use " in any case, and then ' inside that doesn't need escaping at all.

Surely this will get the wrong answer, because it's not paying attention to whether the whitespace is in quotes? I don't really understand why this new piece of code is separate from parse_cmdline_buf at all, let alone separate and completely different.

Wouldn't it be better to use the same code for both passes, with writing of output conditionalized? Then you could trust them to agree with each other.

Yes, this function overestimates by using very simple logic that does not track quotes.

The problem with parse_cmdline_buf is that it modifies cmdline in place to create null-terminated arguments, thus if run second time over the same cmdline buffer it would stop after the first, now null-terminated, argument.

So a few trade-off here between how complex the parsing logic is vs using 2x memory for the buffer to avoid updating in pace vs using a bit more memory for argv. I guess, most of the time quotes will not be used, e.g. in testing.

Let me give a try at implementing single quotes, otherwise stick with only one type of quotes.

Yes, by "writing of output conditionalized" I meant conditionalizing both writing to the argv array and overwriting the output string.

Although another possibility is that pass 1 processes the output string in place to leave the arguments dequoted and packed together at the start of the string separated by '\0 (which requires no allocation because it guarantees to use at most as much space as the original string did), and it returns the total number of arguments it found. Then pass 2 just has to find that many NUL-terminated strings and write pointers to them into the argv array. Then the two passes split the work between them, without having to duplicate the same analysis as each other.

Style suggestion:

If you make p a const char *, then the compiler would catch accidental writes through what's supposed to be the read pointer. That means you can't initialize w straight from p in the loop handling each argument, so instead you could initialize it to the start of buf here.

(You would also have to increment w when writing the trailing NUL on line 181; assign argv[argc] = w rather than = p; and skip_spaces would now take a const char *&.)

This way, you end up with the arguments packed at the start of the buffer, instead of having dead space between them. But that makes no difference to what the strings are.

(This is also the basis of my suggestion yesterday about not even doing the parsing twice. If you did this and didn't pass argv to this function at all, then you could have a much simpler pass 2 that just looks for '\0' and sets the argv pointers.)

Style suggestion: changing the semantics of the return value depending on which pass you're in is confusing! It even confused me when I pasted this function into a quick standalone test program.

Perhaps it would be better to just return argc unconditionally, and move the +1 to where the malloc happens.