Skip to content

Disallow delim and quotechar matching#1066

Merged
quinnj merged 1 commit intomainfrom
jq/disallow-quote-delim-match
Jan 4, 2023
Merged

Disallow delim and quotechar matching#1066
quinnj merged 1 commit intomainfrom
jq/disallow-quote-delim-match

Conversation

@quinnj
Copy link
Copy Markdown
Member

@quinnj quinnj commented Jan 4, 2023

Upon review of Parsers.Options validations (ref), @nickrobinson251, @Drvi, and I agree that this shouldn't be valid. The problem is that there's an ambiguity in the case of a missing field vs. an empty quoted field vs. a quoted field with newlines. For example:

a"b"c
1"2"3   # normal row
"2"3    # 1st field is missing, but ambiguous with 1st field as quoted `"2"`
"1""2"3 # Also ambiguous because the 2nd quote escapes the 3rd, so value is `1"2`

Upon review of Parsers.Options validations ([ref](JuliaData/Parsers.jl#152)),
@nickrobinson251, @Drvi, and I agree that this shouldn't be valid. The problem is that there's
an ambiguity in the case of a _missing_ field vs. an empty quoted field vs. a quoted field
with newlines. For example:

```julia
a"b"c
1"2"3   # normal row
"2"3    # 1st field is missing, but ambiguous with 1st field as quoted `"2"`
"1""2"3 # Also ambiguous because the 2nd quote escapes the 3rd, so value is `1"2`
```
@quinnj quinnj merged commit 2b07821 into main Jan 4, 2023
@quinnj quinnj deleted the jq/disallow-quote-delim-match branch January 4, 2023 14:18
Copy link
Copy Markdown
Collaborator

@nickrobinson251 nickrobinson251 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do you want to release this? i suppose technically it is breaking but given the unintended and undesirable ambiguities that results i suppose we could consider it a bug that this was ever allowed?

@test f.rows == 0

# delim same as quotechar
f = CSV.File(IOBuffer("a\"b\n1\"2\n"); delim='"')
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we replace these with an @test_throws test?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather have the tests in the Parsers.jl package where we do the validation checks

@quinnj
Copy link
Copy Markdown
Member Author

quinnj commented Jan 4, 2023

How do you want to release this? i suppose technically it is breaking but given the unintended and undesirable ambiguities that results i suppose we could consider it a bug that this was ever allowed?

Yeah, I think we have to treat it as a bug.

quinnj added a commit that referenced this pull request Jan 12, 2026
Upon review of Parsers.Options validations ([ref](JuliaData/Parsers.jl#152)),
@nickrobinson251, @Drvi, and I agree that this shouldn't be valid. The problem is that there's
an ambiguity in the case of a _missing_ field vs. an empty quoted field vs. a quoted field
with newlines. For example:

```julia
a"b"c
1"2"3   # normal row
"2"3    # 1st field is missing, but ambiguous with 1st field as quoted `"2"`
"1""2"3 # Also ambiguous because the 2nd quote escapes the 3rd, so value is `1"2`
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants