{"id":1210,"date":"2018-10-30T16:24:15","date_gmt":"2018-10-30T23:24:15","guid":{"rendered":"http:\/\/cknotes.com\/?p=1210"},"modified":"2020-05-04T05:21:00","modified_gmt":"2020-05-04T12:21:00","slug":"never-handle-non-text-binary-data-as-a-string","status":"publish","type":"post","link":"https:\/\/cknotes.com\/never-handle-non-text-binary-data-as-a-string\/","title":{"rendered":"Never Handle non-Text Binary Data as a String"},"content":{"rendered":"

The bytes of a binary file, such as a JPG, PDF, etc. should never be treated as a string.\u00a0 Loading a binary file into a string, and then saving back to a binary file will surely result in a different file that is corrupted. \u00a0 This rule should be followed for all programming languages.\u00a0 Don’t treat binary non-text bytes as text characters.<\/strong><\/p>\n

Consider this Visual FoxPro code:<\/p>\n

LOCAL x\r\nx = FILETOSTR( \"in.pdf\" )\r\nSTRTOFILE( x, \"out.pdf\" )\r\n<\/pre>\n
It is likely that out.pdf is corrupt. Here’s why:<\/p>\n
When reading a text file, the bytes must be interpreted according to some character encoding.<\/p>\n
For example, consider this character: \u00c9
\nIn the windows-1252 character encoding, it is represented by a single byte: 0xC9
\nIn the utf-8 character encoding, it is represented by a two bytes: 0xC3 0x89
\nIn the utf-16 character encoding, it is represented by a two bytes: 0x00 0xC9<\/p>\n
In this case, FoxPro is probably assuming ANSI (i.e. Windows-1252 for USA\/Western European computers, 1 byte per char). Internally, FoxPro most likely holds strings in the utf-16 byte representation. Therefore, each incoming ANSI byte is converted to 2-byte per char utf-16.<\/p>\n
Now have a look at the Windows-1252 charset:<\/p>\n
\"windows-1252\"<\/a><\/p>\n
Notice the “NOT USED” bytes, such as 8D, 9D, 8E, 9E, etc.<\/p>\n
These byte values will never appear in valid Windows-1252 text.\u00a0 However, they will likely appear in a binary non-text file.\u00a0 If the binary file is large enough, you can be sure these bytes will be present.\u00a0 They’ll likely get converted to a “?” char.\u00a0 That’s why you see “?” or some other standard char when non-text is loaded.<\/p>\n
When you write the text back to the file, all of the “NOT USED” bytes are written as “?” chars.\u00a0 This is the corruption.\u00a0 By trying to handle binary data as text, incoming bytes are implicitly converted to the byte representation used to hold strings (likely utf-8 or utf-16).\u00a0 Writing the file (in the case of STRTOFILE) involves an implicit conversion to the 1-byte per char ANSI representation.<\/p>\n
The round-trip of ANSI –> Internal Representation –> ANSI corrupts the data.<\/p>\n
Also, it doesn’t matter what charsets are involved.\u00a0 It could be utf-8, utf-16, etc.\u00a0 Reading a text file implicitly involves interpreting bytes according to a charset, and if those bytes don’t actually represent text in the given charset, impossible byte values or byte sequences will be present that cause some sort of error char to be substituted (or the error sequences are simply dropped), and the round-trip always results in corruption.<\/p>\n
The rule that should never be broken is: never treat binary data as text.\u00a0 Don’t use string data types to hold non-text binary data.<\/p>\n
 <\/p>\n","protected":false},"excerpt":{"rendered":"
The bytes of a binary file, such as a JPG, PDF, etc. should never be treated as a string.\u00a0 Loading a binary file into a string, and then saving back to a binary file will surely result in a different file that is corrupted. \u00a0 This rule should be followed for all programming languages.\u00a0 Don’t […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[752],"tags":[666,664,665,667],"class_list":["post-1210","post","type-post","status-publish","format-standard","hentry","category-character-encoding","tag-binary","tag-filetostr","tag-strtofile","tag-text"],"_links":{"self":[{"href":"https:\/\/cknotes.com\/wp-json\/wp\/v2\/posts\/1210","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cknotes.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cknotes.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cknotes.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/cknotes.com\/wp-json\/wp\/v2\/comments?post=1210"}],"version-history":[{"count":1,"href":"https:\/\/cknotes.com\/wp-json\/wp\/v2\/posts\/1210\/revisions"}],"predecessor-version":[{"id":1212,"href":"https:\/\/cknotes.com\/wp-json\/wp\/v2\/posts\/1210\/revisions\/1212"}],"wp:attachment":[{"href":"https:\/\/cknotes.com\/wp-json\/wp\/v2\/media?parent=1210"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cknotes.com\/wp-json\/wp\/v2\/categories?post=1210"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cknotes.com\/wp-json\/wp\/v2\/tags?post=1210"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}