For the first experiment, we have written and compiled a set of programs that allow us to measure and compare the accuracy of flawfinder, cwe_checker, cppcheck<\/em>, and WPA. Each binary program contains various function definitions, all of which use the memcpy<\/code> <\/em>function. The memcpy<\/code> <\/em>function is potentially dangerous because it can result in a buffer overflow\/segfault. This can occur if the memory in a large buffer is copied into a smaller buffer. If memcpy<\/code> <\/em>is used properly, it will not result in a buffer overflow\/segfault. It is up to the developer to ensure proper usage of memcpy<\/code><\/em>. Developer error, however, can easily result in a vulnerable usage of memcpy<\/code><\/em>.<\/p>\nThe source code presented tot he right illustrates one of these programs. In it, one function is unsafe and results in a segfault, while the other two are safe.<\/p>\n
In each sample binary program, all the functions except one are safe (they do not result in memory corruption via a buffer overflow). The size of each program is defined by the number of safe-to-use functions it defines. For example, size 10 indicates that the program contains 10 safe functions and 1 unsafe function (12 functions in total, when including main()<\/code><\/em>). Each program calls all its defined functions in order, with the unsafe function being called last.<\/p>\nAlthough each program has some number of potentially dangerous functions, they each only have a single use of memcpy<\/code> <\/em>that can result in a vulnerability\/segfault. A surface level analysis would be incapable of distinguishing between a safe use of memcpy<\/code><\/em>, and a dangerous use of memcpy<\/code><\/em>.<\/p>\nFor each of these programs, we ran flawfinder, cwe_checker, cppcheck,<\/em> and WPA on either the binary or source code representation of the file (depending on the kind of input required by the tool). We recorded runtimes, total reported vulnerabilities\/weaknesses, and accuracy rates. This data is summarized in the tables below.<\/p>\n<\/div><\/div><\/div>\n
\r\n#include <stdio.h>\r\n#include <stdlib.h>\r\n#include <string.h>\r\n\r\nconst int LARGE = 131072;\r\nconst int SMALL = 4;\r\n\r\n\/\/ causes a heap overflow and segfault\r\nvoid heap_overflow() {\r\n printf(\"Performing unsafe heap operation...\\n\");\r\n char* src = (char*)malloc(sizeof(char) * LARGE);\r\n char* dst = (char*)malloc(sizeof(char) * SMALL);\r\n memcpy(dst, src, LARGE);\r\n free(src);\r\n free(dst);\r\n}\r\n\r\n\/\/ performs a heap safe operation\r\nvoid heap_safe_1() {\r\n printf(\"Performing safe heap operation...\\n\");\r\n char* src = (char*)malloc(sizeof(char) * SMALL);\r\n char* dst = (char*)malloc(sizeof(char) * SMALL);\r\n memcpy(dst, src, SMALL);\r\n free(src);\r\n free(dst);\r\n}\r\n\r\n\/\/ performs a heap safe operation\r\nvoid heap_safe_2() {\r\n printf(\"Performing safe heap operation...\\n\");\r\n char* src = (char*)malloc(sizeof(char) * SMALL);\r\n char* dst = (char*)malloc(sizeof(char) * SMALL);\r\n memcpy(dst, src, SMALL);\r\n free(src);\r\n free(dst);\r\n}\r\n\r\nint main() {\r\n heap_safe_1();\r\n heap_safe_2();\r\n heap_overflow();\r\n}\r\n\r\n<\/code><\/pre>\n<\/div>\n<\/div><\/div><\/div><\/div><\/div>\n
\n
\n\n\nSize (# of functions)<\/em><\/span><\/td>\nRuntime (sec)<\/em><\/span><\/td>\nTotal Reported Items<\/em><\/span><\/td>\nVulnerable Reported Items<\/em><\/span><\/td>\nAccuracy<\/em><\/span><\/td>\n<\/tr>\n\n| 5<\/span><\/td>\n | 0.031559<\/span><\/td>\n | 6<\/span><\/td>\n | 1<\/span><\/td>\n | 16.67%<\/span><\/td>\n<\/tr>\n\n| 10<\/span><\/td>\n | 0.030592<\/span><\/td>\n | 11<\/span><\/td>\n | 1<\/span><\/td>\n | 9.09%<\/span><\/td>\n<\/tr>\n\n| 20<\/span><\/td>\n | 0.034237<\/span><\/td>\n | 21<\/span><\/td>\n | 1<\/span><\/td>\n | 4.76%<\/span><\/td>\n<\/tr>\n\n| 50<\/span><\/td>\n | 0.0363287<\/span><\/td>\n | 51<\/span><\/td>\n | 1<\/span><\/td>\n | 1.96%<\/span><\/td>\n<\/tr>\n\n| 100<\/span><\/td>\n | 0.044598<\/span><\/td>\n | 101<\/span><\/td>\n | 1<\/span><\/td>\n | 0.99%<\/span><\/td>\n<\/tr>\n\n| 500<\/span><\/td>\n | 0.098342<\/span><\/td>\n | 501<\/span><\/td>\n | 1<\/span><\/td>\n | 0.20%<\/span><\/td>\n<\/tr>\n\n| 721<\/span><\/td>\n | 0.12650418<\/span><\/td>\n | 722<\/span><\/td>\n | 1<\/span><\/td>\n | 0.14%<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n Table 1 \u2013 flawfinder Accuracy Results<\/span><\/p>\n<\/div>\n\n\n \n\n\nSize (# of functions)<\/em><\/span><\/td>\nRuntime (sec)<\/em><\/span><\/td>\nTotal Reported Items<\/em><\/span><\/td>\nVulnerable Reported Items<\/em><\/span><\/td>\nAccuracy<\/em><\/span><\/td>\n<\/tr>\n\n| 5<\/span><\/td>\n | 0.0074203<\/span><\/td>\n | 9<\/span><\/td>\n | 2<\/span><\/td>\n | 22.22%<\/span><\/td>\n<\/tr>\n\n| 10<\/span><\/td>\n | 0.009376<\/span><\/td>\n | 14<\/span><\/td>\n | 2<\/span><\/td>\n | 14.29%<\/span><\/td>\n<\/tr>\n\n| 20<\/span><\/td>\n | 0.0119152<\/span><\/td>\n | 24<\/span><\/td>\n | 2<\/span><\/td>\n | 8.33%<\/span><\/td>\n<\/tr>\n\n| 50<\/span><\/td>\n | 0.02241158<\/span><\/td>\n | 54<\/span><\/td>\n | 2<\/span><\/td>\n | 3.70%<\/span><\/td>\n<\/tr>\n\n| 100<\/span><\/td>\n | 0.0396542<\/span><\/td>\n | 104<\/span><\/td>\n | 2<\/span><\/td>\n | 1.92%<\/span><\/td>\n<\/tr>\n\n| 500<\/span><\/td>\n | 0.261377<\/span><\/td>\n | 504<\/span><\/td>\n | 2<\/span><\/td>\n | 0.40%<\/span><\/td>\n<\/tr>\n\n| 721<\/span><\/td>\n | 0.476597<\/span><\/td>\n | 725<\/span><\/td>\n | 2<\/span><\/td>\n | 0.28%<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n Table 3 \u2013 cppcheck Accuracy Results<\/p>\n<\/div>\n<\/div><\/div> \n \n \n\n\nSize (# of functions)<\/em><\/span><\/td>\nRuntime (sec)<\/em><\/span><\/td>\nTotal Reported Items<\/em><\/span><\/td>\nVulnerable Reported Items<\/em><\/span><\/td>\nAccuracy<\/em><\/span><\/td>\n<\/tr>\n\n| 5<\/span><\/td>\n | 6.8047869<\/span><\/td>\n | 19<\/span><\/td>\n | 2<\/span><\/td>\n | 10.53%<\/span><\/td>\n<\/tr>\n\n| 10<\/span><\/td>\n | 7.09194<\/span><\/td>\n | 34<\/span><\/td>\n | 2<\/span><\/td>\n | 5.88%<\/span><\/td>\n<\/tr>\n\n| 20<\/span><\/td>\n | 7.124346<\/span><\/td>\n | 64<\/span><\/td>\n | 2<\/span><\/td>\n | 3.13%<\/span><\/td>\n<\/tr>\n\n| 50<\/span><\/td>\n | 7.5739443<\/span><\/td>\n | 154<\/span><\/td>\n | 2<\/span><\/td>\n | 1.30%<\/span><\/td>\n<\/tr>\n\n| 100<\/span><\/td>\n | 8.4504086<\/span><\/td>\n | 304<\/span><\/td>\n | 2<\/span><\/td>\n | 0.66%<\/span><\/td>\n<\/tr>\n\n| 500<\/span><\/td>\n | 14.143265<\/span><\/td>\n | 1504<\/span><\/td>\n | 2<\/span><\/td>\n | 0.13%<\/span><\/td>\n<\/tr>\n\n| 721<\/span><\/td>\n | 17.14055538<\/span><\/td>\n | 2167<\/span><\/td>\n | 2<\/span><\/td>\n | 0.09%<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n Table 2 \u2013 cwe_checker Accuracy Results<\/span><\/p>\n<\/div>\n\n\n \n\n\nSize (# of functions)<\/em><\/span><\/td>\nRuntime (sec)<\/em><\/span><\/td>\nTotal Reported Items<\/em><\/span><\/td>\nVulnerable Reported Items<\/em><\/span><\/td>\nAccuracy<\/em><\/span><\/td>\n<\/tr>\n\n| 5<\/span><\/td>\n | 56.37603<\/span><\/td>\n | 1<\/span><\/td>\n | 1<\/span><\/td>\n | 100%<\/span><\/td>\n<\/tr>\n\n| 10<\/span><\/td>\n | 63.9488<\/span><\/td>\n | 1<\/span><\/td>\n | 1<\/span><\/td>\n | 100%<\/span><\/td>\n<\/tr>\n\n| 20<\/span><\/td>\n | 74.829524<\/span><\/td>\n | 1<\/span><\/td>\n | 1<\/span><\/td>\n | 100%<\/span><\/td>\n<\/tr>\n\n| 50<\/span><\/td>\n | 89.03786<\/span><\/td>\n | 1<\/span><\/td>\n | 1<\/span><\/td>\n | 100%<\/span><\/td>\n<\/tr>\n\n| 100<\/span><\/td>\n | 108.076774<\/span><\/td>\n | 1<\/span><\/td>\n | 1<\/span><\/td>\n | 100%<\/span><\/td>\n<\/tr>\n\n| 500<\/span><\/td>\n | 292.4567799<\/span><\/td>\n | 1<\/span><\/td>\n | 1<\/span><\/td>\n | 100%<\/span><\/td>\n<\/tr>\n\n| 721<\/span><\/td>\n | 312.5943696<\/span><\/td>\n | 1<\/span><\/td>\n | 1<\/span><\/td>\n | 100%<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n Table 4 \u2013 Weak Pointers Accuracy Results<\/p>\n<\/div>\n<\/div><\/div><\/div><\/div> Accuracy is measured as the number of vulnerable reported items divided by the total number of reported items. You can think of this as a measure of the “noisiness” of each tool. Out of everything the tool reports, how much of it reflects a truly troublesome vulnerability.<\/p>\n As an example of what the static analyzers output, consider the output shown on the right from flawfinder when ran against the size 500 program.<\/p>\n<\/div><\/div><\/div> ![](\"https:\/\/i0.wp.com\/objectsecurity.com\/wp-content\/uploads\/2024\/05\/Picture16.png?resize=624%2C172&ssl=1\" "\\"Picture16\\"") <\/span><\/div><\/div><\/div>Flawfinder<\/em>, as well as each of the other static analyzers, reported all uses of memcpy<\/code> <\/em>as being unsafe, even though only one of them was truly unsafe. Some of the static analyzers also reported the same unsafe usage of memcpy<\/code> <\/em>twice, hence the 2<\/em>\u00a0in some the Vulnerable Reported Items<\/em> columns in the tables above.<\/p>\n | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |