New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Metabug: Improving C-level coverage #94808
Comments
|
@mdboom: I think a better approach would be to create sub-checklists or a single issue per file with another checklist for the paths that need coverage, instead of opening dozen of issues for each individual path. Even having an issue per file will result in 548 new issues though, so I'm not sure if we want to do that preemptively for each file. I'd say it's better to keep the checklist with the files and paths here, and then directly create PRs for each (or multiple) paths. Another option is to create a project, and handle it there. You can add a new custom field to specify the file, and create draft issues for each path without creating actual issues here (let me know if you need more help with that). |
I thought about that, but most of the uncovered areas will have independent fixes, and bug-per-area sets up that work to happen. I'm starting with files that have seen a lot of changes lately, so we're seeing quite a few issues in them. I suspect most files will not be that way -- most will probably have no issues, and we can just check the box here and not create a flood of issues where they aren't needed.
Agreed.
I'm happy to use a project instead if you'd prefer. IIUC, it's pretty easy to move the existing issues already created into it. I know I can't create a project, but once it's created, I don't know what my limited permissions will allow me to do. |
|
Maybe it would be better to start by trimming down the list to remove files that have no issues, and see how many files are left first. The remaining files and their paths could be listed here, and if someone starts working on them and wants to discuss the approach they could create the issues lazily or directly create PRs that refers to this meta issues if the fix is straightforward enough. I don't think having lot of almost empty issues (like #94817) will help this effort. |
I programmatically removed files that have 100% coverage, and then also (manually) removed anything platform specific and all of
My thinking was this checklist was to say "someone has read through the file and identified all of the potential issues". Some of the issues will have simple tests that can be written, some are dead code, some might reveal bugs, but we need a place to have those discussions and deal with them individually (not here, ideally). For really simple ones, if it's ok to just reference this bug, that's fine by me, but we still need a way to keep track of which files have been vetted to track progress.
Maybe not in general. The ones I've filed so far are directly tied to the faster CPython work, and the ability to move with more confidence there. So there are motivated people who want to close these bugs. But I empathize with the concern of just creating bugs for the sake of creating them. |
This is much better, thanks for doing this!
My idea was to do something like:
Once people start working on these items, there are different options. They can:
The first two options work for straightforward cases, the third works if the problems are similar, and the last for more complex cases that require discussions. You can also do this incrementally, i.e. start with a list like the one above, then convert some to PRs and some to issues as needed. In addition, if you hover with the mouse over a checklist item, a ⨀ icon will appear to the right which will allow you to quickly convert the item to an actual issue if/when you decide to open a discussion about a specific file or problem.
That's great to hear, I just wanted to avoid creating a bunch of issues that might end up sitting there indefinitely :) |
|
Thanks. The revised plan you suggested should work just fine. I didn't know about the feature to automatically open an issue from a checklist item. |
|
@pablogsal, should we be backporting these coverage-improving tests to 3.11? I think we should be consistent in our handling of all of them. I don't think we usually backport test-only PRs, but perhaps these are sort of a special case. |
`bool_new` had no coverage. Automerge-Triggered-By: GH:brandtbucher
|
Test only PRs are generally ok to backport |
…gits is provided (pythonGH-94860) (cherry picked from commit 625ba9b) Co-authored-by: Michael Droettboom <mdboom@gmail.com>
`bool_new` had no coverage. Automerge-Triggered-By: GH:brandtbucher (cherry picked from commit df4d53a) Co-authored-by: Michael Droettboom <mdboom@gmail.com>
|
@mdboom I have several pending PRs to get merged, and after that I propose to update the meta information / coverage data. There will be a lot of changes in coverage. Plus, there are new changes to the source code. What do you think? :) |
|
@sobolevn could you request review from me on these PRs? I'll see if I can work through some of them. |
…honGH-98300) (cherry picked from commit b5f7111) Co-authored-by: Nikita Sobolev <mail@sobolevn.me>
(cherry picked from commit b5f7111) Co-authored-by: Nikita Sobolev <mail@sobolevn.me>
…nGH-99123) (cherry picked from commit c5c4077) Co-authored-by: Nikita Sobolev <mail@sobolevn.me>
(cherry picked from commit c5c4077) Co-authored-by: Nikita Sobolev <mail@sobolevn.me>
…erator is already running (#97672)
|
WIP:
|
mdboom commentedJul 13, 2022
•
edited by bedevere-bot
This bug is going to be used to track work in a other bugs to improve the C-level coverage of the CPython test suite.
There is a set of baseline coverage results on main that can be used to find coverage gaps.
The plan, discussed on discuss.python.org is as follows:
Related work:
There is related work to publish coverage results from CPython on a regular basis, but this issue is concerned with using those results to actually reduce our gaps in coverage.
List of source files:
Include/internal/pycore_asdl.hInclude/internal/pycore_bitutils.hInclude/internal/pycore_call.hInclude/internal/pycore_code.hInclude/internal/pycore_frame.hInclude/internal/pycore_moduleobject.hInclude/internal/pycore_object.hInclude/internal/pycore_pymath.hInclude/internal/pycore_pymem.hInclude/internal/pycore_pystate.hInclude/object.hInclude/pydtrace.hObjects/abstract.cPyBuffer_FromContiguous,PyObject_CopyData,PyBuffer_FillContiguousStridesPyNumber_Checkdoesn't testcomplexPySequence_RepeatandPySequence_InPlaceRepeathave no coveragePySequence_SetItemwith a negative index is untestedPySequence_SetSliceandPySequence_DelSliceare untestedPyMapping_HasKeyandPyMapping_HasKeyStringare untestedObjects/accu.cObjects/boolobject.cObjects/bytearrayobject.cObjects/bytes_methods.cObjects/bytesobject.cInIt's only the case where the underlying libc is broken that isn't tested.PyBytes_FromFormatV, the special handling of%pisn't tested.Objects/call.cPyEval_CallObjectWithKeywordshas no coverage_PyObject_CallMethodId_SizeThas no coverageObjects/capsule.cObjects/cellobject.cObjects/classobject.cObjects/codeobject.c_PyLineTable_StartsLinewas not used #96609Objects/complexobject.cObjects/descrobject.cObjects/dictobject.cdictresizeconvert split table into new combined table" is uncovered._PyDict_GetItemHinthas no coverageObjects/enumobject.cObjects/exceptions.cObjects/fileobject.cPyFile_FromFdhas no coveragePyFile_GetLineoverbytesinput has no coverageObjects/floatobject.cObjects/frameobject.cObjects/funcobject.cPyFunction_GetCode,PyFunction_GetGlobals,PyFunction_GetModule,PyFunction_GetDefaults,PyFunction_SetDefaults,PyFunction_GetKwDefaults,PyFunction_SetKwDefaults,PyFunction_GetClosure,PyFunction_SetClosure,PyFunction_GetAnnotations,PyFunction_SetAnnotations\PyFunction_GetDefaultsandPyFunction_SetDefaults#98449PyFunction_GetCode,PyFunction_GetGlobals,… #98317Objects/genericaliasobject.cObjects/genobject.cgen_new_with_qualnameand APIPyGen_NewWithQualNameandPyGen_Newhave no coverage.PyCoro_Newhas no coveragePyAsyncGen_Newhas no coverageasync_gen_athrow_sendhas poor coverageObjects/interpreteridobject.cObjects/iterobject.cObjects/listobject.cObjects/longobject.c_PyLong_Sing_t_Converterhas no coveragelong_format_binarydoesn't test outputting to UCS2 or UCS4int_bit_length_implandint_bit_count_impldoesn't cover the case where expression overflowsObjects/memoryobject.cinit_sliceis not well-coveredObjects/methodobject.cObjects/moduleobject.cPyModule_GetFilenamehas no coverageObjects/namespaceobject.cObjects/object.cPyObject_Printhas no coveragePyObject_Bytesdoes not test the case where there is a__bytes__PyObject_SetAttrStringdoesn't test when object has atp_setattrPyObject_GetAttrStringdoesn't test when object has atp_getattr_PyObject_LookupAttrdoesn't test when object has atp_getattrObjects/obmalloc.cObjects/odictobject.cObjects/picklebufobject.cPyPickleBuffer_FromObject,PyPickleBuffer_Releasehas no coverageObjects/rangeobject.cObjects/setobject.cObjects/sliceobject.cPySlice_GetIndices/PySlice_GetIndicesExhas no coverageObjects/stringlib/codecs.hObjects/stringlib/count.hObjects/stringlib/ctype.hObjects/stringlib/eq.hObjects/stringlib/fastsearch.hObjects/stringlib/find.hObjects/stringlib/find_max_char.hObjects/stringlib/join.hObjects/stringlib/localeutil.hObjects/stringlib/partition.hObjects/stringlib/replace.hObjects/stringlib/split.hObjects/stringlib/transmogrify.hObjects/stringlib/undef.hObjects/stringlib/unicode_format.hObjects/structseq.cObjects/tupleobject.cObjects/typeobject.cwrap_sq_setitemhas no coverageObjects/unicodectype.cObjects/unicodeobject.cxmlcharrefreplacedoesn't test for codepoints < 100 (This seems almost impossible to occur).resize_inplacehas no coverageunicode_kind_namewhen!PyUnicode_IS_COMPACTisn't covered -- low priority used by consistency check onlyunicode_write_cstrdoesn't test writing into UCS2 or UCS4%pinPyUnicode_FromFormat#96677PyUnicode_AsDecodedObject,PyUnicode_AsDecodedUnicode,PyUnicode_AsEncodedObject,PyUnicode_AsEncodedUnicodehas no coverage_Py_DecodeUTF8Exand_Py_EncodeUTF8Exhas no coverage forerror == surrogateescapePyUnicode_BuildEncodingMapdoesn't handle theneed_dictcaseucs1lib_find_sliceanducs1lib_rfind_slicearen't covered.PyUnicode_Counthas no coveragestr.rsplitfor UCS1, UCS2 or UCS4 #98228PyUnicode_CompareWithASCIIStringhas no coverage for comparing with UCS2 or UCS4_PyUnicode_EqualToASCIIIdhas no coverageObjects/unicodetype_db.hObjects/unionobject.cObjects/weakrefobject.cParser/action_helpers.c_PyPegen_set_expr_contextdoesn't cover "starred kind"_PyPegen_get_expr_nameswitch statement coverage is non-exhaustiveParser/myreadline.c(N/A Windows-only)Parser/parser.cParser/peg_api.cParser/pegen.cParser/pegen.hParser/pegen_errors.cmock.Mock()._is_coroutine = Falseworkarounds #94926Parser/string_parser.cParser/tokenizer.cGenerated codePython/Python-ast.cPython/Python-tokenize.cPython/_warnings.cshow_warningdoesn't cover the case where there is asourceline.PyErr_WarnExplicithas no coveragePython/asdl.cPython/ast.censure_literal_*functions aren't coveredvalidate_pattern_match_valuedoesn't cover all elements of switchPython/ast_opt.ccheck_complexitydoesn't cover thefrozensetcaseast_foldbodyisn't coveredPython/ast_unparse.cPython/bltinmodule.cPython/bootstrap_hash.cPython/ceval.cPyEval_AquireLockandPyEval_ReleaseLockare uncoveredSTORE_ATTR_WITH_HINTdoesn't cover the case where the dictionary doesn't have Unicode keysCALL_FUNCTION_EXdoesn't cover the case where kwargs is not an exact dictPyEval_EvalCodeExdoesn't cover the case where kwargs are passed inPyEval_GetFramehas no coveragePyEval_GetFuncDescfunction #98300Python/ceval_gil.hPython/codecs.cPython/compile.cPython/condvar.hPython/context.cPyContext_Copy,PyContext_Enter,PyContext_Exithave no coveragePython/deepfreeze/deepfreeze.cPython/dtoa.cPython/dup2.cPython/dynamic_annotations.cPython/errors.cPython/fileutils.cis_valid_wide_chardoesn't test error branchesencode_ascii/decode_asciihas no coverage (probably very low priority -- comment says only for platforms with a broken mbstowcs (FreeBSD, OpenIndiana)_Py_stathas no coveragePython/formatter_unicode.cPython/frame.cPython/frozenmain.cPython/future.cPython/getargs.cPython/getopt.cPython/hamt.cPython/hashtable.cPython/import.cPython/importdl.cPython/initconfig.cPython/marshal.cPython/modsupport.cPython/mysnprintf.cPython/mystrtoul.cPython/pathconfig.cPython/preconfig.cPython/pyarena.cPython/pyfpe.cPython/pyhash.cPython/pylifecycle.cPython/pystate.cPython/pystrcmp.cPython/pystrhex.cPython/pystrtod.cPython/pythonrun.cPython/pytime.cPython/specialize.cPython/structmember.cPython/suggestions.cPython/symtable.cPython/sysmodule.cPython/thread.cPython/traceback.cdepth > limit_PyTraceBack_Print_Indentedwith overflowingtracebacklimit_Py_DumpDecimal,_Py_DumpHexadecimal,_Py_DumpASCII,dump_frame,dump_traceback,_Py_DumpTraceback,write_thread_id,_Py_DumpTracebackThreads-- possibly they have tests which are disabled under some circumstances.LOAD_GLOBALfor custom dict subtypes #96767LOAD_GLOBALfor custom dict subtypes (GH-96767) #99091PySequence_{Set,Del}Slice#99123PyEval_GetFuncDescfunction #98300PyFunction_GetKwDefaultsandPyFunction_SetKwDefaults#98809PyEval_GetFuncDescfunction (GH-98300) #99126PySequence_{Set,Del}Slice(GH-99123) #99133PySequence_[InPlace_]Repeat#99196PySequence_[InPlace]Concat#99319PyFunction_{Get,Set}Closure#99429The text was updated successfully, but these errors were encountered: