Skip to content

Test whether function should use Yields instead of Returns#7258

Open
eriknw wants to merge 23 commits intonetworkx:mainfrom
eriknw:yields_not_returns
Open

Test whether function should use Yields instead of Returns#7258
eriknw wants to merge 23 commits intonetworkx:mainfrom
eriknw:yields_not_returns

Conversation

@eriknw
Copy link
Contributor

@eriknw eriknw commented Jan 30, 2024

This is an issue in PR form (but maybe it could be merged since it serves as a test?)

This is a more thorough report (as a followup to #7252 and #7253) of functions that perhaps ought to use Yields instead of Returns in their docstrings. I actually don't know the best recommended practice here--is there a difference whether a function uses e.g. for x in spam: yield x or yield from spam compared to e.g. return (x for x in spam)?

Context: I was poking around return types in #7191, and it would have been helpful to easily know (using grep) which functions return iterators.

@dschult
Copy link
Member

dschult commented Feb 7, 2024

When would this "test" get run? Not during CI?

I guess this means that for each new generator added, we should update this list in this file?
It looks like there is the start of a list of functions that "return" also... Is that a precursor to another list of functions?

@eriknw
Copy link
Contributor Author

eriknw commented Feb 7, 2024

This currently gets tested during the "test / dispatch" CI run. It actually inspects the return type.

And, yeah, as docs get updated, the sets will need updated too. The error message will tell you what to do.

should_be_returns set is for functions that use Yields but should use Returns, of which there are none.

@dschult
Copy link
Member

dschult commented Feb 8, 2024

I see -- this is just a list of the function with docs that have it wrong... :)
Somehow I thought it was a list of all functions that yield... So, now I understand Ross's remarks that once we get this fixed up, it will be easier to manage. :)

@rossbar
Copy link
Contributor

rossbar commented Feb 11, 2024

I went through the exercise of testing out what the numpydoc validation checks would say about this by turning them on during the sphinx build. This can be achieved by adding the following two lines1 to the conf.py:

numpydoc_validation_checks = {"YD01"}                                           
numpydoc_validation_exclude = [r".coreviews."]

Running the docbuild then spits out an additional 94 warnings - see details below

Numpydoc validation log: return should be yield
WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.assortativity.node_attribute_xy':
  YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.assortativity.node_degree_xy':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.coloring.strategy_connected_sequential':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.coloring.strategy_independent_set':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.coloring.strategy_saturation_largest_first':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.community.kclique.k_clique_communities':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.community.centrality.girvan_newman':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.connectivity.kcutsets.all_node_cuts':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.connectivity.disjoint_paths.edge_disjoint_paths':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.connectivity.disjoint_paths.node_disjoint_paths':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.euler.eulerian_circuit':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.connectivity.edge_kcomponents.EdgeComponentAuxGraph.k_edge_components':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.connectivity.edge_kcomponents.EdgeComponentAuxGraph.k_edge_subgraphs':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.isomorphism.ISMAGS.isomorphisms_iter':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.assortativity.node_attribute_xy':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.assortativity.node_degree_xy':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.coloring.strategy_connected_sequential':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.coloring.strategy_independent_set':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.coloring.strategy_saturation_largest_first':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.community.centrality.girvan_newman':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.community.kclique.k_clique_communities':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.connectivity.disjoint_paths.edge_disjoint_paths':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.connectivity.disjoint_paths.node_disjoint_paths':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.connectivity.edge_kcomponents.EdgeComponentAuxGraph.k_edge_components':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.connectivity.edge_kcomponents.EdgeComponentAuxGraph.k_edge_subgraphs':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.connectivity.kcutsets.all_node_cuts':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.euler.eulerian_circuit':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.isomorphism.DiGraphMatcher.candidate_pairs_iter':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.isomorphism.DiGraphMatcher.isomorphisms_iter':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.isomorphism.DiGraphMatcher.match':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.isomorphism.DiGraphMatcher.subgraph_isomorphisms_iter':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.isomorphism.DiGraphMatcher.subgraph_monomorphisms_iter':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.isomorphism.GraphMatcher.candidate_pairs_iter':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.isomorphism.GraphMatcher.isomorphisms_iter':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.isomorphism.GraphMatcher.match':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.isomorphism.GraphMatcher.subgraph_isomorphisms_iter':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.isomorphism.GraphMatcher.subgraph_monomorphisms_iter':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.isomorphism.ISMAGS.isomorphisms_iter':
YD01: No Yields section found

/home/ross/repos/networkx/networkx/algorithms/operators/product.py:docstring of networkx.algorithms.operators.product.modular_product:16: WARNING: Literal block ends without a blank line; unexpected unindent.
WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.shortest_paths.unweighted.all_pairs_shortest_path':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.shortest_paths.unweighted.all_pairs_shortest_path_length':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.shortest_paths.weighted.all_pairs_bellman_ford_path':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.shortest_paths.weighted.all_pairs_bellman_ford_path_length':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.shortest_paths.weighted.all_pairs_dijkstra_path':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.shortest_paths.weighted.all_pairs_dijkstra_path_length':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.similarity.generate_random_paths':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.similarity.optimize_edit_paths':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.similarity.optimize_graph_edit_distance':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.simple_paths.all_simple_edge_paths':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.simple_paths.all_simple_paths':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.traversal.breadth_first_search.bfs_predecessors':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.traversal.breadth_first_search.bfs_successors':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.traversal.depth_first_search.dfs_labeled_edges':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.isomorphism.GraphMatcher.isomorphisms_iter':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.isomorphism.GraphMatcher.subgraph_isomorphisms_iter':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.isomorphism.GraphMatcher.subgraph_monomorphisms_iter':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.isomorphism.GraphMatcher.candidate_pairs_iter':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.isomorphism.GraphMatcher.match':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.isomorphism.DiGraphMatcher.isomorphisms_iter':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.isomorphism.DiGraphMatcher.subgraph_isomorphisms_iter':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.isomorphism.DiGraphMatcher.subgraph_monomorphisms_iter':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.isomorphism.DiGraphMatcher.candidate_pairs_iter':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.isomorphism.DiGraphMatcher.match':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.shortest_paths.unweighted.all_pairs_shortest_path':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.shortest_paths.unweighted.all_pairs_shortest_path_length':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.shortest_paths.weighted.all_pairs_dijkstra_path':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.shortest_paths.weighted.all_pairs_dijkstra_path_length':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.shortest_paths.weighted.all_pairs_bellman_ford_path':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.shortest_paths.weighted.all_pairs_bellman_ford_path_length':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.similarity.optimize_graph_edit_distance':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.similarity.optimize_edit_paths':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.similarity.generate_random_paths':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.simple_paths.all_simple_paths':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.simple_paths.all_simple_edge_paths':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.traversal.depth_first_search.dfs_labeled_edges':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.traversal.breadth_first_search.bfs_predecessors':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.algorithms.traversal.breadth_first_search.bfs_successors':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.classes.function.non_edges':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.classes.function.non_edges':
YD01: No Yields section found

/home/ross/repos/networkx/networkx/generators/nonisomorphic_trees.py:docstring of networkx.generators.nonisomorphic_trees.nonisomorphic_trees:22: ERROR: Unexpected indentation.
WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.utils.rcm.cuthill_mckee_ordering':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.readwrite.adjlist.generate_adjlist':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.readwrite.edgelist.generate_edgelist':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.readwrite.adjlist.generate_adjlist':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.readwrite.edgelist.generate_edgelist':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.readwrite.gexf.generate_gexf':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.readwrite.gml.generate_gml':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.readwrite.graphml.generate_graphml':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.readwrite.multiline_adjlist.generate_multiline_adjlist':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.readwrite.pajek.generate_pajek':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.readwrite.gexf.generate_gexf':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.readwrite.gml.generate_gml':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.readwrite.graphml.generate_graphml':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.readwrite.multiline_adjlist.generate_multiline_adjlist':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.readwrite.pajek.generate_pajek':
YD01: No Yields section found

WARNING: [numpydoc] Validation warnings while processing docstring for 'networkx.utils.rcm.cuthill_mckee_ordering':
YD01: No Yields section found

The conclusion I draw from the above experiment is that numpydoc validation is sufficient to catch these errors. The course of action that I'd propose is to:

  1. Address these 94 cases, then
  2. Add those two numpydoc validation lines to the sphinx build conf as a test for new docstrings

Footnotes

  1. The second line is necessary due to a bug in numpydoc when dealing with multiple generations of inheritance

@eriknw
Copy link
Contributor Author

eriknw commented Feb 12, 2024

Thanks for running that experiment @rossbar. I'm glad to be able to compare.

The conclusion I draw from the above experiment is that numpydoc validation is sufficient to catch these errors.

I think that's the wrong conclusion.

First, your count is wrong. It found 47, not 94 (it reports each one twice?).

Second, it appears to miss about a two-thirds of the functions identified in this PR, so it actually doesn't do a very good job (why not?!):

adamic_adar_index
all_pairs_all_shortest_paths
all_shortest_paths
all_triads
all_triplets
asyn_fluidc
asyn_lpa_communities
attracting_components
biconnected_component_edges
biconnected_components
bridge_components
cn_soundarajan_hopcroft
common_neighbor_centrality
compute_v_structures
connected_components
dfs_postorder_nodes
dfs_preorder_nodes
edge_boundary
enumerate_all_cliques
fast_label_propagation_communities
find_cliques
find_cliques_recursive
isolates
jaccard_coefficient
k_edge_components
k_edge_subgraphs
kosaraju_strongly_connected_components
maximum_spanning_edges
minimum_spanning_edges
preferential_attachment
ra_index_soundarajan_hopcroft
resource_allocation_index
shortest_path
shortest_path_length
shortest_simple_paths
single_source_all_shortest_paths
strongly_connected_components
strongly_connected_components_recursive
tree_all_pairs_lowest_common_ancestor
weakly_connected_components
within_inter_cluster

TBH, I would expect it to find more.

Third, it's a more relaxed check than done in this PR. It looks if "Yields" is missing regardless if "Returns" is present. This PR could trivially be updated to also check for no "Yields" regardless of "Returns". However, I think it's less work (and more important) to change "Returns" to "Yields" vs. adding a new "Yields" section, but of course the latter could be done too.

@eriknw
Copy link
Contributor Author

eriknw commented Feb 12, 2024

Actually, I take back what I said. There is a difference between yielding and returning an iterator. Consider:

def f1():
    raise ValueError
    yield from [1, 2]

def f2():
    raise ValueError
    yield 1
    yield 2

def f3():
    raise ValueError
    return iter([1, 2])

f1()
f2()
f3()

Do you know what happens?

So, there is a difference whether we check isinstance(result, Iterator) or isinstance(result, Generator). The latter is more strict. Stylistically, I really don't know whether "Yields" or "Returns" is preferred if an iterator but not a generator is returned. To me, this distinction feels like an implementation detail, and I think "Yields" is a more helpful description. I think it's actually really nice to raise early and not when trying to get the first item in the generator, so I would prefer to return an iterator when given a choice.

@eriknw
Copy link
Contributor Author

eriknw commented Feb 12, 2024

Alright, I grouped the function names by whether they are generator functions, return generators, or return iterators. Generator functions is the largest group and should definitely use "Yields" (also, numpydoc validation misses a lot of these).

@rossbar
Copy link
Contributor

rossbar commented Feb 12, 2024

Second, it appears to miss about a two-thirds of the functions identified in this PR, so it actually doesn't do a very good job (why not?!):

Thanks @eriknw for checking this, I never actually compared to the list here 🙃 . Indeed you're right, there are quite a few generators missed by the numpydoc validation.

Diving a little deeper --- the numpydoc validation is implemented with inspect.isgeneratorfunction. I would definitely expect this to return True for a lot of the listed generators that it's currently missing. Take for example nx.all_triads:

>>> import inspect
>>> import networkx as nx
>>> inspect.isgeneratorfunction(nx.all_triads)  # Expect true
False

I think the decorators are the culprit. A simple test is to define an un-decorated version of all_triads to verify that isgeneratorfunction works as expected:

>>> def mytriads(G):
...:     triplets = combinations(G.nodes(), 3)
...:     for triplet in triplets:
...:         yield G.subgraph(triplet).copy()
>>> inspect.isgeneratorfunction(mytriads)
True

Now, why this is happening for some of these functions but not all (since everything in NetworkX is decorated with nx._dispatchable) I haven't dug into yet. Maybe it has something to do with stacked decorators? The ordering of the stacking? Perhaps some of the decorators aren't properly functools.wraps-ed? More investigation needed, but I think this is the right tack!

Stylistically, I really don't know whether "Yields" or "Returns" is preferred if an iterator but not a generator is returned. To me, this distinction feels like an implementation detail, and I think "Yields" is a more helpful description.

Agreed - I think "Yields" is appropriate even in the return <generator> case as what really matters is how the items in the iterable are created (i.e. are they already stored in memory or computed on the fly).

@eriknw
Copy link
Contributor Author

eriknw commented Feb 12, 2024

👍

It looks like inspect.isgeneratorfunction is actually pretty limited. It doesn't work on wrapped functions. It only unwraps __func__ attribute of methods and partial objects. Notably, it ignores __wrapped__. It is correct for us to use __wrapped__ and not __func__. Hence, I think inspect.isgeneratorfunction is fundamentally limited for this use case, and therefore so is numpydoc validation.

Perhaps numpydoc could be updated to unwrap __wrapped__ before calling inspect.isgeneratorfunction, which should make it work better for us.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

3 participants