Still a WIP. Need to decide how much of the *.pxd/*.h/*.so to expose (and commit to supporting, since exposing it means it is now publc). For instance, we currently do not build a distributions pxd file c-extension module so it cannot be used in cimport from cython, and it seems the src/aligned_malloc files do not really belong to random.

Edit: pxd -> c-extension module

xref gh-5312 for moving the aligned allocations into numpy itself

Sorry this is a bit long for a PR comment. Should it be an issue?

Line 75 says "An example ... is in the examples folder" which is part of the sources under numpy.random but not distributed in the wheel.

It contains examples of using the BitGenerator interfaces next_double and next_uint32 and the Generator from cython. To support this via cimport, we should ship a _bit_generator.pxd (which we already do) and a _generator.pxd (TBD). Can we remove the use of PyCapsule since we can directly cimport the BitGenerators?

I am not a numba expert. One of the examples is very close to the one in the exending.rst document, so I think is covered. The other in that directory seems to be a cffi example, not a numba one. It is simple enough we could put it in the extending.rst document.

TBD:

• add a _generator.pxd file for cython
  • with or without an accompanying distributions.h file? If with then where should it live?
• add a cffi example to extending.rst
• change the examples to not use PyCapsule (is this possible and advisable?)

Are we answering the user stories in these examples?

One of the mailing list comments linked to the scipy.special documentation, should we try to create a page like that?

User stories - we should try to relate to these (from the mailing list thread)

Someone asked how to use random in a ufunc.

"An ideal API would allow projects like https://github.com/deepmind/torch-randomkit/tree/master/randomkit or numba to consume the code in NumPy without vendoring it."

"There are c++ applications which use boost::random, would be nice to be able to swap it for numpy.random."

"Using the existing distributions from Cython was a requested feature and an explicit goal, yes. There are users waiting for this." (mattip: but isn't this supported via Generator?)

"Numba would definitely appreciate C functions to access the random distribution implementations, and have a side-project (numba-scipy) that is making the Cython wrapped functions in SciPy visible to Numba" (mattip: I think we do this after gh-14608 from _generator.so via these cdef extern declarations, but that requires the H file.

Maybe someone who wants to write a new BitGenerator, i.e., the numpy/bitgenerator repo without needing to have the numpy code as a git submodule

Sorry this is a bit long for a PR comment. Should it be an issue?

Might indeed make sense to merge this quickly (there's only some doc fixes) here that improve the current state of master), and then use a new issue for the rest.

An example ... is in the examples folder

Good point. I'd suggest moving the examples into the documentation, shipping them in the tarball and including them in the rendered docs as well.

One of the mailing list comments linked to the scipy.special documentation, should we try to create a page like that?

I think so. It's no different than any other public function in the Python and C API: every function and object should be listed.

User stories - we should try to relate to these

Nice collection. Yes, those would make good examples/subsections in the user guide section on this topic.

The cython code does not compile, it needs include/bitgen.h. Now we need to decide how (and if) to add these include files, or if there is a way to work around it

Something is fishy with macos Azure runs. It is printing:

Installed /Users/runner/hostedtoolcache/Python/3.6.9/x64/lib/python3.6/site-packages/numpy-1.18.0.dev0+20c8d7c-py3.6-macosx-10.14-x86_64.egg
Processing dependencies for numpy==1.18.0.dev0+20c8d7c
Searching for numpy==1.18.0.dev0+20c8d7c
Reading https://pypi.org/simple/numpy/
No local packages or working download links found for numpy==1.18.0.dev0+20c8d7c
Running from numpy source directory.
/Users/runner/hostedtoolcache/Python/3.6.9/x64/lib/python3.6/distutils/dist.py:261: UserWarning: Unknown distribution option: 'define_macros'
  warnings.warn(msg)
error: Could not find suitable distribution for Requirement.parse('numpy==1.18.0.dev0+20c8d7c')

I think this is the result of python setup.py build -j 4 build_src --verbose-cfg install, but when I run that locally on python3.6, ubuntu 18.04 it runs to completion.

Anyone with MacOS care to try to duplicate it? @tylerjereddy any ideas?

In 37bbe58 I added include/bitgen.h and include/distributions.h. Thoughts?

rebased off master to resolve conflicts after merge of #13794. Also:

• moved public random headers to numpy/core/include/numpy/ranom
• add first draft of random C-API public methods, rendered here

I removed the WIP from the title. The example works, we have the new c-api documentation (link in the comment above) and examples both in documentation and in tests. Reviews would be welcome. More examples and documentation will be generated in gh-14778

Rebased to clear a merge confilict. @rkern @bashtage could you review this?

Thanks Matti. I'm not convinced that the examples are in the right place but they can probably be moved later without problems.

The examples might better be included somewhere in the /doc directory, the current location might be a development artifact. @rgommers Any thoughts on the examples?

I put the examples there so they could be tested. Pieces of them are included in the docs

I put the examples there so they could be tested. Pieces of them are included in the docs

That sounds like a good idea actually. Please move them to _examples though. There's a reason I added a test for unwanted public submodule API changes recently, you should never have to add to the whitelist in that test.