To avoid seeing all the indent changes due to putting functions in classes:

https://github.com/astropy/astropy/pull/2063/files?w=0

@astrofrog - Sorry, I'm a bit swamped with things at the moment but will try to take a look this week.

@astrofrog - I've been looking at this in the background and thinking about it. I don't have any fully formulated ideas, but the first thing that strikes me is that it seems this doesn't take full advantage of using classes in the sense that it still just registers the reader/writer/identifier functions (though they are now methods). I was originally thinking more about registering classes or class instances. There is some opportunity to be lightweight in the sense of lazy loading where possible.

We still haven't solved the inline documentation problem. I'm stuck because at the moment because to do that would require essentially loading all of io.fits, io.ascii, etc in order to know all the available readers and their docs. I don't want this to happen as an immediate side effect if importing Table, but it could happen at the moment you ask Table.read what formats are supported.

Anyway, just want to let you know I haven't forgotten this.

@taldcroft - I also was thinking about this and coming to the same conclusion that we could just register the classes and have the registry automatically read all the information from the class.

There is one solution to the documentation problem, which while not elegant, would certainly save us a lot of hassle. The solution is simply to move registry.py to a proper sub-package in astropy/io/registry and then move all the connect.py files to astropy/io/registry rather than have them live in the different sub-packages. Then it would be much easier to have dynamic documentation, while not necessarily importing the different sub-pacakges if the imports are run-time imports. Yes, this may not be 'ideal' but at the moment we have to hard-code the list of formats in the registry anyway. Anyway, at the moment import_connectors() (in registry.py) is importing the sub-packages already, which is not ideal either.

In some ways it'd also be easier to keep all the connectors in one place, as it's easier to make them more consistent with each others. Do you have any thoughts on this?

I think I'm also on the same page as @taldcroft on this, and it looks like you might be coming to the same conclusions too, so that's good. The way I saw it, I figured you were doing things the way you were for backwards compatibility with the original reader/writer registry. But I don't think that system is used much outside of Astropy itself...? (do any affiliated packages use it?)

I would want to keep it around a bit longer for backwards compatibility anyways, but the new registry should probably avoid it and take more advantage of the use of classes.

I'm also okay on the astropy.io.registry subpackage. I see the connect drivers as being rather separate and distinct from the packages like astropy.io.fits that implement the low-level interfaces for individual file types.

Sounds good - I'll give the astropy.io.registry approach a try and will see if I can get the auto-documentation stuff to work at the same time.

There is this conflict between keeping things light via external definition of readers vs. importing the actual code.

With io.ascii I took the approach of using the import of io.ascii to define the connect drivers. Part of the motivation is to allow for local definition of the allowed read/write kwargs within the relevant source code. We can dive into one of the most stressing cases, the LaTeX reader/writer. In addition to all the standard args in io.ascii.read, it takes several format-specific args.

So imagine having a BaseIO class, from which the io.ascii MetaBaseReader is derived. That MetaBaseReader can have specific knowledge of all the standard io.ascii.read/write args, and then the LaTeX class could provide (via class attributes?) the modifications to the standard args for purposes of generating docs. That keeps the definition of everything local to the code, which is good. For io.ascii it would also allow improved behavior by providing a mechanism for io.ascii.read(.., format='latex') explicitly know what the allowed args are. Putting all the arg definitions outside in the registry would be a bit of a problem.

@astrofrog - I might have an idea of how to get all this to congeal together, but it's a little fuzzy still. Can I take a crack at this? If you are psyched to do this then that's fine of course.

@taldcroft - please do feel free to have a go at this! You can start from scratch and don't need to include the commits here. I'm short on time over the next week or so.

@taldcroft - just to check, are you still planning to have a go at this?

@astrofrog - errr, I got distracted before getting a chance at this and now I'm thinking about coordinates and table stuff.. I can give you back the baton on this.

@taldcroft - no problem, just wanted to check. I'll see if I can work on it during a flight at the weekend, so if you have any other comments in addition to those you added above, let me know!

As I'm further investigating import times, one thing I've discovered is that importing Table, in order to support the generic reader/writer interface, must import all of io.fits, io.votable, io.ascii etc., just to be able to use a Table. One thing that might be nice as part of this refactoring (if you think it's related) would be to allow the registry callbacks to import the modules that support their formats lazily. As it stands, you can't really do anything with Table without importing the entire io enchilada, which adds a 300% overhead over just importing astropy.table.

@mdboom - keeping the io enchilada out of the initial import is definitely in the plan. It will take some doing, especially for io.ascii, but it's the right thing to do.

@mdboom See, relatedly, #1402 . I've had that open for a while but haven't really addressed it.

This is superseded by #3090