If we want such a feature (and why not), then I would suggest to implement it in BaseHeader and BaseData so that formats other than fixed-width can also profit from it.

Note however, that there is no way the Reader can know if the line with the units is part of the header or part of the data. Therefore, reading the tables @astrofog has written will always require a manual setting of data_start=2. Unfortunately, that brings us a little further from the goal of tables that round-trip automatically.
On the other hand, I don't think we will be able to round-trip most of the formats anyway, so maybe that's not an issue.

Note that LaTeX and AASTex already have a mechanism to print units, because those tables have a lot more options then most output format and we should aim to (in some way) make that consistent - however, that might be done in a subsequent PR.

from astropy.table import Table, Column
import astropy
from copy import deepcopy
COSobs = Table([Column(name='date', data=['a','b']),
                Column(name='NUV exp.time', data=[1,2]),
                ])
latexdict = deepcopy(astropy.io.ascii.latexdicts['AA'])
latexdict['units'] = {'NUV exp.time':'s', 'FUV ext.time':'s'}
COSobs.write(sys.stdout, format='ascii.aastex', latexdict=latexdict,
             caption=r"\label{tab:obsCOS}Observations log of \emph{HST}/COS observations")

I was going to agree with @hamogu to put this into BaseHeader, but then I thought of some things...

In a real way this is inventing a new format. Maybe I'm being too conservative, but I think that any file we output should actually be readable by at least astropy, and preferably by other tools. If you write a file like this then your collaborators either need to use astropy (assuming we write the corresponding reader), or jump through hoops to tell their reader applications what to do.

I think that Pandas won't be able to read such a file at all, and even astropy won't be able to read the second example from @astrofrog unless the user explicitly specifies column start/stop values. (Of course you can just say "don't use space delimiter here").

I guess if the goal is to have something that is human-readable, then it works, but I've spent so much time trying to get away only-human-readable formats...

A secondary concern is #2716. @AmrAS1 - is this written so that new params can be added to read/write and it will see them as unknown params and drop to the Python reader?

Anyway just feeling crufty now, continue discussing. 😄

@taldcroft - I can see your point, but do we really have to tell beginners there is simply no way that they can write their data to a table where the units are in the header without using a format with a yaml/json header etc? Which is not an option anyway in some cases if they want to send it to someone who has say IDL...

I'm not saying we should be able to read this in to Astropy, and in fact my second example indeed breaks your paradigm of being able to read in what you wrote out (not because of the units, just as you said because of the delimiter). I think it's ok if not everything round-trips. If someone wants to write it out to a file and read it back in to astropy, they shouldn't be using the basic ASCII formats anyway, they should be using FITS or HDF5 or VO tables. I see ASCII-writing as mostly a one-way thing as far as astropy is concerned. So I'm firmly in the camp that not all formats have to round trip, and we can make it obvious in the docs which formats can guarantee round tripping and which can't.

I think it's more important to have a way to allow the user to format the table how they want, and you shouldn't tell them they can't do something just because they wouldn't be able to read it back in. Maybe they don't want to read it back in! :)

@hamogu - you mention that LaTeX outputs units - how does the reader deal with this? Does it try and read in with units and then without? If you managed to make that round-trip, we should be able to do the same here too, right?

@taldcroft - This should be fine with #2716, since the Cython writing class will just raise an error given a keyword it doesn't recognize, and anyway it won't try to use the fast writer for fixed-width format. I think there shouldn't be any case in which the fast engine doesn't behave as expected if a new keyword for a specific format is introduced, although I guess there could be a problem if a new general-purpose keyword were added to ascii.read.

I like the idea in principle. Possible ways to help ensure it somewhat more easily roundtrips is to use "-" for "no units, or to enclose the unit in parentheses (or square brackets), which is closer to how a tables are usually typeset anyway. One might think of including this on the first line (though that would be less clear), i.e., either

      a                b                     c  
   (mJy)               ()   (ergs/cm^2/s/Hz/ster)

or

      a(mJy)          b()   c(ergs/cm^2/s/Hz/ster)

For further round-trip heaven, is the #-is-comment convention sufficiently widespread that one could use that?

I strongly suggest to write things such that the output table can be read in some way by io.ascii, even if it does not round-trip perfectly - making the unit line a comment line would be fine for me.

LaTeX - Units don't round-trip. The LaTeX reader is fairly basic, e.g. it fails if the line breaks don't coincide with \\ . For the writer that's not a limitation (the user will just get lines that break exactly after the \\). I have some ideas how to improve that, but I'm not sure it's worth it. I don't want to re-implement LaTeX. I read LaTeX tables seldom and when I read them, they are typically long, but simple data tables with no special constructs or they are so special that I need to edit them by hand before I can read them anyway (e.g. using \multicolumn{} which has no equivalent in an astropy.table.Table).
LaTeX tables are definitely a format that you should not use for temporary on-disk storage of data sets.

@astrofrog I forgot: In typical LaTeX tables in astronomy you can see that the units are part of the header and not that data. It's easy in deluxetables, where units would be set with \colhead{erg} above the \startdata and even in default LaTeX almost all tables separate header and data with \hline\hline, since that's what the A&A and I think also the MNRAS typeseters use (that's of course a convention and not a law) .

In contrast, in normal ascii tables you don't know:

distance  | type | color
    pc        | SpT |mag
  N/A        |  M2V| 12.4

Is the first column a string column with ['pc', 'N/A']? Is the second a string column? Is the third a numeric column with a unit? How do you know?

BTW, you can do something pretty similar to this with 0.4:

In [1]: from astropy.table import Table
In [2]: t = Table([[1], [2]], names=['a', 'b'])
In [3]: t['a'].unit = 'm'
In [4]: t['b'].unit = 'm/s'
In [5]: print('\n'.join(t.pformat()))
 a    b  
 m  m / s
--- -----
  1     2

On the whole, if we want a simple way of conveying units that would be human- and machine-readable with the least effort, I would vote for the format proposed by @eteq. In that case we put the units as separate commented-out lines. The reason is that even though humans intuitively notice that the second line is still "header", pretty much all CSV parsers in the world need to be explicitly told to skip it, and then the user needs to write some custom code to pull out the second line and parse into units.

If you just want a fixed width table with units that isn't necessarily machine readable then the default pformat output should work, no?

Finally if we decide that we really want a global header_unit_line capability, then the easiest thing to do for no unit is "" as a placeholder. That happens automatically for Basic output.

And BTW, IDL does have a JSON parser. 😄

One thing that's bugged me for a while is that there is no easy way to insert comments into a Basic format file. What if the Basic writer automatically emitted any lines that are in the table meta['comments'] list? Something like

t = Table()
...
t.meta['comments'] = ['{}: {}'.format(col.name, col.unit) for col in t.columns.values()]
t.write(sys.stdout, format='ascii')
# a: m
# b: m/s
a b
1 2

Beyond this application, supporting a generic way of getting comments in and out would be good. Likewise the basic reader should set t.meta['comments'].

I'd agree with the proposal to make the unit line commented out for now. Just to brainstorm a little further, if we do make the unit line a global setting, then presumably in order to round-trip you simply also set the setting on the reader, right? I agree it wouldn't necessarily work with the guessing, but if you specify what format it is and that there are units, it should work, correct? (can we also include the presence/absence of unit line in the guessing?)

With the commented-out unit line, might it even make sense to make adding the units the default? (Having been bitten many times by "guessed" units, I do like the idea of making that less likely.) Or, more extremely, could we always add a line if any unit is defined?

@taldcroft's example of using comments gives a format that would work well for tables with lots of columns, and nicely avoids making columns very wide if the unit is very long; do think this is a good alternative. (This of course is independent of the implementation).

For round-trip guessing, the parentheses around the units would likely help.

Just to be controversial, one could argue that not outputting the units means that we can't round-trip tables since the units are a pretty essential part ;) So I'm actually +1 for @mhvk's suggestion to output by default, but I realize it is controversial.

By the way, square brackets might be better around the units than round (I've seen that used in other places, and see http://physics.stackexchange.com/questions/77690/square-bracket-notation-for-dimensions-and-units-usage-and-conventions for example)

I don't mind the idea of using the comments before the table, so count me as +0, but at that point we can then go for @taldcroft's proposed new format.

So how would people feel about encoding units in the header comments as:

# columns:
# - {name: a, unit: null}
# - {name: b, unit: erg / (cm2 s)}

If you didn't guess, this is valid YAML and will be a compliant subset of the full DTIF/APE-6 format [1]. This implies round tripping within astropy and compliance with "existing" standards. The downside is a slightly more verbose (but still human-readable) output.

[1]: The plan is also that DTIF will serialize column metadata in a way that is compliant with the ASDF format specs for embedded ASCII tables. I've been swayed by ASDF to use YAML instead of JSON for DTIF. The human-readability is compelling. Is that enough 4-letter acronyms for ya?

@taldcroft - I'd be happy with that format. Compared to your examples in APE6, I particularly like it that each column would just have one line; less useful to have a readable header if the header is out of sight by the time you look at the data!

Nevertheless, it would still be good if one could easily produce an output file similar to t.pprint(), but perhaps in a way more similar to what you gave above, using pformat or pprint.

Aside, while redirecting standard output and using your print statement works, it doesn't seem to work with pprint: it prints the header on the terminal still, while storing the values in the output file:

In [15]: with redirect_stdout(open("a.out", "w")):
    t.pprint(show_name=True, show_unit=True)
   ....:     
  a         b                 c         
 mJy                 ergs/cm^2/s/Hz/ster
----- -------------- -------------------

In [16]: !cat a.out
    1      1293494.0                   a
    2           11.0                   c
33133 0.333333333333              wqdqwd

I agree with @mhvk - while APE6 looks great, and a YAML header sounds nice, I would still like to argue that we should allow the user to customize their output so they can include the unit line in a 'normal' ascii format. Now I don't think that it's the best format they can use, but we can't just tell users they are wrong to want to output it in a certain way (maybe they have no choice due to requirements by a third party).

That's weird with table pprint and redirect_stdout. Can you open an issue?

This is getting pretty far off topic, but what about adding a file_out option to Table.pprint?

At the rink of sounding like a broken record, just to go back to the original issue here, we advertise astropy.io.ascii as:

astropy.io.ascii provides methods for reading and writing a wide range of ASCII data table formats via built-in Extension Reader classes. The emphasis is on flexibility and ease of use.

Note that the emphasis is on flexibility, but if we tell users that they cannot write units on the second line under the column names, then I think we are not being flexible enough. Caring about round-tripping is fine, but I don't think it should so severely restrict the writer classes. We've catered for some pretty messed up formats for input, so why not allow a little more flexibility in output? Sometimes it's not up to the user what format they need to write to. Maybe they need to pass it to another application that does expect units on the second line. We can't tell them 'no, that's a bad format' because maybe they don't have control over the next tool in the pipeline that needs to read the table.

I don't think that adding a file_out option to pprint is the right way to go here.

@astrofrog - I see your points and will likely concede eventually, but just to make a couple of redirects:

• Have any users (other than core developers) actually asked for this feature?
• @eteq has frequently argued that one ulterior driver for the way astropy should do things is to encourage best practices. Writing out files in the proposed format is a bad idea because it is an impediment to reproducible science, in the sense that most existing tools can't read it.

About flexibility on input vs. output, one could cite Postel's law here.

@taldcroft - yes, I've had one user ask me explicitly for this request (yesterday) and could not understand why this would not be an option given that units are pretty essential to tabular data.

Regarding encouraging best-practices, I completely agree and that's why I think the defaults should be sensible, and I'm fine with not having units shown by default if it's going to cause reading issues. But I'll just point out that to me best practices means that I shouldn't be allowed to write out a table without units! :) If we tell users they can't output units we are not encouraging best practices.

@taldcroft - I completely understand why we don't want to just allow any output format by the way, I just think at the moment the output is a mix of allowing things that are definitely not round-trippable (e.g. ' ' delimiter in fixed width) and disallowing things (units) that would actually help round-trip the contents of the table.

Just for the record - ideally I'd much prefer if no-one even output any tables to ASCII. There are plenty of nice binary formats that are perfectly adequate and support e.g. units and meta-data, and preserve floating-point accuracy. To me, best practices would be 'don't use ASCII!' :)

Out of curiosity did the user specifically need units in a header row for some other application, or just need to have units somewhere in the file? I.e. would the user be OK with the YAML solution? Where I'm going is that sometimes people propose a feature because it's similar to what they have seen before (based on not-so-best practices), but perhaps we can propose a better way that works just as well.

@taldcroft they said they wanted one row with the column names, then one row with units, then the data. So conceptually the same as what one would do in a paper table. But I can ask them what they would think about the YAML solution.

It seems there may be some convergence! To, hopefully, help that, I think there is a conceptual difference between "writing" and "printing", where for the former would one expect that what is written can also be read, while for the latter one only wants inspection. I would advocate having separate solutions for these:

1. writing: like we are converging to: possible both to have units in the first line, so that it will get read reasonably sensibly by excel, etc., and to have a more elaborate (and preferred option?) of the YAML header.
2. Ensure that we can redirect pprint. This may be through a file_out option, though I'm also fine with just documenting how to do it using standard python in the pprint docstring.

In fact there are format where we put units in a certain lines, see IPAC or LaTeX. There might be a good reason for a user to ask for the concept (line 1: column names, line 2: units, later: data) in plain ascii to e.g. convert the plain ascii to Latex or a table in MS Word manually to format it for publication.
So, if we allow for this at all 👍 for also allowing it in the default / basic writer (not be default).

Trying to catch up on some old discussions here... @astrofrog and @taldcroft : I think I've said this before, but I'm strongly in favor of getting units into ASCII representations of tables. I'm a little unclear what the "convergence" of this discussion is that @mhvk mentioned... But I actually think that units should default to on, although with an easy way to turn them off.

My viewpoint comes from the fact that >50% of the time, when I download an ASCII table, I have to go in and modify it anyway (i.e. add or remove comments, change tabs to spaces in the header, etc.). So long as the meaning is immediately visible on visual inspection, and it can be automatically read by astropy, or relatively easily modified via a text editor for other readers, then I don't see what the problem is with having units default to on when present.

@eteq - I'm very strongly against any regressions in the output format of the io.ascii basic formats (character delimited Basic formats). We've advertised io.ascii as stable and we should keep it that way. In your particular workflow these differences don't matter, but for many people they will. Even something like # commented lines are not ubiquitously supported.

io.ascii is built to be extensible by adding new format classes. So DTIF in #2319 has gotten very recent activity and as a standalone format I think it could be merged in a week if the community agreed. This is very close to what you originally requested. In theory we could add a new keyword units_only for that format so that it only outputs column name and unit and never anything else.

To the extent that people really want the units encoded within the table proper, putting them in the column names is the least objectionable to me. A related option that I would actually prefer is new format like basic_units, with a keyword option e.g. units_in_colnames (if true put units in column names, if false then do it as the second line). Doing it this way moves this corner-case functionality into its own namespace. It also is more in line with the spirit of a "format" in the rest of io.ascii. For instance whether the column names are there or not, or whether they are in a commented header line. Arguably there are issues with this strategy, but at least you don't end up with the mess of keywords like pandas.read_csv().

@taldcroft - hmm, I see your point about the "basic" wanting to be stable. I have no problem with this being a separate format along the lines of basic_units. I also don't have a strong opinion on units-in-colnames vs. units-elsewhere, just that it should be somewhere that can be auto-read by astropy.

What I am saying, though, is that I think new users (or those who just don't care very much) should have the path-of-least-resistance be to get units, rather than to not get them. If the first exposure to writing to tables in astropy doesn't do this, I'm guessing most people won't look deep enough to find it. And ASCII tables without units floating around leads to a lot of science problems down the road.

One thing I'm not clear on, though, @taldcroft: are you saying you're against the default being the version with units, or just that Basic shouldn't be altered? The place where a "default" really applies is ascii.write, which is currently effectively format='basic'. I'm saying what I would like to see is the Basic formats left alone, but consider changing the default format from 'basic' to 'basic_units'. Then users can still get 'basic' by just asking for the right format. Or do you still see that as breaking backwards compatibility?

For Table.write (which is what I tend to use more often), one has to specify a format anyway, so just changing the basic examples to use a ascii_units would mostly address that. It might be that the way to have our cake and eat it to is to just make the examples lean on Table.write over ascii.write, and have the examples feature the use of units quite prominently.

If #2319 does indeed happen, I agree that could be considered as taking the place of basic_units. But my statement about defaults still stands.

@taldcroft - Also I just looked back a bit and realized I am getting pretty off-topic here from the actual intent of this PR. So if you'd rather this discussion be elsewhere (#2319 or #756 ), I'm find with that.

I should have been more precise in saying that I want the default to be stable. I think of the default io.ascii output as being a minimal format that can be faithfully read by virtually any CSV reader out there. This has been the case since asciitable 0.1 came out nearly 5 years ago. Putting in a row for units or modifying the column names violates that.

I think of the row for units or column names with units as being solutions for a specialized problem, mostly related to generating human-editable/viewable content, maybe for pasting into another document or whatever. They aren't great as far as a table interface "standard" that should be recommended for people to use with collaborators [1]. That's why a new format like basic_units would be fine, but not as a default.

With regards to #2319, I'm fully convinced this is the right way forward. Along the lines of your suggestion for changing documentation for what to recommend, what if there was a top-level table read/write format asciitable that used DTIF, so t.write(format='ascii') would go to t.write(format='asciitable') and that would produce a fully-serialized version of the table. Or at risk of overloading an over-used word, call the DTIF format table within astropy, so you have:

>>> io.ascii.write(t, format='table')
>>> t.write(format='ascii.table')

Anyway, we don't need to find all the right names now, but recommending to users that they write tables this way, in the DTIF format, would be a good step in my mind.

[1]: It's not a defined or even common-use standard as far as I know. So every person receiving such a file either has to use astropy or roll their own reader. You can say the same of DTIF, but that is at least a well-defined standard and an so it's more likely that developers would be inclined to support it in other languages / apps. I think it's more likely that TOPCAT would support DTIF than the unit formats.

@taldcroft - I see your point about wanting stability here [1], and that something like #2319 is better.

[1] Unfortunately, people still do write out tables all the time like this and send them to collaborators... And I personally know plenty of examples where that lead to problems due to incorrectly assumed units. So I still disagree on the philosophical point that we shouldn't be trying to persuade them to use units, maybe even in slightly nefarious ways. But practicality beats purity, and you're right it would be more annoying at this point to suddenly start changing the default table format around.

Not that #3064 just changed the default for latex tables to include the units in the output by default. Again, this is a slightly different case since latex does not need to round trip.

Closes #386.

Rescheduling for 1.1.

@astrofrog - Just reviewing assigned PRs, finally... This one got a huge amount of discussion without any firm resolution. @mhvk had one nice summary comment mentioning "convergence", but since Python2 doesn't have redirect_stdout I'm not sure this is the best solution. So if you have time and want to finish off this PR with tests then go for it!

I wonder if the round-tripping could be solved using a corresponding header_unit_line keyword in the reader. It also occurred to me the fixed_width_two_line format would be able to "find" a unit line without requiring a keyword because it can see if there is a line with units prior to the --- ---- --- delimiter line. That would allow the output of pprint() to be parsed and return the units.

@taldcroft - that sounds reasonable. I won't have time to do this for 1.1 though, so have removed the milestone.

Hi humans 👋 - this pull request hasn't had any new commits for approximately 3 years. I plan to close this in a month if the pull request doesn't have any new commits by then.

In lieu of a stalled pull request, please close this and open an issue instead to revisit in the future. Maintainers may also choose to add keep-open label to keep this PR open but it is discouraged unless absolutely necessary.

If you believe I commented on this issue incorrectly, please report this here.

@taldcroft @hamogu @mhvk - I don't really care enough about this anymore to push ahead with it (it's no longer relevant for me), so I'll let the bot auto-close it unless you object.

⏰ Time's up! ⏰

I'm going to close this pull request as per my previous message. If you think what is being added/fixed here is still important, please remember to open an issue to keep track of it. Thanks!

If this is the first time I am commenting on this issue, or if you believe I closed this issue incorrectly, please report this here.