Skip to content

Add support for parsing unicode in the units module#9348

Merged
adrn merged 23 commits into
astropy:masterfrom
maxnoe:unicode_micro
Oct 24, 2019
Merged

Add support for parsing unicode in the units module#9348
adrn merged 23 commits into
astropy:masterfrom
maxnoe:unicode_micro

Conversation

@maxnoe

@maxnoe maxnoe commented Oct 9, 2019

Copy link
Copy Markdown
Member

Description

As a first simple step in adding feature #9347, this PR adds two unicode symbols
as possible representations for the micro prefix MICRO SIGN and GREEK SMALL LETTER MU.

While the second one is strictly speaking not correct, it probably appears often enough in the wild to be useful to support.

@maxnoe

maxnoe commented Oct 9, 2019

Copy link
Copy Markdown
Member Author

I though about which representation would be best, basically there are three options:
Just inserting the unicode character: 'µ', using the code point '\u00b5' or the name of the character '\N{MICRO SIGN}'.

I went for the most verbose one, as it is both ascii in the code and human readable. Any other opinions on this?

@maxnoe maxnoe closed this Oct 9, 2019
@maxnoe maxnoe reopened this Oct 9, 2019
@pllim pllim requested a review from mhvk October 9, 2019 16:13
@pllim

pllim commented Oct 9, 2019

Copy link
Copy Markdown
Member

If it is "strictly speaking not correct," should we even support it? 🤔

@pllim pllim added this to the v4.0 milestone Oct 9, 2019
@maxnoe

maxnoe commented Oct 9, 2019

Copy link
Copy Markdown
Member Author

Semantically correct would always be to use 'MICRO SIGN', however I think as

  • It's very common to use GREEK SMALL LETTER MU
  • It's not in conflict with anything
  • The two characters look the same in very many fonts

it's probably better to add both

@mhvk

mhvk commented Oct 9, 2019

Copy link
Copy Markdown
Contributor

The PR looks OK, but I'm a bit with @pllim on whether we should support the greek letter as well. My sense would be that it is OK to interpret it on input, but that we should not expose it in astropy.units. I think the easiest way to do this is to, inside the parser, convert one to another. I'll try to see if I can make a quick PR to your PR.

@maxnoe

maxnoe commented Oct 9, 2019

Copy link
Copy Markdown
Member Author

I see the potential of "Why does µ work but μ not"? I don't really see a harm of supporting both as long as u.µm == u.μm

At closer inspection, the two characters look exactly the same in the mono font but are subtly different in the sans font on github.

@pllim

pllim commented Oct 9, 2019

Copy link
Copy Markdown
Member

They do look very similar... 👀

@mhvk

mhvk commented Oct 9, 2019

Copy link
Copy Markdown
Contributor

My main hesitation is creating yet more entries in the units registry - indeed, I'm not completely sold on even having it there (but very sold on being able to parse it; I think we should roundtrip in unicode as well).

How about we start simple and just allow the mu in strings? I made a PR to your PR...

@mhvk

mhvk commented Oct 9, 2019

Copy link
Copy Markdown
Contributor

p.s. Off for a bit, back in about 1.5 hours.

@maxnoe

maxnoe commented Oct 9, 2019

Copy link
Copy Markdown
Member Author

I also fixed the used character for the micro sign in the unicode output format. See, @pllim commonly used 🤣

Comment thread astropy/units/format/generic.py
Comment thread astropy/units/format/generic.py Outdated
@mhvk

mhvk commented Oct 10, 2019

Copy link
Copy Markdown
Contributor

@maxnoe - I made another PR to your branch, where superscripts are still translated, but as part of a regex that surrounds the translations with parentheses - which are recognized as powers and avoids the mix of regular and superscript numbers just working.

p.s. I had hoped to do it with the lexer/parser itself, but did not manage. Might still be better...

@mhvk

mhvk commented Oct 10, 2019

Copy link
Copy Markdown
Contributor

p.s. It might be fun to see if the roundtrip tests that are done now also work in unicode!

p.s.2 If there are superscript slashes and periods, possibly we should add those too...

@maxnoe

maxnoe commented Oct 10, 2019

Copy link
Copy Markdown
Member Author

@mhvk I checked the unicode superscripts table, seems there is just numbers, +, - and a few selected letters. So I think we got everything covered.

This ensures that mixes of superscripts and normal characters fail.
@mhvk

mhvk commented Oct 10, 2019

Copy link
Copy Markdown
Contributor

How about something like the following (sorry, in a hurry... off for a bit).

if sys.version...
  str_is_ascii = str.is_ascii
else:
   def str_is_ascii(x):
      try:
         x.encode('ascii')
         return True
      except...
         return False

@maxnoe

maxnoe commented Oct 10, 2019

Copy link
Copy Markdown
Member Author

Yes, that's the SO answer I found, but where to put, is there some astropy compat module, or should we just put it in the parser file?

Comment thread astropy/units/format/generic.py Outdated
@mhvk

mhvk commented Oct 10, 2019

Copy link
Copy Markdown
Contributor

Beyond the one last comment this all looks great!

@maxnoe

maxnoe commented Oct 10, 2019

Copy link
Copy Markdown
Member Author

I had a Version in between that replaced the omega with Ohm. Make trans only supports single character replacements. Will fix tomorrow.

@mhvk

mhvk commented Oct 10, 2019

Copy link
Copy Markdown
Contributor

I just made the small change. I think it is all OK now, but since I'm arguably now a co-committer, probably best to get an independent sign-off. @adrn or @dhomeier - might you be able to review?

@adrn

adrn commented Oct 11, 2019

Copy link
Copy Markdown
Member

This looks nice! Unfortunately I won't be able to give it a thorough review until ~Tuesday next week, as I'm away with family for the long (US) weekend.

@dhomeier

Copy link
Copy Markdown
Contributor

I haven't had time for a full review, but tested it under Python 3.8; from a quick overview looks good to me too - @mhvk's last commit certainly does.
One minor question - is it policy not to support prefixed Ångström variants? It's understandable, since the base unit is already deprecated, but with the unicode versions supported, one might more likely encounter u.Unit('mÅ') as well.

@maxnoe

maxnoe commented Oct 17, 2019

Copy link
Copy Markdown
Member Author

I think this is by design. Personally, I would be opposed to enabling prefixes for Ångrström.

@mhvk

mhvk commented Oct 17, 2019

Copy link
Copy Markdown
Contributor

For , I'm not in favour either, but also not really against; my sense would be to let it be driven by user request (i.e., @dhomeier, if you would like to see it, it's fine by me).

@mhvk

mhvk commented Oct 17, 2019

Copy link
Copy Markdown
Contributor

@adrn - would you still be able to have a look? With @dhomeier's quicker review and @taldcroft's liking of the general idea in #9347 (comment), I think there has been enough review to merge, but a good look is always helpful.

@dhomeier

Copy link
Copy Markdown
Contributor

For , I'm not in favour either, but also not really against; my sense would be to let it be driven by user request (i.e., @dhomeier, if you would like to see it, it's fine by me).

I am not strongly pushing for it either, but in the spirit of the earlier discussion on which µ variants to accept, I think it's worth at looking at the potential harm vs. benefits. Stuff copied from older literature might very well come in milliangstroms, too.

@mhvk

mhvk commented Oct 17, 2019

Copy link
Copy Markdown
Contributor

True. Though note that this is somewhat orthogonal to the discussion here - one would just set prefixes=['m'] in the definition of Angstrom and things would just work (just like mOhm would now).

@bsipocz

bsipocz commented Oct 24, 2019

Copy link
Copy Markdown
Member

@mhvk or @adrn - would either of you approve and merge this now?

@mhvk

mhvk commented Oct 24, 2019

Copy link
Copy Markdown
Contributor

As a co-author, I perhaps shouldn't, but if @adrn doesn't get to it, I'l, do it tomorrow.

@adrn adrn left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks reasonable to me - I don't have any comments!

@adrn adrn merged commit 78d5599 into astropy:master Oct 24, 2019
@maxnoe maxnoe deleted the unicode_micro branch October 24, 2019 15:13
@mhvk

mhvk commented Oct 24, 2019

Copy link
Copy Markdown
Contributor

@maxnoe - thanks for the nice work and good collaboration!

@maxnoe

maxnoe commented Oct 24, 2019

Copy link
Copy Markdown
Member Author

@mhvk Was a pleisure, thanks for all the support!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants