[geotrans] upgrade to version 3.8 by StarGate-One · Pull Request #13243 · microsoft/vcpkg

StarGate-One · 2020-08-30T23:57:41Z

Describe the pull request

Upgrade port geotrans to version 3.8
Delete old style CONTROL file
Add new style vcpkg.json file
Update CI to reflect platforms not supported

What does your PR fix? Fixes [geotrans] update to 3.8 #13241 Upgrade geotrans to version 3.8
Which triplets are supported/not supported?
geotrans:arm-uwp=fail
geotrans:arm64-uwp=fail
geotrans:arm64-windows=fail
geotrans:x64-osx=fail
geotrans:x64-uwp=fail
geotrans:x64-windows-static=fail

Have you updated the CI baseline? Yes

Does your PR follow the maintainer guide? Yes it should

**Describe the pull request** 1. Upgrade port geotrans to version 3.8 2. Delete old style CONTROL file 3. Add new style vcpkg.json file 4. Update CI to reflect platforms not supported geotrans:arm-uwp=fail geotrans:arm64-uwp=fail geotrans:arm64-windows=fail geotrans:x64-osx=fail geotrans:x64-uwp=fail geotrans:x64-windows-static=fail

StarGate-One · 2020-08-31T01:04:50Z

@JackBoosY @strega-nil

The CI runs (x86-windows, x64-windows and x64-linux) failed because of downloading the file, as geotrans is located on a US government server and they are not known for speed...
The URL is: ftp://ftp.nga.mil/pub2/gandg/website/wgs84/apps/geotrans/current-version/sw/dev_version/linux_dev.tgz
The file average about and hour give/take a few minutes depending on server traffic, so I do not know what to do from here.
The port builds fine on my x86-windows, x64-windows and x64-linux instances (Windows 10 and OpenSUSE Leap 15.2 both in WSL2 and Hyper-V).
I do not know how to:
a. add a message notifying the user the download will take a while 60+- minutes depending on network traffic
b. change the timeout period for vcpkg_download_distfile() function

scripts/ci.baseline.txt

ports/geotrans/vcpkg.json

ports/geotrans/portfile.cmake

cenit · 2020-08-31T07:14:43Z

@StarGate-One what about using the master file? ftp://ftp.nga.mil/pub2/gandg/website/wgs84/apps/geotrans/current-version/sw/master_version/master.tgz

It's described as identical between linux and windows.
It might ease the load for people having to build both in an automated way (sharing the download folders between multiple vcpkg instances is very common).
Is it possible, in your opinion? Or is it a different beast?

1. Add message to portfile.cmake about long download times 2. Remove port-version from vcpkg.json 3. Remove the following exceptions from ci.baseline.txt geotrans:arm-uwp=fail geotrans:arm64-uwp=fail geotrans:arm64-windows=fail geotrans:x64-osx=fail geotrans:x64-uwp=fail

… geotrans

StarGate-One · 2020-08-31T17:17:42Z

@StarGate-One what about using the master file? ftp://ftp.nga.mil/pub2/gandg/website/wgs84/apps/geotrans/current-version/sw/master_version/master.tgz

It's described as identical between linux and windows.
It might ease the load for people having to build both in an automated way (sharing the download folders between multiple vcpkg instances is very common).
Is it possible, in your opinion? Or is it a different beast?

Hi @cenit

The source in all 3 developer files are the same, it is just the compiled binaries included in each file that makes the difference:

The windows version windows binaries included with the source
The linux version has linux binaries included with the source
The master contains both windows and linux binaries with the source (slightly larger than the each of the files above).

I also looked at the MGRS Versions (windows/linux) but they are only a sub-system of the complete GEOTRANS package, missing some of the source.
The problem is not really the size of the binaries even though we really do not care about those as we are recompiling them - they take up relatively little space in the compressed packages.
The issue is this is a .mil website (I think US army mirror - but it points to the same fie/ftp file) and is most likely an old fashion ftp server - I think they are running either Windows 2000 or 2003, with very slow network cards, I think, at least act like 10 or 100 kps cards on the ftp server.
There is a file in buildtrees\geotrans\src\geotrans-3-1821303982\data\Und_min2.5x2.5_egm2008_WGS84_TideFree_reformatted is 145mb, the bulk of the download, but is needed to use the package after recompiling.
Now I was reading the license/copyright buildtrees\geotrans\src\geotrans-3-1821303982\GEOTRANS3\docs\MSP_Geotrans_Terms_Of_Use - attached
MSP_Geotrans_Terms_Of_Use.txt, it seems it can be distributed as long as the package stays intact and is not modifed (modifed or enhanced versions can not be distributed) and credit is given to National Geospatial-Intelligence Agency (NGA).
So I was thinking, if we had somewhere to host the file, on github maybe in its original form, the download of a 150mb file would only take a couple of minutes.
I searched for where the file may be hosted somewhere already, but it seems the people over on CONAN are having the same issue [request] geotrans/3.8 conan-io/conan-center-index#1779.
I guess this would be something github and/or microsoft legal department would need to weigh in on..

cenit · 2020-08-31T17:37:48Z

The problem is not really the size of the binaries even though we really do not care about those as we are recompiling them - they take up relatively little space in the compressed packages.

exactly for this, the master file was the most interesting, at least for me.
It allowed to install both the Linux and the windows version with only one download, which was especially useful for people that have the buildsystem automated and that share the download folder to cache sources, because in the end the portfile is pointing to the same file and so there is no need to download two :)
Yeah I saw very disappointing speeds when downloading from that website, and that was my first thought when trying to minimize the download (for vcpkg CI and also for geotrans multiplatform users).

StarGate-One · 2020-08-31T18:42:21Z

The problem is not really the size of the binaries even though we really do not care about those as we are recompiling them - they take up relatively little space in the compressed packages.

exactly for this, the master file was the most interesting, at least for me.

If we could just download source, that would be the way to go as the Windows and Linux source directories are identical. It builds either one. Making such a change would be beyond my technical abilities.

Note: The previous geotrans-3.7 copied then uses the CMakeLists.txt file from the ports\geotrans directory and it determines based on the triplet. All I did was add the additional .h .c files, of source updated the SHA512 and download from location.

It allowed to install both the Linux and the windows version with only one download, which was especially useful for people that have the buildsystem automated and that share the download folder to cache sources, because in the end the portfile is pointing to the same file and so there is no need to download two :)

It does not download both files now or before, just the linux source (its a little smaller), the x86-windows, x64-linux and x64-windows all use the same linux file.

Yeah I saw very disappointing speeds when downloading from that website, and that was my first thought when trying to minimize the download (for vcpkg CI and also for geotrans multiplatform users).

I was able to download the Windows file a little faster (14 minutes), but I guess it is cheating using aria2c:

aria2c -d C:\Downloads -o geotrans-3.8.zip -j 16 -x 16 -l C:\Downloads\geotrans-3.8.log -t 600 ftp://ftp.nga.mil/pub2/gandg/website/wgs84/apps/geotrans/current-version/sw/dev_version/win_dev.zip
Logfile: geotrans-3.8.log.zip

StarGate-One · 2020-08-31T18:46:44Z

@JackBoosY It still fails with a timeout downloading the file...
I was able to download the file faster, but I maxed out the number of connections per server and connections per session to accomplish it, although using more than 4 or each is considered inconsiderate to internet norms :(
#13243 (comment)

Download master.tgz containing both Windows/Linux Binaries

JackBoosY · 2020-09-01T02:01:52Z

Elapsed time for package geotrans:x64-windows: 21.74 s
It seems that the pipeline cannot access this url at all.
@BillyONeal Can you take a look?

Thanks.

StarGate-One · 2020-09-01T20:43:33Z

@cenit I changed the portfile.cmake to use the master.tgz file, so both the linux/windows is now present - it is only a 5mb difference.
We still need to figure out why the CI can not connect to the download site, hmm...
@JackBoosY cc @BillyONeal - could it be that since the nga.mil site is running some old hardware/operating systems that it could be an SSL/TLS issue? Just throwing that out there...

BillyONeal · 2020-09-01T21:26:22Z

It can't be TLS because it's an FTP link.

Hmmmm.... it doesn't look like we open the FTP port: https://github.com/microsoft/vcpkg/blob/master/scripts/azure-pipelines/windows/create-vmss.ps1#L51

Is this available on an HTTP(s) endpoint or are we stuck with FTP?

Thanks!

StarGate-One · 2020-09-01T22:01:12Z

It can't be TLS because it's an FTP link.

Hmmmm.... it doesn't look like we open the FTP port: https://github.com/microsoft/vcpkg/blob/master/scripts/azure-pipelines/windows/create-vmss.ps1#L51

Is this available on an HTTP(s) endpoint or are we stuck with FTP?

Thanks!

The home web site is https: https://earth-info.nga.mil/GandG/update/index.php?action=home

When you click on any of the file download links it becomes ftp://ftp.nga.mil in our case the file is: ftp://ftp.nga.mil/pub2/gandg/website/wgs84/apps/geotrans/current-version/sw/master_version/master.tgz

I tried changing the ftp to http https and sftp ://ftp.nga.mil/pub2/gandg/website/wgs84/apps/geotrans/current-version/sw/master_version/master.tgz but appears port 22, 80 and 443 are closed on their ftp server :(

BillyONeal · 2020-09-01T22:23:30Z

Marking depends:vm-update to open the FTP port.

JackBoosY · 2020-09-03T10:16:19Z

~~Seems still have many regressions, get log here.~~
Nope, they are download issue.

BillyONeal · 2020-09-03T18:41:32Z

@JackBoosY Yes it's failing to download because we don't open the FTP port on the VMS.

StarGate-One · 2020-09-03T18:47:59Z

I thought was it the FTP port?

BillyONeal · 2020-09-03T18:52:10Z

@StarGate-One Yes, brain fart 😅

StarGate-One · 2020-09-03T18:54:23Z

LOL no worries, we all have those days.

NancyLi1013 · 2020-10-26T10:08:29Z

@StarGate-One
It seems that the repository doesn't exist now.

Could you please help take a look and resolve the conflicts?

StarGate-One · 2020-10-26T16:09:26Z

@NancyLi1013 cc: @BillyONeal
The download is still available.
Has the ftp port been opened on the CI VM?
The port has been on hold waiting for the ftp ported to be opened/configured on the CI VM.

BillyONeal · 2020-10-26T19:28:02Z

I believe FTP is indeed not open:

vcpkg/scripts/azure-pipelines/windows/create-vmss.ps1

Lines 73 to 147 in e9f8cc6

    
           $allowHttp = New-AzNetworkSecurityRuleConfig ` 
        
             -Name AllowHTTP ` 
        
             -Description 'Allow HTTP(S)' ` 
        
             -Access Allow ` 
        
             -Protocol Tcp ` 
        
             -Direction Outbound ` 
        
             -Priority 1008 ` 
        
             -SourceAddressPrefix * ` 
        
             -SourcePortRange * ` 
        
             -DestinationAddressPrefix * ` 
        
             -DestinationPortRange @(80, 443) 
        
           $allowDns = New-AzNetworkSecurityRuleConfig ` 
        
             -Name AllowDNS ` 
        
             -Description 'Allow DNS' ` 
        
             -Access Allow ` 
        
             -Protocol * ` 
        
             -Direction Outbound ` 
        
             -Priority 1009 ` 
        
             -SourceAddressPrefix * ` 
        
             -SourcePortRange * ` 
        
             -DestinationAddressPrefix * ` 
        
             -DestinationPortRange 53 
        
           $allowGit = New-AzNetworkSecurityRuleConfig ` 
        
             -Name AllowGit ` 
        
             -Description 'Allow git' ` 
        
             -Access Allow ` 
        
             -Protocol Tcp ` 
        
             -Direction Outbound ` 
        
             -Priority 1010 ` 
        
             -SourceAddressPrefix * ` 
        
             -SourcePortRange * ` 
        
             -DestinationAddressPrefix * ` 
        
             -DestinationPortRange 9418 
        
           if (-Not $Unstable) { 
        
             $allowStorage = New-AzNetworkSecurityRuleConfig ` 
        
               -Name AllowStorage ` 
        
               -Description 'Allow Storage' ` 
        
               -Access Allow ` 
        
               -Protocol * ` 
        
               -Direction Outbound ` 
        
               -Priority 1011 ` 
        
               -SourceAddressPrefix VirtualNetwork ` 
        
               -SourcePortRange * ` 
        
               -DestinationAddressPrefix Storage ` 
        
               -DestinationPortRange * 
        
           } 
        
           $denyEverythingElse = New-AzNetworkSecurityRuleConfig ` 
        
             -Name DenyElse ` 
        
             -Description 'Deny everything else' ` 
        
             -Access Deny ` 
        
             -Protocol * ` 
        
             -Direction Outbound ` 
        
             -Priority 1012 ` 
        
             -SourceAddressPrefix * ` 
        
             -SourcePortRange * ` 
        
             -DestinationAddressPrefix * ` 
        
             -DestinationPortRange * 
        
           $NetworkSecurityGroupName = $ResourceGroupName + 'NetworkSecurity' 
        
           $securityRules = @($allowHttp, $allowDns, $allowGit); 
        
           if (-Not $Unstable) { 
        
             $securityRules += @($allowStorage) 
        
           } 
        
           $securityRules += @($denyEverythingElse) 
        
           $NetworkSecurityGroup = New-AzNetworkSecurityGroup ` 
        
             -Name $NetworkSecurityGroupName ` 
        
             -ResourceGroupName $ResourceGroupName ` 
        
             -Location $Location ` 
        
             -SecurityRules $securityRules

Is there an https endpoint for these bits?

StarGate-One · 2020-10-26T20:19:46Z

No https.
The old version 3.7 is http but the server it is located on is being decommissioned in near the future (at least according to the old web site) in favor of the new nga.mil site.

The new site uses http for web pages access, but uses ftp for all file downloads.

URL is ftp://ftp.nga.mil/pub2/gandg/website/wgs84/apps/geotrans/current-version/sw/master_version/master.tgz and is very slow, I mean dial-up modem slow :(

It might be a good discussion point to support 3.7 until the old server is shutdown, and then remove the port from vcpkg when it is no longer available?

I have v3.8 as a local port in my vcpkg instances, so it does not impact me either way and I leave the decision up to you all.

Thanks,

BillyONeal

After discussion today the team has no objection to accepting this FTP source location into the tree, although we still massively prefer an HTTP endpoint should it become available, because this port will ignore proxy settings, etc.

Of course, we need to wait to merge it until we actually open that port, next VM update.

The edits to aria2 that were done to attempt to troubleshoot the closed port should be reverted.

BillyONeal · 2020-11-05T22:10:02Z

ports/geotrans/portfile.cmake

-    URLS "http://earth-info.nga.mil/GandG/geotrans/geotrans3.7/linux_dev.tgz"
-    FILENAME "geotrans-3.7.tgz"
-    SHA512 20bdc870026e95154f1d7f9560cbfa2c0b2dc39042aa544f093b502a0609121cb47df5729248e0d79ccf8f9908bf01bbcea8e777ae4f45e25472b7ce2bcb9742
+    URLS "ftp://ftp.nga.mil/pub2/gandg/website/wgs84/apps/geotrans/current-version/sw/master_version/master.tgz"


It's concerning that this URI is not versioned so it will break whenever they update it.

BillyONeal · 2020-11-05T22:16:42Z

@StarGate-One It looks like somehow the branch is toast so we can't resolve the merge conflict; can you fix that or let us know that you're no longer interested in pursuing this?

Thanks!

StarGate-One · 2020-11-05T22:38:25Z

I have it locally on my customized versions of vcpkg
Yes, I would prefer an https download, even an sftp would be better plain ftp 😢
I will take a look and create a new branch in my local version of the vanilla vcpkg, test to make sure everything is OK, and push to my github fork, then create a PR to merge into microsoft/vcpkg.
@JackBoosY made the aria2c changes, but I do not think we need them... Although the ftp server is very slow, it takes about an hour to download the 156mb file 😢
I think what happened is I had to rebuild my laptop and I did accidently do a git push --force up to my github fork and the local copy was just a fresh clone of microsoft/vcpkg and it did not have the branch with geotrans in it.

StarGate-One added 2 commits August 30, 2020 19:50

Update vcpkg.json by formatting it with vcpkg x-format-manifest

414c595

NancyLi1013 self-assigned this Aug 31, 2020

JackBoosY self-assigned this Aug 31, 2020

NancyLi1013 added the category:port-update The issue is with a library, which is requesting update new revision label Aug 31, 2020

JackBoosY suggested changes Aug 31, 2020

View reviewed changes

scripts/ci.baseline.txt Outdated Show resolved Hide resolved

ports/geotrans/vcpkg.json Outdated Show resolved Hide resolved

JackBoosY suggested changes Aug 31, 2020

View reviewed changes

ports/geotrans/portfile.cmake Show resolved Hide resolved

[vcpkg_download_distfile] Set timeout to 120s

0681fdb

JackBoosY added the category:tool-update The issue is with build tool or build script, which requires update or should be executed correctly label Aug 31, 2020

NancyLi1013 added the requires:author-response label Aug 31, 2020

StarGate-One added 2 commits August 31, 2020 12:44

Merge branch 'geotrans' of https://github.com/StarGate-One/vcpkg into…

8b20f0f

… geotrans

StarGate-One requested a review from JackBoosY August 31, 2020 18:43

Change portfile.cmake to download geotrans master.tgz

bd503be

Download master.tgz containing both Windows/Linux Binaries

BillyONeal added the depends:vm-update PR contains changes to the VM provisioning scripts label Sep 1, 2020

BillyONeal closed this Sep 1, 2020

BillyONeal reopened this Sep 1, 2020

JackBoosY removed the requires:author-response label Sep 2, 2020

NancyLi1013 mentioned this pull request Sep 9, 2020

Build osgearth with optional dependence dacro #13381

Closed

BillyONeal added the requires:discussion label Oct 28, 2020

BillyONeal reviewed Nov 5, 2020

View reviewed changes

BillyONeal removed the requires:discussion label Nov 5, 2020

StarGate-One closed this Nov 5, 2020

StarGate-One mentioned this pull request Nov 5, 2020

[geotrans] update to version 3.8 (on-hold waiting for CI/VM port 21 for ftp to be opened or alternate host site) #14412

Merged

Conversation

StarGate-One commented Aug 30, 2020

Uh oh!

StarGate-One commented Aug 31, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cenit commented Aug 31, 2020 • edited by JackBoosY Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

StarGate-One commented Aug 31, 2020

Uh oh!

cenit commented Aug 31, 2020

Uh oh!

StarGate-One commented Aug 31, 2020

Uh oh!

StarGate-One commented Aug 31, 2020

Uh oh!

JackBoosY commented Sep 1, 2020

Uh oh!

StarGate-One commented Sep 1, 2020

Uh oh!

BillyONeal commented Sep 1, 2020

Uh oh!

StarGate-One commented Sep 1, 2020

Uh oh!

BillyONeal commented Sep 1, 2020

Uh oh!

JackBoosY commented Sep 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BillyONeal commented Sep 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

StarGate-One commented Sep 3, 2020

Uh oh!

BillyONeal commented Sep 3, 2020

Uh oh!

StarGate-One commented Sep 3, 2020

Uh oh!

NancyLi1013 commented Oct 26, 2020

Uh oh!

StarGate-One commented Oct 26, 2020

Uh oh!

BillyONeal commented Oct 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

StarGate-One commented Oct 26, 2020

Uh oh!

BillyONeal left a comment

Choose a reason for hiding this comment

Uh oh!

BillyONeal Nov 5, 2020

Choose a reason for hiding this comment

Uh oh!

BillyONeal commented Nov 5, 2020

Uh oh!

StarGate-One commented Nov 5, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

cenit commented Aug 31, 2020 •

edited by JackBoosY

Loading

JackBoosY commented Sep 3, 2020 •

edited

Loading

BillyONeal commented Sep 3, 2020 •

edited

Loading

BillyONeal commented Oct 26, 2020 •

edited

Loading