htslib 1.11 cannot read any files using HTTPS or S3 on macOS. I tested with the following files:
$ htsfile -vv https://github.com/brentp/cyvcf2/raw/master/cyvcf2/tests/test.vcf.gz
[D::init_add_plugin] Loaded "knetfile"
[D::init_add_plugin] Loaded "mem"
[D::init_add_plugin] Loaded "crypt4gh-needed"
[W::hfile_add_scheme_handler] Couldn't register scheme handler for s3
[W::hfile_add_scheme_handler] Couldn't register scheme handler for s3+http
[W::hfile_add_scheme_handler] Couldn't register scheme handler for s3+https
[D::init_add_plugin] Loaded "/Users/liang/miniconda3/envs/cyvcf2-test/libexec/htslib/hfile_s3.bundle"
[W::hfile_add_scheme_handler] Couldn't register scheme handler for s3w
[W::hfile_add_scheme_handler] Couldn't register scheme handler for s3w+http
[W::hfile_add_scheme_handler] Couldn't register scheme handler for s3w+https
[D::init_add_plugin] Loaded "/Users/liang/miniconda3/envs/cyvcf2-test/libexec/htslib/hfile_s3_write.bundle"
[W::hfile_add_scheme_handler] Couldn't register scheme handler for dict
[W::hfile_add_scheme_handler] Couldn't register scheme handler for file
[W::hfile_add_scheme_handler] Couldn't register scheme handler for ftp
[W::hfile_add_scheme_handler] Couldn't register scheme handler for ftps
[W::hfile_add_scheme_handler] Couldn't register scheme handler for gopher
[W::hfile_add_scheme_handler] Couldn't register scheme handler for http
[W::hfile_add_scheme_handler] Couldn't register scheme handler for https
[W::hfile_add_scheme_handler] Couldn't register scheme handler for imap
[W::hfile_add_scheme_handler] Couldn't register scheme handler for imaps
[W::hfile_add_scheme_handler] Couldn't register scheme handler for pop3
[W::hfile_add_scheme_handler] Couldn't register scheme handler for pop3s
[W::hfile_add_scheme_handler] Couldn't register scheme handler for rtsp
[W::hfile_add_scheme_handler] Couldn't register scheme handler for scp
[W::hfile_add_scheme_handler] Couldn't register scheme handler for sftp
[W::hfile_add_scheme_handler] Couldn't register scheme handler for smb
[W::hfile_add_scheme_handler] Couldn't register scheme handler for smbs
[W::hfile_add_scheme_handler] Couldn't register scheme handler for smtp
[W::hfile_add_scheme_handler] Couldn't register scheme handler for smtps
[W::hfile_add_scheme_handler] Couldn't register scheme handler for telnet
[W::hfile_add_scheme_handler] Couldn't register scheme handler for tftp
[D::init_add_plugin] Loaded "/Users/liang/miniconda3/envs/cyvcf2-test/libexec/htslib/hfile_libcurl.bundle"
[W::hfile_add_scheme_handler] Couldn't register scheme handler for gs
[W::hfile_add_scheme_handler] Couldn't register scheme handler for gs+http
[W::hfile_add_scheme_handler] Couldn't register scheme handler for gs+https
[D::init_add_plugin] Loaded "/Users/liang/miniconda3/envs/cyvcf2-test/libexec/htslib/hfile_gcs.bundle"
htsfile: can't open "https://github.com/brentp/cyvcf2/raw/master/cyvcf2/tests/test.vcf.gz": Protocol not supported
This error persists using the bioconda's htslib 1.11 or the one I built on macOS using the following steps:
./configure --prefix=/tmp/local --enable-libcurl --with-libdeflate --enable-plugins --with-plugin-dir=/tmp/local/htslib --enable-gcs --enable-s3
make -j4
make install
/tmp/local/bin/htsfile ...
It seems that the plugin bundle is loaded after registering the protocol handler. I came across with this issue while investigating brentp/cyvcf2#174 which links to htslib 1.10. Version 1.10 can read the two files fine using the build from bioconda or self-compiled:
$ htsfile -vv 'https://github.com/brentp/cyvcf2/raw/master/cyvcf2/tests/test.vcf.gz'
[D::init_add_plugin] Loaded "knetfile"
[D::init_add_plugin] Loaded "mem"
[D::init_add_plugin] Loaded "/Users/liang/miniconda3/envs/cyvcf2/libexec/htslib/hfile_s3.bundle"
[D::init_add_plugin] Loaded "/Users/liang/miniconda3/envs/cyvcf2/libexec/htslib/hfile_s3_write.bundle"
[D::init_add_plugin] Loaded "/Users/liang/miniconda3/envs/cyvcf2/libexec/htslib/hfile_libcurl.bundle"
[D::init_add_plugin] Loaded "/Users/liang/miniconda3/envs/cyvcf2/libexec/htslib/hfile_gcs.bundle"
https://github.com/brentp/cyvcf2/raw/master/cyvcf2/tests/test.vcf.gz: VCF version 4.1 BGZF-compressed variant calling data
$ set -x AWS_DEFAULT_REGION us-east-1
$ htsfile -vv 's3://3kricegenome/test/test.vcf.gz'
[D::init_add_plugin] Loaded "knetfile"
[D::init_add_plugin] Loaded "mem"
[D::init_add_plugin] Loaded "/Users/liang/miniconda3/envs/cyvcf2/libexec/htslib/hfile_s3.bundle"
[D::init_add_plugin] Loaded "/Users/liang/miniconda3/envs/cyvcf2/libexec/htslib/hfile_s3_write.bundle"
[D::init_add_plugin] Loaded "/Users/liang/miniconda3/envs/cyvcf2/libexec/htslib/hfile_libcurl.bundle"
[D::init_add_plugin] Loaded "/Users/liang/miniconda3/envs/cyvcf2/libexec/htslib/hfile_gcs.bundle"
s3://3kricegenome/test/test.vcf.gz: VCF version 4.1 BGZF-compressed variant calling data
htslib 1.11 cannot read any files using HTTPS or S3 on macOS. I tested with the following files:
This error persists using the bioconda's htslib 1.11 or the one I built on macOS using the following steps:
It seems that the plugin bundle is loaded after registering the protocol handler. I came across with this issue while investigating brentp/cyvcf2#174 which links to htslib 1.10. Version 1.10 can read the two files fine using the build from bioconda or self-compiled: