• Sourcegraph version: 3.40.0
• Platform information: sourcegraph/deploy-sourcegraph@v3.40.0

Steps to reproduce:

1. Host GitHub Enterprise Server with a TLS certificate that is issued by an internal Certificate Authority.
2. Configure two GitHub code hosts, one for the public GitHub.com and another for your GitHub Enterprise Server:

   "GITHUB": [
     {
       "url": "https://github.com",
       "token": "REDACTED",
       "orgs": ["sourcegraph"]
     },
     {
       "url": "https://github.internal",
       "token": "REDACTED",
       "orgs": ["sourcegraph"]
     }
   ]

3. Add the internal CA’s certificate to Sourcegraph’s site configuration under experimentalFeatures, so gitserver can connect to github.internal:

   "experimentalFeatures": {
     "tls.external": {
       "certificates": [
         // Internal CA
         "-----BEGIN CERTIFICATE-----\n…\n-----END CERTIFICATE-----"        
       ]
     }
   }

4. Wait for gitserver to clone all the repositories.

Expected behavior:

1. repo-updater discovers all the repositories in github.com/sourcegraph and github.internal/sourcegraph
2. gitserver clones all the repositories in github.com/sourcegraph and github.internal/sourcegraph

Actual behavior:

1. repo-updater discovers all the repositories in github.com/sourcegraph and github.internal/sourcegraph
2. gitserver only clones all the repositories in github.internal/sourcegraph
3. gitserver fails to clone any repository in github.com/sourcegraph with the following warning:

   t=2022-07-01T20:29:33+0000
   lvl=warn msg="error cloning repo"
   repo=github.com/sourcegraph/sourcegraph
   err="error cloning repo: repo github.com/sourcegraph/sourcegraph not cloneable: exit status 128 (output follows)
   
   fatal: unable to access 'https://github.com/sourcegraph/sourcegraph/': SSL certificate problem: unable to get local issuer certificate"
   

Analysis

This happens because GitHub.com uses a TLS certificate whose root is DigiCert Global Root CA, which is a system certificate that is not mentioned in experimentalFeatures.tls.enabled.certificates.

This bug was initially mentioned in #71, but never got fixed because we had a workaround. I’m re-reporting it because I needed to update the workaround again.

@keegancsmith added a debug log in #22285, in case SystemCertPool fails to load, but I found no evidence of this error in our logs. I’d also like to point out that repo-updater was totally fine using system certificates, so my hypothesis is that gitserver isn’t going down this codepath.

Reading the error message, it really looks like it is coming from the actual git binary, which makes me suspect the GIT_SSL_CAINFO environment variable, which is set by #8092:
https://github.com/sourcegraph/sourcegraph/blob/3518a54ce82699d0e1fe3b19b21dd6eb2412ce5e/cmd/gitserver/server/serverutil.go#L115-L129

Note that this only writes the contents of the configuration, but doesn’t prepend /etc/ssl/certs/ca-certificates.crt, which exists in sourcegraph/gitserver@3.40.0. If this were prepended, all the system certificates would be included and I bet this would fix the bug.

Workaround

Add the DigiCert Global Root CA certificate alongside your internal CA certificate:

"experimentalFeatures": {
  "tls.external": {
    "certificates": [
      // DigiCert Global Root CA
      "-----BEGIN CERTIFICATE-----\nMIIDrzCCApegAwIBAgIQCDvgVpBCRrGhdWrJWZHHSjANBgkqhkiG9w0BAQUFADBh\nMQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3\nd3cuZGlnaWNlcnQuY29tMSAwHgYDVQQDExdEaWdpQ2VydCBHbG9iYWwgUm9vdCBD\nQTAeFw0wNjExMTAwMDAwMDBaFw0zMTExMTAwMDAwMDBaMGExCzAJBgNVBAYTAlVT\nMRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2VydC5j\nb20xIDAeBgNVBAMTF0RpZ2lDZXJ0IEdsb2JhbCBSb290IENBMIIBIjANBgkqhkiG\n9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4jvhEXLeqKTTo1eqUKKPC3eQyaKl7hLOllsB\nCSDMAZOnTjC3U/dDxGkAV53ijSLdhwZAAIEJzs4bg7/fzTtxRuLWZscFs3YnFo97\nnh6Vfe63SKMI2tavegw5BmV/Sl0fvBf4q77uKNd0f3p4mVmFaG5cIzJLv07A6Fpt\n43C/dxC//AH2hdmoRBBYMql1GNXRor5H4idq9Joz+EkIYIvUX7Q6hL+hqkpMfT7P\nT19sdl6gSzeRntwi5m3OFBqOasv+zbMUZBfHWymeMr/y7vrTC0LUq7dBMtoM1O/4\ngdW7jVg/tRvoSSiicNoxBN33shbyTApOB6jtSj1etX+jkMOvJwIDAQABo2MwYTAO\nBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUA95QNVbR\nTLtm8KPiGxvDl7I90VUwHwYDVR0jBBgwFoAUA95QNVbRTLtm8KPiGxvDl7I90VUw\nDQYJKoZIhvcNAQEFBQADggEBAMucN6pIExIK+t1EnE9SsPTfrgT1eXkIoyQY/Esr\nhMAtudXH/vTBH1jLuG2cenTnmCmrEbXjcKChzUyImZOMkXDiqw8cvpOp/2PV5Adg\n06O/nVsJ8dWO41P0jmP6P6fbtGbfYmbW0W5BjfIttep3Sp+dWOIrWcBAI+0tKIJF\nPnlUkiaY4IBIqDfv8NZ5YBberOgOzW6sRBc4L0na4UU+Krk2U886UAb3LujEV0ls\nYSEY1QSteDwsOoBrp+uvFRTp2InBuThs4pFsiv9kuXclVzDAGySj4dzp30d8tbQk\nCAUw7C29C79Fv1C5qfPrmAESrciIxpg0X40KPMbp1ZWVbd4=\n-----END CERTIFICATE-----",
      // Internal CA
      "-----BEGIN CERTIFICATE-----\n…\n-----END CERTIFICATE-----"        
    ]
  }
}