UTF-8 and GBK encoding bug in gitlab-runner + virtualbox + pwsh
## Summary
<!--
Briefly summarize the bug
-->
gitlab-runner in ubuntu is UTF-8 encoding. pwsh in win10 is gb2312 encoding. My project information & username include Chinese. In Ubuntu gitlab-runner + virtualbox + win10-21H2(Simplified Chinese) + pwsh, I got a pwsh syntax error. I guess it is related to character encoding.
## Steps to reproduce
<!--
What do you need to do to reproduce the bug? Please include job definitions or git repository structure if relevant
-->
<!--
Please add the definition of the job from `.gitlab-ci.yml` that is failing
inside of the code blocks (```) below.
-->
<details>
<summary> .gitlab-ci.yml </summary>
```yml
stages:
- build
- test
- deploy
build-bin:
stage: build
tags:
- IAR
script:
- make COMPILER=iar SYSTEM_TYPE=Bare PLATFORM=Stm32f10x
- cp output/Bare/Stm32f10x/iar/prjname/Release/Exe/test.bin test.bin
artifacts:
name: "$CI_JOB_NAME"
paths:
- test.bin
```
</details>
In Ubuntu 18.04, I register a gitlab-runner(v15.1.0) with ```executor = "virtualbox"``` and ```shell = "pwsh"```. The virtualbox has installed Win10-21H2(Simplified Chinese) and PowerShell v7.2.5. When I trigger the CI/CD, it will got an error:
```powershell
ParserError:
Line |
211 | $CI_DISPOSABLE_ENVIRONMENT="true"
| ~~~~~
| Unexpected token 'true" $env:CI_DISPOSABLE_ENVIRONMENT=$CI_DISPOSABLE_ENVIRONMENT
| $CI_RUNNER_VERSION="15.1.0" $env:CI_RUNNER_VERSION=$CI_RUNNER_VERSION $CI_RUNNER_REVISION="76984217"
| $env:CI_RUNNER_REVISION=$CI_RUNNER_REVISION $CI_RUNNER_EXECUTABLE_ARCH="linux/amd64"
| $env:CI_RUNNER_EXECUTABLE_ARCH=$CI_RUNNER_EXECUTABLE_ARCH $GIT_LFS_SKIP_SMUDGE="1"
| $env:GIT_LFS_SKIP_SMUDGE=$GIT_LFS_SKIP_SMUDGE echo "n expression or statement.
```
I wrote an article to analyze and fix this problem: https://blog.csdn.net/IChen86/article/details/125549627
However, my solution is not the best. Please fix this problem officially.
## Actual behavior
<!--
What actually happens
-->
I got a pwsh syntax error.
## Expected behavior
<!--
What you should see instead
-->
It should run and exit with ok.
## Relevant logs and/or screenshots
<!--
Paste the job logs inside of the code blocks (```) below so it would be
easier to read.
-->
<details>
<summary> job log </summary>
```sh
Running with gitlab-runner 15.1.0 (76984217)
on virtual_box_win10 WXfibXkD
Preparing the "virtualbox" executor
00:19
Using VirtualBox version 6.1.34r150636 executor...
Restoring VM from snapshot...
Starting VM...
Waiting for VM to become responsive...
Starting SSH command...
Preparing environment
00:04
Running on DESKTOP-4AA79FO via
PowerEdge-R740xd...
Getting source from Git repository
00:02
ParserError:
Line |
211 | $CI_DISPOSABLE_ENVIRONMENT="true"
| ~~~~~
| Unexpected token 'true" $env:CI_DISPOSABLE_ENVIRONMENT=$CI_DISPOSABLE_ENVIRONMENT
| $CI_RUNNER_VERSION="15.1.0" $env:CI_RUNNER_VERSION=$CI_RUNNER_VERSION $CI_RUNNER_REVISION="76984217"
| $env:CI_RUNNER_REVISION=$CI_RUNNER_REVISION $CI_RUNNER_EXECUTABLE_ARCH="linux/amd64"
| $env:CI_RUNNER_EXECUTABLE_ARCH=$CI_RUNNER_EXECUTABLE_ARCH $GIT_LFS_SKIP_SMUDGE="1"
| $env:GIT_LFS_SKIP_SMUDGE=$GIT_LFS_SKIP_SMUDGE echo "n expression or statement.
error: could not lock config file builds/xxx/yyy.tmp/git-template/config: No such file or directory
ERROR: Job failed: Process exited with status 1
```
</details>
## Environment description
<!--
Are you using shared Runners on GitLab.com? Or is it a custom installation?
Which executors are used? Please also provide the versions of related tools
like `docker info` if you are using the Docker executor.
-->
<!--
Please add the contents of `config.toml` inside of the code blocks (```)
below, remember to remove any secret tokens!
-->
<details>
<summary> config.toml contents </summary>
```toml
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "virtual_box_win10"
url = "http://xxx.yyy.zzz/"
token = "WXfibXkDcdGiYCbNBuoz"
executor = "virtualbox"
shell = "pwsh"
debug_trace_disabled = false
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.ssh]
user = "AAAA"
password = "BBBBCCCC"
identity_file = "/root/.ssh/virtualbox_id_rsa"
disable_strict_host_key_checking = true
[runners.virtualbox]
base_name = "Windows10_x64"
base_folder = ""
disable_snapshots = false
```
</details>
### Used GitLab Runner version
<!--
Please run and paste the output of `gitlab-runner --version`. If you are using
a Runner where you don't have access to, please paste at least the first lines
the from build log, like:
```
Running with gitlab-ci-multi-runner 1.4.2 (bcc1794)
Using Docker executor with image golang:1.8 ...
```
-->
gitlab-runner 15.1.0 (76984217)
## Possible fixes
<!--
(If you can, link to the line of code that might be responsible for the problem)
--->
In executors/virtualbox/virtualbox.go, an incompatible fix:
```go
package virtualbox
import (
"errors"
"fmt"
"time"
"gitlab.com/gitlab-org/gitlab-runner/common"
"gitlab.com/gitlab-org/gitlab-runner/executors"
"gitlab.com/gitlab-org/gitlab-runner/executors/vm"
"gitlab.com/gitlab-org/gitlab-runner/helpers/ssh"
vbox "gitlab.com/gitlab-org/gitlab-runner/helpers/virtualbox"
"github.com/NuoMinMin/mahonia" // UTF-8 转 GBK
)
......
func (s *executor) Run(cmd common.ExecutorCommand) error {
// s.Println("s.BuildShell.CmdLine: " + s.BuildShell.CmdLine)
// s.Println("cmd.Script: " + cmd.Script)
// 将 UTF-8 转换成 gbk, 因为 pwsh 用的是 gbk 编码, 不这么修改的话, 有些字会被解析成 " 导致执行 pwsh 报错
cmd.Script, _ = mahonia.NewEncoder("gbk").ConvertStringOK(cmd.Script)
err := s.sshCommand.Run(cmd.Context, ssh.Command{
Command: s.BuildShell.CmdLine,
Stdin: cmd.Script,
})
if exitError, ok := err.(*ssh.ExitError); ok {
exitCode := exitError.ExitCode()
err = &common.BuildError{Inner: err, ExitCode: exitCode}
}
return err
}
......
```
issue