Skip to content

Print step output into a single buffer#619

Merged
mrnugget merged 3 commits into
mainfrom
mrn/step-output-single-buffer
Sep 17, 2021
Merged

Print step output into a single buffer#619
mrnugget merged 3 commits into
mainfrom
mrn/step-output-single-buffer

Conversation

@mrnugget

@mrnugget mrnugget commented Sep 17, 2021

Copy link
Copy Markdown
Contributor

This is a side-quest to EM2 https://github.com/sourcegraph/sourcegraph/issues/24421 because we realised that we'll make things harder for ourselves if we separate stdout/stderr into separate buffers.

We'd have to zip them up together in the UI but that's hard without having additional timing information.

But timing information is also a bit overkill (at least for now) so we thought we'd use a single buffer.

That's what this PR here contains.

Stdout and stderr are logged into a single buffer, each line prefixed with stdout: and stderr: respectively.

That works because the process.PipeOutput function only writes lines (not chunks) to the passed in writers. So it works even if commands print half a line on stdout, then half a line on stderr, and only then the rest.

Example

Steps

steps:
  - run: |
      echo "this is step 1" >> README.txt
      echo "this is step 1 stdout"
      echo "this is step 1 stderr" 1>&2
      echo -n "this is step 1 stdout half a line..."
      echo -n "this is step 1 stderr half a line..." 1>&2
      echo -n "...this the other half of step 1 stdout line"
      echo -n "...this the other half of step 1 stderr line" 1>&2
    container: alpine:3
  - run: |
      echo "this is step 2" >> README.md
      echo "this is step 2 stdout"
      echo "this is step 2 stderr" 1>&2
    container: alpine:3
  - run: |
      echo "this is step 3" >> README.md
      echo "this is step 3 stdout"
      echo "this is step 3 stderr" 1>&2
    container: alpine:3
    outputs:
      myOutput:
        value: "my-output.txt"
  - run: |
      echo "this is step 4" >> README.md
      cat README.md
      echo "previous_step.modified_files=${{ previous_step.modified_files }}" >> README.md
      ls -lrt
    container: alpine:3
  - run: echo "this is step 5" >> ${{ outputs.myOutput }}
    container: alpine:3

The resulting TASK_STEP JSON lines that contain output:

{"operation":"TASK_STEP","timestamp":"2021-09-17T09:27:45.928Z","status":"PROGRESS","metadata":{"out":"stdout: this is step 1 stdout\nstdout: \nstderr: this is step 1 stderr\nstderr: \n","step":0,"taskID":"65qO4Zovx7V"}}
{"operation":"TASK_STEP","timestamp":"2021-09-17T09:27:45.947Z","status":"PROGRESS","metadata":{"out":"stderr: this is step 1 stderr half a line......this the other half of step 1 stdout line\nstderr: \nstdout: this is step 1 stdout half a line......this the other half of step 1 stdout line\nstdout: \n","step":0,"taskID":"65qO4Zovx7V"}}
{"operation":"TASK_STEP","timestamp":"2021-09-17T09:27:47.357Z","status":"PROGRESS","metadata":{"out":"stderr: this is step 2 stderr\nstderr: \nstdout: this is step 2 stdout\nstdout: \n","step":1,"taskID":"65qO4Zovx7V"}}
{"operation":"TASK_STEP","timestamp":"2021-09-17T09:27:48.835Z","status":"PROGRESS","metadata":{"out":"stderr: this is step 3 stderr\nstderr: \nstdout: this is step 3 stdout\nstdout: \n","step":2,"taskID":"65qO4Zovx7V"}}
{"operation":"TASK_STEP","timestamp":"2021-09-17T09:27:50.235Z","status":"PROGRESS","metadata":{"out":"stdout: # automation-testing\nstdout: \nstdout: This repository is used to test opening and closing pull request with Automation\nstdout: \nstdout: \nstdout: \nstdout: (c) Copyright Sourcegraph 2013-2020.\nstdout: \nstdout: (c) Copyright Sourcegraph 2013-2020.\nstdout: \nstdout: (c) Copyright Sourcegraph 2013-2020.this is step 2\nstdout: \nstdout: this is step 3\nstdout: \nstdout: this is step 4\nstdout: \nstdout: total 56\nstdout: \nstdout: -rw-rw-rw-    1 1000     1000            17 Sep 17 09:27 test.md\nstdout: \nstdout: drwxrwxrwx    4 1000     1000          4096 Sep 17 09:27 project2\nstdout: \nstdout: drwxrwxrwx    2 1000     1000          4096 Sep 17 09:27 project1\nstdout: \nstdout: drwxrwxrwx    2 1000     1000          4096 Sep 17 09:27 gopkg\nstdout: \nstdout: -rw-rw-rw-    1 1000     1000             0 Sep 17 09:27 foobar\nstdout: \nstdout: -rw-rw-rw-    1 1000     1000            17 Sep 17 09:27 file_without_newline_at_eof.txt\nstdout: \nstdout: -rw-rw-rw-    1 1000     1000           434 Sep 17 09:27 file_with_multiple_lines.txt\nstdout: \nstdout: -rw-rw-rw-    1 1000     1000            23 Sep 17 09:27 file3.txt\nstdout: \nstdout: -rw-rw-rw-    1 1000     1000            23 Sep 17 09:27 file2.txt\nstdout: \nstdout: -rw-rw-rw-    1 1000     1000            23 Sep 17 09:27 file1.txt\nstdout: \nstdout: drwxrwxrwx    3 1000     1000          4096 Sep 17 09:27 examples\nstdout: \nstdout: -rw-rw-rw-    1 1000     1000           706 Sep 17 09:27 circle-ci.yml\nstdout: \nstdout: -rw-rw-rw-    1 1000     1000           153 Sep 17 09:27 Dockerfile\nstdout: \nstdout: -rw-r--r--    1 root     root            15 Sep 17 09:27 README.txt\nstdout: \nstdout: -rw-rw-rw-    1 1000     1000           299 Sep 17 09:27 README.md\nstdout: \n","step":3,"taskID":"65qO4Zovx7V"}}

@mrnugget
Print step output into a single buffer
16aed76 
This is a side-quest to EM2
https://github.com/sourcegraph/sourcegraph/issues/24421 because we
realised that we'll make things harder for ourselves if we separate
stdout/stderr into separate buffers.

We'd have to zip them up together
in the UI but that's hard without having additional timing information.

But timing information is also a bit overkill (at least for now) so we
thought we'd use a single buffer.

That's what this PR here contains.

Stdout and stderr are logged into a single buffer, each line prefixed with
`stdout: ` and `stderr: ` respectively.

That works because the `process.PipeOutput` function only writes lines
(not chunks) to the passed in writers. So it works even if commands print half a
line on stdout, then half a line on stderr, and only then the rest.
@mrnugget mrnugget requested a review from eseliger September 17, 2021 09:36
@mrnugget mrnugget requested a review from a team September 17, 2021 11:47
@mrnugget mrnugget merged commit 157a86d into main Sep 17, 2021
@mrnugget mrnugget deleted the mrn/step-output-single-buffer branch September 17, 2021 12:58
scjohns pushed a commit that referenced this pull request Apr 24, 2023
@mrnugget
Print step output into a single buffer (#619)
71644a1 
This is a side-quest to EM2
https://github.com/sourcegraph/sourcegraph/issues/24421 because we
realised that we'll make things harder for ourselves if we separate
stdout/stderr into separate buffers.

We'd have to zip them up together
in the UI but that's hard without having additional timing information.

But timing information is also a bit overkill (at least for now) so we
thought we'd use a single buffer.

That's what this PR here contains.

Stdout and stderr are logged into a single buffer, each line prefixed with
`stdout: ` and `stderr: ` respectively.

That works because the `process.PipeOutput` function only writes lines
(not chunks) to the passed in writers. So it works even if commands print half a
line on stdout, then half a line on stderr, and only then the rest.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@eseliger eseliger eseliger approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@mrnugget @eseliger