Skip to content

[bug] Fix time log error in PipelineEngine #1934

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
jeffra merged 1 commit into from
May 3, 2022
Merged

[bug] Fix time log error in PipelineEngine #1934

jeffra merged 1 commit into from
May 3, 2022

Conversation

@Codle
Copy link
Contributor

@Codle Codle commented May 3, 2022

In engine.py line 360 to 366, the codes want to print something in seconds:

  elapsed = self.timers('train_batch').elapsed(reset=True)
  iter_time = elapsed / self.steps_per_print()
  tput = self.train_batch_size() / iter_time
  print(f'steps: {self.global_steps} '
        f'loss: {self.agg_train_loss:0.4f} '
        f'iter time (s): {iter_time:0.3f} '
        f'samples/sec: {tput:0.3f}')

But in timer, it get elapsed time by _get_elapsed_msec method which returns time in msec.

def elapsed(self, reset=True):
"""Calculate the elapsed time."""
    started_ = self.started_
    # If the timing in progress, end it first.
    if self.started_:
        self.stop()
    # Get the elapsed time.
    elapsed_ = self._get_elapsed_msec()
    # Reset the elapsed time
    if reset:
        self.reset()
    # If timing was in progress, set it back.
    if started_:
        self.start()
    return elapsed_

@jeffra jeffra merged commit a3b9003 into deepspeedai:master May 3, 2022
@yzs981130 yzs981130 mentioned this pull request Aug 19, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tjruwase tjruwase tjruwase approved these changes

@jeffra jeffra Awaiting requested review from jeffra

@samyam samyam Awaiting requested review from samyam

@ShadenSmith ShadenSmith Awaiting requested review from ShadenSmith

@conglongli conglongli Awaiting requested review from conglongli

@awan-10 awan-10 Awaiting requested review from awan-10

@cli99 cli99 Awaiting requested review from cli99

@eltonzheng eltonzheng Awaiting requested review from eltonzheng

@minjiaz minjiaz Awaiting requested review from minjiaz

@RezaYazdaniAminabadi RezaYazdaniAminabadi Awaiting requested review from RezaYazdaniAminabadi

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Codle @tjruwase @jeffra