Skip to content

Telemetry changes#17304

Merged
daxian-dbw merged 18 commits intoPowerShell:masterfrom
JamesWTruher:Telemetry04
May 17, 2022
Merged

Telemetry changes#17304
daxian-dbw merged 18 commits intoPowerShell:masterfrom
JamesWTruher:Telemetry04

Conversation

@JamesWTruher
Copy link
Copy Markdown
Collaborator

@JamesWTruher JamesWTruher commented May 10, 2022

PR Summary

  • Stop sending telemetry for application type
  • Send telemetry for parameters used when running pwsh
  • Include version number when reporting module import

PR Context

Our telemetry ingestion is currently very high with between 75-80% being taken up by ApplicationType metrics. The data doesn't seem to be showing a lot of variation over time, and isn't providing the value it did initially for its cost. We also have questions which are difficult to answer with the current telemetry (primarily disambiguating automation from interactive sessions). This PR includes metrics of how parameters of pwsh.exe are used. To reduce the payload size, the parameters are created as a bitmap which can then be investigated via kusto queries. In addition to collecting the used parameters, we are now capturing the value of ExecutionPolicy to help us understand the distribution of how this is being used. Lastly, this PR will also collect the version number of the loaded module where we can report it. With this we can make determinations as to how new modules are picked up, and how long older modules remain in use.

PR Checklist

@ghost ghost assigned daxian-dbw May 10, 2022
StevenBucher98
StevenBucher98 reviewed May 11, 2022
View reviewed changes
StevenBucher98
StevenBucher98 reviewed May 11, 2022
View reviewed changes
StevenBucher98
StevenBucher98 reviewed May 11, 2022
View reviewed changes
TravisEz13
TravisEz13 reviewed May 13, 2022
View reviewed changes
Copy link
Copy Markdown
Member

@TravisEz13 TravisEz13 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but would like @daxian-dbw

@TravisEz13 TravisEz13 added the CL-Engine Indicates that a PR should be marked as an engine change in the Change Log label May 13, 2022
SteveL-MSFT
SteveL-MSFT requested changes May 13, 2022
View reviewed changes
@ghost ghost added Waiting on Author The PR was reviewed and requires changes or comments from the author before being accept and removed Waiting on Author The PR was reviewed and requires changes or comments from the author before being accept labels May 13, 2022
daxian-dbw
daxian-dbw requested changes May 13, 2022
View reviewed changes
@ghost ghost added the Waiting on Author The PR was reviewed and requires changes or comments from the author before being accept label May 13, 2022
@ghost ghost removed the Waiting on Author The PR was reviewed and requires changes or comments from the author before being accept label May 16, 2022
PaulHigin
PaulHigin suggested changes May 16, 2022
View reviewed changes
@ghost ghost added Waiting on Author The PR was reviewed and requires changes or comments from the author before being accept and removed Waiting on Author The PR was reviewed and requires changes or comments from the author before being accept labels May 16, 2022
SteveL-MSFT
SteveL-MSFT approved these changes May 16, 2022
View reviewed changes
Copy link
Copy Markdown
Member

@SteveL-MSFT SteveL-MSFT left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

daxian-dbw
daxian-dbw requested changes May 16, 2022
View reviewed changes
src/Microsoft.PowerShell.ConsoleHost/host/msh/CommandLineParameterParser.cs Outdated
Copy link
Copy Markdown
Member

@daxian-dbw daxian-dbw May 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here, this line should be moved to _staMode = false; to keep consistent with the rest.

@ghost ghost added the Waiting on Author The PR was reviewed and requires changes or comments from the author before being accept label May 16, 2022
PaulHigin
PaulHigin approved these changes May 16, 2022
View reviewed changes
Copy link
Copy Markdown
Contributor

@PaulHigin PaulHigin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with switch statement suggestion.

@ghost ghost removed the Waiting on Author The PR was reviewed and requires changes or comments from the author before being accept label May 16, 2022
daxian-dbw
daxian-dbw approved these changes May 16, 2022
View reviewed changes
Copy link
Copy Markdown
Member

@daxian-dbw daxian-dbw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

JamesWTruher added 17 commits May 16, 2022 16:28
@JamesWTruher
Stop sending telemetry for the type of entity being executed.
738252e 
Also add telemetry for the parameters provided to pwsh.
@JamesWTruher
Remove the telemetry send from the pipeline.
45d2a55 
But keep support for it in the telemetry client.
@JamesWTruher
change parameter bitmap to something more sensible.
61a4ce9 
Tidy up alignment.
@JamesWTruher
Turn off developer mode for telemetry send.
fd29ad7 
Update comments to improve understanding.
@JamesWTruher
Do not send the start up event out of the send metric method.
8e501a1 
This is not needed and can result in us not sending the telemetry that includes the parameter data.
@JamesWTruher
first round of CodeFactor issues.
a498b4e
@JamesWTruher
Use a default value for the node if we can't create the telemetry.uui…
1bf3cba 
…d file.

This handles the case where we have a read-only file system and want to continue to capture telemetry.
@JamesWTruher
Fix test to reflect new behavior in startup telemetry send.
f6a0781
@JamesWTruher
improve comment about why we have a default node identifier.
d9d3f45
@JamesWTruher
Add more values to the parameter bitmap.
fc935f0 
Address some code review issues:
- Changed the logic of how a uuid file is created, now if we can't create the file, we'll use a default identifier.
- Make the assignment of the parameter bitmap more consistent.
 - update some comments to elaborate a bit more.
@JamesWTruher
Turn off developer mode.
d1dd59a
@JamesWTruher
Protect from a null Version property in the PSModuleInfo object.
2b8e572
@JamesWTruher
fix up code factor issue to use string instead of String.
6e63d35
@JamesWTruher
Change get execution policy map to use a switch.
4edd2ef
@JamesWTruher
another codefactor nit.
3982243
@JamesWTruher
change execution policy parsing back to if/else.
6ac116c
@JamesWTruher
Add Az.Tools.Predictor to list of reported modules (requested by Stev…
964e50e 
…en).
@pull-request-quantifier-deprecated
Copy link
Copy Markdown

pull-request-quantifier-deprecated bot commented May 16, 2022

This PR has 189 quantified lines of changes. In general, a change size of upto 200 lines is ideal for the best PR experience!

Quantification details 

Label      : Medium
Size       : +137 -52
Percentile : 57.8%

Total files changed: 6

Change summary by file extension:
.cs : +132 -39
.ps1 : +5 -13

Change counts above are quantified counts, based on the PullRequestQuantifier customizations.

Why proper sizing of changes matters

Optimal pull request sizes drive a better predictable PR flow as they strike a
balance between between PR complexity and PR review overhead. PRs within the
optimal size (typical small, or medium sized PRs) mean:

  • Fast and predictable releases to production:
    • Optimal size changes are more likely to be reviewed faster with fewer
      iterations.
    • Similarity in low PR complexity drives similar review times.
  • Review quality is likely higher as complexity is lower:
    • Bugs are more likely to be detected.
    • Code inconsistencies are more likely to be detetcted.
  • Knowledge sharing is improved within the participants:
    • Small portions can be assimilated better.
  • Better engineering practices are exercised:
    • Solving big problems by dividing them in well contained, smaller problems.
    • Exercising separation of concerns within the code changes.

What can I do to optimize my changes

  • Use the PullRequestQuantifier to quantify your PR accurately
    • Create a context profile for your repo using the context generator
    • Exclude files that are not necessary to be reviewed or do not increase the review complexity. Example: Autogenerated code, docs, project IDE setting files, binaries, etc. Check out the Excluded section from your prquantifier.yaml context profile.
    • Understand your typical change complexity, drive towards the desired complexity by adjusting the label mapping in your prquantifier.yaml context profile.
    • Only use the labels that matter to you, see context specification to customize your prquantifier.yaml context profile.
  • Change your engineering behaviors
    • For PRs that fall outside of the desired spectrum, review the details and check if:
      • Your PR could be split in smaller, self-contained PRs instead
      • Your PR only solves one particular issue. (For example, don't refactor and code new features in the same PR).

How to interpret the change counts in git diff output

  • One line was added: +1 -0
  • One line was deleted: +0 -1
  • One line was modified: +1 -1 (git diff doesn't know about modified, it will
    interpret that line like one addition plus one deletion)
  • Change percentiles: Change characteristics (addition, deletion, modification)
    of this PR in relation to all other PRs within the repository.

Was this comment helpful? 👍  :ok_hand:  :thumbsdown: (Email)
Customize PullRequestQuantifier for this repository.

@daxian-dbw daxian-dbw merged commit 65d8f11 into PowerShell:master May 17, 2022
@ghost
Copy link
Copy Markdown

ghost commented May 23, 2022

🎉v7.3.0-preview.4 has been released which incorporates this pull request.:tada:

Handy links:

@sdwheeler sdwheeler mentioned this pull request May 27, 2022
Closed
2 tasks
@TravisEz13 TravisEz13 mentioned this pull request Oct 14, 2022
Merged
22 tasks
@michaeltlombardi michaeltlombardi mentioned this pull request Oct 26, 2022
Closed
2 tasks
@sdwheeler sdwheeler mentioned this pull request Nov 17, 2022
Closed
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@TravisEz13 TravisEz13 TravisEz13 left review comments

@daxian-dbw daxian-dbw daxian-dbw approved these changes

@SteveL-MSFT SteveL-MSFT SteveL-MSFT approved these changes

@anmenaga anmenaga Awaiting requested review from anmenaga

+3 more reviewers

@PaulHigin PaulHigin PaulHigin approved these changes

@StevenBucher98 StevenBucher98 StevenBucher98 left review comments

@SydneyhSmith SydneyhSmith SydneyhSmith approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@daxian-dbw daxian-dbw

Labels

CL-Engine Indicates that a PR should be marked as an engine change in the Change Log Medium

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@JamesWTruher @daxian-dbw @TravisEz13 @SteveL-MSFT @PaulHigin @StevenBucher98 @SydneyhSmith