{"id":560,"date":"2018-08-20T08:00:56","date_gmt":"2018-08-20T14:00:56","guid":{"rendered":"https:\/\/practicalpowershell.com\/?p=560"},"modified":"2020-03-14T16:10:41","modified_gmt":"2020-03-14T22:10:41","slug":"new-script-be-prepared","status":"publish","type":"post","link":"https:\/\/practicalpowershell.com\/new-script-be-prepared\/","title":{"rendered":"New Script? Be Prepared!"},"content":{"rendered":"

\"\"<\/a>While writing up some new scripts I ran into some issues which resulting in some troubleshooting, brainstorming and re-writing of the original code.\u00a0 This article was written to pass along some of the ‘lessons learned’:

\n(A) Large Data Sets<\/em> – Scripts that work in small environments will often fail or run very slowly in a larger environment.\u00a0 Additional considerations, coding and workarounds may be needed when designing and writing scripts for larger environments.

\nExamples<\/u>

\nProgress bar:<\/em> If you cannot quantify the entire set of data, using Write-Progress might be a waste of time. Once the Write-Progress process in PowerShell gets over a certain percentage (say 200%) errors begin to occur. The script is still running in the background, but ref error messages will appear for this.

\nI have run into this specifically when using paging with Office 365 connections. For message tracing paging means that we can examine 5000 messages at a time up to a total of 200 pages of these. The problem is that unless you know the total number of pages, then calculating the max value to use for a counter and calculate a percentage will cause write-progress to be inaccurate. Similarly, when pulling information via EWS, there are limits of around 1000 for some and 10000 for others where calculating totals can cause similar issues.

\nThe lesson here is if you can calculate totals and thus percentages, then Write-Progress will be your friend. Otherwise the Write-Progress cmdlet will generate a lot of errors.

\nOffice 365 Throttling: <\/em>This one is hard to quantify because Microsoft will throttle large data sets. There is not set limit of value that will trigger this throttling. Best that you can do is to try not to query a large number of objects at a time. Perhaps building in some sort of pauses. However, this is also not guaranteed to remove Microsoft’s throttling. In other words, be prepared for when it happens and plan to deal with those delays if the data is more urgent.

\nExamples of Throttling:<\/u>
\nPowerShell:<\/em>

\n\"\"<\/a>
\nAnd
\n\"\"<\/a>
\nAdmin Console (Yes. The Admin Console can also be throttled)
\n\"\"<\/a>
\nFilter vs. Where: Use Filter whenever possible and use Where as a last result. This will help with querying large data sets.

\n(B) Files ingested for processing were incorrect<\/em> – when working with a set of files for data process in PowerShell, it is better to make sure that the files pulled will be as accurate as possible. What does this mean?

\nFor example if a script exports a series of CSV files to a given path and then another part of the script dumps CSV files to the same file location, then the same script ingests all CSV files instead of a particular subset that needs to be compared.
\nFor example:
\nA PowerShell script exports these CSV files:
\n[sourcecode language=”powershell”]
\nUser1-MailItems.csv
\nUser2-MailItems.csv
\nUser1-Senders.csv
\nUser2-Senders.csv
\n[\/sourcecode]
\nAs you can tell by the naming, there are two sets of data. Now if we were to accidentally tell PowerShell to grab all *.csv files, we would have too many for our comparison. Make sure to refine your file search to exactly the ones you need, so the query would be something like ‘*-MailItems.csv’ or ‘*-Senders.csv’. That would create two sets of data with two CSV files in each.
\n(C) Unique isn’t always Unique:<\/em> PowerShell has many ways to handle data to ensure uniqueness in a particular set. I won’t dig too far into this as there is a great article already out there that can explain this:

\nhttps:\/\/blogs.technet.microsoft.com\/heyscriptingguy\/2012\/01\/15\/use-powershell-to-choose-unique-objects-from-a-sorted-list\/<\/a>

\nSo make sure you choose the correct ‘unique’ that you need.
\n(D) Break up your Script:<\/em> While working on troubleshooting issue, pull out chunks of code to validate they work when not inside a larger script. This could include function code or code run from within a loop.

\nExample<\/u>
\nFunction 1 creates reference files for Function 2
\nFunction 2 fails.

\nAn easy way to troubleshoot the second function is to copy the files created by Function 1 and drop them into a working file directory (i.e. c:\\Scripts\\Working). Then take the PowerShell code out of Function 2, add some additional error checking (Try\/Catch) \/ reporting (write-host or write-verbose) depending on what Function 2 does and re-run the script against these files. Once this function is fixed, place the code back into the original function shell (Function \n","protected":false},"excerpt":{"rendered":"

While writing up some new scripts I ran into some issues which resulting in some troubleshooting, brainstorming and re-writing of the original code.\u00a0 This article was written to pass along some of the ‘lessons learned’: (A) Large Data Sets – Scripts that work in small environments will often fail or run very slowly in a […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rop_custom_images_group":[],"rop_custom_messages_group":[],"rop_publish_now":"initial","rop_publish_now_accounts":[],"rop_publish_now_history":[],"rop_publish_now_status":"pending","_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-560","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"\nNew Script? Be Prepared! - Practical PowerShell<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/practicalpowershell.com\/new-script-be-prepared\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"New Script? Be Prepared! - Practical PowerShell\" \/>\n<meta property=\"og:description\" content=\"While writing up some new scripts I ran into some issues which resulting in some troubleshooting, brainstorming and re-writing of the original code.\u00a0 This article was written to pass along some of the ‘lessons learned’: (A) Large Data Sets – Scripts that work in small environments will often fail or run very slowly in a […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/practicalpowershell.com\/new-script-be-prepared\/\" \/>\n<meta property=\"og:site_name\" content=\"Practical PowerShell\" \/>\n<meta property=\"article:published_time\" content=\"2018-08-20T14:00:56+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2020-03-14T22:10:41+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/practicalpowershell.com\/wp-content\/uploads\/2020\/03\/BePrepared.jpg\" \/>\n<meta name=\"author\" content=\"damian\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"damian\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/practicalpowershell.com\\\/new-script-be-prepared\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/practicalpowershell.com\\\/new-script-be-prepared\\\/\"},\"author\":{\"name\":\"damian\",\"@id\":\"https:\\\/\\\/practicalpowershell.com\\\/#\\\/schema\\\/person\\\/4d0733c81966e744aabbb49f56d64deb\"},\"headline\":\"New Script? Be Prepared!\",\"datePublished\":\"2018-08-20T14:00:56+00:00\",\"dateModified\":\"2020-03-14T22:10:41+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/practicalpowershell.com\\\/new-script-be-prepared\\\/\"},\"wordCount\":733,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/practicalpowershell.com\\\/#\\\/schema\\\/person\\\/4d0733c81966e744aabbb49f56d64deb\"},\"image\":{\"@id\":\"https:\\\/\\\/practicalpowershell.com\\\/new-script-be-prepared\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/practicalpowershell.com\\\/wp-content\\\/uploads\\\/2020\\\/03\\\/BePrepared.jpg\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/practicalpowershell.com\\\/new-script-be-prepared\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/practicalpowershell.com\\\/new-script-be-prepared\\\/\",\"url\":\"https:\\\/\\\/practicalpowershell.com\\\/new-script-be-prepared\\\/\",\"name\":\"New Script? Be Prepared! - Practical PowerShell\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/practicalpowershell.com\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/practicalpowershell.com\\\/new-script-be-prepared\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/practicalpowershell.com\\\/new-script-be-prepared\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/practicalpowershell.com\\\/wp-content\\\/uploads\\\/2020\\\/03\\\/BePrepared.jpg\",\"datePublished\":\"2018-08-20T14:00:56+00:00\",\"dateModified\":\"2020-03-14T22:10:41+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/practicalpowershell.com\\\/new-script-be-prepared\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/practicalpowershell.com\\\/new-script-be-prepared\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/practicalpowershell.com\\\/new-script-be-prepared\\\/#primaryimage\",\"url\":\"https:\\\/\\\/i0.wp.com\\\/practicalpowershell.com\\\/wp-content\\\/uploads\\\/2020\\\/03\\\/BePrepared.jpg?fit=150%2C99&ssl=1\",\"contentUrl\":\"https:\\\/\\\/i0.wp.com\\\/practicalpowershell.com\\\/wp-content\\\/uploads\\\/2020\\\/03\\\/BePrepared.jpg?fit=150%2C99&ssl=1\",\"width\":150,\"height\":99},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/practicalpowershell.com\\\/new-script-be-prepared\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/practicalpowershell.com\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"New Script? Be Prepared!\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/practicalpowershell.com\\\/#website\",\"url\":\"https:\\\/\\\/practicalpowershell.com\\\/\",\"name\":\"Practical PowerShell\",\"description\":\"PowerShell books written by experts\",\"publisher\":{\"@id\":\"https:\\\/\\\/practicalpowershell.com\\\/#\\\/schema\\\/person\\\/4d0733c81966e744aabbb49f56d64deb\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/practicalpowershell.com\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\\\/\\\/practicalpowershell.com\\\/#\\\/schema\\\/person\\\/4d0733c81966e744aabbb49f56d64deb\",\"name\":\"damian\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/d5a8cc64a5aa27558a897b108e3be1a89859511a3fd26176dac292f26e7a4ae4?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/d5a8cc64a5aa27558a897b108e3be1a89859511a3fd26176dac292f26e7a4ae4?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/d5a8cc64a5aa27558a897b108e3be1a89859511a3fd26176dac292f26e7a4ae4?s=96&d=mm&r=g\",\"caption\":\"damian\"},\"logo\":{\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/d5a8cc64a5aa27558a897b108e3be1a89859511a3fd26176dac292f26e7a4ae4?s=96&d=mm&r=g\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"New Script? Be Prepared! - Practical PowerShell","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/practicalpowershell.com\/new-script-be-prepared\/","og_locale":"en_US","og_type":"article","og_title":"New Script? Be Prepared! - Practical PowerShell","og_description":"While writing up some new scripts I ran into some issues which resulting in some troubleshooting, brainstorming and re-writing of the original code.\u00a0 This article was written to pass along some of the ‘lessons learned’: (A) Large Data Sets – Scripts that work in small environments will often fail or run very slowly in a […]","og_url":"https:\/\/practicalpowershell.com\/new-script-be-prepared\/","og_site_name":"Practical PowerShell","article_published_time":"2018-08-20T14:00:56+00:00","article_modified_time":"2020-03-14T22:10:41+00:00","og_image":[{"url":"https:\/\/practicalpowershell.com\/wp-content\/uploads\/2020\/03\/BePrepared.jpg","type":"","width":"","height":""}],"author":"damian","twitter_card":"summary_large_image","twitter_misc":{"Written by":"damian","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/practicalpowershell.com\/new-script-be-prepared\/#article","isPartOf":{"@id":"https:\/\/practicalpowershell.com\/new-script-be-prepared\/"},"author":{"name":"damian","@id":"https:\/\/practicalpowershell.com\/#\/schema\/person\/4d0733c81966e744aabbb49f56d64deb"},"headline":"New Script? Be Prepared!","datePublished":"2018-08-20T14:00:56+00:00","dateModified":"2020-03-14T22:10:41+00:00","mainEntityOfPage":{"@id":"https:\/\/practicalpowershell.com\/new-script-be-prepared\/"},"wordCount":733,"commentCount":0,"publisher":{"@id":"https:\/\/practicalpowershell.com\/#\/schema\/person\/4d0733c81966e744aabbb49f56d64deb"},"image":{"@id":"https:\/\/practicalpowershell.com\/new-script-be-prepared\/#primaryimage"},"thumbnailUrl":"https:\/\/practicalpowershell.com\/wp-content\/uploads\/2020\/03\/BePrepared.jpg","inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/practicalpowershell.com\/new-script-be-prepared\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/practicalpowershell.com\/new-script-be-prepared\/","url":"https:\/\/practicalpowershell.com\/new-script-be-prepared\/","name":"New Script? Be Prepared! - Practical PowerShell","isPartOf":{"@id":"https:\/\/practicalpowershell.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/practicalpowershell.com\/new-script-be-prepared\/#primaryimage"},"image":{"@id":"https:\/\/practicalpowershell.com\/new-script-be-prepared\/#primaryimage"},"thumbnailUrl":"https:\/\/practicalpowershell.com\/wp-content\/uploads\/2020\/03\/BePrepared.jpg","datePublished":"2018-08-20T14:00:56+00:00","dateModified":"2020-03-14T22:10:41+00:00","breadcrumb":{"@id":"https:\/\/practicalpowershell.com\/new-script-be-prepared\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/practicalpowershell.com\/new-script-be-prepared\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/practicalpowershell.com\/new-script-be-prepared\/#primaryimage","url":"https:\/\/i0.wp.com\/practicalpowershell.com\/wp-content\/uploads\/2020\/03\/BePrepared.jpg?fit=150%2C99&ssl=1","contentUrl":"https:\/\/i0.wp.com\/practicalpowershell.com\/wp-content\/uploads\/2020\/03\/BePrepared.jpg?fit=150%2C99&ssl=1","width":150,"height":99},{"@type":"BreadcrumbList","@id":"https:\/\/practicalpowershell.com\/new-script-be-prepared\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/practicalpowershell.com\/"},{"@type":"ListItem","position":2,"name":"New Script? Be Prepared!"}]},{"@type":"WebSite","@id":"https:\/\/practicalpowershell.com\/#website","url":"https:\/\/practicalpowershell.com\/","name":"Practical PowerShell","description":"PowerShell books written by experts","publisher":{"@id":"https:\/\/practicalpowershell.com\/#\/schema\/person\/4d0733c81966e744aabbb49f56d64deb"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/practicalpowershell.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/practicalpowershell.com\/#\/schema\/person\/4d0733c81966e744aabbb49f56d64deb","name":"damian","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/d5a8cc64a5aa27558a897b108e3be1a89859511a3fd26176dac292f26e7a4ae4?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/d5a8cc64a5aa27558a897b108e3be1a89859511a3fd26176dac292f26e7a4ae4?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/d5a8cc64a5aa27558a897b108e3be1a89859511a3fd26176dac292f26e7a4ae4?s=96&d=mm&r=g","caption":"damian"},"logo":{"@id":"https:\/\/secure.gravatar.com\/avatar\/d5a8cc64a5aa27558a897b108e3be1a89859511a3fd26176dac292f26e7a4ae4?s=96&d=mm&r=g"}}]}},"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/practicalpowershell.com\/wp-json\/wp\/v2\/posts\/560","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/practicalpowershell.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/practicalpowershell.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/practicalpowershell.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/practicalpowershell.com\/wp-json\/wp\/v2\/comments?post=560"}],"version-history":[{"count":1,"href":"https:\/\/practicalpowershell.com\/wp-json\/wp\/v2\/posts\/560\/revisions"}],"predecessor-version":[{"id":566,"href":"https:\/\/practicalpowershell.com\/wp-json\/wp\/v2\/posts\/560\/revisions\/566"}],"wp:attachment":[{"href":"https:\/\/practicalpowershell.com\/wp-json\/wp\/v2\/media?parent=560"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/practicalpowershell.com\/wp-json\/wp\/v2\/categories?post=560"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/practicalpowershell.com\/wp-json\/wp\/v2\/tags?post=560"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}