• Language

• Vision

• Todo List

• Install

• Shell script coding tricks

• Usage

• az-setup-cli.sh

• References

• Maintainers

• Contributing

• License

• Ethics

All code, comments, and documentation here are written in U.S. English. So we can share our learnings with developers around the world, translations are thanks to https://github.com/dephraiim/translate-readme

• English
• 简体中文
• 繁体中文
• हिंदी
• Française
• عربى
  
  

Most scripts in this repo are Bash shell scripts that run natively on MacOS and Linux.

Script code can run on Windows Git Bash Shell. PowerShell scripts are used in cases where that's the only solution.

To handle more complexity, programs coded in Python or Go language may be called when appropriate to take advantage of their capabilities.

There is a risk that CLI functions may not work when its underlying Azure REST API does work. So Postman files are included when appropriate.

Using this repo can save you time and money and give you some peace of mind and happiness.

People leave resources running because they don't want to spend their life repeating the manual toil of clicking though the Azure Portal GUI, as described by most Azure tutorials.

This repo (at https://github.com/wilsonmar/azure-quickly) contains automation scripts that enable you to confidently delete Resource Groups when you're resting or playing because it enables you to get resources back with just a few commands, even with a brand-new Subscription.

You save money because you don't have to leave resources running, consuming credits or running up your credit card bill.

It's stressful to accurately repeat the manual mousing and typing through the Azure Portal (which is not quickly repeatable).

So, this README explains how you can initiate your custom environment from scratch using the CLI Bash terminal.

• Add alias.sh
• Test to ensure script code can run on Windowsw Git Bash Shell.
• A variables vetting script to validate the memory variables before invocation.

The following are one-time activities for each Azure subscription, covered in my deep yet concise tutorial at:

https://wilsonmar.github.io/azure-onboarding

It covers the creation of free Azure accounts and Azure Storage accounts to hold files in a clouddrive:

1. Get an Azure subscription (such as by buying a Visual Studio license).

2. Use an internet browser (Google Chrome) to address https://portal.azure.com and login.

3. Be in https://shell.azure.com or click the CLI icon within the portal.

4. Create a storage account for a clouddrive within the shell.

   Setup an environment for a new Subscription

5. Triple-click the command below to highlight it:

   bash -c "$(curl -fsSL https://raw.githubusercontent.com/wilsonmar/azure-quickly/master/az-setup-cli.sh)" -v -i

6. Right-click the highlighted and select "Copy" to save the highlighted temporarily in your Clipboard.

7. Get in https://shell.azure.com or click the CLI icon after entering https://portal.azure.com

8. Right-click anywhere on the CLI Bash terminal window, then press command+V to paste from the Clipboard.

9. Press Enter to run the script. It takes several minutes to run.

   Steps executed by the script "az-setup-cli.sh" are described below.

   When done, you should see the folder containing scripts and the prompt at the left, where it will stay after each command (instead of at the end of the folder path):

   ~/clouddrive/azure-quickly
   $ _

   Customize setmem.sh values

   The init script also establishes from a sample the setmem script which defines environment variable values in memory to control script operation. Note that the setmen file is in a higher folder than where script code may be pushed to github. That's because

10. Use the built-in Visual Studio Code editor to edit file setmem.sh:

    code ../setmem.sh
    

11. Switch to the Portal GUI.

12. Open the Subscription blade. Click on your current Subscription.

13. Click the copy icon next to the Subscription code (so it gets saved to your Clipboard).

14. Switch to the Code editor window. Highlight the existing text in the variable MY_SUBSCRIPTION_ID and press Paste (Command+V on Macs or Ctrl+V on Windows).

    There may be a different file for each environment (such as dev vs. prod).

15. Click the "..." menu to the right of the Code Editor window to save, then exit.

    Now you can run scripts to create and manage resources. Most of the scripts reference a tutorial at Microsoft Learn, CloudAcademy, Pluralsight, Coursera, etc.

    To run in the Azure Machine Learning studio, you need one or more instances an Azure ML Workspace, Compute, Insights, and Key Vault. To create them:

    Create Workspace using az ml cli v2

    The ML CLI v2 (Preview) provides one automation approach:
    github.com/Azure/azureml-examples/blob/main/cli/setup.sh.

16. Run the script I've modified from that to use environment variables and additional edits:

    ./az-mlcli2.sh
    

    The script performs a git clone into the environment.

    Configure aliases to your liking

17. Edit the aliases.sh file and remove or add keyboard macros.

Now that you have the Resources needed:

Contents within Bash scripts within this repo are written using coding conventions defined at https://wilsonmar.github.io/bash-coding which include:

• source ./az-all-start.sh sets up environment variables and utility functions.

• set -o errexit makes it so that the script stops on the first error (instead of running on).

• A new Resource Group and all resources are created new every run to reduce the complexity of coding for idempotency (where the status is the same at the end of every re-run).

• --resource-group is a required argument on many commands. It's last so that missing slash line a line above it would cause the command to fail.

• A backslash \ character at the end of a line within the same az shell command continues that command.

• Variable (specification) data controlling Python programs are passed to Python programs by saving them as variables in an .env file in the same folder as the Python program.

Jupyter's Gallery of Python Notebooks:

• https://github.com/jupyter/jupyter/wiki/A-gallery-of-interesting-Jupyter-Notebooks

Try each of the activities below for each session (assuming you performed the install above):

According to How to run Jupyter:

1. Go to the Azure Machine Learning Studio

2. Click the instance created.

3. Click the "https://ml.azure.com/?tid=..." URL under the "Studio web URL".

4. If the "get started" pop-up dialog appears, click X to dismiss it.

5. Click "Notebooks".

   Clone Git repositories into your workspace file system

   *

6. Mouse over your user name (among Users) and click the "..." to choose "Upload files".

   

7. Navigate to the folders as https://github.com/Azure/azureml-examples/tree/main/cli/jobs/train:

   • fastai
   • lightbm
   • pytorch
   • r
   • tensorflow
     
     

   MNIST

az ml job create -f jobs/train/lightgbm/iris/job.yml --set compute.target=local --web --stream

1. Navigate to https://github.com/Azure/azureml-examples/blob/main/cli/jobs/train/fastai/mnist/

2. Select "job.yml".

3. Click "Upload".

4. Select the compute where to run the Notebook.

   A stopped compute instance will automatically start when the first cell is run.

5. Click the "Run" icon.

   Alternately

6. In the User files section of your workspace. Click on the cell you wish to edit. If you don't have any notebooks in this section, see Create and manage files in your workspace.

More scripts from my azure-quickly repo:

• Run a Bing Search using API:

  ./az-bing-cli.sh
   

• Create an Azure Key Vault for use by scripts to follow:

  ./az-keyvault-cli.sh
   

  Optionally, put a secret in it; show secret; delete secret; recover secret; create a vm; Managed Service Identity; update permissions; Custom Script Extension; Apply the Custom Script Extension:

• Create a Machine Learning Workspace to run iPython Notebooks using JupyterLab:

  export MY_MLWORKSPACE_NAME="mela"
   ./az-mlworkspace-cli.sh
   

• Use Azure Cognitive Services:

  export MY_COG_ACCT="cogme"
   export MY_COG_PRICING_TIER="F0"  # or S0
   ./az-cog-cli.sh
   

• Use Helm charts

  ./az-helm-cli.sh
   

• Create a VM with a public IP address:

  ./az-vm-cli.sh
   

• Create an App Service Plan, Azure Web App, Deployment, to show MY_APPNAME.

  ./az-webapp-cli.sh
   

• Create a network with two subnets and a network security group that secures inbound traffic. One subnet is for remote access traffic, one is web traffic for VMs that run a web server. Two VMs are then created. One allows SSH access and has the appropriate network security group rules applied. You use this VM as an SSH jumpbox to then connect to the second VM which can be used as a web server:

  ./az-vm-jumpbox-cli.sh
   

• Create a VM with a public IP address. Enabled are a storage account, boot diagnostics with the VM diagnostics extension applied:

  ./az-vm-diag-cli.sh
   

• Create a VM; Recovery Services vault, a backup policy, then creates a VM and applies the backup policy before starting the initial backup job.

  ./az-vm-backup-cli.sh
   

• Create a Docker container from a Dockerfile; Create AKS; Scale up replicas

  ./az-aks-cli.sh
   

  The IP shows the "Month of Pizza Lunches in a container" website (load balanced).

• Create IoT for WebApp:

  export MY_PROJECT_FOLDER="iot-project"
  export MY_IOT_HUB_NAME="hubahuba"
  export MY_IOT_HUB_GROUP="hubgroupie"
  ./az-iot-cli.sh
   

• Create Azure Functions:

  ./az-functions-temp.sh
   

  Several Functions components are not available in the Azure CLI, so manual actions are needed on Azure portal to fill in the gaps. See the "Month of Lunches" ebook.

https://github.com/Azure/azure-quickstart-templates

The script should do all the steps below:

1. OPTIONAL: Edit the .bashrc file to customize the prompt:

   If you're in the cloud Shell (which runs the Linux operating system), add these lines to the bottom of the .bashrc:

   export PS1="\n  \w\[\033[33m\]\n$ "
   #

   The PS1 sets the prompt so it appears in the same spot on the screen every line, under the current folder and file path (rather than to the right of it at various points on the screen).

   "#" at the last line of the file is a hack to make a comment out of the PS1 the system adds on its own.

2. Navigate into a folder which holds repository to be downloaded:

   Within Cloud Shell, it's
   cd cloudshell

   Alternately, on my laptop, I use
   cd gmail_acct

3. Remove the previous repo folder:

   PROTIP: A time proxy command is added in front of commands to identify how many time was taken to run the command each time. For example, "0m4.559s" means about 4.6 seconds.

4. Download this repo to establish a run environment:

   git clone https://github.com/wilsonmar/azure-quickly.git --depth 1 
   cd azure-quickly
   ls
   

   --depth 1 specifies download of only the latest version, to save space used.

   ls lists folders and files to confirm the download actually occurred.

5. Give all the shell file permissions to run:

   chmod +x *.sh
   

6. Run script to setup Azure Providers:

   source az-providers-setup.sh
   

   The response is a list of providers added.

   >>> Microsoft.AlertsManagement already Registered.
   

Microsoft.BotService already Registered. Microsoft.ChangeAnalysis already Registered. Microsoft.CognitiveServices already Registered. Microsoft.Compute already Registered. Microsoft.ContainerInstance already Registered. Microsoft.ContainerRegistry already Registered. Microsoft.Devices already Registered. Microsoft.Insights already Registered. Microsoft.KeyVault already Registered. Microsoft.Notebooks already Registered. Microsoft.MachineLearningServices already Registered. Microsoft.ManagedIdentity already Registered. Microsoft.Search already Registered. Microsoft.Storage already Registered. Microsoft.Web already Registered.

The above only needs to be done once, but running it again won't be harmful.

7. Give setmem.sh permissions to run and run it :

   chmod +x ../setmem.sh
   source ../setmem.sh
   

8. Move (copy and rename) "sample-setmem.sh" to file "setmem.sh"

   mv setmem-sample.sh ../setmem.sh
   

   PROTIP: We move the file where it will never be uploaded to any repository (GitHub, GitLab, etc.).

   Manually customize values in setmem.sh

9. Open the file for edit using program "code" (Visual Studio Code):

   code ../setmem.sh
   

   .. is used because the file, containing secrets, is in a folder which should never be pushed to GitHub.

10. Use a text editor program to edit the ../setmem.sh file:

    Scripts have been generalized by environment variables substituting for hard-coded values in scripts. PROTIP: Using variable instead of hard-coding avoids typos and misconfigurations.

    Lines below define values for each variable so that multiple runs can use different values, without need to change the script file.

11. In portal.azure.com Subscription blade, select the Subscription you wnat to use, then click the icon to Copy to Clipboard.

    

    In the file, highlight the ID and paste it:

    export MY_SUBSCRIPTION_NAME="Azure Pass - Sponsorship"
    export MY_SUBSCRIPTION_ID="11cb040d-4e32-4524-bc8e-0bee213dddae"  # for gmail 
    

12. In portal.azure.com Tenant blade, select the Tenant you want to use, then click the icon to Copy to Clipboard.

    

    In the file, highlight the ID and paste it:

    export TENANT_ID="22223348-f7f0-4cc2-addc-11021d882720"           # for gmail, in Portal
    

    TODO: Substitute export statements of secrets with calls to retrieve them from a long-running Azure KeyVault. But no one else would be at this file unless they are properly logged into Azure under your account.

13. Edit the MY_LOC (Location = Region) and other defaults.

14. At the bottom of the file, add a statement that prints out one of the variables, so you know the export statements took:

    In a Bash script:

    echo "MY_RG=$MY_RG"

15. Save the file. In Cloud Shell, press command+Q or click the "..." to press Save, then Close.

    Each work session

16. At the beginning of each session invoke the script in the folder just above your custom scripts repo:

    source ../setmem.sh
    

    NOTE: Using "source" to run the script so that environment variables defined in the script will be visible after the script is done, and be inherited by any programs you launch from it. That's because the source runs the script in the current shell. But note that any exit statement would stop the run.

    Alternately,

    source <(curl -s -L https://example.com/install.sh)

    After execution, you can still override variable values before running another script.

    That is how you can run scripts for several regions/locations - by changing just the MY_LOC environment variable's value and running the script again.

17. PROTIP: Delete resource groups to stop charges from accumulating on Virtual Servers:

    time az group delete --name "${MY_RG}" --yes   # takes several minutes
    

    --yes before the az command feeds a "y" to automatically answer the request:
    Are you sure you want to perform this operation? (y/n): y

Scripts here are adapted from various experts generous with sharing their code:

• https://github.com/fouldsy/azure-mol-samples-2nd-ed by Iain Foulds, explained in https://aka.ms/monthoflunches published 4/30/2020.

• https://github.com/MicrosoftLearning/AZ-303-Microsoft-Azure-Architect-Technologies

• https://github.com/MicrosoftLearning/AZ500-AzureSecurityTechnologies

• https://github.com/Azure/azure-cli by Microsoft

• https://github.com/timothywarner/az400 & az303 by Tim Warner

• https://github.com/zaalion/oreilly-azure-app-security by Reza Salehi

• https://github.com/Azure/azure-quickstart-templates (ARM Templates)

• https://github.com/johnthebrit/AzureMasterClass PowerShell scripts

• https://github.com/terraform-providers/terraform-provider-azurerm

• Skylines Academy

• Gruntwork (Terraform)

• CloudPosse (Terraform for AWS)
  
  

@wilsonmar

PRs accepted.

If editing this README, please conform to the standard-readme specification.

MIT © 2021 Wilson Mar

This project operates under the W3C's Code of Ethics and Professional Conduct:

W3C is a growing and global community where participants choose to work together and in that process experience differences in language, location, nationality, and experience. In such a diverse environment, misunderstandings and disagreements happen, which in most cases can be resolved informally. In rare cases, however, behavior can intimidate, harass, or otherwise disrupt one or more people in the community, which W3C will not tolerate.

A Code of Ethics and Professional Conduct is useful to define accepted and acceptable behaviors and to promote high standards of professional practice. It also provides a benchmark for self-evaluation and acts as a vehicle for better identity of the organization.

We hope that our community group act according to these guidelines, and that participants hold each other to these high standards. If you have any questions or are worried that the code isn't being followed, please contact the owner of the repository.