scripts for secure workspace

add-cert-vpn-gateway.sh

echo "updating VPN gateway with client IP pool and root cert"
az network vnet-gateway update -g $RG \
    -n $GATEWAY \
    --address-prefixes $VPN_CLIENT_ADDR \
    -o none
# Upload the cert
az network vnet-gateway root-cert create -g $RG \
    --gateway-name $GATEWAY \
    -n 'rootcert' \
    --public-cert-data rootcert.cer \
    -o none
echo "Generating URL for client information download."
echo "Note: This URL expires after a while. You can run the command again to generate a new URL/client package."
az network vnet-gateway vpn-client generate -g $RG \
    -n $GATEWAY

create-vpn-gateway.sh

#-----VPN gateway
echo "Creating 'GatewaySubnet' and VPN gateway"
az network vnet subnet create \
    --vnet-name $VNET \
    -n 'GatewaySubnet' \
    -g $RG \
    --address-prefix $GATEWAY_ADDR \
    -o none
#    Request a public IP address for the gateway
az network public-ip create \
    -n "$GATEWAY-ip" \
    -g $RG \
    --allocation-method 'Dynamic' \
    -o none
# Create the gateway
az network vnet-gateway create -g $RG \
    -l $LOC \
    -n $GATEWAY \
    --public-ip-address "$GATEWAY-ip" \
    --vnet $VNET \
    --gateway-type 'Vpn' \
    --sku 'VpnGw1' \
    --vpn-type 'RouteBased' \
    --no-wait 

echo "It can take up to 45 minutes before the VPN gateway creation is done!"
echo "Use the following command to check the status of the gateway."
echo "It will return a value of 'Succeeded' when done:"
echo "az network vnet-gateway show -g $RG -n $GATEWAY --query 'provisioningState'"

get-private-ip.sh

# Get the private IPs, which need to be added to client DNS resolution.
# For example, to your hosts file
echo "# Private IPs and hosts for Azure Machine Learning"
az network private-endpoint show -g $RG \
    -n "$WS-pe" \
    --query 'customDnsConfigs[*].{IPAddress: ipAddresses[0], FQDN: fqdn}' \
    -o tsv
# Need this also for notebooks
az network private-dns record-set list -g $RG \
    --zone-name 'privatelink.notebooks.azure.net' \
    --query '[1].{IPAddress: aRecords[0].ipv4Address, FQDN: fqdn}' \
    -o tsv
# find IPs of other services.
echo "# Private IPs and hosts for Azure ML dependentzzs services"
# Storage
az network private-endpoint show -g $RG \
    -n "$STORAGE-blob-pe" \
    --query 'customDnsConfigs[*].{IPAddress: ipAddresses[0], FQDN: fqdn}' \
    -o tsv
az network private-endpoint show -g $RG \
    -n "$STORAGE-file-pe" \
    --query 'customDnsConfigs[*].{IPAddress: ipAddresses[0], FQDN: fqdn}' \
    -o tsv
# Key Vault
az network private-endpoint show -g $RG \
    -n "$KV-pe" \
    --query 'customDnsConfigs[*].{IPAddress: ipAddresses[0], FQDN: fqdn}' \
    -o tsv
# ACR
az network private-endpoint show -g $RG \
    -n "$ACR-pe" \
    --query 'customDnsConfigs[*].{IPAddress: ipAddresses[0], FQDN: fqdn}' \
    -o tsv

securews.sh

# BEFORE RUNNING THIS SCRIPT:
# 1. Install the Azure CLI - https://docs.microsoft.com/en-us/cli/azure/install-azure-cli
# 2. Install the Azure CLI extension for ML - https://docs.microsoft.com/en-us/azure/machine-learning/reference-azure-machine-learning-cli
# 3. Login to your Azure account - https://docs.microsoft.com/en-us/cli/azure/authenticate-azure-cli
# 4. Set the subscription you want to use - https://docs.microsoft.com/en-us/cli/azure/manage-azure-subscriptions-azure-cli
# 5. Make sure that the following resource providers are registered:
#    * Microsoft.MachineLearningServices
#    * Microsoft.ContainerRegistry
#    * Microsoft.Storage
#    * Microsoft.KeyVault
#    * Microsoft.Netework
#    For information on registering these, see https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/resource-providers-and-types

# Based on info from:
# https://docs.microsoft.com/en-us/azure/storage/common/storage-account-create?tabs=azure-cli
# https://docs.microsoft.com/en-us/azure/storage/files/storage-files-networking-endpoints?tabs=azure-cli
# https://docs.microsoft.com/en-us/azure/key-vault/general/private-link-service?tabs=cli
# https://docs.microsoft.com/en-us/azure/container-registry/container-registry-private-link
# https://docs.microsoft.com/en-us/azure/machine-learning/how-to-manage-workspace-cli
# https://docs.microsoft.com/en-us/azure/private-link/private-endpoint-dns

# Private link DNS zone names
# If you're in a sovereign cloud/region (Azure China 21Vianet for example,) change these as needed.
# See 
export FILE_ZONE='privatelink.file.core.windows.net'
export BLOB_ZONE='privatelink.blob.core.windows.net'
export ACR_ZONE='privatelink.azurecr.io'
export KV_ZONE='privatelink.vaultcore.azure.net'

# The name of the resource group that everything is created in
export RG='my'
# The Azure region to create everything in
# NOTE: Use the one-word region names (no spaces) as this value is used
#       as part of a unique name for ACR data zone record
export LOC='eastus'

# The name of the Azure Virtual Network that will be created
export VNET='myvnet'
# Address space for the virtual network
export VNET_ADDR='10.150.0.0/16'
# Subnets 
export TRAINING='training'
export SCORING='scoring'
# Training subnet address range
export TRAINING_ADDR='10.150.0.0/24'
# Scoring subnet address range
export SCORING_ADDR='10.150.1.0/24'
# Network security group name that will be created
export NSG='mynsg'
# VPN gateway name
export GATEWAY='myvpngateway'
# VPN subnet address range
export GATEWAY_ADDR='10.150.255.0/27'
# VPN client addresses
export VPN_CLIENT_ADDR='172.16.201.0/24'

# Key Vault name
export KV='mykeyvault'

# Azure Storage name
export STORAGE='mystore'

# Azure Container Registry name
export ACR='myacr'

# Azure Machine Learning
export WS='myworkspace'


#----Resource Group
echo "Creating resource group: $RG"
az group create -n $RG \
    -l $LOC \
    -o none

#----Virtual Network
echo "Creating virtual network: $VNET"
az network vnet create -g $RG \
    -n $VNET \
    --address-prefix $VNET_ADDR \
    --subnet-name $TRAINING \
    --subnet-prefix $TRAINING_ADDR \
    -o none
az network vnet subnet create -g $RG \
    --vnet-name $VNET \
    -n $SCORING \
    --address-prefixes $SCORING_ADDR \
    -o none

echo "Updating subnets to disable private endpoint policies"
az network vnet subnet update -g $RG \
    --vnet-name $VNET \
    -n $TRAINING \
    --service-endpoints Microsoft.Storage Microsoft.KeyVault Microsoft.ContainerRegistry \
    --disable-private-endpoint-network-policies \
    -o none
az network vnet subnet update -g $RG \
    --vnet-name $VNET \
    -n $SCORING \
    --service-endpoints Microsoft.Storage Microsoft.KeyVault Microsoft.ContainerRegistry \
    --disable-private-endpoint-network-policies \
    -o none

echo "Creating network security group: $NSG"
az network nsg create -n $NSG \
    -g $RG \
    -o none

echo "Setting NSG rules for compute instance/compute cluster"
az network nsg rule create -g $RG \
    --nsg-name $NSG \
    -n "AzureBatch" \
    --priority 1040 \
    --access 'Allow' \
    --direction 'Inbound' \
    --protocol 'Tcp' \
    --destination-address-prefixes '*' \
    --destination-port-ranges '29876-29877' \
    --source-address-prefixes 'BatchNodeManagement' \
    --source-port-ranges '*' \
    -o none

az network nsg rule create -g $RG \
    --nsg-name $NSG \
    -n "AzureMachineLearning" \
    --priority 1050 \
    --access 'Allow' \
    --direction 'Inbound' \
    --protocol 'Tcp' \
    --destination-address-prefixes '*' \
    --destination-port-ranges '44224' \
    --source-address-prefixes 'AzureMachineLearning' \
    --source-port-ranges '*' \
    -o none

echo "Updating subnets to use the NSG"
az network vnet subnet update -g $RG \
    --vnet-name $VNET \
    -n $TRAINING \
    --network-security-group $NSG \
    -o none
az network vnet subnet update -g $RG \
    --vnet-name $VNET \
    -n $SCORING \
    --network-security-group $NSG \
    -o none

#----Storage Account
echo "Creating storage account: $STORAGE"
STORAGE_ID=$(az storage account create -g $RG \
    -l $LOC \
    -n $STORAGE \
    --sku Standard_LRS \
    --kind StorageV2 \
    --query 'id' \
    -o tsv)
echo "Storage account Azure Resource Manager ID: $STORAGE_ID"
# Create DNS zone and link for file storage
echo "Enabling private endpoint for file storage"
# Create the zone
az network private-dns zone create -g $RG \
    --name $FILE_ZONE \
    -o none
# Create the link
az network private-dns link vnet create -g $RG \
    --zone-name $FILE_ZONE \
    --name "$STORAGE-f-lnk" \
    --virtual-network $VNET \
    --registration-enabled false \
    -o none
#     Create file storage private endpoint in the training subnet
FILE_IP=$(az network private-endpoint create -g $RG \
    --name "$STORAGE-file-pe" \
    --vnet-name $VNET \
    --subnet $TRAINING \
    --private-connection-resource-id $STORAGE_ID \
    --group-id 'file' \
    --connection-name "$STORAGE-file-conn" \
    -l $LOC \
    --query 'customDnsConfigs[].ipAddresses[]' \
    -o tsv)
#   Create/set the DNS record
az network private-dns record-set a add-record \
    -g $RG \
    -z $FILE_ZONE \
    -n $STORAGE \
    -a $FILE_IP \
    -o none

# Create DNS zone and link for blob storage
echo "Enabling private endpoint for blob storage"
# Create the zone
az network private-dns zone create -g $RG \
    --name $BLOB_ZONE \
    -o none
# Create the link
az network private-dns link vnet create -g $RG \
    --zone-name $BLOB_ZONE \
    --name "$STORAGE-b-lnk" \
    --virtual-network $VNET \
    --registration-enabled false \
    -o none
#     Create blob storage private endpoint in the training subnet
BLOB_IP=$(az network private-endpoint create -g $RG \
    --name "$STORAGE-blob-pe" \
    --vnet-name $VNET \
    --subnet $TRAINING \
    --private-connection-resource-id $STORAGE_ID \
    --group-id 'blob' \
    --connection-name "$STORAGE-blob-conn" \
    -l $LOC\
    --query 'customDnsConfigs[].ipAddresses[]' \
    -o tsv)
#   Create/set the DNS record
az network private-dns record-set a add-record \
    -g $RG \
    -z $BLOB_ZONE \
    -n $STORAGE \
    -a $BLOB_IP \
    -o none

echo "Disabling public endpoint access for storage"
az storage account update -g $RG \
    -n $STORAGE \
    --bypass 'AzureServices' \
    --default-action 'Deny' \
    -o none

#----Key Vault
echo "Creating key vault"
KV_ID=$(az keyvault create -g $RG \
    -l $LOC \
    -n $KV \
    --query 'id' \
    -o tsv)
echo "Key vault Azure Resource Manager ID: $KV_ID"

# Create DNS zone and link for vault
echo "Enabling private endpoint for key vault"
# Create the zone
az network private-dns zone create -g $RG \
    --name $KV_ZONE \
    -o none
# Create the link
az network private-dns link vnet create -g $RG \
    --zone-name $KV_ZONE \
    --name "$KV-lnk" \
    --virtual-network $VNET \
    --registration-enabled false \
    -o none
#     Create the private endpoint
KV_IP=$(az network private-endpoint create -g $RG \
    --name "$KV-pe" \
    --vnet-name $VNET \
    --subnet $TRAINING \
    --private-connection-resource-id $KV_ID \
    --group-id vault \
    --connection-name "$KV-conn" \
    -l $LOC \
    --query 'customDnsConfigs[].ipAddresses[]' \
    -o tsv)
#     Create/set the DNS record
az network private-dns record-set a add-record \
    -g $RG \
    -z $KV_ZONE \
    -n $KV \
    -a $KV_IP \
    -o none

#----Container Registry
echo "Creating container registry"
ACR_ID=$(az acr create -g $RG \
    -l $LOC \
    -n $ACR \
    --sku 'Premium' \
    --query 'id' \
    -o tsv)
echo "container registry Azure Resource Manager ID: $ACR_ID"
# Create DNS zone and link for ACR
echo "Enabling private endpoint for container registry"
# Create the zone
az network private-dns zone create -g $RG \
    --name $ACR_ZONE \
    -o none
# Create the link
az network private-dns link vnet create -g $RG \
    --zone-name $ACR_ZONE \
    --name "$ACR-lnk" \
    --virtual-network $VNET \
    --registration-enabled false \
    -o none
#     Create ACR private endpoint
#     NOTE: There's a different way to get the IPs for ACR. See 5b below.
az network private-endpoint create -g $RG \
    --name "$ACR-pe" \
    --vnet-name $VNET \
    --subnet $TRAINING \
    --private-connection-resource-id $ACR_ID \
    --group-id registry \
    --connection-name "$ACR-conn" \
    -l $LOC \
    -o none
# 5b. Get private IP addresses
#     Get the network interface ID
ACR_NET_IID=$(az network private-endpoint show -g $RG \
    --name "$ACR-pe" \
    --query 'networkInterfaces[0].id' \
    -o tsv)
#     Get the private IP for the registry
ACR_IP=$(az resource show \
  --ids $ACR_NET_IID \
  --api-version '2019-04-01' \
  --query 'properties.ipConfigurations[1].properties.privateIPAddress' \
  -o tsv)
#     Get the private IP for the data endpoint
ACR_DATA_IP=$(az resource show \
  --ids $ACR_NET_IID \
  --api-version '2019-04-01' \
  --query 'properties.ipConfigurations[0].properties.privateIPAddress' \
  -o tsv)
# 5c. Create/set the DNS records
az network private-dns record-set a add-record -g $RG \
    -z $ACR_ZONE \
    -n $ACR \
    -a $ACR_IP \
    -o none
#     The name parameter here (-n) is what the ACR docs recommend; name.region.data
az network private-dns record-set a add-record -g $RG \
    -z $ACR_ZONE \
    -n "$ACR.$LOC.data" \
    -a $ACR_DATA_IP \
    -o none

#-----Azure Machine Learning workspace
echo "Creating Azure Machine Learning workspace: $WS"
echo "Private endpoint is enabled during creation"
az ml workspace create -g $RG \
    -l $LOC \
    -w $WS \
    --storage-account $STORAGE_ID \
    --container-registry $ACR_ID \
    --keyvault $KV_ID \
    --pe-name "$WS-pe" \
    --pe-vnet-name $VNET \
    --pe-subnet-name $TRAINING \
    --pe-auto-approval \
    -o none


# ------- compute cluster
echo "Creating Azure ML compute cluster to use for building Docker images"
echo "Required only when ACR for workspace is behind a VNet"
az ml computetarget create amlcompute -g $RG \
    --vnet-resourcegroup-name $RG \
    --max-nodes 1 \
    -n 'imagebuild' \
    -s 'Standard_NC6' \
    --min-nodes 0 \
    -w $WS \
    --vnet-name $VNET \
    --subnet-name $TRAINING \
    -o none

# Update workspace to use the compute instance for building Docker images
az ml workspace update -g $RG \
    -w $WS \
    --image-build-compute 'imagebuild' \
    -o none

Looks great. Few comments:

• Remove "en-us" from Docs urls in securews
• The Bash shell piece mostly applies to the env variable setting and console output. The Az commands use Az CLI & AML Extension. Does it
  make sense to have a PowerShell equivalent for securews?
• You define the gateway name and address env variables in securews but don't use them there.
  Would it make sense to instead move them to create-vp-gateway
  • Same goes for VPN client address. It's defined in securews but not used until add-cert-vpn-gateway
• Would it make sense to have a separate file for setting environment variables? The reason for this is, let's say someone closes the terminal. Then the env variables won't be persisted in which case they'll have to run securews every time. If env variable setting is in a separate file, they can run that script without having to rerun the resource creation scripts.

The chunking of steps makes sense to me.

• Securews is a long script, it would be helpful to add a comment block at the beginning describing high-level steps. Similar to what you sent via e-mail; "creates a vnet, storage, container registry, key vault, workspace, and private endpoints for all those. Also creates a compute cluster for use building docker images; for some vague technical reason, once ACR is behind a vnet we can no longer build images directly on it and have to use a compute instead."
• Securews Line 25: Is "See" all you want in that line? Do you mean to reference a section or doc here?
• Securews vnet section Line 77-147: Readability is better with the extra line breaks between each command, but they don't exist in the other sections. I think it would be good to be consistent, either add breaks to other sections or remove them from the VNET section.
• Securews Line 304: Where does the "5b" numbering come from? Is there supposed to be an outline structure in these scripts?