Slicing and dicing RESTler speccov.json

analyze_speccov.md

This gist will show how to extract information from the RESTler speccov.json to get a quick view of what operations succeeded and failed in a RESTler run and why.

The primary tool for this is jq.

Here I will use the results of a RESTler test I ran on the Azure DataBricks service.

The test summary reported at the end of the run was:

Starting task Test...
Using python: 'python3' (Python 3.10.9)
Request coverage (successful / total): 13 / 24
Attempted requests: 20 / 24
No bugs were found.

So now the question is ... which operations failed and why? This information is contained in the speccov.json file that RESTler writes to the test ouptut. You can find the file with:

f=$(find Test -name 'speccov.json')

Now you can get a compact view of the 24 operations and their results with:

jq -r '.[] | "\(.valid) \(.verb_endpoint)", if .valid == 0 then "- status_code=\(.status_code)\n- \(.error_message)" else empty end' $f

This command will show every operation, whether or not it was "valid" (got a success response), and for unsuccessful operations shows the status code and error message from the response. Here's what that looks like for my run on DataBricks:

1 PUT /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Databricks/workspaces/{workspaceName}
1 PUT /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Databricks/accessConnectors/{connectorName}
1 GET /providers/Microsoft.Databricks/operations
1 GET /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Databricks/accessConnectors
1 GET /subscriptions/{subscriptionId}/providers/Microsoft.Databricks/workspaces
1 GET /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Databricks/workspaces
1 GET /subscriptions/{subscriptionId}/providers/Microsoft.Databricks/accessConnectors
0 GET /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Databricks/workspaces/{workspaceName}/outboundNetworkDependenciesEndpoints
- status_code=400
- {"error":{"code":"NonVNetInjectedWorkspaceNotSupported","message":"The workspace 'workspacen76c7639836' is not VNet injected. Outbound network endpoints are only supported for VNET injected workspaces."}}
0 GET /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Databricks/workspaces/{workspaceName}/privateLinkResources/{groupId}
- status_code=400
- {"error":{"code":"InvalidWorkspaceState","message":"The workspace 'workspacen639bd23bb4' is in invalid state"}}
1 GET /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Databricks/accessConnectors/{connectorName}
1 DELETE /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Databricks/accessConnectors/{connectorName}
1 GET /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Databricks/workspaces/{workspaceName}
0 PUT /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Databricks/workspaces/{workspaceName}/privateEndpointConnections/{privateEndpointConnectionName}
- status_code=400
- {"error":{"code":"InvalidWorkspaceState","message":"The workspace 'workspacen01a3b89d9e' is in invalid state"}}
0 PUT /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Databricks/workspaces/{workspaceName}/virtualNetworkPeerings/{peeringName}
- status_code=403
- {"error":{"code":"LinkedAuthorizationFailed","message":"The client '[email protected]' with object id '1e0fb022-38b1-4992-8475-44b8181e0016' has permission to perform action 'Microsoft.Databricks/workspaces/virtualNetworkPeerings/write' on scope '/subscriptions/{subscriptionId}/resourceGroups/mkistler/providers/Microsoft.Databricks/workspaces/workspacence214a745d/virtualNetworkPeerings/peeringnam5c8a1968c0'; however, it does not have permission to perform action 'peer/action' on the linked scope(s) '/subscriptions/0140911e-1040-48da-8bc9-b99fb3dd88a6/resourceGroups/subramantest/providers/Microsoft.Network/virtualNetworks/subramanvnet' or the linked scope(s) are invalid."}}
0 GET /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Databricks/workspaces/{workspaceName}/privateEndpointConnections
- status_code=400
- {"error":{"code":"InvalidWorkspaceState","message":"The workspace 'workspacene4837e56c1' is in invalid state"}}
0 GET /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Databricks/workspaces/{workspaceName}/privateLinkResources
- status_code=400
- {"error":{"code":"InvalidWorkspaceState","message":"The workspace 'workspacend1a96dc49e' is in invalid state"}}
1 PATCH /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Databricks/workspaces/{workspaceName}
1 DELETE /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Databricks/workspaces/{workspaceName}
1 PATCH /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Databricks/accessConnectors/{connectorName}
0 GET /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Databricks/workspaces/{workspaceName}/virtualNetworkPeerings
- status_code=404
- {"error":{"code":"WorkspaceNotInitialized","message":"Operation is not supported until workspace (/subscriptions/{suscriptionId}/resourceGroups/mkistler/providers/Microsoft.Databricks/workspaces/workspacend96c4b7b7b) is initialized. Please click \"Launch Workspace\" in the Portal to initialize it."}}

0 DELETE /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Databricks/workspaces/{workspaceName}/privateEndpointConnections/{privateEndpointConnectionName}
- status_code=null
- null
0 GET /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Databricks/workspaces/{workspaceName}/privateEndpointConnections/{privateEndpointConnectionName}
- status_code=null
- null
0 DELETE /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Databricks/workspaces/{workspaceName}/virtualNetworkPeerings/{peeringName}
- status_code=null
- null
0 GET /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Databricks/workspaces/{workspaceName}/virtualNetworkPeerings/{peeringName}
- status_code=null
- null