The pet store app is a interesting way to show how the basics of OctoPerf scripting works. As part of my learning path I chose to jump into a real app that I know pretty well to see what the other end of the spectrum looks like.
ORVIS has a mobile web app that was designed to work very well with the mobile and tablet platforms that are popular in 2019. It is reactive web app that relies very heavily on:
- JavaScript frameworks
- Real time client side rendering
- JSON data
- Micro services
- sub-domains for specific data purposes
- Client side storage of cached information
- Lazy Loading of images
Most of the logic in the web app is written into the client, not the server. The servers are there to answer specific requests and the client then processes the data from the servers into a cohesive experience for the user. For instance, the servers send back a large JSON response that includes all products in a category and it is up to the client to query that data for products that are available. If a product is not available the client side logic is there to decide if the product is shown on the screen or what message to give to the user about the product availability. The client will do things like removing a size option when the user picks a blue shirt that is not available in medium but is available in small, large, and extra large.
Other client-side logic exists to show or hide controls for special handling for certain products. Orvis sells high end fishing reels and when the customer goes to purchase one the client reads from the JSON data and decides to show additional choices in the UI for a right or left handed mount for the reel. This is yet another example of how the server just sends the data to the client and the client-side code has to adapt to the changing data.
For the order object to work properly the client code adds or removes data as the customer shops. As the customer adds items to their cart the JSON order object then grows in size and different values change like, total order amount, taxes, item count and so on. The client side is responsible for adding in properly formatted JSON to the object for:
- Shipping address
- Billing address
- Customer information: name, phone, etc
- Payment information
It is on the person writing the script to write the code needed to complete the order. The details that are missing from the order JSON object have to added in by the client side code in the script, which emulates what is happening in the web app. When we tested this web app on BlazeMeter we had to write some custom Groovy code in JMeter. When we switched over to NeoLoad we had to do the same logic in JavaScript. How does OctoPerf make this any easier or harder than the other tools?
As a side note, this kind of scripting would have been pretty easy for me to do in LoadRunner and given my experience on that platform it might have been faster to script with VuGen. VuGen supports writing scripts in their entirety in JavaScript. The branching logic and handling of JSON is all native in VuGen. I do understand that a lot of folks want to avoid ever writing any code for their scripts but that seems unrealistic given the platforms we have tested in the last 5 years.
OctoPerf support JSON extractors which is imperative for this type of testing. Orvis, like many others, replies with JSON responses that require a query that is beyond the ability of regex to extract the needed values.
Consider this flow through the web app: Mobile Home -> Select Category (Mens) -> Select Sub-Category (Long Sleeve Shirts) -> Scroll through products
When the user scrolls down on their device to look at more products the images are loaded in groups. The images to be loaded are stored in the JSON data not the HTML so telling the script to dynamically load resources does not work. In fact, we have not tackled this problem for most of our customers and relied on either loading the recorded images (static) or just removing the image loading from our scripts completely. We have to use caveats with our response times like, "These times represent only the round trip of the services with no page loading" or something like that.
[ ] Does OctoPerf have way to address this that differentiates them from NeoLoad, LoadRunner, BlazeMeter, etc?
The most important correlation in this flow is capturing the values needed to add a product to the cart. There are 6 values needed for the step that are correlated from the product selection. It is difficult in JMeter, and OctoPerf, to figure out the syntax needed for this kind of multi-value if-then-else logic. One way we have used in the past is to insert a JSR223 Groovy step and write the code that evaluates all 6 or 7 variable values then sets 1 single flag for the IF step to evaluate.
It gets even more in-depth because there also exists times when the variable being evaluated does not exist yet. This means the Groovy code needs to consider first, does the variable exist, and 2nd does the variable value equal "NOT_FOUND", and 3rd is the variables value NULL or empty? All of those conditions have caused errors when trying add an item to the cart.
The site being discussed for these examples uses a URL pattern that is strange but still works in the browser. This URL makes a call to the content service and gets taxonomy information.
/api/taxonomy/|category|dogs-in-the-house|2JE4|
Notice the use of the pipe symbol | to separate words or phrases. It appears that this works fine in browsers but fails in JMeter with an illegal character exception. The conversion process from HAR to JMX does not convert the pipe symbol to something safer like %7C (the hexadecimal representation of | ). Now the tester using OctoPerf or JMeter needs to know this is happening and somehow substitute the pipe symbol manually.
OctoPerf use timestamps to figure out where to put containers in the scripts. The guidance in the tutorials is to pause between steps so that the gaps show up in the capture and the containers get placed properly. In our real world example this logic does not work because there is a 3rd party framework that polls for updates and that places HTTP requests where there should be none. Once OctoPerf places the containers and requests within them, it is very difficult to figure out where they should be and correct them.
One way to deal with this is to used Charles or Fiddler to edit the HAR file and manually delete the 3rd party resources that throw off the OctoPerf logic. This approach assumes a certain level of knowledge about what requests to keep and which ones to delete. In this example the web app under test has over 60 unique domains that it makes requests to but we are only interested in those that affect how the web app functions, a list of more than 1 and less than 10 domains.
- What is the correct list of domains?
- How does one decide which ones to keep and which ones to remove?
- Is it better to edit the capture before importing to OctoPerf?