Abstract. In the context of software development, with agile methods, having continuous and fast feedback is crucial for project success. One way to ensure quick bug detection is through automated tests. Given the high value of automated testing, it is necessary to evaluate the various frameworks, currently available in the market, to determine which one better suits the project’s needs. In this work, it’s proposed a comparative analysis between three testing frameworks: Selenium, one of the most traditional automation tools; Cypress, a tool that has gained popularity in recent years; and Playwright, a newer tool that has attracted a lot of attention. To perform this analysis, automated test suites were developed for a web application, with the aim of comparing the generated scripts based on selected criteria. Thus, the pros and cons of working with each of these frameworks will be presented.
Software testing, among other objectives, seeks to detect existing failures and defects in order to reduce software quality risk levels [ISTQB-CTFL]. Myers' Rule of Ten states that the cost of correcting defects tends to increase the later the defect is detected. Defects found during production tend to cost much more than defects found in data models and other software project documents. Therefore, by treating testing as an organized process that is often parallel and integrated with the development process, maintenance costs will be reduced. [Bastos 2007]
Nowadays, many software options are being released constantly, so the cost of replacing software is no longer as high as it used to be. For this reason, the adoption of agile development methods is becoming increasingly common, to meet customer needs with constant improvements and quick fixes. As a result, the deadline for launching an application is becoming increasingly shorter and, consequently, the time available for testing is becoming more limited.
Due to the shorter time required to develop and launch an application, automated testing becomes necessary to find defects and ensure quality during development cycles. Test automation is the use of software to control the execution of software tests through the application of strategies and tools, comparing expected results with actual results. The objectives are to reduce human involvement in manual activities, the time required, and final costs. [Oliveira 2018]
There are currently many options for test automation frameworks, many of them open source, but even selecting an open source framework for adoption in a company is not without cost. After all, one must take into account the learning curve of a framework, which will affect the time required to train an employee to effectively use it. In addition, it is necessary to check whether the framework meets all the needs of use, because otherwise it may need modifications through customizations or by installing third-party plugins. These factors make choosing a framework a complex task, requiring a good analysis.
One of the most traditional and widely known frameworks is Selenium, a comprehensive project for a variety of tools and libraries that enable and support web browser automation [Selenium 2023a]. Although its first version was created in 2004, the framework is currently in version 4.0 and has great community support, as can be seen in Table 1 [Selenium 2024a].
| Framework | Supported Languages | Copies on GitHub | Stars on GitHub |
|---|---|---|---|
| Selenium | Java, Javascript, Python, Ruby, C# | 7.9K | 29K |
Table 1. Data about the Selenium Framework
Cypress is a new front-end testing tool built for modern web applications. It is worth noting that it addresses the main pain points faced by developers and QA engineers when testing modern applications [Cypress 2024a]. Despite having been created in 2017, Cypress currently has a high number of active users, as shown in Table 2. [Cypress 2024b]
| Framework | Supported Languages | Copies on GitHub | Stars on GitHub |
|---|---|---|---|
| Cypress | Javascript | 3.1K | 45.9K |
Table 2. Data about the Cypress Framework
Playwright was created specifically to meet the needs of end-to-end testing. Capable of supporting all modern rendering engines, including Chromium, WebKit, and Firefox [Playwright 2024a], it was conceived in 2020 by Microsoft and has been attracting a lot of attention from the community, as can be seen in Table 3 [Playwright 2024b].
| Framework | Supported Languages | Copies on GitHub | Stars on GitHub |
|---|---|---|---|
| Playwright | Java, Javascript, Python, C# | 3.2K | 60.1K |
Table 3. Data about the Playwright Framework
Even though it is newer, Playwright has already surpassed Selenium in number of downloads and despite having reduced the distance from Cypress over the past year, it is still far from the number of downloads of Cypress, a situation indicated in Figure 1.
Figure 1. Number of downloads of each framework in the last 5 years. [NPM Trends 2024]
Therefore, in view of the above, this work aims to identify the main functionalities and limitations of the selected frameworks, perform the environment configurations, develop the architecture of the automation project, in order to follow the good practices of software engineering, develop the same test scripts in each framewrok, execute and validate them to later compare the results obtained.
Selenium is a collection of different automation tools that enable browser-based application testing. Released in 2004, it was developed by Thoughtworks and comprises key components such as Selenium Grid, Selenium IDE, Selenium RC, and Selenium WebDriver. It supports all major browsers in the market, including Chrome, Firefox, Safari, Internet Explorer, Edge, Opera, among others. Its scripts are compatible with JavaScript, Java, Ruby, C#, and Python. [Akhtar 2023]
Since it was initially developed in an era when browsers were much simpler compared to today, a common complaint is that Selenium tends to be slower when compared to more recent frameworks [Raggo 2023]. Additionally, it is often less stable due to flaky tests, which arise from the need to include implicit or explicit waits in the script. [Pathak 2023]
Cypress, unlike most automation tools, was not built on Selenium and, for this reason, is not limited by its capabilities. It focuses on end-to-end testing, with test scripts written exclusively in JavaScript. Cypress is a comprehensive testing suite with its own runner and reporting capabilities. Its supports browsers like Firefox and all Chromium-based browsers, such as Chrome, Edge, and Electron. The tool includes built-in features for video recording and screenshoot capture during test execution [Döring and Brueckner 2022].
Cypress performs best in local testing but may not be the fastest option for running tests on live websites. Its high startup time can interfere in high-frequency monitoring scenarios, although this likely makes no real difference in the context of traditional E2E test builds [Raggo 2024].
Playwright, like Cypress, was not built on Selenium and is an entirely independent solution. It supports all major browsers and multiple programming languages, such as Java, JavaScript, and Python. Furthermore, it provides built-in options for screenshots, videos, and execution tracing. Playwright includes its own test runner but can be integrated with other runners of the user's choice. It also allows tests to run across multiple tabs, origins, and users within a single test [Döring and Brueckner 2022].
Playwright supports API testing but does not allow disabling redirects. It also emulates mobile devices by using desktop browsers instead of real devices. [Akhtar 2023]
The studies Puppeteer vs Selenium vs Playwright, a speed comparison [Raggo 2023], Cypress vs Selenium vs Playwright vs Puppeteer speed comparison [Raggo 2024], Why we favor Playwright over Selenium or Cypress [Döring and Brueckner 2022], Cypress vs Selenium vs Playwright vs Puppeteer: Core Differences [Akhtar 2023], Playwright vs Selenium vs Cypress: A Detailed Comparison [Pathak 2023], and Automação de testes funcionais para aplicações web: comparativo dos frameworks Cypress e Playwright [França 2023] conduct studies similar to those presented here regarding the comparison of Selenium, Cypress, and Playwright.
The metrics used in the cited studies are similar to those employed in this research. These include comparison of script coding and execution time; analysis of support for different programming languages and browsers; ease of initial project setup; need for external libraries to capture screenshots and videos; the architecture on which the tool was built; the ability to integrate with CI/CD tools; and, finally, support for API and mobile testing.
The experiment conducted in this study will involve automating the same test suite for the Sauce Demo website across the different frameworks, using JavaScript and the Page Objects design pattern, to ensure that the use of a different language does not influence test execution times. However, it should be noted that the research inspiring this study also evaluates Puppeteer1, which was excluded here due to its strong similarity to Playwright.
The Sauce Demo website (http://www.saucedemo.com), designed for test automation practice, was chosen as a web application example because it allows the creation of E2E test scenarios.
In this study, an experiment was conduct in which the Selenium, Cypress, and Playwright frameworks were examined, and an automated test suite was implemented using each of them. The purpose of developing these suites is to identify the main characteristics of each framework during the test script development process, enabling the identification of their pros and cons.
To ensure an effective comparison, specific metrics were defined for analysis to evaluate and identify the differences among the frameworks, highlighting their strengths and weaknesses. The metrics used in the analysis were:
- Ease of initial project setup and need for external library installations;
- Framework architecture;
- Time spent coding the automation project;
- Test script execution time.
The Sauce Demo website (http://www.saucedemo.com/), used to perform the automation comparison, is a simple web application that allows the execution of an E2E flow simulating product purchases on an e-commerce site. A complete suite comprising 23 test cases covering all site functionalities was created to compare performance using a comprehensive suite. This approach was chosen as a single test case might overlook performance differences that could emerge in larger test volumes. The test scripts created for the analysis are listed in Table 4.
| Feature | Test Case | Test Objective |
|---|---|---|
| Login | TC01 - Log in with a standard user | Verify that logging in with a standard user is successful. |
| Login | TC02 - Log in with a blocked user | Verify that an error message prevents the login of a blocked user. |
| Login | TC03 - Log in with a problematic user | Verify that logging in with a problematic user displays a site version where all product images are incorrect. |
| Login | TC04 - Log in with performance issues | Verify that logging in with a user with performance issues is slow. |
| Login | TC05 - Log in with layout errors | Verify that logging in with a user with layout issues displays a site version where some product images are of incorrect sizes. |
| Products | TC06 - Add a product to the cart and verify it was saved correctly | Validate that a product can be added to the cart via the product list page. |
| Products | TC07 - Remove a product from the cart via the product page | Validate that a product can be removed from the cart via the product list page. |
| Products | TC08 - Add a product to the cart via the product details page and verify it was saved correctly | Validate that a product can be added to the cart via the product details page. |
| Products | TC09 - Remove a product from the cart via the product details page and return to the product list page | Validate that a product can be removed from the cart via the product details page and then return to the product list page. |
| Products | TC10 - Validate adding and removing all products to/from the cart | Validate that all add and remove buttons on the product list page work correctly. |
| Products | TC11 - Validate default sorting in ascending alphabetical order | Validate that the default sorting in ascending alphabetical order is displayed on the product list page. |
| Products | TC12 - Sort products in descending alphabetical order | Validate that changing the sorting to descending alphabetical order displays the products correctly. |
| Products | TC13 - Sort products by price (low to high) | Validate that changing the sorting to price (low to high) displays the products correctly. |
| Products | TC14 - Sort products by price (high to low) | Validate that changing the sorting to price (high to low) displays the products correctly. |
| Your Cart | TC15 - Validate the "Continue Shopping" button | Validate that the "Continue Shopping" button redirects back to the product list page. |
| Your Cart | TC16 - Validate the "Remove Product" button | Validate that the "Remove Product" button removes the product from the cart. |
| Your Cart | TC17 - Validate the "Checkout" button | Validate that the "Checkout" button redirects to the checkout information page. |
| Checkout Your Information | TC18 - Clicking "Cancel" returns to the cart without saving information | Validate that clicking the "Cancel" button does not save entered information. |
| Checkout Your Information | TC19 - Validate filling text fields and clicking "Continue" | Validate that clicking the "Continue" button saves the entered information. |
| Checkout Your Information | TC20 - Validate mandatory text fields | Validate that clicking the "Continue" button triggers validations for mandatory fields. |
| Checkout Overview | TC21 - The "Cancel" button should return to the product list page | Validate that clicking the "Cancel" button redirects the user to the product list page. |
| Checkout Overview | TC22 - The "Continue" button should finalize the order | Validate that clicking the "Continue" button completes the order and redirects to the success page. |
| Checkout Complete | TC23 - Clicking the "Back Home" button should return to the product list page | Validate that clicking the "Back Home" button on the order completion page redirects to the product list page. |
Table 4. Automated Test Cases
To ensure a fair analysis without discrepancies due to execution environment differences, all test suites were executed locally on a single machine. The machine's specifications are as follows:
- Computer: Samsung Expert X50 NP350XBE-XH3BR
- Processor: QuadCore Intel Core i7-8565U, 4050 MHz (42x96)
- Memory: 20 GB 2667 MHz DDR4
- Graphics Card: NVIDIA GeForce MX 110 2048MB
- Operation System: Microsoft Windows 11 Home Single Language
- Browser: Google Chrome
The study began with the installation and configuration of the Cypress framework, which proved to be straightforward. The only prerequisite for Cypress is having Node.js installed on the machine where the automation project will be set up and configured.
The commands "npm init" to initialize the project and "npm install cypress" to install Cypress dependencies were used sequentially,resulting in the package.json file shown in Figure 2.
Figure 2. Package.json file of the poc_cypress project.
No additional dependencies were required for the Cypress project, as the framework includes its own libraries for testing, reporting, and assertions, and natively supports screenshot and video recording of test executions.
The initial project setup is also simplified by running the "npx cypress open" command, which opens the Cypress runner and creates the basic folder structure on its first execution. The project is publicly available on GitHub: https://github.com/AndradeTC86/poc_cypress
The Playwright installation and configuration were initiated next. This was also straightforward, with Node.js being the only prerequisite. After creating the directory, the "npm init playwright@latest" command was executed, which installed the framework and prompted several configuration options for the project. These options included selecting the programming language (JavaScript or TypeScript), defining the folder for test scripts, enabling GitHub Actions integration, and choosing to install supported browsers.
In addition to the basic installation, the @types/node dependency was added to address JavaScript's limitations with type definitions for Node libraries, such as using require for importing pages and custom commands. After completing the installation, the package.json file appeared as shown in Figure 3.
Figure 3. Package.json file of the poc_playwright project.
No additional libraries were required because Playwright has built-in support for testing, reporting, screenshot capture, and assertions. It also supports integration with other testing frameworks if needed.
Regarding project organization, only the predefined folder for test scripts was created, requiring the manual setup of other project folders. This could pose a challenge for individuals with less experience in test automation.
Playwright natively runs tests in headless mode on the installed browsers available on the local machine. It also runs tests in parallel, utilizing the number of sessions supported by the system's memory and processor. On my local machine, up to four sessions were used simultaneously. By default, Playwright generates a failure report automatically when a test fails; if all tests pass, the report isn't automatically displayed but can be accessed with the command "npx playwright show-report". This project is plubicly available on GitHub: https://github.com/AndradeTC86/poc_playwright
The last framework installed was Selenium, which required more initial configurations. Selenium has no strict prerequisites and can be configured for projects using any of the supported programming languages. For consistency with the other projects, Node.js was used as the base for this automation project.
The "npm install selenium" was used to install the framework. Since Sleenium lacks native support for certain features, additional dependencies like Axios (for API requests to verify login response times), Chai (for assertion libraries), and Mocha (for test execution) were installed. This also required changing the file extensions from .js para .mjs, resulting in the package.json file shown in Figure 4.
Figure 4. Package.json file of the projeto poc_selenium project.
The entire folder structure for the project had to be created manually, as the framework does not assist with this task.
The latest version of Selenium introduced improvements, such as automatically downloading the appropriate browser drivers, simplifying the process compared to earlier versions that required manually saving the driver version in a specific project directory to run tests. This project is publicly available on GitHub: https://github.com/AndradeTC86/poc_selenium
Among the frameworks evaluated. Cypress proved to be the easiest to install, as it required no external dependencies and automatically configured the project folders on the first execution.
Playwright was also simple to install, with no dependencies and a basic setup created after running the initialization command.
Selenium was the most complex to install, requiring external dependencies and entirely manual initial setup.
Cypress is a testing framework used for web application automation. It has a unique architecture designed to execute tests quickly and reliably. During user interface testing, all commands are executed directly in the browser without relying on any driver. Since the tests run directly in the browser, execution is faster, unaffected by network speed. Figure 5 provides an overview of the Cypress architecture.
Figure 5. Cypress Architecture [Tutorials Point 2020]
There is a Node server behind Cypress, and the Node process continuously communicates, synchronizes, and performs tasks on behalf of one another. The Node server and the browser communicate via WebSocket, which is initiated as soon as the proxy is created. Due to this method of running tests with the browser, it becomes easier to work directly with the DOM2, network layer, and local storage. This ease of access with Cypress makes ti advantageous for test execution compared to other frameworks. [Pathak 2023]
Playwright operates directly with WebSocket, meaning that once a test is initiated, the code is converted into JSON format and sent to the server using the WebSocket protocol. When the connection is established, commands are exchanged between the test and the Playwright server. The Playwright architecture is shown in Figure 6.
Figure 6. Playwright Architecture [Learn Microsoft 2023]
The connections between the client and server remain active until one or both parties terminate them. After the connection is closed, it is shut down on both ends. One reason Playwright is fast is that the connection remains active as long as neither party ends it [Pathak 2023]
Selenium is an open-source testing framework used for web test automation. The JSON Wire protocol was removed in Selenium 4 in favor of the W3C WebDriver, which is now the official standard for controlling web browsers. However, the Selenium team continues to provide support for the outdated protocol.
The new protocol, called W3C WebDriver, has been adopted by the “World Wide Web Consortium” (W3C). In the architecture of Selenium 4, there is direct communication between the client and server, eliminating the need for the JSON Wire protocol. The same protocol is used by the Selenium WebDriver and web browsers, making test case execution faster and significantly reducing instability. An example of the Selenium architecture is shown in Figure 7.
Figure 7. Selenium Architecture [Pathak 2023]
There are many benefits to adopting the W3C WebDriver protocol, which is more robust compared to JSON Wire. Notably, the actions API has been redesigned to align with the WebDriver specification, now enabling multitouch actions, zooming in and out, pressing two keys simultaneously, and more. [Pathak 2023].
The Playwright architecture has proven to be the most modern among the evaluated frameworks, offering excellent performance by maintaining an active connection during test execution. This results in higher execution speed but with slightly increased complexity.
Cypress architecture, by running directly in the browser, also provides speed and offers a simplified way to communicate with the page under test.
The Selenium architecture, while robust and widely used, may lag in speed and simplicity compared to the other tools.
This sections aims to assess the ease of coding tests in each framework, comparing the amount of code required to automate the same tests and the difficulty of learning the syntax for each.
Cypress supports only JavaScript, which may require testers to learn a new programming language. However, its simple syntax makes the learning curve quite steep.
Cypress offers a user-friendly interface through the Test Runner, which helps configure E2E or component tests for unit testing. When selecting the desired test type for the first time, the Test Runner creates folder structures with simple test examples that serve as templates for new tests. The Test Runner interface is shown in Figure 8.
Figure 8. Initial screen of the Cypress Test Runner
During E2E test execution, Cypress automatically loads the browsers installed on the machine. Currently, the framework supports Chrome, Edge, Firefox, Safari, and Electron—the latter being a browser integrated into Cypress. After selecting the desired browser, the existing test files are displayed for execution in debug mode.
The debug mode greatly aids testers in understanding what happens at each step of the test, as it provides screenshots of each step, an API call log, and a log of entered values. By clicking the crosshair icon and selecting an element, Cypress displays the best selector for that element following programming best practices. This feature helps less-experienced users correctly identify selectors. Figure 9 shows an example of a successfully executed test in debug mode and a selector being identified.
Figure 9. Test executed by the Test Runner
Basic project settings, such as the base URL, security configurations, and other necessary information for test execution, are stored in the cypress.config.js file, located in the project root. Notably, Cypress encountered compatibility issues with the site used in this study, freezing during login. To work around this, Cypress was configured to ignore browser security settings when accessing the site—a limitation in real-world testing scenarios.
Cypress, by default, waits for elements to become visible before attempting interactions, reducing the number of test failures due to timeout issues. If additional waits are required, the cy.wait command can be used; however, in my automation experience, this was unnecessary.
Cypress automatically closes the browser after test execution, eliminating the need to create a method for closing it after each test. The framework also natively supports API testing through the cy.request command, which executes API requests, and the cy.intercept command, which captures and modifies requests. This is particularly useful for simulating errors that are difficult to replicate in regular testing.
Thanks to its ease of mapping page objects and coding tests without requiring many external libraries, the project had the fewest lines of code compared to other frameworks, as shown in Figure 10.
Figure 10. Number of lines of code in the poc_cypress project [Code Tabs 2024a]
Playwright supports Java, C#, Python, JavaScript, and TypeScript, making it easier to adopt without significant language-specific learning requirements.
After initializing a Playwright project, test folders are created with example tests to assist in creating new ones. Basic configurations, such as the base URL, test directories, report format, and browsers for parallel execution, are stored in the playwright.config.js file, which is also created during initialization.
Selectors are manually captured, requiring users to have basic knowledge of finding elements on a page under test. However, running tests in debug mode is very helpful as it shows the location of each searched element at each executed step. Figure 11 illustrates the debug mode feature.
Figure 11. Debug mode in Playwright
Playwright executes tests asynchronously. Therefore, function names must include async, and calls must use await. Waits are handled automatically, requiring no additional configuration. Tests can be run on Chromium, Firefox, or WebKit, making it easy to work with multiple browsers concerning performance and wait times.
Playwright automatically closes the browser after completing each test and generates automatic reports for each execution. If a test fails, the report opens in the browser to highlight the step where the error occurred.
Although Playwright natively supports assertion and API interaction libraries, these classes must be imported, adding some extra lines of code. This, however, does not significantly impact project size. Another factor contributing to the number of lines of code, as shown in Figure 12, was the need to initialize variables so that each new page inherits the driver from the previous page.
Figure 12. Number of lines of code in the poc_playwright project [Code Tabs 2024b]
Selenium, being older, supports more languages than the other frameworks analyzed. It can be used with Java, C#, Python, PHP, Perl, Ruby, JavaScript, and TypeScript, reducing the need to learn a new programming language for test automation.
As the project was developed in VS Code, the entire structure had to be built from scratch, as Selenium does not provide tools to assist with project creation or generate example tests like other frameworks do. This requires basic knowledge of test automation and makes Selenium more suitable for experienced users.
There is no standard configuration file for Selenium projects, and the framework works best with the Page Objects Model. Therefore, basic methods for filling fields, clicking buttons, retrieving element values, and other common functions are created in a base page class to be reused across other pages.
It is also necessary to create a base test with methods to initialize and terminate the driver before and after execution, as the browser will not close automatically at the end of a test unless explicitly programmed to do so.
Element identification is also done manually, but Selenium does not provide a native tool to help identify selectors. Testers need to rely on alternatives available in their preferred browser. Additionally, Selenium lacks a native debug tool, requiring testers to create logs or use the debugging capabilities of their IDE (e.g., VS Code) to diagnose issues during test execution.
Selenium does not include built-in waits for test execution. As a result, testers must implement implicit or explicit waits in the code to ensure the page fully loads before interacting with elements. To address this, Mocha was added to the project, enabling asynchronous test execution, similar to Playwright. This required the use of async in function definitions and await for method calls.
Even with Mocha, the project encountered issues during certain flow tests, producing false positives. For example, logs indicated that an action could not be performed even when it had been successfully executed. Selenium supports Chrome, Firefox, Edge, and Safari, though the latter requires additional configurations. In standard mode, the test for adding and removing all products from the shopping cart consistently returned false positives, as the framework failed to click the last button to remove an item. This test ran without errors in headless mode, but other tests that worked in standard mode then produced false positives.
Due to its lack of native support, Mocha was added to execute tests, Axios to make requests for response time assessments, and Chai for more efficient validations. The need to import external dependencies, combined with the fact that even native Selenium classes must be explicitly imported, significantly increased the codebase size. Additionally, creating base methods for reuse and initializing variables to ensure each new page inherited the driver from the previous one further expanded the code. Consequently, the Selenium project codebase was substantially larger than the others, as shown in Figure 13.
Figure 13. Number of lines of code in the poc_selenium project [Code Tabs 2024c]
Thanks to its simple and intuitive syntax, Cypress stood out as the easiest option to learn, resulting in the smallest codebase among the evaluated frameworks.
Playwright also performed well in terms of ease of learning, though it had a slightly higher level of complexity compared to Cypress.
Selenium has a steeper learning curve. Combined with the need to add external dependencies, it requires greater knowledge of both the framework and the additional libraries.
To ensure the most accurate time measurement, all projects were coded using the same programming language (JavaScript), executed locally on the same machine, and run in the Chrome browser.
Starting with Cypress, the total execution time was 1 minute and 17 seconds, with all tests passing without any errors. The Cypress Dashboard report is shown in Figures 14, 15, and 16 below.
Figure 14. Test suite execution for poc_cypress
Figure 15. Test suite execution for poc_cypress
Figure 16. Test suite execution for poc_cypress Source: https://cloud.cypress.io/projects/xsy89k/runs/1/overview?roarHideRunsWithDiffGroupsAndTags=1
By default, Playwright executes tests in parallel across multiple instances. However, for comparison purposes, it was configured to run with just one instance to align with the other frameworks. The total execution time was 31 seconds, with all tests passing without issues. The native execution report is shown in Figures 17 and 18.
Figure 17. Test suite execution for poc_playwright
Figure 18. Test suite execution for poc_playwright
For Selenium, the execution time was approximately 1 minute, with one test resulting in a false positive. The report generated in the VS Code terminal is shown in Figures 19 and 20.
Figure 19. Test suite execution for poc_selenium
Figure 20. Test suite execution for poc_selenium
When converting the execution times of all tests to milliseconds for more precise evaluation, Selenium exhibited better speed performance for the tests that passed. However, it was not possible to verify the execution time of the test that failed. Notably, in shorter flows, Cypress demonstrated faster validations compared to Playwright. However, in longer flows, Playwright achieved significant performance gains. Figure 21 compares the execution times. The Cypress and Playwright tests were executed in headless mode, while Selenium tests had to be run in headed mode because, in headless mode, several tests that functioned correctly were returning errors.
Figure 21. Comparison of test execution times
Table 5 provides a comparison of the functionalities supported by each framework.
| Playwright | Selenium | Cypress | |
|---|---|---|---|
| Supported languages | JavaScript, Java, C#, Python, Ruby, TypeScript | JavaScript, Java, C#, Python, Ruby, TypeScript, PHP, Perl | JavaScript, TypeScript |
| Supported browsers | Chrome, Edge, Firefox, Safari | Chrome, Edge, Firefox, Safari | Chrome, Edge, Firefox, Safari |
| Supported frameworks | Jest/Jasmine, AVA, Mocha, Vitest | Mocha, Jest/Jasmine, TestNG, JUnit, Cucumber, NUnit | Mocha, Jest/Jasmine, Cucumber |
| Continuous integration | Easily integrates with CI tools | Easily integrates with CI tools | Easily integrates with CI tools |
| Ease of use | User-friendly interface, minimal configuration required | Requires more setup, steeper learning curve | User-friendly interface, minimal configuration required |
| Test writing experience | Intuitive | Moderate | Intuitive |
| DOM handling | Easy | Moderate | Easy |
| Community support | Growing community | Large, active community with excellent documentation and support resources | Active community with excellent documentation and support resources |
| Headless execution | Yes | Yes | Yes |
| Parallel execution | Supports parallel execution | Supports parallel execution | Supports parallel execution using CI/CD tools |
| Built-in network traffic control | Yes | No | Yes |
| Initial setup complexity | Easy setup | Requires effort to set up | Easy setup |
| iFrame support | Yes | Yes | Supported via plugins such as cypress-iframe |
| Driver requirement | No driver needed | Requires a driver for each browser (from Selenium 4, the driver is downloaded automatically) | No driver needed |
| Multiple tab support | Yes | No | Yes |
| Drag-and-drop support | Yes | Yes | Yes |
| Assertion libraries | Mocha, Chai | PyUnit, JUnit, TestNG, and almost any language-specific testing framework can be adapted | Mocha, Chai |
| Embedded reports | Yes | No | Default report is Spec but can be customized with other supported reports |
| Cross-domain communication | Yes | Yes | Yes |
| Debugging tools | Built-in debugging tools with functionality to rewind specific execution moments for easier debugging | No built-in debugging tools | Built-in debugging tools with functionality to rewind specific execution moments for easier debugging |
| Automatic waits | Yes | No | Yes |
| Dashboard | No | No | Yes, as a paid feature |
| Built-in screenshot and video capture | Yes | Not available by default, requires customization to include this functionality | Yes |
| Pricing | Free for open-source projects, paid for commercial use | Free for all use cases | Free for open-source projects, paid for commercial use |
Table 5. Comparative table
Due to its ease of use and configuration, Cypress is recommended for teams with beginner testers who are just starting in test automation, enabling the creation of projects that test both web functionalities and APIs. Although Cypress tests rarely produce false positives, it is worth noting that the execution time of very large tests tends to be quite high, which can be a drawback for test suites containing many substantial tests.
Playwright performed better across most criteria, offering robust support for multiple programming languages, ease of learning, built-in resources, fast test execution, support for both web and API testing, default parallel test execution across multiple browsers, and mobile application simulation. This makes it an excellent choice for test automation regardless of the team’s experience level.
While Selenium demonstrated the best performance in terms of execution time and supports the largest number of programming languages among the evaluated frameworks, it proved to be the most complex to automate due to its higher configuration and learning requirements. Moreover, the lack of built-in resources necessitates the installation of numerous dependencies, making Selenium more suitable for teams with advanced experience in test automation.
[ISTQB-CTFL] ISTQB Foundation Level Syllabus, Version 4.0.
Bastos, A.; Rios, E.; Cristalli, R. and Moreira, T. (2007): Base de conhecimento em teste de software. São Paulo: Editora Martins.
França, Emanuel Victor (2023). Automação de testes funcionais para aplicações web: comparativo dos frameworks Cypress e Playwright
Oliveira, T. (2018). Automação de testes: o que é, quando e por que automatizar. https://medium.com/venturus/quais-as-raz%C3%B5es-para-automa%C3%A7%C3%A3o-de-testes-c177cbd9397
Selenium (2023a). The Selenium Browser Automation Project https://www.selenium.dev/documentation/
Selenium (2024a). Selenium https://github.com/SeleniumHQ/selenium
Cypress (2024a). Why Cypress? https://docs.cypress.io/guides/overview/why-cypress
Cypress (2024b). Cypress https://github.com/cypress-io/cypress
Playwright (2024a). Installation https://playwright.dev/docs/intro
Playwright (2024b). Playwright https://github.com/microsoft/playwright
Puppeteer (2024). Puppeteer https://pptr.dev/
Raggo, G. (2023). Puppeteer vs Selenium vs Playwright, a speed comparison https://www.checklyhq.com/blog/puppeteer-vs-selenium-vs-playwright-speed-comparison/
Raggo, G. (2024). Cypress vs Selenium vs Playwright vs Puppeteer speed comparison https://www.checklyhq.com/blog/cypress-vs-selenium-vs-playwright-vs-puppeteer-speed-comparison/
Döring, P. and Brueckner K. (2022). Why we favor Playwright over Selenium or Cypress https://medium.com/tech-p7s1/why-favor-playwright-over-selenium-or-cypress-e96df84c08e1
Akhtar, H (2023). Cypress vs Selenium vs Playwright vs Puppeteer: Core Differences https://www.browserstack.com/guide/cypress-vs-selenium-vs-playwright-vs-puppeteer
Pathak, K (2023). Playwright vs Selenium vs Cypress: A Detailed Comparison https://www.lambdatest.com/blog/playwright-vs-selenium-vs-cypress/
NPM Trends: Comparative Analysis of Download Numbers for Cypress vs Playwright vs Selenium WebDriver. https://npmtrends.com/cypress-vs-playwright-vs-selenium-webdriver Acessado em: 23/04/2024
Tutorials Point (2020): Cypress Architecture (Test Automation) https://www.tutorialspoint.com/cypress-architecture-test-automation
Learn Microsoft (2023): O que é o Microsoft Playwright Testing? https://learn.microsoft.com/pt-br/azure/playwright-testing/overview-what-is-microsoft-playwright-testing
Code Tabs (2024a): Lines of Code for Poc Cypress Project. https://api.codetabs.com/v1/loc/?github=AndradeTC86/poc_cypress
Code Tabs (2024b): Lines of Code for Poc Playwright Project. https://api.codetabs.com/v1/loc/?github=AndradeTC86/poc_playwright
Code Tabs (2024c): Lines of Code for Poc Selenium Project. https://api.codetabs.com/v1/loc/?github=AndradeTC86/poc_selenium
Footnotes
-
Puppeteer is a JavaScript library that provides a high-level API to control Chrome or Firefox via the DevTools protocol or WebDriver BiDi. Puppeteer runs in headless mode by default. [Puppeteer 2024] ↩
-
Document Object Model (DOM) is a programming interface for HTML, XML, and SVG documents, providing a structured representation of the document as a tree. ↩
Addendum to the Article:
Following the completion of the comparative analysis, a refactoring of the automation projects was carried out, during which the root cause of the failing tests with Selenium was identified.
The test in question was attempting to locate the button using the CSS class and was failing intermittently, resulting in a false positive error. This was corrected by switching to using the ID selector, and the issue ceased to occur. It is worth noting that in the Cypress and Playwright projects, no errors occurred when using the CSS class, highlighting another limitation of Selenium.
After the correction, the total execution time for the three projects was validated. Playwright remained the fastest in terms of execution time, followed by Selenium, with a difference of approximately 20 seconds more. Cypress continued to be the slowest, taking twice the time of Playwright.