diff --git a/docs/advanced.md b/docs/advanced.md index b1ff65939..b11f1d821 100644 --- a/docs/advanced.md +++ b/docs/advanced.md @@ -81,37 +81,6 @@ Data(function*() { *HINT: If you don't use DataTable. add `toString()` method to each object added to data set, so the data could be pretty printed in a test name* -## Tags - -Append `@tag` to your test name, so - -```js -Scenario('update user profile @slow') -``` - -Alternativly, use `tag` method of Scenario to set additional tags: - -```js -Scenario('update user profile', ({ }) => { - // test goes here -}).tag('@slow').tag('important'); -``` - -All tests with `@tag` could be executed with `--grep '@tag'` option. - -```sh -codeceptjs run --grep '@slow' -``` - -Use regex for more flexible filtering: - -* `--grep '(?=.*@smoke2)(?=.*@smoke3)'` - run tests with @smoke2 and @smoke3 in name -* `--grep "\@smoke2|\@smoke3"` - run tests with @smoke2 or @smoke3 in name -* `--grep '((?=.*@smoke2)(?=.*@smoke3))|@smoke4'` - run tests with (@smoke2 and @smoke3) or @smoke4 in name -* `--grep '(?=.*@smoke2)^(?!.*@smoke3)'` - run tests with @smoke2 but without @smoke3 in name -* `--grep '(?=.*)^(?!.*@smoke4)'` - run all tests except @smoke4 - - ## Debug @@ -185,167 +154,3 @@ You can use this options for build your own [plugins](https://codecept.io/hooks/ ... }); ``` - -## Timeout - -Tests can get stuck due to various reasons such as network connection issues, crashed browser, etc. -This can make tests process hang. To prevent these situations timeouts can be used. Timeouts can be set explicitly for flaky parts of code, or implicitly in a config. - -> Previous timeout implementation was disabled as it had no effect when dealing with steps and promises. - -### Steps Timeout - -It is possible to limit a step execution to specified time with `I.limitTime` command. -It will set timeout in seconds for the next executed step: - -```js -// limit clicking to 5 seconds -I.limitTime(5).click('Link') -``` - -It is possible to set a timeout for all steps implicitly (except waiters) using [stepTimeout plugin](/plugins/#steptimeout). - -### Tests Timeout - -Test timeout can be set in seconds via Scenario options: - -```js -// limit test to 20 seconds -Scenario('slow test that should be stopped', { timeout: 20 }, ({ I }) => { - // ... -}) -``` - -This timeout can be set globally in `codecept.conf.js` in seconds: - -```js -exports.config = { - - // each test must not run longer than 5 mins - timeout: 300, - -} -``` - -### Suites Timeout - -A timeout for a group of tests can be set on Feature level via options. - -```js -// limit all tests in this suite to 30 seconds -Feature('flaky tests', { timeout: 30 }) -``` - -### Timeout Confguration - - - -Timeout rules can be set globally via config. - -To set a timeout for all running tests provide a **number of seconds** to `timeout` config option: - - -```js -// inside codecept.conf.js or codecept.conf.ts -timeout: 30, // limit all tests in all suites to 30 secs -``` - -It is possible to tune this configuration for a different groups of tests passing options as array and using `grep` option to filter tests: - -```js -// inside codecept.conf.js or codecept.conf.ts - -timeout: [ - 10, // default timeout is 10secs - - // but increase timeout for slow tests - { - grep: '@slow', - Feature: 50 - }, -] -``` - -> ℹ️ `grep` value can be string or regexp - -It is possible to set a timeout for Scenario or Feature: - -```js -// inside codecept.conf.js or codecept.conf.ts -timeout: [ - - // timeout for Feature with @slow in title - { - grep: '@slow', - Feature: 50 - }, - - // timeout for Scenario with 'flaky0' .. `flaky1` in title - { - // regexp can be passed to grep - grep: /flaky[0-9]/, - Scenario: 10 - }, - - // timeout for all suites - { - Feature: 20 - } -] -``` - -Global timeouts will be overridden by explicit timeouts of a test or steps. - -### Disable Timeouts - -To execute tests ignoring all timeout settings use `--no-timeouts` option: - -``` -npx codeceptjs run --no-timeouts -``` - -## Dynamic Configuration - -Helpers can be reconfigured per scenario or per feature. -This might be useful when some tests should be executed with different settings than others. -In order to reconfigure tests use `.config()` method of `Scenario` or `Feature`. - -```js -Scenario('should be executed in firefox', ({ I }) => { - // I.amOnPage(..) -}).config({ browser: 'firefox' }) -``` - -In this case `config` overrides current config of the first helper. -To change config of specific helper pass two arguments: helper name and config values: - -```js -Scenario('should create data via v2 version of API', ({ I }) => { - // I.amOnPage(..) -}).config('REST', { endpoint: 'https://api.mysite.com/v2' }) -``` - -Config can also be set by a function, in this case you can get a test object and specify config values based on it. -This is very useful when running tests against cloud providers, like BrowserStack. This function can also be asynchronous. - -```js -Scenario('should report to BrowserStack', ({ I }) => { - // I.amOnPage(..) -}).config((test) => { - return { desiredCapabilities: { - project: test.suite.title, - name: test.title, - }} -}); -``` - -Config changes can be applied to all tests in suite: - -```js -Feature('Admin Panel').config({ url: 'https://mysite.com/admin' }); -``` - -Please note that some config changes can't be applied on the fly. For instance, if you set `restart: false` in your config and then changing value `browser` won't take an effect as browser is already started and won't be closed untill all tests finish. - -Configuration changes will be reverted after a test or a suite. - diff --git a/docs/basics.md b/docs/basics.md index e7d11ca0a..d42febdff 100644 --- a/docs/basics.md +++ b/docs/basics.md @@ -16,996 +16,400 @@ Scenario('check Welcome page on site', ({ I }) => { }) ``` -Tests are expected to be written in **ECMAScript 7**. +Tests are written as **ES modules** using modern JavaScript syntax. Each test is described inside a `Scenario` function with the `I` object passed into it. The `I` object is an **actor**, an abstraction for a testing user. The `I` is a proxy object for currently enabled **Helpers**. -## Architecture - -CodeceptJS bypasses execution commands to helpers. Depending on the helper enabled, your tests will be executed differently. - -The following is a diagram of the CodeceptJS architecture: - -![architecture](/img/architecture.png) - -All helpers share the same API, so it's easy to migrate tests from one backend to another. -However, because of the difference in backends and their limitations, they are not guaranteed to be compatible with each other. For instance, you can't set request headers in WebDriver but you can do so in Playwright or Puppeteer. +## How It Works -**Pick one helper, as it defines how tests are executed.** If requirements change it's easy to migrate to another. +### Command Delegation to Helpers ---- +**CodeceptJS delegates all test commands to helper backends.** Tests written with the `I` object (the actor) don't directly execute actions. Instead, CodeceptJS routes them through configurable helpers: -Refer to following guides to more information on: +- **Playwright** - Chromium, Firefox, WebKit automation +- **WebDriver** - Native browser automation via WebDriver Protocol +- **Appium** - Mobile testing on iOS/Android +- **Puppeteer** - Chromium automation via DevTools Protocol -- [▶ Playwright](/playwright) -- [▶ WebDriver](/webdriver) -- [▶ Puppeteer](/puppeteer) +All helpers share the same API, so it's easy to switch backends. However, due to backend differences and limitations, they aren't guaranteed to be compatible with each other. For example, you can set request headers in Playwright or Puppeteer, but not in WebDriver. -> ℹ Depending on a helper selected a list of available actions may change. +> **Pick one helper to define how your tests execute.** If requirements change, it's straightforward to migrate to another. -To list all available commands for the current configuration run `codeceptjs list` -or enable [auto-completion by generating TypeScript definitions](#intellisense). -> 🤔 It is possible to access API of a backend you use inside a test or a [custom helper](/helpers/). For instance, to use Puppeteer API inside a test use [`I.usePuppeteerTo`](/helpers/Puppeteer/#usepuppeteerto) inside a test. Similar methods exist for each helper. +### Promise Chain & Async/Await -## Writing Tests +Tests appear synchronous but **all actions are wrapped in promises and chained together** in a global promise chain. This means: -Tests are written from a user's perspective. There is an actor (represented as `I`) which contains actions taken from helpers. A test is written as a sequence of actions performed by an actor: +- **You usually don't need `await` for regular actions** - commands are automatically queued +- Each `I.*` command is appended to the promise chain +- Setup, teardown, and all test steps execute in sequence ```js +// These execute in order WITHOUT await I.amOnPage('/') I.click('Login') -I.see('Please Login', 'h1') -// ... -``` - -### Opening a Page - -A test should usually start by navigating the browser to a website. - -Start a test by opening a page. Use the `I.amOnPage()` command for this: - -```js -// When "http://site.com" is url in config -I.amOnPage('/') // -> opens http://site.com/ -I.amOnPage('/about') // -> opens http://site.com/about -I.amOnPage('https://google.com') // -> https://google.com -``` - -When an URL doesn't start with a protocol (http:// or https://) it is considered to be a relative URL and will be appended to the URL which was initially set-up in the config. - -> It is recommended to use a relative URL and keep the base URL in the config file, so you can easily switch between development, stage, and production environments. - -### Locating Element - -Element can be found by CSS or XPath locators. - -```js -I.seeElement('.user') // element with CSS class user -I.seeElement('//button[contains(., "press me")]') // button -``` - -By default CodeceptJS tries to guess the locator type. -In order to specify the exact locator type you can pass an object called **strict locator**. - -```js -I.seeElement({ css: 'div.user' }) -I.seeElement({ xpath: '//div[@class=user]' }) -``` - -Strict locators allow to specify additional locator types: - -```js -// locate form element by name -I.seeElement({ name: 'password' }) -// locate element by React component and props -I.seeElement({ react: 'user-profile', props: { name: 'davert' } }) -``` - -In [mobile testing](https://codecept.io/mobile/#locating-elements) you can use `~` to specify the accessibility id to locate an element. In web application you can locate elements by their `aria-label` value. - -```js -// locate element by [aria-label] attribute in web -// or by accessibility id in mobile -I.seeElement('~username') -``` - -> [▶ Learn more about using locators in CodeceptJS](/locators). - -### Clicking - -CodeceptJS provides a flexible syntax to specify an element to click. - -By default CodeceptJS tries to find the button or link with the exact text on it - -```js -// search for link or button -I.click('Login') -``` - -If none was found, CodeceptJS tries to find a link or button containing that text. In case an image is clickable its `alt` attribute will be checked for text inclusion. Form buttons will also be searched by name. - -To narrow down the results you can specify a context in the second parameter. - -```js -I.click('Login', '.nav') // search only in .nav -I.click('Login', { css: 'footer' }) // search only in footer -``` - -> To skip guessing the locator type, pass in a strict locator - A locator starting with '#' or '.' is considered to be CSS. Locators starting with '//' or './/' are considered to be XPath. - -You are not limited to buttons and links. Any element can be found by passing in valid CSS or XPath: - -```js -// click element by CSS -I.click('#signup') -// click element located by special test-id attribute -I.click('//dev[@test-id="myid"]') -``` - -> ℹ If click doesn't work in a test but works for user, it is possible that frontend application is not designed for automated testing. To overcome limitation of standard click in this edgecase use `forceClick` method. It will emulate click instead of sending native event. This command will click an element no matter if this element is visible or animating. It will send JavaScript "click" event to it. - -### Filling Fields - -Clicking the links is not what takes the most time during testing a web site. If your site consists only of links you can skip test automation. The most waste of time goes into the testing of forms. CodeceptJS provides several ways of doing that. - -Let's submit this sample form for a test: - -![](https://user-images.githubusercontent.com/220264/80355863-494a8280-8881-11ea-9b41-ba1f07abf094.png) - -```html -
- -
- -
- -
- -
- -
-
-``` - -We need to fill in all those fields and click the "Update" button. CodeceptJS matches form elements by their label, name, or by CSS or XPath locators. - -```js -// we are using label to match user_name field -I.fillField('Name', 'Miles') -// we can use input name -I.fillField('user[email]', 'miles@davis.com') -// select element by label, choose option by text -I.selectOption('Role', 'Admin') -// click 'Save' button, found by text -I.checkOption('Accept') -I.click('Save') -``` - -> ℹ `selectOption` works only with standard ` HTML elements. If your selectbox is created by React, Vue, or as a component of any other framework, this method potentially won't work with it. Use `click` to manipulate it. - -> ℹ `checkOption` also works only with standard `` HTML elements. If your checkbox is created by React, Vue, or as a component of any other framework, this method potentially won't work with it. Use `click` to manipulate it. - -Alternative scenario: - -```js -// we are using CSS -I.fillField('#user_name', 'Miles') -I.fillField('#user_email', 'miles@davis.com') -// select element by label, option by value -I.selectOption('#user_role', '1') -// click 'Update' button, found by name -I.click('submitButton', '#update_form') -``` - -To fill in sensitive data use the `secret` function, it won't expose actual value in logs. - -```js -I.fillField('password', secret('123456')) -``` - -> ℹ️ Learn more about [masking secret](/secrets/) output - -### Assertions - -In order to verify the expected behavior of a web application, its content should be checked. -CodeceptJS provides built-in assertions for that. They start with a `see` (or `dontSee`) prefix. - -The most general and common assertion is `see`, which checks visilibility of a text on a page: - -```js -// Just a visible text on a page -I.see('Hello') -// text inside .msg element -I.see('Hello', '.msg') -// opposite -I.dontSee('Bye') -``` - -You should provide a text as first argument and, optionally, a locator to search for a text in a context. - -You can check that specific element exists (or not) on a page, as it was described in [Locating Element](#locating-element) section. - -```js -I.seeElement('.notice') -I.dontSeeElement('.error') +I.see('Welcome') ``` -Additional assertions: - +Behind the scenes, this is equivalent to: ```js -I.seeInCurrentUrl('/user/miles') -I.seeInField('user[name]', 'Miles') -I.seeInTitle('My Website') +Promise.resolve() + .then(() => I.amOnPage('/')) + .then(() => I.click('Login')) + .then(() => I.see('Welcome')) ``` -To see all possible assertions, check the helper's reference. +When You DO Need `await`: -> ℹ If you need custom assertions, you can install an assertion libarary like `chai`, use grabbers to obtain information from a browser and perform assertions. However, it is recommended to put custom assertions into a helper for further reuse. - -### Grabbing - -Sometimes you need to retrieve data from a page to use it in the following steps of a scenario. -Imagine the application generates a password, and you want to ensure that user can login using this password. +Use `await` only with **grab** actions (methods that retrieve data from the page): ```js -Scenario('login with generated password', async ({ I }) => { - I.fillField('email', 'miles@davis.com') +Scenario('use data from page', async ({ I }) => { + I.fillField('email', 'user@example.com') I.click('Generate Password') - const password = await I.grabTextFrom('#password') - I.click('Login') - I.fillField('email', 'miles@davis.com') - I.fillField('password', password) - I.click('Log in!') - I.see('Hello, Miles') -}) -``` - -The `grabTextFrom` action is used to retrieve the text from an element. All actions starting with the `grab` prefix are expected to return data. In order to synchronize this step with a scenario you should pause the test execution with the `await` keyword of ES6. To make it work, your test should be written inside a async function (notice `async` in its definition). - -```js -Scenario('use page title', async ({ I }) => { - // ... + // grab actions return data - use await here const password = await I.grabTextFrom('#password') I.fillField('password', password) + I.click('Login') }) ``` -### Waiting - -In modern web applications, rendering is done on the client-side. -Sometimes that may cause delays. A test may fail while trying to click an element which has not appeared on a page yet. -To handle these cases, the `wait*` methods has been introduced. - -```js -I.waitForElement('#agree_button', 30) // secs -// clicks a button only when it is visible -I.click('#agree_button') -``` - -## How It Works - -Tests are written in a synchronous way. This improves the readability and maintainability of tests. -While writing tests you should not think about promises, and instead should focus on the test scenario. - -However, behind the scenes **all actions are wrapped in promises**, inside of the `I` object. -[Global promise](https://github.com/codeceptjs/CodeceptJS/blob/master/lib/recorder.js) chain is initialized before each test and all `I.*` calls will be appended to it, as well as setup and teardown. - -> 📺 [Learn how CodeceptJS](https://www.youtube.com/watch?v=MDLLpHAwy_s) works with promises by watching video on YouTube - -If you want to get information from a running test you can use `await` inside the **async function**, and utilize special methods of helpers started with the `grab` prefix. - -```js -Scenario('try grabbers', async ({ I }) => { - let title = await I.grabTitle() -}) -``` - -then you can use those variables in assertions: - -```js -var title = await I.grabTitle() -var assert = require('assert') -assert.equal(title, 'CodeceptJS') -``` - -It is important to understand the usage of **async** functions in CodeceptJS. While non-returning actions can be called without await, if an async function uses `grab*` action it must be called with `await`: - -```js -// a helper function -async function getAllUsers(I) { - const users = await I.grabTextFrom('.users') - return users.filter(u => u.includes('active')) -} - -// a test -Scenario('try helper functions', async ({ I }) => { - // we call function with await because it includes `grab` - const users = await getAllUsers(I) -}) -``` - -If you miss `await` you get commands unsynchrhonized. And this will result to an error like this: +Also **use `await` with imported functions and page object methods**, as they may contain async operations that aren't part of the promise chain (e.g., `await loginPage.login()` if it contains `I.grab` operations inside). -``` -(node:446390) UnhandledPromiseRejectionWarning: ... - at processTicksAndRejections (internal/process/task_queues.js:95:5) -(node:446390) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2) -``` - -If you face that error please make sure that all async functions are called with `await`. - -## Running Tests - -To launch tests use the `run` command, and to execute tests in [multiple threads](/advanced/parallel) using `run-workers` command. - -### Level of Detail - -To see the step-by-step output of running tests, add the `--steps` flag: - -``` -npx codeceptjs run --steps -``` - -To see a more detailed output add the `--debug` flag: - -``` -npx codeceptjs run --debug -``` - -To see very detailed output informations use the `--verbose` flag: - -``` -npx codeceptjs run --verbose -``` - -### Filter - -A single test file can be executed if you provide a relative path to such a file: - -``` -npx codeceptjs run github_test.js - -# or +> **Rule:** If an action starts with `grab`, or if calling an imported function/page object method, you must `await` it. Regular actions (`I.click()`, `I.fillField()`, `I.see()`) don't need await. -npx codeceptjs run admin/login_test.js -``` - -To filter a test by name use the `--grep` parameter, which will execute all tests with names matching the regex pattern. - -To run all tests with the `slow` word in it: - -``` -npx codeceptjs run --grep "slow" -``` - -It is recommended to [filter tests by tags](/advanced/#tags). - -> For more options see [full reference of `run` command](/commands/#run). - -### Parallel Run - -Tests can be executed in parallel mode by using [NodeJS workers](https://nodejs.org/api/worker_threads.html). Use `run-workers` command with the number of workers (threads) to split tests into different workers. - -``` -npx codeceptjs run-workers 3 -``` - -Tests are split by scenarios, not by files. Results are aggregated and shown up in the main process. - -## Configuration - -Configuration is set in the `codecept.conf.js` file which was created during the `init` process. -Inside the config file you can enable and configure helpers and plugins, and set bootstrap and teardown scripts. - -```js -exports.config = { - helpers: { - // enabled helpers with their configs - }, - plugins: { - // list of used plugins - }, - include: { - // current actor and page objects - }, -} -``` - -> ▶ See complete [configuration reference](/configuration). - -You can have multiple configuration files for a the same project, in this case you can specify a config file to be used with `-c` when running. - -``` -npx codeceptjs run -c codecept.ci.conf.js -``` - -Tuning configuration for helpers like WebDriver, Puppeteer can be hard, as it requires good understanding of how these technologies work. Use the [`@codeceptjs/configure`](https://github.com/codeceptjs/configure) package with common configuration recipes. - -For instance, you can set the window size or toggle headless mode, no matter of which helpers are actually used. - -```js -const { setHeadlessWhen, setWindowSize } = require('@codeceptjs/configure') - -// run headless when CI environment variable set -setHeadlessWhen(process.env.CI) -// set window size for any helper: Puppeteer, WebDriver, Playwright -setWindowSize(1600, 1200) - -exports.config = { - // ... -} -``` - -> ▶ See more [configuration recipes](https://github.com/codeceptjs/configure) - -## Debug - -CodeceptJS allows to write and debug tests on the fly while keeping your browser opened. -By using the interactive shell you can stop execution at any point and type in any CodeceptJS commands. +## Writing Tests -This is especially useful while writing a new scratch. After opening a page call `pause()` to start interacting with a page: +Tests are written from a user's perspective. There is an actor (represented as `I`) which contains actions taken from helpers. A test is written as a sequence of actions performed by an actor: ```js I.amOnPage('/') -pause() -``` - -Try to perform your scenario step by step. Then copy succesful commands and insert them into a test. - -### Pause - -Test execution can be paused in any place of a test with `pause()` call. -Variables can also be passed to `pause({data: 'hi', func: () => console.log('hello')})` which can be accessed in Interactive shell. - -This launches the interactive console where you can call any action from the `I` object. - -``` - Interactive shell started - Press ENTER to resume test - Use JavaScript syntax to try steps in action - - Press ENTER to run the next step - - Press TAB twice to see all available commands - - Type exit + Enter to exit the interactive shell - - Prefix => to run js commands - I. -``` - -Type in different actions to try them, copy and paste successful ones into the test file. - -Press `ENTER` to resume test execution. - -To **debug test step-by-step** press Enter, the next step will be executed and interactive shell will be shown again. - -To see all available commands, press TAB two times to see list of all actions included in the `I` object. - -> The interactive shell can be started outside of test context by running `npx codeceptjs shell` - -PageObjects and other variables can also be passed to as object: - -```js -pause({ loginPage, data: 'hi', func: () => console.log('hello') }) -``` - -Inside a pause mode you can use `loginPage`, `data`, `func` variables. -Arbitrary JavaScript code can be executed when used `=> ` prefix: - -```js -I.=> loginPage.open() -I.=> func() -I.=> 2 + 5 -``` - -### Pause on Fail - -To start interactive pause automatically for a failing test you can run tests with [pauseOnFail Plugin](/plugins/#pauseonfail). -When a test fails, the pause mode will be activated, so you can inspect current browser session before it is closed. - -> **[pauseOnFail plugin](/plugins/#pauseOnFail) can be used** for new setups - -To run tests with pause on fail enabled use `-p pauseOnFail` option - -``` -npx codeceptjs run -p pauseOnFail +I.click('Login') +I.see('Please Login', 'h1') +// ... ``` -> To enable pause after a test without a plugin you can use `After(pause)` inside a test file. - -### Screenshot on Failure - -By default CodeceptJS saves a screenshot of a failed test. -This can be configured in [screenshotOnFail Plugin](/plugins/#screenshotonfail) - -> **[screenshotOnFail plugin](/plugins/#screenshotonfail) is enabled by default** for new setups - -### Step By Step Report - -To see how the test was executed, use [stepByStepReport Plugin](/plugins/#stepbystepreport). It saves a screenshot of each passed step and shows them in a nice slideshow. - -## Before - -Common preparation steps like opening a web page or logging in a user, can be placed in the `Before` or `Background` hooks: +A complete test file looks like this: ```js -Feature('CodeceptJS Demonstration') +// suite declaration, like describe() in other frameworks +Feature('User Authentication') +// before each hook Before(({ I }) => { - // or Background - I.amOnPage('/documentation') -}) - -Scenario('test some forms', ({ I }) => { - I.click('Create User') - I.see('User is valid') - I.dontSeeInCurrentUrl('/documentation') -}) - -Scenario('test title', ({ I }) => { - I.seeInTitle('Example application') + I.amOnPage('/') }) -``` - -Same as `Before` you can use `After` to run teardown for each scenario. - -## BeforeSuite -If you need to run complex a setup before all tests and have to teardown this afterwards, you can use the `BeforeSuite` and `AfterSuite` functions. -`BeforeSuite` and `AfterSuite` have access to the `I` object, but `BeforeSuite/AfterSuite` don't have any access to the browser, because it's not running at this moment. -You can use them to execute handlers that will setup your environment. `BeforeSuite/AfterSuite` will work only for the file it was declared in (so you can declare different setups for files) - -```js -BeforeSuite(({ I }) => { - I.syncDown('testfolder') +// a test +Scenario('user can login with valid credentials', ({ I }) => { + I.click('Login') + I.fillField('email', 'user@example.com') + I.fillField('password', 'password123') + I.click('Sign In') + I.see('Welcome, User') }) -AfterSuite(({ I }) => { - I.syncUp('testfolder') - I.clearDir('testfolder') +// after each hook +After(({ I }) => { + // ... }) ``` -## Retries - -### Auto Retry - -Each failed step is auto-retried by default via [retryFailedStep Plugin](/plugins/#retryfailedstep). -If this is not expected, this plugin can be disabled in a config. - -> **[retryFailedStep plugin](/plugins/#retryfailedstep) is enabled by default** incide global configuration +> CodeceptJS doesn't allow nested suites or multiple suites in one file. -### Retry Step - -Unless you use retryFailedStep plugin you can manually control retries in your project. - -If you have a step which often fails, you can retry execution for this single step. -Use the `retry()` function before an action to ask CodeceptJS to retry it on failure: +### Opening a Page -```js -I.retry().see('Welcome') -``` +A test should usually start by navigating the browser to a website. -If you'd like to retry a step more than once, pass the amount as a parameter: +Start a test by opening a page. Use the `I.amOnPage()` command for this: ```js -I.retry(3).see('Welcome') +// When "http://site.com" is url in config +I.amOnPage('/') // -> opens http://site.com/ +I.amOnPage('/about') // -> opens http://site.com/about +I.amOnPage('https://google.com') // -> https://google.com ``` -Additional options can be provided to `retry`, so you can set the additional options (defined in [promise-retry](https://www.npmjs.com/package/promise-retry) library). - -```js -// retry action 3 times waiting for 0.1 second before next try -I.retry({ retries: 3, minTimeout: 100 }).see('Hello') - -// retry action 3 times waiting no more than 3 seconds for last retry -I.retry({ retries: 3, maxTimeout: 3000 }).see('Hello') - -// retry 2 times if error with message 'Node not visible' happens -I.retry({ - retries: 2, - when: err => err.message === 'Node not visible', -}).seeElement('#user') -``` +When an URL doesn't start with a protocol (http:// or https://) it is considered to be a relative URL and will be appended to the URL which was initially set-up in the config. -Pass a function to the `when` option to retry only when an error matches the expected one. +> It is recommended to use a relative URL and keep the base URL in the config file, so you can easily switch between development, stage, and production environments. -### Retry Multiple Steps +### Locating Elements -To retry a group of steps enable [retryTo plugin](/plugins/#retryto): +CodeceptJS supports multiple locating strategies. Most actions accept locators as strings or objects: ```js -// retry these steps 5 times before failing -await retryTo(tryNum => { - I.switchTo('#editor frame') - I.click('Open') - I.see('Opened') -}, 5) -``` +// Basic CSS/XPath +I.click('.login-btn') +I.fillField('//input[@type="email"]', 'user@test.com') -### Retry Scenario +// Semantic - by text/labels (searches multiple strategies) +I.click('Login') +I.fillField('Email', 'user@test.com') -When you need to rerun scenarios a few times, add the `retries` option to the `Scenario` declaration. +// ARIA locators - by role and accessible name (most reliable) +I.click({ role: 'button', name: 'Login' }) +I.fillField({ role: 'textbox', name: 'Email Address' }, 'user@test.com') +I.seeElement({ role: 'button', name: 'Submit' }) -CodeceptJS implements retries the same way [Mocha does](https://mochajs.org#retry-tests); -You can set the number of a retries for a feature: +// Accessibility ID or aria-label +I.click('~login-button') +I.seeElement({ aria: 'Username' }) -```js -Scenario('Really complex', ({ I }) => { - // test goes here -}).retry(2) +// Complex locators with locate() +I.click(locate('button').withText('Save').inside('.modal')) +I.seeElement(locate({ role: 'button', name: 'Delete' }).inside('.toolbar')) -// alternative -Scenario('Really complex', { retries: 2 }, ({ I }) => {}) +// ARIA + Context - combine role with section to target specific area +I.click({ role: 'button', name: 'Save' }, '#detail-panel') +I.fillField({ role: 'textbox', name: 'Title' }, 'New Task', '.modal') +I.seeElement({ role: 'button', name: 'Delete' }, { css: '.toolbar' }) +I.click({ role: 'button', name: 'Close' }, '.sidebar') ``` -This scenario will be restarted two times on a failure. -Unlike retry step, there is no `when` condition supported for retries on a scenario level. - -### Retry Before - -To retry `Before`, `BeforeSuite`, `After`, `AfterSuite` hooks, call `retry()` after declaring the hook. +**Best practices:** -- `Before().retry()` -- `BeforeSuite().retry()` -- `After().retry()` -- `AfterSuite().retry()` +- **Use ARIA locators** `{ role: 'button', name: '...' }` - resilient to CSS changes and support accessibility +- **Combine with context** when multiple similar elements exist: `I.click({ role: 'button', name: 'Delete' }, '.toolbar')` +- **Use semantic CSS classes and IDs** when available (e.g., `.btn-save`, `#login-form`); avoid style-based names like `.bg-green` +- **Use `locate()` function** for complex queries: `locate(selector).withText(...).inside(...)` -For instance, to retry Before hook 3 times before failing: +> [▶ Learn more about using locators in CodeceptJS](/locators). -```js -Before(({ I }) => { - I.amOnPage('/') -}).retry(3) -``` +### Clicking Elements -Same applied for `BeforeSuite`: +Use the locator strategies from [Locating Elements](#locating-elements) section to click any element: ```js -BeforeSuite(() => { - // do some prepreations -}).retry(3) +I.click('Login') // by text +I.click({ role: 'button', name: 'Save' }) // by ARIA role +I.click('#signup') // by ID +I.click('Delete', '.toolbar') // with context ``` -Alternatively, retry options can be set on Feature level: +| | | | | +|---|---|---|---| +| [click](/web-api#iclick) | [forceClick](/web-api#iforcecclick) | [doubleClick](/web-api#idoubleclick) | [rightClick](/web-api#irightclick) | +| [forceRightClick](/web-api#iforcerightclick) | [moveCursorTo](/web-api#imovecursorto) | [dragAndDrop](/web-api#idraganddrop) | [dragSlider](/web-api#idragslider) | -```js -Feature('my tests', { - retryBefore: 3, - retryBeforeSuite: 2, - retryAfter: 1, - retryAfterSuite: 3, -}) -``` +Use **forceClick** when standard click fails (e.g., hidden elements, animations). Use **rightClick** for context menus, **doubleClick** for multi-select. -### Retry Feature +### Interacting with Forms -To set this option for all scenarios in a file, add `retry` to a feature: +Use form methods to interact with inputs, selects, checkboxes, and other form elements. Fields can be located by label, name, CSS, XPath, or aria-label: ```js -Feature('Complex JS Stuff').retry(3) -// or -Feature('Complex JS Stuff', { retries: 3 }) -``` +// Fill fields - by label, name, CSS, or aria-label +I.fillField('Email', 'user@test.com') +I.fillField('My Address', 'Home Sweet Home') // matches aria-label or aria-labelledby +I.fillField('LoginForm[username]', 'davert') // by field name attribute +I.fillField('Password', secret('123456')) // use secret() for sensitive data -Every Scenario inside this feature will be rerun 3 times. -You can make an exception for a specific scenario by passing the `retries` option to a Scenario. +// Use context (3rd parameter) to narrow search to specific form +I.fillField('Email', 'user@test.com', '#login-form') +I.fillField('Email', 'admin@test.com', '#registration-form') -### Retry Configuration - -It is possible to set retry rules globally via `retry` config option. The configuration is flexible and allows multiple formats. -The simplest config would be: - -```js -// inside codecept.conf.js -retry: 3 -``` +// Append or clear +I.appendField('Comments', ' — updated') +I.appendField('Message', ' P.S. Thank you!', '.contact-form') // with context +I.clearField('#search-input') -This will enable Feature Retry for all executed feature, retrying failing tests 3 times. +// Type into focused field - use when fillField doesn't trigger JS events +I.click('Card Number') +I.type('4111111111111111', 100) // optional delay in ms between keystrokes -An object can be used to tune retries of a Before/After hook, Scenario or Feature +// Selects - pass array for multi-select +I.selectOption('Role', 'Admin') +I.selectOption('Role', 'User', '.user-form') // with context +I.selectOption('Tags', ['Important', 'Urgent']) -```js -// inside codecept.conf.js -retry: { - Feature: ..., - Scenario: ..., - Before: ..., - BeforeSuite: ..., - After: ..., - AfterSuite: ..., -} +// Checkboxes and radios - supports context +I.checkOption('I Agree to Terms and Conditions') +I.checkOption('Remember me', '#login-form') // with context +I.uncheckOption('Subscribe') ``` -Multiple retry configs can be added via array. To use different retry configs for different subset of tests use `grep` option inside a retry config: +| | | | | +|---|---|---|---| +| [fillField](/web-api#ifillfield) | [clearField](/web-api#iclearfield) | [appendField](/web-api#iappendfield) | [type](/web-api#itype) | +| [selectOption](/web-api#iselectoption) | [checkOption](/web-api#icheckoption) | [uncheckOption](/web-api#iuncheckoption) | [focus](/web-api#ifocus) | +| [blur](/web-api#iblur) | | | | -```js -// inside codecept.conf.js -retry: [ - { - // enable this config only for flaky tests - grep: '@flaky', - Before: 3 - Scenario: 3 - }, - { - // retry less when running slow tests - grep: '@slow' - Scenario: 1 - Before: 1 - }, { - // retry all BeforeSuite - BeforeSuite: 3 - } -] -``` +> Use `secret()` for sensitive data: `I.fillField('password', secret('123456'))` - [won't expose in logs](/secrets/). +> -When using `grep` with `Before`, `After`, `BeforeSuite`, `AfterSuite`, a suite title will be checked for included value. +> [selectOption](/web-api#iselectoption) works with native `` -* {xpath: "//input[@type='submit'][contains(@value, 'foo')]"} matches `` -* {class: 'foo'} matches `
` -* { pw: '_react=t[name = "="]' } +* `{id: 'foo'}` matches `
` +* `{name: 'foo'}` matches `
` +* `{css: 'input[type=input][value=foo]'}` matches `` +* `{xpath: "//input[@type='submit'][contains(@value, 'foo')]"}` matches `` +* `{class: 'foo'}` matches `
` +* `{ pw: '_react=t[name = "="]' }` +* `{ role: 'button', name: 'Submit' }` matches `` Writing good locators can be tricky. The Mozilla team has written an excellent guide titled [Writing reliable locators for Selenium and WebDriver tests](https://blog.mozilla.org/webqa/2013/09/26/writing-reliable-locators-for-selenium-and-webdriver-tests/). -If you prefer, you may also pass a string for the locator. This is called a "fuzzy" locator. +If you prefer, you may also pass a string for the locator. This is called a **fuzzy** locator. In this case, CodeceptJS uses a variety of heuristics (depending on the exact method called) to determine what element you're referring to. If you are locating a clickable element or an input element, CodeceptJS will use [semantic locators](#semantic-locators). For example, here's the heuristic used for the `fillField` method: -1. Does the locator look like an ID selector (e.g. "#foo")? If so, try to find an input element matching that ID. +1. Does the locator look like an ID selector (e.g. `#foo`)? If so, try to find an input element matching that ID. 2. If nothing found, check if locator looks like a CSS selector. If so, run it. 3. If nothing found, check if locator looks like an XPath expression. If so, run it. 4. If nothing found, check if there is an input element with a corresponding name. 5. If nothing found, check if there is a label with specified text for input element. 6. If nothing found, throw an `ElementNotFound` exception. -> ⚠ Be warned that fuzzy locators can be significantly slower than strict locators. If speed is a concern, it's recommended you stick with explicitly specifying the locator type via object syntax. +> ⚠ Fuzzy locators can be significantly slower than strict locators. If speed is a concern, stick with explicitly specifying the locator type via object syntax. It is recommended to avoid using implicit CSS locators in methods like `fillField` or `click`, where semantic locators are allowed. Use locator type to speed up search by various locator strategies. @@ -58,7 +60,7 @@ I.fillField({ css: 'input[type=password]' }, '123456'); ## CSS and XPath -Both CSS and XPath is supported. Usually CodeceptJS can guess locator's type: +Both CSS and XPath are supported. Usually CodeceptJS can guess the locator's type: ```js // select by CSS @@ -84,15 +86,33 @@ I.seeElement({ xpath: 'descendant::table/tr' }); ## Semantic Locators CodeceptJS can guess an element's locator from context. -For example, when clicking CodeceptJS will try to find a link or button by their text -When typing into a field this field can be located by its name, placeholder. +For example, when clicking CodeceptJS will try to find a link or button by their text. +When typing into a field, the field can be located by its name or placeholder. ```js I.click('Sign In'); I.fillField('Username', 'davert'); ``` -Various strategies are used to locate semantic elements. However, they may run slower than specifying locator by XPath or CSS. +Various strategies are used to locate semantic elements. However, they may run slower than specifying the locator by XPath or CSS. + +## ARIA and Role Locators + +ARIA locators find elements by their ARIA role and accessible name. They are the most resilient to markup changes and the most meaningful for accessibility. + +Pass an object with a `role` key and an optional `name` key: + +```js +I.click({ role: 'button', name: 'Login' }); +I.fillField({ role: 'textbox', name: 'Email Address' }, 'user@test.com'); +I.seeElement({ role: 'button', name: 'Submit' }); +``` + +The `name` matches the element's accessible name — its visible text, `aria-label`, or `aria-labelledby` target. + +Common ARIA roles include `button`, `link`, `textbox`, `checkbox`, `radio`, `combobox`, `listbox`, `menuitem`, `tab`, `dialog`, `alert`, `heading`, and `navigation`. + +> ℹ ARIA locators are supported by Playwright and other helpers that implement accessibility tree queries. ## Locator Builder @@ -250,6 +270,31 @@ ID locators are best to select the exact semantic element in web and mobile test * `#user` or `{ id: 'user' }` finds element with id="user" * `~user` finds element with accessibility id "user" (in Mobile testing) or with `aria-label=user`. +* `{ aria: 'user' }` finds element with `aria-label="user"` in web testing. + +## Context + +Most CodeceptJS actions accept a **context** locator as a last parameter. The context narrows the search to a specific part of the page. + +```js +I.click('Edit', '.user-profile'); +I.fillField('Email', 'user@test.com', '#login-form'); +I.see('Error', '.alert'); +I.seeElement({ role: 'button', name: 'Delete' }, '.toolbar'); +``` + +Use context when the same element appears in multiple places on a page: + +```js +// click Delete only inside the toolbar, not the sidebar +I.click({ role: 'button', name: 'Delete' }, '.toolbar'); + +// fill Email in login form, not registration form +I.fillField('Email', 'user@test.com', '#login-form'); +I.fillField('Email', 'admin@test.com', '#registration-form'); +``` + +The context accepts any locator type: CSS, XPath, strict object, or `locate()` builder result. ## Custom Locators diff --git a/docs/mobile-react-native-locators.md b/docs/mobile-react-native-locators.md deleted file mode 100644 index f0df4ba48..000000000 --- a/docs/mobile-react-native-locators.md +++ /dev/null @@ -1,67 +0,0 @@ -## Automating React Native apps - -### Problem - -> ⚠️ **NOTE**: This problem is not actual starting from `react-native@0.65.x` [CHANGELOG](https://github.com/react-native-community/releases/blob/master/CHANGELOG.md#android-specific-9), [#381fb3](https://github.com/facebook/react-native/commit/381fb395ad9d2d48717a5d082aaedbecdd804554) - -Let's say we have a React Native app with component defined like this -```html - -``` - -If you will try to execute a simple test like -```js -I.tap('~someButton') -``` -it will work correctly on iOS (with XUITest driver), but on Android's UIAutomator2 it will give you an error -``` -Touch actions like "tap" need at least some kind of position information like "element", "x" or "y" options, you've none given. -``` - -This happens because of the way React Native implements `testID` for Android, it puts provided value into the [View tag](https://developer.android.com/reference/android/view/View#tags), -as could be found in the [source code](https://github.com/facebook/react-native/blob/19a88d7f4addcd9f95fd4908d50db37b3604b5b1/ReactAndroid/src/main/java/com/facebook/react/uimanager/BaseViewManager.java#L114). -But UIAutomator doesn't have any means to work with view tags. - -### Solutions -As many resources suggest (like [here](https://github.com/appium/appium/issues/6025#issuecomment-406141946) or [here](https://github.com/facebook/react-native/issues/7135)), -you could use `testID` for iOS and `accesibilityLabel` for Android, but `accessibilityLabel` is a user-facing text that's intended for ... accesibility, -not for UI tests, and you will need to switch it off for production builds. - -Another way to solve this issue is to use Espresso driver for Android. -At first you need to enable Espresso driver for your Android configuration. -You could do it just by changing `automationName` in the `helpers` section of the config file: -```js -{ - //... - helpers: { - Appium: { - app: '/path/to/apk.apk', - platform: 'Android', - desiredCapabilities: { - automationName: 'Espresso', - platformVersion: '9', - deviceName: 'Android Emulator' - } - } - } - //... -} -``` -Then you could locate components using XPath expression: -```js -I.tap({android: '//*[@view-tag="someButton"]', ios: '~someButton'}) -``` -This way test would work for both platforms without any changes in code. -To simplify things further you could write a helper function: -```js -function tid(id) { - return { - android: `//*[@view-tag="${id}"]`, - ios: '~' + id - } -} -``` -Now test will look more concise -```js -I.tap(tid('someButton')); -``` diff --git a/docs/retry.md b/docs/retry.md index c1b1b4ceb..5de5f0622 100644 --- a/docs/retry.md +++ b/docs/retry.md @@ -1,370 +1,274 @@ +--- +permalink: /retry +title: Retry Mechanisms +--- + # Retry Mechanisms -CodeceptJS provides flexible retry mechanisms to handle flaky tests at different levels. +CodeceptJS provides flexible retry mechanisms to handle flaky tests. Use retries when dealing with unstable environments, network delays, or timing issues — not to mask bugs in your code. + +## Helper Retries -## Overview +Browser automation helpers (Playwright, Puppeteer, WebDriver) have **built-in retry mechanisms** for element interactions. When you call `I.click('Button')`, Playwright automatically waits for the element to exist, be visible, stable, and enabled — retrying for up to 5 seconds. -CodeceptJS supports retries at **four levels** with a **priority system** to prevent conflicts: +Configure the timeout in your helper settings: + +```js +helpers: { + Playwright: { + timeout: 5000, // retry actions for up to 5 seconds + waitForAction: 100 // wait 100ms before each action + } +} +``` -| Priority | Level | Value | Description | -|----------|-------|-------|-------------| -| **Highest** | Manual Step (`I.retry()`) | 100 | Explicit retries in test code | -| | Step Plugin (`retryFailedStep`) | 50 | Automatic step-level retries | -| | Scenario Config | 30 | Retry entire scenarios | -| | Feature Config | 20 | Retry all scenarios in feature | -| **Lowest** | Hook Config | 10 | Retry failed hooks | +**Learn more:** [Playwright Helper](/helpers/Playwright), [Timeouts](/timeouts) -**Rule:** Higher priority retries cannot be overwritten by lower priority ones. +## CodeceptJS Retry Levels -## Step-Level Retries +When helper retries aren't enough, CodeceptJS adds retry layers on top. -### Manual Retry: `I.retry()` +### 1. Manual Step Retry -Retry specific steps in your tests: +Retry a specific step known to be flaky: ```js -// Retry up to 5 times -I.retry().click('Submit') +import step from 'codeceptjs/steps' -// Custom options -I.retry({ - retries: 3, - minTimeout: 1000, // 1 second - maxTimeout: 5000, // 5 seconds -}).see('Welcome') +Scenario('checkout', ({ I }) => { + I.amOnPage('/cart') + I.click('Proceed to Checkout', step.retry(5)) // retry up to 5 times + I.see('Payment') +}) +``` -// Infinite retries -I.retry(0).waitForElement('Dashboard') +Configure timing with exponential backoff: + +```js +I.click('Submit', step.retry({ + retries: 3, + minTimeout: 1000, // wait 1 second before first retry + maxTimeout: 5000, // max 5 seconds between retries + factor: 1.5 // exponential backoff multiplier +})) ``` -### Automatic Retry: `retryFailedStep` Plugin +Pass `0` for infinite retries. -Automatically retry all failed steps without modifying test code. +### 2. Automatic Step Retry -**Basic configuration:** +Automatically retry all failed steps without modifying test code: ```js -// codecept.conf.js plugins: { retryFailedStep: { enabled: true, - retries: 3, + retries: 3 } } ``` -**Advanced options:** +Steps matching `amOnPage`, `wait*`, `send*`, `execute*`, `run*`, `have*` are skipped by default. + +When a scenario has its own retries, step retries are disabled by default (`deferToScenarioRetries: true`). This prevents excessive execution time: ```js -plugins: { - retryFailedStep: { - enabled: true, - retries: 3, - factor: 1.5, // exponential backoff factor - minTimeout: 1000, // 1 second before first retry - maxTimeout: 5000, // 5 seconds max between retries - - // Steps to ignore (never retry these) - ignoredSteps: [ - 'scroll*', // ignore all scroll steps - /Cookie/, // ignore by regexp - ], - - // Defer to scenario retries to prevent excessive retries (default: true) - deferToScenarioRetries: true, - } -} +Scenario('test', { retries: 2 }, ({ I }) => { + I.click('Button') // step retries disabled; scenario retries run instead +}) ``` -**Ignored steps by default:** `amOnPage`, `wait*`, `send*`, `execute*`, `run*`, `have*` - -**Disable per test:** +To disable step retries for a specific test: ```js -Scenario('test', { disableRetryFailedStep: true }, () => { - I.retry(5).click('Button') // Use manual retries instead +Scenario('manual retries only', { disableRetryFailedStep: true }, ({ I }) => { + I.click('Button', step.retry(5)) }) ``` -## Scenario-Level Retries +Full plugin options: -Configure retries for individual test scenarios. +| Option | Default | Description | +|--------|---------|-------------| +| `retries` | — | Retries per step | +| `minTimeout` | — | Milliseconds before first retry | +| `maxTimeout` | `Infinity` | Max milliseconds between retries | +| `factor` | — | Exponential backoff multiplier | +| `randomize` | `false` | Randomize timeout intervals | +| `ignoredSteps` | `[]` | Patterns/regex of steps to never retry | +| `deferToScenarioRetries` | `true` | Disable step retries when scenario retries exist | +| `when` | `() => true` | Function receiving error; return `true` to retry | -```js -// Simple: All scenarios retry 3 times -{ - retry: 3 -} +### 3. Multiple Steps Retry (retryTo) -// Advanced: By pattern -{ - retry: [ - { - Scenario: 2, - grep: 'Login', // Only scenarios containing "Login" - }, - { - Scenario: 5, - grep: 'API', - }, - ] -} +Retry a group of steps together as a single operation: -// In-code -Scenario('my test', { retries: 3 }, () => { - I.amOnPage('/') - I.click('Login') -}) +```js +import { retryTo } from 'codeceptjs/effects' + +await retryTo(() => { + I.click('Load More') + I.see('New Content') +}, 3) ``` -## Feature-Level Retries +If any step inside fails, the entire block retries. Use this for sequences that must succeed together — switching into an iframe and filling a form, for example. -Retry all scenarios within a feature file: +**Learn more:** [Effects](/effects#retryto) + +### 4. Self-Healing Steps + +When a step fails, a healing recipe runs recovery actions and continues the test — without touching test code. With AI healing enabled: ```js -{ - retry: [ - { - Feature: 3, - grep: 'Authentication', // Only features containing "Authentication" - }, - ] -} +Scenario('checkout', ({ I }) => { + I.click('Proceed to Checkout') + I.see('Payment') +}) ``` -## Hook-Level Retries +- `I.click('Proceed to Checkout')` fails — button was renamed or moved + - failed step, error message, and page HTML are sent to an LLM + - AI scans page elements and suggests valid replacement actions + - CodeceptJS executes the suggestions until one succeeds +- test continues with `I.see('Payment')` -Configure retries for failed hooks: +Run with `--ai` to activate: -```js -{ - retry: [ - { - BeforeSuite: 2, // Retry setup hook - Before: 1, // Retry test setup - After: 1, // Retry teardown - }, - ] -} +```bash +npx codeceptjs run --ai ``` -## Retry Coordination +You can also write custom recipes for non-UI failures — network errors, data glitches, UI migrations. -### How Different Retries Work Together +**Learn more:** [Self-Healing Tests](/heal), [AI Configuration](/ai) -When multiple retry mechanisms are configured, they work together based on priorities: +### 5. Scenario Retry -**Example 1: Step Plugin + Scenario Retries (default behavior)** +Retry an entire test when it fails: ```js -plugins: { - retryFailedStep: { - enabled: true, - retries: 3, - deferToScenarioRetries: true, // default - } -} - -Scenario('API test', { retries: 2 }, () => { - I.sendPostRequest('/api/users', { name: 'John' }) +Scenario('API integration', { retries: 3 }, ({ I }) => { + I.sendGetRequest('/api/users') + I.seeResponseCodeIs(200) }) ``` -**Result:** Step retries are **disabled**. Only scenario retries run (2 times). -**Total attempts:** 1 initial + 2 retries = **3 attempts** - -**Example 2: Step Plugin without Defer** +Retry all scenarios globally, or by grep pattern: ```js -plugins: { - retryFailedStep: { - enabled: true, - retries: 3, - deferToScenarioRetries: false, - } +export const config = { + retry: [ + { Scenario: 3, grep: 'API' }, // retry scenarios containing "API" 3 times + { Scenario: 5, grep: '@flaky' } // retry @flaky-tagged scenarios 5 times + ] } - -Scenario('API test', { retries: 2 }, () => { - I.sendPostRequest('/api/users', { name: 'John' }) - I.seeResponseCodeIs(200) -}) ``` -**Result:** Each step can retry 3 times, scenario can retry 2 times. -**⚠️ Warning:** Can lead to excessive execution time +### 6. Feature Retry -**Example 3: Manual Retry + Plugin** +Retry all scenarios within a feature: ```js -plugins: { - retryFailedStep: { - enabled: true, - retries: 3, - } -} +Feature('Payment Processing', { retries: 2 }) -Scenario('test', () => { - I.retry(5).click('Button') // Manual (priority 100) - I.click('AnotherButton') // Plugin (priority 50) -}) +Scenario('credit card payment', ({ I }) => { ... }) // retries 2 times +Scenario('paypal payment', ({ I }) => { ... }) // retries 2 times ``` -**Result:** -- First button: **5 retries** (manual takes precedence) -- Second button: **3 retries** (plugin) - -## Common Patterns - -### External API Flakiness +Or target features by pattern in config: ```js -{ +export const config = { retry: [ - { - Scenario: 3, - grep: 'API', - }, - ], - plugins: { - retryFailedStep: { - enabled: true, - deferToScenarioRetries: true, // Let scenario retries handle it - }, - }, + { Feature: 3, grep: 'Integration' } + ] } ``` -### UI Element Intermittent Visibility +### 7. Hook Retries + +Retry `Before`/`After` hooks when they fail: ```js -Scenario('form submission', () => { - I.amOnPage('/form') - I.fillField('email', 'test@example.com') +Before(({ I }) => { + I.amOnPage('/') +}).retry(2) +``` - // This specific button is sometimes not immediately clickable - I.retry(3).click('Submit') +Set per feature: - I.see('Success') +```js +Feature('My Suite', { + retryBefore: 2, + retryAfter: 1, + retryBeforeSuite: 3, + retryAfterSuite: 1 }) ``` -### Flaky Feature Suite +Or globally: ```js -{ +export const config = { retry: [ - { - Feature: 2, - grep: 'ThirdPartyIntegration', - }, - ], + { BeforeSuite: 2, Before: 1, After: 1 } + ] } ``` -## Best Practices - -1. **Use `deferToScenarioRetries: true`** (default) to avoid excessive retries -2. **Prefer scenario retries** over step retries for general flakiness -3. **Use manual `I.retry()`** for specific problematic steps -4. **Avoid combining** step plugin with scenario retries unless necessary -5. **Don't over-retry** - it can mask real bugs and slow down tests - -## Troubleshooting +## Retry Priority -### Tests Taking Too Long +When multiple retry configurations exist, higher-priority retries take precedence: -**Solutions:** -- Enable `deferToScenarioRetries: true` -- Reduce retry counts -- Use more specific retry patterns (grep) -- Fix the root cause instead of retrying +| Priority | Type | Description | +|----------|------|-------------| +| **Highest** | Manual Step (`step.retry()`) | Explicit retries in test code | +| | Automatic Step | `retryFailedStep` plugin | +| | Multiple Steps (`retryTo`) | Retry groups of steps together | +| | Scenario Config | Retry entire scenarios | +| | Feature Config | Retry all scenarios in a feature | +| **Lowest** | Hook Config | Retry failed hooks | -### Retries Not Working +`retryTo` operates independently from step-level retries. If a step inside `retryTo` fails, the entire block retries. -**Check:** -1. Verify configuration syntax -2. Check if higher priority retry is overriding -3. Ensure `disableRetryFailedStep: true` isn't set -4. Run with `DEBUG_RETRY_PLUGIN=1`: +## Best Practices -```bash -DEBUG_RETRY_PLUGIN=1 npx codeceptjs run -``` +1. **Understand helper retries first** — Playwright/Puppeteer/WebDriver already retry actions internally +2. **Start with scenario retries** — simpler and less likely to cause issues +3. **Use manual retries for known flaky steps** — most predictable behavior +4. **Enable `deferToScenarioRetries`** — prevents excessive retries (default) +5. **Don't over-retry** — if tests fail consistently, fix the root cause +6. **Use grep patterns** — apply retries only where needed +7. **Retry timeouts, not bugs** — retries handle environmental issues, not code defects +8. **Consider healing for complex recovery** — see [Self-Healing Tests](/heal) -### Too Many Retries +## Troubleshooting -**Solutions:** -1. Set `deferToScenarioRetries: true` -2. Add problematic steps to `ignoredSteps` -3. Use scenario retries instead of step retries -4. Add `when` condition to filter errors: +### Tests running too long -```js -plugins: { - retryFailedStep: { - enabled: true, - when: (err) => { - // Only retry network errors - return err.message.includes('ECONNREFUSED') - }, - } -} -``` +- Confirm `deferToScenarioRetries: true` (the default) +- Reduce retry counts +- Use `grep` patterns to target specific tests +- Add problematic steps to `ignoredSteps` -## Configuration Reference +### Retries not working -### Global Retry Options +1. Check configuration syntax +2. Check the priority table — a higher-priority retry may be overriding +3. Confirm `disableRetryFailedStep: true` is not set on the scenario +4. Confirm the step isn't in `ignoredSteps` -```js -// Simple -{ - retry: 3 -} +Debug with: -// Advanced -{ - retry: [ - { - Feature: 2, - grep: 'Auth', - }, - { - Scenario: 5, - grep: 'Payment', - }, - { - BeforeSuite: 3, - }, - ] -} +```bash +DEBUG_RETRY_PLUGIN=1 npx codeceptjs run ``` -### retryFailedStep Plugin Options - -| Option | Type | Default | Description | -|--------|------|---------|-------------| -| `retries` | number | `3` | Number of retries per step | -| `factor` | number | `1.5` | Exponential backoff factor | -| `minTimeout` | number | `1000` | Min milliseconds before first retry | -| `maxTimeout` | number | `Infinity` | Max milliseconds between retries | -| `randomize` | boolean | `false` | Randomize timeouts | -| `ignoredSteps` | array | `[]` | Additional steps to ignore | -| `deferToScenarioRetries` | boolean | `true` | Disable step retries when scenario retries exist | -| `when` | function | - | Custom condition (receives error) | +### `step.retry()` is undefined -### I.retry() Options +Import `step` from codeceptjs: ```js -I.retry({ - retries: 3, // number of retries (0 = infinite) - minTimeout: 1000, // milliseconds - maxTimeout: 5000, // milliseconds - factor: 1.5, // exponential backoff -}) +import step from 'codeceptjs/steps' ``` - -## Related - -- [Plugins](plugins.md) - Plugin system overview -- [Configuration](configuration.md) - Full configuration reference -- [Hooks](hooks.md) - Test hooks and lifecycle diff --git a/docs/sessions.md b/docs/sessions.md new file mode 100644 index 000000000..0f5eb5702 --- /dev/null +++ b/docs/sessions.md @@ -0,0 +1,80 @@ +--- +permalink: /sessions +title: Multiple Sessions +--- + +# Multiple Sessions + +CodeceptJS can run several browser sessions in a single test. This is useful for testing real-time features like chat, notifications, or multi-user workflows. + +```js +Scenario('two users chat', ({ I }) => { + I.amOnPage('/chat') + I.fillField('name', 'davert') + I.click('Sign In') + I.see('Hello, davert') + + session('john', () => { + I.amOnPage('/chat') + I.fillField('name', 'john') + I.click('Sign In') + I.see('Hello, john') + }) + + // back in default session (davert) + I.fillField('message', 'Hi, john') + I.see('me: Hi, john', '.messages') + + session('john', () => { + I.see('davert: Hi, john', '.messages') + }) +}) +``` + +Switch back to a session at any point by using the same name. + +## Session Configuration + +Override the browser config for a specific session: + +```js +session('mobile', { browser: 'webkit', viewport: { width: 375, height: 812 } }, () => { + I.amOnPage('/') + I.see('Mobile View') +}) +``` + +## Opening Sessions Without Switching + +Open sessions in the background without switching to them immediately: + +```js +Scenario('multi-user test', ({ I }) => { + session('john') + session('mary') + session('jane') + + I.amOnPage('/') + + session('mary', () => { + I.amOnPage('/login') + }) +}) +``` + +## Returning Values + +`session` can return a value for use in the main scenario: + +```js +const profileName = await session('john', () => { + I.amOnPage('/profile') + return I.grabTextFrom({ css: 'h1' }) +}) +I.fillField('Recipient', profileName) +``` + +## Notes + +- Sessions can use the `I` object, page objects, and any other objects declared for the scenario +- `within` can be used inside a session, but `session` cannot be called from inside `within` diff --git a/docs/test-structure.md b/docs/test-structure.md new file mode 100644 index 000000000..69a6e9b4c --- /dev/null +++ b/docs/test-structure.md @@ -0,0 +1,275 @@ +--- +permalink: /test-structure +title: Test Structure +--- + +# Test Structure + +A CodeceptJS test file contains one **Feature** (suite) and one or more **Scenarios** (tests). + +```js +Feature('User Authentication') + +Scenario('user logs in', ({ I }) => { + I.amOnPage('/login') + I.fillField('Email', 'user@example.com') + I.fillField('Password', secret('123456')) + I.click('Sign In') + I.see('Welcome') +}) +``` + +## Feature + +`Feature(title, config?)` declares a suite. Each test file contains exactly one Feature. + +```js +Feature('User Authentication') +``` + +An optional config object sets defaults for all scenarios: + +```js +Feature('Payment Processing', { + retries: 2, + timeout: 30000 +}) +``` + +Available options: + +- `retries` — number of times to retry failed scenarios before marking as failed (see [Retry Mechanisms](/retry)) +- `timeout` — maximum time in milliseconds for each scenario to complete (see [Timeouts](/advanced#timeout)) +- `retryBefore` — number of times to retry the Before hook if it fails +- `retryAfter` — number of times to retry the After hook if it fails +- `retryBeforeSuite` — number of times to retry the BeforeSuite hook if it fails +- `retryAfterSuite` — number of times to retry the AfterSuite hook if it fails + +> Unlike Mocha/Jest, nesting suites is not allowed — each file maps to exactly one feature. + +## Scenario + +`Scenario(title, config?, fn)` declares a test. The function receives an object with `I` (the actor), `test` object, and any page objects declared in `include` config: + +```js +Scenario('guest checkout', ({ I, checkoutPage }) => { + checkoutPage.open() + I.see('Order Summary') +}) +``` + +Access the `test` object to store metadata and artifacts for custom reporting: + +```js +Scenario('payment processing', ({ I, test }) => { + test.meta.transactionId = '12345' + test.artifacts.receipt = 'receipts/order-12345.pdf' + I.amOnPage('/checkout') +}) +``` + +Available properties: + +- `test.title` — test name +- `test.tags` — extracted tags from test name (e.g., `@smoke`, `@critical`) +- `test.steps` — array of executed steps +- `test.artifacts` — store screenshots, videos, logs, or files +- `test.meta` — custom metadata for reporters +- `test.notes` — array for adding notes or annotations +- `test.file` — path to test file +- `test.state` — current state (pending, passed, failed) +- `test.duration` — execution time in milliseconds +- `test.fullTitle()` — full title including suite name + +An optional config object can customize the scenario: + +```js +Scenario('slow test', { + timeout: 60000, + retries: 3 +}, ({ I }) => { + // ... +}) +``` + +Available options: + +- `timeout` — maximum time in milliseconds for scenario to complete (see [Timeouts](/timeouts)) +- `retries` — number of times to retry the scenario if it fails (see [Retry Mechanisms](/retry)) +- `meta` — metadata object with key-value pairs for reporting or filtering +- `[helperName]` — helper-specific configuration (e.g., `Playwright: { headless: false }`) +- `cookies` — pre-loaded cookies for authentication (used by auth plugin) +- `user` — user identifier for session management (used by auth plugin) +- `disableRetryFailedStep` — disable automatic step retries for this scenario + +### Dynamic Configuration + +Override helper config for a single scenario using `.config()`: + +```js +Scenario('run in firefox', ({ I }) => { + // ... +}).config({ browser: 'firefox' }) +``` + +To target a specific helper, pass its name as the first argument: + +```js +Scenario('use v2 API', ({ I }) => { + // ... +}).config('REST', { endpoint: 'https://api.mysite.com/v2' }) +``` + +Pass a function to derive config from the test object — useful for cloud providers: + +```js +Scenario('report to BrowserStack', ({ I }) => { + // ... +}).config((test) => ({ + desiredCapabilities: { + project: test.suite.title, + name: test.title, + } +})) +``` + +Apply config to all scenarios in a suite via `Feature`: + +```js +Feature('Admin Panel').config({ url: 'https://mysite.com/admin' }) +``` + +Config changes are reverted after the test or suite completes. Some options — such as `browser` when `restart: false` — cannot be changed after the browser has started. + +### Data-Driven Scenarios + +Use `Data().Scenario` to run the same scenario with multiple datasets: + +```js +const users = new DataTable(['role', 'email']) +users.add(['admin', 'admin@example.com']) +users.add(['user', 'user@example.com']) + +Data(users).Scenario('user can log in', ({ I, current }) => { + I.fillField('Email', current.email) + I.click('Login') + I.see(`Logged in as ${current.role}`) +}) +``` + +> ▶ See [Data Driven Tests](/data) for more details. + +### Tags + +Append a tag to the scenario title: + +```js +Scenario('update user profile @slow', ...) +``` + +Or use the `tag()` method: + +```js +Scenario('update user profile', ({ I }) => { + // ... +}).tag('@slow').tag('important') +``` + +Run tagged tests with `--grep`: + +```sh +npx codeceptjs run --grep '@slow' +``` + +Use regex for complex filtering: + +```sh +# both @smoke2 and @smoke3 +npx codeceptjs run --grep '(?=.*@smoke2)(?=.*@smoke3)' +# @smoke2 or @smoke3 +npx codeceptjs run --grep '@smoke2|@smoke3' +# all except @smoke4 +npx codeceptjs run --grep '(?=.*)^(?!.*@smoke4)' +``` + +### Skipping & Focusing + +```js +xScenario('skipped test', ...) // skip +Scenario.skip('skipped test', ...) // skip +Scenario.only('focused test', ...) // run only this test + +xFeature('Skipped Suite') // skip entire file +Feature.skip('Skipped Suite') // skip entire file +Feature.only('Run Only This Suite') // focus entire file +``` + +### Todo Scenarios + +Mark scenarios as planned but not yet implemented: + +```js +Scenario.todo('user can reset password') + +Scenario.todo('user can change avatar', ({ I }) => { + /** + * 1. Open profile settings + * 2. Upload new avatar + * Result: avatar is updated + */ +}) +``` + +## Hooks + +### Before / After + +Run code before or after **each scenario** in the file: + +```js +Before(({ I }) => { + I.amOnPage('/') +}) + +After(({ I }) => { + I.clearCookie() +}) +``` + +These are equivalent to `beforeEach` / `afterEach` in Mocha/Jest. + +Hooks can be retried on failure: + +```js +Before(({ I }) => { + I.amOnPage('/dashboard') +}).retry(3) +``` + +### BeforeSuite / AfterSuite + +Run code once before or after **all scenarios** in the file — equivalent to `beforeAll` / `afterAll`: + +```js +BeforeSuite(async ({ I }) => { + // seed test data before any scenario runs + await I.executeScript(() => window.resetDatabase()) +}) + +AfterSuite(async ({ I }) => { + await I.executeScript(() => window.cleanupDatabase()) +}) +``` + +> **Note:** The browser is available in `BeforeSuite` when using Playwright or Puppeteer helpers. + +Hooks can also be configured at Feature level: + +```js +Feature('My Suite', { + retryBefore: 3, + retryBeforeSuite: 2, + retryAfter: 1, + retryAfterSuite: 3, +}) +``` diff --git a/docs/timeouts.md b/docs/timeouts.md new file mode 100644 index 000000000..d0cf605b6 --- /dev/null +++ b/docs/timeouts.md @@ -0,0 +1,183 @@ +--- +permalink: /timeouts +title: Timeouts +--- + +# Timeouts + +CodeceptJS provides multiple timeout configurations to control test execution time at different levels. + +## Step Timeout + +Set timeout for individual steps using `step.timeout()`: + +```js +import step from 'codeceptjs/steps' + +Scenario('test with step timeouts', ({ I }) => { + I.amOnPage('/') + I.click('Slow Button', step.timeout(30)) + I.fillField('Email', 'user@example.com', step.timeout(10)) +}) +``` + +## Scenario Timeout + +Override timeout for specific scenarios: + +```js +Scenario('quick test', { timeout: 5 }, ({ I }) => { + I.amOnPage('/') +}) + +Scenario('slow test', { timeout: 120 }, ({ I }) => { + I.amOnPage('/dashboard') +}) +``` + +## Feature Timeout + +Set timeout for all scenarios in a feature: + +```js +Feature('Slow Operations', { timeout: 60 }) + +Scenario('first test', ({ I }) => { + // has 60 second timeout +}) + +Scenario('second test', ({ I }) => { + // has 60 second timeout +}) +``` + +## Global Timeout + +Set default timeout for all tests in `codecept.conf.js`: + +```js +export const config = { + timeout: 10 // 10 seconds for all tests +} +``` + +Use advanced configuration with grep patterns for conditional timeouts: + +```js +export const config = { + timeout: [ + 10, // default timeout + { + grep: '@slow', + Feature: 60 // 60 seconds for features tagged @slow + }, + { + grep: /critical/, + Scenario: 30 // 30 seconds for scenarios matching pattern + } + ] +} +``` + +## Helper Timeouts + +Each helper (Playwright, Puppeteer, WebDriver, Appium, REST) has its own timeout settings for browser actions, API requests, and element interactions. These are configured separately from test timeouts and are measured in milliseconds. + +Refer to helper-specific documentation for timeout configuration: + +- [Playwright Helper](/helpers/Playwright) — `timeout`, `waitForTimeout`, `getPageTimeout` +- [Puppeteer Helper](/helpers/Puppeteer) — `waitForTimeout`, `getPageTimeout` +- [WebDriver Helper](/helpers/WebDriver) — `waitForTimeout`, `smartWait`, `timeouts` object +- [Appium Helper](/helpers/Appium) — `timeouts` object +- [REST Helper](/helpers/REST) — `timeout` + +## stepTimeout Plugin + +Automatically apply timeouts to all steps: + +```js +plugins: { + stepTimeout: { + enabled: true, + timeout: 150, // default step timeout in seconds + overrideStepLimits: false, // override step.timeout() if true + noTimeoutSteps: [ + 'amOnPage', + 'wait*' // wildcard patterns supported + ], + customTimeoutSteps: [ + ['slowAction*', 30], // 30 seconds for matching steps + [/critical.*/, 60] // regex patterns supported + ] + } +} +``` + +### Timeout Priority + +When multiple timeouts are configured, CodeceptJS applies them in priority order: + +1. **stepTimeoutHard** — plugin with `overrideStepLimits: true` +2. **codeLimitTime** — `step.timeout()` or `I.limitTime()` +3. **stepTimeoutSoft** — plugin with `overrideStepLimits: false` +4. **testOrSuite** — global test/suite timeout + +Higher priority timeouts override lower priority ones. + +## Disabling Timeouts + +Disable all timeouts for debugging: + +```bash +npx codeceptjs run --no-timeouts +``` + +## Complete Example + +```js +// codecept.conf.js +export const config = { + timeout: [ + 10, + { grep: '@slow', Feature: 60 }, + { grep: '@critical', Scenario: 30 } + ], + + helpers: { + Playwright: { + timeout: 5000, + waitForTimeout: 1000, + getPageTimeout: 30000 + } + }, + + plugins: { + stepTimeout: { + enabled: true, + timeout: 150, + noTimeoutSteps: ['amOnPage', 'wait*'], + customTimeoutSteps: [ + ['slowAction*', 30] + ] + } + } +} +``` + +```js +// test file +import step from 'codeceptjs/steps' + +Feature('User Management @slow', { timeout: 60 }) + +Scenario('create user', { timeout: 20 }, ({ I }) => { + I.amOnPage('/users') + I.click('Create', step.timeout(10)) + I.fillField('Name', 'John', step.timeout(5).retry(2)) +}) +``` + +## Timeout Units + +- **Global, Feature, Scenario, Step, stepTimeout plugin** — seconds +- **Helper timeouts** — milliseconds diff --git a/docs/ui.md b/docs/ui.md deleted file mode 100644 index d091c27fa..000000000 --- a/docs/ui.md +++ /dev/null @@ -1,59 +0,0 @@ ---- -title: CodeceptUI -permalink: /ui ---- - - -CodeceptUI - - -## CodeceptUI - -CodeceptJS has an interactive, graphical test runner. We call it CodeceptUI. It works in your browser and helps you to manage your tests. - -CodeceptUI can be used for - -* running tests by groups or single -* get test reports -* review tests -* edit tests and page objects -* write new tests -* reuse one browser session accross multiple test runs -* easily switch to headless/headful mode - - -![](https://user-images.githubusercontent.com/220264/93860826-4d5fbc80-fcc8-11ea-99dc-af816f3db466.png) - -## Installation - -CodeceptUI is already installed with `create-codeceptjs` command but you can install it manually via: - -``` -npm i @codeceptjs/ui --save -``` - -## Usage - -To start using CodeceptUI you need to have CodeceptJS project with a few tests written. -If CodeceptUI was installed by `create-codecept` command it can be started with: - -``` -npm run codeceptjs:ui -``` - -CodeceptUI can be started in two modes: - -* **Application** mode - starts Electron application in a window. Designed for desktop systems. -* **Server** mode - starts a webserver. Deigned for CI systems. - -To start CodeceptUI in application mode: - -``` -npx codecept-ui --app -``` - -To start CodeceptUI in server mode: - -``` -npx codecept-ui -``` diff --git a/docs/videos.md b/docs/videos.md deleted file mode 100644 index 1835f5412..000000000 --- a/docs/videos.md +++ /dev/null @@ -1,28 +0,0 @@ ---- -permalink: /videos -layout: Section -sidebar: false -title: Videos -editLink: false ---- - -> Add your own videos to our [Wiki Page](https://github.com/codeceptjs/CodeceptJS/wiki/Videos) -[![](http://i3.ytimg.com/vi/BRMWstiOTks/maxresdefault.jpg)](https://www.youtube.com/watch?v=BRMWstiOTks) - -## [An Introduction, Getting started and working with CodeceptJS & Puppeteer (EAWeekend)](https://www.youtube.com/watch?v=BRMWstiOTks) - -## [CodeceptJS Official YouTube Channel](https://www.youtube.com/channel/UCEs4030bmtonyDhTHEXa_2g) - -## [Introductory Videos](https://www.youtube.com/watch?v=FPFG1rBNJ64&list=PLcFXthgti9Lt4SjSvL1ALDg6dOeTC0TvT) - -Free educational videos provided by our community member **[@ontytoom](http://github.com/ontytoom)**. - -1. [Installation](https://www.youtube.com/watch?v=FPFG1rBNJ64) -1. [Creating a Test](https://www.youtube.com/watch?v=mdQZjL3h9d0) -1. [Using Page Objects](https://www.youtube.com/watch?v=s677_6VctjQ) - -## [Practical E2E Testing with CodeceptJS](https://www.udemy.com/practical-e2e-testing-with-codeceptjs/) - -Udemy course by Luke Beilharz - - diff --git a/docs/visual.md b/docs/visual.md deleted file mode 100644 index 5663dcc56..000000000 --- a/docs/visual.md +++ /dev/null @@ -1,202 +0,0 @@ ---- -permalink: /visual -title: Visual Testing ---- - -# Visual Testing - -How does one test if the UI being rendered appears correctly to the users or how to test if each UI element appears in the right position and size? The traditional way to test the UI of the application has always been manually, which is time consuming. - -Visual testing with help of CodeceptJS will help in improving such use cases for the QA folks. - -By default CodeceptJS uses [WebDriver](/helpers/WebDriver/) helper and **Selenium** to automate browser. It is also capable of taking screenshots of the application and this could be used for visual testing. - -Currently there are two helpers available for Visual testing with CodeceptJS - -## Using Resemble helper - -[Resemble.js](https://github.com/rsmbl/Resemble.js) is a great tool for image comparison and analysis, which can be used with CodeceptJS - -### Setup - -To install the package, just run - -``` -npm install codeceptjs-resemblehelper --save -``` - -### Configuring - -This helper should be added to `codecept.conf.js` config file. - -Example: - -```json -{ - "helpers": { - "ResembleHelper" : { - "require": "codeceptjs-resemblehelper", - "screenshotFolder" : "./tests/output/", - "baseFolder": "./tests/screenshots/base/", - "diffFolder": "./tests/screenshots/diff/" - } - } -} -``` - -To use the Helper, users must provide the three parameters: - -* `screenshotFolder` : This will always have the same value as `output` in Codecept configuration, this is the folder where WebDriver saves a screenshot when using `I.saveScreenshot` method -* `baseFolder`: This is the folder for base images, which will be used with screenshot for comparison -* `diffFolder`: This will the folder where resemble would try to store the difference image, which can be viewed later. - -### Usage - -Details about the helper can be found on the [Github Repo](https://github.com/puneet0191/codeceptjs-resemblehelper) - -Base Image is compared with the screenshot image and test results are derived based on the `mismatch tolerance` level provided by the user for the comparison - -### Example - -Lets consider visual testing for [CodeceptJS Home](https://codecept.io) - -```js -Feature('To test screen comparison with resemble Js Example test'); - -Scenario('Compare CodeceptIO Home Page @visual-test', async ({ I }) => { - I.amOnPage("/"); - I.saveScreenshot("Codecept_IO_Screenshot_Image.png"); - I.seeVisualDiff("Codecept_IO_Screenshot_Image.png", {tolerance: 2, prepareBaseImage: false}); -}); -``` - -In this example, we are setting the expected mismatch tolerance level as `2` - -`Base Image` (Generated by User) -![Base Image](/img/Codecept_IO_Base_Image.png) - -`Screenshot Image` (Generated by Test) -![Screenshot Image](/img/Codecept_IO_Screenshot_Image.png) - -Clearly the difference in both the images visible to human eye is the section about `Scenario Driven` - -![Difference Image](/img/difference_Image_Codecept_Home.png) - -`Diff Image` generated by the helper clearly highlights the section which don't match - -![Highlight](/img/Difference%20Image%20Focus.png) - -`Failed Test output` -``` -To test screen comparison with resemble Js Example test -- - Compare CodeceptIO Home Page @visual-test - I see Visual Diff "Codecept_IO_Screenshot_Image.png", {tolerance: 2, prepareBaseImage: false} -MisMatch Percentage Calculated is 2.85 - ✖ FAILED in 418ms - - --- FAILURES: - - 1) To test screen comparison with resemble Js Example test - Compare CodeceptIO Home Page @visual-test: - - MissMatch Percentage 2.85 - + expected - actual - - -false - +true -``` - -`Codeceptjs-resemblehelper` basically comes with two major functions - -1) `seeVisualDiff` which can be used to compare two images and calculate the misMatch percentage. -2) `seeVisualDiffForElement` which can be used to compare elements on the two images and calculate misMatch percentage. - -## Using Applitools - -Applitools helps Test Automation engineers, DevOps, and FrontEnd Developers continuously test and validate visually perfect mobile, web, and native apps. Now it can be used with CodeceptJS. - -### Setup - -Create an account at [Applitools](https://applitools.com/users/register) and install the npm packages - -``` -npm i codeceptjs-applitoolshelper --save -npm i webdriverio@5 --save -``` - -### Configuring - -```js -... - helpers: { - WebDriver: { - url: 'https://applitools.com/helloworld', - browser: 'chrome', - desiredCapabilities: { - chromeOptions: { - args: [ '--headless', '--disable-extensions', '--disable-gpu', '--no-sandbox', '--disable-dev-shm-usage'] - } - }, - windowSize: '1920x600', - smartWait: 5000, - timeouts: { - 'script': 60000, - 'page load': 10000 - }, - }, - ApplitoolsHelper: { - require: 'codeceptjs-applitoolshelper', - applitoolsKey: 'YOUR_API_KEY' - } - }, -... -``` - -#### To use this helper you need to provide the following info: - -- applitoolsKey (Required): You can find your API key under the user menu located at the right hand side of the test manager toolbar -- windowSize (Optional): the windows size as for instance 1440x700, if not provided, the default 1920x600 will be used. The windowSize will follow this precedence: ApplitoolsHelper, Webdriver. -- appName (Optional): you can either provide your desired application name, if not provided, the default 'Application Under Test' will be used. - -### Usage - -```javascript -/** - * @param pageToCheck {string} Name of page you want to check - */ -I.eyeCheck(pageToCheck); - -``` - -The first time you run this test a new baseline will be created, and subsequent test runs will be compared to this baseline. If any screenshot mismatch its baseline image in a perceptible way, there will be a `DiffsFoundException` which includes a URL that points to a detailed report where you can see the detected differences and take appropriate actions such as reporting bugs, updating the baseline and more. - -```js --- FAILURES: - - 1) Applitools functionality - Check home page @test: - Test 'Applitools functionality' of 'Application Under Test' detected differences!. See details at: https://eyes.applitools.com/app/batches/00000251831777088983/00000251831777088717?accountId=KO-Oh9tXI0e8VF8Ha_GLVA~~ -``` - -> You can find the latest documentation here [Applitools Docs](https://applitools.com/tutorials/webdriverio5.html#run-your-first-test) - -### Example - -Lets consider visual testing for [CodeceptJS Home](https://codecept.io). -You can also find example repo here: https://github.com/PeterNgTr/codeceptjs-applitoolshelper - -```js -const { I } = inject(); - -Feature('Applitools functionality'); - -Before(() => { - I.amOnPage('https://applitools.com/helloworld'); -}); - -Scenario('Check home page @test', async ({ }) => { - await I.eyeCheck('Homepage'); -}); -``` - diff --git a/docs/vue.md b/docs/vue.md deleted file mode 100644 index e09f921f2..000000000 --- a/docs/vue.md +++ /dev/null @@ -1,143 +0,0 @@ ---- -permalink: /vue -layout: Section -sidebar: false -title: Testing Vue Apps ---- - - -# vue-cli-plugin-e2e-codeceptjs - -*Hey, how about some end 2 end testing for your Vue apps?* 🤔 - -*Let's do it together! Vue, me, [CodeceptJS](https://codecept.io) & [Puppeteer](https://pptr.dev).* 🤗 - -*Browser testing was never that simple. Just see it!* 😍 - -```js -I.amOnPage('/'); -I.click('My Component Button'); -I.see('My Component'); -I.say('I am happy!'); -// that's right, this is a valid test! -``` - -## How to try it? - -**Requirements:** - -* NodeJS >= 8.9 -* NPM / Yarn -* Vue CLI installed globally - -``` -npm i vue-cli-plugin-codeceptjs-puppeteer --save-dev -``` - -This will install CodeceptJS, CodeceptUI & Puppeteer with Chrome browser. - -To add CodeceptJS to your project invoke installer: - -``` -vue invoke vue-cli-plugin-codeceptjs-puppeteer -``` - -> You will be asked about installing a demo component. If you start a fresh project **it is recommended to agree and install a demo component**, so you could see tests passing. - - -## Running Tests - -We added npm scripts: - -* `test:e2e` - will execute tests with browser opened. If you installed test component, and started a test server, running this command will show you a brower window passed test. - * Use `--headless` option to run browser headlessly - * Use `--serve` option to start a dev server before tests - - -Examples: - -``` -npm run test:e2e -npm run test:e2e -- --headless -npm run test:e2e -- --serve -``` - -> This command is a wrapper for `codecept run --steps`. You can use the [Run arguments and options](/commands#run) here. - -* `test:e2e:parallel` - will execute tests headlessly in parallel processes (workers). By default runs tests in 2 workers. - * Use an argument to set number of workers - * Use `--serve` option to start dev server before running - -Examples: - -``` -npm run test:e2e:parallel -npm run test:e2e:parallel -- 3 -npm run test:e2e:parallel -- 3 --serve -``` - -> This command is a wrapper for `codecept run-workers 2`. You can use the [Run arguments and options](/commands#run-workers) here. - -* `test:e2e:open` - this opens interactive web test runner. So you could see, review & run your tests from a browser. - -![](https://user-images.githubusercontent.com/220264/70399222-b7a1bc00-1a2a-11ea-8f0b-2878b0328161.gif) - -``` -npm run test:e2e:open -``` - -## Directory Structure - -Generator has created these files: - -```js -codecept.conf.js 👈 codeceptjs config -jsconfig.json 👈 enabling type definitons -tests -├── e2e -│   ├── app_test.js 👈 demo test, edit it! -│   ├── output 👈 temp directory for screenshots, reports, etc -│   └── support -│   └── steps_file.js 👈 common steps -└── steps.d.ts 👈 type definitions -``` - -If you agreed to create a demo component, you will also see `TestMe` component in `src/components` folder. - -## Locators - -For Vue apps a special `vue` locator is available. It allows to select an element by its component name, and props. - -```js -{ vue: 'MyComponent' } -{ vue: 'Button', props: { title: 'Click Me' }} -``` - -With Playwright, you can use Vue locators in any method where locator is required: - -```js -I.click({ vue: 'Tab', props: { title: 'Click Me!' }}); -I.seeElement({ vue: 't', props: { title: 'Clicked' }}); -``` - -To find Vue element names and props in a tree use [Vue DevTools](https://chromewebstore.google.com/detail/vuejs-devtools/nhdogjmejiglipccpnnnanhbledajbpd) extension. - -> Turn off minification for application builds otherwise component names will be uglified as well - -Vue locators work via [Playwright Vue Locator](https://playwright.dev/docs/other-locators#vue-locator). - -## How to write tests? - -* Open `tests/e2e/app_js` and see the demo test -* Execute a test & use interactive pause to see how CodeceptJS works -* [Learn CodeceptJS basics](/basics) -* [Learn how to write CodeceptJS tests with Puppeteer](/puppeteer) -* [See full reference for CodeceptJS Puppeteer Helper](/helpers/Puppeteer) -* Ask your questions in [Slack](https://bit.ly/chat-codeceptjs) & [Forum](https://codecept.discourse.group/) - -## Enjoy testing! - -Testing is simple & fun, enjoy it! - -With ❤ [CodeceptJS Team](https://codecept.io) - diff --git a/docs/within.md b/docs/within.md new file mode 100644 index 000000000..6ce7513e5 --- /dev/null +++ b/docs/within.md @@ -0,0 +1,55 @@ +--- +permalink: /within +title: Within +--- + +# Within + +`within` scopes all actions inside it to a specific element on the page — useful when working with repeated UI components or narrowing interaction to a specific section. + +```js +within('.js-signup-form', () => { + I.fillField('user[login]', 'User') + I.fillField('user[email]', 'user@user.com') + I.fillField('user[password]', 'user@user.com') + I.click('button') +}) +I.see('There were problems creating your account.') +``` + +> ⚠ `within` can cause problems when used incorrectly. If you see unexpected behavior, refactor to use the context parameter on individual actions instead (e.g. `I.click('Login', '.nav')`). Keep `within` for the simplest cases. +> Since `within` returns a Promise, always `await` it when you need its return value. + +## IFrames + +Use a `frame` locator to scope actions inside an iframe: + +```js +within({ frame: '#editor' }, () => { + I.see('Page') + I.fillField('Body', 'Hello world') +}) +``` + +Nested iframes _(WebDriver & Puppeteer only)_: + +```js +within({ frame: ['.content', '#editor'] }, () => { + I.see('Page') +}) +``` + +> ℹ IFrames can also be accessed via `I.switchTo` command. + +## Returning Values + +`within` can return a value for use in the scenario: + +```js +const val = await within('#sidebar', () => { + return I.grabTextFrom({ css: 'h1' }) +}) +I.fillField('Description', val) +``` + +When running steps inside a `within` block, they will be shown indented in the output.