Skip to main content

Test Automation Frameworks and Its Types

· 10 min read
Divya Manohar
Divya Manohar
Co-Founder and CEO, DevAssure

An automation framework is a collection of multiple tools and libraries that are structured together to support automated testing of an application.

Advantages of an automation framwework​

  1. Less manual work, saves time, validates 100s of test cases in a shorter span
  2. Early bug detection
  3. Regression and sanity testing
  4. Improved test coverage
  5. Minimal manual intervention
  6. Ability to perform parallel testing
  7. Shift left testing
  8. Lower maintenance costs.
  9. Reusability
  10. Minimize cost, Maximize ROI

Framework design should be like building a product, you need to think through how it maps with your application under test, identify generic ways of handling the tools and libraries.

To be able to do that it's important to understand the different types of automation frameworks. This will help decide the basis of building the automation framework.

Types of Automation Frameworks

Linear Automation Framework​

This is the simplest form of any automation framework that involves record and playback. There is no modularity involved, and you would simply write the test code as you would record and keep it going.

Sample use case​

Consider the following scenario

  1. Login to https://www.saucedemo.com/
  2. Add 6 items to cart
  3. Validate the number of items in the cart
  4. Complete checkout and validate the total amount.

Automation code​

For this example, I've used playwright to record the entire use case. The code is written in the same flow as the use case would be tested manually. But when there are multiple use cases, the same steps will be added for different cases, code maintenance will be a challenge.

import { test, expect } from '@playwright/test';

test('test', async ({ page }) => {
  await page.goto('https://www.saucedemo.com/');
  await page.locator('[data-test="username"]').click();
  await page.locator('[data-test="username"]').fill('standard_user');
  await page.locator('[data-test="password"]').click();
  await page.locator('[data-test="password"]').fill('secret_sauce');
  await page.locator('[data-test="login-button"]').click();
  await page.locator('[data-test="add-to-cart-sauce-labs-backpack"]').click();
  await page.locator('[data-test="add-to-cart-sauce-labs-bike-light"]').click();
  await page.locator('[data-test="add-to-cart-sauce-labs-bolt-t-shirt"]').click();
  await page.locator('[data-test="add-to-cart-sauce-labs-fleece-jacket"]').click();
  await page.locator('[data-test="add-to-cart-sauce-labs-onesie"]').click();
  await page.locator('[data-test="add-to-cart-test\\.allthethings\\(\\)-t-shirt-\\(red\\)"]').click();
  await page.locator('[data-test="shopping-cart-link"]').click();
  await expect(page.locator('[data-test="shopping-cart-badge"]')).toContainText('6');
  await page.locator('[data-test="checkout"]').click();
  await page.locator('[data-test="firstName"]').click();
  await page.locator('[data-test="firstName"]').fill('Test');
  await page.locator('[data-test="lastName"]').click();
  await page.locator('[data-test="lastName"]').fill('Test');
  await page.locator('[data-test="postalCode"]').click();
  await page.locator('[data-test="postalCode"]').fill('600017');
  await page.locator('[data-test="continue"]').click();
  await expect(page.locator('[data-test="total-label"]')).toContainText('Total: $140.34');
  await page.locator('[data-test="finish"]').click();
  await expect(page.locator('[data-test="complete-header"]')).toContainText('Thank you for your order!');
});

Advantages​

  1. Simple and straightforward
  2. Works well for basic applications
  3. Minimal time to design and build the framework

Disadvantages​

  1. Cannot solve complex use cases
  2. Not modular
  3. Error prone
  4. Test code maintenance is challenging

Modular Driven Framework​

This is an enhancement to the linear framework, where in test steps repeated across multiple use cases will be extracted into multiple modules. In essence, you would be breaking down the interactions with the application into smaller modules.

Sample use case​

Consider the following scenario

  1. Login to https://www.saucedemo.com/
  2. Add 6 items to cart
  3. Validate the number of items in the cart
  4. Complete checkout and validate the total amount.

Automation code​

The same code from Linear Automation Framework, will be transformed into smaller modules.

  1. Login
  2. Add item to cart
  3. Checkout
  4. Validate

Actions

import { Page, expect } from '@playwright/test';
import selectors from './selectors';

export async function login(page: Page, username: string, password: string) {
  await page.goto('https://www.saucedemo.com/');
  await page.locator(selectors.username).fill(username);
  await page.locator(selectors.password).fill(password);
  await page.locator(selectors.loginButton).click();
}

export async function addItemToCart(page: Page, itemSelector: string) {
  await page.locator(itemSelector).click();
}

export async function checkout(page: Page, firstName: string, lastName: string, postalCode: string) {
  await page.locator(selectors.cartLink).click();
  await page.locator(selectors.checkoutButton).click();
  await page.locator(selectors.firstName).fill(firstName);
  await page.locator(selectors.lastName).fill(lastName);
  await page.locator(selectors.postalCode).fill(postalCode);
  await page.locator(selectors.continueButton).click();
}

export async function verifyOrder(page: Page) {
  await expect(page.locator(selectors.totalLabel)).toContainText('Total: $140.34');
  await page.locator(selectors.finishButton).click();
  await expect(page.locator(selectors.completeHeader)).toContainText('Thank you for your order!');
}

Selectors

Contains all the selectors in a single file for easy maintenance.

const selectors = {
    username: '[data-test="username"]',
    password: '[data-test="password"]',
    loginButton: '[data-test="login-button"]',
    addToCartBackpack: '[data-test="add-to-cart-sauce-labs-backpack"]',
    addToCartBikeLight: '[data-test="add-to-cart-sauce-labs-bike-light"]',
    addToCartBoltTshirt: '[data-test="add-to-cart-sauce-labs-bolt-t-shirt"]',
    addToCartFleeceJacket: '[data-test="add-to-cart-sauce-labs-fleece-jacket"]',
    addToCartOnesie: '[data-test="add-to-cart-sauce-labs-onesie"]',
    addToCartTestAllTheThingsTshirt: '[data-test="add-to-cart-test\\.allthethings\\(\\)-t-shirt-\\(red\\)"]',
    cartLink: '[data-test="shopping-cart-link"]',
    cartBadge: '[data-test="shopping-cart-badge"]',
    checkoutButton: '[data-test="checkout"]',
    firstName: '[data-test="firstName"]',
    lastName: '[data-test="lastName"]',
    postalCode: '[data-test="postalCode"]',
    continueButton: '[data-test="continue"]',
    totalLabel: '[data-test="total-label"]',
    finishButton: '[data-test="finish"]',
    completeHeader: '[data-test="complete-header"]',
  };
  
  export default selectors;

Test Steps

import { test, expect } from '@playwright/test';
import { login, addItemToCart, checkout, verifyOrder } from './actions';
import selectors from './selectors';

test('test', async ({ page }) => {
  await login(page, 'standard_user', 'secret_sauce');

  await addItemToCart(page, selectors.addToCartBackpack);
  await addItemToCart(page, selectors.addToCartBikeLight);
  await addItemToCart(page, selectors.addToCartBoltTshirt);
  await addItemToCart(page, selectors.addToCartFleeceJacket);
  await addItemToCart(page, selectors.addToCartOnesie);
  await addItemToCart(page, selectors.addToCartTestAllTheThingsTshirt);

  await page.locator(selectors.cartLink).click();
  await expect(page.locator(selectors.cartBadge)).toContainText('6');

  await checkout(page, 'Test', 'Test', '600017');

  await verifyOrder(page);
});

Advantages​

  1. Reusability
  2. When a test step changes, multiple tests will not be impacted.
  3. Ease of test maintenance

Disadvantages​

  1. Difficult to identify the modules whenever multiple engineers work together, leading to code duplication.

Behaviour Driven framework​

Tests in the framework are written in plain english which can be easily understood by technical and non-technical people. Devs, QA, PMs can equally contribute to automation.

Sample use case​

Consider the following scenario

  1. Login to https://www.saucedemo.com/
  2. Add 6 items to cart
  3. Validate the number of items in the cart
  4. Complete checkout and validate the total amount.

Automation code​

A feature file defines the various behaviours in the use case. But there is a need for a framework, that will translate the keywords used, to actual executable test scripts.

Feature file

Feature: Saucedemo Shopping Flow

  Scenario: User adds items to the cart and completes the checkout process
    Given the user is on the Saucedemo login page
    When the user logs in with username "standard_user" and password "secret_sauce"
    And the user adds items to the cart
    And the user goes to the cart page
    Then the cart should have 6 items
    When the user checks out
    And the user enters personal information
    Then the total should be "$140.34"
    When the user completes the order
    Then the order should be successful

Each of the keywords are mapped to the playwright scripts.

Spec file

import { setWorldConstructor, Before, After, IWorldOptions } from '@cucumber/cucumber';
import { Browser, chromium, Page } from '@playwright/test';
import { Given, When, Then } from '@cucumber/cucumber';
import { expect } from '@playwright/test';

class CustomWorld {
  browser: Browser;
  page: Page;

  constructor() {
    this.browser = null!;
    this.page = null!;
  }
}

setWorldConstructor(CustomWorld);

Before(async function (this: CustomWorld) {
  this.browser = await chromium.launch();
  this.page = await this.browser.newPage();
});

After(async function (this: CustomWorld) {
  await this.page.close();
  await this.browser.close();
});

Given('the user is on the Saucedemo login page', async function () {
  await this.page.goto('https://www.saucedemo.com/');
});
  
When('the user logs in with username {string} and password {string}', async function (username: string, password: string) {
  await this.page.locator('[data-test="username"]').fill(username);
  await this.page.locator('[data-test="password"]').fill(password);
  await this.page.locator('[data-test="login-button"]').click();
});
  
When('the user adds items to the cart', async function () {
  await this.page.locator('[data-test="add-to-cart-sauce-labs-backpack"]').click();
  await this.page.locator('[data-test="add-to-cart-sauce-labs-bike-light"]').click();
  await this.page.locator('[data-test="add-to-cart-sauce-labs-bolt-t-shirt"]').click();
  await this.page.locator('[data-test="add-to-cart-sauce-labs-fleece-jacket"]').click();
  await this.page.locator('[data-test="add-to-cart-sauce-labs-onesie"]').click();
  await this.page.locator('[data-test="add-to-cart-test\\.allthethings\\(\\)-t-shirt-\\(red\\)"]').click();
});
  
When('the user goes to the cart page', async function () {
  await this.page.locator('[data-test="shopping-cart-link"]').click();
});
  
Then('the cart should have {int} items', async function (itemCount: number) {
  await expect(this.page.locator('[data-test="shopping-cart-badge"]')).toContainText(String(itemCount));
});
  
When('the user checks out', async function () {
  await this.page.locator('[data-test="checkout"]').click();
});
  
When('the user enters personal information', async function () {
  await this.page.locator('[data-test="firstName"]').fill('Test');
  await this.page.locator('[data-test="lastName"]').fill('Test');
  await this.page.locator('[data-test="postalCode"]').fill('600017');
  await this.page.locator('[data-test="continue"]').click();
});
  
Then('the total should be {string}', async function (total: string) {
  await expect(this.page.locator('[data-test="total-label"]')).toContainText(total);
});
  
When('the user completes the order', async function () {
  await this.page.locator('[data-test="finish"]').click();
});
  
Then('the order should be successful', async function () {
  await expect(this.page.locator('[data-test="complete-header"]')).toContainText('Thank you for your order!');
});

Advantages​

  1. Ease of use & understanding
  2. Non technical people can also contribute to automation.
  3. Shorter learning curve.

Disadvantages​

  1. Potential for duplicate step definitions.
  2. Building and maintaining such a framework is complex.

Data-Driven Framework​

Tests in this framework depend on the test data. These frameworks are typical useful when you have multiple test data coming from different sources like files, database and APIs for a single test case.

Sample use case​

Consider the following scenario

  1. Login to https://www.saucedemo.com/
  2. Add 6 items to cart
  3. Validate the number of items in the cart
  4. Complete checkout and validate the total amount.

Automation code​

testData.json can contain multiple data sets, which means that the same automated test can be used to validate the flows for different data sets.

testData.json

[
    {
      "username": "standard_user",
      "password": "secret_sauce",
      "firstName": "Test",
      "lastName": "Test",
      "postalCode": "600017",
      "total": "Total: $140.34"
    }
]

Spec file

import { test, expect } from '@playwright/test';
import * as fs from 'fs';
import * as path from 'path';

const testDataPath = path.resolve(__dirname, 'testData.json');
const testData = JSON.parse(fs.readFileSync(testDataPath, 'utf-8'));

test.describe('Data Driven Tests', () => {
  testData.forEach(data => {
    test(`test with user: ${data.username}`, async ({ page }) => {
      await page.goto('https://www.saucedemo.com/');
      await page.locator('[data-test="username"]').click();
      await page.locator('[data-test="username"]').fill(data.username);
      await page.locator('[data-test="password"]').click();
      await page.locator('[data-test="password"]').fill(data.password);
      await page.locator('[data-test="login-button"]').click();
      await page.locator('[data-test="add-to-cart-sauce-labs-backpack"]').click();
      await page.locator('[data-test="add-to-cart-sauce-labs-bike-light"]').click();
      await page.locator('[data-test="add-to-cart-sauce-labs-bolt-t-shirt"]').click();
      await page.locator('[data-test="add-to-cart-sauce-labs-fleece-jacket"]').click();
      await page.locator('[data-test="add-to-cart-sauce-labs-onesie"]').click();
      await page.locator('[data-test="add-to-cart-test\\.allthethings\\(\\)-t-shirt-\\(red\\)"]').click();
      await page.locator('[data-test="shopping-cart-link"]').click();
      await expect(page.locator('[data-test="shopping-cart-badge"]')).toContainText('6');
      await page.locator('[data-test="checkout"]').click();
      await page.locator('[data-test="firstName"]').click();
      await page.locator('[data-test="firstName"]').fill(data.firstName);
      await page.locator('[data-test="lastName"]').click();
      await page.locator('[data-test="lastName"]').fill(data.lastName);
      await page.locator('[data-test="postalCode"]').click();
      await page.locator('[data-test="postalCode"]').fill(data.postalCode);
      await page.locator('[data-test="continue"]').click();
      await expect(page.locator('[data-test="total-label"]')).toContainText(data.total);
      await page.locator('[data-test="finish"]').click();
      await expect(page.locator('[data-test="complete-header"]')).toContainText('Thank you for your order!');
    });
  });
});

Advantages​

Avoid creating multiple scripts for different types of test data

Disadvantages​

Build and maintaining such a framework is complex

Keyword-Driven Framework​

This is an extension of the data driven approach. With this framework, you couple the test data and specific keywords stored in files to execute the tests. The keywords and test data are stored in a tabular format.

Sample use case​

Consider the following scenario

  1. Login to https://www.saucedemo.com/
  2. Add 6 items to cart
  3. Validate the number of items in the cart
  4. Complete checkout and validate the total amount.

Automation code​

keywords.ts

This file defines functions for each keyword, ensuring type safety with TypeScript.

import { Page, expect } from '@playwright/test';

export async function openUrl(page: Page, url: string) {
  await page.goto(url);
}

export async function clickElement(page: Page, selector: string) {
  await page.locator(selector).click();
}

export async function fillElement(page: Page, selector: string, text: string) {
  await page.locator(selector).fill(text);
}

export async function assertTextContains(page: Page, selector: string, expectedText: string) {
  await expect(page.locator(selector)).toContainText(expectedText);
}

testCases.csv

This CSV file contains the test steps using the keywords and selectors.

keyword,selector,value
openUrl,,https://www.saucedemo.com/
clickElement,[data-test="username"],
fillElement,[data-test="username"],standard_user
clickElement,[data-test="password"],
fillElement,[data-test="password"],secret_sauce
clickElement,[data-test="login-button"],
clickElement,[data-test="add-to-cart-sauce-labs-backpack"],
clickElement,[data-test="add-to-cart-sauce-labs-bike-light"],
clickElement,[data-test="add-to-cart-sauce-labs-bolt-t-shirt"],
clickElement,[data-test="add-to-cart-sauce-labs-fleece-jacket"],
clickElement,[data-test="add-to-cart-sauce-labs-onesie"],
clickElement,[data-test="add-to-cart-test\\.allthethings\\(\\)-t-shirt-\\(red\\)"],
clickElement,[data-test="shopping-cart-link"],
assertTextContains,[data-test="shopping-cart-badge"],6
clickElement,[data-test="checkout"],
clickElement,[data-test="firstName"],
fillElement,[data-test="firstName"],Test
clickElement,[data-test="lastName"],
fillElement,[data-test="lastName"],Test
clickElement,[data-test="postalCode"],
fillElement,[data-test="postalCode"],600017
clickElement,[data-test="continue"],
assertTextContains,[data-test="total-label"],Total: $140.34
clickElement,[data-test="finish"],
assertTextContains,[data-test="complete-header"],Thank you for your order!

Spec file

This script reads the test steps from the CSV file and executes them using the functions defined in keywords.ts.


import { test, Page } from '@playwright/test';
import fs from 'fs';
import csvParser from 'csv-parser';
import * as keywords from './keywords';

interface TestStep {
  keyword: string;
  selector: string;
  value: string;
}

const testSteps: TestStep[] = [];

fs.createReadStream('testCases.csv')
  .pipe(csvParser())
  .on('data', (row) => {
    testSteps.push(row as TestStep);
  })
  .on('end', () => {
    test('Keyword Driven Test', async ({ page }) => {
      for (const step of testSteps) {
        const { keyword, selector, value } = step;
        if ((keywords as any)[keyword]) {
          await (keywords as any)[keyword](page, selector, value);
        } else {
          console.error(`Keyword "${keyword}" not defined.`);
        }
      }
    });
  });


Advantages​

  1. Easy to build
  2. Simple to follow
  3. Data driven
  4. Reusable

Disadvantages​

Keyword management for large and complex applications is challenging.

Hybrid Testing Framework​

This is a combination of modular, data and keyword driven frameworks. This is the framework that's most commonly used. Although it will involve a considerable amount of time to build and set up - once the baseline is set, it's easy to automate and maintain the tests.

Advantages​

  1. Data driven
  2. Ease of maintenance
  3. Complex apps
  4. Customizations are easy

Disadvantages​

  1. Framework build and maintenance, needs someone who is technically strong.

DevAssure​

DevAssure is an automation app that is built using the best practices of a hybrid testing framework. The tool has a very powerful record and playback, which supports automating complex and simple applications. DevAssure's support for custom functions make it easy to automate and maintain.

Get Started for Free