Managing Test Case

Overview

Test cases are the cornerstone of the Testany Platform, enabling users to define specific scenarios for testing their applications. Every user, regardless of their role, can register and manage their own test cases. By default, only the owner of a test case can update or delete it. Global Admins also have the ability to edit and delete any test case. Other users can only view it. If needed, the owner or a Global Admin can transfer ownership to another user (see Transfer Ownership below).

Note: The voice in this video is AI-generated, not original.

Understanding a Qualified Test Case

A qualified test case should meet the following criteria:

1. Clear Objective: It should have a well-defined purpose and target a specific aspect of the application.
2. Reproducibility: It must be consistent in its execution, producing the same results under the same conditions.
3. Independence: It should be self-contained, not relying on other test cases to function.
4. Measurable Outcome: The results should be quantifiable and measurable to determine success or failure.
5. Compliance with Platform Standards: It must adhere to the specific guidelines and formats defined by the Testany Platform.

Test case on Testany Platform

Test frameworks the platform supports

A key feature of the Testany Platform is its support for multiple testing frameworks, and the ability to assemble test cases from different testing frameworks within a single test pipeline. Currently, the platform supports the following test executors: Postman, Playwright, Python Unit Test, PyRes, and JMeter. Maven and Gradle executors are no longer supported.

Understanding the Concepts of Test Case on Testany Platform

Definition

In our product, a "Test Case Package" refers to a bundled unit of a Test Case that encompasses both the Test Case Meta and Test Case Code. The Test Case Meta provides important context and information about the test case such as author, status, priority, version, and other necessary parameters. The Test Case Code is the executable script that outlines the steps for the test. By packaging the metadata and code together, users can register and manage tests cases more effectively on Testany Platform, and adjust parameters according to different testing scenarios.

1. Test Case Code File

The Test Case Code File contains the actual code that defines the test. It can vary depending on the test framework chosen by the creator.

Key Features:

• Framework Dependent: The file type and structure may vary based on the selected test framework.
• Logic Definition: Contains the step-by-step instructions for executing the test.
• Language Support: Written in the supported programming language(s) of the chosen framework.

2. Test Case Package File

The Test Case Package File is a zip file that encapsulates the Test Case Code File.

Key Features:

• Zip Format: Bundled as a zip file.
• Includes Code File(s): Combines the code file(s) into a single package.
• Portability: Enables easy sharing and registration of the test case within the platform.
• Integrity: Ensures that the code and meta files are consistently linked and managed as a unified entity.

Understanding the Concepts of Case Visibility on Testany Platform

In the Testany platform, Case Visibility refers to the level of access control applied to a Case, determining who can view and interact with it. There are two types of Case Visibility: Global Visibility and Workspace-Restricted Visibility.

• Global Visibility: Cases are accessible to all active users within the tenant.
• Workspace-Restricted Visibility: Cases are only accessible to members of the workspace(s) they are assigned to.

To further illustrate "Workspace-Restricted Visibility," here is an example:

A user has member permissions for the workspaces: WKS, ABC, FTT, and admin permissions for the workspaces: ACT, CPE.

• Case 1 is associated with workspaces: ACT, WKS, FTT.
• Case 2 is associated with workspaces: FTT, ABD.
• Case 3 is associated with workspaces: ABD, CPE.
• Case 4 is associated with workspaces: ABD, LBI.

In this scenario:

• The user can see Case 1 because their workspace member permissions (WKS, FTT) overlap with the Case's associated workspaces (WKS, FTT).
• The user can see Case 2 because their workspace member permissions (FTT) overlap with the Case's associated workspaces (FTT).
• The user cannot see Case 3 because their workspace member permissions does NOT overlap with the Case's associated workspaces (ABD, CPE).
• The user cannot see Case 4 because there is no overlap between their workspace permissions and the Case's associated workspaces (ABD, LBI).

Register Test Case

Step-by-Step Guide

1. Login to Testany Platform

Login and click "Test case library" button in the top bar.

2. Click "Register Test Case" button

3. Fill the form

After the "Register a new Test Case" pop-up appears, please fill out the form step-by-step:

3-1: Name: The name is an important attribute of a test case and will be frequently used in the Testany Platform. To enhance effectiveness and convenience for you and your colleagues, please ensure the name's correctness, completeness, and conciseness.

3-2: Description: You can describe your test case in more detail in the Description, making it easier for other users to utilize it. Listing the elements like the test object, preconditions, input data, execution steps, and expected results in the Description will greatly facilitate other users.

3-3: Select Runtime: You can select a Runtime for the test case to be executed.

3-4: Make Private?: You can set the case as global visible or restricted to workspaces.

3-4-1: Global Visible: keep "Make Private?" as No

3-4-2: Workspace Restricted Visibility: set "Make Private?" as Yes. Then, utilize the dropdown list to select the workspaces with which you want to share restricted cases.

4. Click "Register & Next" button

Click "Register & Next" button at the bottom right corner of the pop up to register the case. Then, you'll be redirected to the case detail page.

Edit Test Case

Specification of Test Case Package file

1. The file must have a .zip extension.

2. The zip file must contain your test case code file(s).

   1. at least one .json file (Postman collection file) if you choose Postman executor
   2. at least one .py file if you choose PyRes or Python executor
   3. at least one .jmx file if you choose JMeter executor
   4. at least one .js or .ts file if you choose Playwright executor

3. If you have multiple files for your test case, the file amount must not exceed 100. (Postman can only support one .json collection file)

4. The entire test case package .zip file size must not exceed 1MB.

5. A single test case can only have one environment. If you have a test case which is able to test multiple environments (by adjust environment variables), you need to register them to multiple test cases on Testany Platform.

Step-by-Step Guide

• Only the owner or a Global Admin has the authority to make edits.
• The test case key is immutable and cannot be altered under any circumstances.
• The runtime of the test case cannot be changed.

1. Navigate to the case detail page

2. Fill the required fields

2-1: Test Executor: you can choose the executor for your test case. After selecting the framework, you must provide either a Path or a Command depending on the executor type:

2-2: Definition file: This is where you can upload your test case code file(s). Please ensure that the file(s) are contained within a .zip archive.

2-3: Environments: This is an important attribute of a test case, describing the environment(s) of the service being tested by the test case. On the Testany Platform, each test case can correspond to multiple Environments.

2-4: Environment variables: you can fill in other environment variables required to run the test case. This section now supports three types in one place:

• Normal: regular key/value environment variables.
• Relay: variables used for pipeline relay orchestration.
• Secret: references to workspace credentials instead of raw secret values.

If a parameter needs to be temporarily changed during a certain execution, you can use an appropriate method to turn it into an environment variable and fill it in here.

If you have an environment that you wish to share in your case, you can specify the relay's name and value within the code. For guidance, please refer to the sample code in the section Add code to your test script. After defining the relay, make sure to expose the relay name as a relay environment variable by clicking the relay icon. This will enable its use during pipeline orchestration.

Read Test Case

You can view the test cases in the Test Case Library by card, list or directory views, as illustrated in the picture below. If you're looking to locate a specific test case quickly, simply utilize the search boxes located in the top right corner of the page.

In the case of a relay, you will notice a relay icon located in the top right corner of the case card when viewing in card format. Alternatively, in list view, the icon will be found on the leftmost side of the row corresponding to the case.

You may click the test case to view its detail information. There're several key elements in this page:

1. Test case key, please note the key won't (and can't) be changed in any circumstances.
2. Owner
3. Visibility
4. Environments
5. Labels
6. Version
7. Runtime
8. Meta (including environment_variables, executor and path/command)
9. Output

When a relay-related environment variable is defined in a case, as illustrated above, you will see the blue relay icon corresponding to that specific environment in the 'Type' column.

Dry Run Test Case

If a case has not been executed since its last modification, you will see the following indicators:

1. A message bar located beneath the case name serves as a reminder for you and other users who have access to this case, indicating that a successful dry run has not yet been completed.
2. The "DryRun" button is activated, signaling that as the case owner, you can revisit this page at any time and click it to perform the Dry Run.
3. The output indicates that a successful dry run for this case has not been completed yet.

By clicking "Dry Run" button, the platform will promptly perform a dry run to verify whether the test case is qualified and compatible. Following this, you will be directed to the dry run page of the test case, where you can review the displayed information to ensure it aligns with your expectations.

Transfer Ownership

If you need to hand over a test case to another user, the current owner or a Global Admin can transfer ownership.

1. Navigate to the case detail page.
2. In the Details panel on the right side, find the Owned by field and click on the current owner's name.
3. A search box will appear. Type to search for the new owner, then select them from the list.
4. Click Confirm to complete the transfer.

Delete Test Case

1. Navigate to case detail page

2. As case owner, click "Delete" button at the top right of the case

Test Case Packaging FAQ

Q1: What's the difference between "Path" and "Command"?

Q2: Will dependencies in requirements.txt be automatically installed?

Yes. For Python/PyRes executors, Testany automatically runs pip install -r requirements.txt before executing your test if the file exists in your package.

Q3: How do I package a multi-file Python test?

my-test.zip
├── tests/
│   ├── __init__.py
│   ├── test_login.py
│   └── test_order.py
├── utils/
│   └── helpers.py
└── requirements.txt

Command: python -m pytest tests/

Q4: Why does Postman only support a single .json file?

The Postman executor only supports one collection file. If you need environment variables, configure them in the Environment Variables section on the test case detail page instead of using a separate environment file.

Q5: What if my .zip file exceeds 1MB?

• Remove unnecessary files (.git, __pycache__, node_modules, etc.)
• Don't include large test data files; use environment variables instead
• If you genuinely need larger files, contact Testany support

Q6: My DryRun failed with "File not found". What should I check?

1. Check your Path/Command: Ensure it matches the actual file location inside your .zip
2. Check folder structure: If your test file is in a subdirectory, include the path (e.g., tests/test_api.py not just test_api.py)
3. Check file extension: Ensure the correct extension for your executor (.py for Python, .json for Postman, etc.)

×