Glossary

Assertion

This refers to a statement in a test setup that verifies whether the output of a particular test case matches the expected output or behavior. If the assertion is true, the test case passes; if it is false, the test case fails. Assertions are an essential part of creating robust and effective automated testing scenarios, as they allow for precise and immediate feedback on the functionality of the product or system being tested.

Case Visibility

In the Testany platform, Case Visibility refers to the level of access control applied to a Case, determining who can view and interact with it. There are two types of Case Visibility: Global Visibility and Workspace-Restricted Visibility

CloudPrime

CloudPrime, in contrast to TR, is a Runtime managed by Testany first-party. Each tenant has only one CloudPrime Runtime.

Command

In Testany Platform, the Command refers to the executable instruction or script command required to run a specific test case. It is the exact line of code or command that triggers the test execution within the specified environment, often including parameters or options tailored to the test framework being used.

Currently, “Command” will be required if Executor “python” or “Pyres” is selected.

Credential Safe

“Credential Safe” is a virtual object representing a third-party credential store.

Cumulative View

One of two main filtering modes of Directory View for Case Library. In Cumulative View, when you select a directory, the list of test cases on the right includes test cases from that directory as well as all of its subdirectories. In other words, if a test case appears in any subdirectory, it will also appear in the parent directory's test case list.

To learn more about filtering modes, please click here.

(user) Disabled

Disabled is a user status. In this state, the user will not be allowed to log in, access any resources under that Tenant, or perform any operations on any resources. Furthermore, the user will not be able to receive any notifications sent through any notification channel (method) by Testany. At the same time, the user will be removed from all the ‘global admin', 'workspace admin', 'workspace member’ and the 'owner', 'watcher’ fields of all resources. Disabled is an irreversible state which can be only performed by Global Admin.

Environment

The term "environment" refers to the application environment targeted by the user's testing, such as stage, prod, etc.

Exclusive View

One of two main filtering modes of Directory View for Case Library. In Exclusive View, the list of test cases on the right only shows those directly associated with the chosen directory, excluding any subdirectory content. Put simply, only test cases explicitly labeled for this directory (and not just its subdirectories) will be displayed in the list.

To learn more about filtering modes, please click here.

Execution

Each execution is a run of a pipeline including the result of the pipeline execution. It contains the real-time status and the result of the run of a pipeline.

Execution Unit

A system controlled running environment to execute test cases.

Executor

An Executor in Testany Platform is the core component responsible for running test cases within a pipeline. It executes user-defined test scripts and configurations in specified environments, processes test data, and generates results. Executors support various testing frameworks and ensure seamless integration with the user’s testing environment, enabling efficient and reliable execution of automated tests.

Gatekeeper

In our product, "Gatekeeper" is a feature that enables users to trigger a specific group of tests using a webhook. Users can create a Gatekeeper and receive a webhook from it, which can then be integrated into various Dev/Ops tools. Whenever a user-specified event occurs, the Gatekeeper will trigger the execution of a pre-set pipeline group. This empowers users to execute tests in specific scenarios, facilitating efficient, ongoing product optimization and issue monitoring.

Gatekeeper Key

Gatekeeper Key is a unique identifier used on the Testany platform to label Gatekeepers. Each Gatekeeper receives a distinct Key upon creation. This Key comprises 10 characters in total: the initial character is 'G', signifying 'Gatekeeper', followed by a hyphen. The next three characters represent the Key value of the corresponding Workspace for the Gatekeeper. After another hyphen, the final four characters are numerical, automatically generated by the system. For example, a Gatekeeper Key might look like this: G-WSK-5678. Once a Gatekeeper is successfully created, its Key cannot be modified or deleted by any role. This identifier remains constant, regardless of any changes or deletion of the Gatekeeper itself.

Gatekeeper Owner

The Gatekeeper Owner in our system is a user who has the authority to edit and manage Gatekeepers for triggering test executions. The Gatekeeper Owner determines the specific conditions under which tests are triggered, such as defining events or criteria that activate the Gatekeeper. They also have the responsibility to add users to the watcher list, ensuring that the right team members are notified and involved in the process. This role is crucial for maintaining high efficiency and effectiveness in automated testing, facilitating proactive product optimization and issue monitoring.

Global Admin

In Testany, it’s a system default role. To learn more about permission policy of Testany, please click here.

Global Visibility

Cases with Global Visibility are accessible to all active users within the tenant. These Cases can be viewed and managed by any tenant active user (with existing permission control mechanism), regardless of their workspace membership. This option is ideal for collaborative scenarios where Cases need to be shared across multiple teams.

Import History

An Import History is created each time you perform a bulk import of test cases from a Git repository. It stores key details such as repository URL, selected branch, commit ID, the list of imported test cases, and the import timestamp. By reviewing or modifying the associated Import History, users can track and manage the imported test cases and conveniently handle subsequent changes.

Import History Owner

The Import History Owner is the user who performed the bulk import of test cases from Git. This user has administrative privileges over that particular import’s configuration and maintenance within Testany, such as deleting or editing the Import History and updating the imported test cases.

Output relay

The Output relay feature enables the seamless transfer of data between test cases within the same pipeline during execution. It allows one or more variables from a completed test case (which is called “Relay case”) to be used as input for one or more variables in subsequent test cases. This data relay occurs in real-time, utilizing values obtained dynamically during the execution process.

By leveraging the Output Relay, users can efficiently manage test data across multiple test cases without the need for manual intervention or external data sources. This feature optimizes test automation workflows by allowing test cases to rely on actual execution results, improving the accuracy and relevance of downstream test scenarios. It also enhances the overall efficiency of test data management, particularly in environments requiring interdependent test cases.

Path

In Testany Platform, the Path refers to the location in the .zip file of the test case script, allowing the platform to locate and execute the test case. The path must match the location where the script is stored in the testing environment.

Currently, “Path” will be required if Executor “Postman”, “Maven” or “Gradle” is selected.

Plan

In our product, “Plan" is a feature that allows users to pre-set the execution time and frequency of tests. Users can create plans by setting precise dates and times or setting repeating intervals. This empowers users to execute tests at ideal times and with appropriate frequency, enabling efficient, ongoing product optimization and issue monitoring.

Plan Key

It is an identifier used on the Testany platform to label Plans and possesses uniqueness. When each Plan is created on the platform, the system assigns it a value. This Key is made up of 10 characters in total: the first character is 'P', representing 'Plan', the second character is a hyphen, the third to fifth characters are the Key value of the Plan's corresponding Workspace, the sixth character is a hyphen, and the last four characters are numbers automatically generated by the system. (Sample: P-TAN-1234) Once the Plan is successfully created, all roles cannot modify or delete its Key. Even if the Plan itself is altered or deleted, this value remains unchanged.

Plan Owner

This refers to a user in our system who have the permission to edits and manages plans for test execution. The Plan Owner sets the precise time and frequency of tests, add user to watcher list, ensuring optimal efficiency and effectiveness in ongoing product testing and issue monitoring.

Relay case

“Relay” is an attribute of a test case. A test case with this attribute can pass the value of its pre-declared variable(s) as input to other cases within the same pipeline upon its execution completion. Relay Cases facilitate data transmission and sharing, ensuring that dependencies between test cases are properly managed, thereby enhancing the automation and efficiency of the testing process.

Characteristics:

Data Generation: A Relay case can be responsible for generating specific output data.

Data Transmission: This output data is automatically passed to designated downstream test cases.

Dependency Management: Relay cases enable the management and handling of dependencies between test cases.

Automation: Increases the level of automation in the test pipeline, reducing the need for manual intervention.

Runtime

In the context of the Testany Platform, a Runtime refers to a set of specialized services that manage test start/stop, test running, credentials for testing, and access to test logs. Additionally, these services are encapsulated within a Kubernetes Operator, which can be deployed on public clouds, private clouds, or even physical servers. The Runtime directly receives commands from the Testany Platform's scheduling system to run tests, stores logs after testing is complete, and sends the test results (success, fail, skip, etc.) to the Testany Platform's notification system.

Secure UUID

This is UUID for the secret key name prefix. Each test case registered in Testany will have one unique Secure UUID. It used for TSS retrieve test secrets from Trusted Credential Safe. If you have a secret stored in your Trusted Credential Safe, the secret name should follow <Secure UUID>-<secretName> format, which means, if the secret named client-secret, then the secret stored in your Trusted Credential Safe named <Secure UUID>-client-secret

Subscription

(user) Suspended

Suspended is a user status. In this state, the user will not be allowed to log in, access any resources under that Tenant, or perform any operations on any resources. Moreover, the user will not be able to receive any notifications sent through any notification channel (method) by Testany. However, unless modified by a Global Admin, all properties and data of the user under that Tenant remain unchanged. Suspension is a reversible action which can be only performed by Global Admin. When the 'suspended' attribute is reset to false, the user regains all the operational permissions they had before being suspended.

Test Case

A "test case" is a set of conditions or variables under which a tester will determine whether an application, software system, or one of its features is working as it was originally established to do. The test case includes specific variables or conditions, along with expected results, which help the tester to determine whether the system's performance is in line with its design and requirements.

Test Case Library

In Testany Platform, the Test Case Library is a repository for storing, organizing, and managing test cases. As one of the top-level structures in the Testany Platform, it provides users with an organized space to create, edit, categorize, and maintain test cases, enabling efficient reuse, version control, and team collaboration.

Test Case Owner

This is a user role in our product that enables users to edit and manage test case on Testany Platform. Test Case Owner can edit test scripts, define input parameters, expected output and/or behavior. The managed test cases are included in the Test Pipeline, used for continuously verifying the quality and performance of the product or system.

Test Script

This refers to the executable script in a test case that outlines the specific steps and conditions to be followed during the testing process. It includes the test inputs, expected outcomes, and assertion statements, among other things. This script is designed to be run in an automated testing framework, and it's part of the overall Test Case Package that also includes metadata related to the test case.

Test Case Key

Test Case Key is an identifier used on the Testany platform to mark Test Cases. Every Test Case, when registered on the Testany platform, is automatically assigned an 8-digit Key value by the system. (Sample: 20231024) This Key will accompany the entire lifecycle of the Test Case on the Testany platform, and even if the Test Case is updated, this value will not change.

Test Case Meta

This refers to the metadata associated with a test case in the Test Case Package. It provides essential context and information about the test case, including but not limited to, the author, test environment, creation date, status, priority, version, and potentially sensitive information like the way of accessing credentials of applications need to be tested. This metadata is crucial for effectively managing, executing, and interpreting test results, and it complements the Test Case Code in the overall Test Case Package.

Test Case Step

A test case step is a step in the process of a test case. One test case may contain multiple test cases (But we strongly NOT RECOMMEND). Generally, a test case step is an api call.

Test Execution

This term refers to the act of running a test pipeline. It involves initiating the test, monitoring its progress, and may also include live tracking of results. Essentially, it is the operation of carrying out the predefined tests on your software components.

Test Execution Engine

(Aka TEE) The service in Testany which responsible for testing execution.

Test Execution Result

This is the outcome generated after a test execution has been completed. It provides a comprehensive view of what passed, what failed, and any other metrics or diagnostics captured during the test. Test execution results are usually what teams review to assess quality, performance, and reliability.

Test Execution UUID

The "Test Execution UUID" on Testany Platform is a composite string that uniquely identifies each test execution run. It follows the format {test_pipeline_key}-{5_digit_auto_incrementing_number}. In this format, the "test pipeline key" is denoted as "xxx-1234," and it is paired with a 5-digit auto-incrementing number. This unique identifier ensures that each test run within a specific test pipeline is distinct, facilitating easier tracking, querying, and management of the test results and associated metadata.

Test Pipeline

In our product, a "Test Pipeline" is a feature for implementing automated testing. It consists of a series of predefined automated testing cases that are triggered by various test frameworks and can be executed in a specific order and logic. The goal of the Test Pipeline is to continuously verify the quality and performance of the product or system. Each execution produces test results, which can be reviewed in “Executions” and pushed to Watchers.

Test Pipeline Owner

This is a user role in our product that enables users to edit and manage Test Pipelines. Test Pipeline Owner can select and arrange predefined automated testing cases and define their execution order and logic. Their goal is to create Test Pipelines that continuously verify the quality and performance of the product or system.

Test Pipeline Key

Test Pipeline Key is an identifier used on the Testany platform to label Test Pipelines and is unique. When each Test Pipeline is created on the platform, the system assigns it a value. This Key consists of 8 characters in total: the first three are the Key value of the Test Pipeline's corresponding Workspace, the fourth character is a hyphen, and the last four characters are numbers automatically generated by the system. (TAN-5678) Once the Test Pipeline is successfully created, all roles are unable to change or delete the Key, and even if the Test Pipeline itself is changed or deleted, this value remains unchanged.

Test Orchestration Engine

 (Aka TOE) This is a key component of our product that automates and manages the execution of complex workflows, like Test Pipelines. The orchestration engine coordinates the order, timing, and dependencies of each step in the pipeline, ensuring tests are performed efficiently and accurately. It can handle tasks like triggering tests at specified times, managing resources, handling errors, and sending results to Watchers.

Testany Secrets Service

(Aka TSS) Testany Secrets Service is a specialized service designed to fetch secrets from Trusted Certificate Safes, such as Azure Key Vault and AWS Secrets Manager. It facilitates secure retrieval of sensitive information like keys and certificates, ensuring data security and compliance with industry standards. The service operates by receiving requests to extract specific secrets identified by keys and responds with the requested data, thus playing a crucial role in secure information management.

Trusted Runtime

(aka “TR”) Trusted Runtime refers to a Runtime that is installed and managed by users themselves. The TR communicates with the Testany Platform SaaS that the user subscribes to, receives instructions, and outputs results. Each tenant can have multiple TR simultaneously to satisfy different testing requirements.

User

In our product, an "User" refers to a user's profile within the system. It contains the user information, login credentials, and settings. The User also determines the user's permissions and access rights within the product, such as what workspaces, test pipelines, and plans they can access and manage.

Watcher

In our product, a "Watcher" refers to a role associated with Test Pipelines and Plans. Watchers are users who receive active push notifications of the test pipeline execution results. This enables these users to keep track of and understand the progress and results of the tests in real-time, without the need to manually check or inquire. The watcher list of a specific Test Pipeline is created and managed by Plan Creator.

Workspace

In our product, a “Workspace" is an isolated working environment that includes Test Pipeline, Test Execution, and Plan. Each workspace is isolated from others, offering a distinct resource and data environment. Access to specific workspaces is granted to users by Global Admin or Workspace Admin through our product-defined Role-Based Access Control (RBAC) policies.

Workspace Key

Workspace is an identifier used on the Testany platform to label Workspaces. When each Workspace is created on the platform, the system requires the creator to specify a value. This Key is three characters long, with the first character being a letter, and the following two characters can be either letters or numbers. (Sample: TA9) However, the key does not support spaces, symbols, special characters, or emojis. This Key will accompany the entire lifecycle of the Workspace on the Testany platform, and even if the Workspace is updated or archived, this value will not change.

Roles with the authority to create Workspaces can specify the Key when creating the Workspace for the first time. After creation, the Key cannot be changed or deleted. Other roles do not have the permission to create, modify, or delete this Key.

Workspace Admin

In Testany, it’s a system default role. To learn more about permission policy of Testany, please click here.

Workspace Member

In Testany, it’s a system default role. To learn more about permission policy of Testany, please click here.

Workspace-Restricted Visibility

Cases with Workspace-Restricted Visibility are only accessible to members of the workspace(s) they are assigned to. These Cases cannot be viewed or accessed by users outside the specified workspace(s), ensuring data isolation and confidentiality.

For Cases with Workspace-Restricted Visibility, a user can view the Case if there is any overlap between the set of workspaces where the user has member permissions and the set of workspaces associated with the Case.

To learn more about the workspace-restricted visibility, please click here.