Glossary
π¦ A-D
| Name | Definition |
|---|---|
| Assertion | This refers to a statement in a test setup that verifies whether the output of a particular test case matches the expected output or behavior. If the assertion is true, the test case passes; if it is false, the test case fails. Assertions are an essential part of creating robust and effective automated testing scenarios, as they allow for precise and immediate feedback on the functionality of the product or system being tested. |
| Case Visibility | In the Testany platform, Case Visibility refers to the level of access control applied to a Case, determining who can view and interact with it. There are two types of Case Visibility: Global Visibility and Workspace-Restricted Visibility |
| CloudPrime | CloudPrime, in contrast to TR, is a Runtime managed by Testany first-party. Each tenant has only one CloudPrime Runtime. |
| Command | In Testany Platform, the Command refers to the executable instruction or script command required to run a specific test case. It is the exact line of code or command that triggers the test execution within the specified environment, often including parameters or options tailored to the test framework being used. Currently, "Command" will be required if Executor "python" or "Pyres" is selected. |
| Credential Safe | "Credential Safe" is a virtual object representing a third-party credential store. |
| Cumulative View | One of two main filtering modes of Directory View for Case Library. In Cumulative View, when you select a directory, the list of test cases on the right includes test cases from that directory as well as all of its subdirectories. In other words, if a test case appears in any subdirectory, it will also appear in the parent directory's test case list. To learn more about filtering modes, please click here. |
| (user) Disabled | Disabled is a user status. In this state, the user will not be allowed to log in, access any resources under that Tenant, or perform any operations on any resources. Furthermore, the user will not be able to receive any notifications sent through any notification channel (method) by Testany. At the same time, the user will be removed from all the 'global admin', 'workspace admin', 'workspace member' and the 'owner', 'watcher' fields of all resources. Disabled is an irreversible state which can be only performed by Global Admin. |
πͺ E-K
| Name | Definition |
|---|---|
| Environment | The term "environment" refers to the application environment targeted by the user's testing, such as stage, prod, etc. |
| Exclusive View | One of two main filtering modes of Directory View for Case Library. In Exclusive View, the list of test cases on the right only shows those directly associated with the chosen directory, excluding any subdirectory content. Put simply, only test cases explicitly labeled for this directory (and not just its subdirectories) will be displayed in the list. To learn more about filtering modes, please click here. |
| Execution | Each execution is a run of a pipeline including the result of the pipeline execution. It contains the real-time status and the result of the run of a pipeline. |
| Execution Unit | A system controlled running environment to execute test cases. |
| Executor | An Executor in Testany Platform is the core component responsible for running test cases within a pipeline. It executes user-defined test scripts and configurations in specified environments, processes test data, and generates results. Executors support various testing frameworks and ensure seamless integration with the user's testing environment, enabling efficient and reliable execution of automated tests. |
| Gatekeeper | In our product, "Gatekeeper" is a feature that enables users to trigger a specific group of tests using a webhook. Users can create a Gatekeeper and receive a webhook from it, which can then be integrated into various Dev/Ops tools. Whenever a user-specified event occurs, the Gatekeeper will trigger the execution of a pre-set pipeline group. This empowers users to execute tests in specific scenarios, facilitating efficient, ongoing product optimization and issue monitoring. |
| Gatekeeper Key | Gatekeeper Key is a unique identifier used on the Testany platform to label Gatekeepers. Each Gatekeeper receives a distinct Key upon creation. This Key comprises 10 characters in total: the initial character is 'G', signifying 'Gatekeeper', followed by a hyphen. The next three characters represent the Key value of the corresponding Workspace for the Gatekeeper. After another hyphen, the final four characters are numerical, automatically generated by the system. For example, a Gatekeeper Key might look like this: G-WSK-5678. Once a Gatekeeper is successfully created, its Key cannot be modified or deleted by any role. This identifier remains constant, regardless of any changes or deletion of the Gatekeeper itself. |
| Gatekeeper Owner | The Gatekeeper Owner in our system is a user who has the authority to edit and manage Gatekeepers for triggering test executions. The Gatekeeper Owner determines the specific conditions under which tests are triggered, such as defining events or criteria that activate the Gatekeeper. They also have the responsibility to add users to the watcher list, ensuring that the right team members are notified and involved in the process. This role is crucial for maintaining high efficiency and effectiveness in automated testing, facilitating proactive product optimization and issue monitoring. |
| Global Admin | In Testany, it's a system default role. To learn more about permission policy of Testany, please click here. |
| Global Visibility | Cases with Global Visibility are accessible to all active users within the tenant. These Cases can be viewed and managed by any tenant active user (with existing permission control mechanism), regardless of their workspace membership. This option is ideal for collaborative scenarios where Cases need to be shared across multiple teams. |
| GitHub App Installation | A GitHub App Installation is the authorization binding created when a GitHub App is installed on a personal account or organization. In Import from Git V2, each installation corresponds to one repository-owner scope that Testany can use to discover importable repositories and manage repository access. |
| Import History | An Import History is the persistent record Testany creates for each Import from Git operation. It stores the source repository, selected branch, version strategy, import mode, selected files, runtime, owner, timestamps, and other lifecycle data required to maintain the imported test cases over time. By reviewing or modifying the Import History, users can track and manage the ongoing relationship between Testany test cases and the underlying Git source. |
| Import History Owner | The Import History Owner is the user responsible for creating and maintaining a specific Import History record in Testany. This user manages the ongoing configuration and lifecycle of that Git-linked relationship, including editing or deleting the Import History and updating the linked test cases. |
π± L-O
| Name | Definition |
|---|---|
| Managed Import | Managed Import is an Import from Git V2 mode where Testany imports repository code into a platform-managed Mirror. Execution reads scripts from that Mirror instead of directly from Git. In track-latest mode, the Mirror is advanced by manual Sync or GitHub Webhook rather than by implicit execution-time fetches. |
| Manual Trigger | Manual Trigger is a trigger template in Testany used for on-demand execution. A user can create a Manual Trigger, bind one or more pipelines, and run it whenever needed without waiting for a schedule or webhook event. Each run creates a trigger instance that can be traced in Execution history. |
| Manual Trigger Key | Manual Trigger Key is the immutable identifier of a Manual Trigger resource. It is generated by the platform when the trigger is created, starts with prefix "M", includes workspace key, and ends with a 4~5 uppercase hexadecimal suffix. The key is used in URLs, API queries, and execution traceability. |
| Manual Trigger Owner | Manual Trigger Owner is the user responsible for a specific Manual Trigger template. The owner can update trigger metadata, maintain bound pipelines, and manage notification configuration according to workspace permission policies. This role is different from Manual Trigger Executor (the operator who clicks Run for a specific trigger instance). |
| Manual Trigger Executor | Manual Trigger Executor is the user who initiates a specific Manual Trigger run. In execution data, this identity is tracked per trigger instance and can be different from the Manual Trigger Owner. |
| Mirror | Mirror is the platform-managed copy of repository code used by Managed Import records in Import from Git V2. Testany refreshes this copy to newer commits through explicit lifecycle actions such as Sync or Webhook-driven refresh. If the managed import history is deleted, the corresponding Mirror is deleted as well. |
| Output relay | The Output relay feature enables the seamless transfer of data between test cases within the same pipeline during execution. It allows one or more variables from a completed test case (which is called "Relay case") to be used as input for one or more variables in subsequent test cases. This data relay occurs in real-time, utilizing values obtained dynamically during the execution process. By leveraging the Output Relay, users can efficiently manage test data across multiple test cases without the need for manual intervention or external data sources. This feature optimizes test automation workflows by allowing test cases to rely on actual execution results, improving the accuracy and relevance of downstream test scenarios. It also enhances the overall efficiency of test data management, particularly in environments requiring interdependent test cases. |
π΅ P-S
| Name | Definition |
|---|---|
| Path | In Testany Platform, the Path refers to the location in the .zip file of the test case script, allowing the platform to locate and execute the test case. The path must match the location where the script is stored in the testing environment. Currently, "Path" is required for path-based executors such as "Postman", "Playwright" and "JMeter". |
| Plan | In our product, "Plan" is a feature that allows users to pre-set the execution time and frequency of tests. Users can create plans by setting precise dates and times or setting repeating intervals. This empowers users to execute tests at ideal times and with appropriate frequency, enabling efficient, ongoing product optimization and issue monitoring. |
| Plan Key | It is an identifier used on the Testany platform to label Plans and possesses uniqueness. When each Plan is created on the platform, the system assigns it a value. This Key is made up of 10 characters in total: the first character is 'P', representing 'Plan', the second character is a hyphen, the third to fifth characters are the Key value of the Plan's corresponding Workspace, the sixth character is a hyphen, and the last four characters are numbers automatically generated by the system. (Sample: P-TAN-1234) Once the Plan is successfully created, all roles cannot modify or delete its Key. Even if the Plan itself is altered or deleted, this value remains unchanged. |
| Plan Owner | Plan Owner is the owner of a Plan resource. This user is responsible for maintaining the plan configuration, including schedule, linked pipelines, metadata (such as labels), and notification setup, subject to the platform's permission controls. |
| Pinned Commit | Pinned Commit is a version strategy where an import history is locked to one specific commit. The record does not follow later branch commits until the user explicitly switches to another commit or changes back to Track Latest. |
| Repository Access Scope | Repository Access Scope is the set of repositories exposed to Testany through a specific GitHub App installation. In Import from Git V2, it determines which repositories appear in the wizard for a selected owner and which repositories can actually be imported. |
| Relay case | "Relay" is an attribute of a test case. A test case with this attribute can pass the value of its pre-declared variable(s) as input to other cases within the same pipeline upon its execution completion. Relay Cases facilitate data transmission and sharing, ensuring that dependencies between test cases are properly managed, thereby enhancing the automation and efficiency of the testing process. Characteristics: Data Generation: A Relay case can be responsible for generating specific output data. Data Transmission: This output data is automatically passed to designated downstream test cases. Dependency Management: Relay cases enable the management and handling of dependencies between test cases. Automation: Increases the level of automation in the test pipeline, reducing the need for manual intervention. |
| source_deleted | source_deleted is a binding state in Import from Git V2. It means a previously tracked source file no longer exists in the current source snapshot or Mirror snapshot. Marking a file as source_deleted updates the binding state, but does not automatically delete the corresponding Test Case. |
| Sync Link | Sync Link is a Git-linked mode where Testany keeps the relationship between Test Cases and Git files, while execution reads scripts directly from Git at run time. In Import from Git V1, Sync Link is the only supported mode. In Import from Git V2, it is one of the available modes. In track-latest mode, execution follows the branch latest commit, but relationship changes such as added files or source-deleted files still require explicit handling. |
| Runtime | In the context of the Testany Platform, a Runtime refers to a set of specialized services that manage test start/stop, test running, credentials for testing, and access to test logs. Additionally, these services are encapsulated within a Kubernetes Operator, which can be deployed on public clouds, private clouds, or even physical servers. The Runtime directly receives commands from the Testany Platform's scheduling system to run tests, stores logs after testing is complete, and sends the test results (success, fail, skip, etc.) to the Testany Platform's notification system. |
| Secure UUID | This is UUID for the secret key name prefix. Each test case registered in Testany will have one unique Secure UUID. It used for TSS retrieve test secrets from Trusted Credential Safe. If you have a secret stored in your Trusted Credential Safe, the secret name should follow client-secret, then the secret stored in your Trusted Credential Safe named <Secure UUID>-client-secret |
| Subscription | |
| (user) Suspended | Suspended is a user status. In this state, the user will not be allowed to log in, access any resources under that Tenant, or perform any operations on any resources. Moreover, the user will not be able to receive any notifications sent through any notification channel (method) by Testany. However, unless modified by a Global Admin, all properties and data of the user under that Tenant remain unchanged. Suspension is a reversible action which can be only performed by Global Admin. When the 'suspended' attribute is reset to false, the user regains all the operational permissions they had before being suspended. |
πΉ T-Z
| Name | Definition |
|---|---|
| Test Case | A "test case" is a set of conditions or variables under which a tester will determine whether an application, software system, or one of its features is working as it was originally established to do. The test case includes specific variables or conditions, along with expected results, which help the tester to determine whether the system's performance is in line with its design and requirements. |
| Test Case Library | In Testany Platform, the Test Case Library is a repository for storing, organizing, and managing test cases. As one of the top-level structures in the Testany Platform, it provides users with an organized space to create, edit, categorize, and maintain test cases, enabling efficient reuse, version control, and team collaboration. |
| Test Case Owner | This is a user role in our product that enables users to edit and manage test case on Testany Platform. Test Case Owner can edit test scripts, define input parameters, expected output and/or behavior. The managed test cases are included in the Test Pipeline, used for continuously verifying the quality and performance of the product or system. |
| Test Script | This refers to the executable script in a test case that outlines the specific steps and conditions to be followed during the testing process. It includes the test inputs, expected outcomes, and assertion statements, among other things. This script is designed to be run in an automated testing framework, and it's part of the overall Test Case Package that also includes metadata related to the test case. |
| Test Case Key | Test Case Key is an identifier used on the Testany platform to mark Test Cases. Every Test Case, when registered on the Testany platform, is automatically assigned an 8-digit Key value by the system. (Sample: 20231024) This Key will accompany the entire lifecycle of the Test Case on the Testany platform, and even if the Test Case is updated, this value will not change. |
| Test Case Meta | This refers to the metadata associated with a test case in the Test Case Package. It provides essential context and information about the test case, including but not limited to, the author, test environment, creation date, status, priority, version, and potentially sensitive information like the way of accessing credentials of applications need to be tested. This metadata is crucial for effectively managing, executing, and interpreting test results, and it complements the Test Case Code in the overall Test Case Package. |
| Test Case Step | A test case step is a step in the process of a test case. One test case may contain multiple test cases (But we strongly NOT RECOMMEND). Generally, a test case step is an api call. |
| Test Execution | This term refers to the act of running a test pipeline. It involves initiating the test, monitoring its progress, and may also include live tracking of results. Essentially, it is the operation of carrying out the predefined tests on your software components. |
| Test Execution Engine | (Aka TEE) The service in Testany which responsible for testing execution. |
| Test Execution Result | This is the outcome generated after a test execution has been completed. It provides a comprehensive view of what passed, what failed, and any other metrics or diagnostics captured during the test. Test execution results are usually what teams review to assess quality, performance, and reliability. |
| Test Execution UUID | The "Test Execution UUID" on Testany Platform is a composite string that uniquely identifies each test execution run. It follows the format {test_pipeline_key}-{5_digit_auto_incrementing_number}. In this format, the "test pipeline key" is denoted as "xxx-1234," and it is paired with a 5-digit auto-incrementing number. This unique identifier ensures that each test run within a specific test pipeline is distinct, facilitating easier tracking, querying, and management of the test results and associated metadata. |
| Test Pipeline | In our product, a "Test Pipeline" is a feature for implementing automated testing. It consists of a series of predefined automated testing cases that are triggered by various test frameworks and can be executed in a specific order and logic. The goal of the Test Pipeline is to continuously verify the quality and performance of the product or system. Each execution produces test results, which can be reviewed in "Executions" and pushed to Watchers. |
| Test Pipeline Owner | This is a user role in our product that enables users to edit and manage Test Pipelines. Test Pipeline Owner can select and arrange predefined automated testing cases and define their execution order and logic. Their goal is to create Test Pipelines that continuously verify the quality and performance of the product or system. |
| Test Pipeline Key | Test Pipeline Key is an identifier used on the Testany platform to label Test Pipelines and is unique. When each Test Pipeline is created on the platform, the system assigns it a value. This Key consists of 8 characters in total: the first three are the Key value of the Test Pipeline's corresponding Workspace, the fourth character is a hyphen, and the last four characters are numbers automatically generated by the system. (TAN-5678) Once the Test Pipeline is successfully created, all roles are unable to change or delete the Key, and even if the Test Pipeline itself is changed or deleted, this value remains unchanged. |
| Test Orchestration Engine | (Aka TOE) This is a key component of our product that automates and manages the execution of complex workflows, like Test Pipelines. The orchestration engine coordinates the order, timing, and dependencies of each step in the pipeline, ensuring tests are performed efficiently and accurately. It can handle tasks like triggering tests at specified times, managing resources, handling errors, and sending results to Watchers. |
| Testany Secrets Service | (Aka TSS) Testany Secrets Service is a specialized service designed to fetch secrets from Trusted Certificate Safes, such as Azure Key Vault and AWS Secrets Manager. It facilitates secure retrieval of sensitive information like keys and certificates, ensuring data security and compliance with industry standards. The service operates by receiving requests to extract specific secrets identified by keys and responds with the requested data, thus playing a crucial role in secure information management. |
| Trigger ID | Trigger ID identifies a trigger object/template. For Plan and Gatekeeper, this is the object key itself (for example, P-WKS-7351 or G-WKS-0605). For Manual Trigger, this is M-<workspace_key>-<seq5> (for example, M-WKS-00001). |
| Trigger Instance ID | Trigger Instance ID identifies one concrete run under the same Trigger ID (for example, 00007). It is used with Trigger ID to form Trigger Instance Key (for example, M-WKS-00001 + 00007 => M-WKS-00001-00007). |
| Trigger Instance | Trigger Instance represents one concrete trigger event in execution history (for example, one manual run, one plan schedule hit, or one gatekeeper webhook trigger). A single trigger instance can include multiple pipeline executions, and each pipeline execution can include multiple test case executions. |
| Track Latest | Track Latest is a version strategy where an import history follows the latest commit on the selected branch. In Import from Git V2, this controls how the code source moves forward, but it does not automatically apply relationship changes such as creating new Test Cases for newly discovered files or marking deleted source files. |
| Trusted Runtime | (aka "TR") Trusted Runtime refers to a Runtime that is installed and managed by users themselves. The TR communicates with the Testany Platform SaaS that the user subscribes to, receives instructions, and outputs results. Each tenant can have multiple TR simultaneously to satisfy different testing requirements. |
| User | In our product, an "User" refers to a user's profile within the system. It contains the user information, login credentials, and settings. The User also determines the user's permissions and access rights within the product, such as what workspaces, test pipelines, and plans they can access and manage. |
| Watcher | In our product, a "Watcher" refers to a role associated with Test Pipelines and Plans. Watchers are users who receive active push notifications of the test pipeline execution results. This enables these users to keep track of and understand the progress and results of the tests in real-time, without the need to manually check or inquire. The watcher list of a specific Test Pipeline is created and managed by Plan Creator. |
| Webhook Dispatch | Webhook Dispatch is a notification feature in Testany that sends test execution results to external systems via HTTP POST requests. It can be configured on both Plan and Gatekeeper to automatically deliver structured JSON payloads containing execution details (status, test case results, timestamps, etc.) to any webhook-compatible endpoint such as Slack, Jira, PagerDuty, or custom services. This enables automated workflows like team notifications, ticket creation, incident escalation, or CI/CD quality gates. |
| Workspace | In our product, a "Workspace" is an isolated working environment that includes Test Pipeline, Test Execution, and Plan. Each workspace is isolated from others, offering a distinct resource and data environment. Access to specific workspaces is granted to users by Global Admin or Workspace Admin through our product-defined Role-Based Access Control (RBAC) policies. |
| Workspace Key | Workspace is an identifier used on the Testany platform to label Workspaces. When each Workspace is created on the platform, the system requires the creator to specify a value. This Key is three characters long, with the first character being a letter, and the following two characters can be either letters or numbers. (Sample: TA9) However, the key does not support spaces, symbols, special characters, or emojis. This Key will accompany the entire lifecycle of the Workspace on the Testany platform, and even if the Workspace is updated or archived, this value will not change. Roles with the authority to create Workspaces can specify the Key when creating the Workspace for the first time. After creation, the Key cannot be changed or deleted. Other roles do not have the permission to create, modify, or delete this Key. |
| Workspace Admin | In Testany, it's a system default role. To learn more about permission policy of Testany, please click here. |
| Workspace Member | In Testany, it's a system default role. To learn more about permission policy of Testany, please click here. |
| Workspace-Restricted Visibility | Cases with Workspace-Restricted Visibility are only accessible to members of the workspace(s) they are assigned to. These Cases cannot be viewed or accessed by users outside the specified workspace(s), ensuring data isolation and confidentiality. For Cases with Workspace-Restricted Visibility, a user can view the Case if there is any overlap between the set of workspaces where the user has member permissions and the set of workspaces associated with the Case. To learn more about the workspace-restricted visibility, please click here. |
Still have questions?
Our team is here to help. Get in touch and we'll get back to you as soon as possible.