Pipeline YAML Schema Reference

This document provides a complete technical reference for the Test Pipeline YAML schema. For usage examples and best practices, see Test Pipeline Writing Guideline and Samples.

Schema Version

Testany currently supports both rule/v1.2 and rule/v1.3.

Version Selection Guidance

Default Concurrency in rule/v1.3

When a pipeline uses rule/v1.3, the platform decides the default actual concurrency by instance type:

• Community Edition defaults to 2
• Paid Community users default to 4
• Enterprise Edition defaults to 8

If the number of semantically parallel-ready cases is greater than the concurrency available to the current workspace, the platform queues and refills slots instead of starting everything at once. When slots are limited, dispatch order within the same ready wave follows YAML order.

Top-Level Structure

YAMLCopy
kind: rule/v1.3
spec:
  rules:
    - run: <TEST-CASE-KEY>
      # ... rule fields

Rule Object

Each rule in the rules array defines a Test Case to execute and its conditions.

YAMLCopy
- run: AC2F5A50
  whenPassed: 9686C618
  expect: fail
  relay:
    - key: ACCESS_TOKEN
      refKey: 9686C618/ACCESS_TOKEN
      nonSecret: true

Rule Fields

Condition Fields: whenPassed vs whenFailed

Validation Rules:

• The referenced Test Case Key must be defined in a preceding rule.
• If neither whenPassed nor whenFailed is specified, the rule executes unconditionally (or after the test script that provides relay data finishes, if relay is defined).

The expect Field

The expect field changes how the test result is reported to the pipeline.

Key Point: A case with expect: fail always reports Passed to the pipeline. Therefore, subsequent cases should use whenPassed (not whenFailed) to follow it.

YAMLCopy
- run: A1B2C3D4             # Test case: validate setup
- run: E5F6A7B8             # Test case: test invalid input
  whenPassed: A1B2C3D4
  expect: fail              # Always reports "Passed" to pipeline
- run: C9D0E1F2             # Test case: next step
  whenPassed: E5F6A7B8      # Use whenPassed, not whenFailed

Relay Object

The relay field enables Output Relay - passing environment variables from one case to another.

YAMLCopy
relay:
  - key: ACCESS_TOKEN
    refKey: 5DC1106D/ACCESS_TOKEN
    nonSecret: true

Relay Fields

Relay Validation Rules

1. Missing fields: Both key and refKey are required.
2. Undefined source: The source case (refKey prefix) must be defined in a preceding rule.
3. Relay with whenFailed: You cannot relay from a case that you reference with whenFailed.
4. Parallel ambiguity: In rule/v1.3, the test script that provides relay data must be guaranteed to complete before the test script that consumes relay data starts. If both could land in the same parallel-ready wave, the configuration is rejected.

Invalid Example:

YAMLCopy
# ERROR: Cannot relay from a failed case
- run: 5385B189
- run: A3D9405F
  whenFailed: 5385B189      # This case failed
  relay:
    - key: NEW_URL
      refKey: 5385B189/NEW_URL  # Cannot relay from failed case

Implicit whenPassed Behavior

If a rule has relay defined but no explicit whenPassed or whenFailed, the system automatically adds whenPassed referencing the first test script that provides relay data.

YAMLCopy
# These two configurations are equivalent:

# Explicit
- run: 5DC1106D
- run: DE96D6A0
  whenPassed: 5DC1106D
  relay:
    - key: TOKEN
      refKey: 5DC1106D/TOKEN

# Implicit (system adds whenPassed automatically)
- run: 5DC1106D
- run: DE96D6A0
  relay:
    - key: TOKEN
      refKey: 5DC1106D/TOKEN

Complete Validation Rules

The platform validates Pipeline YAML against these rules:

Quick Reference

Minimal Pipeline

YAMLCopy
kind: rule/v1.3
spec:
  rules:
    - run: A1B2C3D4           # Your test case key

Sequential Execution (Pass Chain)

YAMLCopy
kind: rule/v1.3
spec:
  rules:
    - run: A1B2C3D4           # Step 1
    - run: E5F6A7B8           # Step 2
      whenPassed: A1B2C3D4
    - run: C9D0E1F2           # Step 3
      whenPassed: E5F6A7B8

Parallel Execution (No Conditions, recommended in rule/v1.3)

YAMLCopy
kind: rule/v1.3
spec:
  rules:
    - run: A1B2C3D4           # Test A
    - run: E5F6A7B8           # Test B
    - run: C9D0E1F2           # Test C

With Output Relay

YAMLCopy
kind: rule/v1.3
spec:
  rules:
    - run: A1B2C3D4           # Login test case
    - run: E5F6A7B8           # Create order test case
      whenPassed: A1B2C3D4
      relay:
        - key: ORDER_ID
          refKey: A1B2C3D4/ORDER_ID
        - key: ITEM_COUNT
          refKey: A1B2C3D4/ITEM_COUNT
          nonSecret: true

Negative Testing

YAMLCopy
kind: rule/v1.3
spec:
  rules:
    - run: A1B2C3D4           # Test case: test invalid input
      expect: fail            # Always reports "Passed" regardless of actual result
    - run: E5F6A7B8           # Test case: next test
      whenPassed: A1B2C3D4    # Use whenPassed since expect:fail always reports Passed

See Also

• Test Pipeline Writing Guideline and Samples - Usage examples and best practices
• Build your tests more robust & flexible with Output Relay - Detailed Output Relay concepts
• Managing Test Pipeline - Pipeline management operations