Skunit

# skUnit

skUnit is a semantic testing framework for .NET — write AI tests in plain Markdown, run them with xUnit, NUnit, or MSTest.

Test anything that talks to AI:

• AI Agents (Microsoft Agent Framework, A2A)
• IChatClient (Microsoft Extensions AI)
• MCP servers (Model Context Protocol)
• Custom AI integrations

dotnet add package skUnit

Quick Start

Imagine you have an AIAgent configured to answer questions about a bank account. Here's a test scenario in Markdown, balance-test.md:

# SCENARIO Balance Test

## [USER]
What is the account balance for John Doe?

## [ASSISTANT]

### ASSERT SemanticCondition
The answer mentions that the account balance is $1,234.56.

And the C# test:

[Fact]
public async Task SimpleTest()
{
    var markdown = File.ReadAllText("balance-test.md");
    var scenario = ChatScenario.Parse(markdown);

    await agent.RunAsync(scenario);
}

That's it. skUnit drives the conversation, calls your AI, and semantically verifies the response.

Tip — Parameterized scenarios: Use {placeholders} in your Markdown and replace them with C# string manipulation to run the same scenario with different inputs.

## [USER]
What is the account balance for {CustomerName}?

## [ASSISTANT]

### ASSERT SemanticCondition
The answer mentions that the account balance is {CustomerBalance}.

Key Features

1. Function Call Assertion

Verify that your agent calls the right tool with the right arguments:

# SCENARIO Balance Test

## [USER]
What is the account balance for John Doe?

## [ASSISTANT]

### ASSERT SemanticCondition
The answer mentions that the account balance is $1,234.56.

### ASSERT FunctionCall
{
  "function_name": "get-account-balance"
}

You can also assert on the arguments passed to the function, using exact or semantic matching:

{
  "function_name": "get-account-balance",
  "arguments": {
    "accountHolder": ["Equals", "John Doe"]
  }
}

{
  "function_name": "get-account-balance",
  "arguments": {
    "accountHolder": ["SemanticSimilar", "John Doe"]
  }
}

See ASSERT JsonCheck for full JSON schema validation.

2. Multi-Turn Conversations

Test realistic back-and-forth conversations — each turn gets its own assertions:

# SCENARIO Multi-Turn Balance Test

## [USER]
What is the account balance for John Doe?

## [ASSISTANT]
The account balance for John Doe is $1,234.56.

### ASSERT SemanticCondition
The answer mentions that the account balance is $1,234.56.

## [USER]
What about his credit score?

## [ASSISTANT]
His credit score is 750.

### ASSERT SemanticCondition
It mentions that the credit score is 750.

Each [ASSISTANT] turn can carry multiple ASSERT statements to verify different aspects of the response independently.

3. MCP Server Testing

Test Model Context Protocol servers end-to-end — same Markdown format, no extra APIs to learn:

# SCENARIO MCP Time Server

## [USER]
What time is it?

## [ASSISTANT]
It's currently 2:30 PM PST

### ASSERT FunctionCall
{
  "function_name": "current_time"
}

### ASSERT SemanticCondition
It mentions a specific time

Wire up your MCP server in the test setup:

var mcp = await McpClientFactory.CreateAsync(clientTransport);
var tools = await mcp.ListToolsAsync();

var baseChatClient = // your IChatClient (Azure OpenAI, OpenAI, etc.)

var agent = baseChatClient.AsAgent(
    instructions: "You are a helpful assistant that can answer questions and call tools.",
    tools: tools.ToArray());

await agent.ExecuteScenarioAsync(scenario, assertionClient: baseChatClient);
// assertionClient = the AI that evaluates responses semantically
// agent           = the system under test that actually calls tools

Need to test multiple MCP servers together? Merge their tool lists before creating the agent — see MCP Testing Guide.

4. Hallucination Mitigation with ScenarioRunOptions

LLM outputs vary between runs. Don't let a single flaky response break your build.

Run each scenario multiple times and require a minimum number of passes:

var options = new ScenarioRunOptions
{
    TotalRuns = 3,           // Run the scenario 3 times
    RequiredSuccessRuns = 2  // At least 2 must pass
};

await agent.ExecuteScenarioAsync(scenario, assertionClient: baseChatClient, options: options);

Recommended thresholds:

When a scenario falls below the threshold, you get a clear failure message:

Only 2 of 5 runs passed, which is below the required success runs of 4.

See Scenario Run Options for the full guide.

Setup

public class BankAccountTests
{
    private readonly ChatScenarioRunner ScenarioRunner;
    private readonly IChatClient systemUnderTestClient;

    public BankAccountTests(ITestOutputHelper output)  // xUnit
    {
        var assertionClient = new AzureOpenAIClient(endpoint, credential)
            .GetChatClient(deploymentName)
            .AsIChatClient();

        ScenarioRunner = new ChatScenarioRunner(assertionClient, output.WriteLine);
    }

    [Fact]
    public async Task TestBalance()
    {
        var agentUnderTest = /* the client/agent you are testing */
        var scenarios = ChatScenario.Parse(File.ReadAllText("balance-test.md"));
        await ScenarioRunner.RunAsync(scenarios, systemUnderTestClient);
    }
}

Two clients, two roles:

• assertionClient — evaluates semantic assertions (SemanticCondition, SemanticSimilar, etc.)
• agentUnderTest — the agent/client whose behavior you are testing

MSTest / NUnit: Replace ITestOutputHelper.WriteLine with TestContext.WriteLine. Everything else stays the same. See Test Framework Integration.

Documentation

Examples

See the /demos folder for complete, runnable projects:

• Moody Chef AI: A step-by-step tutorial project that shows how to build and test an AI agent using skUnit.

Requirements

• .NET 8.0 or higher
• An AI provider (Azure OpenAI, OpenAI, Anthropic, ...) for semantic assertions
• A test framework of your choice — xUnit, NUnit, or MSTest

Contributing

Contributions are welcome! Check open issues or submit a PR.