Skip to content

Guide: Adding a New Industry

This guide provides a step-by-step process for adding a new industry to the MultiAgentEval.

Step 1: Create the Directory Structure

  1. Navigate to the industries/ directory in the root of the project.
  2. Create a new folder for your industry. The name should be lowercase and use underscores instead of spaces (e.g., real_estate, not Real Estate).
  3. Inside your new industry folder, create two sub-folders:
    • scenarios
    • datasets

Step 2: Add Your First Scenario

  1. Inside the scenarios folder, you may want to create sub-folders for different use cases (e.g., crop_management in the agriculture industry).
  2. Create your first scenario .json file. The name should be numbered and descriptive (e.g., 01_mortgage_application_check.json).
  3. Populate the JSON file using the structure defined in the Evaluation Guide. Use existing scenarios as a reference.

Step 3: Add The Core Functions Description to the Core Functions Guide

  1. Locate the alphabetic spot for your industry (e.g., aerospace industry resides between accounting industry and agriculture industry).
  2. Add your new industry, its use cases and core functions, by following the markdown format of the other industries. These constructs relate to the industry, use_case, and core_function fields in your JSON files.
  3. Define the core functions to document their scope.

Step 4: Include Synthetic Datasets

Realistic environments require data. If your scenarios require data (e.g. a CSV of customer transactions), you should link them via the dataset attribute. To easily generate highly realistic dummy data for your new industry, use the builtin generation script: 

python scripts/generate_industry_datasets.py --industry YOUR_INDUSTRY
This script uses faker libraries to create a records.csv with industry-specific schemas (e.g., policy numbers for insurance, flight routes for airlines) and places it directly in your datasets/ folder for immediate use.

All scenarios are validated at load time against schemas/scenario.schema.json. Your scenario must include these required fields:

Scenario level: - scenario_id, title, description, use_case, core_function, industry, tasks - (Optional) initial_state, policies

Task level (each item in tasks): - task_id, description, expected_outcome, success_criteria - (Optional) expected_state_changes

Run schema validation locally: 

python -c "from eval_runner.loader import load_scenario; from pathlib import Path; load_scenario(Path('industries/YOUR_INDUSTRY/scenarios/YOUR_SCENARIO.json'))"

Step 5: Verify in Visual Debugger

Launch the dashboard to ensure your new industry and scenarios are appearing correctly in the library:

multiagent-eval console

Navigate to the Library tab to view your JSON structure and trigger a test run visually.

Step 6: Submit a Pull Request

Once you have created the directory structure and added at least one complete scenario, please submit a Pull Request. In the description, briefly explain the industry and the use cases you are starting to build out.

Thank you for helping expand the reach of this project!