Skip to content

DOC: Add scenario doc pages for all 7 undocumented scenarios#1558

Open
varunj-msft wants to merge 3 commits intomicrosoft:mainfrom
varunj-msft:varunj-msft/7933-Migrate-cookbooks-to-scenarios
Open

DOC: Add scenario doc pages for all 7 undocumented scenarios#1558
varunj-msft wants to merge 3 commits intomicrosoft:mainfrom
varunj-msft:varunj-msft/7933-Migrate-cookbooks-to-scenarios

Conversation

@varunj-msft
Copy link
Copy Markdown
Contributor

Description

PR 1 of 3 for User Story 7933 - Migrate Cookbooks to Scenarios.

Creates a doc page for each of the 7 scenarios.

New pages:

2_content_harms — ContentHarms (7 harm categories)
3_psychosocial — Psychosocial (crisis escalation, therapist impersonation). Incorporates key concepts from cookbook 5: rubric design, MHFA/ALGEE methodology, conversation-level scoring via create_conversation_scorer.
4_cyber — Cyber (malware/exploitation)
5_jailbreak — Jailbreak (template-based prompt injection)
6_leakage — Leakage (IP/sensitive data). Incorporates key concepts from cookbook 3: copyright testing with FirstLetterConverter, PlagiarismScorer with LCS/Levenshtein/Jaccard metrics.
7_scam — Scam (phishing/fraud)
8_garak_encoding — GarakEncoding (17 encoding-based probes)
Each page follows the pattern from 1_configuring_scenarios.py: overview, strategy table, CLI usage, programmatic usage, result interpretation.

Also updates doc/myst.yml TOC with the 7 new entries.

Subsequent PRs will migrate remaining cookbook content (PR 2) and delete the cookbooks directory with cross-reference fixes (PR 3).

Tests and Documentation

This is a documentation-only PR. No source code or test changes.

All 7 notebooks were executed locally committed with outputs. Execution times ranged from 23s (Jailbreak) to ~2m (ContentHarms, Psychosocial). Ruff check passes on all files. All .py/.ipynb pairs round-trip correctly.

@hannahwestra25
Copy link
Copy Markdown
Contributor

wondering if we should add integration tests for this (obviously we should skip most for the day to day integration tests). If not, we should add this to the release notes that these should be validated

@hannahwestra25
Copy link
Copy Markdown
Contributor

we should also update the configuring scenarios docs bc they're specific to foundry so I would add another redteam scenario notebook and maybe make that one more generic or merge it into the 0_scenarios files

Copy link
Copy Markdown
Contributor

@ValbuenaVC ValbuenaVC left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is our plan for documenting changes to scenarios? For extensibility should we add something about it to a style guide or release notes

@jsong468
Copy link
Copy Markdown
Contributor

jsong468 commented Apr 2, 2026

wondering if we should add integration tests for this (obviously we should skip most for the day to day integration tests). If not, we should add this to the release notes that these should be validated

As long as each notebook is running a small subset of objectives/strategies, it might be okay to run regularly like our other notebooks? @varunj-msft probably has better grasp on how long these take though

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants