Conversation
This comment was marked as resolved.
This comment was marked as resolved.
There was a problem hiding this comment.
Pull request overview
Updates documentation (and one benchmark project) to quote --filter glob patterns so shell globbing doesn’t unexpectedly expand * before BenchmarkDotNet receives it.
Changes:
- Add quoting around
--filtervalues in various docs/examples. - Update
PowerShell.Benchmarks.csprojStartArgumentsto quote--filter.
Show a summary per file
| File | Description |
|---|---|
| src/tools/ResultsComparer/README.md | Quotes the --filter example in ResultsComparer usage. |
| src/benchmarks/real-world/PowerShell.Benchmarks/PowerShell.Benchmarks.csproj | Quotes --filter in StartArguments. |
| src/benchmarks/micro/README.md | Quotes --filter in microbenchmark CLI examples. |
| scripts/BENCHMARKS_LOCAL_README.md | Quotes --filter in benchmarks_local.py examples. |
| docs/microbenchmark-design-guidelines.md | Quotes --filter in an ETW profiling example. |
| docs/crank-to-helix-workflow.md | Quotes --filter inside bdnArgs example + description. |
| docs/benchmarking-workflow-dotnet-runtime.md | Quotes many --filter examples, including ones using shell variables. |
| docs/benchmarkdotnet.md | Updates the main filtering docs to use quoted --filter examples and refines the Unix-shell globbing note. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comments suppressed due to low confidence (7)
docs/crank-to-helix-workflow.md:107
- This bullet repeats the same problematic nested quoting as the example above: the inner single quotes will likely be preserved all the way to BenchmarkDotNet because the tooling splits args without shell-style quote processing. Recommend documenting
bdnArgs="--filter *Linq*"instead (outer quotes only).
- `--profile win-x64`: Configures crank to a local Windows x64 build of the runtime, and sets the Helix Queue to a Windows x64 queue.
- `--variable bdnArgs="--filter '*Linq*'"`: Sets arguments to pass to BenchmarkDotNet that will filter it to only Linq benchmarks
- `--profile msft-internal`: Sets the crank agent endpoint to the internal hosted crank agent
docs/benchmarking-workflow-dotnet-runtime.md:260
- Same issue as above:
--filter '$TestToRun*'in the Linux command prevents$TestToRunfrom expanding. Use double quotes so the variable expands but*is not pathname-expanded, e.g.--filter "$TestToRun*".
# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net11.0 --no-restore --no-build -- --filter '$TestToRun*' --anyCategories Libraries Runtime "" --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog "" --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200
docs/benchmarking-workflow-dotnet-runtime.md:141
--filter '$someFilter'is problematic in both common cases: in cmd.exe the single quotes are literal, and in bash the single quotes prevent$someFilterfrom expanding. Prefer--filter "$someFilter"(allows variable expansion but avoids glob expansion) or use a placeholder like<someFilter>without$if it's not meant to expand.
```cmd
dotnet run -c Release -f net11.0 --filter '$someFilter' \
--coreRun C:\Projects\runtime\artifacts\bin\testhost\net11.0-windows-Release-x64\shared\Microsoft.NETCore.App\11.0.0\CoreRun.exe
**docs/benchmarking-workflow-dotnet-runtime.md:298**
* This Windows section is in a ```cmd block; single quotes are not removed by cmd.exe, so `--filter '$TestToRun*'` will include quotes and may not match. Prefer `--filter *` (no quotes) or `--filter "..."` as needed; if `$TestToRun` is meant to expand, use the appropriate syntax for the documented shell.
Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net11.0 --no-restore --no-build -- --filter '$TestToRun*' --anyCategories Libraries Runtime "" --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog "" --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200
docs/benchmarking-workflow-dotnet-runtime.md:488
- These commands are shown in a ```cmd block but use single-quoted filters. In cmd.exe the quotes are literal; prefer double quotes (e.g.
"System.IO.Pipes*") or no quotes for Windows-only examples so the filter value passed to BenchmarkDotNet does not include `'` characters.
C:\Projects\performance\src\benchmarks\micro> dotnet run -c Release -f net11.0 \
--artifacts "C:\results\before" \
--coreRun "C:\Projects\runtime\artifacts\bin\testhost\net11.0-windows-Release-x64\shared\Microsoft.NETCore.App\11.0.0\CoreRun.exe" \
--filter 'System.IO.Pipes*'
docs/benchmarking-workflow-dotnet-runtime.md:547
- In a ```cmd block,
--filter '$YourFilter'will pass the single quotes through on cmd.exe, and on bash it prevents `$YourFilter` expansion if it’s meant to be a variable. Prefer double quotes (or a `` placeholder) depending on whether the text is intended as a real variable.
To profile the benchmarked code and produce an ETW Trace file ([read more](./benchmarkdotnet.md#Profiling)):
```cmd
dotnet run -c Release -f net11.0 --profiler ETW --filter '$YourFilter'
docs/benchmarking-workflow-dotnet-runtime.md:571
- This sentence suggests running all benchmarks using
--filter '*'. In cmd.exe, single quotes are literal, so this can be interpreted as a filter containing quotes and match nothing. Prefer documenting--filter *(Windows) or--filter "*"/--filter '*'explicitly as shell-specific alternatives.
When you identify and fix the regression, you should use [ResultsComparer](../src/tools/ResultsComparer/README.md) to confirm that you have solved the problem. Please remember that if the regression was found in a very common type like `Span<T>` and you are not sure which benchmarks to run, you can run all of them using `--filter '*'`.
- Files reviewed: 8/8 changed files
- Comments generated: 8
| ```cmd | ||
| dotnet run -c Release matrix decompress --input D:\results\Performance-Runs.zip --output D:\results\net7.0-preview3 | ||
| dotnet run -c Release matrix --input D:\results\net7.0-preview3 --base net7.0-preview2 --diff net7.0-preview3 --threshold 10% --noise 2ns --filter System.IO* | ||
| dotnet run -c Release matrix --input D:\results\net7.0-preview3 --base net7.0-preview2 --diff net7.0-preview3 --threshold 10% --noise 2ns --filter 'System.IO*' |
| <ImplicitUsings>enable</ImplicitUsings> | ||
| <AllowUnsafeBlocks>true</AllowUnsafeBlocks> | ||
| <StartArguments>--filter *</StartArguments> | ||
| <StartArguments>--filter '*'</StartArguments> |
| To filter the benchmarks using a glob pattern applied to namespace.typeName.methodName ([read more](../../../docs/benchmarkdotnet.md#Filtering-the-Benchmarks)): | ||
|
|
||
| ```cmd | ||
| dotnet run -c Release -f net11.0 --filter *Span* | ||
| dotnet run -c Release -f net11.0 --filter '*Span*' | ||
| ``` | ||
|
|
||
| To profile the benchmarked code and produce an ETW Trace file ([read more](../../../docs/benchmarkdotnet.md#Profiling)): | ||
|
|
||
| ```cmd | ||
| dotnet run -c Release -f net11.0 --filter $YourFilter --profiler ETW | ||
| dotnet run -c Release -f net11.0 --filter '$YourFilter' --profiler ETW | ||
| ``` |
| Here is an example command line that runs the MonoJIT RunType from a local runtime for the tests matching `*Span.IndexerBench.CoveredIndex2*`: | ||
|
|
||
| `python .\benchmarks_local.py --local-test-repo "<absolute path to runtime folder>/runtime" --run-types MonoJIT --filter *Span.IndexerBench.CoveredIndex2*` | ||
| `python .\benchmarks_local.py --local-test-repo "<absolute path to runtime folder>/runtime" --run-types MonoJIT --filter '*Span.IndexerBench.CoveredIndex2*'` | ||
|
|
||
| Here is an example command line that runs the MonoInterpreter and MonoJIT RunTypes using commits `dd079f53` and `69702c37` for the tests `*Span.IndexerBench.CoveredIndex2*` with the commits being cloned to the `--repo-storage-path` for building, it also passes `--join` to BenchmarkDotNet so all the reports from a single run will be joined into a single report: | ||
|
|
||
| `python .\benchmarks_local.py --commits dd079f53b95519c8398d8b0c6e796aaf7686b99a 69702c372a051580f76defc7ba899dde8fcd2723 --repo-storage-path "<absolute path to where you want to store runtime clones>" --run-types MonoInterpreter MonoJIT --filter *Span.IndexerBench.CoveredIndex2* *WriteReadAsync* --bdn-arguments="--join"` | ||
| `python .\benchmarks_local.py --commits dd079f53b95519c8398d8b0c6e796aaf7686b99a 69702c372a051580f76defc7ba899dde8fcd2723 --repo-storage-path "<absolute path to where you want to store runtime clones>" --run-types MonoInterpreter MonoJIT --filter '*Span.IndexerBench.CoveredIndex2*' '*WriteReadAsync*' --bdn-arguments="--join"` | ||
|
|
|
|
||
| ```cmd | ||
| dotnet run -c Release -f netcoreapp3.1 --filter *.Reverse --profiler ETW | ||
| dotnet run -c Release -f netcoreapp3.1 --filter '*.Reverse' --profiler ETW |
| ```cmd | ||
| crank --config .\helix.yml --scenario micro --profile win-x64 --variable bdnArgs="--filter *Linq*" --profile msft-internal --variable buildNumber="myalias-20230811.1" | ||
| crank --config .\helix.yml --scenario micro --profile win-x64 --variable bdnArgs="--filter '*Linq*'" --profile msft-internal --variable buildNumber="myalias-20230811.1" | ||
| ``` |
| #### Filtering the Benchmarks | ||
|
|
||
| You can filter the benchmarks using `--filter $globPattern` console line argument. The filter is **case insensitive**. | ||
| You can filter the benchmarks using `--filter '$globPattern'` console line argument. The filter is **case insensitive**. | ||
|
|
||
| The glob patterns are applied to full benchmark name: namespace.typeName.methodName. Examples (all in the `src\benchmarks\micro` folder): | ||
|
|
||
| - Run all the benchmarks from BenchmarksGame namespace: | ||
|
|
||
| ```cmd | ||
| dotnet run -c Release -f net9.0 --filter BenchmarksGame* | ||
| dotnet run -c Release -f net9.0 --filter 'BenchmarksGame*' |
|
|
||
| ```cmd | ||
| dotnet run -c Release -f net11.0 --filter System.Memory* | ||
| dotnet run -c Release -f net11.0 --filter 'System.Memory*' |
|
@copilot apply changes based on the comments in this thread |
No description provided.