-
Notifications
You must be signed in to change notification settings - Fork 295
Add MAUI iOS Inner Loop measurements for CI #5187
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
davidnguyen-tech
wants to merge
69
commits into
dotnet:main
Choose a base branch
from
davidnguyen-tech:nguyendav/maui-ios-inner-loop
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
69 commits
Select commit
Hold shift + click to select a range
727782e
Add iOSInnerLoopParser, wire Startup.cs, fix Reporter.cs
davidnguyen-tech ee23cf0
Add shared Python infrastructure for iOS inner loop
davidnguyen-tech 664f061
Add mauiiosinnerloop scenario directory (pre/test/post.py)
davidnguyen-tech e20e8a4
Add setup_helix.py for iOS Helix bootstrap
davidnguyen-tech cb6315e
Add maui_scenarios_ios_innerloop.proj Helix workitem definition
davidnguyen-tech 7c9e0c6
Wire iOS inner loop into CI pipeline YAML and routing
davidnguyen-tech 0d86772
Add physical device support for iOS inner loop
davidnguyen-tech edab33f
Fix review findings: devicectl commands, UDID regex, edit paths, Xcod…
davidnguyen-tech 9942170
Quality fixes: tempfile.mkstemp, devicectl fallback, report guard
davidnguyen-tech d21e9b7
[TEMP] Disable all pipeline jobs except iOS Inner Loop
davidnguyen-tech 25a2846
Fix iOS Inner Loop build output capture and logging
davidnguyen-tech 53e647e
Capture and print dotnet build output for iOS Inner Loop
davidnguyen-tech bb1e6d2
Increase iOS Inner Loop Helix timeout to 2.5 hours
davidnguyen-tech 565f8eb
Bypass Xcode version validation for iOS Inner Loop
davidnguyen-tech 60d8954
Fix iOS simulator RID for Intel x64 Helix machines
davidnguyen-tech 259c10e
Fix FileExistsError when copying traces on incremental iterations
davidnguyen-tech f62020a
Remove verbose build output from iOS Inner Loop
davidnguyen-tech 62aecd2
Fix ILLink error on Helix machines with older Xcode
davidnguyen-tech 68d6b4c
Re-enable all pipeline jobs after iOS inner loop CI validation
davidnguyen-tech ba4e0d1
Add install timing, CoreCLR config, and device support for iOS inner …
davidnguyen-tech 60f1106
Add fallback for workload install version skew in pre.py
davidnguyen-tech 995fe94
Add workload install fallback to setup_helix.py
davidnguyen-tech ae4c34c
Exclude simulator work item from device jobs
davidnguyen-tech 3fbe535
Clean up iOS inner loop: unify ioshelper API, simplify runner and pos…
davidnguyen-tech 92a538a
Fix device job failures: iOSRid propagation and setup_helix exit hand…
davidnguyen-tech 5c092e4
Pass iOSRid as explicit MSBuild property to fix device job dispatch
davidnguyen-tech edc2ccb
Rename simulator job identifiers for clarity
davidnguyen-tech d8b4ff1
Switch ioshelper.py from simctl/devicectl to mlaunch for install/launch
davidnguyen-tech c398150
Use --installsim for granular simulator install timing
davidnguyen-tech 8e89a1f
Re-select Xcode to match iOS SDK's required version after workload in…
davidnguyen-tech 5d80e1f
Resolve booted simulator UDID for mlaunch and remove ValidateXcodeVer…
davidnguyen-tech 6756d03
Run --launchsim as background process to prevent timeout
davidnguyen-tech 803ec21
Add post-build code signing for physical device deployment
davidnguyen-tech 2bcbb64
Make Xcode handling diagnostic-only on Helix — match Device Startup s…
davidnguyen-tech c2a65e3
Add --skip-manifest-update to workload install retry on Helix
davidnguyen-tech 4253fcd
Resolve sign tool path for Helix device deployment
davidnguyen-tech bf97725
Remove /usr/local/bin from PATH - shadows venv python
davidnguyen-tech 259573f
Restore active Xcode selection in setup_helix.py — fixes Xcode versio…
davidnguyen-tech 6d7dfc8
Fix sign tool resolution — fall back to login shell execution
davidnguyen-tech e74cb12
Fix Xcode version parsing — regex greedily captured trailing dot
davidnguyen-tech 15ed711
Use in-box manifest for workload install — avoid Xcode version skew
davidnguyen-tech 189b555
Use in-box manifest on build agent — stop shipping incompatible SDK p…
davidnguyen-tech 482d22e
Revert to --from-rollback-file and make Xcode diagnostic-only
davidnguyen-tech c8ca867
Add timeout to workload install to prevent NuGet download hangs
davidnguyen-tech 2a6cdf7
Temporarily pin iOS workload to 26.2 for Xcode compatibility
davidnguyen-tech 2f8fd09
Add active Xcode selection and version validation to setup_helix.py
davidnguyen-tech c29f631
fixup! Add active Xcode selection and version validation to setup_hel…
davidnguyen-tech 9f4ddab
Fix Xcode selection: match workload-required version, not highest
davidnguyen-tech b859032
fixup! Fix Xcode selection: match workload-required version, not high…
davidnguyen-tech 2c1d0e7
Remove temporary workload pin — restore dynamic NuGet feed resolution
davidnguyen-tech ece266a
Remove silent workload fallback — fail loud if nightly packs unavailable
davidnguyen-tech c380cbb
Add manifest patching fallback for missing net10.0 packs
davidnguyen-tech c64ec2f
Fix review findings: packs type handling, timeouts, diagnostics
davidnguyen-tech ca4f030
Document Helix manifest-patching dependency in setup_helix.py
davidnguyen-tech 4148296
Fix iOS inner loop local device support
davidnguyen-tech 7ca397b
Improve code quality: deduplicate workload install, fix correctness b…
davidnguyen-tech 28f94ac
Handle missing sign tool gracefully for local device deployment
davidnguyen-tech afe6351
Add run-local.sh for local iOS inner loop measurements
davidnguyen-tech 625c595
Add SDK-aware Xcode selection for local measurements
davidnguyen-tech 5882cfa
Preserve binlogs in results directory
davidnguyen-tech b3f8ed6
Copy versions.json to results directory
davidnguyen-tech e2657fb
Uninstall stale app from physical device before first deploy
davidnguyen-tech c3d1ab6
Clean stale app/traces dirs before setup to fix XAML duplicate errors
davidnguyen-tech a0d7d63
Validate cold startup measurement before including in results
davidnguyen-tech 321dcb9
Track ILLink, CompileNativeCode, R2R targets in iOS parser
davidnguyen-tech bb15078
Fix correctness bugs found during code review
davidnguyen-tech 78cb6b6
Measure .app bundle size in iOS inner loop results
davidnguyen-tech e2f7b89
Fix watchdog event ordering and find_app_bundle simulator/device pref…
davidnguyen-tech 5844d6a
Add restore timeout to prevent hung NuGet restores from consuming Hel…
davidnguyen-tech File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,111 @@ | ||
| <Project Sdk="Microsoft.DotNet.Helix.Sdk" DefaultTargets="Test"> | ||
|
|
||
| <Import Project="Scenarios.Common.props" /> | ||
|
|
||
| <!-- No XHarness needed — iOS inner loop uses xcrun simctl directly for | ||
| simulator deploy and does not use the XHarness test infrastructure. --> | ||
|
|
||
| <PropertyGroup> | ||
| <!-- Map RuntimeFlavor to the UseMonoRuntime MSBuild property. | ||
| Same pattern as maui_scenarios_ios.proj. --> | ||
| <_MSBuildArgs Condition="'$(RuntimeFlavor)' == 'mono'">/p:UseMonoRuntime=true</_MSBuildArgs> | ||
| <_MSBuildArgs Condition="'$(RuntimeFlavor)' == 'coreclr'">/p:UseMonoRuntime=false</_MSBuildArgs> | ||
|
|
||
| <!-- RuntimeIdentifier: simulator by default. Physical device (ios-arm64) | ||
| can be set via pipeline parameter when device support is enabled. | ||
| Default is iossimulator-x64 because the Mac.iPhone.17.Perf Helix | ||
| queue runs on Intel x64 machines (e.g. DNCENGMAC045). --> | ||
| <iOSRid Condition="'$(iOSRid)' == ''">iossimulator-x64</iOSRid> | ||
| <_MSBuildArgs>$(_MSBuildArgs) /p:RuntimeIdentifier=$(iOSRid)</_MSBuildArgs> | ||
| <!-- Device builds: disable automatic code signing during 'dotnet build'. | ||
| Signing is handled post-build by ioshelper.sign_app_for_device() using | ||
| the provisioning profile and 'sign' tool pre-installed on the Helix | ||
| machines (same flow as maui_scenarios_ios.proj device startup). --> | ||
| <_MSBuildArgs Condition="'$(iOSRid)' == 'ios-arm64'">$(_MSBuildArgs) /p:EnableCodeSigning=false</_MSBuildArgs> | ||
| <!-- Disable managed linker for inner loop builds. Debug configuration | ||
| doesn't need linking, and it avoids ILLink errors (MT0180) on Helix | ||
| machines with older Xcode versions that lack the required SDK headers. --> | ||
| <_MSBuildArgs>$(_MSBuildArgs) /p:MtouchLink=None</_MSBuildArgs> | ||
|
|
||
| <RunConfigsString>$(RuntimeFlavor)_Default</RunConfigsString> | ||
| <InnerLoopIterations Condition="'$(InnerLoopIterations)' == ''">3</InnerLoopIterations> | ||
| </PropertyGroup> | ||
|
|
||
| <!-- Remove .NET SDK packs from the Helix correlation payload to reduce | ||
| transfer size. The maui-ios workload packs (~1-2 GB) are installed on | ||
| the Helix machine by setup_helix.py, so shipping them is unnecessary | ||
| and would exceed Helix payload limits. --> | ||
| <Target Name="RemoveDotnetFromCorrelationStaging" BeforeTargets="BeforeTest"> | ||
| <Message Text="Removing Dotnet Packs from Correlation Staging" Importance="high" /> | ||
| <RemoveDir Directories="$(CorrelationPayloadDirectory)dotnet/packs" /> | ||
| </Target> | ||
|
|
||
| <ItemDefinitionGroup> | ||
| <HelixWorkItem> | ||
| <!-- iOS builds are slower than Android (AOT compilation, code signing, | ||
| native linking) and the workitem also includes workload install + | ||
| NuGet restore on the Helix machine. --> | ||
| <Timeout>02:30</Timeout> | ||
| </HelixWorkItem> | ||
| </ItemDefinitionGroup> | ||
|
|
||
| <ItemGroup> | ||
| <MAUIiOSInnerLoopScenario Include="MAUI iOS Inner Loop"> | ||
| <ScenarioDirectoryName>mauiiosinnerloop</ScenarioDirectoryName> | ||
| <PayloadDirectory>$(ScenariosDir)%(ScenarioDirectoryName)</PayloadDirectory> | ||
| </MAUIiOSInnerLoopScenario> | ||
| </ItemGroup> | ||
|
|
||
| <!-- PreparePayloadWorkItem: runs on the build agent (macOS). | ||
| Selects the highest versioned Xcode, then runs pre.py to create the MAUI | ||
| app template, strip non-iOS TFMs, and prepare modified source files for | ||
| the incremental deploy simulation. NuGet restore is NOT done here — | ||
| packages are restored on the Helix machine by setup_helix.py. --> | ||
| <ItemGroup> | ||
| <PreparePayloadWorkItem Include="@(MAUIiOSInnerLoopScenario)"> | ||
| <!-- Select highest versioned Xcode_*.app and avoid runner-image symlink | ||
| aliases that do not work with the iOS SDK (same pattern as | ||
| maui_scenarios_ios.proj PreparePayloadWorkItem). --> | ||
| <Command>XCODE_PATH=`find /Applications -maxdepth 1 -type d -name 'Xcode_*.app' | sort -t_ -k2 -V | tail -1` && echo "Selected Xcode: $XCODE_PATH" && sudo xcode-select -s "$XCODE_PATH" && $(Python) pre.py default -f $(PERFLAB_Framework)-ios</Command> | ||
| <WorkingDirectory>%(PreparePayloadWorkItem.PayloadDirectory)</WorkingDirectory> | ||
| </PreparePayloadWorkItem> | ||
| </ItemGroup> | ||
|
|
||
| <!-- Environment variables for the macOS Helix machine. | ||
| Must be set via shell 'export' in PreCommands because env vars set via | ||
| os.environ in a Python script (setup_helix.py) do not persist to | ||
| subsequent commands (test.py, post.py) in the Helix shell session. --> | ||
| <PropertyGroup> | ||
| <_MacEnvVars>export DOTNET_ROOT=$HELIX_CORRELATION_PAYLOAD/dotnet;export DOTNET_CLI_TELEMETRY_OPTOUT=1;export DOTNET_MULTILEVEL_LOOKUP=0;export NUGET_PACKAGES=$HELIX_WORKITEM_ROOT/.packages;export PATH=$HELIX_CORRELATION_PAYLOAD/dotnet:$PATH</_MacEnvVars> | ||
| </PropertyGroup> | ||
|
|
||
| <!-- Simulator workitem: only when NOT targeting physical devices. | ||
| When iOSRid=ios-arm64, _MSBuildArgs bakes in the device RID which | ||
| would produce an ARM64 binary incompatible with the simulator. --> | ||
| <ItemGroup Condition="'$(iOSRid)' != 'ios-arm64'"> | ||
| <HelixWorkItem Include="@(MAUIiOSInnerLoopScenario -> 'Inner Loop Simulator - %(Identity)')"> | ||
| <PreCommands>$(_MacEnvVars);export IOS_RID=$(iOSRid);$(Python) setup_helix.py $(PERFLAB_Framework)-ios "$(_MSBuildArgs)" || exit $?</PreCommands> | ||
| <Command>$(Python) test.py iosinnerloop --csproj-path app/MauiiOSInnerLoop.csproj --edit-src "src/MainPage.xaml.cs;src/MainPage.xaml" --edit-dest "app/Pages/MainPage.xaml.cs;app/Pages/MainPage.xaml" --bundle-id com.companyname.mauiiosinnerloop -f $(PERFLAB_Framework)-ios -c Debug --msbuild-args "$(_MSBuildArgs)" --device-type simulator --inner-loop-iterations $(InnerLoopIterations) --scenario-name "%(Identity)" $(ScenarioArgs)</Command> | ||
| <PostCommands>export IOS_RID=$(iOSRid);$(Python) post.py</PostCommands> | ||
| <DownloadFilesFromResults>output.log</DownloadFilesFromResults> | ||
| </HelixWorkItem> | ||
| </ItemGroup> | ||
|
|
||
| <!-- Physical device workitem: runs on macOS machines with USB-connected | ||
| iPhones (Mac.iPhone.17.Perf queue). | ||
| Code signing is disabled during 'dotnet build' (EnableCodeSigning=false) | ||
| and handled post-build: ioshelper copies embedded.mobileprovision into | ||
| the .app and runs the Helix-provided 'sign' tool — same flow as | ||
| maui_scenarios_ios.proj device startup scenarios. --> | ||
| <ItemGroup Condition="'$(iOSRid)' == 'ios-arm64'"> | ||
| <HelixWorkItem Include="@(MAUIiOSInnerLoopScenario -> 'Inner Loop Device - %(Identity)')"> | ||
| <PreCommands>$(_MacEnvVars);export IOS_RID=$(iOSRid);$(Python) setup_helix.py $(PERFLAB_Framework)-ios "$(_MSBuildArgs)" || exit $?</PreCommands> | ||
| <Command>$(Python) test.py iosinnerloop --csproj-path app/MauiiOSInnerLoop.csproj --edit-src "src/MainPage.xaml.cs;src/MainPage.xaml" --edit-dest "app/Pages/MainPage.xaml.cs;app/Pages/MainPage.xaml" --bundle-id com.companyname.mauiiosinnerloop -f $(PERFLAB_Framework)-ios -c Debug --msbuild-args "$(_MSBuildArgs)" --device-type device --inner-loop-iterations $(InnerLoopIterations) --scenario-name "%(Identity)" $(ScenarioArgs)</Command> | ||
| <PostCommands>export IOS_RID=$(iOSRid);$(Python) post.py</PostCommands> | ||
| <DownloadFilesFromResults>output.log</DownloadFilesFromResults> | ||
| </HelixWorkItem> | ||
| </ItemGroup> | ||
|
|
||
| <Import Project="PreparePayloadWorkItems.targets" /> | ||
|
|
||
| </Project> | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The comment says the inner loop scenario uses
xcrun simctldirectly for simulator deploy, but the implementation usesmlaunch --installsim/--launchsimfor deploy/launch (withsimctlmainly for boot/terminate/uninstall). Please update the comment to match the actual tooling so future maintainers aren’t misled about how deploy is performed.