[dotnet test] Add androidtest project template and dotnet run instrumentation support#10862
Open
jonathanpeppers wants to merge 31 commits intomainfrom
Open
[dotnet test] Add androidtest project template and dotnet run instrumentation support#10862jonathanpeppers wants to merge 31 commits intomainfrom
androidtest project template and dotnet run instrumentation support#10862jonathanpeppers wants to merge 31 commits intomainfrom
Conversation
rmarinho
reviewed
Mar 3, 2026
…trumentation support Context: #10683 Add a new `androidtest` .NET project template that creates an Android test project using MSTest with Microsoft.Testing.Platform. The template includes a TestInstrumentation class that runs tests via `am instrument` and reports results (passed/failed/skipped) back through Android's instrumentation protocol. Add `--instrument` option to Microsoft.Android.Run so `dotnet run` can launch test projects via `am instrument -w` instead of `am start`. Fix an issue where NuGet packages like MSTest add assemblies as both properly-resolved publish assets (with DestinationSubPath metadata) and as None items (without DestinationSubPath) that flow into `_SourceItemsToCopyToPublishDirectory`. When NuGet conflict resolution arbitrarily picks the None-based copy, the assembly loses its DestinationSubPath and TargetPath metadata, causing it to either be missing from the APK or deployed to the wrong directory on device. The fix conditionally removes only items without DestinationSubPath, then re-adds any assemblies that were completely stripped with a cleared TargetPath so that downstream targets (ProcessAssemblies, FastDeploy) can set the correct per-architecture paths. Changes: - New `androidtest` template under src/Microsoft.Android.Templates/ - New GetAndroidInstrumentationName MSBuild task to resolve the instrumentation runner class from AndroidManifest.xml - Updated Microsoft.Android.Sdk.Application.targets to pass `--instrument` when EnableMSTestRunner is true - Added AdbHelper class to Microsoft.Android.Run for shared adb process management - Added `--instrument` CLI option to Microsoft.Android.Run/Program.cs - Fixed _ComputeFilesToPublishForRuntimeIdentifiers to preserve properly-resolved publish assets and recover missing assemblies - Added DotNetNewAndroidTest device integration test TODO: - In a future PR, add support for `dotnet test` end-to-end.
jonathanpeppers
force-pushed
the
dev/peppers/androidtest
branch
from
16a61dc to
f121c7c
Compare
Check pe.HasMetadata before calling GetMetadataReader() to avoid InvalidOperationException when processing native DLLs (e.g. from BenchmarkDotNet/TraceEvent NuGet packages) that don't have .NET metadata. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jonathanpeppers
added a commit
to jonathanpeppers/iOSDotNetTest
that referenced
this pull request
Mar 10, 2026
MSTest test platform assemblies fail AOT compilation because the IL linker trims types (DoesNotReturnAttribute, MemberNotNullWhenAttribute) from Microsoft.TestPlatform.CoreUtilities that are still referenced via custom attributes in other test platform DLLs. Use MtouchInterpreter=all to bypass AOT and run all assemblies through the interpreter instead. See: dotnet/android#10862 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
18 tasks
…tifiers Compare by filename instead of full ItemSpec path when checking if assemblies from _SourceItemsToCopyToPublishDirectory already exist in ResolvedFileToPublish. Native DLLs like KernelTraceControl.dll (from Microsoft.Diagnostics.Tracing.TraceEvent NuGet) can exist under different paths, causing the old Remove-by-path to miss them and re-add duplicates that crash GetPerArchAssemblies with XALNS7004. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…meIdentifiers When _SourceItemsToCopyToPublishDirectory contains multiple items with the same filename from different source paths (e.g., KernelTraceControl.dll), _MissingAssemblyNames ends up with duplicate entries. These duplicates get re-added to ResolvedFileToPublish, causing a Dictionary duplicate key error in GetPerArchAssemblies (XALNS7004). Use RemoveDuplicates task to deduplicate by filename before re-adding, following the same pattern already used for NativeAOT deduplication. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
NuGet packages like microsoft.diagnostics.tracing.traceevent (a
dependency of BenchmarkDotNet) include native Windows DLLs under
lib/native/{arch}/ paths. These were flowing through
_SourceItemsToCopyToPublishDirectory into _MissingAssemblyNames and
being re-added to ResolvedFileToPublish. When the linker tried to
load them with Mono.Cecil, it threw BadImageFormatException
(XALNS7000/XA0009).
Filter out items whose OriginalItemSpec contains a /native/ path
segment, which covers both lib/native/ and runtimes/*/native/ NuGet
layout conventions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
NuGet packages like Microsoft.Testing.Extensions.CodeCoverage ship native .so files under runtimes/linux-x64/native/. The .NET SDK stamps RuntimeIdentifier=android-arm64 metadata on all ResolvedFileToPublish items during inner builds, causing ProcessNativeLibraries to treat these non-Android libraries as valid Android native libs. This leads to spurious XA0141 warnings about 16 KB page alignment. Use %(PathInPackage) metadata to detect the actual source RID and exclude .so files from non-Android runtimes (linux-x64, linux-musl-x64, osx-x64, win-x64, etc.) while preserving android-* and linux-bionic-* RIDs which are valid Android targets. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jonathanpeppers
added a commit
that referenced
this pull request
Mar 16, 2026
…native Windows DLLs (#10938) Context: #10862 ## Summary Extracted from PR #10862: Add `PEReader.HasMetadata` checks before calling `GetMetadataReader()` across all code paths that use `PEReader` to prevent exceptions when processing native (non-.NET) PE DLLs. ## Problem When NuGet packages (like MSTest) include native Windows `.dll` files that flow into `ResolvedFileToPublish`, calling `PEReader.GetMetadataReader()` on these files throws an exception because they are not .NET assemblies and have no CLI metadata. ## Fix Added `PEReader.HasMetadata` guards before `GetMetadataReader()` calls across all relevant code paths. In MSBuild tasks with log access, a `LogDebugMessage()` call is included when skipping non-.NET assemblies. Co-authored-by: jonathanpeppers <840039+jonathanpeppers@users.noreply.github.com>
…esToPublishForRuntimeIdentifiers Instead of removing all _SourceItemsToCopyToPublishDirectory items and then recovering missing assemblies through temp item groups and dedup, just set DestinationSubPath on .dll/.pdb items that lack it before the Remove. Items with DestinationSubPath survive; everything else is stripped as before. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Some NuGet packages (e.g. MSTest.TestFramework) add assemblies as @(Reference) items with CopyToOutputDirectory metadata via their .targets files. This causes the assemblies to flow into @(_SourceItemsToCopyToPublishDirectory) in addition to the normal reference resolution path, creating duplicate entries in @(ResolvedFileToPublish) that crash downstream tasks like GetPerArchAssemblies with 'duplicate key' errors. Add _PatchNuGetReferenceMetadata target that strips CopyToOutputDirectory from all @(Reference) items during the inner per-RID publish build, so they only flow through normal reference resolution. This allows reverting _ComputeFilesToPublishForRuntimeIdentifiers back to the simple Remove of @(_SourceItemsToCopyToPublishDirectory). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CancelledTestNodeStateProperty is obsolete (CS0618). Test frameworks should throw OperationCanceledException using the cancellation token passed by Microsoft.Testing.Platform instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Some NuGet packages (e.g. MSTest.TestAdapter) add assemblies as both
@(Reference) items (via buildTransitive .targets) and @(None) items.
The @(None) copy flows into @(_SourceItemsToCopyToPublishDirectory)
and creates a duplicate in @(ResolvedFileToPublish). NuGet conflict
resolution picks arbitrarily between them ('Could not determine winner
due to equal file and assembly versions'). When the @(None)-based copy
wins, _ComputeFilesToPublishForRuntimeIdentifiers removes it via
Remove=@(_SourceItemsToCopyToPublishDirectory), and the assembly is
completely missing from the APK.
Fix by removing @(None) items that match @(Reference) HintPaths in
_PatchNuGetReferenceMetadata, so they never enter the publish pipeline
as duplicates.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This reverts commit 27796e3.
MSTest.TestAdapter adds MSTestAdapter.PlatformServices.dll and Microsoft.TestPlatform.AdapterUtilities.dll only as @(None) items with CopyToOutputDirectory (not as @(Reference)) when EnableMSTestRunner=true. These flow exclusively through @(_SourceItemsToCopyToPublishDirectory) and get blanket-removed from @(ResolvedFileToPublish) by _ComputeFilesToPublishForRuntimeIdentifiers, causing them to be missing from the APK at runtime. Fix by promoting @(None) assembly items with CopyToOutputDirectory to @(Reference) in _PatchNuGetReferenceMetadata, then stripping CopyToOutputDirectory from all References so they flow through normal reference resolution instead of _SourceItemsToCopyToPublishDirectory. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…Newest The previous condition matched any non-empty CopyToOutputDirectory value, which would incorrectly promote items with CopyToOutputDirectory=Never. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Unqualified %(CopyToOutputDirectory) requires all @(None) items to define the metadata. Items like AndroidManifest.xml don't have it, causing MSB4096. Qualify with %(None.CopyToOutputDirectory) so MSBuild treats missing metadata as empty instead of erroring. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tion NuGet packages like microsoft.diagnostics.tracing.traceevent add native Windows DLLs (KernelTraceControl.dll, msdia140.dll, etc.) as @(None) items with CopyToOutputDirectory=PreserveNewest and %(Link) containing a platform-specific subdirectory (e.g. amd64\KernelTraceControl.dll). The _PatchNuGetReferenceMetadata target was promoting ALL @(None) .dll items to @(Reference), causing ResolveAssemblyReferences to emit MSB3246 warnings for these native DLLs (PE image does not have metadata). Filter by checking if %(Link) equals its own filename — items with a subdirectory in %(Link) are platform-specific and cannot be used on Android, which packs assemblies flat into a single native library. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds first-class support for Android device test projects driven by MSTest + Microsoft.Testing.Platform by introducing a new androidtest template and enabling dotnet run to launch via Android instrumentation (am instrument) instead of launching an activity.
Changes:
- Adds an
androidtestproject template that includes anInstrumentationentry point which runs tests and reports results via instrumentation bundles. - Extends
Microsoft.Android.Runwith a new--instrumentmode foram instrument -wexecution. - Updates MSBuild targets to locate instrumentation runners from the manifest and adjusts inner-build publish/assembly resolution behavior for certain NuGet package asset patterns.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/MSBuildDeviceIntegration/Tests/InstallAndRunTests.cs | Adds a device integration test validating dotnet new androidtest + dotnet run instrumentation output. |
| src/Xamarin.Android.Build.Tasks/Tasks/GetAndroidInstrumentationName.cs | New MSBuild task to extract the instrumentation runner from the manifest. |
| src/Xamarin.Android.Build.Tasks/Microsoft.Android.Sdk/targets/Microsoft.Android.Sdk.AssemblyResolution.targets | Adds a target that mutates @(None)/@(Reference) metadata to influence publish asset selection. |
| src/Xamarin.Android.Build.Tasks/Microsoft.Android.Sdk/targets/Microsoft.Android.Sdk.Application.targets | Wires up the new task and passes --instrument to Microsoft.Android.Run when applicable. |
| src/Microsoft.Android.Templates/androidtest/* | New androidtest template files (csproj, manifest, tests, instrumentation runner, template config). |
| src/Microsoft.Android.Run/Program.cs | Adds --instrument option and async instrumentation execution path. |
| src/Microsoft.Android.Run/AdbHelper.cs | Introduces shared adb process start/run helpers. |
src/Xamarin.Android.Build.Tasks/Tasks/GetAndroidInstrumentationName.cs
Outdated
Show resolved
Hide resolved
src/Xamarin.Android.Build.Tasks/Tasks/GetAndroidInstrumentationName.cs
Outdated
Show resolved
Hide resolved
...d.Build.Tasks/Microsoft.Android.Sdk/targets/Microsoft.Android.Sdk.AssemblyResolution.targets
Show resolved
Hide resolved
jonathanpeppers
commented
Apr 2, 2026
Member
Author
jonathanpeppers
left a comment
jonathanpeppers
left a comment
There was a problem hiding this comment.
🤖 AI Review Summary
Verdict:
CI Status: All checks are still pending/in-progress.
Found 4 issues: 1 error, 3 warnings
- ❌ MSBuild tasks:
Log.LogErrorwithout XA error code in GetAndroidInstrumentationName (GetAndroidInstrumentationName.cs:29,36) ⚠️ Nullable: Missing#nullable enablein new AdbHelper.cs (AdbHelper.cs:1)⚠️ Thread safety: Non-atomic counter increments in TestInstrumentation template with parallel execution (TestInstrumentation.cs:93)⚠️ Error handling: Empty catch blocks in RunInstrumentationAsync (Program.cs:235,247)
👍 What's good:
- Clean refactor of sync
RunAdb→ asyncAdbHelper.RunAsyncwith proper async/await throughout - Well-documented
_PatchNuGetReferenceMetadatatarget with clear comments explaining the NuGet assembly resolution fix - Good use of
XmlReader(streaming) and.IsNullOrEmpty()NRT extension in the new MSBuild task - Comprehensive
DotNetNewAndroidTestintegration test covering pass/fail/skip scenarios - Clean mutual exclusion between
--activityand--instrumentwith clear user-facing error messages
Review generated by android-reviewer from review guidelines.
src/Xamarin.Android.Build.Tasks/Tasks/GetAndroidInstrumentationName.cs
Outdated
Show resolved
Hide resolved
src/Microsoft.Android.Templates/androidtest/TestInstrumentation.cs
Outdated
Show resolved
Hide resolved
- GetAndroidInstrumentationName: use Log.LogCodedError with XA1042/XA1043, load manifest via AndroidAppManifest.Load following GetAndroidActivityName pattern, use AndroidAppManifest.AndroidXNamespace - TestInstrumentation: use Interlocked.Increment for thread-safe counters since tests run with method-level parallelization - Program.cs: replace ConfigureAwait(SuppressThrowing) with try/catch for OperationCanceledException, log exceptions in cleanup catch blocks when verbose - InstallAndRunTests: remove unused processExited variable, add process.WaitForExit() after timeout to drain async output events Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Apr 2, 2026
GetAndroidInstrumentationName runs during ComputeRunArguments (before build), so it reads the source AndroidManifest.xml, not the merged one. The [Instrumentation] C# attribute only generates the element during build (_GenerateJavaStubs). Add it to the source manifest so dotnet run can find it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This reverts commit cfccb64.
The [Instrumentation] C# attribute generates the <instrumentation> element in the merged manifest during Build (_GenerateJavaStubs). ComputeRunArguments runs before Build, so the merged manifest doesn't have <instrumentation> yet. Fix by: 1. Adding Build and DeployToDevice to _AndroidComputeRunArgumentsDependsOn when EnableMSTestRunner=true, ensuring the merged manifest exists 2. Making GetAndroidInstrumentationName return empty instead of erroring when no <instrumentation> element is found (matching the pattern used by GetAndroidActivityName) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
GenerateMainAndroidManifest writes the manifest with <instrumentation> (from the [Instrumentation] C# attribute) to \AndroidManifest.xml The manifest merger then copies it to: \android\AndroidManifest.xml but appears to strip the <instrumentation> element. Read the pre-merger manifest instead, which is guaranteed to have the element if AddInstrumentations found the type. Also reverts the Build;DeployToDevice dependency workaround which is no longer needed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context: #10683
Add a new
androidtest.NET project template that creates an Android test project using MSTest with Microsoft.Testing.Platform. The template includes a TestInstrumentation class that runs tests viaam instrumentand reports results (passed/failed/skipped) back through Android's instrumentation protocol.Add
--instrumentoption to Microsoft.Android.Run sodotnet runcan launch test projects viaam instrument -winstead ofam start.Fix an issue where NuGet packages like MSTest add assemblies as both properly-resolved publish assets (with DestinationSubPath metadata) and as None items (without DestinationSubPath) that flow into
_SourceItemsToCopyToPublishDirectory. When NuGet conflict resolution arbitrarily picks the None-based copy, the assembly loses its DestinationSubPath and TargetPath metadata, causing it to either be missing from the APK or deployed to the wrong directory on device. The fix conditionally removes only items without DestinationSubPath, then re-adds any assemblies that were completely stripped with a cleared TargetPath so that downstream targets (ProcessAssemblies, FastDeploy) can set the correct per-architecture paths.Changes:
androidtesttemplate under src/Microsoft.Android.Templates/--instrumentwhen EnableMSTestRunner is true--instrumentCLI option to Microsoft.Android.Run/Program.csTODO:
dotnet testend-to-end.