Unable to build TypeScript program using private scope modules

[cmidgley]

Build environment: Windows (I believe any)
Moddable SDK version: Requires recent version that supports private scope module build system changes
Target device: Any

Description
Private scope module resolution with Moddable requires an artificial path to create a common root (such as implemented by mcpack), which breaks TypeScript compilation. For example, to define the private scope package #example one might use a manifest.json containing a modules section similar to this:

"modules": {
	"root/main": "./src/main",
	"root/#example": "./src/example/index",
	"example/do-something": "./src/example/do-something"
}

The artificial path component root is used to assist module resolution in translating between the file system pathing of JS and the namespace resolution used in Moddable.

mcconfig then carries these artificial roots over to the generated tsconfig.json file:

"compilerOptions": {
	"paths": {
		"root/main": ["/some/path/main"],
		"root/#example": ["/some/path/example/index"],
		"example/do-something": ["/some/path/example/do-something"]
	}
}

This results in a TypeScript compilation error such as:

# tsc tsconfig.json
src/main.ts:1:29 - error TS2792: Cannot find module '#example'. Did you mean to set the 'moduleResolution' option to 'nodenext', or to add aliases to the 'paths' option?

1 import { doSomething } from '#example';
                              ~~~~~~~~~~

Steps to Reproduce

1. Clone this repo
2. Switch to the "typescript" branch
3. Run mcconfig -m -d

Thoughts

I have not come up with an obvious/clean way to indicate to mcconfig that the root is artificial. While the best solution may be to change module resolution to not need artificial roots (easier to implement/understand for devs), that feels like a high-risk and low-reward endeavor. My solution has been to change the typescript / compiler option in manifest.json to run a small script that front-ends tsc and removes the artificial roots from tsconfig.json. This is working well for me, but it's a hack and not generic enough to be useful for others.

[phoddie]

Thanks for the example.

On macOS, the main branch works but the typescript branch fails as follows.

# tsc tsconfig.json
src/main.ts:1:29 - error TS2792: Cannot find module '#example'. Did you mean to set the 'moduleResolution' option to 'node', or to add aliases to the 'paths' option?

1 import { doSomething } from '#example';

I assume this matches what you see. We'll take a look.

[cmidgley]

Yes, and this is because the tsconfig.json has the paths set to root/#module rather than what TypeScript needs which is just #module. My hack post-processor is a simple filter to remove the root/, but that isn't a great solution (though it works, so I'm not held up at all by this).

[phoddie]

As the support for private scope modules was introduced as part of the work on mcpack let's start there before coming back to mcconfig.

Like Node, mcpack does not handle TypeScript sources. You have to run tsc before mcpack. See $MODDABLE/examples/packages/ts-hello for an example and the explanations in $MODDABLE/examples/packages/readme.md.

That uses a tsconfig.json to map files and provide typings for XS and the Moddable SDK. Here attached is private-scope-example-2 that works with:

cd /path/to/private-scope-example-2
npm install
npm run build
mcpack mcconfig -d -m

There is some support for TypeScript in mcconfig, which generates a tsconfig.json file when there are TypeScript sources. The generated tsconfig.json file maps files according to modules in all manifests. That works in most case. When it does not, you can add your own paths in the manifest itself. For instance:

"build": {
	"SRC": "./src"
},
"typescript": {
	"tsconfig": {
		"compilerOptions": {
			"paths": {
				"#example": [ "$(SRC)/example/index" ]
			}
		}
	}
}

Here is private-scope-example-3 that builds with:

cd /path/to/private-scope-example-3
mcconfig -d -m

That revealed a bug in mcconfig. It needs to escape # in make files for TypeScript sources, like it already does for JavaScript sources. The fix for that is included in Moddable SDK 4.9.0.

Finally, about the "artificial path”. Private module specifiers look absolute but are relative to their package. XS does not support Node packages at runtime, so mcpack handles packages and aliases at build time. The “common root” is the package name, and of course there can be several packages that use the same private modules specifiers.

You are definitely working at the outer limits so this stuff is a definitely a bit subtle. I hope this helps.

[cmidgley]

Thanks for taking the time to look at this. I agree that I'm on the outer limits - so I appreciate your time all the more!

I can't use mcpack for my use case because I'm not using NPM modules. I use private scope module specifiers within my application because I need to define imports that can reference different implementations (source files) based on the target platform (Node vs. Moddable, and even simulator vs. ESP32). This way I can import "#SomeLibrary" and the mappings will associate the correct source code to compile for the target platform.

I have an extensive platform portability library that provides all the cross-platform services I depend on in Moddable with compatible implementations in Node (and also do things like simulate Flash on simulator using LittleFS and emulate NOR functionality). My system supported shared code (across all platforms) and XS-ESP32 specific code, XS-SIM specific code, and node-specific code as needed. This allows me to use Jest (on Node, and my private Jest implementation on XS), and the amazing WallabyJS product, to execute my hundreds (soon > 1000) unit and integration tests on Node with rapid iteration, on simulator for fast XS validation, and then on the true target platform (ESP32) for final validation, all 100% automatically (which, FYI, is why I asked for those xsbug-log changes!) It's an amazing development environment, that amplifies the value proposition and time-to-market for microcontrollers, all thanks to Moddable's unbelievable JS compatibility.

This is all working perfectly now that we have private scopes working with Moddable for JavaScript (thank you!), with the one hiccup being that the Moddable-generated tsconfig.json file isn't compatible (per the above). Since Moddable generates this file, I can't pre-compile this with tsc as it doesn't exist until I start the build, but using a post-processor works great (where I replace the tsc compiler command name in my manifest.json with a front-end that forks to tsc and then reformats the generated tsconfig.json to solve this issue).

If my understanding is correct, anybody who wants private scope modules (not using NPM packages) with TypeScript on Moddable will encounter this issue. But seeing as I have a solution that works, and nobody else appears to be using TypeScript with private scope modules right now, perhaps it's best to drop this issue until others also encounter it? Especially since there isn't an obvious answer (at least to me), aside from a post-processor/filter.

Thanks again. I'll leave it up to you if you want to close this or leave it dangling open for future reference.

[ScreamZ]

You can have a look at #1434 (comment) I created a starter template that simplifies development in module resolution in TS.

Also check HipsterBrown/xs-dev#186