Move JS packages away from codebase

[michaelbaudino]

The JS packages should be stored in /node_modules (like Ruby gems are stored in /bundle) rather than in the codebase node_modules sub-directory.

This would avoid ignoring node_modules in docker-sync.yml and wouldn't pollute the host filesystem.

We need to tell Yarn to use /node_modules as modules folder for every Yarn command (as suggested in this issue). The following configuration line in .yarnrc is supposed to do exactly that:

--*.modules-folder "/node_modules"

Unfortunately, not all Yarn commands seem to honour this configuration and it seems to break the .bin folder, it sometimes wipes out node_modules completely, ...

This issue is kept open in order to follow progress on this matter.

ℹ️ The following (unscoped) shorthand is supposed to be the exact equivalent:

--modules-folder "/node_modules"

Unfortunately - once again - it seems still unsure whether it's actually behaving exactly equivalently or not 😕

---

ℹ️ Setting NODE_PATH to the same custom folder seems to be a good idea to support some legacy behaviour:

env:
  NODE_PATH "/node_modules"

When all those issues have been worked around, we may be able to improve and merge the code in the use-absolute-node_modules-folder branch 🤷‍♂️

[jmgarnier]

Hey @michaelbaudino ! Nice to see in the cyber space 😸

Can you help a poor backend dev lost in the yarn / webpack world?

I am trying to run with yarn 1.7.0:

cd client; NODE_ENV=production webpack --config webpack.test.config.js --modules-folder /home/runner/dossier/.semaphore-cache/client-node-modules; cd -

And it does not find any previously packages installed with

cd client; yarn install --modules-folder $SEMAPHORE_CACHE_DIR/client-node-modules; cd -

It sounds like yarn run does not support fully this option yet, correct?

[jmgarnier]

FYI, I think I solved this problem using symlinks.

[michaelbaudino]

Hi @jmgarnier 👋

Sorry for the delay, I'm currently on a paternity leave 👶 🍼

Symlinks sound like a smart solution to this, I didn't even think of trying it, tbh. You did something like ln -s $SEMAPHORE_CACHE_DIR/client-node-modules ./node_modules, I guess?

I'm glad you sorted it out, anyway 💯

[jmgarnier]

Hey Michael! Congrats 😄 🍾 👶 :)

Yes, indeed:

mkdir -p $SEMAPHORE_CACHE_DIR/node_modules
ln -s $SEMAPHORE_CACHE_DIR/node_modules $SEMAPHORE_PROJECT_DIR/client/node_modules

[Shigawire]

On a related sidenote: Due to bad performance on many systems with a non-native filesystem layer (macOS, Windows), mounting node_modules outside the project path is a great idea.
What I usually do is adding a particular volume just for node_modules cache to my docker-compose.yml:

version: "3.7"

services:
  app: &app
    build: 
      context: .
      dockerfile: Dockerfile
      target: build
    volumes:
      - bundle-cache:/bundle-cache
      - node-modules-cache:/app/node_modules
      - type: bind
        source: ./
        target: /app
        volume:
          nocopy: true
        consistency: delegated
volumes:
  bundle-cache:
  node-modules-cache:

This allows for node_modules to be cached, but not synchronized to the host (it's not required, anyways. node_modules should be idempotent and the yarn.lock and package-lock.json make sure you have repeatable results, anyways.

With such a setup (note the delegated consistency option), you don't need additional vodoo tools (rsync, etc.) for more performant file syncs between host and guests.

[michaelbaudino]

Thanks for the feedback @Shigawire 🙏

Yes, using a volume to store node_modules is what we recommend in the default setup too 👍

I've not been using MacOS for years, but I see that mount performance issues are still a work in progress 🤔 I was using docker-sync but as you said: it's another (voodoo 😅) tool to add to your stack.

But you are right: using delagated (or cached, fwiw, since they have never been fully implemented and are still doing exactly the same thing) reduces the performance issues. And since node_modules is usually where most of the IO happens, using a volume to store it is probably enough for 90% people/use-cases (coupled with the delegated strategy, indeed).

[Shigawire]

Thanks for the quick reply @michaelbaudino! Didn't expect this, on such an old issue.

I just realized you're suggesting mounting node_modules in a distinct volume.

But yes, Docker's fs performance on macOS is still a big deal. Also Docker networking is still super limited for more complex scenarios, which is particularly bad for running multiple apps in parallel that want to bind to the same port (https://docs.docker.com/docker-for-mac/networking/#known-limitations-use-cases-and-workarounds).

Btw. I also like your Dotfile-approach! Very cool, solving the real pains of developing inside containers.