Best practices for manging code shared between projects?

[JoshuaEN]

Hello,

I work in a NX monorepo and we have a growing problem with how to organize our shared code.

We're wondering how other teams deal with shared code, particularly a lot of small pieces of independent code which are each used in a few different projects each.

---

If additional context might be helpful, in our particular case we have a general structure of (each leaf is a project):

apps
  foo
  bar
  ...

libs
  foo
    pageA
    pageB
    ...
    shared
  bar
    ...

The shared project for each app is where all of the code shared between two or more pages for that app ends up, which results in these shared projects being rather large, and causes projects to be unnecessarily affected.

A majority (~60%) of the code in these shared projects is fully independent (meaning a file being reference in one project doesn't correlate to another file also being referenced; e.g. a tab UI component, an Angular pipe for checking if an array includes a value, etc...).

Lacking a meaningful way of grouping these, the best option seems to be to make a project for each one.

However, in our experience there is a base overhead to running a task, so tripling the number of projects is a concern in terms of performance, to say nothing of the additional configuration (though Project Crystal may help there) and mental overhead of having so many projects.

We could reduce the total number projects by adding arbitrary groupings, perhaps of vaguely related code we don't expect to change very often (e.g. the generic Angular pipes, the generic Angular form validators), but that seems to lead down the same path because most of these independent pieces are used in maybe 7% of the pages for an app (and not the same 7%).

The other option which comes to mind is making projects with groupings based on the pages the code is used in (e.g. the pageA-and-pageC shared code, the pageG-and-pageO shared code, etc...). This though feels like a manual solution to something which would be better as an automated solution (e.g. linting and testing just the individual files which were affected).

All that said, we are curious what approaches other teams have to used to deal with similar problems.

Thanks,
Joshua

[jaysoo]

in our experience there is a base overhead to running a task, so tripling the number of projects is a concern in terms of performance

As long as the shared libs aren't buildable (e.g. have a build target), then the cost of additional projects shouldn't affect the apps that use the libs. The apps using the shared libs will treat them as if they are building from source, so no building libs shouldn't be necessary.

The other option which comes to mind is making projects with groupings based on the pages the code is used in (e.g. the pageA-and-pageC shared code, the pageG-and-pageO shared code, etc...). This though feels like a manual solution to something which would be better as an automated solution (e.g. linting and testing just the individual files which were affected).

I wouldn't group unrelated concerns into the same library just for performance.

[JoshuaEN]

As long as the shared libs aren't buildable (e.g. have a build target), then the cost of additional projects shouldn't affect the apps that use the libs. The apps using the shared libs will treat them as if they are building from source, so no building libs shouldn't be necessary.

The performance impact we see is with linting and testing.

For example, I ran eslint on 44 projects (in a private repository):

Run Type	Time (Local Dev Machine)	Time (Pipeline Agent)
single run of eslint CLI[1]	174.9s	219.3s
run per-a-project of eslint CLI[2]	410.1s	485.3s
nx run-many --parallel=1	365.7s[3]	466.0s[3]
nx run-many --parallel=4	121.2s	286.4s
nx run-many --parallel=8	91.5s	N/A[4]
nx run-many --parallel=16	93.8s[5]	N/A[4]

(--skip-nx-cache was used for all nx commands)

While Nx is faster with parallelization (compared to the single invocation), it is only around twice as fast running 8 tasks in parallel on the dev machine, and is slower on the pipeline agent (not that this is a bottleneck as linting can be split across agents as well).

This is of course ignoring a major benefit of Nx: affected often significantly reduces the total number of tasks which need to run overall.

My goal was simply to demonstrate there is measurable overhead to splitting code into separate projects.

Though, from some additional testing with a much simpler project, I suspect a significant portion of the overhead is from TypeScript needing to reprocess code imported from other projects repeatedly, whereas with the single global eslint run, each file only needs to be processed once.

With the shared library code generally being a leaf in the dependency graph, this should be less of an issue. And, by splitting up the shared libraries into smaller pieces, consuming projects will also have less TypeScript to process themselves. So, if this theory is right, there should be an overall savings in the worst-case time without even considering the improvements it will have on affected's ability to just mark projects which where really affected by a change.

I wouldn't group unrelated concerns into the same library just for performance.

I agree.

Thanks!

Footnotes

1. This uses a single global eslint config and tsconfig. We don't disable rules, only mark them as warn; for this test, the created global eslint config had all of the warn suppression from all of the projects linted.
2. Each project invocation used the project's ESLint config and tsconfig, the same as was configured in the project.json.
3. It is not clear to me why this is faster than the row before; they are conceptually similar, with I think the only difference being lintFilePatterns (and very little of our source code is not included in the lint file patterns).
4. The agent only had 4 cores.
5. It is a 16 core machine, but parallel 8 hits between 75% and 85% CPU usage from node alone.

[phihochzwei]

Maybe check on #9183 there also some people sharing their best practices