Where Do Incremental Builds Currently Stand?

[johnhwright]

Documentation issue

• Reporting a typo
• Reporting a documentation bug
• Documentation improvement
• Documentation feedback

Is there a specific documentation page you are reporting?

Setup incremental builds for Angular applications

Publishable and Buildable Nx Libraries

Additional context or description

Was attempting to setup incremental builds for our angular monorepo (5 apps, 700+ libs), using pnpm

Was taking it one step at a time to keep track of what exactly was changing. First thing was to convert every library to be buildable, and added the node-gyp-build dependency (not even sure that one was necessary). But once that was complete it seemed like the incremental builds were immediately working without any of the other listed changes, no "postinstall", no .npmrc change (thankfully, as I really didn't want this one), no executor changes. It would now run the dependent tasks first and read from cache for the libs before attempting to build.

The performance however, was a little disappointing. With now the initial build taking substantially longer (8 minutes compared to less than a minute), and any smaller incremental builds taking around the same or slightly longer as non-incremental! For example, making a change in one library used by our largest app it would cause a rebuild of 5 of 508 libraries (Nx read the output from the cache instead of running the command for 503 out of 508 tasks), which was exciting to see, but then once it's done rebuild the small 5, it would still takes about ~35 seconds to construct the final bundle. Meanwhile without incremental builds that same change, which causes that entire app to be rebuilt, can still be done straight through in around ~20-25 seconds. Even with no changes the incremental version takes longer since it has to collect from the cache for 500 libraries which takes a few seconds (~4) compared to the less than 1 second when it retrieves the entire non-incremental build task from the cache.

Because of this I did go back and try the suggested @nx/angular:webpack-browser executor instead of the default @angular-devkit/build-angular:application . Note that you'll need to rename the options->browser property to main in order to use it (something the documentation doesn't mention). Added the postinstall script as suggested (which angular complains about). And tried with different values of buildLibsFromSource. But nothing ever really seemed to behave any differently or help the build times.

A bit rambly but I guess I'm just curious when exactly are incremental builds the correct choice? Closing in on a thousand libraries and the small changes we tend to make, I felt like we were going to be a perfect fit so I'm confused how rebuilding the entire app is still somehow faster. Maybe it's different when you're not running locally and can leverage cloud agents, but even that seems doubtful?

Regardless, curious where the docs stand on what actually needs to be changed for incremental builds, is there an actual reason to use @nx/angular:webpack-browser over @angular-devkit/build-angular:application? Is ngcc still necessary or just from a time before v16? And what about the other little changes listed that weren't needed (or are they)? And some guidance in the docs on when incremental builds are actually the right choice would be great.

Or, have I just gotten something terribly wrong and need to make some changes to unlock the real power of incremental builds?

[Coly010]

Hey.

You do not need ngcc. The docs should be updated to reflect that.
You also must use @nx/angular:webpack-browser for build and @nx/angular:dev-server for serve.
You must set buildLibsFromSource: false to actually use the built output from the buildable libraries.

In practice, getting substantial gains with incremental builds for Angular projects using Webpack can be more difficult than it should be.

This is because of how Angular compiles code.
Angular has a unique compilation that means that it has to run its Ivy compiler over everything one last time before emitting files.

So what happens with buildable libraries is this:

1. When you make your library buildable, you are removing the initial compilation of the library to its own process, outside the application's compilation.
2. The library gets compiled and the build artifact is output to a location on your machine
3. This repeats for all your buildable libraries.
4. The application will then build. It gets an updated tsconfig file that tells typescript that the files for the libraries can be found at their build artifact location.
5. Because they have already been compiled, the typescript compilation aspect should be faster than previous
6. However, Angular + Webpack still need to perform their compilation steps, and they still need to run over everything, buildable libraries included.
7. This adds an overhead to the theoretical gains of buildable libraries for Angular projects.

Because you've converted all your libraries, you have a situation where you are getting some speed benefits, but there's still the overhead impacting each. Not to mention that because there are so many of them, cold builds actually result in a lot more build processes needing to complete before the application build starts.

Due to this, we generally recommend that the best cases for buildable libraries are libraries that may be on a larger side and/or have little to low churn rate, so as to not invalidate the cache of it or many other libraries if they depend on it.

Something like a component library would be a great example of a library that could be turned into a buildable library.

[johnhwright]

@Coly010 Thanks for the well laid out response. That last pass from the ivy compiler really seems to be the undoing of it all in our case.

Amusingly it was recommended when we started this process that we use more of the single component per library (previously SCAM?) approach for our shared component libraries. This was specifically so that a change in a shared component wouldn't require a rebuild of the entire shared project (which was technically the case while serving, which has been feeding us false hope for our builds). Meanwhile, the varying app features themselves are usually a single larger library containing all of the necessary angular components. Which is unfortunately backwards from what would be needed to see gains from the incremental builds, since the larger features are what are most likely to receive changes compared to the individual hundred or so small shared component libraries.

Sadly I feel like at this point we'd really need to take step back and evaluate if our current aggressive use of libraries is actually beneficial anymore, and if we shouldn't consolidate them to the level needed for a module boundary. Though given Nx's suggested use of libraries it would still be difficult since most the majority of our shared components revolve around specific entities leading to directories that looked like this (with each child directory being a library)

fulfillment/
  data-access/
  ui-fulfillment-foo
  ui-fulfillment-bar
  ui-fulfillment-baz
application/
  data-access/
  ui-application-foo
  ui-application-bar
  ui-application-baz

So unless we wanted to give up our grouping folders we still wouldn't be able to combine all the components into a single lib, but just reduce any library type with multiple uses to a single library:

fulfillment/
  data-access/
  ui/
application/
  data-access/
  ui/

Because we're so wide with our number of involved entities it just might not be something we get to take advantage of yet. Food for thought though..

Thanks again!

[Coly010]

You could take advantage of secondary-entry-points to help achieve what you're looking for.

It would result in a single library still, but it would be split into multiple nested "entries".

You can learn more about it here: https://nx.dev/nx-api/angular/generators/library-secondary-entry-point
And there's a very detailed article here: https://olofens.medium.com/secondary-entry-points-in-an-angular-nx-library-8d01a80634cc