Opt out of 'project crystal' zero configuration and support the legacy way as 1st class citizen

[beeman]

Description

I want to be able to upgrade to the latest Nx without making changes required by 'project crystal'.

In addition, I want to be able to create new projects that don't use the 'magic' inference but allow me to manage my project.json configurations.

Motivation

The projects I'm working on are all using Nx Generators and expect certain files to be in certain places to make edits.

With the new magic inference, there's nothing left to edit, and it's completely unclear how someone should go about making updates.

Take for example the following situation:

I want to generate a Next.js project and set the port to 3000

Pre 'crystal':

• npx create-nx-workspace@latest -> generate a Next.js project
• locate project.json, go to targets.serve.options, and set your port.
• done. You can explain this to anyone with a few simple instructions

Post 'crystal'

• npx create-nx-workspace@latest -> generate a Next.js project
• you're now lost as the project.json is empty, no instructions or docs on how to create the blob that was previously there
• The user is blocked?

Any old instructions or knowledge is useless. We're now back to manual managing of stuff again, so ironically 'crystal' completely defeats the purpose and makes it more difficult - except for those that can run with the defaults.

Suggested Implementation

Give us an option to bail out of too much magic. It feels like you're cutting off a limb that worked totally fine.

Alternate Implementations

I understand that Nrwl is trying hard to gain market share and make Nx easier for the newbies, but please take into account that there are a lot of developers who are perfectly capable of managing a few more lines of JSON. Especially considering the current alternative.

I tried to keep an open mind but so far, I don't like this change at all, please consider making it optional.

Edit

For context, I got dragged into this rabbit whole because I can't start a new Next.js project. I shared a repro here.

Edit 2

After some more toying around, I figured you can use the nx show project command to generate it for you, and move on with your life as if nothing ever happened 😮‍💨

In this example, I want to generated a regular project.json for my project called web:

pnpm nx show project web | jq > apps/web/project.json

The command above depends on a cli tool called jq but that's just for the formatting. You can leave it out and format the JSON with prettier and that should work too.

Request for NX: a project-wide toggle to enable this behavior by default.

[stephenwade]

I'm having a similar issue. I don't want to use Project Crystal, and I currently can't upgrade my project to the latest version of Nx because I can't opt out.

The upgrade migration added NX_ADD_PLUGINS=false to my .env file which does opt out, but that isn't a good solution because my .env file is ignored in Git.

Please make opting out be a setting in nx.json.

[TriPSs]

Just wondering here, but, you could still keep the targets right? Everything is merged together, yes the project.json is empty (if you use their plugin everything will still be visible) but you can still set targets.serve.options and it will be merged together with the defaults.

[beeman]

I'm having a similar issue. I don't want to use Project Crystal, and I currently can't upgrade my project to the latest version of Nx because I can't opt out.

The upgrade migration added NX_ADD_PLUGINS=false to my .env file which does opt out, but that isn't a good solution because my .env file is ignored in Git.

Please make opting out be a setting in nx.json.

Indeed, the feature request here is to make the regular way of doing things a first-class citizen.

Considering Nx team members told me "it won't go anywhere" is reassuring, the thing that's missing for me is an option in nx.json that just scaffolds the projects in 'non-dummy mode' for people who are willing and able to manage a few more lines of JSON.

[beeman]

Just wondering here, but, you could still keep the targets right?

Sure, for an existing workspace, you can keep them. However, for new workspaces, I want to be able to generate them in 'full mode', not in 'dummy mode'.

Everything is merged together, yes the project.json is empty (if you use their plugin everything will still be visible) but you can still set targets.serve.options and it will be merged together with the defaults.

Well, this might be true, but I have no clue how. Another sign that this release has been rushed is the lack of proper docs.

I spent about an hour yesterday to try and configure a Next.js port override and didn't manage to do it using project.json. The ways to make it work are the PORT env var and --port param, but this is clearly not the same.

I want to set these things in my project so they are always enforced and not have to instruct people using the project that you need to recall to tack on the --port param.

[frantisekhanzlikbl]

Just chiming in to say that there really needs to be a way to at least partially opt out of Crystal. While I personally like the changes and intend to leverage Crystal as much as I can, it can get in the way. Many times there are inferred tasks that make no sense and there is no way to disable them, which breaks CI since run-all now runs stuff it shouldn't. Also, thanks for all the work on NX. I know it must be hard with so many issues flooding in right now. 🙂

[stephenwade]

#21980 fixes this issue, available in v18.1.1.

Users can set useInferencePlugins: false in nx.json to avoid adding plugins

[rexkenley]

This flag doesn't seem to work 100%.

Our custom plugins fail with "weird text" in the console output unless we have the .env NX_ADD_PLUGINS=false

*After updating the package lock, the weird text came back.

[simondotm]

As a community plugin author, I've watched the recent videos about crystal and the latest changes in Nx 18, and while I haven't yet started to deep think about supporting v18 yet, I feel pretty uncertain about it tbh.

My plugin generates and then customises the targets for a boilerplate nx node application, so if some of this project configuration may or may not be inferred somehow based on user workspace settings, my maintenance and testing surface area is going to become unfeasible for my little non-commercial hobby project to work on.

I'm going to keep a close eye on how these conversations progress. Thank you @beeman for raising this concern, I share your misgivings.

[rudfoss]

This flag doesn't seem to work 100%.

Our custom plugins fail with "weird text" in the console output unless we have the .env NX_ADD_PLUGINS=false

*After updating the package lock, the weird text came back.

This might be the same as #22731

[beeman]

As a community plugin author, I've watched the recent videos about crystal and the latest changes in Nx 18, and while I haven't yet started to deep think about supporting v18 yet, I feel pretty uncertain about it tbh.

My plugin generates and then customises the targets for a boilerplate nx node application, so if some of this project configuration may or may not be inferred somehow based on user workspace settings, my maintenance and testing surface area is going to become unfeasible for my little non-commercial hobby project to work on.

I fully share this concern. Maybe for somebody that manages 1 project you can either opt in or opt out, and move on with your life.

For plugin authors or people that work with other teams that use Nx, there is no way you can ignore it. You now have to support both ways of doing things so you're forced into it and that sucks.

In my case, few of my client teams are now looking at alternative options, which means I have to support things like Turbo repo if I want to keep working with them. Far from great.

I'm going to keep a close eye on how these conversations progress.

One of the issues is that there has been zero conversation from the Nx team here on the GitHub issues. It's been totally ignored, for those that don't have the luxury to be close enough to the team. I find this concerning.

It's been a few months, I gave it a try and I still can't like it. It's fully unclear why this is an improvement and for whom. I heard nobody who is excited about this change, and heard a lot of complaints and breakage. There's never been a more disrupting change in the Nx history.

It still feels like I'm using a Linux box with an empty /etc folder that's now magically 'inferred'. Not something I would ever choose to use.

It's unlikely for any team to revert something like this, but I really wish the Nrwl would make an exception and make 'magic mode' opt-in, and leave the 'normal' mode as the default.

Ironically, Nx has always been proud about 'dog-fooding' their own solutions, but if you look at this repo you will find that none of their packages use 'crystal', they all use the old mode. This is very telling that it's not a good change.

I can rant about it more, but in the end of the day I just wish this would have never happend. My trust in Nx as the stable base it always has been is damaged. :(

[FrozenPandaz]

Hi all, I really appreciate the feedback given here.

Just chiming in to say that there really needs to be a way to at least partially opt out of Crystal. While I personally like the changes and intend to leverage Crystal as much as I can, it can get in the way. Many times there are inferred tasks that make no sense and there is no way to disable them, which breaks CI since run-all now runs stuff it shouldn't. Also, thanks for all the work on NX. I know it must be hard with so many issues flooding in right now. 🙂

Good news for you. Not only did we add the flag to disable adding plugins in the first place. Plugins can also now be configured with include and exclude to scope them to certain areas of the repo. We seem to be missing documentation about it but we'll be sure get that documented.

My plugin generates and then customises the targets for a boilerplate nx node application, so if some of this project configuration may or may not be inferred somehow based on user workspace settings, my maintenance and testing surface area is going to become unfeasible for my little non-commercial hobby project to work on.

For plugin authors or people that work with other teams that use Nx, there is no way you can ignore it. You now have to support both ways of doing things so you're forced into it and that sucks.

I understand that your plugin as it is today does not support projects which do not have the target configured in project.json. While we did make changes so that if other generators used our generators, they would NOT get the new changes, there's little we could do about the current state of the workspace generators are run within.

There are changes you may have to make to your plugin to be fully compatible with Nx 18. We'd be happy to explore different solutions with you and help you get your plugin compatible as soon as possible. Please reach out.

Just like you, we also maintain plugins and reducing the maintenance burden is something that was not only considered but a deliberate goal of Project Crystal. Project Crystal directly addresses Plugin maintenance in several ways:

• Plugins reduce the amount of configuration committed to the repo. Apart from being cleaner and less maintenance for the user this helps us avoid writing migrations entirely
• It will be easier to configure workspaces in a way where tasks will be run faster without incurring a lot of extra code to maintain for the user. Splitting E2E tests is a clear example of this.
• There will be less generator and executor code for us to maintain. For existing plugins, we will continue to support our existing generators and executors but for future plugins, Project Crystal decreases the effort of creating a new plugin as well as maintaining it down the line.
• Because plugins calculate configuration dynamically, we can support a wider range of versions without having multiple versions of generators or executor implementation.

---

One of the issues is that there has been zero conversation from the Nx team here on the GitHub issues. It's been totally ignored, for those that don't have the luxury to be close enough to the team. I find this concerning.

Sorry, that we've been pretty quiet. We are actually actively focusing on addressing Github issues ever since Project Crystal was launched. Nearly all of our effort has been devoted to fixing issues people have found with Project Crystal. We are pumping the brakes a bit on delivering new features to improve the stability of Nx.

---

Ironically, Nx has always been proud about 'dog-fooding' their own solutions, but if you look at this repo you will find that none of their packages use 'crystal', they all use the old mode. This is very telling that it's not a good change.

You are definitely correct. Dogfooding is definitely something core to our identity. The fact that we are not dogfooding Project Crystal in the Nx repo honestly brings me great shame. We definitely do intend to dogfood Project Crystal to the full extent here in the Nx repo and you will see more of it soon. We ran into some issues along the way and have used that experience to make the fixes required.

---

It's unlikely for any team to revert something like this, but I really wish the Nrwl would make an exception and make 'magic mode' opt-in, and leave the 'normal' mode as the default.

I can rant about it more, but in the end of the day I just wish this would have never happend. My trust in Nx as the stable base it always has been is damaged. :(

The entire team, and me especially, is really excited about what Project Crystal brings and will bring to Nx. I am really sad that you feel this way and would like to do whatever I can do to fix it. By all means, please rant. I am here to listen and would like to make something constructive out of it to improve your experience.

Thank you,

• Jason Jean

[JakeGinnivan]

FWIW I have upgraded an internal Terraform plugin I have to use project crystal. I was able to write a migration to drop out a bunch of targets (apply, plan, output etc) which did not change any of the default options and now I can drop an existing Terraform program into a repo, add a package.json with the version and then I can run it as part of my NX repo.

There are definitely been a few rough edges but overall I like the direction and what dynamic targets bring.

[FrozenPandaz]

YES! This is exactly the kind of importing we wanted to enable. Thank you so much for sharing!

[ironsm4sh]

Hello, I would also like to leave a comment in this conversation.

I have been using NX for a few years, and it always did exactly what I wanted. It allowed me to combine front- and back-end into a single repository with shared libraries. Adding new apps or libraries was as simple as running a NX command, and sometimes, manually tweaking a configuration file.

NX 18 (project Crystal) completely broke my workflow, with no obvious way to configure anything. I expect the default boilerplate code to contain features such as the dynamic 'Environment.ts', which allowed me to run command such as nx run my-app:build:development, nx run my-app:build:production. Sometimes I would even add additional environments such as nx run my-app:build:acceptation. When I generate a new project using NX 18, the entire Environment.ts file is gone, which is sad, but acceptable if I could manually add it. However, if I open the project.json, where I would normally configure the fileReplacements -- its completely empty!?

That is just one example. How do I configure fileReplacements, Assets, outputPath or add additional configurations in NX 18? There is no obvious way to change anything, at all. NX_ADD_PLUGINS=false fixes (most of) these things, but, like most of the changes above, its hardly documented anywhere. On top of that, its nearly impossible to look it up on Google or Stackoverflow, because all keywords are associated with the old situation.

Why doesn't the interactive TUI (when using: npx create-nx-workspace@latest) ask the user about using project crystal? The same question should be asked about the NX Daemon, which adds no value to many small projects, but does cause headaches when attempting to run npm ci or other commands that touch the node_modules directory

[FrozenPandaz]

NX 18 (project Crystal) completely broke my workflow, with no obvious way to configure anything. I expect the default boilerplate code to contain features such as the dynamic 'Environment.ts', which allowed me to run command such as nx run my-app:build:development, nx run my-app:build:production. Sometimes I would even add additional environments such as nx run my-app:build:acceptation. When I generate a new project using NX 18, the entire Environment.ts file is gone, which is sad, but acceptable if I could manually add it. However, if I open the project.json, where I would normally configure the fileReplacements -- its completely empty!?

That is just one example. How do I configure fileReplacements, Assets, outputPath or add additional configurations in NX 18? There is no obvious way to change anything, at all. NX_ADD_PLUGINS=false fixes (most of) these things, but, like most of the changes above, its hardly documented anywhere. On top of that, its nearly impossible to look it up on Google or Stackoverflow, because all keywords are associated with the old situation.

As you're saying, modifying these settings has indeed changed with Project Crystal. It should now be done within the webpack.config.js file instead. We have webpack configuration utilities which make it just as simple to change this, just in a place that is more specific to webpack.

@jaysoo Can you please help here on how to change fileReplacements, assets, and outputPath via the webpack configuration?

As for NX_ADD_PLUGINS=false, we changed it to a useInferencePlugins setting in nx.json so that it would be easier to apply for workspaces which do not commit .env files. This is documented here: https://nx.dev/concepts/inferred-tasks#existing-nx-workspaces

Why doesn't the interactive TUI (when using: npx create-nx-workspace@latest) ask the user about using project crystal? The same question should be asked about the NX Daemon, which adds no value to many small projects, but does cause headaches when attempting to run npm ci or other commands that touch the node_modules directory

We can considering adding a flag to create-nx-workspace. Perhaps `create-nx-workspace --no-inference-plugins?

[beeman]

I would love to see this flag in create-nx-workspace for those that frequently create new workspaces as it's one thing less to configure to make it work with the explicit way.

[Ambroos]

I don't want to pile on, but I'll add that Crystal has also made me very wary of continuing my Nx adoption. My "would you recommend Nx to friends and family" feeling has also dropped from an easy yes to a probably not.

I've always really liked Nx for how explicit it is. You can read the configuration JSONs and know why Nx is running things the way it's running things (in terms of caching and dependencies). Project Crystal takes that away by moving things into inferred tasks. New plugin versions could change behavior, which means I need to go check the inferred values for things when upgrading Nx plugins. On top of that, I'll also need to go in and check the inferred values when making significant changes to any of the configuration files Nx infers from. Or I don't check and hope for the best, until something breaks. I'm either looking at extra work or extra uncertainty and I don't want either.

I think this is one of those cases where Crystal will be less work for basic projects and repositories, but actually more work for large monorepos. There's a reason why tools like Bazel, Buck and Pants all rely on very explicit configuration files. Magic does not scale.

Migrations etc are hard to write, I understand that, but I think the Nx team is underestimating the value they have for Nx users: it makes it immediately clear what has changed when they are applied. Crystal loses that benefit.

It's made my work quite a bit more stressful. None of us who maintain our monorepo wants to adopt Crystal because of the concerns I mentioned above about a lack of explicit configuration. Unfortunately opting out of Crystal also feels like something that's not sustainable if it is going to become the Nx default.

For now I think we're going to settle on using Nx in a very lightweight manner: only using the capabilities that something like Turborepo also offers, with almost no plugins and task inference completely turned off. Those feel like they'll remain useable for at least a while, and it also means we wouldn't have to deal with a massive migration if we did move away.

[FrozenPandaz]

New plugin versions could change behavior, which means I need to go check the inferred values for things when upgrading Nx plugins.

This would be as much of a breaking change as changing the executor implementation without migrating the project configuration in project.json.

1. We would NOT make those changes outside of a Major version update.
2. We also add migrations to maintain existing behavior in existing workspaces.

I will also add, that this in general should be less likely. The targets defined by crystal plugins are slimmer and defer more to the tools themselves. The playwright plugin for example, just invokes playwright test in the project root. The outputs are populated by the configuration and as long as playwright is configured to output somewhere, the plugin will configure that. Compared to the executor which could just as easily have changed how playwright was invoked without any configuration changes, there should be less cases when changes to plugins would change behavior for existing workspaces.

On top of that, I'll also need to go in and check the inferred values when making significant changes to any of the configuration files Nx infers from. Or I don't check and hope for the best, until something breaks. I'm either looking at extra work or extra uncertainty and I don't want either.

Making significant changes to tooling configuration remains a task which should have a good amount of scrutiny with it. When making significant changes, to the tooling configuration, you'll of course need to make sure the tool still works as you intend it. How the configuration changes the underlying tool does not change with Project Crystal.

Ensuring that Nx picks this up properly is something I feel has gotten easier with Project Crystal.

With executors and configuration in project.json, you would need to ALSO manually update the settings in project.json if you wanted Nx to behave differently if that tool. To be honest, I will shamefully admit that I personally always forget to do this 😅. When I'm updating the configuration of a tool, configuring Nx is an afterthought for me.

With plugins, Nx will reflect the changes made to the tooling configuration without any manual action. Developers can focus on configuring the tool, ensuring the tool works, and Nx will pick it up.

At this time, it is completely understandable that it may feel uncertain. It's new and you've yet to trust the new plugins. We can discuss some ways to make it even more crystal clear what plugins are doing. The PDV screen: https://nrwl-nx-examples-dep-graph.netlify.app/#/project-details/cart aims to make this crystal clear. We've done our best to surface this as much as we feel necessary. It's also still new and there are more opportunities for it to do more to make it more clear. If you have any ideas, we would definitely be interested in hearing them.

For now I think we're going to settle on using Nx in a very lightweight manner: only using the capabilities that something like Turborepo also offers, with almost no plugins and task inference completely turned off. Those feel like they'll remain useable for at least a while, and it also means we wouldn't have to deal with a massive migration if we did move away.

This is actually a very important use case of Nx that we consider all the time. It will absolutely remain useful for the foreseeable future.

[Ambroos]

@FrozenPandaz Have you considered using Crystal as a generator/migration engine to actually create/update project.json files instead? Rather than use the inferred configuration directly, have an Nx command that can update your local project.json files with the inferred data.

I feel like this could be a good compromise. It still achieves pretty much all of the Project Crystal goals from your comment while also solving almost all of the issues mentioned here. With optional warnings in other Nx commands if the inferred configuration has deviated from what's in project.json.

This effectively would turn the project.json files into Project Crystal 'lockfiles'.

For me this would resolve most of my concerns. I once again have a system that's observable, but that's also potentially easier to maintain. When things are updated I can see in my version control exactly what has changed. And I retain control by being able to override easily (but also see what exactly I'm overriding).

[FrozenPandaz]

It has been considered if plugins could be used in codegen to produce project.json files. We are aware this is how tools like Gazelle for Bazel work.

Why didn't we go with this method? While it definitely is more explicit like you mentioned, it is also more overhead for developers to understand. It's a tool.. to configure Bazel... around other tools. It's another tool/workflow that developers have to understand.

🤔 What I could see though is a command or a UI button which would print the configuration for a project in JSON form. I would not really recommend committing this to project.json though you certainly can. But why I think this would be helpful because such output would be diffable. You could print these either to disk or to any text diff tool and see the differences between versions. Please create a separate feature request and we can discuss more there.

[beeman]

I think having something like this would be very helpful. In my OP here under Edit 2 I found a way to make this explicit which:

• directly unblocked me
• made the project work again

In general, I think what Crystal enables is not bad per-se.

That said... The abrubt way the switch got flipped without opting in, proper testing, dogfooding, breaking changes and up-to-date docs was very unexpected, unlike anything Nx ever did did and mostly an unwelcome interruption of precious and scarce work hours.

I've discussed, loathed, doubted Nx and tried to defend its usage (both for internal and external projects) more in the last 3 months than in the 5/6 years before.

Definitely not worth the magic or faster e2e tests iyam.

I try to remain positive as I don't think there are a lot of alternatives.

So anything that helps smoothening this process is welcome. I still got quite a few upgrades to go through.

[rudfoss]

I've been using Nx for about a year now, and since I updated to version with Crystal I don't think I've ever gotten a project to work out-of-the-box without doing at least a little bit of debugging inside the plugin files. Neither Nest, React nor the Expo plugin seems to just work. Before Crystal it was trivial to figure out what configurations things were using, but now I find myself having to guess, debug and explore a lot more just to get things running.

Just to be clear: I think Nx is an amazing tool and I still love to use it for my projects. I've also learned a lot more about how the internals work together because I had to traverse the code which I guess is a plus. But not being able to know the details of how targets are configured makes it hard to fix problems and even harder to write my own targets myself. I'll most likely keep using Nx because its still pretty amazing, but I've started to look more closely at the other monorepo tools that don't do as much "magic" behind the scenes. I don't think I've had a single update since v18 that I haven't had to tewak some setting or option manually to get things working.

I do appreciate the great work that goes into the project though! I'd just love for things to be more clearly specified.

Edit: As an example. I just updated to v18.3.4 and in my expo app the run-android command suddenly doesn't work. There were no migrations either, just a package update. What would be the proper way to go about debugging something like this when configurations are hidden deep within nested node modules?

Edit: Just discovered that a project with a vite-based react app with implicit build targets doesn't actually build upstream projects. I cannot figure out how to let Project Crystal know that when I build Project C that depends on B and A I want it to build B and A as well. This was trivial when I had access to the entire target in project.json since I could simply add "dependsOn": ["^build"], and it'd work.

[FrozenPandaz]

I've been using Nx for about a year now, and since I updated to version with Crystal I don't think I've ever gotten a project to work out-of-the-box without doing at least a little bit of debugging inside the plugin files.

I don't think I've had a single update since v18 that I haven't had to tewak some setting or option manually to get things working.

It is not intended that you must tweak settings manually. Please open new issues when you run into these issues and we will take a look.

Edit: As an example. I just updated to v18.3.4 and in my expo app the run-android command suddenly doesn't work. There were no migrations either, just a package update. What would be the proper way to go about debugging something like this when configurations are hidden deep within nested node modules?

Create a new issue and @xiongemi or someone else will take a look at this.

Edit: Just discovered that a project with a vite-based react app with implicit build targets doesn't actually build upstream projects. I cannot figure out how to let Project Crystal know that when I build Project C that depends on B and A I want it to build B and A as well. This was trivial when I had access to the entire target in project.json since I could simply add "dependsOn": ["^build"], and it'd work.

This also should not be the case, again, please open an issue and someone will take a look.

[FrozenPandaz]

I've converted this issue to a discussion thread so that it is easier to respond to individual comments. There's been a lot of great discussion so far and we'll still be monitoring for further discussion.

[beeman]

Thanks Jason, this is definitely a better way for discussions 👍