miel.truyen/nx
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity

docs(core): database cache concept page (#28935)

Browse Source 
Adds a concept page describing the database cache

---------

Co-authored-by: FrozenPandaz <jasonjean1993@gmail.com>
This commit is contained in:
Isaac Mann 2024-11-18 10:02:54 -05:00 committed by GitHub
parent eaa458c5dd
commit 0407b7a7b4
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
9 changed files with 54 additions and 36 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
docs/blog/2024-09-25-introducing-nx-powerpack.md
View File

@ -125,7 +125,7 @@ Read all about how to [configure Codeowners for your project in our docs](/nx-en
A continuous effort on our Nx core is to improve speed. Last year, we began **rewriting performance-critical parts of Nx into Rust**, and more core components are being rewritten. As part of this effort, we also changed how we manage local cache, moving from a **file-based to a database-based approach**. In addition to small performance gains from reduced I/O, this opens up many opportunities for improving local cache handling, such as keeping only relevant cache based on usage, more easily controlling maximum cache size, and optimizing task orchestration by running failed tasks earlier.
As part of this new approach we're also going to [deprecate custom task runners](/deprecated/custom-task-runners) in Nx 20. I bring this up because it might affect users that relied on 3rd party tools that hooked into the task runners API.
As part of this new approach we're also going to [deprecate custom task runners](/deprecated/legacy-cache) in Nx 20. I bring this up because it might affect users that relied on 3rd party tools that hooked into the task runners API.
To fill in on the custom task runner API we're providing a new Powerpack plugin that allows you to use S3 or a network drive as your storing mechanism for your Nx cache.

12
docs/generated/manifests/menus.json
View File

@ -4796,9 +4796,9 @@
"disableCollapsible": false
},
{
"name": "Custom Task Runners",
"path": "/deprecated/custom-task-runners",
"id": "custom-task-runners",
"name": "Legacy Cache",
"path": "/deprecated/legacy-cache",
"id": "legacy-cache",
"isExternal": false,
"children": [],
"disableCollapsible": false
@ -4919,9 +4919,9 @@
"disableCollapsible": false
},
{
"name": "Custom Task Runners",
"path": "/deprecated/custom-task-runners",
"id": "custom-task-runners",
"name": "Legacy Cache",
"path": "/deprecated/legacy-cache",
"id": "legacy-cache",
"isExternal": false,
"children": [],
"disableCollapsible": false

18
docs/generated/manifests/nx.json
View File

@ -6572,14 +6572,14 @@
"tags": []
},
{
"id": "custom-task-runners",
"name": "Custom Task Runners",
"id": "legacy-cache",
"name": "Legacy Cache",
"description": "",
"mediaImage": "",
"file": "shared/deprecated/custom-task-runners",
"file": "shared/deprecated/legacy-cache",
"itemList": [],
"isExternal": false,
"path": "/deprecated/custom-task-runners",
"path": "/deprecated/legacy-cache",
"tags": []
},
{
@ -6741,15 +6741,15 @@
"path": "/deprecated/workspace-generators",
"tags": []
},
"/deprecated/custom-task-runners": {
"id": "custom-task-runners",
"name": "Custom Task Runners",
"/deprecated/legacy-cache": {
"id": "legacy-cache",
"name": "Legacy Cache",
"description": "",
"mediaImage": "",
"file": "shared/deprecated/custom-task-runners",
"file": "shared/deprecated/legacy-cache",
"itemList": [],
"isExternal": false,
"path": "/deprecated/custom-task-runners",
"path": "/deprecated/legacy-cache",
"tags": []
},
"/deprecated/workspace-executors": {

6
docs/map.json
View File

@ -1388,9 +1388,9 @@
"file": "shared/deprecated/workspace-generators"
},
{
"name": "Custom Task Runners",
"id": "custom-task-runners",
"file": "shared/deprecated/custom-task-runners"
"name": "Legacy Cache",
"id": "legacy-cache",
"file": "shared/deprecated/legacy-cache"
},
{
"name": "Workspace Executors",

12
docs/shared/deprecated/custom-task-runners.md
View File

@ -1,12 +0,0 @@
# tasksRunnerOptions
As of Nx 20, the `tasksRunnerOptions` property in `nx.json` is deprecated. This property was used to register custom task runners. `tasksRunnerOptions` and custom task runners will cease to function in Nx 21. In Nx 20, the local cache metadata and project graph are stored in a database, rather than using the file system. (Cache artifacts are still stored on the file system.) This has two benefits:
1. Cache reads and writes are faster.
2. The local cache is more secure since other processes with access to the file system can no longer read or modify the cache.
For most organizations, this feature is a net positive. If you are currently using a custom task runner, you are most likely using it to define your own custom [remote cache](/ci/features/remote-cache) storage location. You have several options moving forward:
1. Use [Nx Cloud](/nx-cloud) for your remote cache
2. Use an [Nx Powerpack](/powerpack) plugin to store your remote cache on [Amazon S3](/nx-api/powerpack-s3-cache), [Google Cloud](/nx-api/powerpack-gcs-cache), [Google Cloud](/nx-api/powerpack-gcs-cache) or a [network drive](/nx-api/powerpack-shared-fs-cache)
3. Use the deprecated custom task runner feature until Nx 21

30
docs/shared/deprecated/legacy-cache.md Normal file
View File

@ -0,0 +1,30 @@
# Legacy Cache
In Nx 21, the legacy file system cache will be removed in favor of a new database cache. The new database cache stores metadata in a database, rather than blindly trusting the file system. The database cache has the following benefits:
1. Cache reads and writes are faster.
2. The local cache is more secure since Nx will no longer retrieve artifacts it does not recognize.
The legacy file system cache can still be used in Nx 20 by setting `useLegacyCache: true` in your `nx.json` file. To gain the benefits above, remove `useLegacyCache: true` from your `nx.json`.
If you are currently using a custom task runner or the `NX_REJECT_UNKNOWN_LOCAL_CACHE` environment variable continue reading below.
## tasksRunnerOptions
As of Nx 20, the `tasksRunnerOptions` property in `nx.json` is deprecated. This property was used to register custom task runners. `tasksRunnerOptions` and custom task runners will not work with the new database cache.
If you are using `tasksRunnerOptions`, you have a few options moving forward:
1. Use [Nx Cloud](/nx-cloud) for your remote cache. This is the safest, lowest-maintenance, most recommended option
2. If you cannot use Nx Cloud, consider an [Nx Powerpack](/powerpack) plugin for caching: [Amazon S3](/nx-api/powerpack-s3-cache), [Google Cloud](/nx-api/powerpack-gcs-cache), [Azure](/nx-api/powerpack-azure-cache) or a [shared network drive](/nx-api/powerpack-shared-fs-cache)
3. If there is no Powerpack plugin that supports the service where you want to store the remote cache, [file an issue](https://github.com/nrwl/nx/issues/new)
## NX_REJECT_UNKNOWN_LOCAL_CACHE
The `NX_REJECT_UNKNOWN_LOCAL_CACHE` environment variable does not work with the new database cache. We have introduced a new cache which will recognize artifacts from other machines.
If you are using `NX_REJECT_UNKNOWN_LOCAL_CACHE` to share your local cache on a network drive, you have a few options moving forward:
1. Use [Nx Cloud](/nx-cloud) for your remote cache. This is the safest, lowest-maintenance, most recommended option
2. Use the [Nx Powerpack Shared Fs Cache](/nx-api/powerpack-shared-fs-cache) which recognizes artifacts from other machines
3. Use another [Nx Powerpack](/powerpack) plugin for caching on a cloud provider: [Amazon S3](/nx-api/powerpack-s3-cache), [Google Cloud](/nx-api/powerpack-gcs-cache), or [Azure](/nx-api/powerpack-azure-cache)

7
docs/shared/guides/unknown-local-cache.md
View File

@ -44,14 +44,13 @@ error.
## You Share Cache with Another Machine Using a Network Drive
You can prefix any Nx command with `NX_REJECT_UNKNOWN_LOCAL_CACHE=0` to ignore the errors (
When using the legacy file system cache ([deprecated in Nx 20](/deprecated/legacy-cache)), you can prefix any Nx command with `NX_REJECT_UNKNOWN_LOCAL_CACHE=0` to ignore the errors (
e.g., `NX_REJECT_UNKNOWN_LOCAL_CACHE=0 nx run-many -t build test`). This is similar to
setting `NODE_TLS_REJECT_UNAUTHORIZED=0` to ignore any errors stemming form self-signed certificates. Even though it
will make it work, this approach is discouraged.
Storing Nx's local cache on a network drive can present security risks. When a network drive is shared, every CI run has
access to all the previously created Nx cache artifacts. Hence, it is plausible for every single artifact - for every
single task hash - to be accessed without leaving any trace. This is feasible due to the network drive's capability to
Storing Nx's local cache on a network drive presents security risks, so we prevent you from doing so. When a network drive is shared, every CI run has access to all the previously created Nx cache artifacts. Hence, it is plausible for every single artifact - for every
single task hash - to be accessed without leaving any trace. This is possible due to the network drive's capability to
allow overwrites.
## How Nx Replay Makes Sure Sharing Cache is Safe

2
docs/shared/reference/sitemap.md
View File

@ -221,7 +221,7 @@
- [workspace.json](/deprecated/workspace-json)
- [As Provided vs. Derived](/deprecated/as-provided-vs-derived)
- [Workspace Generators](/deprecated/workspace-generators)
- [Custom Task Runners](/deprecated/custom-task-runners)
- [Legacy Cache](/deprecated/legacy-cache)
- [Workspace Executors](/deprecated/workspace-executors)
- [runtimeCacheInputs](/deprecated/runtime-cache-inputs)
- [cacheableOperations](/deprecated/cacheable-operations)

1
nx-dev/nx-dev/redirect-rules.js
View File

@ -1099,6 +1099,7 @@ const removedDeprecatedUrls = {
'/recipes/storybook/overview-react', // 417 views: mostly people searching "React Storybook" is matching this outdated page that was for Nx 12.7
'/deprecated/storybook/upgrade-storybook-v6-react':
'/recipes/storybook/overview-react', // 80 views
'/deprecated/custom-task-runners': '/deprecated/legacy-cache',
};
const decisionsSection = {