feat(core): support collecting migrations excluding the already applied ones (#14968)

docs/generated/cli/migrate.md

@@ -35,7 +35,7 @@ Update @nrwl/workspace to "9.0.0". This will update other packages and will gene
  nx migrate 9.0.0
 ```
 
-Update @nrwl/workspace and generate the list of migrations starting with version 8.0.0 of @nrwl/workspace and @nrwl/node, regardless of what installed locally:
+Update @nrwl/workspace and generate the list of migrations starting with version 8.0.0 of @nrwl/workspace and @nrwl/node, regardless of what is installed locally:
 
 ```shell
  nx migrate @nrwl/workspace@9.0.0 --from="@nrwl/workspace@8.0.0,@nrwl/node@8.0.0"
@@ -59,6 +59,12 @@ Collect package updates and migrations in interactive mode. In this mode, the us
  nx migrate latest --interactive
 ```
 
+Collect package updates and migrations starting with version 14.5.0 of "nx" (and Nx first-party plugins), regardless of what is installed locally, while excluding migrations that should have been applied on previous updates:
+
+```shell
+ nx migrate latest --from=nx@14.5.0 --exclude-applied-migrations
+```
+
 Run migrations from the provided migrations.json file. You can modify migrations.json and run this command many times:
 
 ```shell
@@ -89,6 +95,14 @@ Default: `false`
 
 Automatically create a git commit after each migration runs
 
+### excludeAppliedMigrations
+
+Type: `boolean`
+
+Default: `false`
+
+Exclude migrations that should have been applied on previous updates. To be used with --from
+
 ### from
 
 Type: `string`

docs/generated/packages/nx/documents/migrate.md

@@ -35,7 +35,7 @@ Update @nrwl/workspace to "9.0.0". This will update other packages and will gene
  nx migrate 9.0.0
 ```
 
-Update @nrwl/workspace and generate the list of migrations starting with version 8.0.0 of @nrwl/workspace and @nrwl/node, regardless of what installed locally:
+Update @nrwl/workspace and generate the list of migrations starting with version 8.0.0 of @nrwl/workspace and @nrwl/node, regardless of what is installed locally:
 
 ```shell
  nx migrate @nrwl/workspace@9.0.0 --from="@nrwl/workspace@8.0.0,@nrwl/node@8.0.0"
@@ -59,6 +59,12 @@ Collect package updates and migrations in interactive mode. In this mode, the us
  nx migrate latest --interactive
 ```
 
+Collect package updates and migrations starting with version 14.5.0 of "nx" (and Nx first-party plugins), regardless of what is installed locally, while excluding migrations that should have been applied on previous updates:
+
+```shell
+ nx migrate latest --from=nx@14.5.0 --exclude-applied-migrations
+```
+
 Run migrations from the provided migrations.json file. You can modify migrations.json and run this command many times:
 
 ```shell
@@ -89,6 +95,14 @@ Default: `false`
 
 Automatically create a git commit after each migration runs
 
+### excludeAppliedMigrations
+
+Type: `boolean`
+
+Default: `false`
+
+Exclude migrations that should have been applied on previous updates. To be used with --from
+
 ### from
 
 Type: `string`

docs/shared/recipes/advanced-update.md

@@ -103,29 +103,21 @@ As the migration runs and collects the package updates, you'll be prompted to ap
 
 Once you have skipped some optional updates, there'll come a time when you'll want to update those packages. To do so, you'll need to generate the package updates and migrations from the Nx version that contained those skipped updates.
 
-Say you skipped updating Jest to **v28.x.x**. That package update was meant to happen as part of the `@nrwl/jest@14.6.0` update, but you decided to skip it at the time. The `nx migrate` command collects updates for a package from versions higher than the currently installed version of the said package in the workspace. Since you need to collect updates and migrations that were meant to happen for version **14.6.0** (a version older that what's currently installed), you need to run the command and "fake" the installed version of the `@nrwl/jest` package to a version lower than **14.6.0**:
+Say you skipped updating Jest to **v28.x.x**. That package update was meant to happen as part of the `@nrwl/jest@14.6.0` update, but you decided to skip it at the time. The recommended way to collect the migrations from such an older version is to run the following:
 
 ```shell
-nx migrate @nrwl/jest@latest --from=@nrwl/jest@14.5.0
+nx migrate latest --from=nx@14.5.0 --exclude-applied-migrations
 ```
 
-The above command will effectively collect any package update and migration meant to run if your workspace had `@nrwl/jest@14.5.0` installed. While collecting package updates and migrations for a specific package works, it's not always the best way to proceed.
+A couple of things are happening there:
 
-In the specific example we have been using, since you manage the `jest` package (because you opted out of its automated migrations), you also need to be mindful of packages that depend on it and act accordingly. You could have different version requirements from different packages that depend on it. An example would be if you decide to stay on Jest v27 while updating your Angular version to v15. The `jest-preset-angular` package doesn't have a version that works with both. All versions of `jest-preset-angular` that support Angular v15 require Jest v28.
+- The `--from=nx@14.5.0` flag tells the `migrate` command to use the version **14.5.0** as the installed version for the `nx` package and all the first-party Nx plugins
+- The `--exclude-applied-migrations` flag tells the `migrate` command not to collect migrations that should have been applied on previous updates
 
-In that case you have a couple of options:
+So, the above command will effectively collect any package update and migration meant to run if your workspace had `nx@14.5.0` installed while excluding those that should have been applied before. You can provide a different older version to collect migrations from.
 
-- Don't update to Angular v15
+{% callout type="warning" title="Automatically excluding previously applied migrations" %}
-- Update Jest to v28 and Angular to v15
+Automatically excluding previously applied migrations doesn't consider migrations manually removed from the `migrations.json` in previous updates. If you've manually removed migrations in the past and want to run them, don't pass the `--exclude-applied-migrations` and collect all previous migrations.
-
-If you choose the latter, you'll need to make sure package updates and migrations from `@nrwl/jest` (manages `jest`) and `@nrwl/angular` (manages `jest-preset-angular`) are collected. You can do that by using the `nx` package that groups the rest of the first-party Nx plugins:
-
-```shell
-nx migrate latest --from=nx@14.5.0
-```
-
-{% callout type="check" title="Collect updates for all packages" %}
-As a general rule, prefer to run `nx migrate latest --from=nx@<version>` so package updates and migrations are collected from all Nx first-party plugins. By doing so, you'll be closer to what a fully-automated Nx migration does and avoid some potential issues. While it might collect migrations previously run in the repo, those migrations should be idempotent and you could always remove them from the `migrations.json` file if you want.
 {% /callout %}
 
 ## Other advanced capabilities

packages/nx/src/command-line/examples.ts

@@ -330,7 +330,7 @@ export const examples: Record<string, Example[]> = {
       command:
         'migrate @nrwl/workspace@9.0.0 --from="@nrwl/workspace@8.0.0,@nrwl/node@8.0.0"',
       description:
-        'Update @nrwl/workspace and generate the list of migrations starting with version 8.0.0 of @nrwl/workspace and @nrwl/node, regardless of what installed locally',
+        'Update @nrwl/workspace and generate the list of migrations starting with version 8.0.0 of @nrwl/workspace and @nrwl/node, regardless of what is installed locally',
     },
     {
       command:
@@ -348,6 +348,11 @@ export const examples: Record<string, Example[]> = {
       description:
         'Collect package updates and migrations in interactive mode. In this mode, the user will be prompted whether to apply any optional package update and migration',
     },
+    {
+      command: 'migrate latest --from=nx@14.5.0 --exclude-applied-migrations',
+      description:
+        'Collect package updates and migrations starting with version 14.5.0 of "nx" (and Nx first-party plugins), regardless of what is installed locally, while excluding migrations that should have been applied on previous updates',
+    },
     {
       command: 'migrate --run-migrations=migrations.json',
       description:

packages/nx/src/command-line/migrate.spec.ts

@@ -1365,6 +1365,112 @@ describe('Migration', () => {
         },
       });
     });
+
+    it('should generate the correct migrations when  "--exclude-applied-migrations"', async () => {
+      const migrator = new Migrator({
+        packageJson: createPackageJson({
+          dependencies: {
+            parent: '1.0.0',
+            pkg1: '1.0.0',
+          },
+        }),
+        getInstalledPackageVersion: (p, overrides) => overrides?.[p] ?? '1.0.0',
+        fetch: (p) => {
+          if (p === 'parent') {
+            return Promise.resolve({
+              version: '2.0.0',
+              packageGroup: [{ package: 'pkg1', version: '*' }],
+              generators: {
+                // previous migration
+                migration1: {
+                  version: '1.0.0',
+                  description: 'migration1 desc',
+                  requires: {
+                    // didn't meet requirements and now meets requirements, should collect it
+                    pkg1: '>=2.0.0',
+                  },
+                },
+                // previous migration
+                migration2: {
+                  version: '1.0.0',
+                  description: 'migration2 desc',
+                  requires: {
+                    // didn't meet requirements and now doesn't meet requirements, should not collect it
+                    pkg1: '>=3.0.0',
+                  },
+                },
+                // previous migration, no requirements, should not collect it
+                migration3: {
+                  version: '1.0.0',
+                  description: 'migration3 desc',
+                },
+                // new migration
+                migration4: {
+                  version: '2.0.0',
+                  description: 'migration4 desc',
+                  requires: {
+                    // meets requirements, should collect it
+                    pkg1: '>=2.0.0',
+                  },
+                },
+                // new migration
+                migration5: {
+                  version: '2.0.0',
+                  description: 'migration5 desc',
+                  requires: {
+                    // doesn't meet requirements, should not collect it
+                    pkg1: '>=3.0.0',
+                  },
+                },
+                // new migrationg, no requirements, should collect it
+                migration6: {
+                  version: '2.0.0',
+                  description: 'migration6 desc',
+                },
+              },
+            });
+          } else if (p === 'pkg1') {
+            return Promise.resolve({ version: '2.0.0' });
+          } else {
+            return Promise.resolve(null);
+          }
+        },
+        from: { parent: '0.1.0' },
+        to: {},
+        excludeAppliedMigrations: true,
+      });
+
+      const result = await migrator.migrate('parent', '2.0.0');
+
+      expect(result).toEqual({
+        migrations: [
+          {
+            version: '1.0.0',
+            name: 'migration1',
+            package: 'parent',
+            description: 'migration1 desc',
+            requires: { pkg1: '>=2.0.0' },
+          },
+          {
+            version: '2.0.0',
+            name: 'migration4',
+            package: 'parent',
+            description: 'migration4 desc',
+            requires: { pkg1: '>=2.0.0' },
+          },
+          {
+            version: '2.0.0',
+            name: 'migration6',
+            package: 'parent',
+            description: 'migration6 desc',
+          },
+        ],
+        packageUpdates: {
+          parent: { version: '2.0.0', addToPackageJson: false },
+          pkg1: { version: '2.0.0', addToPackageJson: false },
+        },
+      });
+    });
   });
 
   describe('normalizeVersions', () => {

packages/nx/src/command-line/migrate.ts

@@ -16,6 +16,7 @@ import {
 import { promisify } from 'util';
 import {
   MigrationsJson,
+  MigrationsJsonEntry,
   PackageJsonUpdateForPackage as PackageUpdate,
   PackageJsonUpdates,
 } from '../config/misc-interfaces';
@@ -31,7 +32,6 @@ import { logger } from '../utils/logger';
 import {
   ArrayPackageGroup,
   NxMigrationsConfiguration,
-  PackageGroup,
   PackageJson,
   readModulePackageJson,
   readNxMigrateConfig,
@@ -100,7 +100,7 @@ export interface MigratorOptions {
   packageJson: PackageJson;
   getInstalledPackageVersion: (
     pkg: string,
-    overrides: Record<string, string>
+    overrides?: Record<string, string>
   ) => string;
   fetch: (
     pkg: string,
@@ -109,6 +109,7 @@ export interface MigratorOptions {
   from: { [pkg: string]: string };
   to: { [pkg: string]: string };
   interactive?: boolean;
+  excludeAppliedMigrations?: boolean;
 }
 
 export class Migrator {
@@ -118,6 +119,7 @@ export class Migrator {
   private readonly installedPkgVersionOverrides: MigratorOptions['from'];
   private readonly to: MigratorOptions['to'];
   private readonly interactive: MigratorOptions['interactive'];
+  private readonly excludeAppliedMigrations: MigratorOptions['excludeAppliedMigrations'];
   private readonly packageUpdates: Record<string, PackageUpdate> = {};
   private readonly collectedVersions: Record<string, string> = {};
 
@@ -128,6 +130,7 @@ export class Migrator {
     this.installedPkgVersionOverrides = opts.from;
     this.to = opts.to;
     this.interactive = opts.interactive;
+    this.excludeAppliedMigrations = opts.excludeAppliedMigrations;
   }
 
   async migrate(targetPackage: string, targetVersion: string) {
@@ -157,7 +160,7 @@ export class Migrator {
               migration.version &&
               this.gt(migration.version, currentVersion) &&
               this.lte(migration.version, version) &&
-              this.areRequirementsMet(migration.requires)
+              this.areMigrationRequirementsMet(packageName, migration)
           )
           .map(([migrationName, migration]) => ({
             ...migration,
@@ -218,7 +221,7 @@ export class Migrator {
     }
 
     if (!this.getPkgVersion(targetPackage)) {
-      this.addPackageJsonUpdate(targetPackage, {
+      this.addPackageUpdate(targetPackage, {
         version: target.version,
         addToPackageJson: target.addToPackageJson || false,
       });
@@ -247,6 +250,11 @@ export class Migrator {
     }
     this.collectedVersions[targetPackage] = targetVersion;
 
+    this.addPackageUpdate(targetPackage, {
+      version: migrationConfig.version,
+      addToPackageJson: target.addToPackageJson || false,
+    });
+
     const { packageJsonUpdates, packageGroupOrder } =
       this.getPackageJsonUpdatesFromMigrationConfig(
         targetPackage,
@@ -254,10 +262,9 @@ export class Migrator {
         migrationConfig
       );
 
-    this.addPackageJsonUpdate(targetPackage, {
+    if (!packageJsonUpdates.length) {
-      version: migrationConfig.version,
+      return [];
-      addToPackageJson: target.addToPackageJson || false,
+    }
-    });
 
     const shouldCheckUpdates = packageJsonUpdates.some(
       (packageJsonUpdate) =>
@@ -424,10 +431,7 @@ export class Migrator {
     return filteredPackageJsonUpdates;
   }
 
-  private addPackageJsonUpdate(
+  private addPackageUpdate(name: string, packageUpdate: PackageUpdate): void {
-    name: string,
-    packageUpdate: PackageUpdate
-  ): void {
     if (
       !this.packageUpdates[name] ||
       this.gt(packageUpdate.version, this.packageUpdates[name].version)
@@ -463,6 +467,55 @@ export class Migrator {
     );
   }
 
+  private areMigrationRequirementsMet(
+    packageName: string,
+    migration: MigrationsJsonEntry
+  ): boolean {
+    if (!this.excludeAppliedMigrations) {
+      return this.areRequirementsMet(migration.requires);
+    }
+
+    return (
+      (this.wasMigrationSkipped(migration.requires) ||
+        this.isMigrationForHigherVersionThanWhatIsInstalled(
+          packageName,
+          migration
+        )) &&
+      this.areRequirementsMet(migration.requires)
+    );
+  }
+
+  private isMigrationForHigherVersionThanWhatIsInstalled(
+    packageName: string,
+    migration: MigrationsJsonEntry
+  ): boolean {
+    const installedVersion = this.getInstalledPackageVersion(packageName);
+
+    return (
+      migration.version &&
+      (!installedVersion || this.gt(migration.version, installedVersion)) &&
+      this.lte(migration.version, this.packageUpdates[packageName].version)
+    );
+  }
+
+  private wasMigrationSkipped(
+    requirements: PackageJsonUpdates[string]['requires']
+  ): boolean {
+    // no requiremets, so it ran before
+    if (!requirements || !Object.keys(requirements).length) {
+      return false;
+    }
+
+    // at least a requirement was not met, it was skipped
+    return Object.entries(requirements).some(
+      ([pkgName, versionRange]) =>
+        !this.getInstalledPackageVersion(pkgName) ||
+        !satisfies(this.getInstalledPackageVersion(pkgName), versionRange, {
+          includePrerelease: true,
+        })
+    );
+  }
+
   private async runPackageJsonUpdatesConfirmationPrompt(
     confirmationPrompt: string
   ): Promise<boolean> {
@@ -604,6 +657,7 @@ type GenerateMigrations = {
   from: { [k: string]: string };
   to: { [k: string]: string };
   interactive?: boolean;
+  excludeAppliedMigrations?: boolean;
 };
 
 type RunMigrations = { type: 'runMigrations'; runMigrations: string };
@@ -623,14 +677,14 @@ export function parseMigrationsOptions(options: {
     const { targetPackage, targetVersion } = parseTargetPackageAndVersion(
       options['packageAndVersion']
     );
-    const interactive = options.interactive;
     return {
       type: 'generateMigrations',
       targetPackage: normalizeSlashes(targetPackage),
       targetVersion,
       from,
       to,
-      interactive,
+      interactive: options.interactive,
+      excludeAppliedMigrations: options.excludeAppliedMigrations,
     };
   } else {
     return {
@@ -647,10 +701,10 @@ function createInstalledPackageVersionsResolver(
 
   function getInstalledPackageVersion(
     packageName: string,
-    overrides: Record<string, string>
+    overrides?: Record<string, string>
   ): string | null {
     try {
-      if (overrides[packageName]) {
+      if (overrides?.[packageName]) {
         return overrides[packageName];
       }
 
@@ -1051,6 +1105,7 @@ async function generateMigrationsJsonAndUpdatePackageJson(
       from: opts.from,
       to: opts.to,
       interactive: opts.interactive,
+      excludeAppliedMigrations: opts.excludeAppliedMigrations,
     });
 
     const { migrations, packageUpdates } = await migrator.migrate(

packages/nx/src/command-line/nx-commands.ts

@@ -972,14 +972,27 @@ function withMigrationOptions(yargs: yargs.Argv) {
       type: 'boolean',
       default: false,
     })
-    .check(({ createCommits, commitPrefix }) => {
+    .option('excludeAppliedMigrations', {
-      if (!createCommits && commitPrefix !== defaultCommitPrefix) {
+      describe:
-        throw new Error(
+        'Exclude migrations that should have been applied on previous updates. To be used with --from',
-          'Error: Providing a custom commit prefix requires --create-commits to be enabled'
+      type: 'boolean',
-        );
+      default: false,
+    })
+    .check(
+      ({ createCommits, commitPrefix, from, excludeAppliedMigrations }) => {
+        if (!createCommits && commitPrefix !== defaultCommitPrefix) {
+          throw new Error(
+            'Error: Providing a custom commit prefix requires --create-commits to be enabled'
+          );
+        }
+        if (excludeAppliedMigrations && !from) {
+          throw new Error(
+            'Error: Excluding migrations that should have been previously applied requires --from to be set'
+          );
+        }
+        return true;
       }
-      return true;
+    );
-    });
 }
 
 function withWatchOptions(yargs: yargs.Argv) {