## Current Behavior aws cli usage in Nx Agents is undocumented ## Expected Behavior aws cli step is documented for Nx Agents preview: https://nx-dev-git-docs-add-aws-cli-workflow-step-nrwl.vercel.app/ci/reference/launch-templates#aws-cli-on-nx-agents --------- Co-authored-by: Nicole Oliver <nicole.oliver.42@gmail.com>
30 KiB
Launch Templates
A launch template defines the setup steps Nx Agents will run before running tasks. A custom launch template isn't required to use Nx Agents.
Nx Cloud provides several pre-built launch templates for common use-cases. You can view available templates in the nx-cloud-workflows repository.
{% github-repository url="https://github.com/nrwl/nx-cloud-workflows/tree/main/launch-templates" title="Pre-Built Launch Templates" /%}
Getting Started with Custom Launch Templates
The easiest way to create a new custom launch template is to modify one of the pre-built ones. To do that, create a file in the
.nx/workflows folder and copy one of the pre-built templates. You can name the file any way you want (e.g., agents.yaml) and customize the steps as needed.
Launch Template Structure
launch-templates
A map of launch template configurations. This value is required.
launch-templates:
launch-templates.<template-name>
The name of your custom launch template. This name is used via --distribute-on="<# of agents> <template-name>" when starting the ci run. Supports one to many uniquely named launch templates.
Multiple launch templates can be useful for setting up different toolchains (rust, java, node versions) or resources classes for your workspace needs.
launch-templates:
template-one:
template-two:
nx-cloud start-ci-run --distribute-on="3 template-one"
launch-templates.<template-name>.resource-class
A launch template's resource-class defines the memory and vCPUs available to each agent machine.
launch-templates:
template-one:
resource-class: 'docker_linux_amd64/medium'
The following resource classes are available:
docker_linux_amd64/smalldocker_linux_amd64/mediumdocker_linux_amd64/medium+docker_linux_amd64/largedocker_linux_amd64/large+docker_linux_amd64/extra_largedocker_linux_amd64/extra_large+docker_linux_arm64/mediumdocker_linux_arm64/largedocker_linux_arm64/extra_largewindows/medium
See their detailed description and pricing at nx.dev/pricing.
launch-templates.<template-name>.image
A launch template's image defines the available base software for the agent machine.
launch-templates:
template-one:
image: 'ubuntu22.04-node20.11-v9'
Nx Cloud provides the following images:
Changes added in previous images are included in newer images unless otherwise denoted Images also have go 1.22 installed
-
ubuntu22.04-node20.11-v5- added elevated permission access via
sudo
- added elevated permission access via
-
ubuntu22.04-node20.11-v6corepackis enabled by default- default global package manager versions:
npmv10yarnv1pnpmv8- See how to install a specific package manger version
-
ubuntu22.04-node20.11-v7- added java version 17
-
ubuntu22.04-node20.11-v9- added nvm
-
windows-2022
Note: Windows-based images can only run on Windows-based resource classes.
Enterprise accounts can use custom images.
launch-templates.<template-name>.env
A launch template's env defines a map of environment variable names and values to be available within all steps of the specific launch template.
launch-templates:
template-one:
env:
MY_ENV_VAR: 'my-var-value'
launch-templates.<template-name>.init-steps
A launch template's init-steps defines the series of steps to perform before an Nx Agent runs. Without a defined init-steps the Nx Agent is unable to process any tasks.
Typical init-steps perform actions such as checking out your workspace source code and installing any necessary dependencies. Any extra setup your workspace needs to run should be defined as an init-step. Once all steps run successfully, the agent machine will inform Nx Cloud that it is ready to accept tasks.
launch-templates:
template-one:
init-steps:
launch-templates.<template-name>.init-steps[*].name
An init-step's name is the label that will be reflected in the Nx Cloud UI. name can be used in conjunction with uses and script
launch-templates:
template-one:
init-steps:
- name: 'My Helpful Step Name'
launch-templates.<template-name>.init-steps[*].uses
When defined, specifies an existing step file to be used. Cannot be used when script is also defined
You can find the list of Nx Cloud reusable steps here. If you cannot find a reusable step that suits your needs, you can create your own custom steps.
launch-templates:
template-one:
init-steps:
- uses: 'nrwl/nx-cloud-workflows/v5/workflow-steps/checkout/main.yaml'
- name: 'Install Node Modules'
uses: 'nrwl/nx-cloud-workflows/v5/workflow-steps/install-node-modules/main.yaml'
launch-templates.<template-name>.init-steps[*].script
When defined, allows an inline script to be run. Cannot be used when uses is also defined
launch-templates:
template-one:
init-steps:
- name: 'Print Node Version and PATH'
script: |
node -v
echo $PATH
launch-templates.<template-name>.init-steps[*].env
An init-step's env is similar to the launch-template.<template-name>.env, except the environment variable map is scoped for the current step only.
launch-templates:
template-one:
init-steps:
- name: 'Print Env'
env:
MY_STEP_ENV: 'step-env-var'
script: |
echo $MY_STEP_ENV # prints "step-env-var"
launch-templates.<template-name>.init-steps[*].inputs
An init-step's inputs is defined by the step file in the launch-template.<template-name>.init-steps[*].uses property. Refer to the step file's documentation for specific inputs.
Validation can also be done to validate the step against the step file's defined inputs
launch-templates:
template-one:
init-steps:
- name: Restore Node Modules Cache
uses: 'nrwl/nx-cloud-workflows/v5/workflow-steps/cache/main.yaml'
inputs:
key: 'package-lock.json|yarn.lock|pnpm-lock.yaml'
paths: |
~/.npm
# or ~/.cache/yarn
# or .pnpm-store
base-branch: 'main'
launch-templates.<template-name>.group-name
You can define "step groups" that run in parallel. This can be useful for taking better advantage of the available CPUs of your chosen resource class. In the example below, we can run NPM Install and install the Rust dependencies in parallel, because they do not depend on each other and they write to different places on the filesystem. Running them in parallel can reduce agent startup time by up to 2 minutes, which can add up to a lot of compute time savings over the month.
⚠️ Tips for using parallel steps:
- don't assume everything can be run in parallel
- you'll notice below we had to run
playwright installafter the parallel group, as it was likely writing to the same locations and fighting for similar resources tonpm install - experiment with different parallel groups and measure startup times until you land on the most optimal config for your use-case
- you'll notice below we had to run
- write to PATH once
- multiple parallel processes can overwrite each other's changes if they write to the same
>> $NX_CLOUD_ENVlocation - below, we only update the PATH variable once at the end
echo "PATH=$CARGO_PATH:$POETRY_PATH:$PATH" >> $NX_CLOUD_ENV
- multiple parallel processes can overwrite each other's changes if they write to the same
launch-templates:
template-one:
init-steps:
- group-name: Install Dependencies
parallel: true
# all the below steps will start at the same time and run in parallel
steps:
- name: Install Rust
script: |
curl --proto '=https' --tlsv1.3 https://sh.rustup.rs -sSf | sh -s -- -y
source "~/workspace/.cargo/env"
# we write the CARGO_PATH now so that it can be set in the "Start Services" step below
echo "CARGO_PATH=~/workspace/.cargo/bin" >> $NX_CLOUD_ENV
rustc --version
rustup target add wasm32-wasip1-threads
cargo fetch
cargo check --locked
- name: NPM Install
uses: 'nrwl/nx-cloud-workflows/main/workflow-steps/install-node-modules/main.yaml'
- name: Install Poetry
script: |
curl -sSL https://install.python-poetry.org | python3 -
# we write the POETRY_PATH now so that it can be set in the "Start Services" step below
export POETRY_PATH="/home/workflows/.local/bin"
export PATH="$POETRY_PATH:$PATH"
echo "POETRY_PATH=$POETRY_PATH" >> $NX_CLOUD_ENV
poetry --version
poetry install
- name: Install Localstack
script: |
curl --output localstack-cli-3.7.0-linux-amd64-onefile.tar.gz --location https://github.com/localstack/localstack-cli/releases/download/v3.7.0/localstack-cli-3.7.0-linux-amd64-onefile.tar.gz
sudo tar xvzf localstack-cli-3.7.0-linux-*-onefile.tar.gz -C /usr/local/bin
# because both playwright and "npm install" use NPM and write to a lot of the same places on the filesystem, it would be slower to run them both in parallel
# so we run playwright after the parallel group above
- name: Install Playwright
script: npx playwright install --with-deps
- name: Start services
script: |
# we create the PATH here
echo "PATH=$CARGO_PATH:$POETRY_PATH:$PATH" >> $NX_CLOUD_ENV
npm run start-docker-services
We can also group steps and run them serially. This is actually the default behaviour when you don't set parallel: true for a group.
This can be useful if we have a set of quick enough steps, such as restoring from cache, where we don't need to optimise for speed, and instead we just want them grouped together logically. The below cache steps will also be collapsed together in the Agents UI, making the config look cleaner:
- group-name: Restore Cache
steps:
- name: Restore Node Modules Cache
uses: 'nrwl/nx-cloud-workflows/v5/workflow-steps/cache/main.yaml'
inputs:
key: 'package-lock.json'
paths: |
~/.npm
base-branch: 'main'
- name: Restore Browser Binary Cache
uses: 'nrwl/nx-cloud-workflows/v5/workflow-steps/cache/main.yaml'
inputs:
key: 'package-lock.json|"browsers"'
paths: |
'~/.cache/Cypress'
base-branch: 'main'
Full Example
This is an example of a launch template using all pre-built features:
common-init-steps: &common-init-steps
- name: Checkout
# using a reusable step in an external GitHub repo,
# this step is provided by Nx Cloud: https://github.com/nrwl/nx-cloud-workflows/tree/main/workflow-steps
uses: 'nrwl/nx-cloud-workflows/v5/workflow-steps/checkout/main.yaml'
# group these steps together as they related (it doesn't change anything functionally, but it helps with organising your steps as they will be collapsed together in the UI)
- group-name: Restore Cache
steps:
- name: Restore Node Modules Cache
uses: 'nrwl/nx-cloud-workflows/v5/workflow-steps/cache/main.yaml'
# the cache step requires configuration via env vars
# https://github.com/nrwl/nx-cloud-workflows/tree/main/workflow-steps/cache#options
inputs:
key: 'package-lock.json|yarn.lock|pnpm-lock.yaml'
paths: |
~/.npm
# or ~/.cache/yarn
# or .pnpm-store
base-branch: 'main'
- name: Restore Browser Binary Cache
uses: 'nrwl/nx-cloud-workflows/v5/workflow-steps/cache/main.yaml'
inputs:
key: 'package-lock.json|yarn.lock|pnpm-lock.yaml|"browsers"'
paths: |
'~/.cache/Cypress'
base-branch: 'main'
- name: Install Node Modules
uses: 'nrwl/nx-cloud-workflows/v5/workflow-steps/install-node-modules/main.yaml'
- name: Install Browsers (if needed)
uses: 'nrwl/nx-cloud-workflows/v5/workflow-steps/install-browsers/main.yaml'
# You can also run a custom script to configure various things on the agent machine
- name: Run a custom script
script: |
git config --global user.email test@test.com
git config --global user.name "Test Test"
# You can also set any other env vars to be passed to the following steps
# by setting their value in the `$NX_CLOUD_ENV` file.
# Most commonly for redefining PATH for further steps
- name: Setting env
script: |
# Update PATH with custom value
echo "PATH=$HOME/my-folder:$PATH" >> $NX_CLOUD_ENV
- name: Print path from previous step
# will include my-folder
script: echo $PATH
- name: Define env var for a step
env:
MY_ENV_VAR: 'env-var-for-step'
# will print env-var-for-step
script: echo $MY_ENV_VAR
# after you're last step Nx Agents will start accepting tasks to process
# no need to manually start up the agent yourself
launch-templates:
# Custom template name, the name is referenced via --distribute-on="3 my-linux-medium-js"
# You can define as many templates as you need, commonly used to make different sizes or toolchains depending on your workspace needs
my-linux-medium-js:
# see the available resource list below
resource-class: 'docker_linux_amd64/medium'
# see the available image list below
image: 'ubuntu22.04-node20.11-v9'
# Define environment variables shared among all steps
env:
MY_ENV_VAR: shared
# list out steps to run on the agent before accepting tasks
# the agent will need a copy of the source code and dependencies installed
# note we are using yaml anchors te reduce duplication with the below launch-template (they have the same init-steps)
init-steps: *common-init-steps
# another template which does the same as above, but with a large resource class
# You're not required to define a template for every resource class, only define what you need!
my-linux-large-js:
resource-class: 'docker_linux_amd64/large'
image: 'ubuntu22.04-node20.11-v9'
env:
MY_ENV_VAR: shared
# note we are using yaml anchors te reduce duplication with the above launch-template (they have the same init-steps)
init-steps: *common-init-steps
# template that installs rust
my-linux-rust-large:
resource-class: 'docker_linux_amd64/large'
image: 'ubuntu22.04-node20.11-v9'
init-steps:
- name: Checkout
uses: 'nrwl/nx-cloud-workflows/v5/workflow-steps/checkout/main.yaml'
- name: Restore Node Modules Cache
uses: 'nrwl/nx-cloud-workflows/v5/workflow-steps/cache/main.yaml'
inputs:
key: 'package-lock.json|yarn.lock|pnpm-lock.yaml'
paths: |
~/.npm
# or ~/.cache/yarn
# or .pnpm-store
base-branch: 'main'
- name: Install Node Modules
uses: 'nrwl/nx-cloud-workflows/v5/workflow-steps/install-node-modules/main.yaml'
- name: Install Rust
script: |
curl --proto '=https' --tlsv1.3 https://sh.rustup.rs -sSf | sh -s -- -y
source "$HOME/.cargo/env"
rustup toolchain install 1.70.0
# persist cargo bin into PATH
echo "PATH=$HOME/.cargo/bin:$PATH" >> $NX_CLOUD_ENV
These templates can be used by passing the number of agents desired, and the template name via --distribute-on when starting your CI run.
nx-cloud start-ci-run --distribute-on="4 my-linux-medium-js"
nx-cloud start-ci-run --distribute-on="4 my-linux-large-js"
nx-cloud start-ci-run --distribute-on="4 my-linux-large-rust"
Validating Launch Templates
{% callout type="note" title="Commit before Validation" %} Before Nx Cloud can validate your custom templates, you must first commit any changes to these templates to your source control repository. Running the validation command from a CI is the recommended approach. {% /callout %} After creating your custom launch template, it's recommended to validate it. This ensures that all necessary fields within the launch template and all respective inputs within each step are appropriately defined.
To do this, run the nx-cloud validate command, with the path to the launch template:
nx-cloud validate --workflow-file=./.nx/workflows/agents.yaml
Pass Environment Variables
If you need to send environment variables to agents, you can use the --with-env-vars flag on the nx-cloud start-ci-run command. You can pass a specific list of environment variables like this:
nx-cloud start-ci-run --distribute-on="8 linux-medium-js" --with-env-vars="VAR1,VAR2"
Pass Values Between Steps
If you need to pass a value from one step to another step, such as assigning the value to an existing or new environment variable. You can write to the NX_CLOUD_ENV environment file.
Commonly used for redefining the PATH or setting options for tooling.
launch-templates:
my-template-name:
init-steps:
- name: Set PATH
script: echo "PATH=$HOME/.cargo/bin:$PATH" >> $NX_CLOUD_ENV
- name: Check PATH
script: |
# now contains $HOME/.cargo/bin
echo $PATH
# can invoke cargo directly because it's in the PATH now.
cargo --version
Private NPM Registry
If your project consumes packages from a private registry, you'll have to set up an authentication step in a custom launch template and authenticate like you normally would, usually this is via a .npmrc or .yarnrc file. You can pass the auth token from your main agent, so it's available to the agent machines.
launch-templates:
my-linux-medium-js:
resource-class: 'docker_linux_amd64/medium'
image: 'ubuntu22.04-node20.11-v9'
init-steps:
- name: Checkout
uses: 'nrwl/nx-cloud-workflows/v5/workflow-steps/checkout/main.yaml'
- name: Auth to Registry
script: |
# create .npmrc with @myorg scoped packages pointing to GH npm registry
echo "@myorg:registry=https://npm.pkg.github.com" >> .npmrc
echo "//npm.pkg.github.com/:_authToken=${SOME_AUTH_TOKEN}" >> .npmrc
- name: Install Node Modules
uses: 'nrwl/nx-cloud-workflows/v5/workflow-steps/install-node-modules/main.yaml'
Pass SOME_AUTH_TOKEN via --with-env-vars
# this assumes SOME_AUTH_TOKEN is already defined on the main agent
nx-cloud start-ci-run --distribute-on="5 my-linux-medium-js" --with-env-vars="SOME_AUTH_TOKEN"
Custom Node Version
Nx Agents come with node LTS installed. If you want to use a different version, you can add a step to install the desired node version.
{% tabs %} {% tab label="Pre-Built Step" %}
Nx Cloud provides a pre-built step to install a custom node version within your workflow. This step is available as of v4 of the workflow steps and requires the minimum image version to be ubuntu22.04-node20.11-v9.
launch-templates:
node-21:
resource-class: 'docker_linux_amd64/medium'
# note the image version of v9,
# earlier versions of the base image will not work
image: 'ubuntu22.04-node20.11-v9'
init-steps:
- name: Checkout
uses: 'nrwl/nx-cloud-workflows/v5/workflow-steps/checkout/main.yaml'
- name: Install Node
# note the step is only released as of v4 of the workflow steps
uses: 'nrwl/nx-cloud-workflows/v5/workflow-steps/install-node/main.yaml'
inputs:
# can omit value if a '.nvmrc' file is within the root of the repo
node_version: '21'
{% /tab %} {% tab label="Manual Installation" %}
Here is a simplified launch template manually installing nvm to manage different node version. Feel free to use your preferred tool, other popular options are volta or fnm. The steps follow the general pattern of:
- install tool
- install node with tool
- persist any environment variable changes for future steps
launch-templates:
node-21:
resource-class: 'docker_linux_amd64/medium'
image: 'ubuntu22.04-node20.11-v9'
init-steps:
- name: Checkout
uses: 'nrwl/nx-cloud-workflows/v5/workflow-steps/checkout/main.yaml'
- name: Install nvm
script: |
# run nvm install script
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash
# source the updated profile to get the nvm command available
source ~/.profile
# install the needed version of node via nvm
nvm install 21.7.3
# echo the current path (which now includes the nvm-provided version of node) to NX_CLOUD_ENV
echo "PATH=$PATH" >> $NX_CLOUD_ENV
- name: Print node version
# confirm that the node version has changed
script: node -v
- name: Install Node Modules
uses: 'nrwl/nx-cloud-workflows/v5/workflow-steps/install-node-modules/main.yaml'
# Continue setup steps as needed
{% /tab %} {% /tabs %}
Specific Package Manager Version
Nx Agents have corepack enabled by default, allowing you to define the yarn or pnpm version via the package.json.
{
"packageManager": "yarn@4.1.1"
}
{% callout type="note" title="Supported Package Managers" %}
Currently, corepack only supports yarn or pnpm as package managers. If you need to use a specific npm version, you will need to create a custom launch template and install the specific npm version, i.e. npm install -g npm@<version>
{%/callout %}
Installing Packages
You can use apt to install popular linux packages. This is helpful in streamlining setting up various toolchains needed for your workspace.
For example, you can install the GitHub CLI on the agents if needed.
launch-templates:
my-linux-medium-js:
resource-class: 'docker_linux_amd64/medium'
image: 'ubuntu22.04-node20.11-v9'
init-steps:
- name: Install Extras
script: |
sudo apt install gh unzip zip -y
{% callout type="note" title="Installing without apt" %}
If you're trying to install a package that isn't available on apt, check that packages install steps for Debian base linux. Usually there are a handful of installation scripts that can be used similar to nvm
{% /callout %}
AWS CLI
First, define the environment variables on your main agent, where you call nx-cloud start-ci-run. Pass the same environment variables to your agents via the --with-env-vars flag.
Minimally the AWS_REGION (or AWS_DEFAULT_REGION), AWS_ACCESS_KEY_ID, and AWS_SECRET_ACCESS_KEY environment variables are required.
{% callout type="check" title="OIDC Provider" %}
If using an OIDC provider, e.g. the AWS GitHub Action recommended setup, then the environment variables are automatically configured for you after authentication to AWS. Since the credentials created for with OIDC provider flow are temporary, you'll need to pass the AWS_SESSION_TOKEN environment variable to your agents.
{% /callout %}
{% tabs %} {% tab label="Pre-Built Step" %}
Using the pre-built step simplifies and provides debugging checks to make sure the AWS CLI is properly authenticated to AWS before continuing. The step is recommended for most use cases.
launch-templates:
my-linux-medium-js:
resource-class: 'docker_linux_amd64/medium'
image: 'ubuntu22.04-node20.11-v9'
init-steps:
- name: Install AWS CLI
uses: 'nrwl/nx-cloud-workflows/v5/workflow-steps/install-aws-cli/main.yaml'
# no additional inputs required, as all configuration is via environment variables via --with-env-vars
{% /tab %} {% tab label="Manual Installation" %}
Manual installation follows installing the AWS CLI using the official directions. In order to decompress the installation archive, you'll need to install the unzip package in your step.
Manual installation is helpful if a custom installation setup is required.
It is helpful to run aws sts get-caller-identity command to make sure you're properly authenticated to AWS before proceeding
launch-templates:
my-linux-medium-js:
resource-class: 'docker_linux_amd64/medium'
image: 'ubuntu22.04-node20.11-v9'
init-steps:
# step assumes you've passed all required AWS_* environment variables from the main agent via --with-env-vars
- name: Install AWS CLI
script: |
# unzip is required to unzip the AWS CLI installer
sudo apt install gh unzip -y
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
# run installer, default location is already in the PATH
# if installing in a custom location,
# then update the $PATH like `PATH=$PATH:[extra-path-items] >> $NX_CLOUD_ENV`
sudo ./aws/install
# print AWS CLI version to verify command is avaiable
aws --version
# verify credentials to AWS is working
aws sts get-caller-identity
{% /tab %} {% /tabs %}
Dynamic Changesets
NxCloud can calculate how big your pull request is based on how many projects in your workspace it affects. You can then configure Nx Agents to dynamically use a different number of agents based on your changeset size.
Here we define a small, medium and large distribution strategy:
distribute-on:
small-changeset: 3 linux-medium-js
medium-changeset: 8 linux-medium-js
large-changeset: 12 linux-medium-js
Then you can pass the path to the file to the --distribute-on parameter.
nx-cloud start-ci-run --distribute-on=".nx/workflows/dynamic-changesets.yaml"
Debugging
You can add steps to your template which print information about your workspace, toolchains, or any other needs. Below are some common steps people use for debugging such as running nx commands, printing file contents and/or listing directory contents.
{% callout type="caution" title="Environment Variables" %} Exercise caution when printing out environment variables, they will be shown in plain text. {% /callout %}
launch-templates:
my-linux-medium-js:
resource-class: 'docker_linux_amd64/medium'
image: 'ubuntu22.04-node20.11-v9'
env:
# enable verbose logging for all steps
NX_VERBOSE_LOGGING: true
init-steps:
- name: 'Debug: Print Nx Report'
script: |
nx report
- name: 'Debug: List Directory Contents'
script: |
echo $HOME
ls -la $HOME
output_dir=$HOME/dist
# if output directory exists list it's contents
if [ -d output_dir ]; then
ls -la $output_dir
else
echo "$output_dir does not exist"
fi
- name: 'Debug: Show File Contents'
script: |
cat $HOME/.profile
- name: 'Debug: Check For Checkout Files'
script: |
git diff
- name: 'Debug: Print Versions'
script: |
# note if you use yarn and try to run pnpm, corepack might throw an error at you
# saying you're using the wrong package manager, in that case just remove the usage of pnpm
echo "Versions:"
echo "Node: $(node -v)"
echo "NPM: $(npm -v)"
echo "Yarn: $(yarn -v)"
echo "PNPM: $(pnpm -v)"
echo "Golang: $(go version)"
echo "Java: $(javac --version)"
# add any other toolchain you want
- name: 'Debug: Print env'
script: |
# !!! DO NOT RUN THIS IF YOU HAVE PASSWORD/ACCESS TOKENS IN YOUR ENV VARS !!!
# this will print your env as plain text values to the terminal
env
# This is a safer approach to prevent leaking tokens/passwords
echo "SOME_VALUE: $SOME_VALUE"