Faster build pipeline using cache task in Azure DevOps
Speed up your YAML build pipeline in Azure DevOps with the Cache task.
Sometimes the builds are taking a long time. Often the NuGet restore/npm install takes a couple of minutes to download all packages that you use in your solution. One way to speed up that process is to use the “new” Cache task in Azure DevOps.
Pipeline caching (https://docs.microsoft.com/en-us/azure/devops/pipelines/release/caching?view=azure-devops)
It took me a long time to understand how you actually used this task (yes I know, I´m kind of solve…). But when I found the example in jraps20 build.yml(https://gist.github.com/jraps20/ea29de8b45493dcfa0a1ec5b0af82337) I finally understood how it works and how to do this.
So, what you could do is add 2 tasks. The first task is a PowerShell task creates a hash of the content in packages.config. That means that as long you don´t change and version of NuGet packages, and/or adding/removing NuGet packages, the hash will be the same. That will decide if the cached packages folder should be used or if it should download Nuget packages into the packages folder.
The second task is actual cache task that use the hash in the cachekey to figure out if there is any cached “packages” folder to reuse.
Note: The Cache task (Cache files between runs) is only supported when you are using YAML.
Before I implemented the caching my YAML file started like this:
And after implemented the caching tasks it looks like this:
Result
You never really know how much this will fast up you build. But an example from my last project made the build pipeline become around 1.5 minutes faster. (If I have no packages.config changes of course). And of course, it depends on how many NuGet packages you use etc.
Example before:
In our case in this project we use around 115 NuGet packages and the folder weight around 400Mb. It takes around 2 minutes (if lucky) to download these packages.
Example after:
And with the cache task we managed to get the same task to run in 30 seconds.
NPM
I have not given you any example of how to do this with NPM packages. Here is an example of YAML that cache both NuGet and NPM packages.
trigger:
batch: true
branches:
include:
- develop
variables:
- group: DXP-variables
- name: webProjectDirectory
value: '$(Build.SourcesDirectory)\Customer.Web'
stages:
- stage: Build
jobs:
- job: DevInte
pool:
name: Azure Pipelines
vmImage: 'windows-latest'
demands:
- npm
- msbuild
- visualstudio
- vstest
variables:
solution: '**/*.sln'
buildPlatform: 'Any CPU'
buildConfiguration: 'Release'
steps:
- task: PowerShell@2
displayName: 'Calculate and save packages.config hash'
inputs:
targetType: 'inline'
script: |
# generates a hash of all packages.config and saves each on a single line on 'packagesconfighash.txt'
Get-FileHash -Algorithm MD5 -Path (Get-ChildItem packages.config -Recurse) >> packagesconfighash.txt
Write-Host "Hash File saved to $(System.DefaultWorkingDirectory)\packagesconfighash.txt"
workingDirectory: '$(System.DefaultWorkingDirectory)'
- task: CacheBeta@0 # speed up builds by caching packages folder
inputs:
key: nuget|1|$(Agent.OS)|$(Build.SourcesDirectory)\packagesconfighash.txt # hash map generated in previous step
path: $(Build.SourcesDirectory)\packages
cacheHitVar: 'nuget_cache_hit' # variable that can be checked to see if it was successful
displayName: Cache nuget packages
continueOnError: true
- task: NuGetToolInstaller@0
displayName: 'Use NuGet 5.*'
inputs:
versionSpec: '5.*'
- task: NuGetCommand@2
displayName: 'NuGet restore'
inputs:
command: 'restore'
restoreSolution: '$(Solution)'
feedsToUse: 'config'
nugetConfigPath: 'NuGet.config'
- task: PowerShell@2
displayName: 'Calculate and save package.json hash'
inputs:
targetType: 'inline'
script: |
# generates a hash of all package.json and saves each on a single line on 'packagejsonhash.txt'
Get-FileHash -Algorithm MD5 -Path (Get-ChildItem package.json -Recurse) >> packagejsonhash.txt
Write-Host "Hash File saved to $(System.DefaultWorkingDirectory)\packagejsonhash.txt"
workingDirectory: '$(System.DefaultWorkingDirectory)'
- task: CacheBeta@0 # speed up builds by caching node_modules folder
inputs:
key: nuget|1|$(Agent.OS)|$(Build.SourcesDirectory)\packagejsonhash.txt # hash map generated in previous step
path: $(WebProjectDirectory)\node_modules
cacheHitVar: 'npm_cache_hit' # variable that can be checked to see if it was successful
displayName: Cache npm packages
continueOnError: true
- task: Npm@1
displayName: 'NPM Set progress false (speedup)'
inputs:
command: custom
verbose: false
customCommand: 'set progress=false'
- task: Npm@1
displayName: 'npm install'
inputs:
workingDir: '$(WebProjectDirectory)'
verbose: false
- task: Npm@1
displayName: 'npm run build'
inputs:
command: custom
workingDir: '$(WebProjectDirectory)'
verbose: false
customCommand: 'run build'
I hope this helps!