Azure DevOps Cache task

Sometimes the builds are taking a long time. Often the NuGet restore/npm install takes a couple of minutes to download all packages that you use in your solution. One way to speed up that process is to use the “new” Cache task in Azure DevOps.

Pipeline caching (https://docs.microsoft.com/en-us/azure/devops/pipelines/release/caching?view=azure-devops)

It took me a long time to understand how you actually used this task (yes I know, I´m kind of solve…). But when I found the example in jraps20 build.yml(https://gist.github.com/jraps20/ea29de8b45493dcfa0a1ec5b0af82337) I finally understood how it works and how to do this.

 

So, what you could do is add 2 tasks. The first task is a PowerShell task creates a hash of the content in packages.config. That means that as long you don´t change and version of NuGet packages, and/or adding/removing NuGet packages, the hash will be the same. That will decide if the cached packages folder should be used or if it should download Nuget packages into the packages folder.

The second task is actual cache task that use the hash in the cachekey to figure out if there is any cached “packages” folder to reuse.

Note: The Cache task (Cache files between runs) is only supported when you are using YAML.

SearchCacheTaskInAzureDevOps.jpg

Before I implemented the caching my YAML file started like this:

trigger:

  batch: true

  branches:

    include:

    - develop

 

variables:

- group: DXP-variables

 

stages:

- stage: Build

  jobs:

  - job: DevInte

 

    pool:

      name: Azure Pipelines

      vmImage: 'windows-latest'

      demands:

      - npm

      - msbuild

      - visualstudio

      - vstest

 

    variables:

      solution: '**/*.sln'

      buildPlatform: 'Any CPU'

      buildConfiguration: 'Release'

 

    steps:

 

    - task: NuGetToolInstaller@0

      displayName: 'Use NuGet 5.*'

      inputs:

        versionSpec: '5.*'

 

    - task: NuGetCommand@2

      displayName: 'NuGet restore'

      inputs:

        command: 'restore'

        restoreSolution: '$(Solution)'

        feedsToUse: 'config'

        nugetConfigPath: 'NuGet.config'

 

And after implemented the caching tasks it looks like this:

trigger:

  batch: true

  branches:

    include:

    - develop

 

variables:

- group: DXP-variables

 

stages:

- stage: Build

  jobs:

  - job: DevInte

 

    pool:

      name: Azure Pipelines

      vmImage: 'windows-latest'

      demands:

      - npm

      - msbuild

      - visualstudio

      - vstest

 

    variables:

      solution: '**/*.sln'

      buildPlatform: 'Any CPU'

      buildConfiguration: 'Release'

 

    steps:

 

    - task: PowerShell@2

      displayName: 'Calculate and save packages.config hash'

      inputs:

        targetType: 'inline'

        script: |

          # generates a hash of all packages.config and saves each on a single line on 'hash.txt'

          Get-FileHash -Algorithm MD5 -Path (Get-ChildItem packages.config -Recurse) >> hash.txt

          Write-Host "Hash File saved to $(System.DefaultWorkingDirectory)\hash.txt"

        workingDirectory: '$(System.DefaultWorkingDirectory)'

 

    - task: CacheBeta@0 # speed up builds by caching packages folder

      inputs:

        key: nuget|1|$(Agent.OS)|$(Build.SourcesDirectory)\hash.txt # hash map generated in previous step

        path: $(Build.SourcesDirectory)\packages

        cacheHitVar: 'nuget_cache_hit' # variable that can be checked to see if it was successful

      displayName: Cache nuget packages

      continueOnError: true

 

    - task: NuGetToolInstaller@0

      displayName: 'Use NuGet 5.*'

      inputs:

        versionSpec: '5.*'

 

    - task: NuGetCommand@2

      displayName: 'NuGet restore'

      inputs:

        command: 'restore'

        restoreSolution: '$(Solution)'

        feedsToUse: 'config'

        nugetConfigPath: 'NuGet.config'

 

Result

You never really know how much this will fast up you build. But an example from my last project made the build pipeline become around 1.5 minutes faster. (If I have no packages.config changes of course). And of course, it depends on how many NuGet packages you use etc.

Example before:

In our case in this project we use around 115 NuGet packages and the folder weight around 400Mb. It takes around 2 minutes (if lucky) to download these packages.

ResultBeforeCache.jpg

 

Example after:

And with the cache task we managed to get the same task to run in 30 seconds.

ResultAfterCache.jpg

NPM

I have not given you any example of how to do this with NPM packages. Here is an example of YAML that cache both NuGet and NPM packages.

trigger:

  batch: true

  branches:

    include:

    - develop

 

variables:

- group: DXP-variables

- name: webProjectDirectory

  value: '$(Build.SourcesDirectory)\Customer.Web'

 

stages:

- stage: Build

  jobs:

  - job: DevInte

 

    pool:

      name: Azure Pipelines

      vmImage: 'windows-latest'

      demands:

      - npm

      - msbuild

      - visualstudio

      - vstest

 

    variables:

      solution: '**/*.sln'

      buildPlatform: 'Any CPU'

      buildConfiguration: 'Release'

 

    steps:

    - task: PowerShell@2

      displayName: 'Calculate and save packages.config hash'

      inputs:

        targetType: 'inline'

        script: |

          # generates a hash of all packages.config and saves each on a single line on 'packagesconfighash.txt'

          Get-FileHash -Algorithm MD5 -Path (Get-ChildItem packages.config -Recurse) >> packagesconfighash.txt

          Write-Host "Hash File saved to $(System.DefaultWorkingDirectory)\packagesconfighash.txt"

        workingDirectory: '$(System.DefaultWorkingDirectory)'

 

    - task: CacheBeta@0 # speed up builds by caching packages folder

      inputs:

        key: nuget|1|$(Agent.OS)|$(Build.SourcesDirectory)\packagesconfighash.txt # hash map generated in previous step

        path: $(Build.SourcesDirectory)\packages

        cacheHitVar: 'nuget_cache_hit' # variable that can be checked to see if it was successful

      displayName: Cache nuget packages

      continueOnError: true

 

    - task: NuGetToolInstaller@0

      displayName: 'Use NuGet 5.*'

      inputs:

        versionSpec: '5.*'

 

    - task: NuGetCommand@2

      displayName: 'NuGet restore'

      inputs:

        command: 'restore'

        restoreSolution: '$(Solution)'

        feedsToUse: 'config'

        nugetConfigPath: 'NuGet.config'

 

    - task: PowerShell@2

      displayName: 'Calculate and save package.json hash'

      inputs:

        targetType: 'inline'

        script: |

          # generates a hash of all package.json and saves each on a single line on 'packagejsonhash.txt'

          Get-FileHash -Algorithm MD5 -Path (Get-ChildItem package.json -Recurse) >> packagejsonhash.txt

          Write-Host "Hash File saved to $(System.DefaultWorkingDirectory)\packagejsonhash.txt"

        workingDirectory: '$(System.DefaultWorkingDirectory)'

 

    - task: CacheBeta@0 # speed up builds by caching node_modules folder

      inputs:

        key: nuget|1|$(Agent.OS)|$(Build.SourcesDirectory)\packagejsonhash.txt # hash map generated in previous step

        path: $(WebProjectDirectory)\node_modules

        cacheHitVar: 'npm_cache_hit' # variable that can be checked to see if it was successful

      displayName: Cache npm packages

      continueOnError: true

 

    - task: Npm@1

      displayName: 'NPM Set progress false (speedup)'

      inputs:

        command: custom

        verbose: false

        customCommand: 'set progress=false'

 

    - task: Npm@1

      displayName: 'npm install'

      inputs:

        workingDir: '$(WebProjectDirectory)'

        verbose: false

 

    - task: Npm@1

      displayName: 'npm run build'

      inputs:

        command: custom

        workingDir: '$(WebProjectDirectory)'

        verbose: false

        customCommand: 'run build'

 

I hope this helps!

Les mer om