Pull Docker image from ACR in pipelines

Azure Containers wallpaper

What does it mean pull Docker image from ACR in pipelines? It was difficult to find a title for this post.

First, I give you the scenario. In Azure, I have created an Azure Container Repository where I store the Docker containers. In my previous posts, I talked how to deploy a ShinyApp via pipeline in Azure DevOps.

Scenario

So, Azure DevOps has a connection with the Azure Container Repository already. The Azure DevOps pipeline builds the Docker container and pushes it in the Azure Container Repository. Then, the pipeline connects to the virtual server where I host all the applications (in this case ShinyApps) via SSH to pull the new image and run the container.

This solution is working. For your benefit, here you have the files.

azure-pipeline-yml

# Docker
# Build and push an image to Azure Container Registry
# https://docs.microsoft.com/azure/devops/pipelines/languages/docker

trigger:
- main

resources:
- repo: self

variables:
  # Container registry service connection established during pipeline creation
  dockerRegistryServiceConnection: 'yourconnection'
  # Insert a name for your container
  imageRepository: 'shinyappdemo'
  containerRegistry: $(ACRLoginServer)
  dockerfilePath: '$(Build.SourcesDirectory)/DOCKERFILE'
  tag: '$(Build.BuildId)'
  
  # Agent VM image name
  vmImageName: 'ubuntu-latest'

stages:
- stage: Build
  displayName: Build and push stage
  jobs:  
  - job: Check
    condition: eq('${{ variables.imageRepository }}', '')
    steps:
      - script: |
          echo '##[error] The imageRepository must have a value!'
          exit 1
  - job: Build
    condition: not(eq('${{ variables.imageRepository }}', ''))
    displayName: Build
    pool:
      vmImage: $(vmImageName)
    steps:
    - task: Docker@2
      displayName: Build and push an image to container registry
      inputs:
        command: buildAndPush
        repository: $(imageRepository)
        dockerfile: $(dockerfilePath)
        containerRegistry: $(dockerRegistryServiceConnection)
        tags: |
          latest
    - task: SSH@0
      displayName: 'Run shell commands on remote machine'
      inputs:
        sshEndpoint: 'ShinyServerDev'
        commands: |
          echo $(SSHPassword) | sudo -S docker pull $(containerRegistry)/$(imageRepository)
        failOnStdErr: false
      continueOnError: true

Dockerfile

FROM openanalytics/r-base

ARG project=testApp

# install Debian dependencies for R
RUN apt-get update && apt-get install -y \
    sudo \
    pandoc \
    pandoc-citeproc \
    libcurl4-gnutls-dev \
    libcairo2-dev \
    libxt-dev \
    libssl-dev \
    libssh2-1-dev \
    libxml2-dev \
    libgit2-dev 

# packages needed renv and install
RUN R -e "install.packages('shiny', repos='https://cran.rstudio.com')"
RUN R -e "install.packages('shinydashboard', repos='https://cran.rstudio.com')"
RUN R -e "install.packages('shinythemes', repos='https://cran.rstudio.com')"

# create root folder for app in container
RUN mkdir /root/${project}

# copy the app to the image
COPY app /root/${project}

COPY Rprofile.site /usr/lib/R/etc/
EXPOSE 3838

# remember to update this line
CMD ["R", "-e", "shiny::runApp('/root/testApp')"]

So, how you see in the Dockerfile I pull the container from openanalytics/r-base. That mean, every time I have to install all the application I need and the pipeline takes ages to build the containers. For this reason, I decided to have a container to use as base for all my applications.

Then, I created this container and push in the Azure Container Registry (ACR). So, I changed the Dockerfile to pull the container from my ACR. An immediately I get an error

Denied: retrieving permissions failed in Azure DevOps pipeline

Denied: retrieving permissions failed in Azure DevOps pipeline - Pull Docker image from ACR in pipelines

Denied: retrieving permissions failed in Azure DevOps pipeline

So, I checked the Service connection in the Project Settings but everything is correct.

Service connection in Azure DevOps - Pull Docker image from ACR in pipelines
Service connection in Azure DevOps

I have tried to change the pipeline adding the login to the ACR but it didn’t work. What can I do?

Solution

So, I don’t want to bored you with everything I tried but I spent hours to find the solution. First, this is the Dockerfile that pull the container from the ACR.

Be careful. The file name must be Dockerfile (no capital letters)

# OS & Base R Set Up
FROM youracr.azurecr.io/myrbase
RUN apt-get update && apt-get install -y  libicu-dev make pandoc pandoc-citeproc && rm -rf /var/lib/apt/lists/*
RUN echo "options(repos = c(CRAN = 'https://cran.rstudio.com/'), download.file.method = 'libcurl')" 
    >> /usr/local/lib/R/etc/Rprofile.site

# Install renv to restore all Shiny app deps
RUN R -e "install.packages('renv'); renv::consent(provided = TRUE)"

# create root folder for app in container
RUN mkdir /root/app

# Define GITHUB_PAT
ARG github_pat=#{github_pat}#

# Restore Shiny app Deps
COPY renv.lock /root/app
RUN R -e "Sys.setenv(GITHUB_PAT='${github_pat}'); renv::restore(lockfile = '/root/app/renv.lock')"

# copy shiny app to image folder
COPY inst/shiny /root/app
EXPOSE 3838

# launch shiny app
CMD ["R", "-e", "options('shiny.port'=3838,shiny.host='0.0.0.0');shiny::runApp('/root/app')"]

Now, to connect properly to the ACR, we have to use a Docker Compose. So, in the root of your project, you have to create a docker-compose.yml and here you have the content

version: "3.7"
services:
  web:
    build: .
    ports: 
      - '3838:3838'

Very basic! Now, the funny part with the pipeline. Edit the pipeline. In the Tasks list, search for docker.

Azure pipeline Docker Compose
Azure pipeline Docker Compose

And here the magic happens. When you click on it, you have a form with a long list of configurations but we are interested in those:

  • Container Registry Type
  • Azure subscription
  • Azure Container Registry
  • Docker Compose file
  • Action

Container Registry Type is already selected with Azure Container Registry. Select your Azure subscription and automatically you can select from the Azure Container Registry list what ACR you want to use. Then, from the dropdown list Action select Build service images. Click Add.

Azure DevOps Docker Compose
Azure DevOps Docker Compose

After this command the container will be created but not pushed to the ACR. So, we have to add another step for it. Then, again add another Docker Compose but this time select as an Action the option Push services.

Now, your pipeline could be similar to this one (azureSubscription and azureContainerRegistry are a long string)

# Docker
# Build and push an image to Azure Container Registry
# https://docs.microsoft.com/azure/devops/pipelines/languages/docker

trigger:
- main

resources:
- repo: self

variables:
  # Container registry service connection established during pipeline creation
  dockerRegistryServiceConnection: '...'
  # Insert a name for your container
  imageRepository: 'p200403'
  containerRegistry: $(ACRLoginServer)
  dockerfilePath: '$(Build.SourcesDirectory)/DOCKERFILE'
  tag: '$(Build.BuildId)'
  
  # Agent VM image name
  vmImageName: 'ubuntu-latest'

stages:
- stage: Build
  displayName: Build and push stage
  jobs:  
  - job: Check
    condition: eq('${{ variables.imageRepository }}', '')
    steps:
      - script: |
          echo '##[error] The imageRepository must have a value!'
          exit 1
  - job: Build
    condition: not(eq('${{ variables.imageRepository }}', ''))
    displayName: Build
    pool:
      vmImage: $(vmImageName)
    steps:
    - task: qetza.replacetokens.replacetokens-task.replacetokens@3
      displayName: 'Replace tokens'
      inputs:
        targetFiles: '**/Dockerfile'
    - task: DockerCompose@0
      inputs:
        containerregistrytype: 'Azure Container Registry'
        azureSubscription: '...'
        azureContainerRegistry: '...'
        dockerComposeFile: '**/docker-compose.yml'
        action: 'Build services'
        projectName: 'p200403'
    - task: DockerCompose@0
      inputs:
        containerregistrytype: 'Azure Container Registry'
        azureSubscription: '...'
        azureContainerRegistry: '...'
        dockerComposeFile: '**/docker-compose.yml'
        action: 'Push services'
        projectName: 'p200403'
    - task: SSH@0
      displayName: 'Run shell commands on remote machine'
      inputs:
        sshEndpoint: 'ShinyServerDev'
        commands: |
          echo $(SSHPassword) | sudo -S docker pull $(containerRegistry)/$(imageRepository)_web
        failOnStdErr: false
      continueOnError: true

The last thing to say is about azureSubscription and azureContainerRegistry and it is a bad news. I am trying to replace them using parameters. I can print any parameter inside the yaml. But when I use parameter with any task that requires Azure Subscription that will make the pipeline always fail with

“The pipeline is not valid. Job myDeployment: Step input azureSubscription references service connection $(mySubscription) which could not be found.”

This is a known issue / limitation. You have to pass the Azure subscription as a literal. No way around it that I know of, unfortunately.

It’s been a point of discussion for literally years on this GitHub issue: https://github.com/microsoft/azure-pipelines-agent/issues/1307

Conclusion

So, what do you think about “Pull Docker image from ACR in pipelines”? Please leave a comment or use our Forum.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.