Automating UI change verification with Android Compose Screenshot Testing — Part2

Introduction

In Part 1, we explored how Compose Screenshot Testing works. However, running screenshot tests manually in a local environment can be inefficient and time-consuming, especially for teams handling frequent UI changes. Automating this process helps catch visual regressions early without manual effort.

In this guide, we will set up GitHub Actions to automatically run screenshot tests when a PR is submitted and display the results as comments. The final output will look similar to the image below, providing clear visibility into UI changes directly within the pull request. Additionally, all code links will be provided at the bottom of the post!

Workflow Overview

The automation follows these steps:

1. A PR is created.
2. GitHub Actions detects the PR and runs validation.
3. If validation fails, images are uploaded to a temporary branch.
4. GitHub Actions posts a comment with markdown-formatted test results, including images.
5. Developers review the changes and merge the PR.
6. Once merged, the temporary branch is deleted.
7. Screenshot expectations are updated based on the main branch.

To implement this, we will create three workflow files:

• screenshot-validate.yml: Runs screenshot validation when a PR is created.
• screenshot-delete-companion-branch.yml: Deletes temporary branches after merging.
• screenshot-update.yml: Updates screenshot expectations upon merging into the main branch.

1. Validating Screenshots (screenshot-validate.yml)

First, we run the screenshot validation process. If the validation fails, an exception is thrown. However, we need to ensure that the GitHub Action does not stop execution, so we use continue-on-error: true.

- name: Run screenshot validation
        run: ./gradlew validateDemoScreenshotTest
        continue-on-error: true

If differences are found, the test results are stored in the rendered/ and diffs/ directories. We extract these images into text files (rendered_results.txt and diffs_results.txt) and set a boolean flag to indicate whether changes were detected.

- name: Find all screenshot test results
        id: find_results
        run: |
          find . -type f -iname "*.png" -path "*/rendered/*" > rendered_results.txt
          find . -type f -iname "*.png" -path "*/diffs/*" > diffs_results.txt
          echo "rendered_results : $(cat rendered_results.txt)"
          echo "diffs_results : $(cat diffs_results.txt)"
          if [ -s rendered_results.txt ]; then
            echo "results_found=true" >> $GITHUB_ENV
          else
            echo "results_found=false" >> $GITHUB_ENV
          fi

If differences are detected, we create a temporary branch from the PR branch to store the images. This branch is solely used for image uploads and does not affect any other processes. If you have an alternative CDN for image storage, you can use that instead of creating a temporary branch. After the temporary branch is created, the images identified in rendered_results.txt and diffs_results.txt are uploaded to this branch. These uploaded images will later be referenced in the PR comment.

      - name: Create companion branch and push images
        if: env.results_found == 'true'
        run: |
          BRANCH_NAME="companion_${{ github.event.pull_request.head.ref }}"
          BRANCH_NAME=$(echo "$BRANCH_NAME" | sed 's/[^a-zA-Z0-9_-]/_/g')
          echo "Creating branch: $BRANCH_NAME"
          
          git branch -D "$BRANCH_NAME" || true
          git checkout --orphan "$BRANCH_NAME"
          git rm -rf .

Next, we generate a markdown-formatted table displaying the test results and images. To do this, we iterate over rendered_results.txt, extract the file names and construct their URLs. The images need to have ?raw=true appended to the URL to be displayed correctly in the PR comment. Once extracted, we format the results into a markdown table.

      - id: generate-diff-reports
        name: Generate diff reports
        if: env.results_found == 'true'
        run: |
          delimiter="$(openssl rand -hex 8)"
          
          rendered_results=$(cat rendered_results.txt)
          diffs_results=$(cat diffs_results.txt)
          
          rendered_array=($rendered_results)
          diffs_array=($diffs_results)
          echo "diffs_array: ${diffs_array[@]}"
          
          echo "Generating markdown table for paired results:"
          echo "Rendered results:"
          echo "$rendered_results"
          echo "Diff results:"
          echo "$diffs_results"
          
          {
            echo "markdown_table<<${delimiter}"
          
            echo "| Rendered File | Rendered Image | Diff Image |"
            echo "|---------------|----------------|------------|"
          
            for i in "${!rendered_array[@]}"; do
              rendered_file="${rendered_array[$i]}"
              diff_file="${diffs_array[$i]:-}"
          
              rendered_file_name=$(basename "$rendered_file" | sed -r 's/(.{20})/\\1<br>/g')
              rendered_url_part="companion_${{ github.event.pull_request.head.ref }}/${rendered_file//#/%23}"
          
              diff_file_name=""
              diff_url_part=""
              if [ -n "$diff_file" ]; then
                diff_file_name=$(basename "$diff_file" | sed -r 's/(.{20})/\\1<br>/g')
                diff_url_part="companion_${{ github.event.pull_request.head.ref }}/${diff_file//#/%23}"
              fi
          
              echo "| [$rendered_file_name](<https://github.com/$>{{ github.repository }}/blob/$rendered_url_part) | ![](<https://github.com/$>{{ github.repository }}/blob/$rendered_url_part?raw=true) | ![](<https://github.com/$>{{ github.repository }}/blob/$diff_url_part?raw=true) |"
            done
          
            echo "${delimiter}"
          } >> "$GITHUB_OUTPUT"
          echo "Markdown table generated successfully."

Finally, we post the generated table as a comment on the PR.

      - name: Post comment to Pull Request
        if: env.results_found == 'true'
        run: |
          markdown_table="${{ steps.generate-diff-reports.outputs.markdown_table }}"
          echo "Posting the following markdown table as a comment:"
          echo "$markdown_table"
          
          body="**Screenshot Test Results**"$'\\n'"$markdown_table"
          curl -X POST \\
            -H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" \\
            -H "Content-Type: application/json" \\
            -d "$(jq -n --arg body "$body" '{ body: $body }')" \\
            ${{ github.event.pull_request.comments_url }}

2. Deleting the Temporary Branch (screenshot-delete-companion-branch.yml)

Once the PR is merged, we need to delete the temporary branch used for storing uploaded images. However, deleting the branch will also remove the images from the PR comment. If this is an issue, you might consider scheduling periodic deletions instead. Alternatively, using an external CDN instead of a GitHub branch will prevent images from being lost.

This workflow requires contents: write permission.

permissions:
  contents: write

Extract the PR branch name and append companion_ to identify the temporary branch.

      - name: Extract branch name
        run: |
          BRANCH_NAME="companion_${{ github.event.pull_request.head.ref }}"
          BRANCH_NAME=$(echo "$BRANCH_NAME" | sed 's/[^a-zA-Z0-9_-]/_/g')
          echo "TARGET_BRANCH=$BRANCH_NAME" >> $GITHUB_ENV

Delete the branch if it exists.

      - name: Delete companion branch
        run: |
          if git ls-remote --exit-code --heads origin "$TARGET_BRANCH"; then
            echo "Deleting branch: $TARGET_BRANCH"
            git push origin --delete "$TARGET_BRANCH"
          else
            echo "Branch $TARGET_BRANCH does not exist. Skipping deletion."
          fi

3. Updating Screenshot Expectations (screenshot-update.yml)

Once a PR is merged into the main branch, we need to update the expected screenshot values. This ensures that future PRs have an up-to-date baseline for comparison.

This process also requires contents: write permission.

permissions:
  contents: write

We then update the screenshot references so that future comparisons use the latest approved visuals.

      - name: Run update screenshot
        run: ./gradlew updateDemoScreenshotTest

Finally, we commit and push the updated screenshots to main, completing the update cycle.

      - name: Configure git for push
        run: |
          git config --global user.name "github-actions[bot]"
          git config --global user.email "github-actions[bot]@users.noreply.github.com"
          git remote set-url origin https://x-access-token:${{ secrets.GITHUB_TOKEN }}@github.com/WonJoongLee/FridayMovie.git

      - name: Push screenshot
        run: |
          git add .
          git commit -m "🤖 Update Screenshot"
          git push origin $(git branch --show-current)

Conclusion

By automating screenshot testing with GitHub Actions, we can streamline our UI validation process, ensuring that visual regressions are caught early and reducing the manual effort required to verify UI changes. This workflow helps maintain a high standard of UI consistency while providing clear visibility into changes directly within pull requests.

With this setup, developers can focus more on feature development and less on manually capturing and comparing screenshots. As a result, teams can work more efficiently, confidently merge UI changes, and maintain a polished user experience across their applications.

All the code mentioned in this post is available in this GitHub repository. You can also find each file for the automation: screenshot-validate.yml, screenshot-delete-companion-branch.yml and screenshot-update.yml.

Reference

• Compose Preview Screenshot Testing Documentation
• DroidKaigi 2023