Create an S3 bucket in a localstack Github Action service

gkatsanos · June 14, 2024, 7:25am

I’m trying to integrate localstack to my cypress github action:

jobs:
  cypress:
    runs-on: default-visable
    container:
      image: cypress/browsers:node-20.13.0-chrome-124.0.6367.155-1-ff-125.0.3-edge-124.0.2478.80-1
    env:
      GITHUB_TOKEN: ${{ secrets.PACKAGES_READ_ONLY_TOKEN_GITHUB }}
      PUBLIC_API_URL: "https://api.staging.visable.io"
      IS_CYPRESS: "true"
      BASE_URL: "https://www.wlw-staging.de"
      REQUESTS_INTERNAL_API: "https://requests-service.internal.wlw-1.staging.visable.cloud/requests_service/internal_api"
      RECAPTCHA_SITE_KEY: "6LeIxAcTAAAAAJcZVRqyHh71UMIEGNQ_MXjiZKhI"
      RECAPTCHA_SECRET_KEY: "6LeIxAcTAAAAAGG-vFI1TnRWxMZNFuojJ4WifJWe"
      S3_BUCKET_ARN: "arn:aws:s3:::requests-frontend-local"
      AWS_REGION: "eu-central-1"
    steps:
      - uses: pnpm/action-setup@v4
        with:
          version: 9.1.1
      - name: Define proper Checkout REF
        shell: bash
        run: echo "ref=$(echo ${GITHUB_HEAD_REF:-${GITHUB_REF}})" >> $GITHUB_OUTPUT
        id: extract_ref
      - uses: actions/checkout@v4
        with:
          ref: "${{ steps.extract_ref.outputs.ref }}"
      - name: setup github packages authentication
        run: 'echo "//npm.pkg.github.com/:_authToken=${{ secrets.PACKAGES_READ_ONLY_TOKEN_GITHUB }}" > ~/.npmrc'
      - uses: cypress-io/github-action@v6
        id: cypress
        with:
          build: pnpm build
          start: pnpm start
          browser: chrome
      - name: cleanup repo files
        if: always()
        run: find . -mindepth 1 -delete
    services:
      localstack:
        image: localstack/localstack
        ports:
          - 4566:4566
        env:
          SERVICES: s3

The problem is that I somehow need to create the s3 bucket “requests-frontend-local” that my application is using. I guess for that I need awslocal ? And this is where I get stuck. The documentation shows there’s a with that includes it, GitHub Actions | Docs , but I can’t use with in services.

Marcel · June 14, 2024, 1:57pm

Hi @gkatsanos, The LocalStack Docker image comes with AWS CLI pre-installed, so there’s no need for additional setup. You can invoke it as desired.

I would still recommend running LocalStack using the steps provided in our documentation, GitHub Actions | Docs (localstack.cloud).

gkatsanos · June 17, 2024, 11:24am

@Marcel do you guys might have a guide on how to start localStack alongside a cypress container?

Marcel · June 17, 2024, 12:08pm

Hi @gkatsanos,

We do not have samples with cypress tool.

I suggest installing the necessary software on the runner and trying again.
You may need to check the Network Troubleshooting section in the LocalStack documentation to establish a connection between the two Docker containers.

gkatsanos · June 17, 2024, 1:27pm

Is there a way to get more support?
I was able to advance by using the action you recommended, but localstack is not found:

jobs:
  cypress:
    runs-on: default-visable
    env:
      GITHUB_TOKEN: ${{ secrets.PACKAGES_READ_ONLY_TOKEN_GITHUB }}
      IS_CYPRESS: "true"
      S3_BUCKET_ARN: "arn:aws:s3:::requests-frontend-local"
      AWS_REGION: "eu-central-1"
    steps:
      - name: Start LocalStack
        uses: LocalStack/setup-localstack@v0.2.0
        with:
          image-tag: 'latest'
          install-awslocal: 'true'
          configuration: DEBUG=1
      - name: Start localstack s3 bucket
        run: awslocal s3api create-bucket --bucket requests-frontend-local
      - uses: pnpm/action-setup@v4
        with:
          version: 9.1.1
      - name: Define proper Checkout REF
        shell: bash
        run: echo "ref=$(echo ${GITHUB_HEAD_REF:-${GITHUB_REF}})" >> $GITHUB_OUTPUT
        id: extract_ref
      - uses: actions/checkout@v4
        with:
          ref: "${{ steps.extract_ref.outputs.ref }}"
      - name: setup github packages authentication
        run: 'echo "//npm.pkg.github.com/:_authToken=${{ secrets.PACKAGES_READ_ONLY_TOKEN_GITHUB }}" > ~/.npmrc'
      - uses: cypress-io/github-action@v6
        id: cypress
        with:
          build: pnpm build
          start: pnpm start
          browser: chrome
      - name: cleanup repo files
        if: always()
        run: find . -mindepth 1 -delete

but it fails with

/opt/actions-runner/_work/_temp/803d36be-4da4-4572-8427-4e304cad23d7.sh: line 12: localstack: command not found

467

Marcel · June 18, 2024, 5:39am

Priority Support is included in our paid plans. For more information, please visit our pricing page at Pricing (localstack.cloud).

It looks like the LocalStack CLI is not installed during the setup steps.

Maybe updating the step would help? Please check the full logs where the step Start LocalStack failed.

      - name: Setup LocalStack
        uses: localstack/setup-localstack@v0.1.2
        with:
          image-tag: latest   
          configuration: DEBUG=1
      - name: Create S3 Bucket
        run: awslocal s3api create-bucket --bucket requests-frontend-local

gkatsanos · June 18, 2024, 6:50am

I updated the setup action to 0.2.1 which was released a couple of days ago. Here’s the full log of the setup step:

Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: pip in /usr/lib/python3.9/site-packages (21.3.1)
Collecting pip
  Downloading pip-24.0-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
  WARNING: The scripts pip, pip3, pip3.10 and pip3.9 are installed in '/home/ec2-user/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed pip-24.0
which: no localstack in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/var/lib/snapd/snap/bin)
Defaulting to user installation because normal site-packages is not writeable
Collecting localstack
  Downloading localstack-3.5.0.tar.gz (5.7 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Installing backend dependencies: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting localstack-core (from localstack)
  Downloading localstack_core-3.5.0-py3-none-any.whl.metadata (5.5 kB)
Collecting localstack-ext==3.5.0 (from localstack)
  Downloading localstack_ext-3.5.0.tar.gz (5.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.5/5.5 MB 56.0 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting dill>=0.3.2 (from localstack-ext==3.5.0->localstack)
  Downloading dill-0.3.8-py3-none-any.whl.metadata (10 kB)
Collecting dnslib>=0.9.10 (from localstack-ext==3.5.0->localstack)
  Downloading dnslib-0.9.24-py3-none-any.whl.metadata (19 kB)
Collecting dnspython>=1.16.0 (from localstack-ext==3.5.0->localstack)
  Downloading dnspython-2.6.1-py3-none-any.whl.metadata (5.8 kB)
Collecting packaging (from localstack-ext==3.5.0->localstack)
  Downloading packaging-24.1-py3-none-any.whl.metadata (3.2 kB)
Collecting build (from localstack-ext==3.5.0->localstack)
  Downloading build-1.2.1-py3-none-any.whl.metadata (4.3 kB)
Collecting plux>=1.10.0 (from localstack-ext==3.5.0->localstack)
  Using cached plux-1.10.0-py3-none-any.whl.metadata (10.0 kB)
Collecting pyaes>=1.6.0 (from localstack-ext==3.5.0->localstack)
  Downloading pyaes-1.6.1.tar.gz (28 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Installing backend dependencies: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: awscli in /usr/lib/python3.9/site-packages (from awscli-local[ver1]) (2.15.30)
Requirement already satisfied: colorama<0.4.7,>=0.2.5 in /usr/lib/python3.9/site-packages (from awscli->awscli-local[ver1]) (0.4.4)
Requirement already satisfied: docutils<0.20,>=0.10 in /usr/lib/python3.9/site-packages (from awscli->awscli-local[ver1]) (0.16)
Requirement already satisfied: cryptography<40.0.2,>=3.3.2 in /usr/lib64/python3.9/site-packages (from awscli->awscli-local[ver1]) (36.0.1)
Requirement already satisfied: ruamel.yaml<=0.17.21,>=0.15.0 in /usr/lib/python3.9/site-packages (from awscli->awscli-local[ver1]) (0.16.6)
Requirement already satisfied: prompt-toolkit<3.0.39,>=3.0.24 in /usr/lib/python3.9/site-packages (from awscli->awscli-local[ver1]) (3.0.24)
Requirement already satisfied: distro<1.9.0,>=1.5.0 in /usr/lib/python3.9/site-packages (from awscli->awscli-local[ver1]) (1.5.0)
Requirement already satisfied: awscrt<=0.19.19,>=0.19.18 in /usr/lib64/python3.9/site-packages (from awscli->awscli-local[ver1]) (0.19.19)
Requirement already satisfied: python-dateutil<=2.8.2,>=2.1 in /usr/lib/python3.9/site-packages (from awscli->awscli-local[ver1]) (2.8.1)
Requirement already satisfied: jmespath<1.1.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from awscli->awscli-local[ver1]) (0.10.0)
Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from awscli->awscli-local[ver1]) (1.25.10)
Collecting boto3 (from localstack-client->awscli-local[ver1])
  Downloading boto3-1.34.128-py3-none-any.whl.metadata (6.6 kB)
Requirement already satisfied: cffi>=1.12 in /usr/lib64/python3.9/site-packages (from cryptography<40.0.2,>=3.3.2->awscli->awscli-local[ver1]) (1.14.5)
Requirement already satisfied: wcwidth in /usr/lib/python3.9/site-packages (from prompt-toolkit<3.0.39,>=3.0.24->awscli->awscli-local[ver1]) (0.2.5)
Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil<=2.8.2,>=2.1->awscli->awscli-local[ver1]) (1.15.0)
Collecting botocore<1.35.0,>=1.34.128 (from boto3->localstack-client->awscli-local[ver1])
  Downloading botocore-1.34.128-py3-none-any.whl.metadata (5.7 kB)
Collecting s3transfer<0.11.0,>=0.10.0 (from boto3->localstack-client->awscli-local[ver1])
  Downloading s3transfer-0.10.1-py3-none-any.whl.metadata (1.7 kB)
Requirement already satisfied: pycparser in /usr/lib/python3.9/site-packages (from cffi>=1.12->cryptography<40.0.2,>=3.3.2->awscli->awscli-local[ver1]) (2.20)
Requirement already satisfied: ply==3.11 in /usr/lib/python3.9/site-packages (from pycparser->cffi>=1.12->cryptography<40.0.2,>=3.3.2->awscli->awscli-local[ver1]) (3.11)
Downloading boto3-1.34.128-py3-none-any.whl (139 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 139.2/139.2 kB 9.3 MB/s eta 0:00:00
Downloading botocore-1.34.128-py3-none-any.whl (12.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.3/12.3 MB 89.6 MB/s eta 0:00:00
Downloading s3transfer-0.10.1-py3-none-any.whl (82 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 82.2/82.2 kB 14.3 MB/s eta 0:00:00
Building wheels for collected packages: awscli-local, localstack-client
  Building wheel for awscli-local (pyproject.toml): started
  Building wheel for awscli-local (pyproject.toml): finished with status 'done'
  Created wheel for awscli-local: filename=awscli_local-0.22.0-py3-none-any.whl size=11972 sha256=6fbe2bc668fcfe0e802400f50f680e6d4cb1233a4e10f9e5fbb9e057712dd33a
  Stored in directory: /home/ec2-user/.cache/pip/wheels/45/ba/e8/d7e3f4d3cb867897a90afcf0967fc990644c133fbe2f17f9dc
  Building wheel for localstack-client (pyproject.toml): started
  Building wheel for localstack-client (pyproject.toml): finished with status 'done'
  Created wheel for localstack-client: filename=localstack_client-2.5-py3-none-any.whl size=10230 sha256=2a65561824e3e8b9988e74726981d7459aff21bd6d2c70db0800d4dc5730c9d9
  Stored in directory: /home/ec2-user/.cache/pip/wheels/2d/56/29/f965e48679a3cfd7e3d611003c78762c6398ade2fbed5200db
Successfully built awscli-local localstack-client
Installing collected packages: botocore, s3transfer, boto3, localstack-client, awscli-local
Successfully installed awscli-local-0.22.0 boto3-1.34.128 botocore-1.34.128 localstack-client-2.5 s3transfer-0.10.1
Run if [ "$USE_PRO" = true ]; then
/opt/actions-runner/_work/_temp/b0461859-4c10-4dff-b5d2-b5d576298dc6.sh: line 14: localstack: command not found
stable: Pulling from localstack/localstack

anon51130488 · June 18, 2024, 7:27am

gkatsanos:

WARNING: The scripts pip, pip3, pip3.10 and pip3.9 are installed in '/home/ec2-user/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.

Could you please add your Python install to the PATH and retry the run?
localstack is installed by pip hence the executable lives in the same bin folder most probably.

gkatsanos · June 18, 2024, 7:48am

I was able to move on from this with

          export PATH=$PATH:/home/ec2-user/.local/bin
          echo "PATH=$PATH" >> $GITHUB_ENV

(the aws linux runner was using a different pip folder)

but now I’m stuck in the awslocal s3api create-bucket --bucket requests-frontend-local command, error/log:

  awslocal s3api create-bucket --bucket requests-frontend-local
  shell: /usr/bin/bash -e {0}
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/var/lib/snapd/snap/bin:/home/ec2-user/.local/bin
    GH_ACTION_ROOT: ./../../_actions/LocalStack/setup-localstack/v0.2.1

cannot import name 'SSOTokenFetcher' from 'botocore.utils' (/home/ec2-user/.local/lib/python3.9/site-packages/botocore/utils.py)
Error: Process completed with exit code 255.

anon51130488 · June 18, 2024, 8:02am

This seems to be a dependency clash as it’s reported here:

I’d try to uninstall the local awscli and install it with pip so the dependencies are in sync.

gkatsanos · June 18, 2024, 9:15am

set install-awslocal: 'false' and added a step for manually installing aws-local :

    - name: install awslocal
      run: pip install awscli-local

if so, this also didn’t work, same error as above:

Run pip install awscli-local
  
Defaulting to user installation because normal site-packages is not writeable
Collecting awscli-local
  Downloading awscli-local-0.22.0.tar.gz (11 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Installing backend dependencies: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting localstack-client (from awscli-local)
  Downloading localstack-client-2.5.tar.gz (10 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Installing backend dependencies: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting boto3 (from localstack-client->awscli-local)
  Downloading boto3-1.34.128-py3-none-any.whl.metadata (6.6 kB)
Collecting botocore<1.35.0,>=1.34.128 (from boto3->localstack-client->awscli-local)
  Downloading botocore-1.34.128-py3-none-any.whl.metadata (5.7 kB)
Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3->localstack-client->awscli-local) (0.10.0)
Collecting s3transfer<0.11.0,>=0.10.0 (from boto3->localstack-client->awscli-local)
  Downloading s3transfer-0.10.1-py3-none-any.whl.metadata (1.7 kB)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python3.9/site-packages (from botocore<1.35.0,>=1.34.128->boto3->localstack-client->awscli-local) (2.8.1)
Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.35.0,>=1.34.128->boto3->localstack-client->awscli-local) (1.25.10)
Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.35.0,>=1.34.128->boto3->localstack-client->awscli-local) (1.15.0)
Downloading boto3-1.34.128-py3-none-any.whl (139 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 139.2/139.2 kB 10.5 MB/s eta 0:00:00
Downloading botocore-1.34.128-py3-none-any.whl (12.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.3/12.3 MB 73.9 MB/s eta 0:00:00
Downloading s3transfer-0.10.1-py3-none-any.whl (82 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 82.2/82.2 kB 11.4 MB/s eta 0:00:00
Building wheels for collected packages: awscli-local, localstack-client
  Building wheel for awscli-local (pyproject.toml): started
  Building wheel for awscli-local (pyproject.toml): finished with status 'done'
  Created wheel for awscli-local: filename=awscli_local-0.22.0-py3-none-any.whl size=11972 sha256=4ee96a26798ef9135a2f06bbe15cc43d249d39cfbb005b9b589dc97ab24d4468
  Stored in directory: /home/ec2-user/.cache/pip/wheels/45/ba/e8/d7e3f4d3cb867897a90afcf0967fc990644c133fbe2f17f9dc
  Building wheel for localstack-client (pyproject.toml): started
  Building wheel for localstack-client (pyproject.toml): finished with status 'done'
  Created wheel for localstack-client: filename=localstack_client-2.5-py3-none-any.whl size=10230 sha256=743888195d7345c7f380632f9fbb730d90d8e061e87f0a6179d7f62ee2e49e2e
  Stored in directory: /home/ec2-user/.cache/pip/wheels/2d/56/29/f965e48679a3cfd7e3d611003c78762c6398ade2fbed5200db
Successfully built awscli-local localstack-client
Installing collected packages: botocore, s3transfer, boto3, localstack-client, awscli-local
Successfully installed awscli-local-0.22.0 boto3-1.34.128 botocore-1.34.128 localstack-client-2.5 s3transfer-0.10.1

Run awslocal s3api create-bucket --bucket requests-frontend-local
  

cannot import name 'SSOTokenFetcher' from 'botocore.utils' (/home/ec2-user/.local/lib/python3.9/site-packages/botocore/utils.py)
Error: Process completed with exit code 255.

anon51130488 · June 18, 2024, 9:35am

No, I meant awslocal has an indirect dependency of the awscli as it’s a thin-wrapper around it. But the awscli package and its dependencies are not managed by the install of awslocal so if awscli is installed somewhere else awslocal is unable to resolve the dependencies.
To resolve this uninstall awscli and install via the same pip the action installs it so awslocal can find the dependencies.
Something similar that this post mentions:

Topic		Replies	Views
Localstack v3.0.0 in Github action Lambda invocation failing Platform localstack	1	511	January 29, 2024
LocalStack GitHub actions in self hosted linux not able to access localhost:4566 Platform localstack , docker-image	1	303	March 4, 2024
LocalStack Release v3.4.0 Announcement localstack , announcement , release	2	2537	June 13, 2024
LocalStack release v0.14.4 Announcement localstack , announcement , release	0	997	June 28, 2022
Set up S3 bucket using Docker / Compose Platform localstack , docker-image	3	7467	December 11, 2023

Create an S3 bucket in a localstack Github Action service

Related topics