Github Actions cannot find my poetry-installed dev dependencies and therefore cannot find the aws_cdk module when trying to call cdk deploy

I have a mangum python service with a cdk stack that I can deploy on my local box just fine. Here are my:

pyproject.toml

[tool.poetry]
name = "dics-core-data-service"
version = "0.1.0"
description = ""
authors = ["Clayton <clayton.stetz@gmail.com>"]
readme = "README.md"
package-mode = false

[tool.poetry.dependencies]
python = "^3.12"
fastapi = "^0.115.0"
uvicorn = "^0.31.0"
python-dotenv = "^1.0.1"
pydantic = {extras = ["email"], version = "^2.9.2"}
phonenumbers = "^8.13.46"
pydantic-extra-types = "^2.9.0"
mangum = "^0.19.0"
constructs = "^10.3.0"
psycopg2-binary = "^2.9.9"

[tool.poetry.dev-dependencies]
pytest = "^8.3.3" 
aws-cdk-lib = "^2.161.1"
aws-cdk-aws-lambda-python-alpha = "^2.161.1a0"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

relevant sections of poetry.lock

[[package]]
name = "aws-cdk-lib"
version = "2.161.1"
description = "Version 2 of the AWS Cloud Development Kit library"
optional = false
python-versions = "~=3.8"
files = [
    {file = "aws_cdk_lib-2.161.1-py3-none-any.whl", hash = "sha256:c7de930396b1b9f0f512a728a1b926c77c3cab28fbc11fd4f81819dd9563bfb3"},
    {file = "aws_cdk_lib-2.161.1.tar.gz", hash = "sha256:e27a427bc6d95dd2eb0500b0de628a88ee587b212f999fa6efb7e9ab17980201"},
]

[package.dependencies]
"aws-cdk.asset-awscli-v1" = ">=2.2.202,<3.0.0"
"aws-cdk.asset-kubectl-v20" = ">=2.1.2,<3.0.0"
"aws-cdk.asset-node-proxy-agent-v6" = ">=2.1.0,<3.0.0"
"aws-cdk.cloud-assembly-schema" = ">=38.0.0,<39.0.0"
constructs = ">=10.0.0,<11.0.0"
jsii = ">=1.103.1,<2.0.0"
publication = ">=0.0.3"
typeguard = ">=2.13.3,<5.0.0"

[[package]]
name = "aws-cdk-aws-lambda-python-alpha"
version = "2.161.1a0"
description = "The CDK Construct Library for AWS Lambda in Python"
optional = false
python-versions = "~=3.8"
files = [
    {file = "aws_cdk.aws_lambda_python_alpha-2.161.1a0-py3-none-any.whl", hash = "sha256:bc50b108080d06c68d0d8468467b59751082e1a7b553452a92175bf03e38d0aa"},
    {file = "aws_cdk_aws_lambda_python_alpha-2.161.1a0.tar.gz", hash = "sha256:3543cbaeabb6fb2c8e694cf5c2525ab5f1965130cc7db79a929c8c38be40db84"},
]

[package.dependencies]
aws-cdk-lib = ">=2.161.1,<3.0.0"
constructs = ">=10.0.0,<11.0.0"
jsii = ">=1.103.1,<2.0.0"
publication = ">=0.0.3"
typeguard = ">=2.13.3,<5.0.0"

If I run poetry install on my local box then run cdk deploy --context stage="dev", it succeeds. It sees my dev dependencies just fine and can run the cdk code I have that requires aws-cdk-lib.

However, I need to run this in github actions as well so that I can deploy changesets when the "dev" and "stage" and "prod" branches get pushed to. This is where I hit issues.

Here’s my github actions yaml:

name: AWS Service CI/CD

on:
  push:
    branches: [dev]
  
jobs:
  build:
    environment: dev
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python 3.12
        uses: actions/setup-python@v4
        with:
          python-version: "3.12"
      - name: Set up Node
        uses: actions/setup-node@v3
        with:
          node-version: "22" 
      - name: Service - Install Python and CDK
        run: |
          cd serverlessservice
          python -m pip install --upgrade pip         
          npm install -g aws-cdk
      - name: Service - Install Poetry
        uses: Gr1N/setup-poetry@v8
      - name: Service - Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@master
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: "us-west-2"
      - name: Service - Install Dependencies
        run: |
          cd serverlessservice  

          poetry install

          cdk deploy --context stage="dev"

The very last line fails and the output is as follows:

Traceback (most recent call last):
  File "/home/runner/work/core-data-service/core-data-service/serverlessservice/./infra.py", line 2, in <module>
    import aws_cdk as cdk
ModuleNotFoundError: No module named 'aws_cdk'
Subprocess exited with error 1

If I manually install aws-cdk-lib, the error changes to complain about the other dev dependency I have listed, which is my main evidence that it’s the dev dependencies that are the issue.

If I manually install all my dev dependencies needed by my cdk-deploy call using pip install, it DOES work but then the deploy package is way too big and gets rejected, because aws-cdk-lib is huge and should not be in production deploys.

This is the final blocker in a long line of issues with poetry and github actions, someone save me 😀

You need to sign in to view this answers

Related Post