Lambda function Containers initialization fails

[matus-hustak]

Description:

sam local start-api fails because When it tryes to initialize Lambda functions containers with 500 error

Steps to reproduce:

1. Enabled WSL and Virtual Machine Platform
2. Restart PC
3. Install Python 3.11, Rancher Desktop 1.17, Ubuntu, AWS CLI, AWS SAM on Windows
4. Initiate SAM project on Windows
5. Install AWS SAM & AWS CLI on Ubuntu
6. Run 'sam local start-api -- debug' from Ubuntu console while in Windows directory where SAM Project was initiated
7. It should work
8. Kill it and run again 'sam local start-api --debug'

Observed result:

john@US-5CG04747JR:/mnt/c/dr-hq/sam/drhq-sam$ sam local start-api --debug
2025-01-15 10:09:00,407 | Config file location: /mnt/c/dr-hq/sam/drhq-sam/samconfig.toml
2025-01-15 10:09:00,416 | Loading configuration values from [default.['local', 'start-api'].parameters] (env.command_name.section) in config file at '/mnt/c/dr-hq/sam/drhq-sam/samconfig.toml'...
2025-01-15 10:09:00,423 | Configuration values successfully loaded.
2025-01-15 10:09:00,429 | Configuration values are: {'stack_name': 'drhq-sam', 'warm_containers': 'EAGER'}
2025-01-15 10:09:00,523 | Using SAM Template at /mnt/c/dr-hq/sam/drhq-sam/.aws-sam/build/template.yaml
2025-01-15 10:09:00,586 | Using config file: samconfig.toml, config environment: default
2025-01-15 10:09:00,588 | Expand command line arguments to:
2025-01-15 10:09:00,590 | --template_file=/mnt/c/dr-hq/sam/drhq-sam/.aws-sam/build/template.yaml --host=127.0.0.1 --port=3000 --static_dir=public --layer_cache_basedir=/home/johnl/.aws-sam/layers-pkg --warm_containers=EAGER
--container_host=localhost --container_host_interface=127.0.0.1
2025-01-15 10:09:00,818 | local start-api command is called
2025-01-15 10:09:01,600 | No Parameters detected in the template
2025-01-15 10:09:01,662 | There is no customer defined id or cdk path defined for resource MyApi, so we will use the resource logical id as the resource id
2025-01-15 10:09:01,664 | Sam customer defined id is more priority than other IDs. Customer defined id for resource HelloWorldFunction is HelloWorldFunction
2025-01-15 10:09:01,666 | Sam customer defined id is more priority than other IDs. Customer defined id for resource GoodbyeWorldFunction is GoodbyeWorldFunction
2025-01-15 10:09:01,668 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use the resource logical id as the resource id
2025-01-15 10:09:01,677 | Unable to resolve property Value: OrderedDict([('Fn::GetAtt', 'HelloWorldFunction.Arn')]). Leaving as is.
2025-01-15 10:09:01,679 | Unable to resolve property Value: OrderedDict([('Fn::GetAtt', 'GoodbyeWorldFunction.Arn')]). Leaving as is.
2025-01-15 10:09:01,682 | Unable to resolve property Value: OrderedDict([('Fn::GetAtt', 'HelloWorldFunctionRole.Arn')]). Leaving as is.
2025-01-15 10:09:01,688 | 0 stacks found in the template
2025-01-15 10:09:01,695 | No Parameters detected in the template
2025-01-15 10:09:01,755 | There is no customer defined id or cdk path defined for resource MyApi, so we will use the resource logical id as the resource id
2025-01-15 10:09:01,757 | Sam customer defined id is more priority than other IDs. Customer defined id for resource HelloWorldFunction is HelloWorldFunction
2025-01-15 10:09:01,759 | Sam customer defined id is more priority than other IDs. Customer defined id for resource GoodbyeWorldFunction is GoodbyeWorldFunction
2025-01-15 10:09:01,762 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use the resource logical id as the resource id
2025-01-15 10:09:01,768 | Unable to resolve property Value: OrderedDict([('Fn::GetAtt', 'HelloWorldFunction.Arn')]). Leaving as is.
2025-01-15 10:09:01,770 | Unable to resolve property Value: OrderedDict([('Fn::GetAtt', 'GoodbyeWorldFunction.Arn')]). Leaving as is.
2025-01-15 10:09:01,774 | Unable to resolve property Value: OrderedDict([('Fn::GetAtt', 'HelloWorldFunctionRole.Arn')]). Leaving as is.
2025-01-15 10:09:01,778 | 4 resources found in the stack
2025-01-15 10:09:01,788 | Found Serverless function with name='HelloWorldFunction' and CodeUri='HelloWorldFunction'
2025-01-15 10:09:01,791 | --base-dir is not presented, adjusting uri HelloWorldFunction relative to /mnt/c/dr-hq/sam/drhq-sam/.aws-sam/build/template.yaml
2025-01-15 10:09:01,802 | Found Serverless function with name='GoodbyeWorldFunction' and CodeUri='GoodbyeWorldFunction'
2025-01-15 10:09:01,805 | --base-dir is not presented, adjusting uri GoodbyeWorldFunction relative to /mnt/c/dr-hq/sam/drhq-sam/.aws-sam/build/template.yaml
2025-01-15 10:09:01,853 | watch resource /mnt/c/dr-hq/sam/drhq-sam/.aws-sam/build/template.yaml
2025-01-15 10:09:01,856 | Create Observer for resource /mnt/c/dr-hq/sam/drhq-sam/.aws-sam/build/template.yaml with recursive True
2025-01-15 10:09:01,873 | watch resource /mnt/c/dr-hq/sam/drhq-sam/.aws-sam/build/template.yaml's parent /mnt/c/dr-hq/sam/drhq-sam/.aws-sam/build
2025-01-15 10:09:01,878 | Create Observer for resource /mnt/c/dr-hq/sam/drhq-sam/.aws-sam/build with recursive False
2025-01-15 10:09:01,902 | Initializing the lambda functions containers.
2025-01-15 10:09:01,905 | Async execution started
2025-01-15 10:09:01,907 | Invoking function functools.partial(<function InvokeContext._initialize_all_functions_containers..initialize_function_container at 0x7fe050eac720>, Function(function_id='HelloWorldFunction',
name='HelloWorldFunction', functionname='HelloWorldFunction', runtime='python3.11', memory=None, timeout=3, handler='app.lambda_handler', imageuri=None, packagetype='Zip', imageconfig=None,
codeuri='/mnt/c/dr-hq/sam/drhq-sam/.aws-sam/build/HelloWorldFunction', environment=None, rolearn=None, layers=[], events={'HelloWorld': {'Type': 'Api', 'Properties': {'Path': '/hello', 'Method': 'get', 'RestApiId':
'ServerlessRestApi'}}}, metadata={'SamResourceId': 'HelloWorldFunction'}, inlinecode=None, codesign_config_arn=None, architectures=['x86_64'], function_url_config=None, function_build_info=<FunctionBuildInfo.BuildableZip:
('BuildableZip', 'Regular ZIP function which can be build with SAM CLI')>, stack_path='', runtime_management_config=None, logging_config=None))
2025-01-15 10:09:01,922 | Invoking function functools.partial(<function InvokeContext._initialize_all_functions_containers..initialize_function_container at 0x7fe050eac720>, Function(function_id='GoodbyeWorldFunction',
name='GoodbyeWorldFunction', functionname='GoodbyeWorldFunction', runtime='python3.11', memory=None, timeout=3, handler='app.lambda_handler', imageuri=None, packagetype='Zip', imageconfig=None,
codeuri='/mnt/c/dr-hq/sam/drhq-sam/.aws-sam/build/GoodbyeWorldFunction', environment=None, rolearn=None, layers=[], events={'GoodbyeWorld': {'Type': 'Api', 'Properties': {'Path': '/goodbye', 'Method': 'get', 'RestApiId':
'ServerlessRestApi'}}}, metadata={'SamResourceId': 'GoodbyeWorldFunction'}, inlinecode=None, codesign_config_arn=None, architectures=['x86_64'], function_url_config=None, function_build_info=<FunctionBuildInfo.BuildableZip:
('BuildableZip', 'Regular ZIP function which can be build with SAM CLI')>, stack_path='', runtime_management_config=None, logging_config=None))
2025-01-15 10:09:01,936 | Waiting for async results
2025-01-15 10:09:01,947 | No environment variables found for function 'HelloWorldFunction'
2025-01-15 10:09:01,955 | Loading AWS credentials from session with profile 'None'
2025-01-15 10:09:01,973 | No environment variables found for function 'GoodbyeWorldFunction'
2025-01-15 10:09:01,983 | Loading AWS credentials from session with profile 'None'
2025-01-15 10:09:02,024 | Resolving code path. Cwd=/mnt/c/dr-hq/sam/drhq-sam/.aws-sam/build, CodeUri=/mnt/c/dr-hq/sam/drhq-sam/.aws-sam/build/HelloWorldFunction
2025-01-15 10:09:02,027 | Resolved absolute path to code is /mnt/c/dr-hq/sam/drhq-sam/.aws-sam/build/HelloWorldFunction
2025-01-15 10:09:02,029 | Resolving code path. Cwd=/mnt/c/dr-hq/sam/drhq-sam/.aws-sam/build, CodeUri=/mnt/c/dr-hq/sam/drhq-sam/.aws-sam/build/HelloWorldFunction
2025-01-15 10:09:02,032 | Resolving code path. Cwd=/mnt/c/dr-hq/sam/drhq-sam/.aws-sam/build, CodeUri=/mnt/c/dr-hq/sam/drhq-sam/.aws-sam/build/GoodbyeWorldFunction
2025-01-15 10:09:02,033 | Resolved real code path to /mnt/c/dr-hq/sam/drhq-sam/.aws-sam/build/HelloWorldFunction
2025-01-15 10:09:02,035 | Resolved absolute path to code is /mnt/c/dr-hq/sam/drhq-sam/.aws-sam/build/GoodbyeWorldFunction
2025-01-15 10:09:02,040 | Resolving code path. Cwd=/mnt/c/dr-hq/sam/drhq-sam/.aws-sam/build, CodeUri=/mnt/c/dr-hq/sam/drhq-sam/.aws-sam/build/GoodbyeWorldFunction
2025-01-15 10:09:02,043 | Resolved real code path to /mnt/c/dr-hq/sam/drhq-sam/.aws-sam/build/GoodbyeWorldFunction
2025-01-15 10:09:06,289 | watch resource /mnt/c/dr-hq/sam/drhq-sam/.aws-sam/build/HelloWorldFunction
2025-01-15 10:09:06,291 | Create Observer for resource /mnt/c/dr-hq/sam/drhq-sam/.aws-sam/build/HelloWorldFunction with recursive True
2025-01-15 10:09:06,748 | watch resource /mnt/c/dr-hq/sam/drhq-sam/.aws-sam/build/HelloWorldFunction's parent /mnt/c/dr-hq/sam/drhq-sam/.aws-sam/build
2025-01-15 10:09:06,753 | Code /mnt/c/dr-hq/sam/drhq-sam/.aws-sam/build/HelloWorldFunction is not a zip/jar file
2025-01-15 10:09:10,168 | watch resource /mnt/c/dr-hq/sam/drhq-sam/.aws-sam/build/GoodbyeWorldFunction
2025-01-15 10:09:10,170 | Create Observer for resource /mnt/c/dr-hq/sam/drhq-sam/.aws-sam/build/GoodbyeWorldFunction with recursive True
2025-01-15 10:09:10,626 | watch resource /mnt/c/dr-hq/sam/drhq-sam/.aws-sam/build/GoodbyeWorldFunction's parent /mnt/c/dr-hq/sam/drhq-sam/.aws-sam/build
2025-01-15 10:09:10,631 | Code /mnt/c/dr-hq/sam/drhq-sam/.aws-sam/build/GoodbyeWorldFunction is not a zip/jar file
2025-01-15 10:09:12,608 | Local image is up-to-date
2025-01-15 10:09:12,624 | Checking free port on 127.0.0.1:5823
2025-01-15 10:09:12,633 | Using local image: public.ecr.aws/lambda/python:3.11-rapid-x86_64.

2025-01-15 10:09:12,635 | Mounting /mnt/c/dr-hq/sam/drhq-sam/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated, inside runtime container
2025-01-15 10:09:17,042 | Exception raised during the execution
2025-01-15 10:09:17,048 | Lambda functions containers initialization failed because of 500 Server Error for http+docker://localhost/v1.35/distribution/public.ecr.aws/lambda/python:3.11-x86_64/json: Internal Server Error ("Head
"https://public.ecr.aws/v2/lambda/python/manifests/3.11-x86_64": Get "https://public.ecr.aws/token/?scope=repository%3Alambda%2Fpython%3Apull&service=public.ecr.aws": EOF")
2025-01-15 10:09:17,055 | Terminating all running warm containers
2025-01-15 10:09:17,057 | Terminate running warm container for Lambda Function 'HelloWorldFunction'
2025-01-15 10:09:17,756 | Cleaning all decompressed code dirs
2025-01-15 10:09:17,954 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2025-01-15 10:09:17,984 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2025-01-15 10:09:17,986 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': 'cdb572c0-b1fb-484c-acea-8be7addc5eb8', 'installationId': '29cdb574-f509-4178-930f-f3e0d4af9b41', 'sessionId': '4f8b2535-b5f0-4e2b-aa63-5aaf62f032aa',
'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.11.10', 'samcliVersion': '1.132.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam local start-api', 'metricSpecificAttributes':
{'projectType': 'CFN', 'gitOrigin': None, 'projectName': 'b2dd070ae3624503be4ceee3ebb86af7ce0d15873d95d5c8e47aaccf903866a8', 'initialCommit': None}, 'duration': 17368, 'exitReason': 'ContainersInitializationException', 'exitCode': 1}}]}
2025-01-15 10:09:17,987 | Unable to find Click Context for getting session_id.
2025-01-15 10:09:18,001 | Sending Telemetry: {'metrics': [{'events': {'requestId': '06e893b6-84aa-4e95-bbfd-0ceb78d0a55d', 'installationId': '29cdb574-f509-4178-930f-f3e0d4af9b41', 'sessionId': '4f8b2535-b5f0-4e2b-aa63-5aaf62f032aa',
'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.11.10', 'samcliVersion': '1.132.0', 'commandName': 'sam local start-api', 'metricSpecificAttributes': {'events': [{'event_name': 'SamConfigFileExtension', 'event_value':
'.toml', 'thread_id': '3e085274ab0649a49a17cbe20c570691', 'time_stamp': '2025-01-15 09:09:00.406', 'exception_name': None}, {'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': '5d106b045d1d4ff08271f77226847ddb',
'time_stamp': '2025-01-15 09:09:00.586', 'exception_name': None}]}}}]}
2025-01-15 10:09:18,742 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Error: Lambda functions containers initialization failed
2025-01-15 10:09:18,883 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)

Expected result:

I expect for Lambda functions containers to be initialized.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Windows 10 Enterprise 22H2 19045.5247, Ubuntu 24.04.1 LTS
2. sam --version: SAM CLI, version 1.131.0
3. AWS region:

{
  "version": "1.131.0",
  "system": {
    "python": "3.12.6",
    "os": "Windows-10-10.0.19045-SP0"
  },
  "additional_dependencies": {
    "docker_engine": "26.1.5",
    "aws_cdk": "Not available",
    "terraform": "Not available"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}

Add --debug flag to command you are running

[roger-zhangg]

Thanks for reporting, For more context, is the docker running in WSL as well? Could you use it to pull other images?
Another Rancher desktop related issue:
#7807

[matus-hustak]

Hello, The docker is not running in WSL. It is Running on Windows. I shall try pulling pulling .NET images and paste here what I've found out.

[matus-hustak]

Yes... this issue is the same with .NET:

2025-01-24 12:19:34,737 | Exception raised during the execution
2025-01-24 12:19:34,739 | Lambda functions containers initialization failed because of 500 Server Error for
http+docker://localhost/v1.35/distribution/public.ecr.aws/lambda/dotnet:8-x86_64/json: Internal Server Error ("Head
"https://public.ecr.aws/v2/lambda/dotnet/manifests/8-x86_64": Get
"https://public.ecr.aws/token/?scope=repository%3Alambda%2Fdotnet%3Apull&service=public.ecr.aws": net/http: TLS handshake timeout")
2025-01-24 12:19:34,742 | Terminating all running warm containers

[roger-zhangg]

this is strange, could you try directly pulling this image?
docker pull public.ecr.aws/lambda/dotnet:8-x86_64

if this also doesn't work, can you try follow this doc to login for ECR?
https://docs.aws.amazon.com/AmazonECR/latest/userguide/registry_auth.html

Thank you