Cannot run ansible-runner with EEs

[Fryguy]

I'm hitting a problem trying to run ansible-runner with a very simple EE / playbook. I'm using ansible-runner 2.4.0 on a MacOS M1 with podman-desktop. I also had @agrare try on Linux with ansible-runner 2.4.0, and he got the same issue

Running

$ ansible-runner run --process-isolation --container-image ghcr.io/ansible-community/community-ee-base:latest --playbook test_localhost.yml . --debug -vvvvv

where test_localhost.yml is

- name: Gather and print local facts
  hosts: localhost
  become: true
  gather_facts: true
  tasks:
  - name: Print facts
    ansible.builtin.debug:
      var: ansible_facts

I get the following

...
command: podman run --rm --tty --interactive --workdir /runner/project -v /Users/jfrey/dev/test/example/dir/:/runner/:Z --env-file /Users/jfrey/dev/test/example/dir/artifacts/ffcba5bb-48d5-4855-925c-1c615f44aa6f/env.list --quiet --name ansible_runner_ffcba5bb-48d5-4855-925c-1c615f44aa6f ghcr.io/ansible-community/community-ee-base:latest ansible-playbook -i /runner/inventory -vvvvv test_localhost.yml
Failed to open ansible stdout callback plugin partial data file /Users/jfrey/dev/test/example/dir/artifacts/ffcba5bb-48d5-4855-925c-1c615f44aa6f/job_events/6548d5ec-636e-47df-bb8a-d640d286751d-partial.json with error [Errno 2] No such file or directory: '/Users/jfrey/dev/test/example/dir/artifacts/ffcba5bb-48d5-4855-925c-1c615f44aa6f/job_events/6548d5ec-636e-47df-bb8a-d640d286751d-partial.json'
ansible-playbook [core 2.18.1]
Failed to open ansible stdout callback plugin partial data file /Users/jfrey/dev/test/example/dir/artifacts/ffcba5bb-48d5-4855-925c-1c615f44aa6f/job_events/eee01b8e-e377-4d23-a86e-2f0dd4802056-partial.json with error [Errno 2] No such file or directory: '/Users/jfrey/dev/test/example/dir/artifacts/ffcba5bb-48d5-4855-925c-1c615f44aa6f/job_events/eee01b8e-e377-4d23-a86e-2f0dd4802056-partial.json'
  config file = None
Failed to open ansible stdout callback plugin partial data file /Users/jfrey/dev/test/example/dir/artifacts/ffcba5bb-48d5-4855-925c-1c615f44aa6f/job_events/8e36548a-123c-417f-98af-24bdf2348740-partial.json with error [Errno 2] No such file or directory: '/Users/jfrey/dev/test/example/dir/artifacts/ffcba5bb-48d5-4855-925c-1c615f44aa6f/job_events/8e36548a-123c-417f-98af-24bdf2348740-partial.json'
  configured module search path = ['/runner/project/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
Failed to open ansible stdout callback plugin partial data file /Users/jfrey/dev/test/example/dir/artifacts/ffcba5bb-48d5-4855-925c-1c615f44aa6f/job_events/c6909887-c007-4043-9dff-bfd09f1d5b0d-partial.json with error [Errno 2] No such file or directory: '/Users/jfrey/dev/test/example/dir/artifacts/ffcba5bb-48d5-4855-925c-1c615f44aa6f/job_events/c6909887-c007-4043-9dff-bfd09f1d5b0d-partial.json'
  ansible python module location = /usr/local/lib/python3.13/site-packages/ansible
Failed to open ansible stdout callback plugin partial data file /Users/jfrey/dev/test/example/dir/artifacts/ffcba5bb-48d5-4855-925c-1c615f44aa6f/job_events/a15f4410-5cf8-4e17-8272-efbded1cacec-partial.json with error [Errno 2] No such file or directory: '/Users/jfrey/dev/test/example/dir/artifacts/ffcba5bb-48d5-4855-925c-1c615f44aa6f/job_events/a15f4410-5cf8-4e17-8272-efbded1cacec-partial.json'
  ansible collection location = /runner/project/.ansible/collections:/usr/share/ansible/collections
Failed to open ansible stdout callback plugin partial data file /Users/jfrey/dev/test/example/dir/artifacts/ffcba5bb-48d5-4855-925c-1c615f44aa6f/job_events/96c25552-f425-4628-8f7f-d821b38b48f6-partial.json with error [Errno 2] No such file or directory: '/Users/jfrey/dev/test/example/dir/artifacts/ffcba5bb-48d5-4855-925c-1c615f44aa6f/job_events/96c25552-f425-4628-8f7f-d821b38b48f6-partial.json'
  executable location = /usr/local/bin/ansible-playbook
Failed to open ansible stdout callback plugin partial data file /Users/jfrey/dev/test/example/dir/artifacts/ffcba5bb-48d5-4855-925c-1c615f44aa6f/job_events/71a36e97-ab75-424f-aa1c-2df7d12d86bd-partial.json with error [Errno 2] No such file or directory: '/Users/jfrey/dev/test/example/dir/artifacts/ffcba5bb-48d5-4855-925c-1c615f44aa6f/job_events/71a36e97-ab75-424f-aa1c-2df7d12d86bd-partial.json'
  python version = 3.13.0 (main, Oct  8 2024, 00:00:00) [GCC 14.2.1 20240912 (Red Hat 14.2.1-3)] (/usr/bin/python3)
Failed to open ansible stdout callback plugin partial data file /Users/jfrey/dev/test/example/dir/artifacts/ffcba5bb-48d5-4855-925c-1c615f44aa6f/job_events/9880d216-02ba-459d-9d0f-944332e8c59d-partial.json with error [Errno 2] No such file or directory: '/Users/jfrey/dev/test/example/dir/artifacts/ffcba5bb-48d5-4855-925c-1c615f44aa6f/job_events/9880d216-02ba-459d-9d0f-944332e8c59d-partial.json'
  jinja version = 3.1.4
Failed to open ansible stdout callback plugin partial data file /Users/jfrey/dev/test/example/dir/artifacts/ffcba5bb-48d5-4855-925c-1c615f44aa6f/job_events/adb7c0bd-5851-48d0-800f-cb5ee78ac106-partial.json with error [Errno 2] No such file or directory: '/Users/jfrey/dev/test/example/dir/artifacts/ffcba5bb-48d5-4855-925c-1c615f44aa6f/job_events/adb7c0bd-5851-48d0-800f-cb5ee78ac106-partial.json'
  libyaml = True
Failed to open ansible stdout callback plugin partial data file /Users/jfrey/dev/test/example/dir/artifacts/ffcba5bb-48d5-4855-925c-1c615f44aa6f/job_events/682f4480-742c-473a-97cd-353ee418db7b-partial.json with error [Errno 2] No such file or directory: '/Users/jfrey/dev/test/example/dir/artifacts/ffcba5bb-48d5-4855-925c-1c615f44aa6f/job_events/682f4480-742c-473a-97cd-353ee418db7b-partial.json'
No config file found; using defaults
Failed to open ansible stdout callback plugin partial data file /Users/jfrey/dev/test/example/dir/artifacts/ffcba5bb-48d5-4855-925c-1c615f44aa6f/job_events/f8740481-e803-4f08-a786-5c02c5f293bf-partial.json with error [Errno 2] No such file or directory: '/Users/jfrey/dev/test/example/dir/artifacts/ffcba5bb-48d5-4855-925c-1c615f44aa6f/job_events/f8740481-e803-4f08-a786-5c02c5f293bf-partial.json'
ERROR! the playbook: test_localhost.yml could not be found

I noticed that my job_events directory that was generated has files with a numeric prefix, and the code is not looking for that. For example, I see

dir/artifacts/ffcba5bb-48d5-4855-925c-1c615f44aa6f/job_events/1-6548d5ec-636e-47df-bb8a-d640d286751d.json

but the error says it's looking for

dir/artifacts/ffcba5bb-48d5-4855-925c-1c615f44aa6f/job_events/6548d5ec-636e-47df-bb8a-d640d286751d-partial.json

[Fryguy]

I had a colleague try on Linux and they reproduced the error as well, so it's not Mac specific - I'll update the OP.

[sivel]

You may need to also execute runner with --container-option="--user=root"

Depending on the container runtime, executing user, and the configuration of the EE, running as the default user may produce an unusable container.

[agrare]

Hey @sivel thanks for the tip, I tried adding that to the command from the description and see the same failure as before:

ansible-runner run --process-isolation --container-image ghcr.io/ansible-community/community-ee-base:latest --container-option="--user=root" --playbook test_localhost.yml . --debug -vvvvv
starting debug logging
file path is /home/grare/adam/env/settings
cache miss, attempting to load file from disk: /home/grare/adam/env/settings
specified path does not exist /home/grare/adam/env/settings
Not loading settings
file path is /home/grare/adam/env/passwords
cache miss, attempting to load file from disk: /home/grare/adam/env/passwords
specified path does not exist /home/grare/adam/env/passwords
Not loading passwords
file path is /home/grare/adam/env/envvars
cache miss, attempting to load file from disk: /home/grare/adam/env/envvars
specified path does not exist /home/grare/adam/env/envvars
Not loading environment vars
file path is /home/grare/adam/env/ssh_key
cache miss, attempting to load file from disk: /home/grare/adam/env/ssh_key
specified path does not exist /home/grare/adam/env/ssh_key
Not loading ssh key
env:
 ANSIBLE_CACHE_PLUGIN: jsonfile
 ANSIBLE_CACHE_PLUGIN_CONNECTION: /runner/artifacts/073838dd-3e35-41ea-80cf-ec4c260ef3e0/fact_cache
 ANSIBLE_CALLBACK_PLUGINS: /runner/artifacts/073838dd-3e35-41ea-80cf-ec4c260ef3e0/callback
 ANSIBLE_HOST_KEY_CHECKING: False
 ANSIBLE_RETRY_FILES_ENABLED: False
 ANSIBLE_STDOUT_CALLBACK: awx_display
 ANSIBLE_UNSAFE_WRITES: 1
 AWX_ISOLATED_DATA_DIR: /runner/artifacts/073838dd-3e35-41ea-80cf-ec4c260ef3e0
file path is /home/grare/adam/args
cache miss, attempting to load file from disk: /home/grare/adam/args
specified path does not exist /home/grare/adam/args
file path is /home/grare/adam/env/cmdline
cache miss, attempting to load file from disk: /home/grare/adam/env/cmdline
specified path does not exist /home/grare/adam/env/cmdline
sandbox disabled
containerization enabled
env:
 ANSIBLE_CACHE_PLUGIN: jsonfile
 ANSIBLE_CACHE_PLUGIN_CONNECTION: /runner/artifacts/073838dd-3e35-41ea-80cf-ec4c260ef3e0/fact_cache
 ANSIBLE_CALLBACK_PLUGINS: /runner/artifacts/073838dd-3e35-41ea-80cf-ec4c260ef3e0/callback
 ANSIBLE_HOST_KEY_CHECKING: False
 ANSIBLE_RETRY_FILES_ENABLED: False
 ANSIBLE_STDOUT_CALLBACK: awx_display
 ANSIBLE_UNSAFE_WRITES: 1
 AWX_ISOLATED_DATA_DIR: /runner/artifacts/073838dd-3e35-41ea-80cf-ec4c260ef3e0
 RUNNER_OMIT_EVENTS: False
 RUNNER_ONLY_FAILED_EVENTS: False
command: podman run --rm --tty --interactive --workdir /runner/project -v /home/grare/adam/:/runner/:Z --env-file /home/grare/adam/artifacts/073838dd-3e35-41ea-80cf-ec4c260ef3e0/env.list --quiet --name ansible_runner_073838dd-3e35-41ea-80cf-ec4c260ef3e0 --user=root ghcr.io/ansible-community/community-ee-base:latest ansible-playbook -i /runner/inventory -vvvvv test_localhost.yml
Failed to open ansible stdout callback plugin partial data file /home/grare/adam/artifacts/073838dd-3e35-41ea-80cf-ec4c260ef3e0/job_events/60f6ff48-25ea-4fa3-94bd-11795a4c35f9-partial.json with error [Errno 2] No such file or directory: '/home/grare/adam/artifacts/073838dd-3e35-41ea-80cf-ec4c260ef3e0/job_events/60f6ff48-25ea-4fa3-94bd-11795a4c35f9-partial.json'
ansible-playbook [core 2.18.1]
Failed to open ansible stdout callback plugin partial data file /home/grare/adam/artifacts/073838dd-3e35-41ea-80cf-ec4c260ef3e0/job_events/a2941c8b-f368-49a0-8978-ab97cd4ae29f-partial.json with error [Errno 2] No such file or directory: '/home/grare/adam/artifacts/073838dd-3e35-41ea-80cf-ec4c260ef3e0/job_events/a2941c8b-f368-49a0-8978-ab97cd4ae29f-partial.json'
  config file = None
Failed to open ansible stdout callback plugin partial data file /home/grare/adam/artifacts/073838dd-3e35-41ea-80cf-ec4c260ef3e0/job_events/345eef8d-c83a-4f43-b2e7-812fe2c4ddce-partial.json with error [Errno 2] No such file or directory: '/home/grare/adam/artifacts/073838dd-3e35-41ea-80cf-ec4c260ef3e0/job_events/345eef8d-c83a-4f43-b2e7-812fe2c4ddce-partial.json'
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
Failed to open ansible stdout callback plugin partial data file /home/grare/adam/artifacts/073838dd-3e35-41ea-80cf-ec4c260ef3e0/job_events/9c3aa463-2789-4c3d-82be-5d0ea3636bad-partial.json with error [Errno 2] No such file or directory: '/home/grare/adam/artifacts/073838dd-3e35-41ea-80cf-ec4c260ef3e0/job_events/9c3aa463-2789-4c3d-82be-5d0ea3636bad-partial.json'
  ansible python module location = /usr/local/lib/python3.13/site-packages/ansible
Failed to open ansible stdout callback plugin partial data file /home/grare/adam/artifacts/073838dd-3e35-41ea-80cf-ec4c260ef3e0/job_events/87263bc0-73eb-4231-8934-b9f935c1943a-partial.json with error [Errno 2] No such file or directory: '/home/grare/adam/artifacts/073838dd-3e35-41ea-80cf-ec4c260ef3e0/job_events/87263bc0-73eb-4231-8934-b9f935c1943a-partial.json'
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
Failed to open ansible stdout callback plugin partial data file /home/grare/adam/artifacts/073838dd-3e35-41ea-80cf-ec4c260ef3e0/job_events/69868b1b-bcc2-4713-94a0-b47b54cd6cb9-partial.json with error [Errno 2] No such file or directory: '/home/grare/adam/artifacts/073838dd-3e35-41ea-80cf-ec4c260ef3e0/job_events/69868b1b-bcc2-4713-94a0-b47b54cd6cb9-partial.json'
  executable location = /usr/local/bin/ansible-playbook
Failed to open ansible stdout callback plugin partial data file /home/grare/adam/artifacts/073838dd-3e35-41ea-80cf-ec4c260ef3e0/job_events/d4a6ac9d-93ce-4c39-953a-4747e697e0a8-partial.json with error [Errno 2] No such file or directory: '/home/grare/adam/artifacts/073838dd-3e35-41ea-80cf-ec4c260ef3e0/job_events/d4a6ac9d-93ce-4c39-953a-4747e697e0a8-partial.json'
  python version = 3.13.0 (main, Oct  8 2024, 00:00:00) [GCC 14.2.1 20240912 (Red Hat 14.2.1-3)] (/usr/bin/python3)
Failed to open ansible stdout callback plugin partial data file /home/grare/adam/artifacts/073838dd-3e35-41ea-80cf-ec4c260ef3e0/job_events/1fcf84be-bfcc-4a24-b35f-6336979d8d5d-partial.json with error [Errno 2] No such file or directory: '/home/grare/adam/artifacts/073838dd-3e35-41ea-80cf-ec4c260ef3e0/job_events/1fcf84be-bfcc-4a24-b35f-6336979d8d5d-partial.json'
  jinja version = 3.1.4
Failed to open ansible stdout callback plugin partial data file /home/grare/adam/artifacts/073838dd-3e35-41ea-80cf-ec4c260ef3e0/job_events/32d8e281-c406-4b02-8838-7991e10918cf-partial.json with error [Errno 2] No such file or directory: '/home/grare/adam/artifacts/073838dd-3e35-41ea-80cf-ec4c260ef3e0/job_events/32d8e281-c406-4b02-8838-7991e10918cf-partial.json'
  libyaml = True
Failed to open ansible stdout callback plugin partial data file /home/grare/adam/artifacts/073838dd-3e35-41ea-80cf-ec4c260ef3e0/job_events/ce76ed36-f396-46e7-99a2-359084435c64-partial.json with error [Errno 2] No such file or directory: '/home/grare/adam/artifacts/073838dd-3e35-41ea-80cf-ec4c260ef3e0/job_events/ce76ed36-f396-46e7-99a2-359084435c64-partial.json'
No config file found; using defaults
Failed to open ansible stdout callback plugin partial data file /home/grare/adam/artifacts/073838dd-3e35-41ea-80cf-ec4c260ef3e0/job_events/9667278d-dd15-4f66-838f-0d791c2d3f78-partial.json with error [Errno 2] No such file or directory: '/home/grare/adam/artifacts/073838dd-3e35-41ea-80cf-ec4c260ef3e0/job_events/9667278d-dd15-4f66-838f-0d791c2d3f78-partial.json'
ERROR! the playbook: test_localhost.yml could not be found

[sivel]

When running containerized, the (cwd) dir structure should look like:

.
├── artifacts
└── project
    └── playbook.yml

So you need a project dir, and then to nest your playbook in the project directory.

[sivel]

fwiw, the directory layout is documented at https://ansible.readthedocs.io/projects/runner/en/latest/intro/#inputdir

[agrare]

Thanks that got past the ERROR! the playbook: test_localhost.yml could not be found I do still see the ENOENT errors Failed to open ansible stdout callback plugin partial data file /home/grare/adam/test/artifacts/bf901602-b740-496a-9e70-ca4be81307a5/job_events/337faf61-3967-4a80-8eaa-98ce21132051-partial.json with error [Errno 2] No such file or directory: '/home/grare/adam/test/artifacts/bf901602-b740-496a-9e70-ca4be81307a5/job_events/337faf61-3967-4a80-8eaa-98ce21132051-partial.json' where the right files have an integer prefix:
artifacts/bf901602-b740-496a-9e70-ca4be81307a5/job_events/27-337faf61-3967-4a80-8eaa-98ce21132051.json vs artifacts/bf901602-b740-496a-9e70-ca4be81307a5/job_events/337faf61-3967-4a80-8eaa-98ce21132051-partial.json

[Fryguy]

Ah, the project worked for me as well! Incidentally, I was trying with --project-dir /runner as well, but that didn't work surprisingly. Thank you!

[Fryguy]

I see similar as @agrare sees...it succeeds, but if I pass --debug I'm seeing warnings like

Failed to open ansible stdout callback plugin partial data file /Users/jfrey/dev/test/ansible-ee/dir/artifacts/9364012f-5561-4692-bb69-8cbe4ace1ff0/job_events/31e0980b-2b78-408e-9a02-1a2bae1730c0-partial.json with error [Errno 2] No such file or directory: '/Users/jfrey/dev/test/ansible-ee/dir/artifacts/9364012f-5561-4692-bb69-8cbe4ace1ff0/job_events/31e0980b-2b78-408e-9a02-1a2bae1730c0-partial.json'

[Shrews]

The partial data file messages are harmless. Those happen while attempting to combine partially received ansible event data that may or may not already be recorded on disk.

[Shrews]

Closing this as it seems the initial problem was resolved by using the correct project structure. Thanks @sivel.