Scan returns no results depending on how Python 3.9 is compiled/configured

[jklewa]

Describe the bug

I am running two seemingly identical virtual environments side-by-side. The same Python version and packages are installed.

One's scan works as expected, and the other's returns no results.

1. /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/bin/python3.9 -m venv env_working
2. PYENV_VERSION=3.9.6 pyenv exec python -m venv env_not_working (pyenv 2.5.0, arch -x86_64 pyenv install 3.9.6)

If Python itself is compiled or configured in such a way that it's incompatible with pyatv it would be helpful to know why and possibly detect it with an error/warning.

Error log

/Users/the347/Library/Application\ Support/ATV\ Remote/env_3.9.6__0.16.0/bin/atvremote scan --debug --mdns-debug
2024-12-29 22:35:56 DEBUG [pyatv.scripts]: Running with pyatv 0.16.0
2024-12-29 22:35:56 DEBUG [pyatv.storage.file_storage]: Loading settings from /Users/the347/.pyatv.conf
2024-12-29 22:35:56 DEBUG [pyatv.support.net]: Binding on *:5353
2024-12-29 22:35:56 DEBUG [pyatv.support.net]: Binding on 127.0.0.1:0
2024-12-29 22:35:56 DEBUG [pyatv.support.net]: Binding on 10.0.1.122:0
2024-12-29 22:35:56 DEBUG [pyatv.core.mdns]: Sending multicast DNS request to 224.0.0.251:5353 (Data=bytearray(b'5\xff\x01 \x00\x05\x00\x00\x00\x00\x00\x00\x0c_device-info\x04_tcp\x05local\x00\x00\x0c\x80\x01\x0c_sleep-proxy\x04_udp\x05local\x00\x00\x0c\x80\x01\x08_airplay\x04_tcp\x05local\x00\x00\x0c\x80\x01\x0f_companion-link\x04_tcp\x05local\x00\x00\x0c\x80\x01\x0c_sleep-proxy\x04_udp\x05local\x00\x00\x0c\x80\x01'))
2024-12-29 22:35:56 DEBUG [pyatv.core.mdns]: Sending multicast DNS request to 224.0.0.251:5353 (Data=bytearray(b'5\xff\x01 \x00\x05\x00\x00\x00\x00\x00\x00\x0f_companion-link\x04_tcp\x05local\x00\x00\x0c\x80\x01\x0b_appletv-v2\x04_tcp\x05local\x00\x00\x0c\x80\x01\x0b_touch-able\x04_tcp\x05local\x00\x00\x0c\x80\x01\x05_hscp\x04_tcp\x05local\x00\x00\x0c\x80\x01\x0c_sleep-proxy\x04_udp\x05local\x00\x00\x0c\x80\x01'))
2024-12-29 22:35:56 DEBUG [pyatv.core.mdns]: Sending multicast DNS request to 224.0.0.251:5353 (Data=bytearray(b'5\xff\x01 \x00\x05\x00\x00\x00\x00\x00\x00\x05_hscp\x04_tcp\x05local\x00\x00\x0c\x80\x01\x0e_mediaremotetv\x04_tcp\x05local\x00\x00\x0c\x80\x01\x05_raop\x04_tcp\x05local\x00\x00\x0c\x80\x01\x08_airport\x04_tcp\x05local\x00\x00\x0c\x80\x01\x0c_sleep-proxy\x04_udp\x05local\x00\x00\x0c\x80\x01'))
2024-12-29 22:35:56 DEBUG [pyatv.core.mdns]: Sending multicast DNS request to 224.0.0.251:5353 (Data=bytearray(b'5\xff\x01 \x00\x02\x00\x00\x00\x00\x00\x00\x08_airport\x04_tcp\x05local\x00\x00\x0c\x80\x01\x0c_sleep-proxy\x04_udp\x05local\x00\x00\x0c\x80\x01'))
2024-12-29 22:35:57 DEBUG [pyatv.core.mdns]: Sending multicast DNS request to 224.0.0.251:5353 (Data=bytearray(b'5\xff\x01 \x00\x05\x00\x00\x00\x00\x00\x00\x0c_device-info\x04_tcp\x05local\x00\x00\x0c\x80\x01\x0c_sleep-proxy\x04_udp\x05local\x00\x00\x0c\x80\x01\x08_airplay\x04_tcp\x05local\x00\x00\x0c\x80\x01\x0f_companion-link\x04_tcp\x05local\x00\x00\x0c\x80\x01\x0c_sleep-proxy\x04_udp\x05local\x00\x00\x0c\x80\x01'))
2024-12-29 22:35:57 DEBUG [pyatv.core.mdns]: Sending multicast DNS request to 224.0.0.251:5353 (Data=bytearray(b'5\xff\x01 \x00\x05\x00\x00\x00\x00\x00\x00\x0f_companion-link\x04_tcp\x05local\x00\x00\x0c\x80\x01\x0b_appletv-v2\x04_tcp\x05local\x00\x00\x0c\x80\x01\x0b_touch-able\x04_tcp\x05local\x00\x00\x0c\x80\x01\x05_hscp\x04_tcp\x05local\x00\x00\x0c\x80\x01\x0c_sleep-proxy\x04_udp\x05local\x00\x00\x0c\x80\x01'))
2024-12-29 22:35:57 DEBUG [pyatv.core.mdns]: Sending multicast DNS request to 224.0.0.251:5353 (Data=bytearray(b'5\xff\x01 \x00\x05\x00\x00\x00\x00\x00\x00\x05_hscp\x04_tcp\x05local\x00\x00\x0c\x80\x01\x0e_mediaremotetv\x04_tcp\x05local\x00\x00\x0c\x80\x01\x05_raop\x04_tcp\x05local\x00\x00\x0c\x80\x01\x08_airport\x04_tcp\x05local\x00\x00\x0c\x80\x01\x0c_sleep-proxy\x04_udp\x05local\x00\x00\x0c\x80\x01'))
2024-12-29 22:35:57 DEBUG [pyatv.core.mdns]: Sending multicast DNS request to 224.0.0.251:5353 (Data=bytearray(b'5\xff\x01 \x00\x02\x00\x00\x00\x00\x00\x00\x08_airport\x04_tcp\x05local\x00\x00\x0c\x80\x01\x0c_sleep-proxy\x04_udp\x05local\x00\x00\x0c\x80\x01'))
2024-12-29 22:35:58 DEBUG [pyatv.core.mdns]: Sending multicast DNS request to 224.0.0.251:5353 (Data=bytearray(b'5\xff\x01 \x00\x05\x00\x00\x00\x00\x00\x00\x0c_device-info\x04_tcp\x05local\x00\x00\x0c\x80\x01\x0c_sleep-proxy\x04_udp\x05local\x00\x00\x0c\x80\x01\x08_airplay\x04_tcp\x05local\x00\x00\x0c\x80\x01\x0f_companion-link\x04_tcp\x05local\x00\x00\x0c\x80\x01\x0c_sleep-proxy\x04_udp\x05local\x00\x00\x0c\x80\x01'))
2024-12-29 22:35:58 DEBUG [pyatv.core.mdns]: Sending multicast DNS request to 224.0.0.251:5353 (Data=bytearray(b'5\xff\x01 \x00\x05\x00\x00\x00\x00\x00\x00\x0f_companion-link\x04_tcp\x05local\x00\x00\x0c\x80\x01\x0b_appletv-v2\x04_tcp\x05local\x00\x00\x0c\x80\x01\x0b_touch-able\x04_tcp\x05local\x00\x00\x0c\x80\x01\x05_hscp\x04_tcp\x05local\x00\x00\x0c\x80\x01\x0c_sleep-proxy\x04_udp\x05local\x00\x00\x0c\x80\x01'))
2024-12-29 22:35:58 DEBUG [pyatv.core.mdns]: Sending multicast DNS request to 224.0.0.251:5353 (Data=bytearray(b'5\xff\x01 \x00\x05\x00\x00\x00\x00\x00\x00\x05_hscp\x04_tcp\x05local\x00\x00\x0c\x80\x01\x0e_mediaremotetv\x04_tcp\x05local\x00\x00\x0c\x80\x01\x05_raop\x04_tcp\x05local\x00\x00\x0c\x80\x01\x08_airport\x04_tcp\x05local\x00\x00\x0c\x80\x01\x0c_sleep-proxy\x04_udp\x05local\x00\x00\x0c\x80\x01'))
2024-12-29 22:35:58 DEBUG [pyatv.core.mdns]: Sending multicast DNS request to 224.0.0.251:5353 (Data=bytearray(b'5\xff\x01 \x00\x02\x00\x00\x00\x00\x00\x00\x08_airport\x04_tcp\x05local\x00\x00\x0c\x80\x01\x0c_sleep-proxy\x04_udp\x05local\x00\x00\x0c\x80\x01'))
Scan Results
========================================

How to reproduce the bug?

brew install pyenv
arch -x86_64 pyenv install 3.9.6
PYENV_VERSION=3.9.6 pyenv exec python -m venv env
./env/bin/pip install --upgrade pip
./env/bin/pip install websockets "pyatv==0.16.0"

./env/bin/atvremote scan --debug --mdns-debug

What is expected behavior?

The scan should return results for both virtual environments.

Operating System

macOS (arm64)

Python

3.9

pyatv

0.16.0

Device

Apple TV 4K (gen 2), tvOS 18.1

Additional context

Packages installed in both virtual environments

Package                    Version
-------------------------- ----------
aiohappyeyeballs           2.4.4
aiohttp                    3.11.11
aiosignal                  1.3.2
annotated-types            0.7.0
async-timeout              5.0.1
attrs                      24.3.0
certifi                    2024.12.14
cffi                       1.17.1
chacha20poly1305-reuseable 0.13.2
charset-normalizer         3.4.1
cryptography               44.0.0
frozenlist                 1.5.0
idna                       3.10
ifaddr                     0.2.0
miniaudio                  1.61
multidict                  6.1.0
pip                        24.3.1
propcache                  0.2.1
protobuf                   5.29.2
pyatv                      0.16.0
pycparser                  2.22
pydantic                   2.10.4
pydantic_core              2.27.2
requests                   2.32.3
setuptools                 56.0.0
six                        1.17.0
srptools                   1.0.1
tabulate                   0.9.0
tinytag                    2.0.0
typing_extensions          4.12.2
urllib3                    2.3.0
websockets                 14.1
yarl                       1.18.3
zeroconf                   0.136.2

[jklewa]

tldr; I think this is an OpenSSL issue

Though not triggered during the scan, this warning suggests an incompatibility with an old version of LibreSSL

env_not_working/lib/python3.9/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning:
urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'.
See: https://github.com/urllib3/urllib3/issues/3020

Inspecting my environment further, this is a holdover from the transition to M2, hiding in an old homebrew environment.

After clearing them (see brew doctor) and installing the brew formula openssl@3, I rebuilt python with a newer openssl.

python3 -c "import ssl; print(ssl.OPENSSL_VERSION)"
OpenSSL 3.4.0 22 Oct 2024

Now scans do return results as expected!

[jklewa]

I haven't been able to isolate this issue to OpenSSL. Recompiling Python was the solution, but maybe it was an old dependency that I removed per the recommendation of brew doctor... it had me brew remove gettext openssl readline tcl-tk.

python3 -m sysconfig from the broken environment, showed references to these library paths:

/usr/local/opt/tcl-tk/lib
/usr/local/opt/ncurses/include
/usr/local/opt/readline/include
/usr/local/opt/openssl/include
/usr/local/opt/openssl@1.1/include