Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bluetooth: Audio/ISO: Remove CAP and GMAP AC tests from CI #83736

Open
Thalley opened this issue Jan 9, 2025 · 1 comment
Open

Bluetooth: Audio/ISO: Remove CAP and GMAP AC tests from CI #83736

Thalley opened this issue Jan 9, 2025 · 1 comment
Assignees
Labels
area: Bluetooth Audio area: Bluetooth Controller area: Bluetooth ISO Bluetooth LE Isochronous Channels area: Bluetooth RFC Request For Comments: want input from the community

Comments

@Thalley
Copy link
Collaborator

Thalley commented Jan 9, 2025

Introduction

For LE Audio we have a set of tests in https://github.com/zephyrproject-rtos/zephyr/tree/main/tests/bsim/bluetooth/audio/test_scripts called e.g. cap_unicast_ac_1.sh or gmap_broadcast_ac_12.sh which tests various codec configurations. From an application/host perspective, these are pretty simple to run and pass as it's effectively just metadata. The Bluetooth controller, however, will need to be able to support scheduling these different configurations without error, which is far from trivial.

Problem description

Historically the Zephyr controller has had trouble running all these tests, and there's actually never been a day where all of them has passed, and small timing from unrelated changes may prevent the tests from passing. This either blocks host development (sometimes for months), or PRs need to disable the failing tests.

Proposed change

Propose to not run the tests in CI. From a host perspective they do not really offer much, and are mainly there to test various scheduling challenges in the controller. The tests should still be available for manually testing.

This can be done by simply prefixing the files with _.

Detailed RFC

N/A

Proposed change (Detailed)

N/A

Dependencies

N/A

Concerns and Unresolved Questions

It seems that the controller is not properly tested without these, so there will be less test coverage for the controller without these tests.

Since most of these configuration may also be tested in PTS testing, they will still have some coverage there. Those tests are run on target, and are thus subject to more dynamic timing and are less likely to suffer from the deterministic timing of babblesim.

Alternatives

N/A

References

#81093 and #83539 are just 2 of several recent cases where tests blocked developement for a significant period of time, and tests were disabled for that reason.

An extra bonus is also saving some CI time, as some of these tests have historically taken up to 3 minutes to run.

@aescolar
Copy link
Member

aescolar commented Jan 21, 2025

Unfortunately the current situation with tests failing when an unrelated change changes a bit their timing is very far from desirable, specially as since #83394 we are also running these tests in the nrf54L15bsim which is also has a slight different timing than a 5340 or 52833.
As discussed with @Thalley in another channel, we also see other, non _ac_, tests failing that required work arounds, like for ex.:
5e48633
or
487fab2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: Bluetooth Audio area: Bluetooth Controller area: Bluetooth ISO Bluetooth LE Isochronous Channels area: Bluetooth RFC Request For Comments: want input from the community
Projects
Status: To do
Status: No status
Development

No branches or pull requests

3 participants