Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MDBF-933 - DockerLatentWorker containers name not unique #695

Merged
merged 1 commit into from
Jan 29, 2025

Conversation

RazvanLiviuVarzaru
Copy link
Collaborator

Buildbot calculate the container name on a worker host based on the sha of the master name.

If the master name is the same between dev / prod, which is true at this moment (both environments running in containers), then when the same builder runs on the same host on both environments a conflict will occur causing one of the builds to fail and retry.

Error: The container name "/buildbot-hz-bbw1-docker-tarball-debian-12-15cc21" is already in use by container

This is an error on the development environment when a build << tarball-docker >> was already running in production on hz-bbw1.

Sometimes builds get stuck in "Preparing worker" state and Production is affected. This is why this patch is important, because we can impact Production during our tests in buildbot.dev.mariadb.org

@RazvanLiviuVarzaru
Copy link
Collaborator Author

RazvanLiviuVarzaru commented Jan 28, 2025

Tests:

I changed the hostname manually for master-protected-branches in DEV
to validate my findings.

$ docker exec -it master-protected-branches bash
root@dev_master-protected-branches:/opt/buildbot# hostname
dev_master-protected-branches

Before the change, tarball-docker container name is one of the two:

buildbot-hz-bbw1-docker-tarball-debian-12-15cc21
buildbot-hz-bbw4-docker-tarball-debian-12-15cc21

After the change:
In GUI:

  • master is reported as: dev_master-protected-branches:/srv/buildbot/master/master-protected-branches

Triggering tarball-docker and observing the hash on the worker host:

$ docker ps
CONTAINER ID   IMAGE                                               COMMAND                  CREATED         STATUS        PORTS     NAMES
6d031843c93a   quay.io/mariadb-foundation/bb-worker:dev_debian12   "dumb-init twistd --…"   2 seconds ago   Up 1 second             buildbot-hz-bbw4-docker-tarball-debian-12-5983c

Success, we now have -5983c instead of -15cc21
and production / development should no more collide.

Buildbot calculate the container name on a worker host
based on the sha of the master name.

If the master name is the same between dev / prod, which is true
at this moment (both environments running in containers),
then when the same builder runs on the same host on both environments
a conflict will occur causing one of the builds to fail and retry.

Error: The container name "/buildbot-hz-bbw1-docker-tarball-debian-12-15cc21" is already in use by container
This is an error on the development environment when a build << tarball-docker >> was already running in production on hz-bbw1.

Sometimes builds get stuck in "Preparing worker" state and Production is affected.
This is why this patch is important, because we can impact Production during our tests in buildbot.dev.mariadb.org
@fauust
Copy link
Collaborator

fauust commented Jan 28, 2025

The only thing that I don't like but we already discussed it is the docker-compose.yml file it's confusing. We should probably remove it from the repo and/or rename it and create 2:

  • docker-compose.dev.yml
  • docker-compose.prod.yml

Anyway, ok to merge ASAP.

@RazvanLiviuVarzaru RazvanLiviuVarzaru merged commit 788f811 into MariaDB:dev Jan 29, 2025
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants