Multi-arch container builds are having some trouble

[shaneutt]

We recently updated our multi-arch container build, and since then I've seen quite a few failures that appear may be flakes and on success, I've seen it take as long as 70 minutes.

This isn't tenable to run on every PR, and each update for that PR. We need to evaluate a different strategy for container builds. Perhaps we can do the full multi-arch build on releases, but for PRs only do a basic x86_64 build?

The purpose of this issue is to explore a bit and see what potential solutions there are, pick one and implement it.

[jokestax]

Hey @shaneutt, could I take this up? I’ve been experimenting with matrix builds and exploring this (https://github.com/taiki-e/upload-rust-binary-action)] I’ll raise a PR with my implementation by this week.

[aryan9600]

the point of doing multi-arch builds for PRs is to verify that we aren't breaking compatibility with arm or x86. since our project relies on quite a bit of third party deps and libraries, its relatively easy to make a change that breaks on a platform.
one thing we can do to speed up CI for PRs, is to split the build into two different jobs, one for each arch. we would build the image for each architecture, but do so natively and in parallel.

we can do a proper multi arch build for releases.

[shaneutt]

@jokestax it would be great if you want to take a look at this, thank you!

/assign @jokestax

Keep in mind what @aryan9600 has said above and confer with him going forward. Particularly:

we can do a proper multi arch build for releases.

I do think we should probably only run these tests (only) on releases.

If this ends up getting very difficult or nebulous for some reason though, please check back in here and we'll support you.

[shaneutt]

hey @jokestax 👋

Just wanted to check in here: how are things going? Need any help?

[jokestax]

Hey @shaneutt ,sorry for not updating this,will make progress by this weekend

[shaneutt]

No worries! Just wanted to check in. If you get blocked or need any help just let us know 🖖

[jokestax]

Hi @shaneutt @aryan9600,

I’ve updated the workflow to parallelize the image build process for AMD64 and ARM64. You can review the changes here: #342.

Since push is set to false, I haven’t included a manifest in this update. I’m unsure if we should add it now or defer it for a future iteration. Let me know your thoughts!

Looking forward to your feedback!

[shaneutt]

Thank you! I'll take a look soon. Don't worry about pushing images, we have #339 to cover that.