qa: Run QA directly from GitHub

[romac]

Description

Currently running QA experiments requires local setup and privileged access to DigitalOcean. To improve the QA process and make it easier to test the impact of code changes on performance and stability, we should add the ability to trigger QA runs directly from GitHub, either through workflow triggers or PR comments.

Proposed Solution

Add GitHub Actions workflow(s) that:

1. Can be triggered either:

   • Manually via a workflow_dispatch event (ie. a manual workflow trigger)
   • By commenting /run-qa on a PR
   • Those should be only available to some privileged users to prevent abuse

2. Runs the QA experiments on DigitalOcean

   • Create the necessary machines
   • Run QA
   • Tear down the machines
   • Alternatively, we could only allow a single QA run at a time and re-use the existing machines

3. Uploads results as GitHub artifacts

   • Store metrics (Prometheus DB), logs, and other relevant outputs
   • Generate a comment on the PR with:
     • Summary of key metrics
     • Link to download full artifacts
     • Any failed checks or warnings

Benefits

• Faster feedback loop for changes
• No need for local setup
• Consistent testing environment
• Easy access to results for reviewers
• Historical record of experiment results

Acceptance Criteria

• QA runs can be triggered from GitHub UI
• PR comments can trigger runs
• Results are automatically uploaded as artifacts
• Clear feedback provided in PR comments
• Proper error handling and reporting

Questions

• How long should we retain artifacts?