Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: improve Docker setup and multi-platform publishing #345

Draft
wants to merge 109 commits into
base: main
Choose a base branch
from

Conversation

5u6r054
Copy link
Contributor

@5u6r054 5u6r054 commented Jan 29, 2025

Description

This PR aims to improve the Docker setup and publishing workflow for the Masa Bittensor project. Currently working through build and publish optimizations.

Current Changes

  • Set up GitHub Actions workflow for multi-platform Docker builds
  • Configured caching strategy using GitHub Actions cache
  • Updated Docker action versions to latest stable releases
  • Maintained SLSA provenance for supply chain security
  • Added configuration for both AMD64 and ARM64 architectures
  • Fixed missing scipy dependency required by scorer module

Current Status

  • 🚧 Working through build time optimization issues
  • �� Testing multi-platform build pipeline
  • ⏳ Docker Hub publishing still needs successful completion
  • ✅ Basic workflow structure is in place
  • ✅ Fixed undeclared scipy dependency used in scorer.py

In Progress

  • Resolve build time performance issues
  • Complete first successful multi-platform build
  • Verify Docker Hub publishing process
  • Validate caching effectiveness

Next Steps

  • Debug and optimize build process
  • Monitor and adjust caching strategy as needed
  • Verify platform-specific builds
  • Test Docker Hub integration

Notes

  • Currently investigating build performance issues
  • Maintaining SLSA provenance despite performance impact
  • Will test caching effectiveness once builds complete successfully
  • Fixed undeclared scipy dependency that was causing test failures in Docker environment

…ly - Remove subnet-config.json dependency - Configure validator directly in docker-compose.yml - Add docker/.gitignore for sensitive files - Switch validator to build from local Dockerfile
… in builder stage - Split RUN commands for better caching - Remove redundant cleanup
…fy to single base image for both validator/miner roles - Remove unnecessary protocol dependency - Configure CI for building and publishing masa-bittensor image - Add local build context for development - Clean up environment variables and service definitions
…pdate wallet paths, enhance logging and registration flow, add async support
… setup with coldkey/hotkey management - Implement burned registration for validators - Fix network/netuid configuration consistency - Add proper error handling and logging - Support Docker container replica numbering - Configure dynamic port allocation for scalability - Remove unnecessary chain_endpoint configuration - Clean up environment variable parsing
…mpose files for better persistence - Add proper volume mounts for wallets and state - Configure environment variables and networking - Update gitignore patterns
…llow bittensor versions >=8.2.0 - Allow masa-ai versions >=0.2.5
… ranges for automatic allocation - Validators: 8091-8154:8091 and metrics 9100-9163:9100 - Miners: 8155-8409:8155 and metrics 9164-9418:9164 - Remove network_mode: host in favor of proper port mapping - Each container gets unique host port from range
…location, implement hotkey mapping, improve registration process, update logging config
@5u6r054 5u6r054 self-assigned this Jan 29, 2025
Copy link

PR description is too short and seems to not fulfill PR template, please fill in

- Update miner replicas from 1 to 3 for better distribution

- Remove monitoring profiles from prometheus, grafana, node-exporter, and portainer

- Successfully deploy stack in swarm mode with proper registration

- Implement retry mechanism for miner registration with 10s delay

- Verify validator and miner services running with correct UIDs

- Configure subnet parameters for organic, synthetic, and healthcheck queries

- Fix connection handling and service discovery in swarm mode
Copy link

PR description is too short and seems to not fulfill PR template, please fill in

Copy link

PR description is too short and seems to not fulfill PR template, please fill in

@5u6r054 5u6r054 changed the title Feature docker compose feat: docker compose, published docker hub images Jan 30, 2025
Copy link

PR description is too short and seems to not fulfill PR template, please fill in

@5u6r054 5u6r054 force-pushed the feature-docker-compose branch from f5dc584 to 9c4e835 Compare January 31, 2025 17:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant