This AWS Cloud Development Kit (CDK) project deploys an AWS FSx for Lustre filesystem and mounts it on AWS Batch tasks to access S3 data from batch workloads with a shared storage.
This project builds the solution as described in the AWS Storage blog: Automatically import Amazon S3 object updates into Amazon FSx for Lustre and combines it with this Knowledge Center article to integrate it with AWS Batch.
This CDK project creates an S3 bucket with dummy content and deploys an FSx for Lustre filesystem into a new VPC, as well as AWS Batch resources to run a sample job. The project includes the following stacks:
NetworkingStack: deploys a VPC with a public subnet, a private subnet, and a NAT Gateway.FSxStack: deploys the FSx for Lustre filesystem into the VPC and creates an S3 bucket with a few objects for testing.BatchStack: creates a Launch Template that mounts the FSx filesystem, a Compute Environment that leverages this Launch Template, a job queue that points to the Compute Environment, and a Job Definition that mounts the filesystem into a volume on the container.
Since all three stacks depend on each other, deploying BatchStack will also deploy the other stacks:
cdk -f --require-approval never deploy BatchStackcdk destroy --allThis project implements the solutions described in the following articles:
- https://docs.aws.amazon.com/fsx/latest/LustreGuide/create-fs-linked-data-repo.html
- https://aws.amazon.com/blogs/storage/automatically-import-amazon-s3-object-updates-into-amazon-fsx-for-lustre/
- https://aws.amazon.com/premiumsupport/knowledge-center/batch-fsx-lustre-file-system-mount/
See CONTRIBUTING for more information.
This library is licensed under the MIT-0 License. See the LICENSE file.