Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot open 750Mb Auspice JSON in Chrome #1622

Closed
Vjimenez-vasquez opened this issue Dec 22, 2022 · 6 comments
Closed

Cannot open 750Mb Auspice JSON in Chrome #1622

Vjimenez-vasquez opened this issue Dec 22, 2022 · 6 comments
Labels
enhancement New feature or request

Comments

@Vjimenez-vasquez
Copy link

Vjimenez-vasquez commented Dec 22, 2022

Hi everyone,

Cannot open a *.json file in https://auspice.us/. My file is about 750Mb (2400 monkeypox complete genomes). I ran augur pipeline more than 5 times trying to solve the issue and I never received a warning message, my pipeline runs good and produces a final "right" file. I made a test with only 600 monkeypox genomes (retrieving json file about 25mb) and auspice.us works well (it opens the file with any problem). Is there any limit size to upload in auspice.us ?

Please find attached the drive link to check my files, my compressed *json file is "ns_monkeypox_221222.rar". You will find my metadata.tsv, auspice_config.json, lat_longs.tsv, raw_tree.nwk, refine_tree.nwk

https://drive.google.com/drive/folders/1zHeNQCCkTndamhUpay_VxPAyBJLnEZ4W?usp=share_link

Please, I would appreciate you to help me with this issue.

@Vjimenez-vasquez Vjimenez-vasquez added the bug Something isn't working label Dec 22, 2022
@joverlee521
Copy link
Contributor

Hi @Vjimenez-vasquez,

Which browser are you using? Your issue sounds similar to #1499.
I am able to load your JSON in Firefox but not Chrome. If you are using Chrome, then this is most likely due to Chrome's limits on memory usage.

In general, I recommend using the --minify-json option for augur export v2 to export the JSON without indentations and line returns to reduce the size of the file.

@Vjimenez-vasquez
Copy link
Author

Thank you so much @joverlee521, you are right Firefox works pretty fine ! I did not notice that solution.

@corneliusroemer
Copy link
Member

Maybe we could add a warning in Export if the file is bigger than what Chrome will tolerate, say 500 MB, and suggest to minify.

@Vjimenez-vasquez
Copy link
Author

Hi @corneliusroemer. Sure ! That would help a lot !

@victorlin victorlin moved this from New to Backlog in Nextstrain planning (archived) Dec 28, 2022
@victorlin victorlin changed the title Cannot open a *.json file in auspice.us Add warning for large dataset files (>500MB?) on Chrome Dec 28, 2022
@victorlin victorlin added enhancement New feature or request and removed bug Something isn't working labels Dec 28, 2022
joverlee521 added a commit to nextstrain/zika that referenced this issue Apr 16, 2024
Based on feedback from @jameshadfield in
#56 (comment)

Looking at the existing dataset files on S3,
the 5 KiB root-sequence.json is pretty negligible when the main
Auspice JSON is only 163 KiB. Nextstrain datasets are limited by the
500MB memory cap in Chrome,¹ so we'd be fine adding the
root sequence inline.

¹ nextstrain/auspice#1622
joverlee521 added a commit to nextstrain/dengue that referenced this issue Apr 16, 2024
Based on feedback from @jameshadfield in
nextstrain/zika#56 (comment)

Looking at the existing dataset files on S3,
the 5-6 KiB root-sequence.jsons are pretty negligible when the main
Auspice JSONs are 600-800 KiB. Nextstrain datasets are limited by the
500MB memory cap in Chrome,¹ so we'd be fine adding the
root sequence inline.

This ensures that our uploads will include the root sequence so that
they don't get out-of-sync with the main Auspice JSON.

¹ nextstrain/auspice#1622
joverlee521 added a commit to nextstrain/measles that referenced this issue Apr 16, 2024
Based on feedback from @jameshadfield in
nextstrain/zika#56 (comment)

Looking at the existing dataset files on S3,
the 8 KiB root-sequence.json is pretty negligible when the main
Auspice JSON is only 163 KiB. Nextstrain datasets are limited by the
500MB memory cap in Chrome,¹ so we'd be fine adding the
root sequence inline.

¹ nextstrain/auspice#1622
joverlee521 added a commit to nextstrain/measles that referenced this issue Apr 16, 2024
Based on feedback from @jameshadfield in
nextstrain/zika#56 (comment)

Looking at the existing dataset files on S3,
the 8 KiB root-sequence.json is pretty negligible when the main
Auspice JSON is only 163 KiB. Nextstrain datasets are limited by the
500MB memory cap in Chrome,¹ so we'd be fine adding the
root sequence inline.

This ensures that our uploads will include the root sequence so that
they don't get out-of-sync with the main Auspice JSON.

¹ nextstrain/auspice#1622
@genehack
Copy link
Contributor

Between nextstrain/augur#1155 and nextstrain/augur#1331, I think the original problem has been solved — closing this out.

@victorlin victorlin changed the title Add warning for large dataset files (>500MB?) on Chrome Cannot open 750Mb Auspice JSON in Chrome Jan 14, 2025
@victorlin
Copy link
Member

As mentioned in nextstrain/augur#1155 (comment), it's still worth adding a warning in Auspice for large JSONs. The discussion in this issue is mostly around a specific example though, so I've updated the title.

Re-opened another issue for adding the warning and closing nextstrain/augur#1155.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
No open projects
Status: Backlog
Development

No branches or pull requests

5 participants