Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[save] Stream Worksheet XML to Disk #1255

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Conversation

JanMarvin
Copy link
Owner

@JanMarvin JanMarvin commented Jan 29, 2025

This is an experimental branch, using a custom C++ function to stream the XML file to disk. It is working, but not extensively tested and might still lack features. It has shown that it can reduce the memory consumption significantly.

@JanMarvin
Copy link
Owner Author

Cherry pick 15daf92

@JanMarvin
Copy link
Owner Author

JanMarvin commented Jan 29, 2025

Using this, is a game changer memory wise. Writing a 500,000 x 100 rnorm data frame previously required - with the optimizations from this branch - ~28GB. Writing the output file with pugixml using a custom xml writer, reduces the required memory amount by 12-14GB.
The changes in the branch skip a few guards that check that the cells and rows are in the correct order.

options("openxlsx2.export_with_pugi" = FALSE)

This branch skips building the XML file in memory before it is written to the disk. The custom XML writer simply flushes what is available into a text file. Without any checks for correct XML etc.

@JanMarvin JanMarvin changed the title [save] No copy sheet data [save] Stream Worksheet XML to disk Jan 29, 2025
@JanMarvin JanMarvin changed the title [save] Stream Worksheet XML to disk [save] Stream Worksheet XML to Disk Jan 29, 2025
@JanMarvin JanMarvin added enhancement 😀 New feature or request help wanted 🙏 Extra attention is needed options ☑️ labels Jan 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement 😀 New feature or request help wanted 🙏 Extra attention is needed options ☑️
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant