[BUG] LoadDataFromFileSystem
gets stuck when used with fsspec
#1110
Labels
bug
Something isn't working
LoadDataFromFileSystem
gets stuck when used with fsspec
#1110
Describe the bug
LoadDataFromFileSystem
never ends loading when used in aPipeline
because call toload
method gets stuck. The source of the problem is thatdistilabel
needs to know the output that will produce the step in advance using theoutputs
property which is accessed from the main process. In theLoadDataFromFileSystem
, theoutputs
property is calling theload
method to load thedatasets.Dataset
and get itscolumn_names
. Then, whenload
method called is again from the child process, it gets stuck. Probably for the same reason as in fsspec/s3fs#464.To reproduce
Expected behavior
The step doesn't get stuck.
Screenshots
No response
Environment
Additional context
No response
The text was updated successfully, but these errors were encountered: