[FEATURE] basic use of pipeline to generate SFT dataset from documents

[burtenshaw]

This is a continuation of this: #1059

It implements a pipeline abstraction template that runs on SelfInstruct step and text generation on a dataset of documents. This should help boot strap basic users to build SFT datasets.

from datasets import Dataset
import wikipedia
from distilabel.pipeline import DatasetInstructionResponsePipeline

pipeline = DatasetInstructionResponsePipeline(num_instructions=5)

distiset = pipeline.run(
    use_cache=False,
    dataset=Dataset.from_list(
        [
            {
                "input": wikipedia.page(title="Transfer_learning").content,
            }
        ]
    ),
)

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[github-actions]

Documentation for this PR has been built. You can view it at: https://distilabel.argilla.io/pr-1076/

[codspeed-hq]

CodSpeed Performance Report

Merging #1076 will degrade performances by 49.04%

_{Comparing feat/dataset-instruction-response-pipeline (0303d0f) with develop (f5ddbc6)}

Summary

❌ 1 regressions

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Benchmark	BASE	HEAD	Change
❌	test_cache_time	2.2 s	4.3 s	-49.04%

[davidberenstein1957]

@burtenshaw can we get rid of the pipeline.pipeline.run? Also, perhaps we could limit the exposure to different classes with something like the following. Under the hood it can still use the same but we just use different arguments. WDYT?

from datasets import Dataset
import wikipedia
from distilabel.pipeline import InstructionResponsePipeline

pipeline = InstructionResponsePipeline(num_instructions=5)

distiset = pipeline.pipeline.run(
    use_cache=False,
    dataset=Dataset.from_list(
        [
            {
                "input": wikipedia.page(title="Transfer_learning").content,
            }
        ]
    ),
)

[davidberenstein1957]

@burtenshaw I think it would be worth it to have a dedicated section on this somewhere in the docs, , after that and resolving the tests we should be able to merge.

I would add it to the quickstart and perhaps to the components gallery under "pipelines" or something more explicit like "ready-to-go pipelines" or something.

[burtenshaw]

@burtenshaw I think it would be worth it to have a dedicated section on this somewhere in the docs, , after that and resolving the tests we should be able to merge.

I would add it to the quickstart and perhaps to the components gallery under "pipelines" or something more explicit like "ready-to-go pipelines" or something.

Thanks. I agree with those suggestions. I'll work on this next week.

[davidberenstein1957]

@burtenshaw perhaps you can add the documentation is this PR?

[davidberenstein1957]

Also, perhaps I like some more explicit naming like InstructionResponseFromDataPipeline or InstructionResponseFromSeedDataPipeline better.

[davidberenstein1957]

[burtenshaw]

@davidberenstein1957 This looks great. Thanks for the help.

Are we ready to go?