Stopped runners are still used to execute jobs

[Sh4d1]

Hey! Not sure this is the right place, feel free to tell me if it needs posting elsewhere.

Describe the bug
When using self hosted runner, it recently began to use already stopped runners to execute workflow. The runner is marked down in the runner page, and the job is stuck with:

Waiting for a runner to pick up this job...
Job is about to start running on the runner: <runner> (organization)

Then, cancelling the job waits until 10 min (the timeout I guess) to really cancel the job. And when I retry it tries again on the same shutdown runner.
Also, I have an error when force removing it from the runner page.
Any ideas?

It's worth noting that the runners get shutdown without too much notice, but imho it should not be an issue.