Stop automatically deleting inactive runners

[pimhakkertgc]

Select Topic Area

Product Feedback

Body

We have runners stationed in 3 different servers. Normally we make use of these on a weekly basis, but this is not the case when I go on vacation. I normally do so for at least 15 days, twice a year. When I am not working there are usually no deployments, and as such the runners do not get activated.

We start our runners manually and do not leave them running in an automated process.

When runners are not ran for at least 14 days, Github removes them from the system and Github Actions. This is incredibly annoying and means that I have to delete the local runner directories and rebuild the runners from scratch twice a year.

Can you please not delete inactive runners? Or let us at least press a button to mark a runner to not be automatically deleted.

[ttolbol]

Agreed! This is very annoying in our company where we use self-hosted runners to deploy updates. This is triggered manually 1-2 times per month and has worked flawlessly for years, but now we have to waste time setting up runners every single time! Github have basically made runners single use for us now.

[rossbu]

make sure it's tracked, guys, and this should never be deleted for no reason, inactivity for a few month or even a bit longer is not a reason for github to do so

[CMM1305]

I agree, there should be a way of marking self-hosted runners to remain permanently unless we decide to manually delete them. I don't understand GitHub's reasoning for making this change. We should be in control of the runners we create and we decide when and if to delete them.

[JimMikola]

The automatic deletion of offline selfhosted runners needs to be configurable by the organization.

We have multiple runners that are staged to cover peaks in automation activities. The runners are in an offline state until required, at which point a low-cost runner spins up one of the staged high-cost runners. Some of the high-cost runners are offline for more than 14 days when automation workloads are low.

[tom-carbontrail]

Disappointing this hasn't been addressed by the GH team for over a year. This is also adding friction for no good reason. I agree this needs to be configurable, or at least allow a previously registered runner to "resurrect" its previous registration...

[nomeelnoj]

Also, we run self hosted code-scanning runners. We dynamically spin up the runners as needed, but if we dont have at least one runner registered with code-scanning as the label (even an offline runner will do), the process does not wait for our dynamic generation of runners and therefore it automatically picks up Github managed runners, and causes issues for us.

Can we please have a way of making sure offline runners do not get deleted after 14 days?!

[stuartjvnorton]

Even if they disabled the runner and it had to be manual re-enabled in the admin console, I wouldn't really have a problem with that. But this business of nuking it on 1 side and just leaving it hanging on the other is just a terrible solution.

[Martiix]

They even say they will be deleted after 14 days, but mine are even deleted long before that. I want to disable the whole removal part and instead remove them manually if needed, to have it controlled and on purpose. Now I just randomly discover missing runners, which is super annoying, don't even get notified that they are deleted.

This apparently took less than 3 days to be removed, and it wasn't even supposed to be inactive :( so in this case github's bug where runners become inactive without reason combined with auto-removal is very frustrating.

[Martiix]

We ended up making a hard-coded "runner-state" file with runners and labels and then made some tests in a nightly workflow that checks that the state is as expected. Annoying to have to do this, but at least now we get notified when our runners are suddenly removed. I hope that we can convert this into a stateful setup where a runner is just auto recreated if it is missing in the future. But honestly, why are they removed in the first place...

[echohack]

Are you sure the runners aren't automatically deregistering themselves?

This is a common workflow for runner templates: https://github.com/myoung34/docker-github-actions-runner/blob/d9094bd6ddf4da8c10f5edfe00ec334e12ce2e27/entrypoint.sh#L22

[ManuelSchenk]

Hi GH, please fix this issue! Its unacceptable behavior!

[FredeJ]

Just adding my voice to the choir here, that this is super annoying.

I've set up hardware-in-the-loop testing using GitHub runners. Just came back from Christmas vacation and suddenly one of the runners is missing, which completely stops our whole release flow as we validate firmware on all previous hardware versions and some of the old ones are in short supply.

I can't remove it, because it needs a removal token, that I can't access because it's been autoremoved. So now I may need to re-flash it or figure out where the configuration of the runner is stored. It's way more hassle than I would want.

This auto-removal "feature" basically cost me a day of work.

[ojnick]

I know this is probably not useful to you at this point, but maybe it'll help someone else. The token seems to be the same for all runners under an organization, so if you go to add a new runner on GitHub and you can use the token provided in the Configure section of the instructions for the removal token.

[MMalawski2]

Same here. To make it worse, it does not say anywhere that GH has removed runners due to inactivity. I was surprised when I came back from holidays and found half of my runners missing. Lost half day to investigate and then another to fix it.

[echohack]

Docs here: https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/removing-self-hosted-runners#removing-a-runner-from-a-repository

[stuartjvnorton]

Our runner boxes are on AWS, so we have a schedule that starts them up once a week during quiet time and shuts down after an hour.
We thought that would be enough for the runner and GitHub to sort themselves out, but we still had runners removed after 2 weeks.
So now we force the issue.
Each runner has a unique label. A script is scheduled to run on machine start: it waits until the runner service is up, then fires a curl call to a GitHub action we wrote just for this, citing the unique label.
Then the action runs a Hello World type script on the agent matching the label and inactivity clock is reset.

[davec-fvr]

Likewise I have lost multiple days of work resetting up a runner more than once because of this decision. THe EC2 instance we're connecting to is still running but no actions were run on it for a month and now we need to make urgent bug fixes to a product we released and I'm wasting time chasing people to get this reconfigured for no reason

[crohr]

I think they're really pushing for the use of ephemeral runners, and/or want to reduce the attack surface if you somehow leave a runner daemon dangling and forget about it.

If using AWS I would advise looking at the various solutions for self-hosting ephemeral runners there, including my own.

[RW-RuneAnkerroLiljegren]

+1 for this! I have a couple of UI testing selfhosted runners that I use in virtual machines. I have to remember to start these up and run them. This is incredibly irritating. If github marked them as inactive I would haev been fine with that so you can recall them. Even, update the keys if need be. but setting them up again and again is a pain.

[nebuk89]

Sorry I haven't seen this one come up before :) and sorry for not spotting this feedback sooner!

This is an interesting challenge, as a rule using ephemeral runners or something that autoscales is better (and doesn't have this challenge) but clearly things like that aren't always feasible (if you would share with me why it's not for y'all I would <3 to hear).

Regardless of the why, the ask to persist certain runners seems fair but I will need to have someone think it through as it still can't be 'forever' (as we would end up with a lot of actually dead runners hanging around).

If todays time to remove was a 'short' what 'long' would work for everyone?

[Martiix]

This is an interesting challenge, as a rule using ephemeral runners or something that autoscales is better (and doesn't have this challenge) but clearly things like that aren't always feasible (if you would share with me why it's not for y'all I would <3 to hear).

We do a lot of hardware testing and are using labels on runners to know how much/many resources are available, not sure if this still isn't possible with ephemeral runners. We for instance need specific amounts of gpu memory, our own products are cameras which also have limitations in which type you get and only one runner can use it at the time etc. Hope that helps.

My first request is to at least have a way of sending a notification to somewhere when a runner is removed, as knowing about it at least helps us be aware. We have atm worked around this by having a hardcoded list of what we expect our runners to look like and we have added a workflow job just to verify that the runners are as expected every night. Using a webhook to send notifications to teams for instance is desirable :)

We use all our runners basically continuously, so don't have that exact problem. However we have experienced runners being deleted and going idle without recovery on multiple occasions, which seems like a bug. However step one to make it "nicer" is to reduce the amount of time it takes us to figure it out so we can recover faster. It might take hours if not days before we notice and it's a big problem with temporary reduced capacity in our CI.

[stuartjvnorton]

Our interactions are not related to build and deployment of code, but interacting with a 3rd party client-server application to build and deploy assets native to that application. A couple of core tasks in that interaction are only possible using tools provided with the windows thick client.
So we have a small number of self-hosted runners on machines that are reasonably expensive to build in terms of time and effort.
Ephemeral runners or autoscaling don't really suit our needs.
Some don't run very often, but that doesn't mean we should be forced to reinstall those runners.

[echohack]

Oh hey @nebuk89 long time no see. Couple of potentially-obvious suggestions based on what I'm seeing around 👋

I think the thing that folks would like the most is a way to send a signal at the end of a self-hosted runner run where there is a retention_policy. I think the assumption that there is some kind of default retention at 14 days is probably a good one, but folks that are running into issues seem to want to retain offline runners for a longer period of time. Perhaps if you had this setting you could be more aggressive about the default retention as a tradeoff.

Basically this way the runner can renew/reset its retention date at the end of a run (regardless of if it was successful or not) as a basic kind of keep-alive policy.

[samuelsadok]

I also lost self-hosted runners at multiple points in time, so chiming in here.

If todays time to remove was a 'short' what 'long' would work for everyone?

Just speaking for my case right now, 3 weeks would have worked.

But what would stop you from making it really long, like 3mo, 1yr or 10yr? Why can't it be forever? As far as I'm aware there are many other things that are not autodeleted after a period of inactivity (repositories, users, collaborators, SSH keys, secrets etc) - what's different about runners?

Anyway, simply increasing the timeout seems like the simplest fix that would help most of us. Making it configurable from the frontend (and API) would be a more advanced option. Sending a special signal at the end of a run seems a bit convoluted to me but if it solves the problem I could live with that too.

Often times, self-hosted runners are physical machines (e.g. hardware-in-the-loop tests, as others have mentioned). Treating them as ephemeral seems unintuitive and autoscaling is also not simply feasible (as opposed to VM-based runners). Yes a user could add automation to add the runner back periodically but that seems like something the runner infrastructure (service, server) should take care of under the hood.