GitHub: Git operation failures
231 comments
·November 18, 2025mandus
__MatrixMan__
This escalator is temporarily stairs, sorry for the convenience.
Akronymus
Tbh, I personally don't trust a stopped escalator. Some of the videos of brake failures on them scared me off of ever going on them.
collingreen
You've ruined something for me. My adult side is grateful but the rest of me is throwing a tantrum right now. I hope you're happy with what you've done.
jimbokun
As long as you didn't go all in on GitHub Actions. Like my company has.
esafak
Then your CI host is your weak point. How many companies have multi-cloud or multi-region CI?
IshKebab
Do you think you'd get better uptime with your own solution? I doubt it. It would just be at a different time.
wavemode
Uptime is much, much easier at low scale than at high scale.
The reason for buying centralized cloud solutions is not uptime, it's to safe the headache of developing and maintaining the thing.
jakewins
“Your own solution” should be that CI isn’t doing anything you can’t do on developer machines. CI is a convenience that runs your Make or Bazel or Just or whatever you prefer builds, that your production systems work fine without.
I’ve seen that work first hand to keep critical stuff deployable through several CI outages, and also has the upside of making it trivial to debug “CI issues”, since it’s trivial to run the same target locally
tcoff91
Compared to 2025 github yeah I do think most self-hosted CI systems would be more available. Github goes down weekly lately.
davidsainez
Doesn’t have to be an in house system, just basic redundancy is fine. eg a simple hook that pushes to both GitHub and gitlab
nightski
I mean yes. We've hosted internal apps that have four nines reliability for over a decade without much trouble. It depends on your scale of course, but for a small team it's pretty easy. I'd argue it is easier than it has ever been because now you have open source software that is containerized and trivial to spin up/maintain.
The downtime we do have each year is typically also on our terms, not in the middle of a work day or at a critical moment.
lopatin
The issue is that GitHub is down, not that git is down.
ElijahLynn
You just lose the "hub" of connecting others and providing a way to collaborate with others with rich discussions.
parliament32
All of those sound achievable by email, which, coincidently, is also decentralized.
Aurornis
Some of my open source work is done on mailing lists through e-mail
It's more work and slower. I'm convinced half of the reason they keep it that way is because the barrier to entry is higher and it scares contributors away.
null
awesome_dude
Wait, email is decentralised?
You mean, assuming everyone in the conversation is using different email providers. (ie. Not the company wide one, and not gmail... I think that covers 90% of all email accounts in the company...)
keybored
I don’t use GitHub that much. I think the thing about “oh no you have centralized on GitHub” point is a bit exaggerated.[1] But generally, thinking beyond just pushing blobs to the Internet, “decentralization” as in software that lets you do everything that is Not Internet Related locally is just a great thing. So I can never understand people who scoff at Git being decentralized just because “um, actually you end up pushing to the same repository”.
It would be great to also have the continuous build and test and whatever else you “need” to keep the project going as local alternatives as well. Of course.
[1] Or maybe there is just that much downtime on GitHub now that it can’t be shrugged off
Conscat
I'm on HackerNews because I can't do my job right now.
y42
I work in the wrong time zone. Good night.
ramon156
SSH also down
gertlex
My pushing was failing for reasons I hadn't seen before. I then tried my sanity check of `ssh git@github.com` (I think I'm supposed to throw a -t flag there, but never care to), and that worked.
But yes ssh pushing was down, was my first clue.
My work laptop had just been rebooted (it froze...) and the CPU was pegged by security software doing a scan (insert :clown: emoji), so I just wandered over to HN and learned of the outage at that point :)
kragen
SSH works fine for me. I'm using it right now. Just not to GitHub!
stevage
Curious whether you actually think this, or was it sarcasm?
0x457
It was sarcasm, but git itself is Decentralized VCS. Technically speaking, every git checkout is a repo of itself. GitHub doesn't stop me from having the entire repo history up to last pull, and I still can push either to the company backup server or my coworker directly.
However, since we use github.com fore more than just a git hosting it is SPOF in most cases, and we treat it as a snow day.
SteveNuts
I have a serious question, not trying to start a flame war.
A. Are these major issues with cloud/SaaS tools becoming more common, or is it just that they get a lot more coverage now? It seems like we see major issues across AWS, GCP, Azure, Github, etc. at least monthly now and I don't remember that being the case in the past.
B. If it's becoming more common, what are the reasons? I can think of a few, but I don't know the answer, so if anyone in-the-know has insight I'd appreciate it.
Operations budget cuts/layoffs? Replacing critical components/workflows with AI? Just overall growing pains, where a service has outgrown what it was engineered for?
Thanks
wnevets
> A. Are these major issues with cloud/SaaS tools becoming more common, or is it just that they get a lot more coverage now? It seems like we see major issues across AWS, GCP, Azure, Github, etc. at least monthly now and I don't remember that being the case in the past.
FWIW Microsoft is convinced moving Github to Azure will fix these outages
Lammy
Everything old is new again.
https://www.zdnet.com/article/ms-moving-hotmail-to-win2000-s...
bovermyer
Microsoft is also convinced that its works are a net benefit for humanity, so I would take that with a grain of salt.
andrewstuart2
I think it would be pretty hard to argue against that point of view, at least thus far. If DOS/Windows hadn't become the dominant OS someone would have, and a whole generation of engineers cut their teeth on their parents' windows PCs.
einsteinx2
The same Azure that just had a major outage this month?
kkarpkkarp
> If it's becoming more common, what are the reasons?
Someone answered this morning, while Cloudflare outage, it's AI vibe coding and I tend to think there is something true in this. At some point there might be some tiny grain of AI engaged which starts the avalanche ending like this.
junon
Been on GitHub for a long time. It feels like they're more often. It used to be yearly if at all that GitHub was noticably impacted. Now it's monthly, and recently, seemingly weekly.
0x457
Definitely not how I remember. First, I remember seeing unicorn page multiple times a day some weeks. There were also time when webhook delivery didn't work, so circle ci users couldn't kick off any builds.
What change is how many services GitHub can be having issues.
zackify
there has been 5 between actions and push pull issues just this month. it is more often
chadac
I suspect that the Azure migration is influencing this one. Just a bunch of legacy stuff being moved around along with Azure not really being the most reliable on top... I can't imagine it's easy.
cmrdporcupine
In the early days of GitHub (like before 2010) outages were extremely common.
bovermyer
I agree, for what that's worth.
However, this is an unexpected bell curve. I wonder if GitHub is seeing more frequent adversarial action lately. Alternatively, perhaps there is a premature reliance on new technology at play.
netghost
I think it was generally news when there were upages and the site was up. Similar with twitter for that matter.
junon
Not from my recollection. Not like this. BitBucket on the other hand had a several day outage at one point. That one I do recall.
AIorNot
well layoffs across tech probably havent helped
https://techrights.org/n/2025/08/12/Microsoft_Can_Now_Stop_R...
ever since Musk greenlighted firing people again.. CEOs can't wait to pull the trigger
tingletech
Years ago on hackernews I saw a link about probability describing a statistical technique that one could use to answer a question about if a specific type of event was becoming more common or not. Maybe related to the birthday paradox? The gist that I remember is that sometimes a rare event will seem to be happening more often, when in reality there is some cognitive bias that makes it non-intuitive to make that decision without running the numbers. I think it was a blog post that went through a few different examples, and maybe only one of them was actually happening more often.
swed420
> B. If it's becoming more common, what are the reasons?
Among other mentioned factors like AI and layoffs: mass brain damage caused by never-ending COVID re-infections.
Since vaccines don't prevent transmission, and each re-infection increases the chances of long COVID complications, the only real protection right now is wearing a proper respirator everywhere you go, and basically nobody is doing that anymore.
There are tons of studies to back this line of reasoning.
smsm42
It certainly feels that way, though it may be an instance of availability bias. Not sure what's causing it - maybe extra load from AI bots (certainly a lot of smaller sites complain about it, maybe major providers feel the pain too), maybe some kind of general quality erosion... It's certainly something that is waiting for a serious research.
pm90
Github isn't in the same reliability class as the hyperscalars or cloudflare; its comically bad now, to the point that at a previous job we invested in building a readonly cache layer specifically to prevent github outages from bringing our system down.
chrsstrm
I thought I was going crazy when I couldn't push changes but now it seems it's time to just call it for the day. Back at it tomorrow.
Mossly
Seeing auth succeed but push fail was an exercise in hair pulling.
curioussquirrel
Same, even started adding new ssh keys to no avail... (I was getting some nondescript user error first, then unhealthy upstream)
chrsstrm
Would love to see a global counter for the number of times ‘ssh -T git@github.com’ was invoked.
peciulevicius
same, i've started pulling my hair out, was about to nuke my setup and set it up all from scratch
keepamovin
lol same. Hilarious when this shit goes down that we all rely on like running water. I'm assuming GitHub was hacked by the NSA because someone uploaded "the UFO files" or sth.
_jab
GitHub is pretty easily the most unreliable service I've used in the past five years. Is GitLab better in this regard? At this point my trust in GitHub is essentially zero - they don't deserve my money any longer.
ecshafer
We self host gitlab, so its very stable. But Gitlab also kind of is enterprise software. It hits every feature checkbox, but they aren't well integrated, and they are kind of half way done. I don't think its as smooth of an experience as Github personally, or as feature rich. But Gitlab can self host your project repos, cicd, issues, wikis, etc. and it does it at least okay.
input_sh
I would argue GitLab CI/CD is miles ahead of the dumpster fire that is GitHub Actions. Also the homepage is actually useful, unlike GitHub's.
tottenhm
Frequently use both `github.com` and self-hosted Gitlab. IMHO, it's just... different.
Self-hosted Gitlab periodically blocks access for auto-upgrades. Github.com upgrades are usually invisible.
Github.com is periodically hit with the broad/systemic cloud-outage. Self-hosted Gitlab is more decentralized infra, so you don't have the systemic outages.
With self-hosted Gitlab, you likely to have to deal with rude bots on your own. Github.com has an ops team that deals with the rude bots.
I'm sure the list goes on. (shrug)
jakub_g
My company self-hosts GitLab. Gitaly (the git server) is a weekly source of incidents, it doesn't scale well (CPU/memory spikes which end up taking down the web interface and API). However we have pretty big monorepos with hundreds of daily committers, probably not very representative.
cactusfrog
There’s this Gitlab incident https://www.youtube.com/watch?v=tLdRBsuvVKc
noosphr
You can make it as reliable as you want by hosting it on prem.
jakub_g
> as reliable as you want
We self-host GitLab but the team owning it is having hard time scaling it. From my understanding talking to them, the design of gitaly makes it very hard to scale it beyond certain repo size and # of pushes per day (for reference: our repos are GBs in size, ~1M commits, hundreds of merges per day)
themafia
Flashbacks to me pushing hard for GitLab self hosting a few months ago. The rest of the team did not feel the lift was worth it.
I utterly hate being at the mercy of a third party with an after thought of a "status page" to stare at.
cindyllm
[dead]
null
yoyohello13
We've been self hosting GitLab for 5 years and it's the most reliable service in our organization. We haven't had a single outage. We use Gitlab CI and security scanning extensively.
markbnj
Ditto, self-hosted for over eight years at my last job. SCM server and 2-4 runners depending on what we needed. Very impressive stability and when we had to upgrade their "upgrade path" tooling was a huge help.
tapoxi
Another GitLab self-hosting user here, we've run it on Kubernetes for 6 years. It's never gone down for us, maybe an hour of downtime yearly as we upgrade Postgres to a new version.
kennysmoothx
FYI in an emergency you can edit files directly on Github without the need to use git.
Edit: ugh... if you rely on GH Actions for workflows though actions/checkout@v4 is also currently experiencing the git issues, so no dice if you depend on that.
ruuda
FYI in an emergency you can `git push` to and `git pull` from any SSH-capable host without the need to use GitHub.
cluckindan
FYI in an emergency you can SSH to your server and edit files and the DB directly.
Where is your god now, proponents of immutable filesystems?!
egeozcan
FYI in an emergency, you can buy a plane ticket and send someone to access the server directly.
I actually had the privilege of being sent to the server.
BadBadJellyBean
I love when people do that because they always say "I will push the fix to git later". They never do and when we deploy a version from git things break. Good times.
I started packing things into docker containers because of that. Makes it a bit more of a hassle to change things in production.
lenerdenator
I'm actually getting "ERROR: no healthy upstream" on `git pull`.
They done borked it good.
avree
If your remote is set to a git@github.com remote, it won't work. They're just pointing out that you could use git to set origin/your remote to a different ssh capable server, and push/pull through that.
shrikant
We're not using Github Actions, but CircleCI is also failing git operations on Github (it doesn't recognise our SSH keys).
rco8786
Yup, we were just trying to hotfix prod and ran into this. What is happening to the internet lately.
vielite1310
True that, and this time Github AI actually have a useful answer to check for githubstatus.com
cjonas
I didn't really want to work today anyways. First cloudflare, now this... Seems like a sign to get some fresh air
dlahoda
we depend too much on usa centralized tech.
we need more soverenity and decentralization.
lorenzleutgeb
Please check out radicle.dev, helping hands always welcome!
letrix
> Repositories are replicated across peers in a decentralized manner
You lost me there
worldsavior
How is this related to them being located in the USA?
CivBase
The sad part is both the web and git were developed as decentralized technologies, both of which we foolishly centralized later.
The underlying tech is still decentralized, but what good does that do when we've made everything that uses it dependent on a few centralized services?
lol768
> We are seeing failures for some git http operations and are investigating
It's not just HTTPS, I can't push via SSH either.
I'm not convinced it's just "some" operations either; every single one I've tried fails.
olivia-banks
A friend of mine was able to get through a few minutes ago, apparently. Everyone else I know is still fatal'ing.
shooker435
https://www.githubstatus.com/incidents/5q7nmlxz30sk
it's up now (the incident, not the outage)
laurentiurad
A lot of failures lately during the aI ReVoLuTiOn.
SOLAR_FIELDS
GitHub has a long history of garbage reliability that long predates AI
personjerry
Looks like Gemini 3 figured out the best way to save costs on its compute time was to shut down github!
arbol
I'm also getting this. Cannot pull or push but can authenticate with SSH
myrepo git:(fix/context-types-settings) gp
ERROR: user:1234567:user
fatal: Could not read from remote repository.
myrepo git:(fix/context-types-settings) ssh -o ProxyCommand=none git@github.com
PTY allocation request failed on channel 0
Hi user! You've successfully authenticated, but GitHub does not provide shell access.
Connection to github.com closed.OptionOfT
It is insane how many failures we've been getting lately, especially related to actions.
* jobs not being picked up
* jobs not being able to be cancelled
* jobs running but showing up as failed
* jobs showing up as failed but not running
* jobs showing containers as pushed successfully to GitHub's registry, but then we get errors while pulling them
* ID token failures (E_FAIL) and timeouts.
manbitesdog
Same! The current self-hosted runner gets hung every so often
veighnsche
Probably because AI-generated reviews has made qa way worse.
Good thing git was designed as a decentralized revision control system, so you don’t really need GitHub. It’s just a nice convenience