Compiler Explorer and the promise of URLs that last forever

How do you manage those? Do you have a way to search them, or a specific way to catalogue them, which will make it easy to find exactly what you need from them?

Thanks. I didn't know about this and it looks great.

A couple of questions:

- do you store them compressed or plain?

- what about private info like bank accounts or health issuance?

I guess for privacy one could train oneself to use private browsing mode.

Regarding compression, for thousands of files don't all those self-extraction headers add up? Wouldn't there be space savings by having a global compression dictionary and only storing the encoded data?

storage is cheap, but if you wanted to improve this:

1. find a way to dedup media

2. ensure content blockers are doing well

3. for news articles, put it through readability and store the markdown instead. if you wanted to be really fancy, instead you could attempt to programatically create a "template" of sites you've visited with multiple endpoints so the style is retained but you're not storing the content. alternatively a good compression algo could do this, if you had your directory like /home/andrew/archive/boehs.org.tar.gz and inside of the tar all the boehs.org pages you visited are saved

4. add fts and embeddings over the pages

Are you automating this in some fashion? Is there another extension you've authored or similar to invoke SingleFile functionality on a new page load or similar?

i was considering a similar setup, but i don’t really trust extensions. Im curious;

- Do you also archive logged in pages, infinite scrollers, banking sites, fb etc? - How many entries is that? - How often do you go back to the archive? is stuff easy to find? - do you have any organization or additional process (eg bookmarks)?

did you try integrating it with llms/rag etc yet?

Have you tried MHTML?

You must have several TB of the internet on disk by now...

This a good suggestion with the caveat that entire domains can and do disappear: https://help.archive.org/help/how-do-i-request-to-remove-som...

recently I've come to believe even IA and especially archive.is are ephermal. I've watched sites I've saved disappear without a trace, except in my selfhosted archives.

A technological conundrum, however, is the fact that I have no way to prove that my archive is an accurate representation of a site at a point in time. Hmmm, or maybe I do? Maybe something funky with cert chains could be done.

> you can configure it to automatically archive every page you visit

What?? I am a heavy user of the Internet Archive services, not just the Wayback Machine, including official and "unofficial" clients and endpoints, and I had absolutely no idea the extension could do this.

To bulk archive I would manually do it via the web interface or batch automate it. The limitations of manually doing it one by one are obvious, and the limitations of doing it in batches requires, well, keeping batches (lists).

It was my solution too, and I liked it, but over the past decade or so, I noticed that even when I remember where to find some stuff, hell, even if I just remember how to find it, when I actually try and find it, it often isn't there anymore. "Search rot" is just as big a problem as link rot.

As for being still alive, by that measure hardly anything anyone does is important in the modern world. It's pretty hard to fail at thinking or remembering so badly that it becomes a life-or-death thing.

I’ve found that whenever I think “why don’t other people just do X” it’s because I’m misunderstanding what’s involved in X for them, and that generally if they could ‘just’ do X then they would.

“Why don’t you just” is a red flag now for me.

I use the Resurrect Pages addon

warc is not a panacea; for example, gemini makes it super annoying to get a transcript of your conversation, so i started saving those as pdf and warc.

turns out that unlike most webpages, the pdf version is only a single page of what is visible on screen.

turns out also that opening the warc immediately triggers a js redirect that is planted in the page. i can still extract the text manually - it’s embedded there - but i cannot “just open” the warc in my browser and expect an offline “archive” version - im interacting with a live webpage! this sucks from all sides - usability, privacy, security.

Admittedly, i don’t use webrecorder - does it solve this problem? did you verify?

You want Zotero.

It's for bibliographies, but it also archives and stores web pages locally with a browser integration.

In theory a content-addressed system like IPFS would be the best: if someone online still has a copy, you can get it too.

These are not just "long URLs". These are URLs where the entire content is stored in the fragment suffix of the URL. They are blobs, and always have been.

They didn't store anything themselves since they encoded the full state in the urls that were given out. So the link shortener was the only place where the "database", the urls, were being stored.

Don't they use https://g.co now? Or are there still new internal goo.gl links created?

Edit: Google is using a g.co link on the "Your device is booting another OS" screen that appears when booting up my Pixel running GrapheneOS. Will be awkward when they kill that service and the hard coded link in the phones bios is just dead

It makes sense to me. After seeing a bunch of AI slop, people started putting no AI buttons and disclaimers. Then some people using AI for little things wanted to clarify it wasn’t AI generated wholesale without falsely claiming AI wasn’t involved at all.

It’s more a claimer than a disclaimer. ;)

This comment was written by a human with no check by any automaton, but how will you check that?

For the author it matters. To which degree do they want to be associated with the resulting text.

And I differentiate between "Matt Godbolt" who is an expert in some areas and in my experience careful about avoiding wrong information and an LLM which may produce additional depth, but may also make up things.

And well, "discern the quality of the contents" - I often read texts to learn new things. On new things I don't have enough knowledge to qualify the statements, but I may have experience with regards to the author or publisher.

Typically services like these are side projects of just a few Google employees, and when the last one leaves they are shut down.

yeah but nobody wants to put "spent two months migrating goo.gl url shortener to work with Sisyphus release manager and Dante 7 SRE monitoring" in their perf packet

that's a negative credit activity

Another possibility is that it's a distraction - whatever the marginal costs, there's a fixed cost to each system in terms of cognitive overhead, if not documentation, legal issues (which can change as laws and regulations change), etc. Removing distractions is basic management.

we'd keep the curiosities around, like so much Ea Nasir Sells Shit Copper. we have room for like 5-10 of those per century. not like 8 billion. much of life is mundane.

I regularly wonder if modern educated people do not journal as much as previous century educated people who were kind of rare.

Maybe we should get a journaling boom going.

But it has to be written, because pen and paper is literally ten times more durable than even good digital storage.