Archivists work to save disappearing data.gov datasets

In terms of tooling there's scoop[0] which does a lot of the capture part of what you're thinking about. The files it creates include request headers and responses, TLS certificates, PDF and screenshots and it has support for signing the whole thing as proof of provenance.

Overall though I think archive.org is probably sufficient proof that a specific page had certain content on a certain day for most purposes today.

0. https://github.com/harvard-lil/scoop

The idea is good, as far as I understand TLS however, the cert / asymmetric key is only used prove the identity/authenticity of the cert and thus the host for this session.

But the main content is not signed / checksummed with it, but rather a symmetrical session key, so one could probably manipulate this in the packet dump anyway.

I read about a Google project named SXG (Signed HTTP exchanges) that might do related stuff, albeit likely requiring the assistance of the publisher

Unfortunately, the standard TLS protocol does not provide a non-repudiation mechanism.

It works by using public key cryptography and key agreement to get both parties to agree on a symmetric key, and then uses the symmetric key to encrypt the actual session data.

Any party who knows the symmetric key can forge arbitrary data, and so a transcript of a TLS session, coupled with the symmetric key, is not proof of provenance.

There are interactive protocols that use multi-party computation (see for example https://tlsnotary.org/) where there are two parties on the client side, plus an unmodified server. tlsnotary only works for TLS1.2. One party controls and can see the content, but neither party has direct access to the symmetric key. At the end, the second party can, by virtue of interactively being part of the protocol, provably know a hash of the transaction. If the second party is a trusted third party, they could sign a certificate.

However, there is not a non-interactive version of the same protocol - you either need to have been in the loop when the data was archived, or trust someone who was.

The trusted third party can be a program running in a trusted execution environment (but note pretty much all current TEEs have known fault injection flaws), or in a cloud provider that offers vTPM attestation and a certificate for the state (e.g. Google signs a certificate saying an endorsement key is authentically from Google, and the vTPM signs a certificate saying a particular key is restricted to the vTPM and only available when the compute instance is running particular known binary code, and that key is used to sign a certificate attesting to a TLS transcript).

I'm working on a simpler solution that doesn't use multiparty computation, and provides cloud attestation - https://lemmy.amxl.com/c/project_uniquonym https://github.com/uniquonym/tls-attestproxy - but it's not usable yet.

Another solution is if the server will cooperate with a TLS extension. TLS-N (https://eprint.iacr.org/2017/578.pdf) provides a solution for this. That provides a trivial solution for provenance.

You may be interested in Reclaim Protocol and perhaps zkTLS. They have something very similar going and the sources are free.

https://github.com/reclaimprotocol

https://drive.google.com/file/d/1wmfdtIGPaN9uJBI1DHqN903tP9c...

https://www.reclaimprotocol.org/

https://docs.lighthouse.storage/lighthouse-1/zktls

It’s an interesting idea for sure. Some drawbacks I can think off:

- bigger resource usage. You will need to maintain a dump of the TLS session AND an easily extractable version

- difficulty of verification. OpenSSL / BoringSSL / etc. will all evolve and say, completely remove support for TLS versions, ciphers, TLS extensions… This might make many dumps unreadable in the future, or requiring the exact same version of a given software to read it. Perhaps adding the decoding binary to the dump would help, but then, you’d get Linux retro-compatibility issues.

- compression issues: new compression algorithms will be discovered and could reduce data usage. You’ll have a hard time doing that since TLS streams will look random to the compression software.

I don’t know. I feel like it’s a bit overkill — what are the incentives for tampering with this kind of data?

Maybe a simpler way of going about it would be to build a separate system that does the « certification » after the data is dumped; combined with multiple orgs actually dumping the data (reproducibility), this should be enough the prove that a dataset is really what it claims to be.

What about https://opentimestamps.org/ ?

How does that work exactly? Does it all still hinge on trusting a know Time Stamp Authority, or is there some way of time stamping in a trustless manner?

As a library, the very high level prioritization framework is "what would patrons find useful." That's how we started with data.gov and federal Github repos as broad but principled collections; there's likely to be something in there that's useful and gets lost. Going forward I think we'll be looking for patron stories along the lines of "if you could get this couple of TB of stuff it would cover the core of what my research field depends on."

In practice it's some mix of, there aren't already lots of copies, it's valuable to people, and it's achievable to preserve.

> Is the second question about figuring out how to prioritize valuable stuff behind two depth traversals?

Right -- how do you look at the 300,000 entries and figure out what's not at depth one, is archivable, and is worth preserving? If we started with everything it would be petabytes of raw datasets that probably shouldn't be at the top of the list.

Hmm, I can put them here for now: https://source.coop/harvard-lil/data-gov-metadata

Unfortunately it's a bit messy because we weren't initially thinking about tracking deletions. data_20241119.jsonl.zip (301k rows) and data_20250130.jsonl.zip (305k rows) are simple captures of the API on those dates. data_db_dump_20250130.jsonl.zip (311k rows) is a sqlite dump of all the entries we saw at some point between those dates. My hunch is there's something like 4,000 false positives and 2,000 deletions between the 311k and 305k set, but that could be way off.

I spent a bunch of time on this project feeling like it was futile to jump in and then just jumped in; messing with data is fun even if it turns out someone else has your data. But the government is huge; if you find an interesting report and then poke around for the .gov data catalog or directory index structure or whatever that contains it, you're likely to find a data gathering approach no one else is working on yet.

There's coordinated efforts starting to come together in a bunch of places -- some on r/datahoarders, some around specific topics like climate data (EDGI) or CDC data, there's datasets being posted on archive.org. I think one way is to find a topic or kind of data that seems important and search around for who's already doing it. Eventually maybe there'll be one answer to rule them all, but maybe not; it's just so big.

China as of late is indeed running circles around the US. We spend 3 decades depending on them for manufacturing, and surprise. They got really good at manufacturing stuff for themselves when the chips are down.

I guess great economies also give it away. RIP USA.

> Why can’t Russia and China build and innovate like the USA can? > Because they spend a lot of time censoring and covering things up.

What a statement! How did you come to this conclusion?

Yeah, what has avoiding another plague ever done for the USA.

[flagged]

It's not in any sort of format to do this kind of analysis unfortunately. I'm also missing some data b/c I throw away certain kinds of datasets that are not useful for me. I can probably write some scripts to diff my archives with the current data.gov and see what's missing, but it won't be "complete". But it might still be useful...

I did however just write a Python script to pull data.gov from archive.org and check the dataset count on the front page for all of 2024, here are the results:

https://postimg.cc/1V0WYtRt

As you can see, there were multiple drops on the order of ~10,000 during 2024. So it's not that unusual. There could be something bad going on right now, but just from the numbers I can't conclude that yet.

(Specifically it takes the first snapshot of every Wednesday of 2024).

If I get around to re-formatting my archives this week, I'll follow up on HN :).

The President's ability to affect spending is definitely limited, and hasn't been exercised really since Reagan, but still exists.

Congress rarely makes spending money it's goal, rather it appropriates money to accomplish some goal. Which is to say that if Congress wants a bridge across a river and appropriate 10 billion to build it, the President is not obligated to spend $10 billion if 7 or 8 or 9 will do. In some cases, Congress does appropriate money toward causes and intends all the money to be spent in furtherance of some guiding principle and in these cases all the money must be spent.

I really don't think "Trump can't do it, if it's illegal" is going to fly this time. I hope so, but I'm not very confident.

Of course, he can't get away with too much alone, but with the right appointees, judges, etc, who knows.

I don't get why there's not legal action then? Maybe its a matter of nobody having standing?

[flagged]

These assholes.

Nah, the whole executive branch is getting Jack Welch'ed. Hopefully your tap water cleanliness regulations are strong on a state level.

I think the question is the nature of the losses in the two cases, the transparency circumstances about them, and who exactly is making the decisions about specific datasets.

Time will tell but loss of public datasets is probably not usually good in general.

This is not a direct quote, the actual quote from the article is

> But archivists who have been working on analyzing the deletions and archiving the data it held say that while some of the deletions are surely malicious information scrubbing, some are likely routine artifacts of an administration change, and they are working to determine which is which. For example, in the days after Joe Biden was inaugurated, data.gov showed about 1,000 datasets being deleted as compared to a day before his inauguration, according to the Wayback Machine.

Why are the datasets deleted though? Biden or Trump, Democrats or Republicans - What do they gain?

And you could also run your own archive bot (x86 only). I've got one running in a docker container, it downloads a webpage and auto-uploads it to archive.org

https://tracker.archiveteam.org/

Edit to add:

docker_compose.yml example:

services:

  archiveteam:   

    image: atdr.meo.ws/archiveteam/warrior-dockerfile   

    ports:  

      - '8101:8001'   

    mem_limit: 4G   

    cpus: 3


    dns:   
      - 9.9.9.10   
      - 8.8.8.8   


    labels:   
      - com.centurylinklabs.watchtower.enable=true   
    container_name: archiveteam-warrior   
    environment:   
      - DOWNLOADER=asdf # Change this to your nickname   
      - SELECTED_PROJECT=auto # Change this to your project of preference or let the archiveteam decide with 'auto'
      - CONCURRENT_ITEMS=6 # Change this to the amount of concurrent download threads you can handle



  watchtower:
    command: '--label-enable --include-restarting --cleanup --interval 3600'
    cpu_shares: 128
    mem_limit: 1G
    cpus: 1

    image: containrrr/watchtower
    volumes:
       - '/var/run/docker.sock:/var/run/docker.sock'

    container_name: watchtower