MemGator is a lookup service for various web archives. Think: Here is a URL, is it on THE Internet Archive or archive.today or some other archive that uses the memento standard?
- Github Repo
- How to install
- List of archives that are searched by default
- Paper that decribes MemGator
I found about it one of the talks of the IIPC Wac 2025 conference, where web archiving scene met. Sawood Alam, now at the Internet Archive mentions the tool he had built together with others in the presentation called „Lost, but Preserved – A Web Archiving Perspective on the Ephemeral Web“. His main argument: Link rot is not a huge of problem, because most URLs are rescued.
A slide from the talk to describe links and their possible states aka the „link-rot terminologies“:
Dead links are aroung ten percent, according to different studies:
Back to MemGator and the tool that looks up URLs in archives.
Some example commands
Let’s get started with the most basic command: get the mementos of a specific URL
memgator https://forummuenchen.org/ueber-uns/
<https://forummuenchen.org/ueber-uns/>; rel="original",
<http://localhost:1208/timemap/link/https://forummuenchen.org/ueber-uns/>; rel="self"; type="application/link-format",
<https://web.archive.org/web/20200924105253/https://forummuenchen.org/ueber-uns/>; rel="first memento"; datetime="Thu, 24 Sep 2020 10:52:53 GMT",
<https://web.archive.org/web/20201029001140/https://forummuenchen.org/ueber-uns/>; rel="memento"; datetime="Thu, 29 Oct 2020 00:11:40 GMT",
<https://web.archive.org/web/20201125045547/https://forummuenchen.org/ueber-uns/>; rel="memento"; datetime="Wed, 25 Nov 2020 04:55:47 GMT",
<https://web.archive.org/web/20210115172647/https://forummuenchen.org/ueber-uns/>; rel="memento"; datetime="Fri, 15 Jan 2021 17:26:47 GMT",
<https://web.archive.org/web/20220118230954/https://forummuenchen.org/ueber-uns/>; rel="memento"; datetime="Tue, 18 Jan 2022 23:09:54 GMT",
<https://web.archive.org/web/20220419132644/https://forummuenchen.org/ueber-uns/>; rel="memento"; datetime="Tue, 19 Apr 2022 13:26:44 GMT",
<https://web.archive.org/web/20220614205809/https://forummuenchen.org/ueber-uns/>; rel="memento"; datetime="Tue, 14 Jun 2022 20:58:09 GMT",
<https://web.archive.org/web/20220619085536/https://forummuenchen.org/ueber-uns/>; rel="memento"; datetime="Sun, 19 Jun 2022 08:55:36 GMT",
<https://web.archive.org/web/20220619205601/https://forummuenchen.org/ueber-uns/>; rel="memento"; datetime="Sun, 19 Jun 2022 20:56:01 GMT",
<https://web.archive.org/web/20220619221259/https://forummuenchen.org/ueber-uns/>; rel="memento"; datetime="Sun, 19 Jun 2022 22:12:59 GMT",
<https://web.archive.org/web/20220803200314/https://forummuenchen.org/ueber-uns/>; rel="memento"; datetime="Wed, 03 Aug 2022 20:03:14 GMT",
<https://web.archive.org/web/20220804063544/https://forummuenchen.org/ueber-uns/>; rel="memento"; datetime="Thu, 04 Aug 2022 06:35:44 GMT",
<https://web.archive.org/web/20220804202952/https://forummuenchen.org/ueber-uns/>; rel="memento"; datetime="Thu, 04 Aug 2022 20:29:52 GMT",
<https://web.archive.org/web/20220805222102/https://forummuenchen.org/ueber-uns/>; rel="memento"; datetime="Fri, 05 Aug 2022 22:21:02 GMT",
<https://web.archive.org/web/20220806005024/https://forummuenchen.org/ueber-uns/>; rel="memento"; datetime="Sat, 06 Aug 2022 00:50:24 GMT",
<https://web.archive.org/web/20220807072606/https://forummuenchen.org/ueber-uns/>; rel="memento"; datetime="Sun, 07 Aug 2022 07:26:06 GMT",
<https://web.archive.org/web/20220808175552/https://forummuenchen.org/ueber-uns/>; rel="memento"; datetime="Mon, 08 Aug 2022 17:55:52 GMT",
<https://web.archive.org/web/20220808220607/https://forummuenchen.org/ueber-uns/>; rel="memento"; datetime="Mon, 08 Aug 2022 22:06:07 GMT",
<https://web.archive.org/web/20220810223220/https://forummuenchen.org/ueber-uns/>; rel="memento"; datetime="Wed, 10 Aug 2022 22:32:20 GMT",
<https://web.archive.org/web/20221026202902/https://forummuenchen.org/ueber-uns/>; rel="memento"; datetime="Wed, 26 Oct 2022 20:29:02 GMT",
<https://web.archive.org/web/20230813122914/https://forummuenchen.org/ueber-uns/>; rel="memento"; datetime="Sun, 13 Aug 2023 12:29:14 GMT",
<https://web.archive.org/web/20231119071931/https://forummuenchen.org/ueber-uns/>; rel="memento"; datetime="Sun, 19 Nov 2023 07:19:31 GMT",
<https://web.archive.org/web/20231214075546/https://forummuenchen.org/ueber-uns/>; rel="memento"; datetime="Thu, 14 Dec 2023 07:55:46 GMT",
<https://web.archive.org/web/20231214084202/https://forummuenchen.org/ueber-uns/>; rel="memento"; datetime="Thu, 14 Dec 2023 08:42:02 GMT",
<https://web.archive.org/web/20240130114017/https://forummuenchen.org/ueber-uns/>; rel="memento"; datetime="Tue, 30 Jan 2024 11:40:17 GMT",
<https://web.archive.org/web/20240301105237/https://forummuenchen.org/ueber-uns>; rel="memento"; datetime="Fri, 01 Mar 2024 10:52:37 GMT",
<https://web.archive.org/web/20240918145222/https://forummuenchen.org/ueber-uns>; rel="memento"; datetime="Wed, 18 Sep 2024 14:52:22 GMT",
<https://web.archive.org/web/20241001005024/https://forummuenchen.org/ueber-uns/>; rel="memento"; datetime="Tue, 01 Oct 2024 00:50:24 GMT",
<https://web.archive.org/web/20241101211937/https://forummuenchen.org/ueber-uns/>; rel="memento"; datetime="Fri, 01 Nov 2024 21:19:37 GMT",
<https://web.archive.org/web/20241111231857/https://forummuenchen.org/ueber-uns/>; rel="memento"; datetime="Mon, 11 Nov 2024 23:18:57 GMT",
<https://web.archive.org/web/20241209192519/https://forummuenchen.org/ueber-uns/>; rel="memento"; datetime="Mon, 09 Dec 2024 19:25:19 GMT",
<https://web.archive.org/web/20241214072249/https://forummuenchen.org/ueber-uns/>; rel="memento"; datetime="Sat, 14 Dec 2024 07:22:49 GMT",
<https://web.archive.org/web/20250307221819/https://forummuenchen.org/ueber-uns/>; rel="memento"; datetime="Fri, 07 Mar 2025 22:18:19 GMT",
<https://web.archive.org/web/20250330013644/https://forummuenchen.org/ueber-uns/>; rel="memento"; datetime="Sun, 30 Mar 2025 01:36:44 GMT",
<https://web.archive.org/web/20250429222132/http://forummuenchen.org/ueber-uns/>; rel="last memento"; datetime="Tue, 29 Apr 2025 22:21:32 GMT",
<http://localhost:1208/timemap/link/https://forummuenchen.org/ueber-uns/>; rel="timemap"; type="application/link-format",
<http://localhost:1208/timemap/json/https://forummuenchen.org/ueber-uns/>; rel="timemap"; type="application/json",
<http://localhost:1208/timemap/cdxj/https://forummuenchen.org/ueber-uns/>; rel="timemap"; type="application/cdxj+ors",
<http://localhost:1208/timegate/https://forummuenchen.org/ueber-uns/>; rel="timegate"Get output as a json:
memgator -f json https://forummuenchen.org/ueber-uns/
{
"original_uri": "https://forummuenchen.org/ueber-uns/",
"self": "http://localhost:1208/timemap/json/https://forummuenchen.org/ueber-uns/",
"mementos": {
"list": [
{
"datetime": "2020-09-24T10:52:53Z",
"uri": "https://web.archive.org/web/20200924105253/https://forummuenchen.org/ueber-uns/"
},
{
"datetime": "2020-10-29T00:11:40Z",
"uri": "https://web.archive.org/web/20201029001140/https://forummuenchen.org/ueber-uns/"
},
{
"datetime": "2020-11-25T04:55:47Z",
"uri": "https://web.archive.org/web/20201125045547/https://forummuenchen.org/ueber-uns/"
},
{
"datetime": "2021-01-15T17:26:47Z",
"uri": "https://web.archive.org/web/20210115172647/https://forummuenchen.org/ueber-uns/"
},
{
"datetime": "2022-01-18T23:09:54Z",
"uri": "https://web.archive.org/web/20220118230954/https://forummuenchen.org/ueber-uns/"
},
....Get the memento that is closest to a certain date:
memgator https://forummuenchen.org/ueber-uns/ 20230111
{
"original_uri": "https://forummuenchen.org/ueber-uns/",
"mementos": {
"first": {
"datetime": "2022-10-26T20:29:02Z",
"uri": "https://web.archive.org/web/20221026202902/https://forummuenchen.org/ueber-uns/"
},
"last": {
"datetime": "2022-10-26T20:29:02Z",
"uri": "https://web.archive.org/web/20221026202902/https://forummuenchen.org/ueber-uns/"
},
"closest": {
"datetime": "2022-10-26T20:29:02Z",
"uri": "https://web.archive.org/web/20221026202902/https://forummuenchen.org/ueber-uns/"
}
},
"timemap_uri": {
"link_format": "http://localhost:1208/timemap/link/https://forummuenchen.org/ueber-uns/",
"json_format": "http://localhost:1208/timemap/json/https://forummuenchen.org/ueber-uns/",
"cdxj_format": "http://localhost:1208/timemap/cdxj/https://forummuenchen.org/ueber-uns/"
},
"timegate_uri": "http://localhost:1208/timegate/https://forummuenchen.org/ueber-uns/"
}Get only the closest link by parsing the result with jq
memgator https://forummuenchen.org/ueber-uns/ 20230111 | jq -r '[.mementos.closest.uri, .mementos.first.uri]'
[
"https://web.archive.org/web/20250204023627/https://tirol.orf.at/stories/3291703/",
"https://web.archive.org/web/20250204023627/https://tirol.orf.at/stories/3291703/"
]