Nightmare on Lemmy Street (A Fediverse GDPR Horror Story)

By Michael Altfield, on March 4th, 2024

This article will describe how lemmy instance admins can purge images from pict-rs (click here if you just want to know how).

This is (also) a horror story about accidentally uploading very sensitive data to Lemmy, and the (surprisingly) difficult task of deleting it.

Intro

tl;dr I (accidentally) uploaded a photo of my State-issued ID to Lemmy, and I couldn’t delete it.

Friends don’t let friends compose jerboa comments in bed before coffee (@theyshane)

A few weeks ago I woke up to my 06:00 AM alarm, snoozed my phone, rubbed my eyes, and started reading /c/worldnews (on Lemmy).

Still half-asleep, I was typing a comment when my thumb accidentally hit the “upload media” button. Up popped a gallery of images. I tried to click the back button, but I missed. I tapped on a photo. The photo that I tapped-on was a KYC selfie image (that I took the previous day for a service that has no business having such PII anyway).

That was all it took — two consecutive mis-taps while half-asleep in bed, and my dumb-ass just inadvertently uploaded a KYC selfie onto the public internet. And thanks to archaic State authentication systems, anyone with the URL can now steal my identity.

Like adrenaline to the heart, I jumped out of bed. In such an incident, at least two actions should follow:

1. Delete the credential from the Internet and
2. Revoke & replace the credential.

Thanks to our moronic State authentication systems, #2 is all but impossible. Sigh.

As for #1, I tried to delete the image — but I discovered that the ability to delete images from the Lemmy WUI is a pending feature. Wow.

Yes, Lemmy users cannot delete images that they’ve uploaded. Well, there is an API call that you can make, but you need to first get the image’s “delete token”.

(Update: after writing this, I found another Lemmy bug preventing lemmy users from deleting their own images via the API using the documented DELETE query.)

But this is time-sensitive. What other options do we have? Well, you can delete your whole account, right? In fact, when you go to the page to delete your account, it “assures” you with the following message

screenshot of the Lemmy website with a button that says "Delete Account" and a subsequent message "Warning: this will permanently delete all of your data from this instance. Your data may not be deleted on other, existing instances. Enter your password to confirm."

Lemmy lies. Deleting your account does not delete the photos you’ve uploaded.

Perfect! That’s what I want. I mean, it’s a bit overkill — but it will certainly delete the credential from being publicly-accessible ASAP 🙂

So I delete the account, confirm the user profile and posts are gone, and refresh the image. It’s still there. I clear cache. It’s still there. I launch a new VM with a fresh install of vanilla firefox. Motherfucker, it’s still there!

Yes, Lemmy has a bug where uploaded images are not deleted when the account is deleted. Good lord.

Well, I guess self-help is out the window. Let’s contact the admin. Oh, but I can’t do that unless I have an account. And I just deleted my account.

I create a new account and message the admin. Of course, PMs sent on Lemmy are not secure. In fact, there’s a message at the top of the PM page that says

Screenshot of a Lemmy "/create_private_message" page that says "Warning: Private messages in Lemmy are not secure. Please create an account on Element.io for secure messaging."

Lemmy PMs are not meant to be secure

Unfortunately, the message doesn’t provide the matrix handle for the admin (but I created a feature request for this). At this point the credential leak is publicly accessible, but only with a known URL. If, however, I send the URL insecurely then I draw more attention to it — making the situation worse. Fortunately, after a few hours, I’m able to open an encrypted matrix chat with the instance admin.

Their response:

I can look into how to delete images, I havent actually done that before. Its supposed to be possible using some REST API call but not sure, need to look into it.

The admin tools in lemmy arent very good. :/

-Lemmy Instance Admin

Actually, I wasn’t surprised by this. I’ve never run a Lemmy server instance, but I was googling while waiting for the admins’s response — and I couldn’t find any information on how to purge images in Lemmy. In fact, admins don’t have a WUI (or lemmy API calls) to moderate images on lemmy (only for posts & comments). In lemmy, images are handled by a third party component pict-rs.

How to purge images in Lemmy

pict-rs is a third-party simple image hosting service that runs along-side Lemmy for instances that allow users to upload media.

At the time of writing, there is no WUI for admins to find and delete images. You have to manually query the pict-rs database and execute an API call from the command-line. Worse: Lemmy has no documentation telling instance admins how to delete images 🤦

For the purposes of this example, let’s assume you’re trying to delete the following image

https://monero.town/pictrs/image/001665df-3b25-415f-8a59-3d836bb68dd1.webp

There are two API endpoints in pict-rs that can be used to delete an image

Method One: /image/delete/{delete_token}/{alias}

This API call is publicly-accessible, but it first requires you to obtain the image’s `delete_token`

The `delete_token` is first returned by Lemmy when POSTing to the `/pictrs/image` endpoint

{
   "msg":"ok",
   "files":[
      {
         "file":"001665df-3b25-415f-8a59-3d836bb68dd1.webp",
         "delete_token":"d88b7f32-a56f-4679-bd93-4f334764d381"
      }
   ]
}

Two pieces of information are returned here:

file (aka the “alias”) is the server filename of the uploaded image
delete_token is the token needed to delete the image

Of course, if you didn’t capture this image’s `delete_token` at upload-time, then you must fetch it from the postgres DB.

First, open a shell on your running postgres container. If you installed Lemmy with docker compose, use `docker compose ps` to get the “SERVICE” name of your postgres host, and then enter it with `docker exec`

docker compose ps --format "table {{.Service}}\t{{.Image}}\t{{.Name}}"
docker compose exec <docker_service_name> /bin/bash

For example:

user@host:/home/user/lemmy# docker compose ps --format "table {{.Service}}\t{{.Image}}\t{{.Name}}"
SERVICE    IMAGE                            NAME
lemmy      dessalines/lemmy:0.19.3          lemmy-lemmy-1
lemmy-ui   dessalines/lemmy-ui:0.19.3       lemmy-lemmy-ui-1
pictrs     docker.io/asonix/pictrs:0.5.4    lemmy-pictrs-1
postfix    docker.io/mwader/postfix-relay   lemmy-postfix-1
postgres   docker.io/postgres:15-alpine     lemmy-postgres-1
proxy      docker.io/library/nginx          lemmy-proxy-1
user@host:/home/user/lemmy# 

user@host:/home/user/lemmy# docker compose exec postgres /bin/bash
postgres:/#

Connect to the database as the `lemmy` user

psql -U lemmy

For example

postgres:/# psql -U lemmy
psql (15.5)
Type "help" for help.

lemmy=#

Query for the image by the “alias” (the filename)

select * from image_upload where pictrs_alias = '<image_filename>';

For example

lemmy=# select * from image_upload where pictrs_alias = '001665df-3b25-415f-8a59-3d836bb68dd1.webp';
 local_user_id | pictrs_alias | pictrs_delete_token | published 
---------------+--------------+---------------------+-----------
1149 | 001665df-3b25-415f-8a59-3d836bb68dd1.webp | d88b7f32-a56f-4679-bd93-4f334764d381 | 2024-02-07 11:10:17.158741+00
(1 row)

lemmy=#

Now, take the `pictrs_delete_token` from the above output, and use it to delete the image.

The following command should be able to be run on any computer connected to the internet.

curl -i "https://<instance_domain>/pictrs/image/delete/<pictrs_delete_token>/<image_filename>"

For example:

user@disp9140:~$ curl -i "https://monero.town/pictrs/image/delete/d88b7f32-a56f-4679-bd93-4f334764d381/001665df-3b25-415f-8a59-3d836bb68dd1.webp"

HTTP/2 204 No Content
server: nginx
date: Fri, 09 Feb 2024 15:37:48 GMT
vary: Origin, Access-Control-Request-Method, Access-Control-Request-Headers
cache-control: private
referrer-policy: same-origin
x-content-type-options: nosniff
x-frame-options: DENY
x-xss-protection: 1; mode=block
X-Firefox-Spdy: h2
user@disp9140:~$

ⓘ Note: If you get an `incorrect_login` error, then try [a] logging into the instance in your web browser and then [b] pasting the “https://<instance_domain>/pictrs/image/delete/<pictrs_delete_token>/<image_filename>” URL into your web browser.

The image should be deleted.

Method Two: /internal/purge?alias={alias}

Alternatively, you could execute the deletion directly inside the pictrs container. This eliminates the need to fetch the `delete_token`.

First, open a shell on your running `pictrs` container. If you installed Lemmy with docker compose, use `docker compose ps` to get the “SERVICE” name of your postgres host, and then enter it with `docker exec`

docker compose ps --format "table {{.Service}}\t{{.Image}}\t{{.Name}}"
docker compose exec <docker_service_name> /bin/sh

For example:

user@host:/home/user/lemmy# docker compose ps --format "table {{.Service}}\t{{.Image}}\t{{.Name}}"
SERVICE    IMAGE                            NAME
lemmy      dessalines/lemmy:0.19.3          lemmy-lemmy-1
lemmy-ui   dessalines/lemmy-ui:0.19.3       lemmy-lemmy-ui-1
pictrs     docker.io/asonix/pictrs:0.5.4    lemmy-pictrs-1
postfix    docker.io/mwader/postfix-relay   lemmy-postfix-1
postgres   docker.io/postgres:15-alpine     lemmy-postgres-1
proxy      docker.io/library/nginx          lemmy-proxy-1
user@host:/home/user/lemmy# 

user@host:/home/user/lemmy# docker compose exec pictrs /bin/sh
~ $

Execute the following command inside the `pictrs` container.

wget --server-response --post-data "" --header "X-Api-Token: ${PICTRS__SERVER__API_KEY}" "http://127.0.0.1:8080/internal/purge?alias=<image_filename>"

For example:

~ $ wget --server-response --post-data "" --header "X-Api-Token: ${PICTRS__SERVER__API_KEY}" "http://127.0.0.1:8080/internal/purge?alias=001665df-3b25-415f-8a59-3d836bb68dd1.webp"
Connecting to 127.0.0.1:8080 (127.0.0.1:8080)
HTTP/1.1 200 OK
content-length: 67
connection: close
content-type: application/json
date: Wed, 14 Feb 2024 12:56:24 GMT

saving to 'purge?alias=001665df-3b25-415f-8a59-3d836bb68dd1.webp'
purge?alias=001665df 100% |*****************************************************************************************************************************************************************************************************************************| 67 0:00:00 ETA
'purge?alias=001665df-3b25-415f-8a59-3d836bb68dd1.webp' saved

~ $

ⓘ Note: There’s an error in the pict-rs reference documentation. It says you can POST to `/internal/delete`, but that just returns 404 Not Found.

The image should be deleted

Enumeration of Bad Things™

When accidents happen, usually there’s multiple things that went wrong.

1. I fucked up

Admittedly, I made some mistakes here.

Don’t use Lemmy in bed

I should probably not have been using Lemmy when I was half-asleep. Good luck adhering to that policy, self.

Device Compartmentalization

I should own a second device for taking pictures of super-sensitive documents like State-issued IDs. I should never copy such sensitive files onto my everyday phone.

Unfortunately, I only own one phone. And it’s a bit unrealistic to expect most people to have more than one phone (though I do hope one day we’ll have a secure way to run dozens of distinct Android VMs on our laptops)

Data Hygiene

Immediately after I take a photo of sensitive data, I should transfer it into encrypted storage and then securely wipe the storage on which it was taken.

2. Lemmy fucked up

But there’s also many bugs in Lemmy that allowed this nightmare to realize.

lemmy users should be able to easily delete photos that they’ve uploaded from the WUI. I created a feature request for this.
lemmy users should at least be able to delete their photos via the API. I created a bug report for this. See also this related ticket.
To facilitate GDPR “Right to Erasure” requests, lemmy admins should also be given an easy-to-use WUI for finding & purging images uploaded by their users. I created a feature request for this.
In the meantime, lemmy should document how lemmy admins can purge photos via the CLI. I created a feature request for this.
And, to contact lemmy admins, the private_message_disclaimer shouldn’t just tell lemmy users to use matrix — it should tell them the matrix handle of their instance’s admin. I created a feature request for this.
Also, deleting a user account should also delete all of the user’s data (as it says it will), including the photos that the user uploaded. I created a bug report for this.
lemmy users should be able to easily delete photos that they’ve uploaded from Jerboa App. I created a feature request for this.
jerboa should add a confirmation step to the UX before uploading files to a lemmy instance. I created a feature request for this.
finally, jerboa should add a setting to never upload any files. I created a feature request for this.

3. Android fucked up

Screenshot of the jerboa app with the "media upload" icon highlighted, a red arrow pointed to it, and the text "DANGER!" under the arrow

If you click this button, you’re one mis-tap away from a world of pain

AOSP needs more granular permission settings.

In this instance, Android says that the phone has “No permissions requested.” It doesn’t, for example, have access to my system’s storage.

If I could deny apps from being able to initiate an ACTION_GET_CONTENT Intent, then I could completely prevent apps from spawning a file-picker — thereby reducing the risk of accidental information leakage.

4. The archaic state of State Auth (in 2024)

Photo of an Estonian eID card with the name BARACK HUSSEIN OBAMA written on it

Obama was issued an eID card by 🇪🇪 Estonia in 2014. What about US? 🍌 Thanks Obama.

The current moronic solution for authentication by State-issued ID is to just give the document to someone else to read. In this design, the card itself (and a picture of the card) serves as the user’s private key. There are glaring flaws with this design:

A picture of the ID is a private key
You give your private key to others
Authentication sessions are vulnerable to replay attacks

These problems are easily solved by implementing a State-issued eID system using public key cryptography.

Ideally, the eID system would allow users to easily revoke & replace their (sub)keys, and allow them to make (encrypted) software backups of their eID.

5. KYC must follow PoLP

Due to the issues above, we should very rarely be handing out KYC selfies.

Businesses have no reason to collect KYC data, except to avoid money laundering. For the purposes of AML, it’s unreasonable to track transactions lower than $10,000. Now, this number should be increased for inflation — but there’s a reason you don’t declare currency when you’re moving less than $10,000 in cash.

Similarly, if I’m paying an artist $600 on Upwork, there’s zero reason for me to risk identity theft by uploading my secret auth PII to your platform, which is hardly regulated to handle such data with the care it deserves.

Asking for KYC data from customers who transact <$10,000 is a clear violation of the Principle of Least Privilege, and we need consumer protection and data protection laws that prohibit companies from discriminating against customers who protect themselves by not providing this information to companies that don’t need it.

Why This Matters

Line chart showing an increase of spending on GDPR fines from Jul 2018 to 4.5 billion EUR on Jan 2024

GDPR Violation fines are often millions of EUR or a percent of revenue — whichever is higher

Exposing basic moderation tools (eg POST /delete-this-thing) to the WUI is neither a bell nor a whistle. It’s a basic component that should be added before shipping.

If admins can’t click to delete an image, but users can click to upload an image, then that feature should not be released.

This isn’t just annoying — it’s of significant moral, legal, and financial concern.

This is a legal risk to Lemmy instance admins (GDPR violation). This law applies to any website operating anywhere in the world (not just to websites or businesses located in the EU) that has users who are residents of the EU (so it likely affects >90% of public Lemmy instances with >100 active users).

The fines for this violation are commonly millions of euros or a percent of revenue, whichever is higher. At the time of writing, websites have been fined 98.4 million EUR for violating this class of GDPR violation (Insufficient fulfillment of data subjects rights).

Making the endpoint accessible over an API is not trivial and therefore doesn’t satisfy this requirement. Google received one fine of 150 million EUR in 2022 simply for making it more than 1-click to reject cookies. Certainly requiring a user to figure out how to make API calls (including figuring out the darn delete token, which makes even seasoned admins scratch their heads) would not satisfy the GDPR’s “Right to Erasure” requirement.