You are not taking the load off with a single user, but you will have a more stable experience since your instance won’t require much resources. I have my own instance and haven’t experienced any of the issues people report on lemmy.world and lemmy.ml.
It’s not great, but here it is: https://github.com/ducksdotdev/LemmySubscriber/tree/main … I’ll revisit the app sometime this week/weekend to make it better. Although, I believe there are a couple other scripts in the wild that do the same task and include their own method for crawling.
edit: https://github.com/Fmstrat/lcs – looks like this is a better one than what I did if you want something different.
I can share it, but right now I have it grabbing the latest artifact from the lemmy-exporter which looks like as of June 3rd they stopped updating their repo with the export artifact. Meaning, I need to implement some sort of crawler myself or use a different method for generating the list of communities.
I can share the code with this caveat if you want to use it anyway/try and change it yourself. It is written in Golang… will post back in a bit with a github repo. need to set it up
You won’t get much old data when you first start subscribing to new communities from your instance. You’ll only get new content. When you add a community you’ll get 20 recent topics with no comments or upvotes. As new content comes in you’ll pick up all of it. At least one member of your instance needs to subscribe to a community to begin syncing with that community. To solve the lack of content on my personal instance, I made a bot that subscribed to the top 300 communities with more than 1000 subscribers. After a day or so I now have tons of content… I may stick the bot script on a weekly cronjob to keep my instance synced with the largest communities.
It’s pretty lightweight. I’ve given each container 1/3Gi of memory and 1CPU limit with low requests. Utilizing kubernetes HPA to scale containers under load up to 4 replicas. It only scales when a user takes large actions (subscribing to hundreds of new to the instance communities at once). But once the initial federation begins it seems to quickly scale back down. The biggest bottleneck is pictrs since it is stateful.
So far the database and pictrs is only about 2Gi of storage but I’ve allocated 25Gi to each since I have a lot to spare at the moment.
I have to play with the HPA more since I’m not happy yet with my settings. I have 2 users and 1 bot on my instance.
I’d like to start contributing to Lemmy’s codebase so I wanted to host my own instance to learn the inner workings.
My postgres is a single replica at the moment but I may scale that if stability is becomes a problem.
k3s with rancher. I was using k8s before but redid everything. K3s is overkill for what I do an causes millions of headaches but I enjoy learning through brute force.
I use k8s at work so it’s good experience to run my own k3s
Yes this was unclear to me until I set up my own instance is may be a big deal breaker to the average user. If you can’t view “All” and really see all communities on all federated instances then there is no ability to surf Lemmy. At the very least it should have all communities of instances from either a list or user subscriptions.
edit: if you can share your script that would be amazing, I was thinking of having to do the same thing…
I’m honestly surprised that Plex has revenue in the “double-digit millions”
Great to hear, that will make it super easy if I start allowing users on my instance.
Pihole Keycloak Lemmy
The “usual” Plex stack:
Plex Sonarr Radarr Readarr Calibre & Calibre-Web Sabnzbd Nzbhydra
I want to throw Nextcloud into the mix, but I haven’t gotten the motivation to do that yet. I have 102TB of disk on a 4 node kubernetes cluster just for fun
You mean pictrs can’t scale, or the other pods cannot as well? I separated lemmy-ui, the backend, and pictrs into different pods. Haven’t tried scaling anything yet though, but I did notice the database issue with pictrs when RollingRestart, had to switch to Recreate.
Also on kubernetes, hopefully this message works. First time testing from my self-hosted instance.
This might not be a popular opinion, but I had a very short period of time between graduating school and starting work where I needed to learn docker and I found this course: https://serversforhackers.com/shipping-docker to be very helpful, they had a sale and I got the course and it taught me everything I needed in a few days.
You absolutely don’t need a course to learn it, but sometimes these can be easy shortcuts.