Arjen Wiersma

Self hosting is a lot of fun. From running your own services to checking that backups work. There are, however, people that make it a sport to try and login on the systems you build for yourself. There is a great tool that you can use to detect and block these actors; fail2ban.

For fail2ban to work with custom services (it already has a lot built-in) you need to create some filters. You place them in /etc/fail2ban/filter.d in a file with a descriptive name, such as forgejo.conf or freshrss.conf.

Filters for Forgejo

Of all my services, forgejo is the most attacked one. It makes sense, it is well known and has excellent documentation. To block anybody that tries to log into my instance I created a couple of filter rules.

[Definition]
failregex = ^.*Failed authentication attempt for .* from <HOST>
            ^.*Invalid user .+ from <HOST> port .*
            ^.*User .+ from <HOST> not allowed because not listed in AllowUsers

Then I created a /etc/fail2ban/jail.d file called forgejo.conf. It contains the instructions what to do with the above filter.

[forgejo]
enabled = true
filter = forgejo
logpath = /var/lib/docker/containers/**/*-json.log
maxretry = 4
bantime = 14400
findtime = 14400
chain = DOCKER-USER
ignoreip = 127.0.0.1/8 10.0.0.0/8 172.27.0.0/16 192.168.1.0/24

As i am running everything as docker containers I am looking at the output of the containers, the *-json.log files. I tell it which filter to use, my aptly named forgejo filter, and finally I tell it to which iptables chain the ban should be added. When running things using docker, you will run into the fact that it will try to control everything, including networking. The DOCKER-USER list is there to add custom rules.

Be sure to add your own IP address to the ignoreip list so that you do not get blocked yourself.

FreshRSS

The same setup goes for FreshRSS. The filter is a little simpler:

[Definition]
failregex = ^{"log":"<HOST> .+\" 403 \d+ .*$

And the jail itself looks the same:

[freshrss]
enabled = true
filter = freshrss
logpath = /var/lib/docker/containers/**/*-json.log
maxretry = 4
bantime = 14400
findtime = 14400
chain = DOCKER-USER
ignoreip = 127.0.0.1/8 10.0.0.0/8 172.27.0.0/16 192.168.1.0/24

Default jails

On my ubuntu instance I also enable the SSH jail, which is done through the defaults jail.

[sshd]
enabled = true

Checking that everything works

You can use the logfile /var/log/fail2ban.log to check that everything works as expected. Also check out iptables -L to see all the IPs being added to the REJECT list.

Sometimes you want to check that your filter works, especially after upgrading versions or when you found a new way you wanted to block someone. The tool fail2ban-regex is very useful here:

fail2ban-regex /var/lib/docker/containers/4131da65deefeb601933cf58e2a8498872a0416192354e3411e447c6857bc384/4131da65deefeb601933cf58e2a8498872a0416192354e3411e447c6857bc384-json.log /etc/fail2ban/filter.d/forgejo.conf

Of course you will need to change the path to the docker container log file.

Self-hosting is a fun and interesting way to learn about the technologies you use. One thing you have to take care of is to have backups of important systems. I had to upgrade forgejo from v13 to v14, but you want to be sure that works before doing it, right?

How backups are made

On my devbox I have a daily backup that creates an off-site backup of the volumes.

#!/bin/bash

# Variables
BACKUP_DIR=/mnt/backups-devbox
FOLDERS_TO_BACKUP=("forgejo")

if [ ! -d $BACKUP_DIR ]; then
  echo "Backup dir ($BACKUP_DIR) does not exist, creating."
  mkdir $BACKUP_DIR
fi

if [ ! -e $BACKUP_DIR/.canary ]; then
  echo "Canary file not found, not doing backups!"
  exit 1
fi

echo "Creating backups of PostgreSQL"

DB_USER=forgejo
CONTAINER_NAME=db

# Get current date and time for backup file
TIMESTAMP=$(date +"%F_%T")
BACKUP_FILE_FORGEJO=$BACKUP_DIR/backup_forgejo_$DB_NAME_$TIMESTAMP.sql

# Run pg_dump inside the PostgreSQL container
docker exec -t $CONTAINER_NAME pg_dump -U $DB_USER forgejo >$BACKUP_FILE_FORGEJO &&
  echo "Backup completed: $BACKUP_FILE_FORGEJO"

echo "Creating compressed tarballs of specified folders"

for FOLDER in "${FOLDERS_TO_BACKUP[@]}"; do
  if [ -d "$FOLDER" ]; then
    # Create a filename-friendly string from the path (e.g., /etc/nginx -> etc_nginx)
    FOLDER_NAME=$(echo "$FOLDER" | sed 's/^\///; s/\//_/g')
    TAR_NAME="$BACKUP_DIR/folder_${FOLDER_NAME}_$TIMESTAMP.tar.gz"
    
    echo "Backing up $FOLDER to $TAR_NAME..."
    tar -czf "$TAR_NAME" "$FOLDER"
  else
    echo "Warning: Folder $FOLDER not found, skipping."
  fi
done

echo "Backups done"

This is of course all very well, but this backup needs to be tested.

An untested backup is no backup.

Running a test setup

I am running Forgejo as a docker container on my devbox. The actual configuration is done with ansible and it sits nicely behind a reverse proxy, but for testing purposes locally this docker-compose file builds the same setup:

services:
  db:
    image: postgres:16
    container_name: db
    restart: always
    ports:
      - "5432:5432"
    environment:
      USER_UID: "1000"
      USER_GID: "1000"
      POSTGRES_USER: "forgejo"
      POSTGRES_PASSWORD: "forgejo"
      POSTGRES_DB: "forgejo"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready", "-d", "forgejo"]
      interval: 30s
      timeout: 60s
      retries: 5
      start_period: 80s
    volumes:
      - ./postgres:/var/lib/postgresql/data
    networks:
      - forgejo

  forgejo:
    image: codeberg.org/forgejo/forgejo:14
    container_name: forgejo
    restart: always
    ports:
      - "3000:3000"
      - "22:22"
    environment:
      USER_UID: "1000"
      USER_GID: "1000"
      FORGEJO__database__DB_TYPE: "postgres"
      FORGEJO__database__HOST: "db:5432"
      FORGEJO__database__NAME: "forgejo"
      FORGEJO__database__USER: "forgejo"
      FORGEJO__database__PASSWD: "forgejo"
    volumes:
      - ./forgejo:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    networks:
      - forgejo

networks:
  forgejo:
    name: forgejo

Restoring postgres

So, first I only run the postgres container so that I can re-create the database. The restore is most easily done by copying the dump into the image and restoring it:

docker cp backup_forgejo_2026-SOMEDATE.sql db:/tmp/dump.sql
docker exec -it db psql -U forgejo -d forgejo -f /tmp/dump.sql

Restoring forgejo

Then it is time to recreate the forgejo directory from the backup:

tar zxvf folder_forgejo_2026-SOMEDATE.tar.gz

Remember to run these commands in the directory of the docker-compose file.

Finally ensure everything is fine by running the forgejo docter:

docker exec -u git -ti forgejo bash
forgejo doctor check --all

When I bought my new phone (a Fairphone 6) I received a discount for Storytel. I never really gave listening to books a try, simply because I like reading, but since I had it for free I thought I would give it a try. For my current assignment I commute to The Hague 2 days a week, a trip that takes 1 ½ hours door-to-door. Plenty of time to listen to a book.

Surprisingly it has been quite nice. I am now on my 3rd book, having already listened to the following Dutch books:

• Je bent wat je doet by Ray Klaassens; a book about taking ownership in your life/work/everything
• Nu of nooit by Dai Carter; a book on mental strength and tools how to stay the course in difficult times
• Explosief; a short story by writing collective Crime Squad

I am currently listening to:

• Ewout: Avonturen en inzichten van een eigenzinnige programmamaker; he shares stories about the various dangerous tv programmes he has made.

While listening to these books I do get the sense that I pick-up on the major points being told, but I do think the nuances (things your read twice or more times) tend to get lost. I am going to stick with it for a little while longer, the main issue I have though is the lack of English books in the library. I like the original version of most books better then the translation, so we will see how long it sticks.

On my e-reader I am also reading a book, The Secret of secrets by Dan Brown. This book takes a bit longer as I only make the time to read when I am going to bed, but still it is very enjoyable.

So, today I did something radical. In an effort to be more conscious with my technology I switched to a self-hosted website (before it was hosted on github). I also switched to Write Freely in an attempt to write more and worry less.

The blog requires some TLC with all the old Hugo shortcodes, but I will updated it going forward for my most important posts.

TLDR: I created an interactive chart of editors, see it in my repository.

Today the news dropped that Jetbrains is cancelling its “new” editor, Fleet. Not that I use the product, but it got me thinking about editors (again). I love editors, specifically ones that I can change and bend to my will, such as Emacs.

In my career I have used many different editors. I started on a Laser VTech 310 in Basic 2.0 and from there I transitioned into the world of Pascal and C/C++ to end up on the JVM with Java and Clojure. During that time I have seen many editors come and go in the work field.

Reminiscing about the editors that I have seen made me think about creating a sort of family tree of editors, starting from where I started and then adding editors from the different domain in which I have been active. It kinda got out of hand, just check out this picture:

I started with my trusty Emacs and its friend (neo)vi(m). Alongside staples such as IntelliJ IDEA and VS Code. Using some LLM magic and Wikipedia knowledge I started assembling lineage of the various products and influence of different editors on eachother. For instance, did you know that Lighttable introduced some of the web based editing concepts we now see in Electron? Or that Visual Age was rebuilt into Eclipse, from which modern day editors take a lot of lessons?

I added several neat features to the graph. First you can hover on nodes and vertexes to show you some more information on the editor or its connection.

Each node can also be clicked, this will lock the popup and will allow you to click the link(s) in the popup. I want to collect some personal stories for different editors, or perhaps link to some form of codified history on the particular editor.

You might also wonder about the big blobs on the graph. While researching I found several names to pop up in a lot of different editors, Anders, Erich, Molenaar and the good people of Jetbrains, so I mapped out their influence in the editor landscape. I found the influence that Anders Hejlsberg has had extremely interesting, it basically traces from the start of the code editor (Turbo Pascal, Delphi) all the way to VS Code (Typescript).

If you have any corrections or editor/concepts/stories to add, drop me a line on Mastodon.

#programming

Day 5 of solving {{< backlink “aoc” “The Advent of Code” >}} in {{< backlink “clojure” “Clojure” >}}.

It was a hectic day, and it is not even over yet. I had multiple trips to the vet while trying to get some work done. To relax a bit I worked on the puzzle of the day. In hindsight I was way too cautious with my solution, but then again, it is to solve a puzzle :D

The first thing to figure out is that a list of numbers is in a collection of ranges. Reading in the file and parsing it to numbers is something that has been done the entire week already, then it is a matter of taking the list of ingredients and mapping over the inventory, filtering out the ones that are not in range.

For part 2 the question becomes how many numbers are in the ranges, and this is a dead give away for something that is so huge that you can not just brute force it. The ranges can overlap, so first thing is to merge adjacent or overlapping ranges. My solution is a bit convoluted, but it works. Then it is a matter of getting the size of each range and adding it up.

(ns day5
  (:require
   [clojure.string :as str]))

(defn parse-input [input]
  (let [[available ingredients]
        (-> input
            slurp
            (str/split #"\n\n"))
        inventory (mapv parse-long (str/split-lines ingredients))
        cleaned (->> available
                     str/split-lines
                     (map #(str/split % #"-"))
                     (mapv (partial mapv #(-> % parse-long))))]
    [cleaned inventory]))

(defn part1 []
  (let [[inventory ingredients] (parse-input "resources/5.in")]
    (->> ingredients
         (map
          #(filter (fn [[min max]]
                     (if (and (>= % min) (<= % max))
                       %
                       nil)) inventory))
         (filter seq)
         count)))

(defn should-merge? [[_ end] [start _]]
  (<= start (inc end)))

(defn merge-two-ranges [[start1 end1] [start2 end2]]
  [(min start1 start2) (max end1 end2)])

(defn merge-overlapping [first-range remaining]
  (let [overlapping (filter (partial should-merge? first-range) remaining)
        merged (reduce merge-two-ranges first-range overlapping)
        unmerged (remove (set overlapping) remaining)]
    [merged unmerged]))

(defn merge-pass [ranges]
  (if (empty? ranges)
    []
    (let [[first-range & remaining] ranges
          [merged remaining-unmerged] (merge-overlapping first-range remaining)]
      (cons merged (merge-pass remaining-unmerged)))))

(defn merge-ranges [ranges]
  (loop [current ranges
         previous nil]
    (let [next-merge (merge-pass current)]
      (if (= next-merge previous)
        next-merge
        (recur next-merge current)))))

(->> "resources/5.in"
     parse-input
     first
     merge-ranges
     (map (fn [[start end]] (- (inc end) start)))
     (reduce +))

#programming

Day 4 of solving {{< backlink “aoc” “The Advent of Code” >}} in {{< backlink “clojure” “Clojure” >}}.

The first grid problem of the season! The first part was really suspiciously easy, read in the grid, find all the rolls and look at its 8 neighbors, eliminating it if there are less than 4. The 2nd part was a nice progression on this.

Instead of just saying how many should be eliminated, we do it until there are no eliminations possible. This is highly reminiscent of a Conway Game of Life puzzle. In Clojure this is quite nicely done by reading the grid as a vector, and then ranging over all the coordinates. On each coordinate just take a look at the neighbors and apply the logic. Its pure nature means that the functions are already working in the right way to do this repeatedly.

(ns day4
  (:require
   [clojure.string :as str]))

(defn get-cell [grid [row col]]
  (when (and (>= row 0) (< row (count grid))
             (>= col 0) (< col (count (first grid))))
    (get-in grid [row col])))

(defn neighbors-8 [grid [row col] & {:keys [filter-fn] :or {filter-fn (constantly true)}}]
  (->> (for [r (range (dec row) (+ row 2))
             c (range (dec col) (+ col 2))
             :when (not= [r c] [row col])]
         [r c])
       (map #(get-cell grid %))
       (remove nil?)
       (filter filter-fn)))

(defn all-coords [grid]
  (for [row (range (count grid))
        col (range (count (first grid)))]
    [row col]))

(defn print-grid-with-marks [grid marked-positions]
  (doseq [row (range (count grid))]
    (doseq [col (range (count (first grid)))]
      (let [pos [row col]
            cell (get-in grid pos)]
        (print (if (some #(= % pos) marked-positions)
                 "X "
                 (str cell " ")))))
    (println)))

(defn mark-movable [grid]
  (let [all-pos (all-coords grid)
        marks (remove nil? (map (fn [pos]
                                  (let [n (neighbors-8 grid pos :filter-fn #(not= % \.))]
                                    (when (and (= \@ (get-cell grid pos)) (< (count n) 4))
                                      pos)))
                                all-pos))]
    marks))

(defn remove-marked [grid marked-positions]
  (let [rows (count grid)
        cols (count (first grid))]
    (vec
     (for [row (range rows)]
       (vec
        (for [col (range cols)]
          (let [pos [row col]
                cell (get-in grid pos)]
            (if (and (= cell \@) (some #(= % pos) marked-positions))
              \.
              cell))))))))

(def play
  (fn [grid]
    (loop [g grid
           moved 0]
      (let [marks (mark-movable g)
            mark-count (count marks)]
        (if (= 0 mark-count)
          moved
          (recur (remove-marked g marks)
                 (+ moved mark-count)))))))

(def grid
  (->> "resources/4.in"
       slurp
       str/split-lines
       (mapv vec)))

;; part 1
(count (mark-movable grid))

;; part 2
(play grid)

Really enjoyed it and well within my daily commute to solve.

#programming

Day 3 of solving {{< backlink “aoc” “The Advent of Code” >}} in {{< backlink “clojure” “Clojure” >}}.

The puzzle today has us figuring out the joltage of a bank of batteries. This is a puzzle of the type “largest sequence in a list”. When you start you already know that part 2 will be something that a naive approach will not be able to handle, but I still did part 1 with a naive combination function.

The {{< backlink “clojure” “Clojure” >}} combinatorics library will not work here, as it will take a look at the distinct pairs that will be in the list. So in a list of (2 8 2) it will find the unique pair of (2 8), resulting in the integer 28 for the puzzle, but it will fail to see that 82 was also an option.

I wrote a simple combinations function to start with, and then puzzle 2 hit me with a combination of 12 digits. The algorithm did not work for that amount. I changed it to search for the largest number n positions from the end. So, first search the highest number at least 12 positions away from the end, from then search for the highest number at least 11 positions from the end, etc. The algorithm works beautifully well.

Original: 12397658
First pass (2 positions remaining): ___97658
First pass (1 positions remaining): ___9___8
Result: 98

While writing this I am wondering if it would be even better to convert the string to numbers before finding the digits, but I will leave that as an exercise to you, dear reader.

(ns day3
  (:require
   [clojure.string :as str]))

(defn find-highest-digit
  "Find the highest digit on to the end of the string.
   This is an optimization to not interpolate the entire string."
  [s start end]
  (loop [max-digit nil
         max-pos nil
         pos start]
    (cond
      (> pos end) ;; break out the loop with the current highest number
      {:digit max-digit :pos max-pos}

      (or (nil? max-digit) (> (Character/digit (nth s pos) 10) max-digit))
      (recur (Character/digit (nth s pos) 10) pos (inc pos))

      :else
      (recur max-digit max-pos (inc pos)))))

(defn find-max-number
  "Find the max number of `remaining-length` size. Keep finding
   the max number at least `remaining-length` from the end."
  [s remaining-length]
  (let [n (count s)]
    (when (and (>= n 1) (>= remaining-length 1))
      (let [end-pos (- n remaining-length)
            {:keys [digit pos]} (find-highest-digit s 0 end-pos)]
        (if (nil? digit)
          nil
          (let [remaining-s (if (< (inc pos) n) (subs s (inc pos)) "")]
            (cons digit (find-max-number remaining-s (dec remaining-length)))))))))

(defn max-n-digit-number
  "Find the maximum number in a sequence of number, allowing for
   dropping intermediate numbers"
  [n s]
  (let [result (find-max-number s n)]
    (Long/parseLong (apply str result))))

(->> "resources/3.in"
     slurp
     (str/split-lines)
     (map #(max-n-digit-number 2 %))
     (reduce +))

(->> "resources/3.in"
     slurp
     (str/split-lines)
     (map #(max-n-digit-number 12 %))
     (reduce +))

#programming

Day 2 of solving {{< backlink “aoc” “The Advent of Code” >}} in {{< backlink “clojure” “Clojure” >}}.

The 2nd day had us do some magic with numbers. The quest was to find some silly patterns in the numbers. I chose to use good old regular expressions to do the job, while a colleague of mine chose to use math. Both work really well. The 2nd part of the puzzle is to find repeating patterns, where the 1st only wanted pairs.

(ns day2
  (:require
   [clojure.string :as str]))

(def parse-int Long/parseLong)

(defn parse-ranges [input]
  (map (fn [in]
         (let [[_ start end] (first (re-seq #"(\d+)-(\d+)" in))]
           (filter #(re-matches #"(\d+)\1" (str %))
              (range (parse-int start) (inc (parse-int end))))))
       input))

(defn parse-ranges2 [input]
  (map (fn [in]
         (let [[_ start end] (first (re-seq #"(\d+)-(\d+)" in))]
           (filter #(re-matches #"(\d+)(\1+)" (str %)) 
               (range (parse-int start) (inc (parse-int end))))))
       input))

(def part1 (reduce +
            (->
            (slurp "resources/two.in")
            (str/trim)
            (str/split #",")
            (parse-ranges)
            (flatten))))

(def part2 (reduce + (->
            (slurp "resources/two.in")
            (str/trim)
            (str/split #",")
            (parse-ranges2)
            (flatten))))

(prn part1 part2)

#programming

Day 1 of solving {{< backlink “aoc” “The Advent of Code” >}} in {{< backlink “clojure” “Clojure” >}}.

Each year I like to participate in the Advent of Code. This year there will be 12 puzzles to solve due to Eric taking care of himself, good on you!

My language this year is {{< backlink “clojure” “Clojure” >}}. My solution is quite simple and straightforward. I read the data using split-instructions, transforming the L into a subtraction. For part 1 it is enough to then take the reductions and then filter out all the times we land on the digit 0.

For part 2 I just generate the sequence of numbers for each step in all-steps and then using reduce to apply it to each step, just like the reductions. The result is a long list of numbers from which I just take the 0s again.

(ns one
  (:require
   [clojure.string :as str]))

(defn split-instruction [instruction]
  (let [[_ dir dist] (re-matches #"([A-Z])(\d+)" instruction)
        steps (Integer/parseInt dist)]
    (cond
      (= dir "L") (* steps -1)
      (= dir "R") steps)))

(defn part1 []
  (let [data (->> (slurp "resources/one.in")
                  (str/split-lines)
                  (map split-instruction))
        steps (reductions #(mod (+ %1 %2) 100) 50 data)]
    (count (filter #(= 0 %) steps))))

(part1)

(defn all-steps [start change]
  (let [step      (if (pos? change) 1 -1)
        seq-start (if (pos? change) (inc start) (dec start))
        seq-end   (+ start change step)]
    (map #(mod % 100) (range seq-start seq-end step))))

(defn trace-all-steps [start-pos changes]
  (let [result (reduce (fn [acc change]
                         (let [current-pos (:pos acc)
                               steps (all-steps current-pos change)
                               new-pos (last steps)]
                           {:pos new-pos :history (into (:history acc) steps)}))
                       {:pos start-pos :history []}
                       changes)]
    (:history result)))

(defn part2 []
  (let [data (->> (slurp "resources/one.in")
                  (str/split-lines)
                  (map split-instruction))
        steps (trace-all-steps 50 data)]
    (count (filter #(= 0 %) steps))))

(part2)

#programming