initial sofurry impl

- Added more directories to `.cursorignore` to exclude temporary and log files
- Updated TODO.md with a new task for downloading E621 user favorites
- Modified Dockerfile.devcontainer to install PostgreSQL 17 client
- Updated VS Code extensions script to add Docker extension
- Added Dockerfile formatter in VS Code settings
- Configured Cursor as the default Git editor in the development container

.cursorignore

@@ -5,28 +5,3 @@ tmp
 log
 public
 .bundle
-gems
-
-# Generated/build artifacts
-node_modules
-user_scripts/dist
-app/assets/builds
-vendor/javascript
-
-# Sorbet generated files
-sorbet/tapioca
-sorbet/rbi/gems
-sorbet/rbi/annotations
-sorbet/rbi/dsl
-
-# Configuration files with secrets
-config/credentials.yml.enc
-config/master.key
-
-# Lock files
-yarn.lock
-Gemfile.lock
-
-# Documentation
-TODO.md
-*.notes.md

.cursorrules

@@ -1,278 +0,0 @@
-# How to use this codebase
-
-- Run `bin/tapioca dsl` after changing a model or concern.
-- Run `bin/tapioca gems` after changing the Gemfile.
-- Run `srb tc` after making changes to Ruby files to ensure the codebase is typechecked.
-- Run `bin/rspec <path_to_spec_file>` after a spec file is modified.
-- Run `tapioca dsl` if models or concerns are modified.
-- Run `bin/rspec <path_to_spec_file>` to run tests for a single file.
-- There are no view-specific tests, so if a view changes then run the controller tests instead.
-- For instance, if you modify `app/models/domain/post.rb`, run `bin/rspec spec/models/domain/post_spec.rb`. If you modify `app/views/domain/users/index.html.erb`, run `bin/rspec spec/controllers/domain/users_controller_spec.rb`.
-- At the end of a long series of changes, run `just test`.
-- If specs are failing, then fix the failures, and rerun with `bin/rspec <path_to_spec_file>`.
-- If you need to add logging to a Job to debug it, set `quiet: false` on the spec you are debugging.
-- Fish shell is used for development, not bash.
-- When running scratch commands, use `bin/rails runner`, not `bin/rails console`.
-
-# Typescript Development
-
-- React is the only frontend framework used.
-- Styling is done with Tailwind CSS and FontAwesome.
-- Put new typescript files in `app/javascript/bundles/Main/components/`
-
-# HTTP Mocking in Job Specs
-
-When writing specs for jobs that make HTTP requests, use `HttpClientMockHelpers.init_with()` instead of manually creating doubles:
-
-```ruby
-# CORRECT: Use HttpClientMockHelpers.init_with
-let(:client_mock_config) do
-  [
-    {
-      uri: "https://example.com/api/first-endpoint",
-      status_code: 200,
-      content_type: "application/json",
-      contents: first_response_body,
-    },
-    {
-      uri: "https://example.com/api/second-endpoint",
-      status_code: 200,
-      content_type: "application/json",
-      contents: second_response_body,
-      caused_by_entry: :any, # Use this for chained requests
-    },
-  ]
-end
-
-before { @log_entries = HttpClientMockHelpers.init_with(client_mock_config) }
-
-# WRONG: Don't create doubles manually
-expect(http_client_mock).to receive(:get).and_return(
-  double(status_code: 200, body: response_body, log_entry: double),
-)
-
-# WRONG: Don't use the old init_http_client_mock method
-@log_entries =
-  HttpClientMockHelpers.init_http_client_mock(
-    http_client_mock,
-    client_mock_config,
-  )
-```
-
-This pattern:
-
-- Uses the preferred `init_with` helper method
-- Automatically uses the global `http_client_mock` from `spec_helper.rb`
-- Creates real HttpLogEntry objects that can be serialized by ActiveJob
-- Follows the established codebase pattern
-- Avoids "Unsupported argument type: RSpec::Mocks::Double" errors
-- Use `caused_by_entry: :any` for HTTP requests that are chained (where one request's log entry becomes the `caused_by_entry` for the next request)
-- No need to manually set up `http_client_mock` - it's handled globally in `spec_helper.rb`
-
-# Job Enqueuing Verification in Specs
-
-Use `SpecUtil.enqueued_job_args()` instead of mocking `perform_later`:
-
-```ruby
-# CORRECT: Test actual job enqueuing
-enqueued_jobs = SpecUtil.enqueued_job_args(SomeJob)
-expect(enqueued_jobs).to contain_exactly(hash_including(user: user))
-expect(enqueued_jobs).to be_empty # For no jobs
-
-# WRONG: Don't mock perform_later (breaks with .set chaining)
-expect(SomeJob).to receive(:perform_later)
-```
-
-Benefits: More robust, tests actual behavior, no cleanup needed (tests run in transactions).
-
-# Testing Jobs
-
-When writing specs for jobs e.g. Domain::Site::SomethingJob, do not invoke `job.perform(...)` directly, always use `perform_now(...)` (defined in spec/helpers/perform_job_helpers.rb)
-
-# === BACKLOG.MD GUIDELINES START ===
-
-# Instructions for the usage of Backlog.md CLI Tool
-
-## 1. Source of Truth
-
-- Tasks live under **`backlog/tasks/`** (drafts under **`backlog/drafts/`**).
-- Every implementation decision starts with reading the corresponding Markdown task file.
-- Project documentation is in **`backlog/docs/`**.
-- Project decisions are in **`backlog/decisions/`**.
-
-## 2. Defining Tasks
-
-### **Title**
-
-Use a clear brief title that summarizes the task.
-
-### **Description**: (The **"why"**)
-
-Provide a concise summary of the task purpose and its goal. Do not add implementation details here. It
-should explain the purpose and context of the task. Code snippets should be avoided.
-
-### **Acceptance Criteria**: (The **"what"**)
-
-List specific, measurable outcomes that define what means to reach the goal from the description. Use checkboxes (`- [ ]`) for tracking.
-When defining `## Acceptance Criteria` for a task, focus on **outcomes, behaviors, and verifiable requirements** rather
-than step-by-step implementation details.
-Acceptance Criteria (AC) define _what_ conditions must be met for the task to be considered complete.
-They should be testable and confirm that the core purpose of the task is achieved.
-**Key Principles for Good ACs:**
-
-- **Outcome-Oriented:** Focus on the result, not the method.
-- **Testable/Verifiable:** Each criterion should be something that can be objectively tested or verified.
-- **Clear and Concise:** Unambiguous language.
-- **Complete:** Collectively, ACs should cover the scope of the task.
-- **User-Focused (where applicable):** Frame ACs from the perspective of the end-user or the system's external behavior.
-  - _Good Example:_ "- [ ] User can successfully log in with valid credentials."
-  - _Good Example:_ "- [ ] System processes 1000 requests per second without errors."
-  - _Bad Example (Implementation Step):_ "- [ ] Add a new function `handleLogin()` in `auth.ts`."
-
-### Task file
-
-Once a task is created it will be stored in `backlog/tasks/` directory as a Markdown file with the format
-`task-<id> - <title>.md` (e.g. `task-42 - Add GraphQL resolver.md`).
-
-### Additional task requirements
-
-- Tasks must be **atomic** and **testable**. If a task is too large, break it down into smaller subtasks.
-  Each task should represent a single unit of work that can be completed in a single PR.
-
-- **Never** reference tasks that are to be done in the future or that are not yet created. You can only reference
-  previous
-  tasks (id < current task id).
-
-- When creating multiple tasks, ensure they are **independent** and they do not depend on future tasks.
-  Example of wrong tasks splitting: task 1: "Add API endpoint for user data", task 2: "Define the user model and DB
-  schema".
-  Example of correct tasks splitting: task 1: "Add system for handling API requests", task 2: "Add user model and DB
-  schema", task 3: "Add API endpoint for user data".
-
-## 3. Recommended Task Anatomy
-
-```markdown
-# task‑42 - Add GraphQL resolver
-
-## Description (the why)
-
-Short, imperative explanation of the goal of the task and why it is needed.
-
-## Acceptance Criteria (the what)
-
-- [ ] Resolver returns correct data for happy path
-- [ ] Error response matches REST
-- [ ] P95 latency ≤ 50 ms under 100 RPS
-
-## Implementation Plan (the how)
-
-1. Research existing GraphQL resolver patterns
-2. Implement basic resolver with error handling
-3. Add performance monitoring
-4. Write unit and integration tests
-5. Benchmark performance under load
-
-## Implementation Notes (only added after working on the task)
-
-- Approach taken
-- Features implemented or modified
-- Technical decisions and trade-offs
-- Modified or added files
-```
-
-## 6. Implementing Tasks
-
-Mandatory sections for every task:
-
-- **Implementation Plan**: (The **"how"**) Outline the steps to achieve the task. Because the implementation details may
-  change after the task is created, **the implementation notes must be added only after putting the task in progress**
-  and before starting working on the task.
-- **Implementation Notes**: Document your approach, decisions, challenges, and any deviations from the plan. This
-  section is added after you are done working on the task. It should summarize what you did and why you did it. Keep it
-  concise but informative.
-
-**IMPORTANT**: Do not implement anything else that deviates from the **Acceptance Criteria**. If you need to
-implement something that is not in the AC, update the AC first and then implement it or create a new task for it.
-
-## 2. Typical Workflow
-
-```bash
-# 1 Identify work
-backlog task list -s "To Do" --plain
-
-# 2 Read details & documentation
-backlog task 42 --plain
-# Read also all documentation files in `backlog/docs/` directory.
-# Read also all decision files in `backlog/decisions/` directory.
-
-# 3 Start work: assign yourself & move column
-backlog task edit 42 -a @{yourself} -s "In Progress"
-
-# 4 Add implementation plan before starting
-backlog task edit 42 --plan "1. Analyze current implementation\n2. Identify bottlenecks\n3. Refactor in phases"
-
-# 5 Break work down if needed by creating subtasks or additional tasks
-backlog task create "Refactor DB layer" -p 42 -a @{yourself} -d "Description" --ac "Tests pass,Performance improved"
-
-# 6 Complete and mark Done
-backlog task edit 42 -s Done --notes "Implemented GraphQL resolver with error handling and performance monitoring"
-```
-
-### 7. Final Steps Before Marking a Task as Done
-
-Always ensure you have:
-
-1. ✅ Marked all acceptance criteria as completed (change `- [ ]` to `- [x]`)
-2. ✅ Added an `## Implementation Notes` section documenting your approach
-3. ✅ Run all tests and linting checks
-4. ✅ Updated relevant documentation
-
-## 8. Definition of Done (DoD)
-
-A task is **Done** only when **ALL** of the following are complete:
-
-1. **Acceptance criteria** checklist in the task file is fully checked (all `- [ ]` changed to `- [x]`).
-2. **Implementation plan** was followed or deviations were documented in Implementation Notes.
-3. **Automated tests** (unit + integration) cover new logic.
-4. **Static analysis**: linter & formatter succeed.
-5. **Documentation**:
-   - All relevant docs updated (any relevant README file, backlog/docs, backlog/decisions, etc.).
-   - Task file **MUST** have an `## Implementation Notes` section added summarising:
-     - Approach taken
-     - Features implemented or modified
-     - Technical decisions and trade-offs
-     - Modified or added files
-6. **Review**: self review code.
-7. **Task hygiene**: status set to **Done** via CLI (`backlog task edit <id> -s Done`).
-8. **No regressions**: performance, security and licence checks green.
-
-⚠️ **IMPORTANT**: Never mark a task as Done without completing ALL items above.
-
-## 9. Handy CLI Commands
-
-| Purpose          | Command                                                                |
-| ---------------- | ---------------------------------------------------------------------- |
-| Create task      | `backlog task create "Add OAuth"`                                      |
-| Create with desc | `backlog task create "Feature" -d "Enables users to use this feature"` |
-| Create with AC   | `backlog task create "Feature" --ac "Must work,Must be tested"`        |
-| Create with deps | `backlog task create "Feature" --dep task-1,task-2`                    |
-| Create sub task  | `backlog task create -p 14 "Add Google auth"`                          |
-| List tasks       | `backlog task list --plain`                                            |
-| View detail      | `backlog task 7 --plain`                                               |
-| Edit             | `backlog task edit 7 -a @{yourself} -l auth,backend`                   |
-| Add plan         | `backlog task edit 7 --plan "Implementation approach"`                 |
-| Add AC           | `backlog task edit 7 --ac "New criterion,Another one"`                 |
-| Add deps         | `backlog task edit 7 --dep task-1,task-2`                              |
-| Add notes        | `backlog task edit 7 --notes "We added this and that feature because"` |
-| Mark as done     | `backlog task edit 7 -s "Done"`                                        |
-| Archive          | `backlog task archive 7`                                               |
-| Draft flow       | `backlog draft create "Spike GraphQL"` → `backlog draft promote 3.1`   |
-| Demote to draft  | `backlog task demote <task-id>`                                        |
-
-## 10. Tips for AI Agents
-
-- **Always use `--plain` flag** when listing or viewing tasks for AI-friendly text output instead of using Backlog.md
-  interactive UI.
-- When users mention to create a task, they mean to create a task using Backlog.md CLI tool.
-
-# === BACKLOG.MD GUIDELINES END ===

.devcontainer/Dockerfile.devcontainer

@@ -1,22 +1,38 @@
-# Primary image
-FROM mcr.microsoft.com/devcontainers/base:debian-12
-
-# apt caching & install packages
-RUN rm -f /etc/apt/apt.conf.d/docker-clean && \
+FROM ruby:3.2.0 AS native-gems
+RUN rm -f /etc/apt/apt.conf.d/docker-clean; \
     echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/keep-cache
-
-RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
+RUN \
+    --mount=type=cache,target=/var/cache/apt,sharing=locked \
     --mount=type=cache,target=/var/lib/apt,sharing=locked \
     apt-get update && \
-    apt-get install --no-install-recommends --no-install-suggests -qqy \
-    abiword \
+    apt-get install --no-install-recommends --no-install-suggests -y \
+    cmake
+
+WORKDIR /usr/src/app
+RUN gem install bundler -v '2.4.5'
+COPY gems gems
+WORKDIR /usr/src/app/gems/xdiff-rb
+RUN bundle install
+RUN rake compile
+WORKDIR /usr/src/app/gems/rb-bsdiff
+RUN bundle install
+RUN rake compile
+
+# Primary image
+FROM mcr.microsoft.com/devcontainers/ruby:1-3.2-bookworm
+
+# apt caching & install packages
+RUN rm -f /etc/apt/apt.conf.d/docker-clean; \
+    echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/keep-cache
+RUN \
+    --mount=type=cache,target=/var/cache/apt,sharing=locked \
+    --mount=type=cache,target=/var/lib/apt,sharing=locked \
+    apt-get update && \
+    apt-get install --no-install-recommends --no-install-suggests -y \
     autoconf \
     build-essential \
     ca-certificates \
     curl \
-    ffmpeg \
-    ffmpegthumbnailer \
-    file \
     gnupg \
     iputils-ping \
     libblas-dev \
@@ -25,82 +41,27 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
     libgdbm-dev \
     libgdbm6 \
     libgmp-dev \
-    libicu-dev \
     liblapack-dev \
     libncurses5-dev \
     libpq-dev \
     libreadline6-dev \
-    libreoffice \
-    libsqlite3-dev \
     libssl-dev \
+    libvips42 \
     libyaml-dev \
     patch \
-    pdftohtml \
-    pkg-config \
     rustc \
     uuid-dev \
-    watchman \
-    zlib1g-dev
+    zlib1g-dev \
+    watchman
 
-# Install vips dependencies
-RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
-    --mount=type=cache,target=/var/lib/apt,sharing=locked \
-    apt-get update && \
-    apt-get install --no-install-recommends --no-install-suggests -qqy \
-    automake \
-    gtk-doc-tools \
-    gobject-introspection \
-    libgirepository1.0-dev \
-    libglib2.0-dev \
-    libexpat1-dev \
-    libjpeg-dev \
-    libpng-dev \
-    libtiff5-dev \
-    libwebp-dev \
-    libheif-dev \
-    libexif-dev \
-    liblcms2-dev \
-    libxml2-dev \
-    libfftw3-dev \
-    liborc-0.4-dev \
-    libcgif-dev \
-    libjxl-dev \
-    libopenjp2-7-dev \
-    meson \
-    ninja-build
-
-
-# Install imagemagick from source
-RUN cd /tmp && \
-    wget -qO- https://imagemagick.org/archive/releases/ImageMagick-7.1.2-1.tar.xz | tar -xJ && \
-    cd ImageMagick-7.1.2-1 && \
-    ./configure && \
-    make -j$(nproc) && \
-    make install && \
-    ldconfig && \
-    cd / && \
-    rm -rf /tmp/ImageMagick-7.1.2-1*
-
-# Install vips from source
-RUN cd /tmp && \
-    wget -qO- https://github.com/libvips/libvips/releases/download/v8.17.1/vips-8.17.1.tar.xz | tar -xJ && \
-    cd vips-8.17.1 && \
-    meson setup build --prefix=/usr/local -Dcgif=enabled && \
-    cd build && \
-    ninja && \
-    ninja install && \
-    ldconfig && \
-    cd / && \
-    rm -rf /tmp/vips-8.17.1*
-
-# Install postgres 15 client
+# Install postgres 17 client
 RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
     --mount=type=cache,target=/var/lib/apt,sharing=locked \
     sudo install -d /usr/share/postgresql-common/pgdg && \
     curl -o /usr/share/postgresql-common/pgdg/apt.postgresql.org.asc --fail https://www.postgresql.org/media/keys/ACCC4CF8.asc && \
     sh -c 'echo "deb [signed-by=/usr/share/postgresql-common/pgdg/apt.postgresql.org.asc] https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list' && \
-    sudo apt update && \
-    sudo apt-get install --no-install-recommends --no-install-suggests -qqy \
+    apt update && \
+    apt-get install --no-install-recommends --no-install-suggests -y \
     postgresql-client-17
 
 # Install & configure delta diff tool
@@ -110,38 +71,29 @@ RUN wget -O- https://github.com/dandavison/delta/releases/download/0.18.2/git-de
 
 RUN git config --system core.pager "delta" && \
     git config --system interactive.diffFilter "delta --color-only" && \
-    git config --system delta.navigate "true" && \
-    git config --system delta.dark "true" && \
-    git config --system delta.side-by-side "true" && \
+    git config --system delta.navigate true && \
+    git config --system delta.dark true && \
+    git config --system delta.side-by-side true && \
     git config --system merge.conflictstyle "zdiff3" && \
-    git config --system core.editor "cursor --wait" && \
-    git config --system diff.algorithm "histogram" && \
-    git config --system diff.colorMoved "default"
+    git config --system core.editor "cursor --wait"
 
-# Install ruby
-USER vscode
-RUN git clone https://github.com/rbenv/rbenv.git ~/.rbenv
-ENV PATH="/home/vscode/.rbenv/bin:/home/vscode/.rbenv/shims:$PATH"
-RUN echo 'eval "$(rbenv init - --no-rehash bash)"' >> ~/.bashrc
-RUN git clone https://github.com/rbenv/ruby-build.git "$(rbenv root)"/plugins/ruby-build
-RUN rbenv install 3.4.4 && \
-    rbenv global 3.4.4
+# Install native gems
+COPY --from=native-gems /usr/src/app/gems/xdiff-rb /gems/xdiff-rb
+COPY --from=native-gems /usr/src/app/gems/rb-bsdiff /gems/rb-bsdiff
 
 ENV RAILS_ENV development
 
-# Pre install gems to speed up container startup
-USER root
-RUN mkdir -p /tmp/bundle-install-cache && \
-    chown -R vscode:vscode /tmp/bundle-install-cache
-WORKDIR /tmp/bundle-install-cache
-USER vscode
-COPY Gemfile.lock Gemfile ./
-COPY gems/has_aux_table ./gems/has_aux_table
-RUN BUNDLE_FROZEN=true MAKE="make -j$(nproc)" bundle install --jobs $(nproc)
+# [Optional] Uncomment this line to install additional gems.
+RUN su vscode -c "gem install bundler -v '2.5.6'" && \
+    su vscode -c "gem install rake -v '13.0.6'" && \
+    su vscode -c "gem install ruby-lsp -v '0.22.1'"
 
 # install exo
-RUN curl -sL https://exo.deref.io/install | bash
+RUN su vscode -c "curl -sL https://exo.deref.io/install | bash"
 ENV PATH "/home/vscode/.exo/bin:$PATH"
 
 # install just (command runner)
-RUN curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | sudo bash -s -- --to /usr/local/bin
+RUN curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | bash -s -- --to /usr/local/bin
+
+RUN su vscode -c "source /usr/local/share/nvm/nvm.sh && nvm install 18 && nvm use 18 && npm install -g yarn" 2>&1
+ENV PATH /usr/local/share/nvm/current/bin:$PATH

.devcontainer/Dockerfile.postgres

@@ -4,4 +4,5 @@ RUN apt-get update && apt-get install -y \
     postgresql-17-pgvector \
     && rm -rf /var/lib/apt/lists/*
 
-RUN echo "CREATE EXTENSION vector;" >> /docker-entrypoint-initdb.d/01-vector.sql
+COPY create-tablespaces.bash /docker-entrypoint-initdb.d/00-create-tablespaces.bash
+RUN echo "CREATE EXTENSION pgvector;" >> /docker-entrypoint-initdb.d/01-pgvector.sql

.devcontainer/create-tablespaces.bash

@@ -0,0 +1,9 @@
+#!/bin/bash -ex
+
+mkdir -p /tablespaces/mirai
+chown postgres:postgres /tablespaces/mirai
+chmod 750 /tablespaces/mirai
+psql -v ON_ERROR_STOP=1 \
+    --username "$POSTGRES_USER" \
+    --dbname "$POSTGRES_DB" \
+    -c "CREATE TABLESPACE mirai LOCATION '/tablespaces/mirai'"

.devcontainer/devcontainer.json

@@ -7,28 +7,7 @@
   "workspaceFolder": "/workspaces/${localWorkspaceFolderBasename}",
   "features": {
     "ghcr.io/meaningful-ooo/devcontainer-features/fish:1": {},
-    "ghcr.io/nikobockerman/devcontainer-features/fish-persistent-data:2": {},
-    "ghcr.io/devcontainers-extra/features/npm-package:1": {
-      "package": "backlog.md"
-    }
-  },
-  "customizations": {
-    "vscode": {
-      "extensions": [
-        "Shopify.ruby-extensions-pack",
-        "dbaeumer.vscode-eslint",
-        "aliariff.vscode-erb-beautify",
-        "bradlc.vscode-tailwindcss",
-        "KoichiSasada.vscode-rdbg",
-        "qwtel.sqlite-viewer",
-        "esbenp.prettier-vscode",
-        "ms-azuretools.vscode-docker",
-        "1YiB.rust-bundle",
-        "rust-lang.rust-analyzer",
-        "saoudrizwan.claude-dev",
-        "ritwickdey.LiveServer"
-      ]
-    }
+    "ghcr.io/nikobockerman/devcontainer-features/fish-persistent-data:2": {}
   },
   // Use 'forwardPorts' to make a list of ports inside the container available locally.
   // This can be used to network with other containers or the host.
@@ -37,8 +16,8 @@
   // "postCreateCommand": "bundle install && rake db:setup",
   "postCreateCommand": ".devcontainer/post-create.sh",
   "forwardPorts": [
-    3000, // rails
-    3001, // thrust
+    3000, // rails development
+    3001, // rails staging
     9394, // prometheus exporter
     "pgadmin:8080", // pgadmin
     "grafana:3100", // grafana

.devcontainer/docker-compose.yml

@@ -23,7 +23,8 @@ services:
       dockerfile: Dockerfile.postgres
     restart: unless-stopped
     volumes:
-      - postgres-17-data:/var/lib/postgresql/data
+      - postgres-data:/var/lib/postgresql/data
+      - postgres-data-tablespaces:/tablespaces
       - ./create-db-user.sql:/docker-entrypoint-initdb.d/create-db-user.sql
     environment:
       POSTGRES_USER: postgres
@@ -31,7 +32,7 @@ services:
       POSTGRES_PASSWORD: postgres
 
   pgadmin:
-    image: dpage/pgadmin4:9
+    image: dpage/pgadmin4:8.14.0
     restart: unless-stopped
     environment:
       PGADMIN_DEFAULT_EMAIL: admin@example.com
@@ -64,52 +65,10 @@ services:
     volumes:
       - devcontainer-redux-grafana-data:/var/lib/grafana
 
-  airvpn-netherlands-proxy:
-    image: qmcgaw/gluetun
-    cap_add:
-      - NET_ADMIN
-    devices:
-      - /dev/net/tun:/dev/net/tun
-    environment:
-      - HTTPPROXY=on
-      - SHADOWSOCKS=on
-      - HTTPPROXY_LOG=on
-      - VPN_SERVICE_PROVIDER=airvpn
-      - VPN_TYPE=wireguard
-      - WIREGUARD_PRIVATE_KEY=INLA6x1gUVLRPKcCBgRmfpJBCXhOpyq3SvRd5EvCE08=
-      - WIREGUARD_PRESHARED_KEY=DR6CBW9yG5y+D+qpo8TZCizo5WKOooC/UFBdWk6lGEg=
-      - WIREGUARD_ADDRESSES=10.165.87.232,fd7d:76ee:e68f:a993:4d1b:a77a:b471:a606
-      - SERVER_COUNTRIES=Netherlands
-
-  airvpn-san-jose-proxy:
-    image: qmcgaw/gluetun
-    cap_add:
-      - NET_ADMIN
-    devices:
-      - /dev/net/tun:/dev/net/tun
-    environment:
-      - HTTPPROXY=on
-      - SHADOWSOCKS=on
-      - HTTPPROXY_LOG=on
-      - VPN_SERVICE_PROVIDER=airvpn
-      - VPN_TYPE=wireguard
-      - WIREGUARD_PRIVATE_KEY=INLA6x1gUVLRPKcCBgRmfpJBCXhOpyq3SvRd5EvCE08=
-      - WIREGUARD_PRESHARED_KEY=DR6CBW9yG5y+D+qpo8TZCizo5WKOooC/UFBdWk6lGEg=
-      - WIREGUARD_ADDRESSES=10.165.87.232/32,fd7d:76ee:e68f:a993:4d1b:a77a:b471:a606/128
-      - SERVER_CITIES="San Jose California, Fremont California"
-
-  tor:
-    image: dockurr/tor
-    volumes:
-      - devcontainer-redux-tor-config:/etc/tor
-      - devcontainer-redux-tor-data:/var/lib/tor
-    restart: always
-
 volumes:
-  postgres-17-data:
+  postgres-data:
+  postgres-data-tablespaces:
   devcontainer-redux-gem-cache:
   devcontainer-redux-blob-files:
   devcontainer-redux-grafana-data:
   devcontainer-redux-prometheus-data:
-  devcontainer-redux-tor-config:
-  devcontainer-redux-tor-data:

.devcontainer/fish-shell-conf-d/bash-in-agent.fish

@@ -1,10 +0,0 @@
-# Agent detection - only activate minimal mode for actual agents
-if test -n "$npm_config_yes"; or test -n "$CI"; or not status --is-interactive
-    set -gx AGENT_MODE true
-else
-    set -gx AGENT_MODE false
-end
-
-if test $AGENT_MODE = true
-    # /usr/bin/bash -l
-end

.devcontainer/fish-shell-conf-d/binstubs.fish

@@ -1 +0,0 @@
-set -gx PATH "/workspaces/redux-scraper/bin" $PATH

.devcontainer/fish-shell-conf-d/rbenv.fish

@@ -1 +0,0 @@
-status --is-interactive; and rbenv init - --no-rehash fish | source

.devcontainer/fish-shell-conf-d/utils.fish

@@ -24,14 +24,4 @@ function blob-files-stats
     set -l files_dir (blob-files-dir || return 1)
     printf "apparent size: %s\n" (du -sh --apparent-size $files_dir)
     printf "actual size:   %s\n" (du -sh $files_dir)
-end
-
-function curl-fa-onion
-    curl \
-    --socks5-hostname tor:9050 \
-    --compressed \
-    -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:128.0) Gecko/20100101 Firefox/128.0" \
-    -H "Accept-Encoding: gzip, deflate" \
-    -H "Connection: keep-alive" \
-    "http://g6jy5jkx466lrqojcngbnksugrcfxsl562bzuikrka5rv7srgguqbjid.onion/$argv[1]"
-end
+end

.devcontainer/install-extensions.sh

@@ -18,5 +18,3 @@ install_extension KoichiSasada.vscode-rdbg
 install_extension qwtel.sqlite-viewer
 install_extension esbenp.prettier-vscode
 install_extension ms-azuretools.vscode-docker
-install_extension 1YiB.rust-bundle
-install_extension rust-lang.rust-analyzer

.dockerignore

@@ -12,78 +12,3 @@ launch.json
 settings.json
 *.export
 .devcontainer
-user_scripts/dist
-backlog
-
-# Test directories (not needed in production)
-spec
-test
-
-# Development and CI/CD files
-.github
-.ruby-lsp
-.aider*
-.cursorignore
-.cursorrules
-.rspec
-.rspec_parallel
-.rubocop.yml
-.prettierrc
-TODO.md
-*.notes.md
-things-to-fix.notes.md
-mf-fitter-commands.notes.md
-.aiderignore
-
-# Sorbet type checking files (not needed in production)
-sorbet
-
-# Storage directory (contains uploaded files/cache)
-storage
-
-# Development database files
-db/*.sqlite3
-db/*.sqlite3-*
-
-# Core dump files
-core
-
-# Yarn/npm cache and lock files that might conflict
-yarn-error.log
-yarn-debug.log*
-.yarn-integrity
-package-lock.json
-
-# OS specific files
-.DS_Store
-Thumbs.db
-
-# Editor specific files
-*.swp
-*.swo
-*~
-
-# Local environment files
-.env.local
-.env.*.local
-
-# Compiled assets (will be rebuilt in Docker)
-public/assets
-public/packs
-public/packs-test
-app/assets/builds/*
-
-# Flame graph files
-flamegraph.svg
-
-# Procfile variants (only need production one)
-Procfile.dev
-Procfile.dev-static
-Procfile.staging
-Procfile.worker
-
-# Development scripts
-justfile
-
-# Documentation
-README.md

.gitignore

@@ -8,14 +8,15 @@ build
 tmp
 core
 *.bundle
-user_scripts/dist
-migrated_files.txt
+lib/xdiff
+ext/xdiff/Makefile
+ext/xdiff/xdiff
 
 # use yarn to manage node_modules
 package-lock.json
 
 *.notes.md
-*.txt
+
 # Ignore bundler config.
 /.bundle
 
@@ -59,4 +60,3 @@ yarn-debug.log*
 .yarn-integrity
 .DS_Store
 *.export
-.aider*

.gitmodules

@@ -1,3 +0,0 @@
-[submodule "gems/has_aux_table"]
-	path = gems/has_aux_table
-	url = ssh://git@git.dy.mk:2221/dymk/has_aux_table.git

.prettierrc

@@ -4,10 +4,6 @@
   "trailingComma": "all",
   "arrowParens": "always",
   "singleQuote": true,
-  "semi": true,
-  "bracketSpacing": true,
-  "bracketSameLine": false,
-  "printWidth": 80,
   "plugins": [
     "prettier-plugin-tailwindcss",
     "@prettier/plugin-ruby",
@@ -15,13 +11,5 @@
     "@4az/prettier-plugin-html-erb"
   ],
   "xmlQuoteAttributes": "double",
-  "xmlWhitespaceSensitivity": "ignore",
-  "overrides": [
-    {
-      "files": ["*.ts", "*.tsx"],
-      "options": {
-        "parser": "typescript"
-      }
-    }
-  ]
+  "xmlWhitespaceSensitivity": "ignore"
 }

.rspec_parallel

@@ -1,2 +0,0 @@
---format progress
---format ParallelTests::RSpec::RuntimeLogger --out tmp/parallel_runtime_rspec.log

.ruby-version

@@ -1 +1 @@
-3.4.4
+system

.vscode/launch.json

@@ -3,8 +3,9 @@
   "configurations": [
     {
       "type": "rdbg",
-      "name": "rdbg - attach",
-      "request": "attach"
+      "name": "Attach rdbg",
+      "request": "attach",
+      "rdbgPath": "export GEM_HOME=/usr/local/rvm/gems/default && bundle exec rdbg"
     }
   ]
 }

.vscode/settings.json

@@ -5,14 +5,6 @@
   "workbench.preferredDarkColorTheme": "Spinel",
   "workbench.preferredLightColorTheme": "Spinel Light",
   "rubyLsp.formatter": "syntax_tree",
-  "rubyLsp.featureFlags": {
-    "fullTestDiscovery": true
-  },
-  "rubyLsp.addonSettings": {
-    "Ruby LSP RSpec": {
-      "debug": true
-    }
-  },
   "files.associations": {
     ".env-cmdrc": "json"
   },
@@ -20,7 +12,7 @@
     "editor.defaultFormatter": "esbenp.prettier-vscode"
   },
   "[erb]": {
-    "editor.defaultFormatter": "aliariff.vscode-erb-beautify"
+    "editor.defaultFormatter": "esbenp.prettier-vscode"
   },
   "[typescript]": {
     "editor.defaultFormatter": "esbenp.prettier-vscode"

Dockerfile

@@ -1,5 +1,25 @@
+FROM ruby:3.2.6 AS native-gems
+RUN rm -f /etc/apt/apt.conf.d/docker-clean; \
+    echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/keep-cache
+RUN \
+    --mount=type=cache,target=/var/cache/apt,sharing=locked \
+    --mount=type=cache,target=/var/lib/apt,sharing=locked \
+    apt-get update && \
+    apt-get install --no-install-recommends --no-install-suggests -y \
+    cmake
+
+WORKDIR /usr/src/app
+RUN gem install bundler -v '2.5.6'
+COPY gems gems
+WORKDIR /usr/src/app/gems/xdiff-rb
+RUN bundle _2.5.6_ install
+RUN rake compile
+WORKDIR /usr/src/app/gems/rb-bsdiff
+RUN bundle _2.5.6_ install
+RUN rake compile
+
 # Primary image
-FROM ruby:3.4.4
+FROM ruby:3.2.6
 USER root
 
 # apt caching & install packages
@@ -14,10 +34,9 @@ RUN \
     libblas-dev liblapack-dev
 
 # preinstall gems that take a long time to install
-RUN MAKE="make -j12" gem install bundler           -v '2.6.7'
-RUN MAKE="make -j12" gem install rice              -v '4.3.3'
-RUN MAKE="make -j12" gem install faiss             -v '0.3.2'
-RUN MAKE="make -j12" gem install rails_live_reload -v '0.3.6'
+RUN MAKE="make -j12" gem install bundler           -v '2.5.6' --verbose
+RUN MAKE="make -j12" gem install faiss             -v '0.3.2' --verbose
+RUN MAKE="make -j12" gem install rails_live_reload -v '0.3.6' --verbose
 RUN bundle config --global frozen 1
 
 # set up nodejs 18.x deb repo
@@ -32,71 +51,14 @@ RUN \
     --mount=type=cache,target=/var/lib/apt,sharing=locked \
     apt-get update && \
     apt-get install --no-install-recommends --no-install-suggests -y \
-    ca-certificates \
-    curl \
-    gnupg \
-    nodejs \
-    libpq-dev \
-    ffmpeg \
-    ffmpegthumbnailer \
-    abiword \
-    pdftohtml \
-    libreoffice
+    libvips42 ca-certificates curl gnupg nodejs libpq-dev
 
-# Install vips dependencies
-RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
-    --mount=type=cache,target=/var/lib/apt,sharing=locked \
-    apt-get update && \
-    apt-get install --no-install-recommends --no-install-suggests -qqy \
-    automake \
-    gtk-doc-tools \
-    gobject-introspection \
-    libgirepository1.0-dev \
-    libglib2.0-dev \
-    libexpat1-dev \
-    libjpeg-dev \
-    libpng-dev \
-    libtiff5-dev \
-    libwebp-dev \
-    libheif-dev \
-    libexif-dev \
-    liblcms2-dev \
-    libxml2-dev \
-    libfftw3-dev \
-    liborc-0.4-dev \
-    libcgif-dev \
-    libjxl-dev \
-    libopenjp2-7-dev \
-    meson \
-    ninja-build
-
-# Install imagemagick from source
-RUN cd /tmp && \
-    wget -qO- https://imagemagick.org/archive/releases/ImageMagick-7.1.2-1.tar.xz | tar -xJ && \
-    cd ImageMagick-7.1.2-1 && \
-    ./configure && \
-    make -j$(nproc) && \
-    make install && \
-    ldconfig && \
-    cd / && \
-    rm -rf /tmp/ImageMagick-7.1.2-1*
-
-# Install vips from source
-RUN cd /tmp && \
-    wget -qO- https://github.com/libvips/libvips/releases/download/v8.17.1/vips-8.17.1.tar.xz | tar -xJ && \
-    cd vips-8.17.1 && \
-    meson setup build --prefix=/usr/local -Dcgif=enabled && \
-    cd build && \
-    ninja && \
-    ninja install && \
-    ldconfig && \
-    cd / && \
-    rm -rf /tmp/vips-8.17.1*
+COPY --from=native-gems /usr/src/app/gems/xdiff-rb /gems/xdiff-rb
+COPY --from=native-gems /usr/src/app/gems/rb-bsdiff /gems/rb-bsdiff
 
 WORKDIR /usr/src/app
 COPY Gemfile Gemfile.lock ./
-COPY gems/has_aux_table ./gems/has_aux_table
-RUN ls -lah gems && BUNDLE_FROZEN=true MAKE="make -j$(nproc)" bundle install --jobs $(nproc)
+RUN bundle _2.5.6_ install
 
 # install js dependencies
 COPY package.json yarn.lock ./
@@ -110,9 +72,6 @@ COPY . .
 RUN RAILS_ENV=production bin/rails assets:precompile
 RUN mkdir -p tmp/pids
 
-# build user scripts
-RUN yarn build:user-scripts
-
 # create user with id=1000 gid=1000
 RUN groupadd -g 1000 app && \
     useradd -m -d /home/app -s /bin/bash -u 1000 -g 1000 app

Gemfile

@@ -1,24 +1,24 @@
 source "https://rubygems.org"
 git_source(:github) { |repo| "https://github.com/#{repo}.git" }
 
-ruby "~> 3"
-gem "bundler", "~> 2.6.7"
+ruby "3.2.6"
+# ruby "3.0.3"
 
 # Bundle edge Rails instead: gem "rails", github: "rails/rails", branch: "main"
 gem "rails", "~> 7.2"
-gem "has_aux_table", path: "gems/has_aux_table"
 
 # The original asset pipeline for Rails [https://github.com/rails/sprockets-rails]
 gem "sprockets-rails"
 
+# Use sqlite3 as the database for Active Record
 gem "pg"
+gem "sqlite3", "~> 1.4"
 
 gem "pry"
 gem "pry-stack_explorer"
 
 # Use the Puma web server [https://github.com/puma/puma]
 gem "puma", "~> 5.0"
-gem "thruster"
 
 # # Use JavaScript with ESM import maps [https://github.com/rails/importmap-rails]
 # gem "importmap-rails"
@@ -55,7 +55,7 @@ gem "bootsnap", require: false
 
 group :development, :test, :staging do
   # See https://guides.rubyonrails.org/debugging_rails_applications.html#debugging-with-the-debug-gem
-  gem "debug", "~> 1.11", platforms: %i[mri mingw x64_mingw], require: false
+  gem "debug", "~> 1.10", platforms: %i[mri mingw x64_mingw]
 end
 
 group :development, :staging do
@@ -65,8 +65,7 @@ group :development, :staging do
   gem "web-console"
 
   # Speed up commands on slow machines / big apps [https://github.com/rails/spring]
-  gem "spring"
-  gem "spring-commands-rspec"
+  # gem "spring"
 
   # Add speed badges [https://github.com/MiniProfiler/rack-mini-profiler]
   gem "memory_profiler"
@@ -100,9 +99,14 @@ end
 
 group :test, :development do
   gem "parallel_tests"
-  gem "spring-commands-parallel-tests"
 end
 
+gem "xdiff", path: "/gems/xdiff-rb"
+
+# for legacy import
+gem "diffy"
+gem "rb-bsdiff", path: "/gems/rb-bsdiff"
+
 gem "addressable"
 gem "colorize"
 gem "concurrent-ruby-edge", require: "concurrent-edge"
@@ -120,20 +124,8 @@ gem "ripcord"
 gem "ruby-prof"
 gem "ruby-prof-speedscope"
 gem "ruby-vips"
-gem "dhash-vips"
-gem "ffmpeg", git: "https://github.com/instructure/ruby-ffmpeg", tag: "v6.1.2"
 gem "table_print"
 gem "zstd-ruby"
-gem "rouge"
-gem "docx"
-gem "ruby-bbcode"
-gem "dtext_rb",
-    git: "https://github.com/e621ng/dtext_rb",
-    ref: "5ef8fd7a5205c832f4c18197911717e7d491494e"
-gem "charlock_holmes"
-
-# Telegram Bot API
-gem "telegram-bot-ruby"
 
 # gem "pghero", git: "https://github.com/dymk/pghero", ref: "e314f99"
 gem "pghero", "~> 3.6"
@@ -147,7 +139,6 @@ gem "attr_json"
 
 group :production, :staging do
   gem "rails_semantic_logger", "~> 4.17"
-  gem "cloudflare-rails"
 end
 
 group :production do
@@ -159,14 +150,10 @@ gem "rack-cors"
 gem "react_on_rails"
 gem "sanitize", "~> 6.1"
 gem "shakapacker", "~> 6.6"
-gem "timeout"
 
 group :development do
   gem "prettier_print"
   gem "syntax_tree", "~> 6.2"
-  gem "unicode_plot" # For terminal-based data visualization (Ruby API)
-  gem "rumale" # Professional machine learning library for Ruby
-  gem "ruby-lsp-rspec", require: false
 end
 
 gem "cssbundling-rails", "~> 1.4"
@@ -181,13 +168,6 @@ gem "pundit", "~> 2.4"
 # Monitoring
 gem "prometheus_exporter", "~> 2.2"
 
-SORBET_VERSION = "0.5.12221"
-gem "sorbet", SORBET_VERSION, group: :development
-gem "sorbet-runtime", SORBET_VERSION
-gem "tapioca", "0.16.6", require: false, group: %i[development test]
+gem "sorbet-static-and-runtime"
+gem "tapioca", require: false, group: %i[development test]
 gem "rspec-sorbet", group: [:test]
-gem "sorbet-struct-comparable"
-
-gem "skyfall", "~> 0.6.0"
-
-gem "didkit", "~> 0.2.3"

Gemfile.lock

@@ -1,18 +1,3 @@
-GIT
-  remote: https://github.com/e621ng/dtext_rb
-  revision: 5ef8fd7a5205c832f4c18197911717e7d491494e
-  ref: 5ef8fd7a5205c832f4c18197911717e7d491494e
-  specs:
-    dtext_rb (1.11.0)
-
-GIT
-  remote: https://github.com/instructure/ruby-ffmpeg
-  revision: a3404b8fa275e2eb9549f074906461b0266a70ea
-  tag: v6.1.2
-  specs:
-    ffmpeg (6.1.2)
-      multi_json (~> 1.8)
-
 GIT
   remote: https://github.com/railsjazz/rails_live_reload
   revision: dcd3b73904594e2c5134c2f6e05954f3937a8d29
@@ -25,12 +10,14 @@ GIT
       websocket-driver
 
 PATH
-  remote: gems/has_aux_table
+  remote: /gems/rb-bsdiff
   specs:
-    has_aux_table (0.1.0)
-      activerecord (>= 7.0)
-      activesupport (>= 7.0)
-      sorbet-runtime (~> 0.5)
+    rb-bsdiff (0.1.0)
+
+PATH
+  remote: /gems/xdiff-rb
+  specs:
+    xdiff (0.0.1)
 
 GEM
   remote: https://rubygems.org/
@@ -110,11 +97,10 @@ GEM
       public_suffix (>= 2.0.2, < 7.0)
     attr_json (2.5.0)
       activerecord (>= 6.0.0, < 8.1)
-    base32 (0.3.4)
-    base64 (0.3.0)
+    base64 (0.2.0)
     bcrypt (3.1.20)
-    benchmark (0.4.1)
-    bigdecimal (3.2.2)
+    benchmark (0.4.0)
+    bigdecimal (3.1.9)
     bindex (0.8.1)
     binding_of_caller (1.0.1)
       debug_inspector (>= 1.2.0)
@@ -130,13 +116,6 @@ GEM
       rack-test (>= 0.6.3)
       regexp_parser (>= 1.5, < 3.0)
       xpath (~> 3.2)
-    cbor (0.5.9.9)
-    charlock_holmes (0.7.9)
-    cloudflare-rails (6.2.0)
-      actionpack (>= 7.1.0, < 8.1.0)
-      activesupport (>= 7.1.0, < 8.1.0)
-      railties (>= 7.1.0, < 8.1.0)
-      zeitwerk (>= 2.5.0)
     coderay (1.1.3)
     colorize (1.1.0)
     concurrent-ruby (1.3.4)
@@ -144,18 +123,17 @@ GEM
       concurrent-ruby (~> 1.3)
     concurrent-ruby-ext (1.3.4)
       concurrent-ruby (= 1.3.4)
-    connection_pool (2.5.3)
+    connection_pool (2.4.1)
     crass (1.0.6)
     cssbundling-rails (1.4.1)
       railties (>= 6.0.0)
-    csv (3.3.5)
     curb (1.0.6)
     daemons (1.4.1)
     date (3.4.1)
     db-query-matchers (0.14.0)
       activesupport (>= 4.0, < 8.1)
       rspec (>= 3.0)
-    debug (1.11.0)
+    debug (1.10.0)
       irb (~> 1.10)
       reline (>= 0.3.8)
     debug_inspector (1.2.0)
@@ -165,47 +143,18 @@ GEM
       railties (>= 4.1.0)
       responders
       warden (~> 1.2.3)
-    dhash-vips (0.2.3.0)
-      ruby-vips (~> 2.0, != 2.1.1, != 2.1.0)
-    didkit (0.2.3)
     diff-lcs (1.5.1)
+    diffy (3.4.3)
     discard (1.4.0)
       activerecord (>= 4.2, < 9.0)
     disco (0.5.1)
       libmf (>= 0.4)
       numo-narray (>= 0.9.2)
-    docx (0.8.0)
-      nokogiri (~> 1.13, >= 1.13.0)
-      rubyzip (~> 2.0)
     domain_name (0.6.20240107)
-    drb (2.2.3)
-    dry-core (1.1.0)
-      concurrent-ruby (~> 1.0)
-      logger
-      zeitwerk (~> 2.6)
-    dry-inflector (1.2.0)
-    dry-logic (1.6.0)
-      bigdecimal
-      concurrent-ruby (~> 1.0)
-      dry-core (~> 1.1)
-      zeitwerk (~> 2.6)
-    dry-struct (1.8.0)
-      dry-core (~> 1.1)
-      dry-types (~> 1.8, >= 1.8.2)
-      ice_nine (~> 0.11)
-      zeitwerk (~> 2.6)
-    dry-types (1.8.3)
-      bigdecimal (~> 3.0)
-      concurrent-ruby (~> 1.0)
-      dry-core (~> 1.0)
-      dry-inflector (~> 1.0)
-      dry-logic (~> 1.4)
-      zeitwerk (~> 2.6)
-    enumerable-statistics (2.0.8)
+    drb (2.2.1)
     erubi (1.13.1)
     et-orbi (1.2.11)
       tzinfo
-    eventmachine (1.2.7)
     execjs (2.10.0)
     factory_bot (6.5.0)
       activesupport (>= 5.0.0)
@@ -215,17 +164,6 @@ GEM
     faiss (0.3.2)
       numo-narray
       rice (>= 4.0.2)
-    faraday (2.13.4)
-      faraday-net_http (>= 2.0, < 3.5)
-      json
-      logger
-    faraday-multipart (1.1.1)
-      multipart-post (~> 2.0)
-    faraday-net_http (3.4.1)
-      net-http (>= 0.5.0)
-    faye-websocket (0.12.0)
-      eventmachine (>= 0.12.0)
-      websocket-driver (>= 0.8.0)
     ffi (1.17.1-aarch64-linux-gnu)
     ffi (1.17.1-aarch64-linux-musl)
     ffi (1.17.1-arm64-darwin)
@@ -247,22 +185,16 @@ GEM
       fugit (>= 1.11.0)
       railties (>= 6.1.0)
       thor (>= 1.0.0)
-    google-protobuf (4.31.1-aarch64-linux-gnu)
+    google-protobuf (4.29.2-aarch64-linux)
       bigdecimal
       rake (>= 13)
-    google-protobuf (4.31.1-aarch64-linux-musl)
+    google-protobuf (4.29.2-arm64-darwin)
       bigdecimal
       rake (>= 13)
-    google-protobuf (4.31.1-arm64-darwin)
+    google-protobuf (4.29.2-x86_64-darwin)
       bigdecimal
       rake (>= 13)
-    google-protobuf (4.31.1-x86_64-darwin)
-      bigdecimal
-      rake (>= 13)
-    google-protobuf (4.31.1-x86_64-linux-gnu)
-      bigdecimal
-      rake (>= 13)
-    google-protobuf (4.31.1-x86_64-linux-musl)
+    google-protobuf (4.29.2-x86_64-linux)
       bigdecimal
       rake (>= 13)
     htmlbeautifier (1.4.3)
@@ -275,9 +207,8 @@ GEM
     http-cookie (1.0.8)
       domain_name (~> 0.5)
     http-form_data (2.3.0)
-    i18n (1.14.7)
+    i18n (1.14.6)
       concurrent-ruby (~> 1.0)
-    ice_nine (0.11.2)
     io-console (0.8.0)
     irb (1.14.3)
       rdoc (>= 4.0.0)
@@ -285,7 +216,6 @@ GEM
     jbuilder (2.13.0)
       actionview (>= 5.0.0)
       activesupport (>= 5.0.0)
-    json (2.13.2)
     kaminari (1.2.2)
       activesupport (>= 4.1.0)
       kaminari-actionview (= 1.2.2)
@@ -298,9 +228,6 @@ GEM
       activerecord
       kaminari-core (= 1.2.2)
     kaminari-core (1.2.2)
-    language_server-protocol (3.17.0.5)
-    lbfgsb (0.6.0)
-      numo-narray (>= 0.9.1)
     libmf (0.4.0)
       ffi
     listen (3.9.0)
@@ -309,7 +236,7 @@ GEM
     llhttp-ffi (0.5.0)
       ffi-compiler (~> 1.0)
       rake (~> 13.0)
-    logger (1.7.0)
+    logger (1.6.4)
     loofah (2.23.1)
       crass (~> 1.0.2)
       nokogiri (>= 1.12.0)
@@ -323,15 +250,10 @@ GEM
     memory_profiler (1.1.0)
     method_source (1.1.0)
     mini_mime (1.1.5)
-    minitest (5.25.5)
-    mmh3 (1.2.0)
+    minitest (5.25.4)
     msgpack (1.7.5)
-    multi_json (1.15.0)
-    multipart-post (2.4.1)
     neighbor (0.5.1)
       activerecord (>= 7)
-    net-http (0.6.0)
-      uri
     net-imap (0.5.4)
       date
       net-protocol
@@ -453,8 +375,6 @@ GEM
     rbi (0.2.2)
       prism (~> 1.0)
       sorbet-runtime (>= 0.5.9204)
-    rbs (3.9.4)
-      logger
     rdoc (6.10.0)
       psych (>= 4.0.0)
     react_on_rails (14.0.5)
@@ -472,7 +392,6 @@ GEM
     rexml (3.4.0)
     rice (4.3.3)
     ripcord (2.0.0)
-    rouge (4.5.1)
     rspec (3.13.0)
       rspec-core (~> 3.13.0)
       rspec-expectations (~> 3.13.0)
@@ -496,14 +415,6 @@ GEM
     rspec-sorbet (1.9.2)
       sorbet-runtime
     rspec-support (3.13.2)
-    ruby-bbcode (2.1.1)
-      activesupport (>= 4.2.2)
-    ruby-lsp (0.25.0)
-      language_server-protocol (~> 3.17.0)
-      prism (>= 1.2, < 2.0)
-      rbs (>= 3, < 5)
-    ruby-lsp-rspec (0.1.26)
-      ruby-lsp (~> 0.25.0)
     ruby-prof (1.7.1)
     ruby-prof-speedscope (0.3.0)
       ruby-prof (~> 1.0)
@@ -512,91 +423,6 @@ GEM
       logger
     rubyzip (2.3.2)
     rufo (0.18.0)
-    rumale (1.0.0)
-      numo-narray (>= 0.9.1)
-      rumale-clustering (~> 1.0.0)
-      rumale-core (~> 1.0.0)
-      rumale-decomposition (~> 1.0.0)
-      rumale-ensemble (~> 1.0.0)
-      rumale-evaluation_measure (~> 1.0.0)
-      rumale-feature_extraction (~> 1.0.0)
-      rumale-kernel_approximation (~> 1.0.0)
-      rumale-kernel_machine (~> 1.0.0)
-      rumale-linear_model (~> 1.0.0)
-      rumale-manifold (~> 1.0.0)
-      rumale-metric_learning (~> 1.0.0)
-      rumale-model_selection (~> 1.0.0)
-      rumale-naive_bayes (~> 1.0.0)
-      rumale-nearest_neighbors (~> 1.0.0)
-      rumale-neural_network (~> 1.0.0)
-      rumale-pipeline (~> 1.0.0)
-      rumale-preprocessing (~> 1.0.0)
-      rumale-tree (~> 1.0.0)
-    rumale-clustering (1.0.0)
-      numo-narray (>= 0.9.1)
-      rumale-core (~> 1.0.0)
-    rumale-core (1.0.0)
-      csv (>= 3.1.9)
-      numo-narray (>= 0.9.1)
-    rumale-decomposition (1.0.0)
-      numo-narray (>= 0.9.1)
-      rumale-core (~> 1.0.0)
-    rumale-ensemble (1.0.0)
-      numo-narray (>= 0.9.1)
-      rumale-core (~> 1.0.0)
-      rumale-linear_model (~> 1.0.0)
-      rumale-model_selection (~> 1.0.0)
-      rumale-preprocessing (~> 1.0.0)
-      rumale-tree (~> 1.0.0)
-    rumale-evaluation_measure (1.0.0)
-      numo-narray (>= 0.9.1)
-      rumale-core (~> 1.0.0)
-    rumale-feature_extraction (1.0.0)
-      mmh3 (~> 1.0)
-      numo-narray (>= 0.9.1)
-      rumale-core (~> 1.0.0)
-    rumale-kernel_approximation (1.0.0)
-      numo-narray (>= 0.9.1)
-      rumale-core (~> 1.0.0)
-    rumale-kernel_machine (1.0.0)
-      numo-narray (>= 0.9.1)
-      rumale-core (~> 1.0.0)
-    rumale-linear_model (1.0.0)
-      lbfgsb (>= 0.3.0)
-      numo-narray (>= 0.9.1)
-      rumale-core (~> 1.0.0)
-    rumale-manifold (1.0.0)
-      numo-narray (>= 0.9.1)
-      rumale-core (~> 1.0.0)
-      rumale-decomposition (~> 1.0.0)
-    rumale-metric_learning (1.0.0)
-      lbfgsb (>= 0.3.0)
-      numo-narray (>= 0.9.1)
-      rumale-core (~> 1.0.0)
-      rumale-decomposition (~> 1.0.0)
-    rumale-model_selection (1.0.0)
-      numo-narray (>= 0.9.1)
-      rumale-core (~> 1.0.0)
-      rumale-evaluation_measure (~> 1.0.0)
-      rumale-preprocessing (~> 1.0.0)
-    rumale-naive_bayes (1.0.0)
-      numo-narray (>= 0.9.1)
-      rumale-core (~> 1.0.0)
-    rumale-nearest_neighbors (1.0.0)
-      numo-narray (>= 0.9.1)
-      rumale-core (~> 1.0.0)
-    rumale-neural_network (1.0.0)
-      numo-narray (>= 0.9.1)
-      rumale-core (~> 1.0.0)
-    rumale-pipeline (1.0.0)
-      numo-narray (>= 0.9.1)
-      rumale-core (~> 1.0.0)
-    rumale-preprocessing (1.0.0)
-      numo-narray (>= 0.9.1)
-      rumale-core (~> 1.0.0)
-    rumale-tree (1.0.0)
-      numo-narray (>= 0.9.1)
-      rumale-core (~> 1.0.0)
     sanitize (6.1.3)
       crass (~> 1.0.2)
       nokogiri (>= 1.12.0)
@@ -616,33 +442,20 @@ GEM
       semantic_range (>= 2.3.0)
     shoulda-matchers (6.4.0)
       activesupport (>= 5.2.0)
-    skyfall (0.6.0)
-      base32 (~> 0.3, >= 0.3.4)
-      base64 (~> 0.1)
-      cbor (~> 0.5, >= 0.5.9.6)
-      eventmachine (~> 1.2, >= 1.2.7)
-      faye-websocket (~> 0.12)
-    sorbet (0.5.12221)
-      sorbet-static (= 0.5.12221)
-    sorbet-runtime (0.5.12221)
-    sorbet-static (0.5.12221-aarch64-linux)
-    sorbet-static (0.5.12221-universal-darwin)
-    sorbet-static (0.5.12221-x86_64-linux)
-    sorbet-static-and-runtime (0.5.12221)
-      sorbet (= 0.5.12221)
-      sorbet-runtime (= 0.5.12221)
-    sorbet-struct-comparable (1.3.0)
-      sorbet-runtime (>= 0.5)
+    sorbet (0.5.11711)
+      sorbet-static (= 0.5.11711)
+    sorbet-runtime (0.5.11711)
+    sorbet-static (0.5.11711-aarch64-linux)
+    sorbet-static (0.5.11711-universal-darwin)
+    sorbet-static (0.5.11711-x86_64-linux)
+    sorbet-static-and-runtime (0.5.11711)
+      sorbet (= 0.5.11711)
+      sorbet-runtime (= 0.5.11711)
     spoom (1.5.0)
       erubi (>= 1.10.0)
       prism (>= 0.28.0)
       sorbet-static-and-runtime (>= 0.5.10187)
       thor (>= 0.19.2)
-    spring (4.3.0)
-    spring-commands-parallel-tests (1.0.1)
-      spring (>= 0.9.1)
-    spring-commands-rspec (1.0.4)
-      spring (>= 0.9.1)
     sprockets (4.2.1)
       concurrent-ruby (~> 1.0)
       rack (>= 2.2.4, < 4)
@@ -650,6 +463,10 @@ GEM
       actionpack (>= 6.1)
       activesupport (>= 6.1)
       sprockets (>= 3.0.0)
+    sqlite3 (1.7.3-aarch64-linux)
+    sqlite3 (1.7.3-arm64-darwin)
+    sqlite3 (1.7.3-x86_64-darwin)
+    sqlite3 (1.7.3-x86_64-linux)
     stackprof (0.2.26)
     stimulus-rails (1.3.4)
       railties (>= 6.0.0)
@@ -664,7 +481,7 @@ GEM
     tailwindcss-ruby (3.4.17-arm64-darwin)
     tailwindcss-ruby (3.4.17-x86_64-darwin)
     tailwindcss-ruby (3.4.17-x86_64-linux)
-    tapioca (0.16.6)
+    tapioca (0.16.5)
       bundler (>= 2.2.25)
       netrc (>= 0.11.0)
       parallel (>= 1.21.0)
@@ -673,25 +490,13 @@ GEM
       spoom (>= 1.2.0)
       thor (>= 1.2.0)
       yard-sorbet
-    telegram-bot-ruby (2.4.0)
-      dry-struct (~> 1.6)
-      faraday (~> 2.0)
-      faraday-multipart (~> 1.0)
-      zeitwerk (~> 2.6)
     thor (1.3.2)
-    thruster (0.1.11-aarch64-linux)
-    thruster (0.1.11-arm64-darwin)
-    thruster (0.1.11-x86_64-darwin)
-    thruster (0.1.11-x86_64-linux)
     timeout (0.4.3)
     turbo-rails (2.0.11)
       actionpack (>= 6.0.0)
       railties (>= 6.0.0)
     tzinfo (2.0.6)
       concurrent-ruby (~> 1.0)
-    unicode_plot (0.0.5)
-      enumerable-statistics (>= 2.0.1)
-    uri (1.0.3)
     useragent (0.16.11)
     warden (1.2.9)
       rack (>= 2.0.9)
@@ -706,8 +511,7 @@ GEM
       selenium-webdriver (~> 4.0, < 4.11)
     webrick (1.9.1)
     websocket (1.2.11)
-    websocket-driver (0.8.0)
-      base64
+    websocket-driver (0.7.6)
       websocket-extensions (>= 0.1.0)
     websocket-extensions (0.1.5)
     xpath (3.2.0)
@@ -733,10 +537,7 @@ DEPENDENCIES
   addressable
   attr_json
   bootsnap
-  bundler (~> 2.6.7)
   capybara
-  charlock_holmes
-  cloudflare-rails
   colorize
   concurrent-ruby-edge
   concurrent-ruby-ext
@@ -744,19 +545,14 @@ DEPENDENCIES
   curb
   daemons
   db-query-matchers (~> 0.14)
-  debug (~> 1.11)
+  debug (~> 1.10)
   devise (~> 4.9)
-  dhash-vips
-  didkit (~> 0.2.3)
+  diffy
   discard
   disco
-  docx
-  dtext_rb!
   factory_bot_rails
   faiss
-  ffmpeg!
   good_job (~> 4.6)
-  has_aux_table!
   htmlbeautifier
   http (~> 5.2)
   http-cookie
@@ -785,49 +581,38 @@ DEPENDENCIES
   rails-controller-testing
   rails_live_reload!
   rails_semantic_logger (~> 4.17)
+  rb-bsdiff!
   react_on_rails
   ripcord
-  rouge
   rspec-rails (~> 7.0)
   rspec-sorbet
-  ruby-bbcode
-  ruby-lsp-rspec
   ruby-prof
   ruby-prof-speedscope
   ruby-vips
   rufo
-  rumale
   sanitize (~> 6.1)
   sd_notify
   selenium-webdriver
   shakapacker (~> 6.6)
   shoulda-matchers
-  skyfall (~> 0.6.0)
-  sorbet (= 0.5.12221)
-  sorbet-runtime (= 0.5.12221)
-  sorbet-struct-comparable
-  spring
-  spring-commands-parallel-tests
-  spring-commands-rspec
+  sorbet-static-and-runtime
   sprockets-rails
+  sqlite3 (~> 1.4)
   stackprof
   stimulus-rails
   syntax_tree (~> 6.2)
   table_print
   tailwindcss-rails (~> 3.0)
-  tapioca (= 0.16.6)
-  telegram-bot-ruby
-  thruster
-  timeout
+  tapioca
   turbo-rails
   tzinfo-data
-  unicode_plot
   web-console
   webdrivers
+  xdiff!
   zstd-ruby
 
 RUBY VERSION
    ruby 3.2.6p234
 
 BUNDLED WITH
-   2.6.7
+   2.6.2

Procfile.dev

@@ -1,5 +1,5 @@
-rails: RAILS_ENV=development HTTP_PORT=3001 thrust ./bin/rails server
+rails: RAILS_ENV=development bundle exec rails s -p 3000
 wp-client: RAILS_ENV=development HMR=true ./bin/webpacker-dev-server
 wp-server: RAILS_ENV=development HMR=true SERVER_BUNDLE_ONLY=yes ./bin/webpacker --watch
-css: tailwindcss -c ./config/tailwind.config.js -i ./app/assets/stylesheets/application.tailwind.css -o ./app/assets/builds/tailwind.css --watch
+css: RAILS_ENV=development yarn "build:css[debug]" --watch
 prometheus_exporter: RAILS_ENV=development bundle exec prometheus_exporter --bind 0.0.0.0 --prefix redux_ --label '{"environment": "development"}'

Procfile.production

@@ -1,3 +1,3 @@
-rails: RAILS_ENV=production HTTP_PORT=3000 TARGET_PORT=3003 thrust ./bin/rails server -p 3003
+rails: RAILS_ENV=production bundle exec rails s -b 0.0.0.0 -p 3000
 tail: tail -f log/production.log
 prometheus_exporter: RAILS_ENV=production bundle exec prometheus_exporter --bind 0.0.0.0 --prefix redux_ --label '{"environment": "production"}'

Procfile.staging

@@ -1,5 +1,5 @@
-rails: RAILS_ENV=staging HTTP_PORT=3001 bundle exec thrust ./bin/rails server
+rails: RAILS_ENV=staging ./bin/rails s -p 3001
 wp-client: RAILS_ENV=development HMR=true ./bin/webpacker-dev-server
 wp-server: RAILS_ENV=development HMR=true SERVER_BUNDLE_ONLY=yes ./bin/webpacker --watch
 css: RAILS_ENV=development yarn "build:css[debug]" --watch
-prometheus-exporter: RAILS_ENV=staging bundle exec prometheus_exporter --bind 0.0.0.0 --prefix redux_ --label '{"environment": "staging"}'
+prometheus_exporter: RAILS_ENV=staging bundle exec prometheus_exporter --bind 0.0.0.0 --prefix redux_ --label '{"environment": "staging"}'

README.md

@@ -1,29 +1,3 @@
-# Redux Scraper
-
-A Ruby on Rails application for scraping and managing various content sources.
-
-## Setup
-
-This application is configured for optimal development and testing performance:
-
-### Performance Optimizations
-
-- **Bootsnap**: Accelerates gem loading and caching for faster boot times
-- **Spring**: Preloads the Rails application for faster command execution
-
-#### Rails Boot Performance
-
-- Development boot time: ~270ms (87% faster than without optimization)
-- Test environment startup: ~211ms (29% faster than without optimization)
-
-To use Spring-optimized commands:
-```bash
-# Use bin/ executables for Spring acceleration
-bin/rails console
-bin/rails runner "puts 'Hello'"
-bin/rspec spec/
-```
-
 # README
 
 This README would normally document whatever steps are necessary to get the

Rakefile

@@ -40,23 +40,22 @@ task periodic_tasks: %i[environment set_logger_stdout] do
   loop { sleep 10 }
 end
 
-# TODO - migrate to Domain::Post / Domain::User
-# namespace :db_sampler do
-#   task export: :environment do
-#     url_names = ENV["url_names"] || raise("need 'url_names' (comma-separated)")
-#     outfile = $stdout
-#     DbSampler.new(outfile).export(url_names.split(","))
-#   ensure
-#     outfile.close if outfile
-#   end
+namespace :db_sampler do
+  task export: :environment do
+    url_names = ENV["url_names"] || raise("need 'url_names' (comma-separated)")
+    outfile = $stdout
+    DbSampler.new(outfile).export(url_names.split(","))
+  ensure
+    outfile.close if outfile
+  end
 
-#   task import: [:environment] do
-#     infile = $stdin
-#     DbSampler.new(infile).import
-#   ensure
-#     infile.close if infile
-#   end
-# end
+  task import: [:environment] do
+    infile = $stdin
+    DbSampler.new(infile).import
+  ensure
+    infile.close if infile
+  end
+end
 
 task good_job: %i[environment set_ar_stdout set_logger_stdout] do
   env_hash = {
@@ -93,302 +92,3 @@ task :reverse_csv do
   in_csv.reverse_each { |row| out_csv << row.map(&:second) }
   out_csv.close
 end
-
-task enqueue_fa_posts_missing_files: %i[environment set_logger_stdout] do
-  Domain::Post::FaPost
-    .where(state: "ok")
-    .where
-    .missing(:file)
-    .find_each(order: :desc) do |post|
-      Domain::Fa::Job::ScanPostJob.perform_now(post:)
-    end
-end
-
-task fix_e621_post_files: :environment do
-  query = Domain::Post::E621Post.where(state: "ok").where.missing(:files)
-  limit = ENV["limit"]&.to_i
-  puts "query: #{query.to_sql}"
-
-  query.find_each(batch_size: 10) do |post|
-    Domain::E621::Task::FixE621PostMissingFiles.new.run(post)
-    if limit
-      limit -= 1
-      if limit.zero?
-        puts "limit reached"
-        break
-      end
-    end
-  end
-end
-
-task fix_ok_e621_posts_missing_files: :environment do
-  query = Domain::Post::E621Post.where(state: "ok").where.missing(:file)
-  progress_bar =
-    ProgressBar.create(total: query.count, format: "%t: %c/%C %B %p%% %a %e")
-  query.find_each(batch_size: 10) do |post|
-    Domain::E621::Job::ScanPostJob.perform_now(post:)
-    progress_bar.progress = [progress_bar.progress + 1, progress_bar.total].min
-  end
-end
-
-task perform_good_jobs: :environment do
-  job_class = ENV["job_class"]
-  job_id = ENV["job_id"]
-  limit = ENV["limit"]&.to_i
-
-  if !job_id.present? && !job_class.present?
-    raise "need 'job_id' or 'job_class'"
-  end
-
-  relation =
-    if job_id
-      job =
-        GoodJob::Job.find_by(id: job_id) ||
-          GoodJob::Execution.find_by(id: job_id)&.job
-      if job.nil?
-        puts "no job found with id #{job_id}"
-        exit 1
-      end
-      puts "found job with id #{job.id}" if job.id != job_id
-      GoodJob::Job.where(id: job.id)
-    else
-      GoodJob::Job.queued.where(job_class: job_class).order(created_at: :asc)
-    end
-
-  relation.find_each(batch_size: 1) do |job|
-    job = T.cast(job, GoodJob::Job)
-
-    # Get the actual job instance and deserialize arguments
-    serialized_args = job.serialized_params["arguments"]
-    if serialized_args.nil?
-      puts "No arguments found for job #{job.id}"
-      next
-    end
-
-    deserialized_args = ActiveJob::Arguments.deserialize(serialized_args)
-    job_instance = job.job_class.constantize.new
-    job_instance.deserialize(job.serialized_params)
-
-    puts "Running job #{job.id} (#{job.job_class})"
-
-    # Create execution record
-    execution =
-      GoodJob::Execution.create!(
-        active_job_id: job.active_job_id,
-        job_class: job.job_class,
-        queue_name: job.queue_name,
-        serialized_params: job.serialized_params,
-        scheduled_at: job.scheduled_at,
-        created_at: Time.now,
-        updated_at: Time.now,
-        process_id: SecureRandom.uuid,
-      )
-
-    start_time = Time.now
-
-    # Temporarily disable concurrency limits
-    job_class = job.job_class.constantize
-    old_config = job_class.good_job_concurrency_config
-    job_class.good_job_concurrency_config = { total_limit: nil }
-
-    begin
-      # Perform the job with deserialized arguments
-      GoodJob::CurrentThread.job = job
-      job.update!(performed_at: Time.now)
-      job_instance.arguments = deserialized_args
-      job_instance.perform_now
-
-      # Update execution and job records
-      execution.update!(
-        finished_at: Time.now,
-        error: nil,
-        error_event: nil,
-        duration: Time.now - start_time,
-      )
-      job.update!(finished_at: Time.now)
-      puts "Job completed successfully"
-    rescue => e
-      puts "Job failed: #{e.message}"
-      # Update execution and job records with error
-      execution.update!(
-        finished_at: Time.now,
-        error: e.message,
-        error_event: "execution_failed",
-        error_backtrace: e.backtrace,
-        duration: Time.now - start_time,
-      )
-      job.update!(
-        error: "#{e.class}: #{e.message}",
-        error_event: "execution_failed",
-      )
-      raise e
-    ensure
-      job.update!(
-        executions_count: GoodJob::Execution.where(active_job_id: job.id).count,
-      )
-      # Restore original concurrency config
-      job_class.good_job_concurrency_config = old_config
-      GoodJob::CurrentThread.job = nil
-    end
-
-    if limit
-      limit -= 1
-      if limit.zero?
-        puts "limit reached"
-        break
-      end
-    end
-  end
-end
-
-task fix_removed_fa_posts: :environment do
-  colorize_state = ->(state) do
-    case state
-    when "ok"
-      "ok".green
-    when "removed"
-      "removed".red
-    else
-      state.to_s
-    end.bold
-  end
-
-  last_fa_id = ENV["start_at"]&.to_i
-  while true
-    query =
-      Domain::Post::FaPost
-        .where(state: "removed")
-        .where.not(title: nil)
-        .order(fa_id: :desc)
-    query = query.where(fa_id: ...last_fa_id) if last_fa_id
-    post = query.first
-    break unless post
-    last_fa_id = post.fa_id
-
-    puts "[before] [post.state: #{colorize_state.call(post.state)}] [post.file.id: #{post.file&.id}] [post.id: #{post.id}] [post.fa_id: #{post.fa_id}] [post.title: #{post.title}]"
-    Domain::Fa::Job::ScanPostJob.perform_now(post: post, force_scan: true)
-    post.reload
-    puts "[after] [post.state: #{colorize_state.call(post.state)}] [post.file.id: #{post.file&.id}] [post.id: #{post.id}] [post.fa_id: #{post.fa_id}] [post.title: #{post.title}]"
-    sleep 2
-  end
-rescue => e
-  puts "error: #{e.message}"
-  binding.pry
-end
-
-task run_fa_user_avatar_jobs: :environment do
-  avatars =
-    Domain::UserAvatar
-      .where(state: "pending")
-      .joins(:user)
-      .where(user: { type: Domain::User::FaUser.name })
-
-  puts "count: #{avatars.count}"
-
-  avatars.each do |avatar|
-    Domain::Fa::Job::UserAvatarJob.perform_now(avatar:)
-    avatar.reload
-    puts "perform avatar job for #{avatar.user.url_name} - #{avatar.state.bold}"
-  end
-end
-
-task create_post_file_fingerprints: %i[environment set_logger_stdout] do
-  task = Tasks::CreatePostFileFingerprintsTask.new
-
-  mode =
-    if ENV["post_file_descending"].present?
-      Tasks::CreatePostFileFingerprintsTask::Mode::PostFileDescending
-    elsif ENV["posts_descending"].present?
-      Tasks::CreatePostFileFingerprintsTask::Mode::PostsDescending
-    elsif ENV["user"].present?
-      Tasks::CreatePostFileFingerprintsTask::Mode::User
-    elsif ENV["users_descending"].present?
-      Tasks::CreatePostFileFingerprintsTask::Mode::UsersDescending
-    else
-      raise "need one of: post_file_descending, posts_descending, user, users_descending"
-    end
-
-  task.run(mode: mode, user_param: ENV["user"], start_at: ENV["start_at"])
-end
-
-task enqueue_pending_post_files: :environment do
-  query = Domain::PostFile.where(state: "pending")
-  puts "enqueueing #{query.count} pending post files"
-  query.find_in_batches(batch_size: 100, start: ENV["start_at"]) do |batch|
-    while (
-            queue_size =
-              GoodJob::Job.where(
-                job_class: "Job::PostFileJob",
-                performed_at: nil,
-                scheduled_at: nil,
-                error: nil,
-              ).count
-          ) > 100
-      puts "queue size: #{queue_size}"
-      sleep 10
-    end
-    batch.each do |post_file|
-      Job::PostFileJob.set(priority: 10).perform_later(post_file:)
-    end
-  end
-end
-
-task find_post_files_with_empty_response: :environment do
-  query =
-    Domain::PostFile
-      .where(state: "ok", retry_count: 0)
-      .joins(:log_entry)
-      .where(http_log_entries: { response_sha256: BlobFile::EMPTY_FILE_SHA256 })
-
-  pb = ProgressBar.create(total: query.count, format: "%t: %c/%C %B %p%% %a %e")
-
-  query.find_each(batch_size: 10) do |post_file|
-    # puts "post_file: #{post_file.id} / '#{post_file.post.to_param}'"
-    post_file.state_pending!
-    post_file.save!
-    Job::PostFileJob.perform_now(post_file:)
-    pb.progress = [pb.progress + 1, pb.total].min
-  end
-end
-
-desc "Enqueue pending post file jobs"
-task enqueue_pending_post_file_jobs: :environment do
-  Tasks::EnqueueDuePostFileJobsTask.new.run
-end
-
-desc "Compute null counter caches for all users"
-task compute_null_user_counter_caches: :environment do
-  counter_caches = {
-    user_post_creations_count: :user_post_creations,
-    user_post_favs_count: :user_post_favs,
-    user_user_follows_from_count: :user_user_follows_from,
-    user_user_follows_to_count: :user_user_follows_to,
-  }
-
-  query =
-    Domain::User.where(
-      counter_caches.map { |col, _| "(\"#{col}\" IS NULL)" }.join(" OR "),
-    )
-  total = query.count
-  query = query.select(:id, *counter_caches.keys)
-
-  puts "computing #{counter_caches.keys.join(", ")} for #{total} users"
-  pb = ProgressBar.create(total:, format: "%t: %c/%C %B %p%% %a %e")
-
-  query.find_in_batches(batch_size: 32) do |batch|
-    ReduxApplicationRecord.transaction do
-      batch.each do |user|
-        nil_caches =
-          counter_caches.keys.filter { |cache| user.send(cache).nil? }
-        Domain::User.reset_counters(
-          user.id,
-          *nil_caches.map { |col| counter_caches[col] },
-        )
-        pb.progress = [pb.progress + 1, total].min
-      end
-    end
-  end
-end
-
-puts "set proc title to #{ARGV.first}"
-Process.setproctitle(ARGV.first) if $0.split("/").last == "rake"

TODO.md

@@ -6,42 +6,4 @@
 - [x] Attach logs to jobs, page to view jobs and their logs
 - [ ] Standardize all the embeddings tables to use the same schema (item_id, embedding)
 - [ ] Bluesky scraper
-- [x] Download favs / votes for E621 users
-- [ ] Automatically enqueue jobs for FA users to do incremental scans of profiles
-- [ ] Fix FA posts that start with "Font size adjustment: smallerlarger"
-- [ ] Convert logger .prefix=... into .tagged(...)
-- [x] `make_tag` should be smart about the objects it takes
-- [ ] Convert all `state: string` attributes to enums in ActiveRecord models
-- [ ] Create `belongs_to_log_entry` macro for ActiveRecord models
-- [x] Use StaticFileJobHelper for Domain::Fa::Job::ScanFileJob
-- [ ] Unify HTTP client configs for all domains, so the same job type can be used for different domains
-- [ ] put abstract `external_url_for_view` in a module
-- [ ] backfill descriptions on inkbunny posts
-- [ ] store deep update json on inkbunny posts
-- [x] limit number of users, or paginate for "users who favorited this post" page
-- [ ] manual good job runner does not indicate if the job threw an exception - check return value of #perform, maybe?
-- [ ] FA user favs job should stop when in incremental mode when all posts on the page are already known favs (e.g. pages with only 47 posts are not a false positive)
-- [x] Factor out FA listings page enqueue logic into common location; use in Gallery and Favs jobs
-- [ ] Add followers / following to FA user show page
-- [x] Parse E621 source url for inkbunny posts & users
-- [x] Parse E621 source url for fa users
-- [ ] Parse BBCode in post descriptions
-  - example post with bbcode: https://refurrer.com/posts/ib/3452498
-- [ ] Show tags on fa posts, ib posts
-- [ ] Sofurry implmentation
-- [ ] Make unified Static file job
-- [ ] Make unified Avatar file job
-- [ ] ko-fi domain icon
-- [ ] tumblr domain icon
-- [ ] Do PCA on user factors table to display a 2D plot of users
-- [ ] Use links found in descriptions to indicate re-scanning a post? (e.g. for comic next/prev links)
-- [ ] fix for IDs that have a dot in them - e.g. https://refurrer.com/users/fa@jakke.
-- [ ] Rich inline links to e621 e.g. https://refurrer.com/posts/fa@60070060
-- [ ] Find FaPost that have favs recorded but no scan / file, enqueue scan
-- [x] Bunch of posts with empty responses: posts = Domain::Post.joins(files: :log_entry).where(files: { http_log_entries: { response_sha256: BlobFile::EMPTY_FILE_SHA256 }}).limit(10)
-- [ ] Create GlobalState entries for last FA id on browse page, periodic scan to scan from the newest FA ID to the stored one
-- [ ] GlobalState entries for long running backfill jobs, automatically restart them if they fail
-- [ ] Flag to pass to jobs to log HTTP requests / responses to a directory, HTTP mock helper to read from that directory
-- [ ] fix IP address incorrect for Cloudflare proxied requests
-- [ ] SOCKS5 proxy for additional workers
-- [ ] Backup FA scraper using foxbot & g6jy5jkx466lrqojcngbnksugrcfxsl562bzuikrka5rv7srgguqbjid.onion
+- [ ] Download favs / votes for E621 users

app/assets/images/domain-icons/sorbet/rbi/dsl/.gitattributes

@@ -1 +0,0 @@
-**/*.rbi linguist-generated=true

app/assets/images/domain-icons/sorbet/rbi/dsl/active_support/callbacks.rbi

@@ -1,23 +0,0 @@
-# typed: true
-
-# DO NOT EDIT MANUALLY
-# This is an autogenerated file for dynamic methods in `ActiveSupport::Callbacks`.
-# Please instead update this file by running `bin/tapioca dsl ActiveSupport::Callbacks`.
-
-
-module ActiveSupport::Callbacks
-  include GeneratedInstanceMethods
-
-  mixes_in_class_methods GeneratedClassMethods
-
-  module GeneratedClassMethods
-    def __callbacks; end
-    def __callbacks=(value); end
-    def __callbacks?; end
-  end
-
-  module GeneratedInstanceMethods
-    def __callbacks; end
-    def __callbacks?; end
-  end
-end

app/assets/images/generic-domain.svg

@@ -1,19 +0,0 @@
-<svg
-  width="16"
-  height="16"
-  viewBox="0 0 16 16"
-  fill="none"
-  xmlns="http://www.w3.org/2000/svg"
->
-  <!-- Background circle -->
-  <circle cx="8" cy="8" r="7" fill="#E0E0E0" />
-
-  <!-- Stylized "www" text -->
-  <path
-    d="M4 8.5C4 6.5 5 5.5 6 5.5C7 5.5 8 6.5 8 8.5C8 6.5 9 5.5 10 5.5C11 5.5 12 6.5 12 8.5"
-    stroke="#666666"
-    stroke-width="1.5"
-    stroke-linecap="round"
-    fill="none"
-  />
-</svg>

app/assets/stylesheets/application.tailwind.css

@@ -7,25 +7,17 @@
 }
 
 .sky-section {
-  @apply divide-y divide-slate-300 overflow-hidden border border-slate-300 bg-slate-100 sm:rounded-lg;
+  @apply divide-y divide-slate-300 overflow-hidden border border-slate-300 bg-slate-100 md:rounded-lg;
 }
 
 .section-header {
   @apply px-4 py-3 font-medium text-slate-900;
 }
 
-.sky-section-header {
-  @apply px-4 py-3 font-medium text-slate-900;
-}
-
 .sky-link {
   @apply text-sky-600 underline decoration-dotted transition-colors hover:text-sky-800;
 }
 
-.blue-link {
-  @apply text-blue-600 transition-colors hover:text-blue-800 hover:underline;
-}
-
 .scroll-shadows {
   background:
     /* Shadow Cover TOP */
@@ -47,15 +39,3 @@
     100% 10px;
   background-attachment: local, local, scroll, scroll;
 }
-
-.log-entry-table-header-cell {
-  @apply bg-slate-50 py-1 text-xs font-medium uppercase tracking-wider text-slate-500;
-}
-
-.log-entry-table-row-cell {
-  @apply flex items-center py-1 text-sm;
-}
-
-.rich-text-content blockquote {
-  @apply my-4 border-s-4 border-gray-300 bg-slate-200 p-4 italic leading-relaxed;
-}

app/assets/stylesheets/good_job_custom.css

@@ -3,25 +3,25 @@
   font-weight: bold;
 }
 .ansi-black {
-  color: #333333;
+  color: #000000;
 }
 .ansi-red {
-  color: #cd3333;
+  color: #cd0000;
 }
 .ansi-green {
-  color: #33cd33;
+  color: #00cd00;
 }
 .ansi-yellow {
-  color: #cdcd33;
+  color: #cdcd00;
 }
 .ansi-blue {
-  color: #3333ee;
+  color: #0000ee;
 }
 .ansi-magenta {
-  color: #cd33cd;
+  color: #cd00cd;
 }
 .ansi-cyan {
-  color: #33cdcd;
+  color: #00cdcd;
 }
 .ansi-white {
   color: #e5e5e5;
@@ -32,63 +32,51 @@
   color: #7f7f7f;
 }
 .ansi-bright-red {
-  color: #990000;
+  color: #ff0000;
 }
 .ansi-bright-green {
-  color: #009900;
+  color: #00ff00;
 }
 .ansi-bright-yellow {
-  color: #999900;
+  color: #ffff00;
 }
 .ansi-bright-blue {
-  color: #5c5c99;
+  color: #5c5cff;
 }
 .ansi-bright-magenta {
-  color: #990099;
+  color: #ff00ff;
 }
 .ansi-bright-cyan {
-  color: #009999;
+  color: #00ffff;
 }
 .ansi-bright-white {
-  color: #999999;
+  color: #ffffff;
 }
 
 .log-uuid {
-  min-width: 20px;
   max-width: 100px;
   overflow: hidden;
+  /* white-space: nowrap; */
   text-overflow: ellipsis;
 }
 
-/* All log lines container */
-.good-job-log-lines {
-  overflow-x: auto;
-}
-
-/* Single log line container */
-.good-job-log-line {
+/* Log line container */
+.log-line {
   font-family: monospace;
   font-size: 0.8rem;
   line-height: 1;
   margin: 2px 0;
   padding: 2px 4px;
   display: flex;
-  white-space: nowrap;
-  width: max-content; /* Make width match the content width */
 }
 
-.good-job-log-line:hover {
-  background-color: #ccc;
-}
-
-.good-job-log-line > span {
+.log-line > span {
   display: inline-block;
   white-space: pre;
 }
 
 .good-job-execution-log {
-  color: #333;
-  background: #f0f0f0;
+  background: #3d3d3d;
 }
 
 .text-truncate-link {
@@ -97,35 +85,3 @@
   overflow: hidden;
   text-overflow: ellipsis;
 }
-
-.good-job-arg-name {
-  white-space: nowrap;
-}
-
-.good-job-arg-grid {
-  display: grid;
-  grid-template-columns: auto 1fr;
-}
-
-.good-job-arg-value,
-.good-job-arg-name {
-  padding: 0.35em 0.4em;
-}
-
-.good-job-arg-name,
-.good-job-arg-value {
-  border-bottom: 1px solid #e0e0e0;
-}
-
-.good-job-arg-row {
-  display: contents;
-}
-
-.good-job-arg-row:hover > * {
-  background-color: #ccc;
-}
-
-/* This ensures the last row doesn't have a bottom border */
-.good-job-arg-grid .good-job-arg-row:last-child * {
-  border-bottom: none;
-}

app/controllers/application_controller.rb

@@ -5,17 +5,6 @@ class ApplicationController < ActionController::Base
   include Pundit::Authorization
   include Devise::Controllers::Helpers::ClassMethods
 
-  sig { returns(T.nilable(IpAddressRole)) }
-  def current_ip_address_role
-    @current_ip_address_role ||= IpAddressRole.for_ip(request.remote_ip)
-  end
-  helper_method :current_ip_address_role
-
-  sig { returns(T.nilable(T.any(User, IpAddressRole))) }
-  def pundit_user
-    current_user || current_ip_address_role
-  end
-
   before_action do
     if Rails.env.development? || Rails.env.staging?
       Rack::MiniProfiler.authorize_request
@@ -29,6 +18,13 @@ class ApplicationController < ActionController::Base
 
   protected
 
+  def set_ivfflat_probes!
+    ReduxApplicationRecord.connection.execute("SET ivfflat.max_probes = 10")
+    ReduxApplicationRecord.connection.execute("SET ivfflat.probes = 10")
+  end
+
+  protected
+
   def prometheus_client
     PrometheusExporter::Client.default
   end

app/controllers/blob_entries_controller.rb

@@ -1,227 +1,69 @@
-# typed: strict
+# typed: false
 class BlobEntriesController < ApplicationController
   skip_before_action :authenticate_user!, only: [:show]
 
-  sig { void }
   def show
     thumb = params[:thumb]
-    if thumb.present? && !thumb_params(thumb)
-      raise ActionController::BadRequest.new("invalid thumbnail #{thumb}")
-    end
+    raise("invalid thumb #{thumb}") if thumb.present? && !thumb_params(thumb)
 
-    if thumb.present?
-      expires_dur = 1.week
-    else
-      expires_dur = 1.year
-    end
+    expires_dur = 1.year
     response.headers["Expires"] = expires_dur.from_now.httpdate
     expires_in expires_dur, public: true
 
-    unless stale?(
-             last_modified: Time.at(0),
-             strong_etag: strong_etag_for_request,
-           )
-      return
-    end
+    sha256 = params[:sha256]
+    etag = sha256
+    etag += "-#{thumb}" if thumb
+    return unless stale?(last_modified: Time.at(0), strong_etag: etag)
 
-    sha256 = T.let(params[:sha256], String)
-    raise ActionController::BadRequest.new("no file specified") if sha256.blank?
+    # images, videos, etc
+    blob_entry = BlobEntry.find(HexUtil.hex2bin(sha256))
+    if helpers.is_send_data_content_type?(blob_entry.content_type)
+      if !thumb.blank? &&
+           helpers.is_thumbable_content_type?(blob_entry.content_type)
+        filename = "thumb-#{thumb}-#{sha256}"
+        filename = filename[..File.extname(filename).length]
+        filename += ".jpeg"
 
-    if show_blob_file(sha256, thumb)
-      return
+        width, height = thumb_params(thumb)
+        image =
+          Vips::Image.thumbnail_buffer(
+            blob_entry.contents,
+            width,
+            height: height,
+          )
+        resized_image_contents = image.jpegsave_buffer
+
+        send_data(
+          resized_image_contents,
+          type: "image/jpg",
+          disposition: "inline",
+          filename: filename,
+        )
+      else
+        ext = helpers.ext_for_content_type(blob_entry.content_type)
+        ext = ".#{ext}" if ext
+        send_data(
+          blob_entry.contents,
+          type: blob_entry.content_type,
+          disposition: "inline",
+          filename: "data#{ext}",
+        )
+      end
+    elsif blob_entry.content_type =~ %r{text/plain}
+      render plain: blob_entry.contents
+    elsif blob_entry.content_type.starts_with? "text/html"
+      render html: blob_entry.contents.html_safe
+    elsif blob_entry.content_type.starts_with? "application/json"
+      pretty_json = JSON.pretty_generate(JSON.parse blob_entry.contents)
+      render html:
+               "<html><body><pre>#{pretty_json}</pre></body></html>".html_safe
     else
-      raise ActiveRecord::RecordNotFound
+      render plain: "no renderer for #{blob_entry.content_type}"
     end
   end
 
   private
 
-  sig { params(sha256: String, thumb: T.nilable(String)).returns(T::Boolean) }
-  def show_blob_file(sha256, thumb)
-    if thumb
-      thumb_params = thumb_params(thumb)
-      if thumb_params.nil?
-        raise ActionController::BadRequest.new("invalid thumbnail: #{thumb}")
-      end
-
-      # if the requested format is gif, and the thumbnail type is content-container, we want to
-      # thumbnail the gif into another gif. Else, always thumbnail into a jpeg.
-      file_ext = "jpeg"
-      if params[:format] == "gif" && thumb == "content-container"
-        file_ext = "gif"
-      end
-
-      # content-container may be pre-thumbnailed, see if the file is on the disk
-      if thumb == "content-container" && file_ext == "jpeg"
-        thumbnail_path =
-          Domain::PostFile::Thumbnail.absolute_file_path(
-            sha256,
-            "content_container",
-            0,
-          )
-        if File.exist?(thumbnail_path)
-          send_file(thumbnail_path, type: "image/jpeg", disposition: "inline")
-          return true
-        end
-      end
-
-      width, height = thumb_params
-      filename = "thumb-#{sha256}-#{thumb}.#{file_ext}"
-      cache_key = "vips:#{filename}"
-      thumb_data =
-        Rack::MiniProfiler.step("vips: load from cache") do
-          Rails
-            .cache
-            .fetch(cache_key, expires_in: 1.day) do
-              blob_file = BlobFile.find_by(sha256: HexUtil.hex2bin(sha256))
-              if blob_file
-                content_type =
-                  blob_file.content_type || "application/octet-stream"
-                if helpers.is_renderable_video_type?(content_type)
-                  thumbnail_video_file(blob_file, width, height, file_ext)
-                elsif helpers.is_renderable_image_type?(content_type)
-                  thumbnail_image_file(blob_file, width, height, file_ext)
-                end
-              end
-            end
-        end
-
-      if !thumb_data
-        Rails.cache.delete(cache_key)
-        return false
-      end
-
-      send_data(
-        thumb_data[0],
-        type: thumb_data[1],
-        disposition: "inline",
-        filename: filename,
-      )
-    else
-      blob_file = BlobFile.find_by(sha256: HexUtil.hex2bin(sha256))
-      return false if !blob_file
-
-      content_type = blob_file.content_type || "application/octet-stream"
-      send_file(
-        blob_file.absolute_file_path,
-        type: content_type,
-        disposition: "inline",
-      )
-    end
-
-    return true
-  end
-
-  sig do
-    params(
-      blob_file: BlobFile,
-      width: Integer,
-      height: Integer,
-      thumb: String,
-    ).returns(T.nilable([String, String]))
-  end
-  def thumbnail_video_file(blob_file, width, height, thumb)
-    video_file = blob_file.absolute_file_path
-    temp_thumb_file = Tempfile.new(%w[video-thumb .png])
-    process_result =
-      system(
-        "ffmpegthumbnailer",
-        "-f", # overlay video strip indicator
-        "-i",
-        video_file,
-        "-o",
-        T.must(temp_thumb_file.path),
-        "-s",
-        "#{width}",
-        "-c",
-        "jpeg",
-      )
-    if !process_result
-      temp_thumb_file.unlink
-      return nil
-    end
-
-    thumb_data_tmp = File.read(T.must(temp_thumb_file.path), mode: "rb")
-    temp_thumb_file.unlink
-    [thumb_data_tmp, "image/jpeg"]
-  end
-
-  # Returns a tuple of the thumbnail data and the content type
-  sig do
-    params(
-      blob_file: BlobFile,
-      width: Integer,
-      height: Integer,
-      file_ext: String,
-    ).returns(T.nilable([String, String]))
-  end
-  def thumbnail_image_file(blob_file, width, height, file_ext)
-    blob_file_path = blob_file.absolute_file_path
-
-    if file_ext == "gif"
-      VipsUtil.try_load_gif(
-        blob_file_path,
-        load_gif: -> do
-          Rack::MiniProfiler.step("vips: load gif") do
-            # Use libvips' gifload with n=-1 to load all frames
-            image = Vips::Image.gifload(blob_file_path, n: -1)
-            num_frames = image.get("n-pages")
-            image_width, image_height = image.width, (image.height / num_frames)
-
-            if width >= image_width && height >= image_height
-              logger.info(
-                "gif is already smaller than requested thumbnail size",
-              )
-              return File.binread(blob_file_path), "image/gif"
-            end
-
-            Rack::MiniProfiler.step("vips: thumbnail gif") do
-              image = image.thumbnail_image(width, height: height)
-              image_buffer =
-                image.gifsave_buffer(
-                  dither: 1,
-                  effort: 1,
-                  interframe_maxerror: 16,
-                  interpalette_maxerror: 10,
-                  interlace: true,
-                )
-              [image_buffer, "image/gif"]
-            end
-          end
-        end,
-        on_load_failed: ->(detected_content_type) do
-          case detected_content_type
-          when %r{image/png}
-            thumbnail_image_file(blob_file, width, height, "png")
-          when %r{image/jpeg}, %r{image/jpg}
-            thumbnail_image_file(blob_file, width, height, "jpeg")
-          else
-            raise
-          end
-        end,
-      )
-    else
-      # Original static image thumbnailing logic
-      image_buffer =
-        Rack::MiniProfiler.step("vips: load image") do
-          T.unsafe(Vips::Image).thumbnail(
-            blob_file.absolute_file_path,
-            width,
-            height: height,
-          )
-        end
-
-      Rack::MiniProfiler.step("vips: thumbnail image") do
-        logger.info("rendering thumbnail as jpeg")
-        [
-          T.let(image_buffer.jpegsave_buffer(interlace: true, Q: 95), String),
-          "image/jpeg",
-        ]
-      end
-    end
-  end
-
-  sig { params(thumb: String).returns(T.nilable([Integer, Integer])) }
   def thumb_params(thumb)
     case thumb
     when "32-avatar"
@@ -234,13 +76,6 @@ class BlobEntriesController < ApplicationController
       [400, 300]
     when "medium"
       [800, 600]
-    when "content-container"
-      [768, 2048]
     end
   end
-
-  sig { returns(String) }
-  def strong_etag_for_request
-    [params[:sha256], params[:thumb], params[:format]].compact.join("-")
-  end
 end

app/controllers/domain/e621/posts_controller.rb

@@ -0,0 +1,6 @@
+# typed: true
+class Domain::E621::PostsController < ApplicationController
+  def show
+    @post = Domain::E621::Post.find_by!(e621_id: params[:e621_id])
+  end
+end

app/controllers/domain/fa/api_controller.rb

@@ -0,0 +1,423 @@
+# typed: true
+class Domain::Fa::ApiController < ApplicationController
+  skip_before_action :authenticate_user!
+  before_action :validate_api_token!
+
+  skip_before_action :verify_authenticity_token,
+                     only: %i[enqueue_objects object_statuses]
+
+  skip_before_action :validate_api_token!, only: %i[search_user_names]
+
+  def search_user_names
+    name = params[:name]
+    limit = (params[:limit] || 5).to_i.clamp(0, 15)
+    users = users_for_name(name, limit: limit)
+    if !Rails.env.production? && name == "error"
+      render status: 500, json: { error: "an error!" }
+    else
+      render json: { users: users }
+    end
+  end
+
+  def object_statuses
+    fa_ids = (params[:fa_ids] || []).map(&:to_i)
+    url_names = (params[:url_names] || [])
+
+    jobs_async =
+      GoodJob::Job
+        .select(:id, :queue_name, :serialized_params)
+        .where(queue_name: "manual", finished_at: nil)
+        .where(
+          [
+            "(serialized_params->'exception_executions' = '{}')",
+            "(serialized_params->'exception_executions' is null)",
+          ].join(" OR "),
+        )
+        .load_async
+
+    users_async = Domain::Fa::User.where(url_name: url_names).load_async
+
+    fa_id_to_post =
+      Domain::Fa::Post
+        .includes(:file)
+        .where(fa_id: fa_ids)
+        .map { |post| [post.fa_id, post] }
+        .to_h
+
+    posts_response = {}
+    users_response = {}
+
+    fa_ids.each do |fa_id|
+      post = fa_id_to_post[fa_id]
+
+      post_response =
+        T.let(
+          {
+            terminal_state: false,
+            seen_at: time_ago_or_never(post&.created_at),
+            scanned_at: "never",
+            downloaded_at: "never",
+          },
+          T::Hash[Symbol, T.untyped],
+        )
+
+      if post
+        post_response[:info_url] = domain_fa_post_url(fa_id: post.fa_id)
+        post_response[:scanned_at] = time_ago_or_never(post.scanned_at)
+
+        if post.file.present?
+          post_response[:downloaded_at] = time_ago_or_never(
+            post.file&.created_at,
+          )
+          post_response[:state] = "have_file"
+          post_response[:terminal_state] = true
+        elsif post.scanned?
+          post_response[:state] = "scanned_post"
+        else
+          post_response[:state] = post.state
+        end
+      else
+        post_response[:state] = "not_seen"
+      end
+
+      posts_response[fa_id] = post_response
+    end
+
+    url_name_to_user = users_async.map { |user| [user.url_name, user] }.to_h
+
+    url_names.each do |url_name|
+      user = url_name_to_user[url_name]
+
+      if user
+        user_response = {
+          created_at: time_ago_or_never(user.created_at),
+          scanned_gallery_at: time_ago_or_never(user.scanned_gallery_at),
+          scanned_page_at: time_ago_or_never(user.scanned_page_at),
+        }
+        states = []
+        states << "page" unless user.due_for_page_scan?
+        states << "gallery" unless user.due_for_gallery_scan?
+        states << "seen" if states.empty?
+
+        user_response[:state] = states.join(",")
+
+        if user.scanned_gallery_at && user.scanned_page_at
+          user_response[:terminal_state] = true
+        end
+      else
+        user_response = { state: "not_seen", terminal_state: false }
+      end
+      users_response[url_name] = user_response
+    end
+
+    queue_depths = Hash.new { |hash, key| hash[key] = 0 }
+
+    jobs_async.each do |job|
+      queue_depths[job.serialized_params["job_class"]] += 1
+    end
+
+    queue_depths =
+      queue_depths
+        .map do |key, value|
+          [
+            key
+              .delete_prefix("Domain::Fa::Job::")
+              .split("::")
+              .last
+              .underscore
+              .delete_suffix("_job")
+              .gsub("_", " "),
+            value,
+          ]
+        end
+        .to_h
+
+    render json: {
+             posts: posts_response,
+             users: users_response,
+             queues: {
+               total_depth: queue_depths.values.sum,
+               depths: queue_depths,
+             },
+           }
+  end
+
+  def enqueue_objects
+    @enqueue_counts ||= Hash.new { |h, k| h[k] = 0 }
+
+    fa_ids = (params[:fa_ids] || []).map(&:to_i)
+    url_names = (params[:url_names] || [])
+    url_names_to_enqueue = Set.new(params[:url_names_to_enqueue] || [])
+
+    fa_id_to_post =
+      Domain::Fa::Post
+        .includes(:file)
+        .where(fa_id: fa_ids)
+        .map { |post| [post.fa_id, post] }
+        .to_h
+
+    url_name_to_user =
+      Domain::Fa::User
+        .where(url_name: url_names)
+        .map { |user| [user.url_name, user] }
+        .to_h
+
+    fa_ids.each do |fa_id|
+      post = fa_id_to_post[fa_id]
+      defer_post_scan(post, fa_id)
+    end
+
+    url_names.each do |url_name|
+      user = url_name_to_user[url_name]
+      defer_user_scan(user, url_name, url_names_to_enqueue.include?(url_name))
+    end
+
+    enqueue_deferred!
+
+    render json: {
+             post_scans: @enqueue_counts[Domain::Fa::Job::ScanPostJob],
+             post_files: @enqueue_counts[Domain::Fa::Job::ScanFileJob],
+             user_pages: @enqueue_counts[Domain::Fa::Job::UserPageJob],
+             user_galleries: @enqueue_counts[Domain::Fa::Job::UserGalleryJob],
+           }
+  end
+
+  def similar_users
+    url_name = params[:url_name]
+    exclude_url_name = params[:exclude_url_name]
+
+    user = Domain::Fa::User.find_by(url_name: url_name)
+    if user.nil?
+      render status: 404,
+             json: {
+               error: "user '#{url_name}' not found",
+               error_type: "user_not_found",
+             }
+      return
+    end
+
+    all_similar_users = helpers.similar_users_by_followed(user, limit: 10)
+
+    if all_similar_users.nil?
+      render status: 500,
+             json: {
+               error:
+                 "user '#{url_name}' has not had recommendations computed yet",
+               error_type: "recs_not_computed",
+             }
+      return
+    end
+
+    all_similar_users = users_list_to_similar_list(all_similar_users)
+
+    not_followed_similar_users = nil
+    if exclude_url_name
+      exclude_folowed_by_user =
+        Domain::Fa::User.find_by(url_name: exclude_url_name)
+      not_followed_similar_users =
+        if exclude_folowed_by_user.nil?
+          # TODO - enqueue a manual UserFollowsJob for this user and have client
+          # re-try the request later
+          {
+            error: "user '#{exclude_url_name}' not found",
+            error_type: "exclude_user_not_found",
+          }
+        elsif exclude_folowed_by_user.scanned_follows_at.nil?
+          {
+            error:
+              "user '#{exclude_url_name}' followers list hasn't been scanned",
+            error_type: "exclude_user_not_scanned",
+          }
+        else
+          users_list_to_similar_list(
+            helpers.similar_users_by_followed(
+              user,
+              limit: 10,
+              exclude_followed_by: exclude_folowed_by_user,
+            ),
+          )
+        end
+    end
+
+    render json: {
+             all: all_similar_users,
+             not_followed: not_followed_similar_users,
+           }
+  end
+
+  private
+
+  def get_best_user_page_http_log_entry_for(user)
+    for_path =
+      proc do |uri_path|
+        HttpLogEntry
+          .where(
+            uri_scheme: "https",
+            uri_host: "www.furaffinity.net",
+            uri_path: uri_path,
+          )
+          .order(created_at: :desc)
+          .first
+          &.response
+      end
+
+    for_hle_id =
+      proc { |hle_id| hle_id && HttpLogEntry.find_by(id: hle_id)&.response }
+
+    # older versions don't end in a trailing slash
+    hle_id = user.log_entry_detail && user.log_entry_detail["last_user_page_id"]
+    for_hle_id.call(hle_id) || for_path.call("/user/#{user.url_name}/") ||
+      for_path.call("/user/#{user.url_name}")
+  end
+
+  def defer_post_scan(post, fa_id)
+    if !post || !post.scanned?
+      defer_manual(Domain::Fa::Job::ScanPostJob, { fa_id: fa_id }, -17)
+    end
+
+    if post && post.file_uri && !post.file.present?
+      return(
+        defer_manual(
+          Domain::Fa::Job::ScanFileJob,
+          { post: post },
+          -15,
+          "static_file",
+        )
+      )
+    end
+  end
+
+  def defer_user_scan(user, url_name, highpri)
+    if !user || user.due_for_page_scan?
+      defer_manual(
+        Domain::Fa::Job::UserPageJob,
+        { url_name: url_name },
+        highpri ? -16 : -6,
+      )
+      return
+    end
+
+    if !user || user.due_for_gallery_scan?
+      defer_manual(
+        Domain::Fa::Job::UserGalleryJob,
+        { url_name: url_name },
+        highpri ? -14 : -4,
+      )
+      return
+    end
+
+    false
+  end
+
+  def defer_manual(klass, args, priority, queue = "manual")
+    @@enqueue_deduper ||= Set.new
+    return unless @@enqueue_deduper.add?([klass, args, priority])
+
+    @deferred_jobs ||= []
+    @deferred_jobs << [klass, args, priority, queue]
+    @enqueue_counts[klass] += 1
+  end
+
+  def enqueue_deferred!
+    GoodJob::Bulk.enqueue do
+      while job = (@deferred_jobs || []).shift
+        klass, args, priority, queue = job
+        klass.set(priority: priority, queue: queue).perform_later(args)
+      end
+    end
+  end
+
+  def time_ago_or_never(time)
+    if time
+      helpers.time_ago_in_words(time, include_seconds: true) + " ago"
+    else
+      "never"
+    end
+  end
+
+  def users_for_name(name, limit: 10)
+    users =
+      Domain::Fa::User
+        .where(
+          [
+            "(name ilike :name) OR (url_name ilike :name)",
+            { name: "#{ReduxApplicationRecord.sanitize_sql_like(name)}%" },
+          ],
+        )
+        .includes(:avatar)
+        .select(:id, :state, :state_detail, :log_entry_detail, :name, :url_name)
+        .select(
+          "(SELECT COUNT(*) FROM domain_fa_posts WHERE creator_id = domain_fa_users.id) as num_posts",
+        )
+        .order(name: :asc)
+        .limit(limit)
+
+    users.map do |user|
+      {
+        id: user.id,
+        name: user.name,
+        url_name: user.url_name,
+        thumb: helpers.fa_user_avatar_path(user, thumb: "64-avatar"),
+        show_path: domain_fa_user_path(user.url_name),
+        # `num_posts` is a manually added column, so we need to use T.unsafe to
+        # access it
+        num_posts: T.unsafe(user).num_posts,
+      }
+    end
+  end
+
+  def users_list_to_similar_list(users_list)
+    users_list.map do |user|
+      profile_thumb_url = user.avatar&.file_uri&.to_s
+      profile_thumb_url ||
+        begin
+          profile_page_response = get_best_user_page_http_log_entry_for(user)
+          if profile_page_response
+            parser =
+              Domain::Fa::Parser::Page.new(
+                profile_page_response.contents,
+                require_logged_in: false,
+              )
+            profile_thumb_url = parser.user_page.profile_thumb_url
+          else
+            if user.due_for_follows_scan?
+              Domain::Fa::Job::UserFollowsJob.set(
+                { priority: -20 },
+              ).perform_later({ user: user })
+            end
+            if user.due_for_page_scan?
+              Domain::Fa::Job::UserPageJob.set({ priority: -20 }).perform_later(
+                { user: user },
+              )
+            end
+          end
+        rescue StandardError
+          logger.error("error getting profile_thumb_url: #{$!.message}")
+        end
+
+      {
+        name: user.name,
+        url_name: user.url_name,
+        profile_thumb_url: profile_thumb_url,
+        url: "https://www.furaffinity.net/user/#{user.url_name}/",
+      }
+    end
+  end
+
+  API_TOKENS = {
+    "a4eb03ac-b33c-439c-9b51-a834d1c5cf48" => "dymk",
+    "56cc81fe-8c00-4436-8981-4580eab00e66" => "taargus",
+    "9c38727f-f11d-41de-b775-0effd86d520c" => "xjal",
+    "e38c568f-a24d-4f26-87f0-dfcd898a359d" => "fyacin",
+    "41fa1144-d4cd-11ed-afa1-0242ac120002" => "soft_fox_lad",
+    "9b3cf444-5913-4efb-9935-bf26501232ff" => "syfaro",
+  }
+
+  def validate_api_token!
+    api_token = request.params[:api_token]
+    api_user_name = API_TOKENS[api_token]
+    return if api_user_name
+    return if VpnOnlyRouteConstraint.new.matches?(request)
+    render status: 403, json: { error: "not authenticated" }
+  end
+end

app/controllers/domain/fa/posts_controller.rb

@@ -0,0 +1,139 @@
+# typed: true
+
+class Domain::Fa::PostsController < ApplicationController
+  before_action :set_ivfflat_probes!, only: %i[show]
+  before_action :set_domain_fa_post, only: %i[show scan_post]
+
+  skip_before_action :verify_authenticity_token,
+                     only: %i[try_scan_post try_scan_posts]
+
+  skip_before_action :authenticate_user!, only: %i[show index]
+
+  # This action is always scoped to a user, so the :user_url_name parameter is required.
+  # GET /domain/fa/users/:user_url_name/posts
+  def index
+    @user = Domain::Fa::User.find_by!(url_name: params[:user_url_name])
+    relation = policy_scope(@user.posts)
+    @posts =
+      relation
+        .includes(:creator, :file)
+        .order(fa_id: :desc)
+        .page(params[:page])
+        .per(50)
+        .without_count
+  end
+
+  # GET /domain/fa/posts/:fa_id
+  def show
+  end
+
+  # GET /domain/fa/posts/:fa_id/favorites
+  def favorites
+    @post = Domain::Fa::Post.find_by!(fa_id: params[:fa_id])
+  end
+
+  def scan_post
+    if try_enqueue_post_scan(@post, @post.fa_id)
+      redirect_to domain_fa_post_path(@post.fa_id), notice: "Enqueued for scan"
+    else
+      redirect_to domain_fa_post_path(@post.fa_id), notice: "Already scanned"
+    end
+  end
+
+  def try_scan_post
+    fa_id = params[:fa_id]&.to_i || raise("need fa_id parameter")
+    post = Domain::Fa::Post.find_by(fa_id: fa_id)
+    enqueued = try_enqueue_post_scan(post, fa_id)
+
+    if post && (file = post.file).present?
+      state_string =
+        "downloaded #{helpers.time_ago_in_words(file.created_at, include_seconds: true)} ago"
+    elsif post && post.scanned?
+      state_string =
+        "scanned #{helpers.time_ago_in_words(post.scanned_at, include_seconds: true)} ago"
+    else
+      state_string = []
+      !post ? state_string << "not seen" : state_string << "#{post.state}"
+
+      state_string << "enqueued" if enqueued
+
+      state_string = state_string.join(", ")
+    end
+
+    render json: {
+             enqueued: enqueued,
+             title: post&.title,
+             state: state_string,
+             is_terminal_state: post&.scanned? && post.file&.present? || false,
+           }
+  end
+
+  def try_scan_posts
+    Rails.logger.info "params: #{params.inspect}"
+    fa_ids = params[:fa_ids].map(&:to_i)
+    fa_id_to_post =
+      Domain::Fa::Post
+        .where(fa_id: fa_ids)
+        .map { |post| [post.fa_id, post] }
+        .to_h
+
+    response = {}
+
+    fa_ids.each do |fa_id|
+      post = fa_id_to_post[fa_id]
+      if post.nil?
+        state = "not_seen"
+      elsif post.file.present?
+        state = "have_file"
+      elsif post.scanned?
+        state = "scanned"
+      else
+        state = "state_#{post.state}"
+      end
+
+      response[fa_id] = {
+        state: state,
+        enqueued: try_enqueue_post_scan(post, fa_id),
+      }
+    end
+    render json: response
+  end
+
+  private
+
+  def try_enqueue_post_scan(post, fa_id)
+    @@already_enqueued_fa_ids ||= Set.new
+    unless @@already_enqueued_fa_ids.add?(fa_id)
+      Rails.logger.info "Already enqueued #{fa_id}, skipping"
+      return false
+    end
+
+    if !post || !post.scanned?
+      Rails.logger.info "Enqueue scan #{fa_id}"
+      Domain::Fa::Job::ScanPostJob.set(
+        priority: -15,
+        queue: "manual",
+      ).perform_later({ fa_id: fa_id })
+      return true
+    end
+
+    if post && post.file_uri && !post.file.present?
+      Rails.logger.info "Enqueue file #{fa_id}"
+      Domain::Fa::Job::ScanFileJob.set(
+        priority: -15,
+        queue: "manual",
+      ).perform_later({ post: post })
+      return true
+    end
+
+    false
+  end
+
+  # Use callbacks to share common setup or constraints between actions.
+  def set_domain_fa_post
+    @post =
+      Domain::Fa::Post.includes(:creator, file: :response).find_by!(
+        fa_id: params[:fa_id],
+      )
+  end
+end

app/controllers/domain/fa/users_controller.rb

@@ -0,0 +1,28 @@
+# typed: true
+
+class Domain::Fa::UsersController < ApplicationController
+  before_action :set_ivfflat_probes!, only: %i[show]
+  before_action :set_user, only: %i[show]
+  skip_before_action :authenticate_user!, only: %i[show]
+
+  # GET /domain/fa/users or /domain/fa/users.json
+  def index
+    authorize Domain::Fa::User
+    @users =
+      policy_scope(Domain::Fa::User).includes({ avatar: [:file] }).page(
+        params[:page],
+      )
+  end
+
+  # GET /domain/fa/users/1 or /domain/fa/users/1.json
+  def show
+    authorize @user
+  end
+
+  private
+
+  # Use callbacks to share common setup or constraints between actions.
+  def set_user
+    @user = Domain::Fa::User.find_by(url_name: params[:url_name])
+  end
+end

app/controllers/domain/inkbunny/posts_controller.rb

@@ -0,0 +1,19 @@
+# typed: false
+class Domain::Inkbunny::PostsController < ApplicationController
+  skip_before_action :authenticate_user!, only: %i[show index]
+
+  def index
+    relation = Domain::Inkbunny::Post.includes(:creator, :files)
+
+    if params[:user_id].present?
+      @user = Domain::Inkbunny::User.find(params[:user_id])
+      relation = relation.where(creator: @user)
+    end
+
+    @posts = relation.order(ib_post_id: :desc).page(params[:page]).per(50)
+  end
+
+  def show
+    @post = Domain::Inkbunny::Post.find_by!(ib_post_id: params[:ib_post_id])
+  end
+end

app/controllers/domain/inkbunny/users_controller.rb

@@ -0,0 +1,6 @@
+# typed: true
+class Domain::Inkbunny::UsersController < ApplicationController
+  def show
+    @user = Domain::Inkbunny::User.find_by(name: params[:name])
+  end
+end

app/controllers/domain/post_groups_controller.rb

@@ -1,26 +0,0 @@
-# typed: true
-
-class Domain::PostGroupsController < DomainController
-  extend T::Sig
-  extend T::Helpers
-
-  skip_before_action :authenticate_user!, only: %i[show]
-  before_action :set_post_group!, only: %i[show]
-
-  # GET /pools/:id
-  sig(:final) { void }
-  def show
-    authorize @post_group
-  end
-
-  private
-
-  sig { override.returns(DomainController::DomainParamConfig) }
-  def self.param_config
-    DomainController::DomainParamConfig.new(
-      post_group_id_param: :id,
-      post_id_param: :domain_post_id,
-      user_id_param: :domain_user_id,
-    )
-  end
-end

app/controllers/domain/posts_controller.rb

@@ -1,254 +0,0 @@
-# typed: true
-
-require "open-uri"
-require "tempfile"
-require "base64"
-
-class Domain::PostsController < DomainController
-  extend T::Sig
-  extend T::Helpers
-
-  skip_before_action :authenticate_user!,
-                     only: %i[
-                       show
-                       index
-                       user_favorite_posts
-                       user_created_posts
-                       visual_search
-                       visual_results
-                     ]
-  before_action :set_post!, only: %i[show]
-  before_action :set_user!, only: %i[user_created_posts]
-  before_action :set_post_group!, only: %i[posts_in_group]
-
-  class PostsIndexViewConfig < T::ImmutableStruct
-    include T::Struct::ActsAsComparable
-
-    const :show_domain_filters, T::Boolean
-    const :show_creator_links, T::Boolean
-    const :index_type_header, String
-  end
-
-  sig { void }
-  def initialize
-    super
-    @posts_index_view_config =
-      PostsIndexViewConfig.new(
-        show_domain_filters: false,
-        show_creator_links: false,
-        index_type_header: "all_posts",
-      )
-  end
-
-  # GET /posts
-  sig(:final) { void }
-  def index
-    @posts_index_view_config =
-      PostsIndexViewConfig.new(
-        show_domain_filters: true,
-        show_creator_links: true,
-        index_type_header: "all_posts",
-      )
-    authorize Domain::Post
-    @posts = posts_relation(Domain::Post.all).without_count
-    active_sources = (params[:sources] || DomainSourceHelper.all_source_names)
-    unless DomainSourceHelper.has_all_sources?(active_sources)
-      postable_types =
-        DomainSourceHelper.source_names_to_class_names(active_sources)
-      @posts = @posts.where(type: postable_types) if postable_types.any?
-    end
-  end
-
-  # GET /posts/:id
-  sig(:final) { void }
-  def show
-    authorize @post
-  end
-
-  sig(:final) { void }
-  def user_created_posts
-    @posts_index_view_config =
-      PostsIndexViewConfig.new(
-        show_domain_filters: false,
-        show_creator_links: false,
-        index_type_header: "user_created",
-      )
-
-    @user = T.must(@user)
-    authorize @user
-    @posts = posts_relation(@user.posts)
-    authorize @posts
-    render :index
-  end
-
-  sig(:final) { void }
-  def posts_in_group
-    @posts_index_view_config =
-      PostsIndexViewConfig.new(
-        show_domain_filters: false,
-        show_creator_links: true,
-        index_type_header: "posts_in_group",
-      )
-
-    authorize @post_group
-    @posts = posts_relation(T.must(@post_group).posts)
-    render :index
-  end
-
-  # GET /posts/visual_search
-  sig(:final) { void }
-  def visual_search
-    authorize Domain::Post
-  end
-
-  sig { params(content_type: T.nilable(String)).returns(T::Boolean) }
-  def check_content_type!(content_type)
-    return false unless content_type
-
-    ret =
-      Domain::PostFile::Thumbnail::THUMBABLE_CONTENT_TYPES.any? do |type|
-        content_type.match?(type)
-      end
-
-    unless ret
-      flash.now[:error] = "The uploaded file is not a valid image format."
-      render :visual_search
-    end
-
-    ret
-  end
-
-  # POST /posts/visual_search
-  sig(:final) { void }
-  def visual_results
-    authorize Domain::Post
-
-    # Process the uploaded image or URL
-    file_result = process_image_input
-    return unless file_result
-    file_path, content_type = file_result
-
-    # Create thumbnail for the view if possible
-    tmp_dir = Dir.mktmpdir("visual-search")
-    thumbs_and_fingerprints =
-      helpers.generate_fingerprints(file_path, content_type, tmp_dir)
-    first_thumb_and_fingerprint = thumbs_and_fingerprints&.first
-    if thumbs_and_fingerprints.nil? || first_thumb_and_fingerprint.nil?
-      flash.now[:error] = "Error generating fingerprints"
-      render :visual_search
-      return
-    end
-    logger.info("generated #{thumbs_and_fingerprints.length} thumbs")
-
-    @uploaded_image_data_uri =
-      helpers.create_image_thumbnail_data_uri(
-        first_thumb_and_fingerprint.thumb_path,
-        "image/jpeg",
-      )
-    @uploaded_detail_hash_value = first_thumb_and_fingerprint.detail_fingerprint
-    before = Time.now
-
-    similar_fingerprints =
-      helpers.find_similar_fingerprints(
-        thumbs_and_fingerprints.map(&:to_fingerprint_and_detail),
-      ).take(10)
-
-    @time_taken = Time.now - before
-
-    @matches = similar_fingerprints
-    @good_matches =
-      similar_fingerprints.select { |f| f.similarity_percentage >= 80 }
-    @bad_matches =
-      similar_fingerprints.select { |f| f.similarity_percentage < 80 }
-
-    @matches = @good_matches if @good_matches.any?
-  ensure
-    # Clean up any temporary files
-    FileUtils.rm_rf(tmp_dir) if tmp_dir
-  end
-
-  private
-
-  # Process the uploaded file or URL and return [image_path, content_type] or nil on failure
-  sig { returns(T.nilable([String, String])) }
-  def process_image_input
-    if params[:image_file].present?
-      process_uploaded_file
-    elsif params[:image_url].present?
-      process_image_url
-    else
-      flash.now[:error] = "Please upload an image or provide an image URL."
-      render :visual_search
-      nil
-    end
-  end
-
-  # Process an uploaded file and return [image_path, content_type] or nil on failure
-  sig { returns(T.nilable([String, String])) }
-  def process_uploaded_file
-    image_file = params[:image_file]
-    content_type = T.must(image_file.content_type)
-
-    return nil unless check_content_type!(content_type)
-
-    image_path = T.must(image_file.tempfile.path)
-    [image_path, content_type]
-  end
-
-  # Process an image URL and return [image_path, content_type] or nil on failure
-  sig { returns(T.nilable([String, String])) }
-  def process_image_url
-    # Download the image to a temporary file
-    image_url = params[:image_url]
-    image_io = URI.open(image_url)
-
-    if image_io.nil?
-      flash.now[:error] = "The URL does not point to a valid image format."
-      render :visual_search
-      return nil
-    end
-
-    content_type = T.must(T.unsafe(image_io).content_type)
-    return nil unless check_content_type!(content_type)
-
-    # Save to temp file
-    extension = helpers.extension_for_content_type(content_type) || "jpg"
-    @temp_file = Tempfile.new(["image", ".#{extension}"])
-    @temp_file.binmode
-    image_data = image_io.read
-    @temp_file.write(image_data)
-    @temp_file.close
-
-    image_path = T.must(@temp_file.path)
-    [image_path, content_type]
-  rescue StandardError => e
-    Rails.logger.error("Error processing image URL: #{e.message}")
-    flash.now[:error] = "Error downloading search image"
-    render :visual_search
-    nil
-  end
-
-  sig { override.returns(DomainController::DomainParamConfig) }
-  def self.param_config
-    DomainController::DomainParamConfig.new(
-      post_id_param: :id,
-      user_id_param: :domain_user_id,
-      post_group_id_param: :domain_post_group_id,
-    )
-  end
-
-  sig(:final) do
-    params(
-      starting_relation: ActiveRecord::Relation,
-      skip_ordering: T::Boolean,
-    ).returns(
-      T.all(ActiveRecord::Relation, Kaminari::ActiveRecordRelationMethods),
-    )
-  end
-  def posts_relation(starting_relation, skip_ordering: false)
-    relation = starting_relation
-    relation = T.unsafe(policy_scope(relation)).page(params[:page]).per(50)
-    relation = relation.order("posted_at DESC NULLS LAST") unless skip_ordering
-    relation
-  end
-end

app/controllers/domain/user_job_events_controller.rb

@@ -1,120 +0,0 @@
-# typed: strict
-class Domain::UserJobEventsController < DomainController
-  extend T::Sig
-
-  before_action :set_user!
-
-  sig { void }
-  def tracked_objects_kinds
-    @kinds =
-      T.let(
-        Domain::UserJobEvent::AddTrackedObject.kinds.keys.map(&:to_s),
-        T.nilable(T::Array[String]),
-      )
-
-    @kind_counts =
-      T.let(
-        @user&.add_tracked_objects&.reorder(nil)&.group(:kind)&.count,
-        T.nilable(T::Hash[String, Integer]),
-      )
-
-    @kinds_most_at =
-      T.let(
-        @user
-          &.add_tracked_objects
-          &.reorder(nil)
-          &.group(:kind)
-          &.maximum(:requested_at),
-        T.nilable(T::Hash[String, Time]),
-      )
-  end
-
-  sig { void }
-  def tracked_objects
-    set_and_validate_kind!
-    @tracked_objects =
-      T.let(
-        T
-          .must(@user)
-          .add_tracked_objects
-          .includes(:log_entry)
-          .where(kind: @kind)
-          .sort_by { |a| -a.requested_at.to_i },
-        T.untyped,
-      )
-  end
-
-  sig { void }
-  def backfill_scan_job
-    set_and_validate_kind!
-    @user = T.must(@user)
-
-    unless @user.is_a?(Domain::User::FaUser)
-      flash[:error] = "This user is not a FurAffinity user"
-      redirect_to tracked_objects_domain_user_job_events_path(
-                    @user,
-                    kind: @kind,
-                  )
-      return
-    end
-
-    case @kind
-    when "favs"
-      now = Time.now
-      stats = Domain::Fa::BackfillTrackedObjectUserFavs.new(user: @user).run
-      flash[
-        :success
-      ] = "Backfilled #{@user.url_name} favs, #{stats.total_created} favs scans created, #{stats.total_favs} favs, loaded #{stats.total_hles} logs, took #{helpers.distance_of_time_in_words_to_now(now, include_seconds: true)}"
-    end
-
-    redirect_to tracked_objects_domain_user_job_events_path(@user, kind: @kind)
-  end
-
-  sig { void }
-  def enqueue_scan_job
-    set_and_validate_kind!
-    @user = T.must(@user)
-
-    unless @user.is_a?(Domain::User::FaUser)
-      flash[:error] = "This user is not a FurAffinity user"
-      redirect_to tracked_objects_domain_user_job_events_path(
-                    @user,
-                    kind: @kind,
-                  )
-      return
-    end
-
-    case @kind
-    when "favs"
-      flash[:success] = "Enqueued scan job for #{@user.url_name} favs"
-      Domain::Fa::Job::FavsJob.set(queue: "manual").perform_later(
-        user: @user,
-        force_scan: true,
-      )
-    else
-      flash[:error] = "Unimplemented kind: #{@kind}"
-    end
-
-    redirect_to tracked_objects_domain_user_job_events_path(@user, kind: @kind)
-  end
-
-  private
-
-  sig { override.returns(DomainController::DomainParamConfig) }
-  def self.param_config
-    DomainController::DomainParamConfig.new(
-      user_id_param: :domain_user_id,
-      post_id_param: :domain_post_id,
-      post_group_id_param: :domain_post_group_id,
-    )
-  end
-
-  sig { void }
-  def set_and_validate_kind!
-    @kind = T.let(params[:kind], T.nilable(String))
-    raise ActionController::RoutingError, "Not Found" if @kind.blank?
-    unless Domain::UserJobEvent::AddTrackedObject.kinds.include?(@kind)
-      raise ActionController::RoutingError, "Not Found"
-    end
-  end
-end

app/controllers/domain/user_post_favs_controller.rb

@@ -1,29 +0,0 @@
-# typed: true
-# frozen_string_literal: true
-
-class Domain::UserPostFavsController < DomainController
-  before_action :set_user!, only: %i[favorites]
-
-  def self.param_config
-    DomainParamConfig.new(
-      post_id_param: :domain_post_id,
-      user_id_param: :domain_user_id,
-      post_group_id_param: :domain_post_group_id,
-    )
-  end
-
-  sig { void }
-  def favorites
-    @posts_index_view_config =
-      Domain::PostsController::PostsIndexViewConfig.new(
-        show_domain_filters: false,
-        show_creator_links: true,
-        index_type_header: "user_favorites",
-      )
-    user = T.cast(@user, Domain::User)
-    @user_post_favs =
-      user.user_post_favs.includes(:post).page(params[:page]).per(50)
-    authorize @user_post_favs
-    render :favorites
-  end
-end

app/controllers/domain/users_controller.rb

@@ -1,290 +0,0 @@
-# typed: true
-class Domain::UsersController < DomainController
-  extend T::Sig
-  extend T::Helpers
-
-  before_action :set_user!,
-                only: %i[show followed_by following monitor_bluesky_user]
-  before_action :set_post!, only: %i[users_faving_post]
-  skip_before_action :authenticate_user!,
-                     only: %i[
-                       show
-                       search_by_name
-                       users_faving_post
-                       similar_users
-                     ]
-
-  # GET /users
-  sig(:final) { void }
-  def index
-    authorize Domain::User
-    @users = policy_scope(Domain::User).order(created_at: :desc)
-  end
-
-  sig(:final) { void }
-  def followed_by
-    @user = T.must(@user)
-    authorize @user
-    @users =
-      @user
-        .followed_by_users
-        .includes(avatar: :log_entry)
-        .page(params[:page])
-        .per(50)
-    @index_type = :followed_by
-    render :index
-  end
-
-  sig(:final) { void }
-  def following
-    @user = T.must(@user)
-    authorize @user
-    @users =
-      @user
-        .followed_users
-        .includes(avatar: :log_entry)
-        .page(params[:page])
-        .per(50)
-    @index_type = :following
-    render :index
-  end
-
-  sig(:final) { void }
-  def users_faving_post
-    @post = T.must(@post)
-    authorize @post
-    @users =
-      T
-        .unsafe(@post)
-        .faving_users
-        .includes(avatar: :log_entry)
-        .page(params[:page])
-        .per(50)
-
-    @index_type = :users_faving_post
-    render :index
-  end
-
-  # GET /users/:id
-  sig(:final) { void }
-  def show
-    authorize @user
-  end
-
-  sig(:final) { void }
-  def search_by_name
-    authorize Domain::User
-    name = params[:name]&.downcase
-    name = ReduxApplicationRecord.sanitize_sql_like(name)
-
-    if name.starts_with?("did:plc:") || name.starts_with?("did:pkh:")
-      @user_search_names =
-        Domain::UserSearchName
-          .select(
-            "domain_user_search_names.*, domain_users.*, domain_users_bluesky_aux.did",
-          )
-          .select(
-            "levenshtein(domain_users_bluesky_aux.did, '#{name}') as distance",
-          )
-          .where(
-            user: Domain::User::BlueskyUser.where("did LIKE ?", "#{name}%"),
-          )
-          .joins(:user)
-          .limit(10)
-      return
-    end
-
-    @user_search_names =
-      Domain::UserSearchName
-        .select("domain_user_search_names.*, domain_users.*")
-        .select("levenshtein(name, '#{name}') as distance")
-        .select(
-          "(SELECT COUNT(*) FROM domain_user_post_creations dupc WHERE dupc.user_id = domain_users.id) as num_posts",
-        )
-        .joins(:user)
-        .where(
-          "(name ilike ?) OR (similarity(dmetaphone(name), dmetaphone(?)) > 0.8)",
-          "%#{name}%",
-          name,
-        )
-        .where(
-          "NOT EXISTS (
-            SELECT 1
-            FROM domain_user_search_names dns2
-            WHERE dns2.user_id = domain_user_search_names.user_id
-            AND levenshtein(dns2.name, ?) < levenshtein(domain_user_search_names.name, ?)
-          )",
-          name,
-          name,
-        )
-        .order("distance ASC")
-        .limit(10)
-  end
-
-  sig { void }
-  def similar_users
-    url_name = params[:url_name]
-    exclude_url_name = params[:exclude_url_name]
-
-    user = Domain::User::FaUser.find_by(url_name: url_name)
-    if user.nil?
-      render status: 404,
-             json: {
-               error: "user '#{url_name}' not found",
-               error_type: "user_not_found",
-             }
-      return
-    end
-
-    all_similar_users =
-      users_similar_to_by_followers(user, limit: 10).map do |u|
-        user_to_similarity_entry(u)
-      end
-
-    if all_similar_users.nil?
-      render status: 500,
-             json: {
-               error:
-                 "user '#{url_name}' has not had recommendations computed yet",
-               error_type: "recs_not_computed",
-             }
-      return
-    end
-
-    not_followed_similar_users = nil
-    if exclude_url_name
-      exclude_followed_by =
-        Domain::User::FaUser.find_by(url_name: exclude_url_name)
-      if exclude_followed_by.nil?
-        render status: 500,
-               json: {
-                 error: "user '#{exclude_url_name}' not found",
-                 error_type: "exclude_user_not_found",
-               }
-        return
-      elsif exclude_followed_by.scanned_follows_at.nil?
-        render status: 500,
-               json: {
-                 error:
-                   "user '#{exclude_url_name}' followers list hasn't been scanned",
-                 error_type: "exclude_user_not_scanned",
-               }
-        return
-      else
-        not_followed_similar_users =
-          users_similar_to_by_followers(
-            user,
-            limit: 10,
-            exclude_followed_by: exclude_followed_by,
-          ).map { |u| user_to_similarity_entry(u) }
-      end
-    end
-
-    render json: {
-             all: all_similar_users,
-             not_followed: not_followed_similar_users,
-           }
-  end
-
-  sig { void }
-  def monitor_bluesky_user
-    user = T.cast(@user, Domain::User::BlueskyUser)
-    authorize user
-    monitor = Domain::Bluesky::MonitoredObject.build_for_user(user)
-    if monitor.save
-      Domain::Bluesky::Job::ScanUserJob.perform_later(user:)
-      Domain::Bluesky::Job::ScanPostsJob.perform_later(user:)
-      flash[:notice] = "User is now being monitored"
-    else
-      flash[
-        :alert
-      ] = "Error monitoring user: #{monitor.errors.full_messages.join(", ")}"
-    end
-    redirect_to domain_user_path(user)
-  end
-
-  private
-
-  sig { override.returns(DomainController::DomainParamConfig) }
-  def self.param_config
-    DomainController::DomainParamConfig.new(
-      user_id_param: :id,
-      post_id_param: :domain_post_id,
-      post_group_id_param: :domain_post_group_id,
-    )
-  end
-
-  # TODO - make a typed ImmutableStruct for the return type
-  sig { params(user: Domain::User::FaUser).returns(T::Hash[Symbol, T.untyped]) }
-  def user_to_similarity_entry(user)
-    profile_thumb_url = user.avatar&.log_entry&.uri_str
-    profile_thumb_url ||=
-      begin
-        pp_log_entry = get_best_user_page_http_log_entry_for(user)
-        if pp_log_entry
-          parser =
-            Domain::Fa::Parser::Page.from_log_entry(
-              pp_log_entry,
-              require_logged_in: false,
-            )
-          parser.user_page.profile_thumb_url
-        end
-      rescue StandardError
-        logger.error("error getting profile_thumb_url: #{$!.message}")
-      end || "https://a.furaffinity.net/0/#{user.url_name}.gif"
-
-    {
-      name: user.name,
-      url_name: user.url_name,
-      profile_thumb_url: profile_thumb_url,
-      external_url: "https://www.furaffinity.net/user/#{user.url_name}/",
-      refurrer_url: request.base_url + helpers.domain_user_path(user),
-    }
-  end
-
-  sig { params(user: Domain::User::FaUser).returns(T.nilable(HttpLogEntry)) }
-  def get_best_user_page_http_log_entry_for(user)
-    for_path =
-      proc do |uri_path|
-        HttpLogEntry
-          .where(
-            uri_scheme: "https",
-            uri_host: "www.furaffinity.net",
-            uri_path: uri_path,
-          )
-          .order(created_at: :desc)
-          .first
-      end
-
-    # older versions don't end in a trailing slash
-    user.last_user_page_log_entry || for_path.call("/user/#{user.url_name}/") ||
-      for_path.call("/user/#{user.url_name}")
-  end
-
-  sig do
-    params(
-      user: Domain::User::FaUser,
-      limit: Integer,
-      exclude_followed_by: T.nilable(Domain::User::FaUser),
-    ).returns(T::Array[Domain::User::FaUser])
-  end
-  def users_similar_to_by_followers(user, limit: 10, exclude_followed_by: nil)
-    factors = Domain::Factors::UserUserFollowToFactors.find_by(user: user)
-    return [] if factors.nil?
-
-    relation =
-      Domain::NeighborFinder
-        .find_neighbors(factors)
-        .limit(limit)
-        .includes(:user)
-
-    if exclude_followed_by
-      relation =
-        relation.where.not(
-          user_id: exclude_followed_by.followed_users.select(:to_id),
-        )
-    end
-
-    relation.map(&:user)
-  end
-end

app/controllers/domain_controller.rb

@@ -1,73 +0,0 @@
-# typed: strict
-class DomainController < ApplicationController
-  extend T::Sig
-  extend T::Helpers
-  abstract!
-
-  class DomainParamConfig < T::ImmutableStruct
-    include T::Struct::ActsAsComparable
-
-    const :post_id_param, Symbol
-    const :user_id_param, Symbol
-    const :post_group_id_param, Symbol
-  end
-
-  sig { void }
-  def initialize
-    super
-    @post = T.let(nil, T.nilable(Domain::Post))
-    @user = T.let(nil, T.nilable(Domain::User))
-    @post_group = T.let(nil, T.nilable(Domain::PostGroup))
-  end
-
-  protected
-
-  sig { abstract.returns(DomainParamConfig) }
-  def self.param_config
-  end
-
-  sig(:final) { void }
-  def set_post!
-    @post =
-      self.class.find_model_from_param(
-        Domain::Post,
-        params[self.class.param_config.post_id_param],
-      ) || raise(ActiveRecord::RecordNotFound)
-  end
-
-  sig(:final) { void }
-  def set_user!
-    @user =
-      self.class.find_model_from_param(
-        Domain::User,
-        params[self.class.param_config.user_id_param],
-      ) || raise(ActiveRecord::RecordNotFound)
-  end
-
-  sig(:final) { void }
-  def set_post_group!
-    @post_group =
-      self.class.find_model_from_param(
-        Domain::PostGroup,
-        params[self.class.param_config.post_group_id_param],
-      ) || raise(ActiveRecord::RecordNotFound)
-  end
-
-  public
-
-  sig(:final) do
-    type_parameters(:Klass)
-      .params(
-        klass:
-          T.all(
-            T.class_of(ReduxApplicationRecord),
-            HasCompositeToParam::ClassMethods[T.type_parameter(:Klass)],
-          ),
-        param: T.nilable(String),
-      )
-      .returns(T.nilable(T.type_parameter(:Klass)))
-  end
-  def self.find_model_from_param(klass, param)
-    klass.find_by_param(param)
-  end
-end

app/controllers/global_states_controller.rb

@@ -11,8 +11,6 @@ class GlobalStatesController < ApplicationController
 
   IB_COOKIE_KEYS = %w[inkbunny-username inkbunny-password inkbunny-sid].freeze
 
-  TELEGRAM_KEYS = %w[telegram-bot-token].freeze
-
   def index
     authorize GlobalState
     @global_states = policy_scope(GlobalState).order(:key)
@@ -184,50 +182,6 @@ class GlobalStatesController < ApplicationController
     end
   end
 
-  def telegram_config
-    authorize GlobalState
-    @telegram_config =
-      TELEGRAM_KEYS.map do |key|
-        GlobalState.find_by(key: key) ||
-          GlobalState.new(key: key, value_type: :string)
-      end
-  end
-
-  def edit_telegram_config
-    authorize GlobalState
-    @telegram_config =
-      TELEGRAM_KEYS.map do |key|
-        GlobalState.find_by(key: key) ||
-          GlobalState.new(key: key, value_type: :string)
-      end
-  end
-
-  def update_telegram_config
-    authorize GlobalState
-
-    begin
-      ActiveRecord::Base.transaction do
-        telegram_config_params.each do |key, value|
-          state = GlobalState.find_or_initialize_by(key: key)
-          state.value = value
-          state.value_type = :string
-          state.save!
-        end
-      end
-
-      redirect_to telegram_config_global_states_path,
-                  notice: "Telegram bot configuration was successfully updated."
-    rescue ActiveRecord::RecordInvalid => e
-      @telegram_config =
-        TELEGRAM_KEYS.map do |key|
-          GlobalState.find_by(key: key) ||
-            GlobalState.new(key: key, value_type: :string)
-        end
-      flash.now[:alert] = "Error updating Telegram bot configuration: #{e.message}"
-      render :edit_telegram_config, status: :unprocessable_entity
-    end
-  end
-
   private
 
   def set_global_state
@@ -247,8 +201,4 @@ class GlobalStatesController < ApplicationController
       *IB_COOKIE_KEYS.reject { |key| key == "inkbunny-sid" },
     )
   end
-
-  def telegram_config_params
-    params.require(:telegram_config).permit(*TELEGRAM_KEYS)
-  end
 end

app/controllers/indexed_posts_controller.rb

@@ -0,0 +1,13 @@
+# typed: false
+class IndexedPostsController < ApplicationController
+  def index
+    @posts = IndexedPost.all
+    active_sources = (params[:sources] || SourceHelper.all_source_names)
+    unless SourceHelper.has_all_sources?(active_sources)
+      postable_types = SourceHelper.source_names_to_class_names(active_sources)
+      @posts =
+        @posts.where(postable_type: postable_types) if postable_types.any?
+    end
+    @posts = @posts.order(created_at: :desc).page(params[:page]).per(50)
+  end
+end

app/controllers/log_entries_controller.rb

@@ -56,7 +56,7 @@ class LogEntriesController < ApplicationController
     HttpLogEntry
       .joins(:response)
       .includes(:response)
-      .select("http_log_entries.*, blob_files.size_bytes")
+      .select("http_log_entries.*, blob_entries_p.size")
       .find_each(batch_size: 100, order: :desc) do |log_entry|
         break if log_entry.created_at < @time_window.ago
         @last_window_count += 1
@@ -76,7 +76,7 @@ class LogEntriesController < ApplicationController
       HttpLogEntry.includes(
         :caused_by_entry,
         :triggered_entries,
-        :response,
+        response: :base,
       ).find(params[:id])
   end
 end

app/controllers/pages_controller.rb

@@ -1,12 +1,8 @@
 # typed: true
 class PagesController < ApplicationController
-  skip_before_action :authenticate_user!, only: %i[root furecs_user_script]
+  skip_before_action :authenticate_user!, only: [:root]
 
   def root
     render :root
   end
-
-  def furecs_user_script
-    render :furecs_user_script
-  end
 end

app/controllers/state/ip_address_roles_controller.rb

@@ -1,85 +0,0 @@
-# typed: true
-class State::IpAddressRolesController < ApplicationController
-  before_action :set_ip_address_role, only: %i[edit update destroy toggle]
-  before_action :authorize_ip_address_roles
-
-  # GET /state/ip_address_roles
-  def index
-    @ip_address_roles = IpAddressRole.all.order(created_at: :desc)
-  end
-
-  # GET /state/ip_address_roles/new
-  def new
-    @ip_address_role = IpAddressRole.new
-  end
-
-  # GET /state/ip_address_roles/1/edit
-  def edit
-  end
-
-  # POST /state/ip_address_roles
-  def create
-    @ip_address_role = IpAddressRole.new(ip_address_role_params)
-
-    if @ip_address_role.save
-      redirect_to state_ip_address_roles_path,
-                  notice: "IP address role was successfully created."
-    else
-      render :new
-    end
-  end
-
-  # PATCH/PUT /state/ip_address_roles/1
-  def update
-    if @ip_address_role.update(ip_address_role_params)
-      redirect_to state_ip_address_roles_path,
-                  notice: "IP address role was successfully updated."
-    else
-      render :edit
-    end
-  end
-
-  # DELETE /state/ip_address_roles/1
-  def destroy
-    @ip_address_role.destroy
-    redirect_to state_ip_address_roles_path,
-                notice: "IP address role was successfully deleted."
-  end
-
-  def toggle
-    @ip_address_role.update!(active: !@ip_address_role.active)
-    redirect_to state_ip_address_roles_path
-  rescue ActiveRecord::RecordInvalid => e
-    redirect_to state_ip_address_roles_path,
-                alert: "Failed to update status: #{e.message}"
-  end
-
-  private
-
-  # Use callbacks to share common setup or constraints between actions
-  def set_ip_address_role
-    @ip_address_role = IpAddressRole.find(params[:id])
-  end
-
-  # Only allow a list of trusted parameters through
-  def ip_address_role_params
-    params.require(:ip_address_role).permit(
-      :ip_address,
-      :role,
-      :description,
-      :active,
-    )
-  end
-
-  # Authorize all actions based on the current action
-  def authorize_ip_address_roles
-    case action_name.to_sym
-    when :index, :new, :edit
-      authorize IpAddressRole, policy_class: State::IpAddressRolePolicy
-    when :create
-      authorize IpAddressRole, policy_class: State::IpAddressRolePolicy
-    when :update, :destroy, :toggle
-      authorize @ip_address_role, policy_class: State::IpAddressRolePolicy
-    end
-  end
-end

app/controllers/telegram_bot_logs_controller.rb

@@ -1,103 +0,0 @@
-# typed: false
-class TelegramBotLogsController < ApplicationController
-  before_action :set_telegram_bot_log, only: %i[show]
-  after_action :verify_authorized
-
-  def index
-    authorize TelegramBotLog
-
-    # Start with policy scope
-    @telegram_bot_logs = policy_scope(TelegramBotLog)
-
-    # Apply filters
-    @telegram_bot_logs = apply_filters(@telegram_bot_logs)
-
-    # Order by most recent first
-    @telegram_bot_logs = @telegram_bot_logs.recent
-
-    # Paginate with Kaminari
-    @limit = (params[:limit] || 50).to_i.clamp(1, 500)
-    @telegram_bot_logs = @telegram_bot_logs.page(params[:page]).per(@limit)
-
-    # Load associations for display
-    @telegram_bot_logs = @telegram_bot_logs.includes(:processed_image)
-
-    # Set up filter options for the view
-    @status_options = TelegramBotLog.statuses.keys
-    @filter_params =
-      params.slice(
-        :telegram_user_id,
-        :status,
-        :start_date,
-        :end_date,
-        :min_results,
-        :max_results,
-        :slow_requests,
-      )
-  end
-
-  def show
-    authorize @telegram_bot_log
-
-    # The processed_image association will be loaded automatically when accessed
-  end
-
-  private
-
-  def set_telegram_bot_log
-    @telegram_bot_log =
-      TelegramBotLog.includes(:processed_image).find(params[:id])
-  rescue ActiveRecord::RecordNotFound
-    redirect_to telegram_bot_logs_path, alert: "Telegram bot log not found."
-  end
-
-  def apply_filters(scope)
-    # Filter by telegram user ID
-    if params[:telegram_user_id].present?
-      scope = scope.for_user(params[:telegram_user_id].to_i)
-    end
-
-    # Filter by status
-    if params[:status].present? && TelegramBotLog.statuses.key?(params[:status])
-      scope = scope.where(status: params[:status])
-    end
-
-    # Filter by date range
-    if params[:start_date].present?
-      begin
-        start_date = Date.parse(params[:start_date])
-        scope =
-          scope.where("request_timestamp >= ?", start_date.beginning_of_day)
-      rescue Date::Error
-        # Ignore invalid date
-      end
-    end
-
-    if params[:end_date].present?
-      begin
-        end_date = Date.parse(params[:end_date])
-        scope = scope.where("request_timestamp <= ?", end_date.end_of_day)
-      rescue Date::Error
-        # Ignore invalid date
-      end
-    end
-
-    # Filter by search results count
-    if params[:min_results].present?
-      scope =
-        scope.where("search_results_count >= ?", params[:min_results].to_i)
-    end
-
-    if params[:max_results].present?
-      scope =
-        scope.where("search_results_count <= ?", params[:max_results].to_i)
-    end
-
-    # Filter by performance metrics
-    if params[:slow_requests].present? && params[:slow_requests] == "true"
-      scope = scope.slow_requests
-    end
-
-    scope
-  end
-end

app/controllers/user_scripts_controller.rb

@@ -1,24 +1,21 @@
 # typed: true
 class UserScriptsController < ApplicationController
   skip_before_action :authenticate_user!, only: [:get]
-  skip_before_action :verify_authenticity_token, only: [:get]
-
-  ALLOWED_SCRIPTS = %w[object_statuses.user.js furecs.user.js].freeze
 
   def get
     expires_in 1.hour, public: true
-    response.cache_control[:public] = true
-    response.cache_control[:private] = false
+    response.cache_control[:public] = false
+    response.cache_control[:private] = true
 
     script = params[:script]
-    unless ALLOWED_SCRIPTS.include?(script)
+    case script
+    when "furecs.user.js"
+      send_file(
+        Rails.root.join("user_scripts/furecs.user.js"),
+        type: "application/json",
+      )
+    else
       render status: 404, text: "not found"
-      return
     end
-
-    send_file(
-      Rails.root.join("user_scripts/dist/#{script}"),
-      type: "application/javascript",
-    )
   end
 end

app/helpers/domain/bluesky_post_helper.rb

@@ -1,210 +0,0 @@
-# typed: strict
-# frozen_string_literal: true
-
-module Domain::BlueskyPostHelper
-  extend T::Sig
-  include ActionView::Helpers::UrlHelper
-  include HelpersInterface
-  include Domain::PostsHelper
-
-  class FacetPart < T::Struct
-    const :type, Symbol
-    const :value, String
-  end
-
-  sig do
-    params(text: String, facets: T.nilable(T::Array[T.untyped])).returns(
-      T.nilable(String),
-    )
-  end
-  def render_bsky_post_facets(text, facets = nil)
-    return text if facets.blank?
-
-    facets =
-      begin
-        facets.map { |facet| Bluesky::Text::Facet.from_hash(facet) }
-      rescue => e
-        Rails.logger.error("error parsing Bluesky facets: #{e.message}")
-        return text
-      end
-
-    result_parts = T.let([], T::Array[FacetPart])
-    last_end = 0
-
-    # Sort facets by start position to handle them in order
-    sorted_facets = facets.sort_by(&:byteStart)
-
-    sorted_facets.each do |facet|
-      if facet.byteStart < 0 || facet.byteEnd <= facet.byteStart ||
-           facet.byteEnd > text.bytesize
-        next
-      end
-
-      # Skip overlapping facets
-      next if facet.byteStart < last_end
-
-      # Add text before this facet
-      if facet.byteStart > last_end
-        before_text = text.byteslice(last_end, facet.byteStart - last_end)
-        if before_text
-          result_parts << FacetPart.new(type: :text, value: before_text)
-        end
-      end
-
-      # Extract the facet text using byteslice for accurate character extraction
-      facet_text =
-        text.byteslice(facet.byteStart, facet.byteEnd - facet.byteStart)
-      next unless facet_text # Skip if byteslice returns nil
-
-      # Process the facet
-      rendered_facet = render_facet(facet, facet_text)
-      result_parts << FacetPart.new(type: :facet, value: rendered_facet)
-
-      last_end = facet.byteEnd
-    end
-
-    # Add remaining text after the last facet
-    if last_end < text.bytesize
-      remaining_text = text.byteslice(last_end, text.bytesize - last_end)
-      if remaining_text
-        result_parts << FacetPart.new(type: :text, value: remaining_text)
-      end
-    end
-
-    result_parts
-      .map do |part|
-        case part.type
-        when :text
-          part.value.gsub("\n", "<br />")
-        when :facet
-          part.value
-        end
-      end
-      .join
-      .html_safe
-  end
-
-  private
-
-  sig do
-    params(facet: Bluesky::Text::Facet, facet_text: String).returns(String)
-  end
-  def render_facet(facet, facet_text)
-    return facet_text unless facet.features.any?
-
-    # Process the first feature (Bluesky facets typically have one feature per facet)
-    feature = facet.features.first
-    return facet_text unless feature.is_a?(Bluesky::Text::FacetFeature)
-
-    case feature
-    when Bluesky::Text::FacetFeatureMention
-      render_mention_facet(feature, facet_text)
-    when Bluesky::Text::FacetFeatureURI
-      render_link_facet(feature, facet_text)
-    when Bluesky::Text::FacetFeatureTag
-      render_tag_facet(feature, facet_text)
-    else
-      # Unknown facet type, return original text
-      facet_text
-    end
-  end
-
-  sig do
-    params(
-      feature: Bluesky::Text::FacetFeatureMention,
-      facet_text: String,
-    ).returns(String)
-  end
-  def render_mention_facet(feature, facet_text)
-    did = feature.did
-    return facet_text unless did.present?
-
-    # Try to find the user in the database
-    user = Domain::User::BlueskyUser.find_by(did: did)
-
-    if user
-      # Render the inline user partial
-      render(
-        partial: "domain/has_description_html/inline_link_domain_user",
-        locals: {
-          user: user,
-          link_text: facet_text,
-          visual_style: "description-section-link-light",
-        },
-      )
-    else
-      # Render external link to Bluesky profile
-      render(
-        partial: "domain/has_description_html/external_link",
-        locals: {
-          link_text: facet_text,
-          url: "https://bsky.app/profile/#{did}",
-        },
-      )
-    end
-  end
-
-  sig do
-    params(feature: Bluesky::Text::FacetFeatureURI, facet_text: String).returns(
-      String,
-    )
-  end
-  def render_link_facet(feature, facet_text)
-    uri = feature.uri
-    return facet_text unless uri.present?
-
-    source = link_for_source(uri)
-    if source.present? && (model = source.model)
-      case model
-      when Domain::Post
-        return(
-          render(
-            partial: "domain/has_description_html/inline_link_domain_post",
-            locals: {
-              post: model,
-              link_text: facet_text,
-              visual_style: "description-section-link-light",
-            },
-          )
-        )
-      when Domain::User
-        return(
-          render(
-            partial: "domain/has_description_html/inline_link_domain_user",
-            locals: {
-              user: model,
-              link_text: facet_text,
-              visual_style: "description-section-link-light",
-            },
-          )
-        )
-      end
-    end
-
-    render(
-      partial: "domain/has_description_html/external_link",
-      locals: {
-        link_text: facet_text,
-        url: uri,
-      },
-    )
-  end
-
-  sig do
-    params(feature: Bluesky::Text::FacetFeatureTag, facet_text: String).returns(
-      String,
-    )
-  end
-  def render_tag_facet(feature, facet_text)
-    tag = feature.tag
-    return facet_text unless tag.present?
-
-    render(
-      partial: "domain/has_description_html/external_link",
-      locals: {
-        link_text: facet_text,
-        url: "https://bsky.app/hashtag/#{tag}",
-      },
-    )
-  end
-end

app/helpers/domain/descriptions_helper.rb

@@ -1,397 +0,0 @@
-# typed: strict
-# frozen_string_literal: true
-require "dtext"
-
-module Domain::DescriptionsHelper
-  extend T::Sig
-  extend T::Helpers
-  include HelpersInterface
-  include Domain::PostsHelper
-  include Domain::DomainsHelper
-  include Domain::UsersHelper
-  requires_ancestor { Object }
-
-  abstract!
-
-  sig do
-    params(assumed_host: String, url_string: String).returns(
-      T.nilable(Addressable::URI),
-    )
-  end
-  def try_parse_uri(assumed_host, url_string)
-    extracted = URI.extract(url_string).first || url_string
-
-    # if the url string starts with a slash, add the assumed host to it
-    extracted = assumed_host + extracted if extracted.starts_with?("/")
-
-    # if the url string doesn't have a protocol, add https:// to it
-    unless extracted.starts_with?("http") && extracted.include?("://")
-      extracted = "https://" + extracted
-    end
-
-    uri = Addressable::URI.parse(extracted)
-    uri.host ||= assumed_host
-    uri.scheme ||= "https"
-    uri
-  rescue Addressable::URI::InvalidURIError
-    nil
-  end
-
-  sig { params(text: String, url: String).returns(T::Boolean) }
-  def text_same_as_url?(text, url)
-    text = text.strip.downcase
-    url = url.strip.downcase
-    ["", "http://", "https://"].any? { |prefix| "#{prefix}#{text}" == url }
-  end
-
-  sig { params(model: HasDescriptionHtmlForView).returns(T.nilable(String)) }
-  def description_section_class_for_model(model)
-    case model
-    when Domain::Post::FaPost, Domain::User::FaUser
-      "bg-slate-700 p-4 text-slate-200 text-sm"
-    when Domain::Post::E621Post, Domain::User::E621User
-      "bg-slate-700 p-4 text-slate-200 text-sm"
-    else
-      nil
-    end
-  end
-
-  WEAK_URL_MATCHER_REGEX =
-    %r{(http(s)?:\/\/)?(www\.)?[-a-zA-Z0-9@:%._\+~#=]{1,256}\.[a-z]{2,6}\b([-a-zA-Z0-9@:%_\+.~#?&//=]*)}
-
-  sig { params(str: String).returns(T.nilable(String)) }
-  def extract_weak_url(str)
-    str.match(WEAK_URL_MATCHER_REGEX)&.[](0)
-  end
-
-  ALLOWED_INFERRED_URL_DOMAINS =
-    T.let(
-      %w[furaffinity.net inkbunny.net e621.net bsky.app]
-        .flat_map { |domain| [domain, "www.#{domain}"] }
-        .freeze,
-      T::Array[String],
-    )
-
-  sig { params(model: HasDescriptionHtmlForView).returns(T.nilable(String)) }
-  def sanitize_description_html(model)
-    html = model.description_html_for_view
-    return nil if html.blank?
-
-    is_bsky_description = model.is_a?(Domain::User::BlueskyUser)
-    visual_style =
-      (
-        if model.is_a?(Domain::User::BlueskyUser)
-          "description-section-link-light"
-        else
-          "description-section-link"
-        end
-      )
-
-    case model
-    when Domain::Post::E621Post
-      dtext_result = DText.parse(html)
-      return nil if dtext_result.blank?
-      html = dtext_result[0]
-    else
-      # profiles often contain bbcode, so first re-parse that
-      # for some reason, lots of duplicate <br> tags appear as well
-      html = html.gsub("<br>", "").strip
-      html = try_convert_bbcode_to_html(html)
-    end
-
-    replacements = {}
-
-    # Transform bare text that is not contained within an anchor tag into an anchor tag
-    text_link_transformer =
-      lambda do |env|
-        node = T.cast(env[:node], Nokogiri::XML::Node)
-        return if env[:is_allowlisted]
-
-        next unless node.text?
-        next unless node.ancestors("a").empty?
-        next unless (node_text = T.cast(node.text, T.nilable(String)))
-        next unless (url_text = extract_weak_url(node_text))
-        next if url_text.blank?
-        unless (
-                 uri =
-                   try_parse_uri(model.description_html_base_domain, url_text)
-               )
-          next
-        end
-        if is_bsky_description
-          unless ALLOWED_EXTERNAL_LINK_DOMAINS.any? { |domain|
-                   url_matches_domain?(domain, uri.host)
-                 }
-            next
-          end
-        elsif ALLOWED_PLAIN_TEXT_URL_DOMAINS.none? do |domain|
-              url_matches_domain?(domain, uri.host)
-            end
-          next
-        end
-
-        before, after = node.text.split(url_text, 2)
-        new_node = "#{before}<a href=\"#{url_text}\">#{url_text}</a>#{after}"
-        node.replace(new_node)
-      end
-
-    tag_class_and_style_transformer =
-      lambda do |env|
-        node = T.cast(env[:node], Nokogiri::XML::Node)
-        node_name = T.cast(env[:node_name], String)
-        return if env[:is_allowlisted] || !node.element?
-
-        # Convert bbcode_center class to text-align: center style
-        # and remove all other styling
-        add_node_styles = []
-        if node["class"]&.include?("bbcode_center")
-          add_node_styles << "text-align: center"
-        end
-
-        node.name = "div" if node_name == "code"
-        node.remove_attribute("class")
-        # add to original styles
-        node["style"] = (node["style"] || "")
-          .split(";")
-          .map(&:strip)
-          .concat(add_node_styles)
-          .map { |s| s + ";" }
-          .join(" ")
-      end
-
-    link_to_model_link_transformer =
-      lambda do |env|
-        node = T.cast(env[:node], Nokogiri::XML::Node)
-        node_name = T.cast(env[:node_name], String)
-        next if env[:is_allowlisted] || !node.element?
-
-        # Only allow and transform FA links
-        if node_name == "a"
-          href_str = node["href"]&.downcase || ""
-          url = try_parse_uri(model.description_html_base_domain, href_str)
-          next { node_whitelist: [] } if url.nil?
-
-          found_link = link_for_source(url.to_s)
-          if found_link.present? && (found_model = found_link.model)
-            partial, locals =
-              case found_model
-              when Domain::Post
-                [
-                  "domain/has_description_html/inline_link_domain_post",
-                  { post: found_model, link_text: node.text, visual_style: },
-                ]
-              when Domain::User
-                [
-                  "domain/has_description_html/inline_link_domain_user",
-                  { user: found_model, link_text: node.text, visual_style: },
-                ]
-              else
-                raise "Unknown model type: #{found_link.model.class}"
-              end
-
-            replacements[node] = Nokogiri::HTML5.fragment(
-              render(partial:, locals:),
-            )
-            next { node_whitelist: [node] }
-          else
-            if ALLOWED_EXTERNAL_LINK_DOMAINS.any? { |domain|
-                 url_matches_domain?(domain, url.host)
-               }
-              if node.text.blank? || text_same_as_url?(node.text, url.to_s)
-                title = title_for_url(url.to_s)
-              else
-                title = node.text
-              end
-
-              replacements[node] = Nokogiri::HTML5.fragment(
-                if is_bsky_description
-                  render(
-                    partial: "domain/has_description_html/external_link",
-                    locals: {
-                      link_text: node.text,
-                      url: url.to_s,
-                    },
-                  )
-                else
-                  render(
-                    partial: "domain/has_description_html/inline_link_external",
-                    locals: {
-                      url: url.to_s,
-                      title:,
-                      icon_path: icon_path_for_domain(url.host),
-                    },
-                  )
-                end,
-              )
-              next { node_whitelist: [node] }
-            end
-          end
-        end
-      end
-
-    disallowed_link_transformer =
-      lambda do |env|
-        node = T.cast(env[:node], Nokogiri::XML::Node)
-        node_name = T.cast(env[:node_name], String)
-        return if env[:is_allowlisted] || !node.element?
-
-        if node_name == "a"
-          # by the time we're here, we know this is not a valid link node,
-          # and it should be replaced with its text
-          node.replace(node.inner_html)
-        end
-      end
-
-    sanitizer =
-      Sanitize.new(
-        elements: %w[a code div br img b i span strong hr p],
-        attributes: {
-          "a" => %w[href class],
-          :all => %w[class style],
-        },
-        css: {
-          properties: %w[font-size color text-align class],
-        },
-        transformers: [
-          text_link_transformer,
-          tag_class_and_style_transformer,
-          link_to_model_link_transformer,
-          disallowed_link_transformer,
-        ],
-      )
-
-    fragment = Nokogiri::HTML5.fragment(sanitizer.send(:preprocess, html))
-    sanitizer.node!(fragment)
-    replacements.each { |node, replacement| node.replace(replacement) }
-    raw fragment.to_html(preserve_newline: true)
-  rescue StandardError
-    raise if Rails.env == "staging" || Rails.env.test? || Rails.env.development?
-    # if anything goes wrong in production, bail out and don't display anything
-    "(error generating description)"
-  end
-
-  sig { params(visual_style: String).returns(String) }
-  def link_classes_for_visual_style(visual_style)
-    case visual_style
-    when "sky-link"
-      "blue-link truncate"
-    when "description-section-link"
-      [
-        "text-sky-200 border-slate-200",
-        "border border-transparent hover:border-slate-300 hover:text-sky-800 hover:bg-slate-100",
-        "rounded-md px-1 transition-all",
-        "inline-flex items-center align-bottom",
-      ].join(" ")
-    when "description-section-link-light"
-      [
-        "text-sky-600 border-slate-300",
-        "border border-transparent hover:border-slate-500 hover:text-sky-800 hover:bg-slate-200",
-        "rounded-md px-1 transition-all",
-        "inline-flex items-center align-bottom",
-      ].join(" ")
-    else
-      "blue-link"
-    end
-  end
-
-  sig do
-    params(user: Domain::User, visual_style: String, icon_size: String).returns(
-      T::Hash[Symbol, T.untyped],
-    )
-  end
-  def props_for_user_hover_preview(user, visual_style, icon_size)
-    cache_key = [
-      user,
-      policy(user),
-      "popover_inline_link_domain_user",
-      icon_size,
-    ]
-    Rails
-      .cache
-      .fetch(cache_key) do
-        num_posts =
-          user.has_created_posts? ? user.user_post_creations.size : nil
-        registered_at = domain_user_registered_at_string_for_view(user)
-        num_followed_by =
-          user.has_followed_by_users? ? user.user_user_follows_to.size : nil
-        num_followed =
-          user.has_followed_users? ? user.user_user_follows_from.size : nil
-        avatar_thumb_size = icon_size == "large" ? "64-avatar" : "32-avatar"
-
-        {
-          iconSize: icon_size,
-          linkText: user.name_for_view,
-          userId: user.to_param,
-          userName: user.name_for_view,
-          userPath: domain_user_path(user),
-          userSmallAvatarPath:
-            domain_user_avatar_img_src_path(
-              user.avatar,
-              thumb: avatar_thumb_size,
-            ),
-          userAvatarPath: domain_user_avatar_img_src_path(user.avatar),
-          userAvatarAlt: "View #{user.name_for_view}'s profile",
-          userDomainIcon: domain_model_icon_path(user),
-          userNumPosts: num_posts,
-          userRegisteredAt: registered_at,
-          userNumFollowedBy: num_followed_by,
-          userNumFollowed: num_followed,
-        }
-      end
-      .then do |props|
-        props[:visualStyle] = visual_style
-        props
-      end
-  end
-
-  sig do
-    params(
-      post: Domain::Post,
-      link_text: String,
-      visual_style: String,
-      domain_icon: T::Boolean,
-      link_params: T::Hash[Symbol, T.untyped],
-    ).returns(T::Hash[Symbol, T.untyped])
-  end
-  def props_for_post_hover_preview(
-    post,
-    link_text,
-    visual_style,
-    domain_icon: true,
-    link_params: {}
-  )
-    cache_key = [
-      post,
-      policy(post),
-      link_text,
-      "popover_inline_link_domain_post",
-    ]
-    Rails
-      .cache
-      .fetch(cache_key) do
-        {
-          linkText: link_text,
-          postId: post.to_param,
-          postTitle: post.title,
-          postPath:
-            Rails.application.routes.url_helpers.domain_post_path(
-              post,
-              link_params,
-            ),
-          postThumbnailPath: thumbnail_for_post_path(post),
-          postThumbnailAlt: "View on #{domain_name_for_model(post)}",
-          postDomainIcon: domain_icon ? domain_model_icon_path(post) : nil,
-        }.then do |props|
-          if creator = post.primary_creator_for_view
-            props[:creatorName] = creator.name_for_view
-            props[:creatorAvatarPath] = user_avatar_path_for_view(creator)
-          end
-          props
-        end
-      end
-      .then do |props|
-        props[:visualStyle] = visual_style
-        props
-      end
-  end
-end

app/helpers/domain/domain_model_helper.rb

@@ -1,42 +0,0 @@
-# typed: strict
-module Domain::DomainModelHelper
-  extend T::Sig
-  extend T::Helpers
-  include HelpersInterface
-  abstract!
-
-  HasDomainTypeType =
-    T.type_alias { T.any(HasDomainType, HasDomainType::ClassMethods) }
-
-  sig { params(model: HasDomainTypeType).returns(String) }
-  def domain_name_for_model(model)
-    case model.domain_type
-    when Domain::DomainType::Fa
-      "FurAffinity"
-    when Domain::DomainType::E621
-      "E621"
-    when Domain::DomainType::Inkbunny
-      "Inkbunny"
-    when Domain::DomainType::Sofurry
-      "Sofurry"
-    when Domain::DomainType::Bluesky
-      "Bluesky"
-    end
-  end
-
-  sig { params(model: HasDomainTypeType).returns(String) }
-  def domain_abbreviation_for_model(model)
-    case model.domain_type
-    when Domain::DomainType::Fa
-      "FA"
-    when Domain::DomainType::E621
-      "E621"
-    when Domain::DomainType::Inkbunny
-      "IB"
-    when Domain::DomainType::Sofurry
-      "SF"
-    when Domain::DomainType::Bluesky
-      "BSKY"
-    end
-  end
-end

app/helpers/domain/domain_type.rb

@@ -1,11 +0,0 @@
-# typed: strict
-# Enum represents the domain of a post or user, e.g. "FurAffinity", "E621", "Inkbunny"
-class Domain::DomainType < T::Enum
-  enums do
-    Fa = new
-    E621 = new
-    Inkbunny = new
-    Sofurry = new
-    Bluesky = new
-  end
-end

app/helpers/domain/domains_helper.rb

@@ -1,142 +0,0 @@
-# typed: strict
-module Domain::DomainsHelper
-  extend T::Sig
-  extend T::Helpers
-  include HelpersInterface
-  abstract!
-
-  # If a URL is detected in plain text and is one of these domains,
-  # it will be converted to an anchor tag.
-  ALLOWED_PLAIN_TEXT_URL_DOMAINS = %w[
-    e621.net
-    furaffinity.net
-    inkbunny.net
-    bsky.app
-  ].freeze
-
-  # If a link is detected in an anchor tag and is one of these domains,
-  # it will be converted to a link.
-  ALLOWED_EXTERNAL_LINK_DOMAINS =
-    T.let(
-      (
-        %w[
-          archiveofourown.org
-          behance.net
-          bigcartel.com
-          boosty.to
-          bsky.app
-          carrd.co
-          deviantart.com
-          discord.gg
-          dribbble.com
-          e621.net
-          facebook.com
-          furaffinity.net
-          gumroad.com
-          hipolink.me
-          inkbunny.net
-          itch.io
-          instagram.com
-          ko-fi.com
-          livejournal.com
-          mstdn.social
-          patreon.com
-          pinterest.com
-          pixiv.net
-          redbubble.com
-          spreadshirt.com
-          spreadshirt.de
-          subscribestar.adult
-          linktr.ee
-          t.me
-          trello.com
-          tumblr.com
-          twitch.tv
-          twitter.com
-          vimeo.com
-          weasyl.com
-          x.com
-          youtube.com
-          sofurry.com
-          aethy.com
-        ] + ALLOWED_PLAIN_TEXT_URL_DOMAINS
-      ).freeze,
-      T::Array[String],
-    )
-
-  DOMAIN_TO_ICON_PATH =
-    T.let(
-      {
-        "bigcartel.com" => "bigcartel.png",
-        "boosty.to" => "boosty.png",
-        "bsky.app" => "bsky.png",
-        "carrd.co" => "carrd.png",
-        "deviantart.com" => "deviantart.png",
-        "e621.net" => "e621.png",
-        "furaffinity.net" => "fa.png",
-        "ib.metapix.net" => "inkbunny.png",
-        "inkbunny.net" => "inkbunny.png",
-        "itaku.ee" => "itaku.png",
-        "ko-fi.com" => "ko-fi.png",
-        "newgrounds.com" => "newgrounds.png",
-        "patreon.com" => "patreon.png",
-        "pixiv.net" => "pixiv.png",
-        "redbubble.com" => "redbubble.png",
-        "spreadshirt.com" => "spreadshirt.png",
-        "spreadshirt.de" => "spreadshirt.png",
-        "subscribestar.com" => "subscribestar.png",
-        "subscribestar.adult" => "subscribestar.png",
-        "gumroad.com" => "gumroad.png",
-        "itch.io" => "itch-io.png",
-        "t.me" => "telegram.png",
-        "tumblr.com" => "tumblr.png",
-        "twitter.com" => "x-twitter.png",
-        "trello.com" => "trello.png",
-        "weasyl.com" => "weasyl.png",
-        "wixmp.com" => "deviantart.png",
-        "x.com" => "x-twitter.png",
-        "linktr.ee" => "linktree.png",
-        "aethy.com" => "aethy.png",
-      }.freeze,
-      T::Hash[String, String],
-    )
-
-  DOMAIN_TITLE_MAPPERS =
-    T.let(
-      [
-        [%r{://t.me/([^/]+)}, ->(match) { match[1] }],
-        [%r{://bsky.app/profile/([^/]+)}, ->(match) { match[1] }],
-        [%r{://(.*\.)?x.com/([^/]+)}, ->(match) { match[2] }],
-        [%r{://(.*\.)?twitter.com/([^/]+)}, ->(match) { match[2] }],
-        [%r{://(.*\.)?patreon.com/([^/]+)}, ->(match) { match[2] }],
-        [%r{://(.*\.)?furaffinity.net/user/([^/]+)}, ->(match) { match[2] }],
-      ],
-      T::Array[[Regexp, T.proc.params(match: MatchData).returns(String)]],
-    )
-
-  sig { params(domain: String, host: String).returns(T::Boolean) }
-  def url_matches_domain?(domain, host)
-    host == domain || host.end_with?(".#{domain}")
-  end
-
-  sig { params(domain: String).returns(T.nilable(String)) }
-  def icon_path_for_domain(domain)
-    for test_domain, icon in DOMAIN_TO_ICON_PATH
-      if url_matches_domain?(test_domain, domain)
-        return asset_path("domain-icons/#{icon}")
-      end
-    end
-    nil
-  end
-
-  sig { params(url: String).returns(String) }
-  def title_for_url(url)
-    url = url.to_s
-    for mapper in DOMAIN_TITLE_MAPPERS
-      if (match = mapper[0].match(url)) && (group = mapper[1].call(match))
-        return group
-      end
-    end
-    url
-  end
-end

app/helpers/domain/e621/posts_helper.rb

@@ -1,5 +1,5 @@
 # typed: false
-module Domain::Posts::E621PostsHelper
+module Domain::E621::PostsHelper
   def icon_asset_for_url(url)
     domain = extract_domain(url)
     return nil unless domain
@@ -81,7 +81,7 @@ module Domain::Posts::E621PostsHelper
     return unless %w[www.furaffinity.net furaffinity.net].include?(uri.host)
     fa_id = uri.path.match(%r{/view/(\d+)})[1]
     return unless fa_id
-    raise("not implemented")
+    Domain::Fa::Post.find_by(fa_id: fa_id)
   rescue StandardError
     nil
   end

app/helpers/domain/fa/posts_helper.rb

@@ -0,0 +1,155 @@
+# typed: strict
+module Domain::Fa::PostsHelper
+  extend T::Sig
+
+  include ActionView::Helpers::DateHelper
+  include ActionView::Helpers::SanitizeHelper
+  include ActionView::Helpers::RenderingHelper
+  include ActionView::Helpers::TagHelper
+
+  sig { params(post: Domain::Fa::Post).returns(String) }
+  def post_state_string(post)
+    if post.have_file?
+      "file"
+    elsif post.scanned?
+      "scanned"
+    else
+      post.state || "unknown"
+    end
+  end
+
+  sig do
+    params(
+      params:
+        T.any(ActionController::Parameters, T::Hash[T.untyped, T.untyped]),
+    ).returns(T.nilable(String))
+  end
+  def page_str(params)
+    if (params[:page] || 1).to_i > 1
+      "(page #{params[:page]})"
+    else
+      nil
+    end
+  end
+
+  sig { params(post: Domain::Fa::Post).returns(T.nilable(HttpLogEntry)) }
+  def guess_scanned_http_log_entry(post)
+    HttpLogEntry.find_all_by_uri(
+      "https://www.furaffinity.net/view/#{post.fa_id}",
+    ).first
+  end
+
+  sig { params(post: Domain::Fa::Post).returns(T.nilable(HttpLogEntry)) }
+  def guess_file_downloaded_http_log_entry(post)
+    if (uri = post.file_uri)
+      HttpLogEntry.find_all_by_uri(uri).first
+    end
+  end
+
+  sig { params(html: String).returns(String) }
+  def fa_post_description_sanitized(html)
+    fa_post_id_to_node = {}
+    fa_user_url_name_to_node = {}
+
+    sanitizer =
+      Sanitize.new(
+        elements: %w[br img b i span strong],
+        attributes: {
+          "span" => %w[style],
+        },
+        css: {
+          properties: %w[font-size color],
+        },
+        transformers: [
+          Kernel.lambda do |env|
+            # Only allow and transform FA links
+            if env[:node_name] == "a"
+              node = env[:node]
+
+              # by default, assume the host is www.furaffinity.net
+              href = node["href"]&.downcase || ""
+              href = "//" + href if href.match?(/^(www\.)?furaffinity\.net/)
+              uri = URI.parse(href)
+              uri.host ||= "www.furaffinity.net"
+              path = uri.path
+
+              fa_host_matcher = /^(www\.)?furaffinity\.net$/
+              fa_post_matcher = %r{^/view/(\d+)/?$}
+              fa_user_matcher = %r{^/user/(\w+)/?$}
+
+              if fa_host_matcher.match?(uri.host) && path
+                if match = path.match(fa_post_matcher)
+                  fa_id = match[1].to_i
+                  fa_post_id_to_node[fa_id] = node
+                  next { node_whitelist: [node] }
+                elsif match = path.match(fa_user_matcher)
+                  fa_url_name = match[1]
+                  fa_user_url_name_to_node[fa_url_name] = node
+                  next { node_whitelist: [node] }
+                end
+              end
+
+              # Don't allow any other links
+              node.replace(node.children)
+            end
+          end,
+        ],
+      )
+
+    fragment = Nokogiri::HTML5.fragment(sanitizer.send(:preprocess, html))
+    sanitizer.node!(fragment)
+
+    if fa_post_id_to_node.any?
+      # Batch load posts and their titles, ensuring fa_post_ids are strings
+      posts_by_id =
+        Domain::Fa::Post.where(fa_id: fa_post_id_to_node.keys).index_by(&:fa_id)
+
+      # Replace the link text with post titles if available
+      fa_post_id_to_node.each do |fa_id, node|
+        if (post = posts_by_id[fa_id])
+          node.replace(
+            Nokogiri::HTML5.fragment(
+              render(
+                partial: "domain/fa/posts/description_inline_link_fa_post",
+                locals: {
+                  post: post,
+                },
+              ),
+            ),
+          )
+        else
+          node.replace(node.children)
+        end
+      end
+    end
+
+    if fa_user_url_name_to_node.any?
+      # Batch load users and their names, ensuring fa_user_url_names are strings
+      users_by_url_name =
+        Domain::Fa::User
+          .where(url_name: fa_user_url_name_to_node.keys)
+          .includes(:avatar)
+          .index_by(&:url_name)
+
+      # Replace the link text with user names if available
+      fa_user_url_name_to_node.each do |fa_url_name, node|
+        if (user = users_by_url_name[fa_url_name])
+          node.replace(
+            Nokogiri::HTML5.fragment(
+              render(
+                partial: "domain/fa/posts/description_inline_link_fa_user",
+                locals: {
+                  user: user,
+                },
+              ),
+            ),
+          )
+        else
+          node.replace(node.children)
+        end
+      end
+    end
+
+    raw fragment.to_html(preserve_newline: true)
+  end
+end

app/helpers/domain/fa/users_helper.rb

@@ -0,0 +1,113 @@
+# typed: false
+module Domain::Fa::UsersHelper
+  def avatar_url(sha256, thumb: "32-avatar")
+    blob_path(HexUtil.bin2hex(sha256), format: "jpg", thumb: thumb)
+  end
+
+  def fa_user_avatar_path(user, thumb: nil)
+    if (sha256 = user.avatar&.file_sha256)
+      blob_path(HexUtil.bin2hex(sha256), format: "jpg", thumb: thumb)
+    else
+      # default / 'not found' avatar image
+      # "/blobs/9080fd4e7e23920eb2dccfe2d86903fc3e748eebb2e5aa8c657bbf6f3d941cdc/contents.jpg"
+      asset_path("user-circle.svg")
+    end
+  end
+
+  def sanitized_fa_user_profile_html(html)
+    # try to preload all the FA usernames in the profile
+    maybe_url_names =
+      Nokogiri
+        .HTML(html)
+        .css("a")
+        .flat_map do |node|
+          href = URI.parse(node["href"])
+          right_host = href.host.nil? || href.host == "www.furaffinity.net"
+          right_path = href.path =~ %r{/user/.+}
+          if right_host && right_path
+            [href]
+          else
+            []
+          end
+        end
+        .map { |href| href.path.split("/")[2]&.downcase }
+
+    preloaded_users =
+      Domain::Fa::User
+        .where(url_name: maybe_url_names)
+        .select(:id, :state, :state_detail, :log_entry_detail, :url_name)
+        .joins(:avatar)
+        .includes(:avatar)
+        .index_by(&:url_name)
+
+    raw Sanitize.fragment(
+          html,
+          elements: %w[br img b i span strong],
+          attributes: {
+            "span" => %w[style],
+            "a" => [],
+          },
+          css: {
+            properties: %w[font-size color],
+          },
+          transformers:
+            lambda do |env|
+              return unless env[:node_name] == "a"
+              node = env[:node]
+              href = URI.parse(node["href"])
+              unless href.host == nil || href.host == "www.furaffinity.net"
+                return
+              end
+              return unless href.path =~ %r{/user/.+}
+              url_name = href.path.split("/")[2]&.downcase
+              Sanitize.node!(
+                node,
+                { elements: %w[a], attributes: { "a" => %w[href] } },
+              )
+              node["href"] = domain_fa_user_path(url_name)
+              node["class"] = "text-slate-200 underline decoration-slate-200 " +
+                "decoration-dashed decoration-dashed decoration-1"
+
+              whitelist = [node]
+
+              user =
+                preloaded_users[url_name] ||
+                  Domain::Fa::User.find_by(url_name: url_name)
+              if user
+                img = Nokogiri::XML::Node.new("img", node.document)
+                img["class"] = "inline w-5"
+                img["src"] = fa_user_avatar_path(user, thumb: "32-avatar")
+                node.prepend_child(img)
+                whitelist << img
+              end
+
+              { node_allowlist: whitelist }
+            end,
+        )
+  end
+
+  def similar_users_by_followed(user, limit: 10, exclude_followed_by: nil)
+    if user.disco.nil?
+      nil
+    else
+      ReduxApplicationRecord.connection.execute("SET ivfflat.probes = 32")
+      user.similar_users_by_followed(
+        exclude_followed_by: exclude_followed_by,
+      ).limit(limit)
+    end
+  end
+
+  def fa_user_account_status(user)
+    log_entry_id = user.log_entry_detail["last_user_page_id"]
+    return "unknown" if log_entry_id.nil?
+    log_entry = HttpLogEntry.find_by(id: log_entry_id)
+    return "unknown" if log_entry.nil?
+    parser =
+      Domain::Fa::Parser::Page.new(
+        log_entry.response.contents,
+        require_logged_in: false,
+      )
+    return "unknown" unless parser.probably_user_page?
+    parser.user_page.account_status
+  end
+end

app/helpers/domain/model_helper.rb

@@ -1,38 +0,0 @@
-# typed: strict
-module Domain::ModelHelper
-  extend T::Sig
-  extend T::Helpers
-  include HelpersInterface
-  include Pundit::Authorization
-  abstract!
-
-  sig do
-    params(
-      model: HasViewPrefix,
-      partial: String,
-      as: Symbol,
-      expires_in: ActiveSupport::Duration,
-    ).returns(T.nilable(String))
-  end
-  def render_for_model(model, partial, as:, expires_in: 1.hour)
-    cache_key = [model, policy(model), partial].compact
-    Rails
-      .cache
-      .fetch(cache_key, expires_in:) do
-        prefixes = lookup_context.prefixes
-        partial_path =
-          prefixes
-            .map { |prefix| "#{prefix}/#{model.class.view_prefix}/_#{partial}" }
-            .find { |view| lookup_context.exists?(view) } ||
-            prefixes
-              .map { |prefix| "#{prefix}/default/_#{partial}" }
-              .find { |view| lookup_context.exists?(view) } ||
-            Kernel.raise("no partial found for #{partial} in #{prefixes}")
-
-        partial_path = partial_path.split("/")
-        T.must(partial_path.last).delete_prefix!("_")
-        partial_path = partial_path.join("/")
-        render partial: partial_path, locals: { as => model }
-      end
-  end
-end

app/helpers/domain/pagination_helper.rb

@@ -1,85 +0,0 @@
-# typed: strict
-module Domain::PaginationHelper
-  extend T::Sig
-  extend T::Helpers
-  include HelpersInterface
-  abstract!
-
-  sig do
-    params(
-      collection: T.untyped,
-      link_text: String,
-      link_opts: T.untyped,
-    ).returns(String)
-  end
-  def collection_previous_page_link(collection, link_text, link_opts)
-    request_url = Addressable::URI.parse(request.url)
-    page = collection.current_page - 1
-    set_uri_page!(request_url, page)
-    path = get_uri_path_and_query(request_url)
-    link_to link_text, path, link_opts
-  end
-
-  sig do
-    params(
-      collection: T.untyped,
-      link_text: String,
-      link_opts: T.untyped,
-    ).returns(String)
-  end
-  def collection_next_page_link(collection, link_text, link_opts)
-    request_url = Addressable::URI.parse(request.url)
-    page = collection.current_page + 1
-    set_uri_page!(request_url, page)
-    path = get_uri_path_and_query(request_url)
-    link_to link_text, path, link_opts
-  end
-
-  sig do
-    params(
-      collection: T.untyped,
-      link_text: String,
-      link_opts: T.untyped,
-    ).returns(String)
-  end
-  def collection_previous_page_link(collection, link_text, link_opts)
-    request_url = Addressable::URI.parse(request.url)
-    page = collection.current_page - 1
-    set_uri_page!(request_url, page)
-    path = get_uri_path_and_query(request_url)
-    link_to link_text, path, link_opts
-  end
-
-  sig { params(uri: Addressable::URI, page: T.nilable(Integer)).void }
-  def set_uri_page!(uri, page)
-    query_values = uri.query_values || {}
-    if page.nil? || page <= 1
-      query_values.delete("page")
-    else
-      query_values["page"] = page
-    end
-
-    uri.query_values = query_values.empty? ? nil : query_values
-  end
-
-  sig { params(uri: Addressable::URI).returns(String) }
-  def get_uri_path_and_query(uri)
-    path = uri.path
-    path += "?#{uri.query}" if uri.query.present?
-    path
-  end
-
-  sig do
-    params(
-      params:
-        T.any(ActionController::Parameters, T::Hash[T.untyped, T.untyped]),
-    ).returns(T.nilable(String))
-  end
-  def page_str(params)
-    if (params[:page] || 1).to_i > 1
-      "(page #{params[:page]})"
-    else
-      nil
-    end
-  end
-end

app/helpers/domain/post_groups_helper.rb

@@ -1,17 +0,0 @@
-# typed: strict
-module Domain::PostGroupsHelper
-  extend T::Sig
-  extend T::Helpers
-  include HelpersInterface
-  abstract!
-
-  sig { params(post_group: Domain::PostGroup).returns(String) }
-  def domain_post_group_posts_path(post_group)
-    "#{domain_post_groups_path}/#{post_group.to_param}/posts"
-  end
-
-  sig { returns(String) }
-  def domain_post_groups_path
-    "/pools"
-  end
-end

app/helpers/domain/posts_helper.rb

@@ -1,672 +0,0 @@
-# typed: strict
-module Domain::PostsHelper
-  extend T::Sig
-  extend T::Helpers
-  include HelpersInterface
-  include LogEntriesHelper
-  include Domain::UsersHelper
-  include PathsHelper
-  include Domain::DomainsHelper
-  include Domain::DomainModelHelper
-  include Pundit::Authorization
-  require "base64"
-  abstract!
-
-  class DomainData < T::Struct
-    include T::Struct::ActsAsComparable
-
-    const :domain_icon_path, String
-    const :domain_icon_title, String
-  end
-
-  DEFAULT_DOMAIN_DATA =
-    DomainData.new(
-      domain_icon_path: "generic-domain.svg",
-      domain_icon_title: "Unknown",
-    )
-
-  DOMAIN_DATA =
-    T.let(
-      {
-        Domain::DomainType::Fa =>
-          DomainData.new(
-            domain_icon_path: "domain-icons/fa.png",
-            domain_icon_title: "Furaffinity",
-          ),
-        Domain::DomainType::E621 =>
-          DomainData.new(
-            domain_icon_path: "domain-icons/e621.png",
-            domain_icon_title: "E621",
-          ),
-        Domain::DomainType::Inkbunny =>
-          DomainData.new(
-            domain_icon_path: "domain-icons/inkbunny.png",
-            domain_icon_title: "Inkbunny",
-          ),
-        Domain::DomainType::Sofurry =>
-          DomainData.new(
-            domain_icon_path: "domain-icons/sofurry.png",
-            domain_icon_title: "SoFurry",
-          ),
-        Domain::DomainType::Bluesky =>
-          DomainData.new(
-            domain_icon_path: "domain-icons/bluesky.png",
-            domain_icon_title: "Bluesky",
-          ),
-      },
-      T::Hash[Domain::DomainType, DomainData],
-    )
-
-  sig { params(model: HasDomainType).returns(String) }
-  def domain_model_icon_path(model)
-    path =
-      if (domain_data = DOMAIN_DATA[model.domain_type])
-        domain_data.domain_icon_path
-      else
-        DEFAULT_DOMAIN_DATA.domain_icon_path
-      end
-    asset_path(path)
-  end
-
-  sig { params(post: Domain::Post).returns(T.nilable(Domain::PostFile)) }
-  def gallery_file_for_post(post)
-    file = post.primary_file_for_view
-    return nil unless file.present?
-    return nil unless file.state_ok? || file.last_status_code == 200
-    return nil unless file.log_entry_id.present?
-    content_type = file.log_entry&.content_type
-    return nil unless content_type.present?
-    return nil unless is_thumbable_content_type?(content_type)
-    file
-  end
-
-  sig { params(post: Domain::Post).returns(T.any(T.nilable(String), Symbol)) }
-  def gallery_file_info_for_post(post)
-    return :post_pending if post.pending_scan?
-    file = post.primary_file_for_view
-    return nil unless file.present?
-    return :file_pending if file.state_pending?
-    return nil unless file.state_ok?
-    return nil unless file.log_entry_id.present?
-    content_type = file.log_entry&.content_type || ""
-    pretty_content_type(content_type)
-  end
-
-  sig { params(post: Domain::Post).returns(T.nilable(String)) }
-  def thumbnail_for_post_path(post)
-    return nil unless policy(post).view_file?
-    file = gallery_file_for_post(post)
-    return nil unless file.present?
-    return nil unless file.state_ok?
-    return nil unless file.log_entry_id.present?
-    if (log_entry = file.log_entry) &&
-         (response_sha256 = log_entry.response_sha256)
-      blob_path(HexUtil.bin2hex(response_sha256), format: "jpg", thumb: "small")
-    end
-  end
-
-  # Create a data URI thumbnail from an image file
-  sig do
-    params(file_path: String, content_type: String, max_size: Integer).returns(
-      T.nilable(String),
-    )
-  end
-  def create_image_thumbnail_data_uri(file_path, content_type, max_size = 180)
-    # Load the Vips library properly instead of using require directly
-    begin
-      # Load the image
-      image = ::Vips::Image.new_from_file(file_path)
-
-      # Calculate the scaling factor to keep within max_size
-      scale = [max_size.to_f / image.width, max_size.to_f / image.height].min
-
-      # Only scale down, not up
-      scale = 1.0 if scale > 1.0
-
-      # Resize the image (use nearest neighbor for speed as this is just a thumbnail)
-      thumbnail = image.resize(scale)
-
-      # Get the image data in the original format
-      # For JPEG use quality 85 for a good balance of quality vs size
-      output_format = content_type.split("/").last
-
-      case output_format
-      when "jpeg", "jpg"
-        image_data = thumbnail.write_to_buffer(".jpg", Q: 85)
-      when "png"
-        image_data = thumbnail.write_to_buffer(".png", compression: 6)
-      when "gif"
-        image_data = thumbnail.write_to_buffer(".gif")
-      else
-        # Default to JPEG if format not recognized
-        image_data = thumbnail.write_to_buffer(".jpg", Q: 85)
-        content_type = "image/jpeg"
-      end
-
-      # Create the data URI
-      "data:#{content_type};base64,#{Base64.strict_encode64(image_data)}"
-    rescue => e
-      Rails.logger.error("Error creating thumbnail: #{e.message}")
-      nil
-    end
-  end
-
-  sig { params(content_type: String).returns(String) }
-  def pretty_content_type(content_type)
-    case content_type
-    when %r{text/plain}
-      "Plain Text Document"
-    when %r{application/pdf}
-      "PDF Document"
-    when %r{application/msword}
-      "Microsoft Word Document"
-    when %r{application/vnd\.openxmlformats-officedocument\.wordprocessingml\.document}
-      "Microsoft Word Document (OpenXML)"
-    when %r{application/vnd\.oasis\.opendocument\.text}
-      "OpenDocument Text"
-    when %r{application/rtf}
-      "Rich Text Document"
-    when %r{image/jpeg}
-      "JPEG Image"
-    when %r{image/png}
-      "PNG Image"
-    when %r{image/gif}
-      "GIF Image"
-    when %r{video/webm}
-      "Webm Video"
-    when %r{audio/mpeg}
-      "MP3 Audio"
-    when %r{audio/mp3}
-      "MP3 Audio"
-    when %r{audio/wav}
-      "WAV Audio"
-    else
-      content_type.split(";").first&.split("/")&.last&.titleize || "Unknown"
-    end
-  end
-
-  sig { params(post: Domain::Post).returns(T.nilable(String)) }
-  def gallery_file_size_for_post(post)
-    file = post.primary_file_for_view
-    return nil unless file.present?
-    return nil unless file.state_ok?
-    return nil unless file.log_entry_id.present?
-    file.log_entry&.response_size&.then { |size| number_to_human_size(size) }
-  end
-
-  sig do
-    params(
-      post_files: T::Array[Domain::PostFile],
-      initial_file_index: T.nilable(Integer),
-    ).returns(T::Hash[Symbol, T.untyped])
-  end
-  def props_for_post_files(post_files:, initial_file_index: nil)
-    files_data =
-      post_files.map.with_index do |post_file, index|
-        thumbnail_path = nil
-        content_html = nil
-        log_entry = post_file.log_entry
-
-        if log_entry && (log_entry.status_code == 200)
-          if (response_sha256 = log_entry.response_sha256)
-            thumbnail_path = {
-              type: "url",
-              value:
-                blob_path(
-                  HexUtil.bin2hex(response_sha256),
-                  format: "jpg",
-                  thumb: "small",
-                ),
-            }
-          end
-
-          # Generate content HTML
-          content_html =
-            ApplicationController.renderer.render(
-              partial: "log_entries/content_container",
-              locals: {
-                log_entry: log_entry,
-              },
-              assigns: {
-                current_user: nil,
-              },
-            )
-        elsif post_file.state_pending?
-          thumbnail_path = {
-            type: "icon",
-            value: "fa-solid fa-file-arrow-down",
-          }
-        end
-
-        {
-          id: post_file.id,
-          fileState: post_file.state,
-          thumbnailPath: thumbnail_path,
-          hasContent: post_file.log_entry&.status_code == 200,
-          index: index,
-          contentHtml: content_html,
-          fileDetails:
-            (
-              if log_entry
-                {
-                  contentType: log_entry.content_type,
-                  fileSize: log_entry.response_size,
-                  responseTimeMs: log_entry.response_time_ms,
-                  responseStatusCode: log_entry.status_code,
-                  postFileState: post_file.state,
-                  logEntryId: log_entry.id,
-                  logEntryPath: log_entry_path(log_entry),
-                }
-              else
-                nil
-              end
-            ),
-        }
-      end
-
-    # Validate initial_file_index
-    validated_initial_index = nil
-    if initial_file_index && initial_file_index >= 0 &&
-         initial_file_index < post_files.count
-      validated_initial_index = initial_file_index
-    end
-
-    { files: files_data, initialSelectedIndex: validated_initial_index }
-  end
-
-  sig { params(url: String).returns(T.nilable(String)) }
-  def icon_asset_for_url(url)
-    domain = extract_domain(url)
-    return nil unless domain
-    icon_path_for_domain(domain)
-  end
-
-  sig { params(category: Symbol).returns(String) }
-  def tailwind_tag_category_class(category)
-    case category
-    when :general
-      "bg-blue-300" # Light blue
-    when :artist
-      "bg-indigo-300" # Light indigo
-    when :copyright
-      "bg-purple-300" # Light purple
-    when :character
-      "bg-green-300" # Light green
-    when :species
-      "bg-teal-300" # Light teal
-    when :invalid
-      "bg-slate-300" # Medium gray
-    when :meta
-      "bg-amber-300" # Light amber
-    when :lore
-      "bg-cyan-300" # Light cyan
-    else
-      "bg-white" # White (default)
-    end
-  end
-
-  sig { returns(T::Array[Symbol]) }
-  def tag_category_order
-    TAG_CATEGORY_ORDER
-  end
-
-  sig { params(category: Symbol).returns(T.nilable(String)) }
-  def font_awesome_category_icon(category)
-    case category
-    when :artist
-      "fa-brush"
-    when :species
-      "fa-paw"
-    when :character
-      "fa-user"
-    when :copyright
-      "fa-copyright"
-    when :general
-      "fa-tag"
-    when :lore
-      "fa-book"
-    when :meta
-      "fa-info"
-    when :invalid
-      "fa-ban"
-    end
-  end
-
-  class LinkForSource < T::ImmutableStruct
-    include T::Struct::ActsAsComparable
-
-    const :model, ReduxApplicationRecord
-    const :title, String
-    const :model_path, String
-    const :icon_path, T.nilable(String)
-  end
-
-  class SourceResult < T::ImmutableStruct
-    include T::Struct::ActsAsComparable
-
-    const :model, ReduxApplicationRecord
-    const :title, String
-  end
-
-  class SourceMatcher < T::ImmutableStruct
-    extend T::Generic
-    include T::Struct::ActsAsComparable
-
-    const :hosts, T::Array[String]
-    const :patterns, T::Array[Regexp]
-    const :find_proc,
-          T
-            .proc
-            .params(helper: Domain::PostsHelper, match: MatchData, url: String)
-            .returns(T.nilable(SourceResult))
-  end
-
-  FA_HOSTS = %w[*.furaffinity.net furaffinity.net]
-  FA_CDN_HOSTS = %w[d.furaffinity.net *.facdn.net facdn.net]
-  IB_HOSTS = %w[*.inkbunny.net inkbunny.net]
-  IB_CDN_HOSTS = %w[*.ib.metapix.net ib.metapix.net]
-  E621_HOSTS = %w[www.e621.net e621.net]
-  BLUESKY_HOSTS = %w[bsky.app]
-
-  URL_SUFFIX_QUERY = T.let(<<-SQL.strip.chomp.freeze, String)
-    lower('url_str') = lower(?)
-  SQL
-
-  MATCHERS =
-    T.let(
-      [
-        # Furaffinity posts
-        SourceMatcher.new(
-          hosts: FA_HOSTS,
-          patterns: [
-            %r{/view/(\d+)/?},
-            %r{/full/(\d+)/?},
-            %r{/controls/submissions/changeinfo/(\d+)/?},
-          ],
-          find_proc: ->(helper, match, _) do
-            if post = Domain::Post::FaPost.find_by(fa_id: match[1])
-              SourceResult.new(model: post, title: post.title_for_view)
-            end
-          end,
-        ),
-        # Furaffinity posts via direct file URL
-        SourceMatcher.new(
-          hosts: FA_CDN_HOSTS,
-          patterns: [/.+/],
-          find_proc: ->(helper, _, url) do
-            url = Addressable::URI.parse(url)
-
-            post_file =
-              Domain::PostFile.where(
-                "lower('url_str') IN (?, ?, ?, ?, ?, ?)",
-                "d.furaffinity.net#{url.host}/#{url.path}",
-                "//d.furaffinity.net#{url.host}/#{url.path}",
-                "https://d.furaffinity.net#{url.host}/#{url.path}",
-                "d.facdn.net#{url.host}/#{url.path}",
-                "//d.facdn.net#{url.host}/#{url.path}",
-                "https://d.facdn.net#{url.host}/#{url.path}",
-              ).first
-
-            if post_file && (post = post_file.post)
-              SourceResult.new(model: post, title: post.title_for_view)
-            end
-          end,
-        ),
-        # Furaffinity users
-        SourceMatcher.new(
-          hosts: FA_HOSTS,
-          patterns: [%r{/user/([^/]+)/?}],
-          find_proc: ->(helper, match, _) do
-            if user = Domain::User::FaUser.find_by(url_name: match[1])
-              SourceResult.new(
-                model: user,
-                title: user.name_for_view || "unknown",
-              )
-            end
-          end,
-        ),
-        # Inkbunny posts
-        SourceMatcher.new(
-          hosts: IB_HOSTS,
-          patterns: [%r{/s/(\d+)/?}, %r{/submissionview\.php\?id=(\d+)/?}],
-          find_proc: ->(helper, match, _) do
-            if post = Domain::Post::InkbunnyPost.find_by(ib_id: match[1])
-              SourceResult.new(model: post, title: post.title_for_view)
-            end
-          end,
-        ),
-        # Inkbunny posts via direct file URL
-        SourceMatcher.new(
-          hosts: IB_CDN_HOSTS,
-          patterns: [//],
-          find_proc: ->(helper, _, url) do
-            url = Addressable::URI.parse(url)
-            if post_file =
-                 Domain::PostFile.where(
-                   "#{URL_SUFFIX_QUERY}",
-                   "ib.metapix.net#{url.path}",
-                 ).first
-              if post = post_file.post
-                SourceResult.new(model: post, title: post.title_for_view)
-              end
-            end
-          end,
-        ),
-        # Inkbunny users
-        SourceMatcher.new(
-          hosts: IB_HOSTS,
-          patterns: [%r{/(\w+)/?$}],
-          find_proc: ->(_, match, _) do
-            if user =
-                 Domain::User::InkbunnyUser.where(
-                   "name = lower(?)",
-                   match[1],
-                 ).first
-              SourceResult.new(
-                model: user,
-                title: user.name_for_view || "unknown",
-              )
-            end
-          end,
-        ),
-        # E621 posts
-        SourceMatcher.new(
-          hosts: E621_HOSTS,
-          patterns: [%r{/posts/(\d+)/?}],
-          find_proc: ->(helper, match, _) do
-            if post = Domain::Post::E621Post.find_by(e621_id: match[1])
-              SourceResult.new(model: post, title: post.title_for_view)
-            end
-          end,
-        ),
-        # E621 users
-        SourceMatcher.new(
-          hosts: E621_HOSTS,
-          patterns: [%r{/users/(\d+)/?}],
-          find_proc: ->(helper, match, _) do
-            if user = Domain::User::E621User.find_by(e621_id: match[1])
-              SourceResult.new(
-                model: user,
-                title: user.name_for_view || "unknown",
-              )
-            end
-          end,
-        ),
-        # Bluesky posts
-        SourceMatcher.new(
-          hosts: BLUESKY_HOSTS,
-          patterns: [%r{/profile/([^/]+)/post/([^/]+)/?$}],
-          find_proc: ->(helper, match, _) do
-            handle_or_did = match[1]
-            post_rkey = match[2]
-            if handle_or_did.start_with?("did:")
-              did = handle_or_did
-            else
-              user = Domain::User::BlueskyUser.find_by(handle: handle_or_did)
-              did = user&.did
-            end
-            next unless did
-            at_uri = "at://#{did}/app.bsky.feed.post/#{post_rkey}"
-            post = Domain::Post::BlueskyPost.find_by(at_uri:)
-            SourceResult.new(model: post, title: post.title_for_view) if post
-          end,
-        ),
-        # Bluesky users
-        SourceMatcher.new(
-          hosts: BLUESKY_HOSTS,
-          patterns: [%r{/profile/([^/]+)\/?$}],
-          find_proc: ->(helper, match, _) do
-            handle_or_did = match[1]
-            user =
-              if handle_or_did.start_with?("did:")
-                Domain::User::BlueskyUser.find_by(did: handle_or_did)
-              else
-                Domain::User::BlueskyUser.find_by(handle: handle_or_did)
-              end
-            next unless user
-            SourceResult.new(
-              model: user,
-              title: user.name_for_view || handle_or_did,
-            )
-          end,
-        ),
-      ],
-      T::Array[SourceMatcher],
-    )
-
-  sig { params(source: String).returns(T.nilable(LinkForSource)) }
-  def link_for_source(source)
-    return nil if source.blank?
-
-    # normalize the source to a lowercase string with a protocol
-    source = source.downcase
-    source = "https://" + source unless source.include?("://")
-    begin
-      uri = URI.parse(source)
-    rescue StandardError
-      return nil
-    end
-    uri_host = uri.host
-    return nil if uri_host.blank?
-
-    for matcher in MATCHERS
-      if matcher.hosts.any? { |host|
-           File.fnmatch?(host, uri_host, File::FNM_PATHNAME)
-         }
-        for pattern in matcher.patterns
-          if (match = pattern.match(uri.to_s))
-            object = matcher.find_proc.call(self, match, uri.to_s)
-            return nil unless object
-            model = object.model
-
-            if model.is_a?(Domain::Post)
-              model_path =
-                Rails.application.routes.url_helpers.domain_post_path(model)
-            elsif model.is_a?(Domain::User)
-              model_path =
-                Rails.application.routes.url_helpers.domain_user_path(model)
-              icon_path =
-                domain_user_avatar_img_src_path(
-                  model.avatar,
-                  thumb: "64-avatar",
-                )
-            else
-              model_path = "#"
-            end
-
-            return(
-              LinkForSource.new(
-                model:,
-                title: object.title,
-                model_path:,
-                icon_path:,
-              )
-            )
-          end
-        end
-      end
-    end
-    nil
-  end
-
-  sig { params(post: Domain::Post::FaPost).returns(T::Array[String]) }
-  def keywords_for_fa_post(post)
-    post.keywords.map(&:strip).reject(&:blank?).compact
-  end
-
-  sig do
-    params(post: Domain::Post::InkbunnyPost).returns(
-      T.nilable(T::Array[String]),
-    )
-  end
-  def keywords_for_ib_post(post)
-    post.keywords&.map { |keyword| keyword["keyword_name"] }&.compact
-  end
-
-  sig do
-    params(time: T.nilable(T.any(ActiveSupport::TimeWithZone, Time))).returns(
-      String,
-    )
-  end
-  def time_ago_in_words_no_prefix(time)
-    return "never" if time.nil?
-    time = time.in_time_zone if time.is_a?(Time)
-    time_ago_in_words(time).delete_prefix("over ").delete_prefix("about ")
-  end
-
-  sig do
-    params(faved_at_type: Domain::UserPostFav::FavedAtType).returns(String)
-  end
-  def faved_at_type_icon(faved_at_type)
-    case faved_at_type
-    when Domain::UserPostFav::FavedAtType::PostedAt
-      "fa-clock" # Clock icon for fallback to posted_at
-    when Domain::UserPostFav::FavedAtType::Explicit
-      "fa-calendar-check" # Calendar check for explicitly set time
-    when Domain::UserPostFav::FavedAtType::Inferred
-      "fa-chart-line" # Chart line for inferred from regression model
-    when Domain::UserPostFav::FavedAtType::InferredNow
-      "fa-bolt" # Lightning bolt for computed on the fly
-    end
-  end
-
-  sig do
-    params(faved_at_type: Domain::UserPostFav::FavedAtType).returns(String)
-  end
-  def faved_at_type_tooltip(faved_at_type)
-    case faved_at_type
-    when Domain::UserPostFav::FavedAtType::PostedAt
-      "Estimated from posted date"
-    when Domain::UserPostFav::FavedAtType::Explicit
-      "Exact time recorded"
-    when Domain::UserPostFav::FavedAtType::Inferred
-      "Estimated from regression model"
-    when Domain::UserPostFav::FavedAtType::InferredNow
-      "Estimated in real-time from regression model"
-    end
-  end
-
-  sig { returns(T::Hash[Symbol, T.untyped]) }
-  def props_for_visual_search_form
-    {
-      actionUrl:
-        Rails.application.routes.url_helpers.visual_results_domain_posts_path,
-      csrfToken: form_authenticity_token,
-    }
-  end
-
-  private
-
-  sig { params(url: String).returns(T.nilable(String)) }
-  def extract_domain(url)
-    URI.parse(url).host
-  rescue URI::InvalidURIError
-    nil
-  end
-
-  TAG_CATEGORY_ORDER =
-    T.let(
-      %i[artist copyright character species general meta lore invalid].freeze,
-      T::Array[Symbol],
-    )
-end

app/helpers/domain/user_job_event_helper.rb

@@ -1,36 +0,0 @@
-# typed: strict
-module Domain::UserJobEventHelper
-  extend T::Sig
-  include HelpersInterface
-
-  sig { params(kind: String).returns(String) }
-  def add_tracked_object_kind_for_view(kind)
-    case kind
-    when "favs"
-      "Favs"
-    else
-      kind.titleize
-    end
-  end
-
-  sig { params(kind: String).returns(String) }
-  def add_tracked_object_kind_event_name(kind)
-    case kind
-    when "favs"
-      "favs scan"
-    else
-      "#{kind} scan"
-    end
-  end
-
-  sig { params(duration: ActiveSupport::Duration).returns(String) }
-  def format_duration_since_last_scan(duration)
-    if duration.in_days >= 1
-      pluralize(duration.in_days.round(0), "day")
-    elsif duration.in_hours >= 1
-      pluralize(duration.in_hours.round(0), "hour")
-    else
-      pluralize(duration.in_minutes.round(0), "min")
-    end
-  end
-end

app/helpers/domain/users/fa_users_helper.rb

@@ -1,32 +0,0 @@
-# typed: true
-module Domain::Users::FaUsersHelper
-  extend T::Sig
-  include HelpersInterface
-
-  # TODO - remove this once we've migrated similarity scores to new user model
-  sig do
-    params(
-      user: Domain::User::FaUser,
-      limit: Integer,
-      exclude_followed_by: T.nilable(Domain::User::FaUser),
-    ).returns(T::Array[Domain::User::FaUser])
-  end
-  def similar_users_by_followed(user, limit: 10, exclude_followed_by: nil)
-    factors = Domain::Factors::UserUserFollowToFactors.find_by(user: user)
-    return [] if factors.nil?
-
-    relation =
-      Domain::NeighborFinder
-        .find_neighbors(factors)
-        .limit(limit)
-        .includes(:user)
-    if exclude_followed_by
-      relation =
-        relation.where.not(
-          user_id: exclude_followed_by.followed_users.select(:to_id),
-        )
-    end
-
-    relation.map { |factor| factor.user }
-  end
-end

app/helpers/domain/users_helper.rb

@@ -1,290 +0,0 @@
-# typed: strict
-module Domain::UsersHelper
-  extend T::Sig
-  extend T::Helpers
-  include HelpersInterface
-  include Pundit::Authorization
-  abstract!
-
-  sig { params(user: Domain::User).returns(T.nilable(String)) }
-  def user_avatar_path_for_view(user)
-    if avatar = user.avatar
-      domain_user_avatar_img_src_path(avatar, thumb: "32-avatar")
-    end
-  end
-
-  sig { params(user: Domain::User).returns(T.nilable(String)) }
-  def user_name_for_view(user)
-    if avatar = user.avatar
-      domain_user_avatar_img_src_path(avatar, thumb: "32-avatar")
-    end
-  end
-
-  sig { params(user: Domain::User).returns(T.nilable(String)) }
-  def domain_user_registered_at_string_for_view(user)
-    ts = domain_user_registered_at_ts_for_view(user)
-    ts ? time_ago_in_words(ts) : nil
-  end
-
-  sig do
-    params(user: Domain::User).returns(T.nilable(ActiveSupport::TimeWithZone))
-  end
-  def domain_user_registered_at_ts_for_view(user)
-    case user
-    when Domain::User::FaUser, Domain::User::E621User, Domain::User::BlueskyUser
-      user.registered_at
-    else
-      nil
-    end
-  end
-
-  sig do
-    params(
-      avatar: T.nilable(Domain::UserAvatar),
-      thumb: T.nilable(String),
-    ).returns(String)
-  end
-  def domain_user_avatar_img_src_path(avatar, thumb: nil)
-    cache_key = ["domain_user_avatar_img_src_path", avatar&.id, thumb]
-    Rails
-      .cache
-      .fetch(cache_key, expires_in: 1.day) do
-        if (sha256 = avatar&.log_entry&.response_sha256)
-          Rails.application.routes.url_helpers.blob_path(
-            HexUtil.bin2hex(sha256),
-            format: "jpg",
-            thumb: thumb,
-          )
-        elsif avatar && avatar.state_file_404? &&
-              (sha256 = avatar.last_log_entry&.response_sha256)
-          Rails.application.routes.url_helpers.blob_path(
-            HexUtil.bin2hex(sha256),
-            format: "jpg",
-            thumb: thumb,
-          )
-        else
-          # default / 'not found' avatar image
-          # "/blobs/9080fd4e7e23920eb2dccfe2d86903fc3e748eebb2e5aa8c657bbf6f3d941cdc/contents.jpg"
-          asset_path("user-circle.svg")
-        end
-      end
-  end
-  sig do
-    params(
-      avatar: T.nilable(Domain::UserAvatar),
-      thumb: T.nilable(String),
-    ).returns(String)
-  end
-  def domain_user_avatar_img_tag(avatar, thumb: nil)
-    if (sha256 = avatar&.log_entry&.response_sha256)
-      image_tag(
-        blob_path(HexUtil.bin2hex(sha256), format: "jpg", thumb: thumb),
-        class: "inline-block h-4 w-4 flex-shrink-0 rounded-sm object-cover",
-        alt: avatar&.user&.name_for_view || "user avatar",
-      )
-    else
-      raw <<~SVG
-      <svg xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" stroke-width="1.5" stroke="currentColor" class="w-4 h-4">
-        <path stroke-linecap="round" stroke-linejoin="round" d="M15.75 6a3.75 3.75 0 11-7.5 0 3.75 3.75 0 017.5 0zM4.501 20.118a7.5 7.5 0 0114.998 0A17.933 17.933 0 0112 21.75c-2.676 0-5.216-.584-7.499-1.632z" />
-      </svg>
-      SVG
-    end
-  end
-
-  sig { params(user: Domain::User).returns(String) }
-  def site_icon_path_for_user(user)
-    case user
-    when Domain::User::E621User
-      asset_path("domain-icons/e621.png")
-    when Domain::User::FaUser
-      asset_path("domain-icons/fa.png")
-    when Domain::User::InkbunnyUser
-      asset_path("domain-icons/inkbunny.png")
-    when Domain::User::SofurryUser
-      asset_path("domain-icons/sofurry.png")
-    when Domain::User::BlueskyUser
-      asset_path("domain-icons/bluesky.png")
-    else
-      Kernel.raise "Unknown user type: #{user.class}"
-    end
-  end
-
-  sig { params(user: Domain::User).returns(String) }
-  def domain_user_path(user)
-    "#{domain_users_path}/#{user.to_param}"
-  end
-
-  sig { returns(String) }
-  def domain_users_path
-    "/users"
-  end
-
-  sig { params(user: Domain::User).returns(String) }
-  def domain_user_posts_path(user)
-    "#{domain_user_path(user)}/posts"
-  end
-
-  sig { params(user: Domain::User).returns(String) }
-  def domain_user_favorites_path(user)
-    "#{domain_user_path(user)}/favorites"
-  end
-
-  sig { params(user: Domain::User).returns(String) }
-  def domain_user_followed_by_path(user)
-    "#{domain_user_path(user)}/followed_by"
-  end
-
-  sig { params(user: Domain::User).returns(String) }
-  def domain_user_following_path(user)
-    "#{domain_user_path(user)}/following"
-  end
-
-  sig { params(user: Domain::User, kind: String).returns(String) }
-  def tracked_objects_domain_user_job_events_path(user, kind:)
-    unless Domain::UserJobEvent::AddTrackedObject.kinds.include?(kind)
-      Kernel.raise "invalid kind: #{kind}"
-    end
-    "#{domain_user_path(user)}/job_events/tracked_objects/#{kind}"
-  end
-
-  sig { params(user: Domain::User).returns(String) }
-  def tracked_objects_domain_user_job_events_kinds_path(user)
-    "#{domain_user_path(user)}/job_events/tracked_objects"
-  end
-
-  sig { params(user: Domain::User, kind: String).returns(String) }
-  def enqueue_scan_job_domain_user_job_events_path(user, kind:)
-    unless Domain::UserJobEvent::AddTrackedObject.kinds.include?(kind)
-      Kernel.raise "invalid kind: #{kind}"
-    end
-    "#{domain_user_path(user)}/job_events/enqueue_scan_job/#{kind}"
-  end
-
-  class StatRow < T::ImmutableStruct
-    include T::Struct::ActsAsComparable
-
-    const :name, String
-    const :value,
-          T.nilable(
-            T.any(String, Integer, HasTimestampsWithDueAt::TimestampScanInfo),
-          )
-    const :link_to, T.nilable(String)
-    const :fa_icon_class, T.nilable(String)
-    const :hover_title, T.nilable(String)
-  end
-
-  sig { params(user: Domain::User).returns(T::Array[StatRow]) }
-  def stat_rows_for_user(user)
-    rows = T.let([], T::Array[StatRow])
-    if user.has_faved_posts?
-      rows << StatRow.new(name: "Favorites", value: user.user_post_favs.size)
-    end
-    if user.has_followed_by_users?
-      can_view_link = policy(user).followed_by?
-      rows << StatRow.new(
-        name: "Followed by",
-        value: user.user_user_follows_to.size,
-        link_to: can_view_link ? domain_user_followed_by_path(user) : nil,
-      )
-    end
-    if user.has_followed_users?
-      can_view_link = policy(user).following?
-      rows << StatRow.new(
-        name: "Following",
-        value: user.user_user_follows_from.size,
-        link_to: can_view_link ? domain_user_following_path(user) : nil,
-      )
-    end
-
-    can_view_timestamps = policy(user).view_page_scanned_at_timestamps?
-    can_view_log_entries = policy(user).view_log_entries?
-    icon_for =
-      Kernel.proc do |due_for_scan|
-        due_for_scan ? "fa-hourglass-half" : "fa-check"
-      end
-
-    if user.is_a?(Domain::User::BlueskyUser) && can_view_timestamps
-      rows << StatRow.new(
-        name: "Page scanned",
-        value: user.profile_scan,
-        link_to:
-          user.last_scan_log_entry && log_entry_path(user.last_scan_log_entry),
-        fa_icon_class: icon_for.call(user.profile_scan.due?),
-        hover_title: user.profile_scan.interval.inspect,
-      )
-
-      rows << StatRow.new(
-        name: "Posts scanned",
-        value: user.posts_scan,
-        link_to:
-          user.last_posts_scan_log_entry &&
-            log_entry_path(user.last_posts_scan_log_entry),
-        fa_icon_class: icon_for.call(user.posts_scan.due?),
-        hover_title: user.posts_scan.interval.inspect,
-      )
-    end
-
-    if user.is_a?(Domain::User::FaUser) && can_view_timestamps
-      if can_view_log_entries && hle = user.guess_last_user_page_log_entry
-        rows << StatRow.new(
-          name: "Page scanned",
-          value: user.page_scan,
-          link_to: log_entry_path(hle),
-          fa_icon_class: icon_for.call(user.page_scan.due?),
-          hover_title: user.page_scan.interval.inspect,
-        )
-      else
-        rows << StatRow.new(
-          name: "Page",
-          value: user.page_scan,
-          fa_icon_class: icon_for.call(user.page_scan.due?),
-          hover_title: user.page_scan.interval.inspect,
-        )
-      end
-
-      rows << StatRow.new(
-        name: "Favs",
-        value: user.favs_scan,
-        link_to:
-          tracked_objects_domain_user_job_events_path(user, kind: "favs"),
-        fa_icon_class: icon_for.call(user.favs_scan.due?),
-        hover_title: user.favs_scan.interval.inspect,
-      )
-      rows << StatRow.new(
-        name: "Followed by",
-        value: user.followed_by_scan,
-        fa_icon_class: icon_for.call(user.followed_by_scan.due?),
-        hover_title: user.followed_by_scan.interval.inspect,
-      )
-      rows << StatRow.new(
-        name: "Following",
-        value: user.follows_scan,
-        fa_icon_class: icon_for.call(user.follows_scan.due?),
-        hover_title: user.follows_scan.interval.inspect,
-      )
-      rows << StatRow.new(
-        name: "Gallery",
-        value: user.gallery_scan,
-        fa_icon_class: icon_for.call(user.gallery_scan.due?),
-        hover_title: user.gallery_scan.interval.inspect,
-      )
-    elsif user.is_a?(Domain::User::E621User) && can_view_timestamps
-      if user.favs_are_hidden
-        rows << StatRow.new(name: "Favorites hidden", value: "yes")
-      else
-        rows << StatRow.new(
-          name: "Server favorites",
-          value: user.num_other_favs_cached,
-        )
-      end
-      rows << StatRow.new(
-        name: "Favorites",
-        value: user.favs_scan,
-        fa_icon_class: icon_for.call(user.favs_scan.due?),
-        hover_title: user.favs_scan.interval.inspect,
-      )
-    end
-
-    rows
-  end
-end

app/helpers/domain/visual_search_helper.rb

@@ -1,190 +0,0 @@
-# typed: strict
-module Domain
-  module VisualSearchHelper
-    extend T::Sig
-
-    # Calculate the similarity percentage between two fingerprint hash values
-    # @param hash_value [String] The hash value to compare
-    # @param reference_hash_value [String] The reference hash value to compare against
-    # @return [Float] The similarity percentage between 0 and 100
-    sig do
-      params(hash_value: String, reference_hash_value: String).returns(Float)
-    end
-    def calculate_similarity_percentage(hash_value, reference_hash_value)
-      # Calculate hamming distance between the two hash values
-      distance =
-        DHashVips::IDHash.distance(
-          hash_value.to_i(2),
-          reference_hash_value.to_i(2),
-        )
-
-      max_distance = 256
-      # # Calculate similarity percentage based on distance
-      ((max_distance - distance) / max_distance.to_f * 100).round(1).clamp(
-        0,
-        100,
-      )
-    end
-
-    # Determine the background color class based on similarity percentage
-    # @param similarity_percentage [Float] The similarity percentage between 0 and 100
-    # @return [String] The Tailwind CSS background color class
-    sig { params(similarity_percentage: Float).returns(String) }
-    def match_badge_bg_color(similarity_percentage)
-      case similarity_percentage
-      when 90..100
-        "bg-green-600"
-      when 70..89
-        "bg-blue-600"
-      when 50..69
-        "bg-amber-500"
-      else
-        "bg-slate-700"
-      end
-    end
-
-    # Determine the text color class based on similarity percentage
-    # @param similarity_percentage [Float] The similarity percentage between 0 and 100
-    # @return [String] The Tailwind CSS text color class
-    sig { params(similarity_percentage: Float).returns(String) }
-    def match_text_color(similarity_percentage)
-      case similarity_percentage
-      when 90..100
-        "text-green-700"
-      when 70..89
-        "text-blue-700"
-      when 50..69
-        "text-amber-700"
-      else
-        "text-slate-700"
-      end
-    end
-
-    # Get the CSS classes for the match percentage badge
-    # @param similarity_percentage [Float] The similarity percentage between 0 and 100
-    # @return [String] The complete CSS classes for the match percentage badge
-    sig { params(similarity_percentage: Float).returns(String) }
-    def match_badge_classes(similarity_percentage)
-      "#{match_badge_bg_color(similarity_percentage)} text-white text-xs rounded-full px-3 py-1 shadow-md"
-    end
-
-    class SimilarFingerprintResult < T::Struct
-      include T::Struct::ActsAsComparable
-      const :fingerprint, Domain::PostFile::BitFingerprint
-      const :similarity_percentage, Float
-    end
-
-    class FingerprintAndDetail < T::Struct
-      include T::Struct::ActsAsComparable
-      const :fingerprint, String
-      const :detail_fingerprint, String
-    end
-
-    # Find similar images based on the fingerprint
-    sig do
-      params(
-        fingerprints: T::Array[FingerprintAndDetail],
-        limit: Integer,
-        oversearch: Integer,
-        includes: T.untyped,
-      ).returns(T::Array[SimilarFingerprintResult])
-    end
-    def find_similar_fingerprints(
-      fingerprints,
-      limit: 32,
-      oversearch: 2,
-      includes: {}
-    )
-      ActiveRecord::Base.connection.execute("SET ivfflat.probes = 20")
-
-      results =
-        fingerprints.flat_map do |f|
-          Domain::PostFile::BitFingerprint
-            .order(
-              Arel.sql "(fingerprint_value <~> '#{ActiveRecord::Base.connection.quote_string(f.fingerprint)}')"
-            )
-            .limit(limit * oversearch)
-            .includes(includes)
-            .to_a
-            .uniq(&:post_file_id)
-            .map do |other_fingerprint|
-              SimilarFingerprintResult.new(
-                fingerprint: other_fingerprint,
-                similarity_percentage:
-                  calculate_similarity_percentage(
-                    f.detail_fingerprint,
-                    T.must(other_fingerprint.fingerprint_detail_value),
-                  ),
-              )
-            end
-            .sort { |a, b| b.similarity_percentage <=> a.similarity_percentage }
-            .take(limit)
-        end
-
-      results
-        .group_by { |s| T.must(s.fingerprint.post_file_id) }
-        .map do |post_file_id, similar_fingerprints|
-          T.must(similar_fingerprints.max_by(&:similarity_percentage))
-        end
-        .sort_by(&:similarity_percentage)
-        .reverse
-    end
-
-    class GenerateFingerprintsResult < T::Struct
-      extend T::Sig
-      include T::Struct::ActsAsComparable
-
-      const :thumb_path, String
-      const :fingerprint, String
-      const :detail_fingerprint, String
-
-      sig { returns(FingerprintAndDetail) }
-      def to_fingerprint_and_detail
-        FingerprintAndDetail.new(
-          fingerprint: fingerprint,
-          detail_fingerprint: detail_fingerprint,
-        )
-      end
-    end
-
-    # Generate a fingerprint from the image path
-    sig do
-      params(image_path: String, content_type: String, tmp_dir: String).returns(
-        T.nilable(T::Array[GenerateFingerprintsResult]),
-      )
-    end
-    def generate_fingerprints(image_path, content_type, tmp_dir)
-      # Use the new from_file_path method to create a fingerprint
-      media = LoadedMedia.from_file(content_type, image_path)
-      return nil unless media
-      thumbnail_options =
-        LoadedMedia::ThumbnailOptions.new(
-          width: 128,
-          height: 128,
-          quality: 95,
-          size: :force,
-          interlace: false,
-          for_frames: [0.0, 0.1, 0.5, 0.9, 1.0],
-        )
-      frame_nums =
-        thumbnail_options
-          .for_frames
-          .map do |frame_fraction|
-            (frame_fraction * (media.num_frames - 1)).to_i
-          end
-          .uniq
-          .sort
-      frame_nums.map do |frame_num|
-        tmp_file = File.join(tmp_dir, "frame-#{frame_num}.jpg")
-        media.write_frame_thumbnail(frame_num, tmp_file, thumbnail_options)
-        GenerateFingerprintsResult.new(
-          thumb_path: tmp_file,
-          fingerprint:
-            Domain::PostFile::BitFingerprint.from_file_path(tmp_file),
-          detail_fingerprint:
-            Domain::PostFile::BitFingerprint.detail_from_file_path(tmp_file),
-        )
-      end
-    end
-  end
-end

app/helpers/domain_source_helper.rb

@@ -1,29 +0,0 @@
-# typed: strict
-module DomainSourceHelper
-  extend T::Sig
-
-  sig { returns(T::Hash[String, String]) }
-  def self.source_name_to_class_name
-    {
-      "furaffinity" => "Domain::Post::FaPost",
-      "e621" => "Domain::Post::E621Post",
-      "inkbunny" => "Domain::Post::InkbunnyPost",
-      "bluesky" => "Domain::Post::BlueskyPost",
-    }
-  end
-
-  sig { returns(T::Array[String]) }
-  def self.all_source_names
-    source_name_to_class_name.keys
-  end
-
-  sig { params(list: T::Array[String]).returns(T::Array[String]) }
-  def self.source_names_to_class_names(list)
-    list.map { |source| source_name_to_class_name[source] }.compact
-  end
-
-  sig { params(list: T::Array[String]).returns(T::Boolean) }
-  def self.has_all_sources?(list)
-    list.sort == all_source_names.sort
-  end
-end

app/helpers/fa_uri_helper.rb

@@ -1,59 +0,0 @@
-# typed: strict
-
-module FaUriHelper
-  extend T::Sig
-
-  FA_CDN_HOSTS = %w[d.facdn.net d.furaffinity.net].freeze
-
-  class FaMediaUrlInfo < T::ImmutableStruct
-    extend T::Sig
-    include T::Struct::ActsAsComparable
-
-    const :url_name, String
-    const :original_file_posted, Integer
-    const :latest_file_posted, Integer
-    const :filename, String
-    const :filename_with_ts, String
-
-    sig { returns(Time) }
-    def original_file_posted_at
-      Time.at(original_file_posted)
-    end
-
-    sig { returns(Time) }
-    def latest_file_posted_at
-      Time.at(latest_file_posted)
-    end
-  end
-
-  sig { params(url_str: String).returns(T.nilable(FaMediaUrlInfo)) }
-  def self.parse_fa_media_url(url_str)
-    uri = Addressable::URI.parse(url_str)
-    return nil unless is_fa_cdn_host?(uri.host)
-    # paths are in the form of `art/<user.url_name>/<latest_file_ts>/<og_file_ts>.<rest_of_filename>`
-    # latest_file_ts is the timestamp of the most up to date file that has been uploaded for the post
-    # og_file_ts is the timestamp of when the post was originally made
-    path = uri.path
-    match =
-      path.match(
-        %r{/art/(?<url_name>[^/]+)/(stories/)?(?<latest_ts>\d+)/(?<original_ts>\d+)\.(?<filename>.*)},
-      )
-    return nil unless match
-    url_name = match[:url_name]
-    latest_ts = match[:latest_ts].to_i
-    original_ts = match[:original_ts].to_i
-    filename = match[:filename]
-    FaMediaUrlInfo.new(
-      url_name:,
-      original_file_posted: original_ts,
-      latest_file_posted: latest_ts,
-      filename:,
-      filename_with_ts: path.split("/").last,
-    )
-  end
-
-  sig { params(host: String).returns(T::Boolean) }
-  def self.is_fa_cdn_host?(host)
-    FA_CDN_HOSTS.include?(host)
-  end
-end

app/helpers/good_job_helper.rb

@@ -4,14 +4,11 @@
 module GoodJobHelper
   extend T::Sig
   extend T::Helpers
-  abstract!
+  extend self
 
   class AnsiSegment < T::Struct
-    include T::Struct::ActsAsComparable
-
-    prop :text, String
-    prop :class_names, T::Array[String], default: []
-    prop :url, T.nilable(String), default: nil
+    const :text, String
+    const :class_names, T::Array[String]
   end
 
   # ANSI escape code pattern
@@ -20,7 +17,7 @@ module GoodJobHelper
 
   sig { params(text: String).returns(T::Array[AnsiSegment]) }
   def parse_ansi(text)
-    segments = T.let([], T::Array[AnsiSegment])
+    segments = []
     current_classes = T::Array[String].new
 
     # Split the text into parts based on ANSI codes
@@ -48,34 +45,23 @@ module GoodJobHelper
       end
     end
 
-    segments.each_with_index do |s0, idx|
-      s1 = segments[idx + 1] || next
-      if s0.text == "[hle " && s1.text.match(/\d+/)
-        segments[idx + 1] = AnsiSegment.new(
-          text: s1.text,
-          class_names: s1.class_names,
-          url: Rails.application.routes.url_helpers.log_entry_path(s1.text),
-        )
-      end
-    end
-
     # go through segments and detect UUIDs, splitting the segment at the uuid
     # and adding them to the segments array. Should result in a <before>, <uuid>,
     # <after> tuple.
     segments.flat_map do |segment|
-      if (idx = segment.text.index(UUID_REGEX))
+      if segment.text.match?(UUID_REGEX)
+        idx = segment.text.index(UUID_REGEX)
         [
           AnsiSegment.new(
-            text: segment.text[0...idx] || "",
+            text: segment.text[0...idx],
             class_names: segment.class_names,
           ),
           AnsiSegment.new(
-            text: segment.text[idx...idx + 36] || "",
+            text: segment.text[idx...idx + 36],
             class_names: ["log-uuid"],
-            url: "/jobs/jobs/#{segment.text[idx...idx + 36]}",
           ),
           AnsiSegment.new(
-            text: segment.text[idx + 36..] || "",
+            text: segment.text[idx + 36..],
             class_names: segment.class_names,
           ),
         ]
@@ -85,94 +71,15 @@ module GoodJobHelper
     end
   end
 
-  class JobArg < T::Struct
-    const :key, Symbol
-    const :value, T.untyped
-    const :inferred, T::Boolean
-  end
-
-  sig { params(job: GoodJob::Job).returns(T::Array[JobArg]) }
+  sig { params(job: GoodJob::Job).returns(T::Hash[String, T.untyped]) }
   def arguments_for_job(job)
-    begin
-      deserialized =
-        T.cast(
-          ActiveJob::Arguments.deserialize(job.serialized_params).to_h,
-          T::Hash[String, T.untyped],
-        )
-    rescue ActiveJob::DeserializationError => e
-      Rails.logger.error(
-        "error deserializing job arguments: #{e.class.name} - #{e.message}",
+    deserialized =
+      T.cast(
+        ActiveJob::Arguments.deserialize(job.serialized_params).to_h,
+        T::Hash[String, T.untyped],
       )
-      return [JobArg.new(key: :error, value: e.message, inferred: true)]
-    end
-    args_hash =
-      T.cast(deserialized["arguments"].first, T::Hash[Symbol, T.untyped])
-    args =
-      args_hash.map { |key, value| JobArg.new(key:, value:, inferred: false) }
-
-    if args_hash[:fa_id] && !args_hash[:post] &&
-         args << JobArg.new(
-           key: :post,
-           value:
-             Domain::Post::FaPost.find_by(fa_id: args_hash[:fa_id]) ||
-               "post not found",
-           inferred: true,
-         )
-    end
-
-    if args_hash[:url_name] && !args_hash[:user]
-      args << JobArg.new(
-        key: :user,
-        value:
-          Domain::User::FaUser.find_by(url_name: args_hash[:url_name]) ||
-            "user not found",
-        inferred: true,
-      )
-    end
-
-    if job_id = args_hash[:caused_by_job_id]
-      job = GoodJob::Job.find_by(id: job_id)
-      if job
-        args << JobArg.new(key: :caused_by_job, value: job, inferred: false)
-        args.delete_if { |arg| arg.key == :caused_by_job_id }
-      end
-    end
-
-    args.sort_by(&:key)
-  end
-
-  sig { params(state: String).returns(String) }
-  def post_file_state_badge_class(state)
-    case state
-    when "ok"
-      "bg-success"
-    when "terminal_error"
-      "bg-danger"
-    when "retryable_error"
-      "bg-warning text-dark"
-    when "pending"
-      "bg-info"
-    else
-      "bg-secondary"
-    end
-  end
-
-  sig { params(state: String).returns(String) }
-  def post_file_state_icon_class(state)
-    base =
-      case state
-      when "ok"
-        "fa-check"
-      when "terminal_error"
-        "fa-circle-xmark"
-      when "retryable_error"
-        "fa-triangle-exclamation"
-      when "pending"
-        "fa-clock"
-      else
-        "fa-question"
-      end
-    "fa-solid #{base}"
+    args = deserialized["arguments"].first
+    args.sort_by { |key, _| key.to_s }.to_h
   end
 
   private

app/helpers/indexable_posts_helper.rb

@@ -0,0 +1,22 @@
+# typed: false
+module IndexablePostsHelper
+  def show_path(indexed_post)
+    case indexed_post.postable_type
+    when "Domain::Fa::Post"
+      # need to use the helper here because the postable is not loaded
+      Rails.application.routes.url_helpers.domain_fa_post_path(
+        indexed_post.postable,
+      )
+    when "Domain::E621::Post"
+      Rails.application.routes.url_helpers.domain_e621_post_path(
+        indexed_post.postable,
+      )
+    when "Domain::Inkbunny::Post"
+      Rails.application.routes.url_helpers.domain_inkbunny_post_path(
+        indexed_post.postable,
+      )
+    else
+      raise("Unsupported postable type: #{indexed_post.postable_type}")
+    end
+  end
+end

app/helpers/ip_address_helper.rb

@@ -1,35 +0,0 @@
-# typed: strict
-module IpAddressHelper
-  extend T::Sig
-
-  # Formats an IPAddr object to display properly with CIDR notation if it's a subnet
-  # @param ip_addr [IPAddr, nil] The IP address object to format or nil
-  # @return [String] A formatted string representation of the IP address
-  sig { params(ip_addr: T.nilable(IPAddr)).returns(String) }
-  def format_ip_address(ip_addr)
-    if ip_addr.nil?
-      ""
-    else
-      # For IPv4, check if the prefix is not 32 (full mask)
-      # For IPv6, check if the prefix is not 128 (full mask)
-      if (ip_addr.ipv4? && ip_addr.prefix < 32) ||
-           (ip_addr.ipv6? && ip_addr.prefix < 128)
-        # This is a CIDR range
-        "#{ip_addr.to_s}/#{ip_addr.prefix}"
-      else
-        # Single IP address
-        ip_addr.to_s
-      end
-    end
-  end
-
-  # Determines if the provided IP address is a CIDR range
-  # @param ip_addr [IPAddr, nil] The IP address to check or nil
-  # @return [Boolean] true if the address is a CIDR range, false otherwise
-  sig { params(ip_addr: T.nilable(IPAddr)).returns(T::Boolean) }
-  def cidr_range?(ip_addr)
-    return false if ip_addr.nil?
-
-    format_ip_address(ip_addr).include?("/")
-  end
-end

app/helpers/log_entries_helper.rb

@@ -1,387 +1,54 @@
-# typed: strict
+# typed: true
 module LogEntriesHelper
-  extend T::Sig
-  extend T::Helpers
-  include HelpersInterface
-  abstract!
-
-  sig { params(content_type: String).returns(T::Boolean) }
   def is_send_data_content_type?(content_type)
     is_renderable_image_type?(content_type) ||
       is_renderable_video_type?(content_type) ||
-      is_renderable_audio_type?(content_type) ||
       is_flash_content_type?(content_type)
   end
 
-  sig { params(uri_path: String).returns(T::Array[[String, String]]) }
   def path_iterative_parts(uri_path)
     path_parts = uri_path.split("/")
-
     (1...path_parts.length).map do |i|
       [
-        T.must(path_parts[i]),
-        T.must(path_parts[0..i]).join("/") +
-          (i == path_parts.length - 1 ? "" : "/"),
+        path_parts[i],
+        path_parts[0..i].join("/") + (i == path_parts.length - 1 ? "" : "/"),
       ]
     end
   end
 
-  sig { params(content_type: String).returns(T.nilable(String)) }
-  def thumbnail_extension_for_content_type(content_type)
-    return nil unless is_thumbable_content_type?(content_type)
-    extension = extension_for_content_type(content_type)
-    if extension == "gif"
-      "gif"
-    else
-      "jpeg"
-    end
-  end
-
-  sig { params(content_type: String).returns(T.nilable(String)) }
-  def extension_for_content_type(content_type)
-    content_type = content_type.split(";")[0]
-    return nil unless content_type
-
-    extension = Rack::Mime::MIME_TYPES.invert[content_type]
-    return extension[1..] if extension
-
+  def ext_for_content_type(content_type)
     case content_type
-    when %r{image/jpeg}
+    when "image/jpeg"
       "jpeg"
-    when %r{image/jpg}
+    when "image/jpg"
       "jpg"
-    when %r{image/png}
+    when "image/png"
       "png"
-    when %r{image/gif}
+    when "image/gif"
       "gif"
-    when %r{video/webm}
+    when "video/webm"
       "webm"
-    when %r{audio/mpeg}
-      "mp3"
-    when %r{audio/mp3}
-      "mp3"
-    when %r{audio/wav}
-      "wav"
-    when %r{application/pdf}
-      "pdf"
-    when %r{application/rtf}
-      "rtf"
-    when %r{application/msword}
-      "doc"
-    when %r{application/vnd\.openxmlformats-officedocument\.wordprocessingml\.document}
-      "docx"
-    when %r{application/vnd\.oasis\.opendocument\.text}
-      "odt"
     else
       nil
     end
   end
 
-  sig { params(content_type: String).returns(T::Boolean) }
   def is_renderable_image_type?(content_type)
     %w[image/jpeg image/jpg image/png image/gif].any? do |ct|
       content_type.starts_with?(ct)
     end
   end
 
-  sig { params(content_type: String).returns(T::Boolean) }
-  def is_json_content_type?(content_type)
-    content_type.starts_with?("application/json")
+  def is_thumbable_content_type?(content_type)
+    %w[video/webm].any? { |ct| content_type.starts_with?(ct) } ||
+      is_renderable_image_type?(content_type)
   end
 
-  sig { params(content_type: String).returns(T::Boolean) }
-  def is_rich_text_content_type?(content_type)
-    %w[
-      application/pdf
-      application/rtf
-      application/msword
-      text/plain
-      application/vnd.oasis.opendocument.text
-      application/vnd.openxmlformats-officedocument.wordprocessingml.document
-    ].any? { |ct| content_type.starts_with?(ct) }
-  end
-
-  sig { params(content_type: String).returns(T::Boolean) }
   def is_renderable_video_type?(content_type)
     %w[video/mp4 video/webm].any? { |ct| content_type.starts_with?(ct) }
   end
 
-  sig { params(content_type: String).returns(T::Boolean) }
-  def is_renderable_audio_type?(content_type)
-    %w[audio/mpeg audio/mp3 audio/wav audio/ogg].any? do |ct|
-      content_type.starts_with?(ct)
-    end
-  end
-
-  sig { params(content_type: String).returns(T::Boolean) }
   def is_flash_content_type?(content_type)
-    content_type.match? %r{application/x-shockwave-flash}
-  end
-
-  sig { params(content_type: String).returns(T::Boolean) }
-  def is_thumbable_content_type?(content_type)
-    is_renderable_video_type?(content_type) ||
-      is_renderable_image_type?(content_type)
-  end
-
-  sig { params(rich_text_body: String).returns(T.nilable(String)) }
-  def convert_with_pdftohtml(rich_text_body)
-    stdin, stdout, stderr, wait_thr =
-      Open3.popen3(
-        "pdftohtml",
-        "-i", # ignore images
-        "-s", # generate single HTML page
-        "-nodrm", # ignore drm
-        "-enc",
-        "UTF-8",
-        "-stdout",
-        "-", # read from stdin
-        "-", # write to stdout (???)
-      )
-    stdin.binmode
-    stdin.write(rich_text_body)
-    stdin.close
-
-    stdout_str = stdout.read
-    exit_status = T.cast(wait_thr.value, Process::Status)
-    return nil unless exit_status.success?
-
-    # For PDFs, handle both HTML entities and Unicode NBSPs
-    # First replace the actual unicode NBSP character (U+00A0)
-    # stdout_str.gsub!(/[[:space:]]+/, " ")
-    stdout_str.gsub!(/\u00A0/, " ")
-    stdout_str.gsub!(/ /i, " ")
-    stdout_str.gsub!(/ /, " ")
-    stdout_str.gsub!(/ /i, " ")
-
-    stdout_str
-  ensure
-    stdin&.close
-    stdout&.close
-    stderr&.close
-  end
-
-  sig { params(rich_text_body: String).returns(T.nilable(String)) }
-  def convert_with_abiword(rich_text_body)
-    stdin, stdout, stderr, wait_thr =
-      Open3.popen3(
-        "abiword",
-        "--display=0",
-        "--to=html",
-        "--to-name=fd://1",
-        "fd://0",
-      )
-    stdin.binmode
-    stdin.write(rich_text_body)
-    stdin.close
-
-    stdout_str = stdout.read
-    exit_status = T.cast(wait_thr.value, Process::Status)
-    return nil unless exit_status.success?
-
-    stdout_str.gsub!(/Abiword HTML Document/, "")
-    stdout_str = try_convert_bbcode_to_html(stdout_str)
-    stdout_str.gsub!(%r{<br\s*/>}, "")
-
-    stdout_str
-  ensure
-    stdin&.close
-    stdout&.close
-    stderr&.close
-  end
-
-  sig { params(rich_text_body: String).returns(T.nilable(String)) }
-  def convert_with_libreoffice(rich_text_body)
-    tempfile = Tempfile.new(%w[test .doc], binmode: true)
-    tempfile.write(rich_text_body)
-    tempfile.flush
-
-    stdin, stdout, stderr, wait_thr =
-      Open3.popen3(
-        "libreoffice",
-        "--display",
-        "0",
-        "--headless",
-        "--convert-to",
-        "html",
-        T.must(tempfile.path),
-        "--cat",
-      )
-    stdin.binmode
-    stdin.write(rich_text_body)
-    stdin.close
-
-    stdout_str = stdout.read
-    exit_status = T.cast(wait_thr.value, Process::Status)
-    return nil unless exit_status.success?
-
-    stdout_str
-  ensure
-    stdin&.close
-    stdout&.close
-    stderr&.close
-    tempfile&.close
-  end
-
-  sig { params(str: String).returns(String) }
-  def reencode_as_utf8_lossy(str)
-    str.encode("UTF-8", invalid: :replace, undef: :replace, replace: "")
-  rescue StandardError
-    str
-  end
-
-  sig { params(rich_text_body: String).returns(String) }
-  def try_convert_bbcode_to_html(rich_text_body)
-    rich_text_body.bbcode_to_html(false)
-  rescue StandardError
-    rich_text_body
-  end
-
-  sig { params(rich_text_body: String).returns(String) }
-  def try_detect_encoding(rich_text_body)
-    encoding = CharlockHolmes::EncodingDetector.detect(rich_text_body)
-    encoding ? encoding[:encoding] : "UTF-8"
-  end
-
-  sig { params(log_entry: HttpLogEntry).returns(T.nilable(String)) }
-  def render_rich_text_content(log_entry)
-    content_type = log_entry.content_type
-    rich_text_body = log_entry.response_bytes
-    return nil if rich_text_body.blank? || content_type.blank?
-
-    is_plain_text = content_type.starts_with?("text/plain")
-
-    if is_plain_text
-      encoding_name = try_detect_encoding(rich_text_body)
-      rich_text_body = rich_text_body.force_encoding(encoding_name)
-      rich_text_body = reencode_as_utf8_lossy(rich_text_body)
-      document_html = try_convert_bbcode_to_html(rich_text_body)
-    elsif content_type.starts_with?("application/pdf")
-      document_html = convert_with_pdftohtml(rich_text_body)
-    else
-      document_html =
-        convert_with_abiword(rich_text_body) ||
-          convert_with_libreoffice(rich_text_body)
-    end
-
-    return nil if document_html.blank?
-    sanitize_rich_text_document_html(document_html, is_plain_text)
-  end
-
-  sig do
-    params(document_html: String, is_plain_text: T::Boolean).returns(String)
-  end
-  def sanitize_rich_text_document_html(document_html, is_plain_text)
-    quote_transformer =
-      Kernel.lambda do |env|
-        node = env[:node]
-        if node["class"]
-          classes = node["class"].split(" ").map(&:strip).compact
-          node.remove_attribute("class")
-          if classes.include?("quote")
-            # write div to be a blockquote
-            node.name = "blockquote"
-          end
-        end
-        node
-      end
-
-    clean_plain_text_node =
-      Kernel.lambda do |node|
-        node = T.cast(node, Nokogiri::XML::Node)
-        if node.text?
-          node_text = node.text.strip
-          if node_text.empty?
-            node.unlink
-          else
-            # collect all the subsequent nodes that are not a block element
-            # and replace the current node with a <p> containing the text
-            # and the collected nodes
-            current_node = node
-            inline_elements = []
-            while (next_sibling = current_node.next_sibling) &&
-                    (next_sibling.name != "br") && (next_sibling.name != "p")
-              inline_elements << next_sibling
-              current_node = next_sibling
-            end
-
-            node_html = [node_text]
-            inline_elements.each do |inline_element|
-              inline_element.unlink
-              node_html << inline_element.to_html
-            end
-            node.replace("<p>#{node_html.join(" ")}</p>")
-          end
-        end
-      end
-
-    plain_text_transformer =
-      Kernel.lambda do |env|
-        # within a div, wrap bare text nodes in a <p>
-        node = T.cast(env[:node], Nokogiri::XML::Node)
-        node_name = T.cast(env[:node_name], String)
-        if node_name == "div"
-          current_child = T.unsafe(node.children.first)
-          while current_child.present?
-            clean_plain_text_node.call(current_child)
-            current_child = current_child.next_sibling
-          end
-        elsif node.text? && node.parent&.name == "#document-fragment"
-          clean_plain_text_node.call(node)
-        end
-        { node_allowlist: [node] }
-      end
-
-    # remove_empty_newline_transformer =
-    #   Kernel.lambda do |env|
-    #     node = env[:node]
-    #     node.unlink if node.text? && node.text.strip.chomp.blank?
-    #   end
-
-    # remove_multiple_br_transformer =
-    #   Kernel.lambda do |env|
-    #     node = env[:node]
-    #     if node.name == "br"
-    #       node.unlink if node.previous_sibling&.name == "br"
-    #     end
-    #   end
-
-    sanitizer =
-      Sanitize.new(
-        elements: %w[span div p i b strong em blockquote br],
-        attributes: {
-          all: %w[style class],
-        },
-        css: {
-          properties: %w[color text-align margin-bottom],
-        },
-        transformers: [
-          quote_transformer,
-          # is_plain_text ? remove_empty_newline_transformer : nil,
-          is_plain_text ? plain_text_transformer : nil,
-          # is_plain_text ? remove_multiple_br_transformer : nil,
-        ].compact,
-      )
-
-    fragment = sanitizer.fragment(document_html).strip
-    if is_plain_text
-      fragment.gsub!("<br>", "")
-      fragment.gsub!("<br />", "")
-    end
-    raw fragment
-  end
-
-  sig { params(performed_by: String).returns(String) }
-  def performed_by_to_short_code(performed_by)
-    case performed_by
-    when "direct"
-      "DR"
-    when "airvpn-1-netherlands"
-      "NL"
-    when "airvpn-2-san-jose"
-      "SJ"
-    else
-      "??"
-    end
+    content_type =~ %r{application/x-shockwave-flash}
   end
 end

app/helpers/paths_helper.rb

@@ -1,18 +0,0 @@
-# typed: strict
-# frozen_string_literal: true
-module PathsHelper
-  extend T::Sig
-  extend T::Helpers
-  include HelpersInterface
-  abstract!
-
-  private
-
-  sig do
-    params(path: String, params: T::Hash[Symbol, T.untyped]).returns(String)
-  end
-  def to_path(path, params = {})
-    params_string = params.reject { |k, v| v.blank? }.to_query
-    "#{path}#{"?#{params_string}" if params_string.present?}"
-  end
-end