fix for routes

This commit introduces comprehensive support for post groups across different domains:
- Created PostGroupsController to handle viewing post groups
- Added new views for displaying post groups and their associated posts
- Implemented policies for post groups
- Enhanced models to support post group functionality
- Updated routes to support post group navigation
- Added helper methods for post group interactions
- Improved GoodJob argument rendering for post groups

The changes provide a unified way to view and interact with post collections across different domains like Inkbunny and E621.

.cursorignore

@@ -5,28 +5,3 @@ tmp
 log
 public
 .bundle
-gems
-
-# Generated/build artifacts
-node_modules
-user_scripts/dist
-app/assets/builds
-vendor/javascript
-
-# Sorbet generated files
-sorbet/tapioca
-sorbet/rbi/gems
-sorbet/rbi/annotations
-sorbet/rbi/dsl
-
-# Configuration files with secrets
-config/credentials.yml.enc
-config/master.key
-
-# Lock files
-yarn.lock
-Gemfile.lock
-
-# Documentation
-TODO.md
-*.notes.md

.cursorrules

@@ -1,278 +0,0 @@
-# How to use this codebase
-
-- Run `bin/tapioca dsl` after changing a model or concern.
-- Run `bin/tapioca gems` after changing the Gemfile.
-- Run `srb tc` after making changes to Ruby files to ensure the codebase is typechecked.
-- Run `bin/rspec <path_to_spec_file>` after a spec file is modified.
-- Run `tapioca dsl` if models or concerns are modified.
-- Run `bin/rspec <path_to_spec_file>` to run tests for a single file.
-- There are no view-specific tests, so if a view changes then run the controller tests instead.
-- For instance, if you modify `app/models/domain/post.rb`, run `bin/rspec spec/models/domain/post_spec.rb`. If you modify `app/views/domain/users/index.html.erb`, run `bin/rspec spec/controllers/domain/users_controller_spec.rb`.
-- At the end of a long series of changes, run `just test`.
-- If specs are failing, then fix the failures, and rerun with `bin/rspec <path_to_spec_file>`.
-- If you need to add logging to a Job to debug it, set `quiet: false` on the spec you are debugging.
-- Fish shell is used for development, not bash.
-- When running scratch commands, use `bin/rails runner`, not `bin/rails console`.
-
-# Typescript Development
-
-- React is the only frontend framework used.
-- Styling is done with Tailwind CSS and FontAwesome.
-- Put new typescript files in `app/javascript/bundles/Main/components/`
-
-# HTTP Mocking in Job Specs
-
-When writing specs for jobs that make HTTP requests, use `HttpClientMockHelpers.init_with()` instead of manually creating doubles:
-
-```ruby
-# CORRECT: Use HttpClientMockHelpers.init_with
-let(:client_mock_config) do
-  [
-    {
-      uri: "https://example.com/api/first-endpoint",
-      status_code: 200,
-      content_type: "application/json",
-      contents: first_response_body,
-    },
-    {
-      uri: "https://example.com/api/second-endpoint",
-      status_code: 200,
-      content_type: "application/json",
-      contents: second_response_body,
-      caused_by_entry: :any, # Use this for chained requests
-    },
-  ]
-end
-
-before { @log_entries = HttpClientMockHelpers.init_with(client_mock_config) }
-
-# WRONG: Don't create doubles manually
-expect(http_client_mock).to receive(:get).and_return(
-  double(status_code: 200, body: response_body, log_entry: double),
-)
-
-# WRONG: Don't use the old init_http_client_mock method
-@log_entries =
-  HttpClientMockHelpers.init_http_client_mock(
-    http_client_mock,
-    client_mock_config,
-  )
-```
-
-This pattern:
-
-- Uses the preferred `init_with` helper method
-- Automatically uses the global `http_client_mock` from `spec_helper.rb`
-- Creates real HttpLogEntry objects that can be serialized by ActiveJob
-- Follows the established codebase pattern
-- Avoids "Unsupported argument type: RSpec::Mocks::Double" errors
-- Use `caused_by_entry: :any` for HTTP requests that are chained (where one request's log entry becomes the `caused_by_entry` for the next request)
-- No need to manually set up `http_client_mock` - it's handled globally in `spec_helper.rb`
-
-# Job Enqueuing Verification in Specs
-
-Use `SpecUtil.enqueued_job_args()` instead of mocking `perform_later`:
-
-```ruby
-# CORRECT: Test actual job enqueuing
-enqueued_jobs = SpecUtil.enqueued_job_args(SomeJob)
-expect(enqueued_jobs).to contain_exactly(hash_including(user: user))
-expect(enqueued_jobs).to be_empty # For no jobs
-
-# WRONG: Don't mock perform_later (breaks with .set chaining)
-expect(SomeJob).to receive(:perform_later)
-```
-
-Benefits: More robust, tests actual behavior, no cleanup needed (tests run in transactions).
-
-# Testing Jobs
-
-When writing specs for jobs e.g. Domain::Site::SomethingJob, do not invoke `job.perform(...)` directly, always use `perform_now(...)` (defined in spec/helpers/perform_job_helpers.rb)
-
-# === BACKLOG.MD GUIDELINES START ===
-
-# Instructions for the usage of Backlog.md CLI Tool
-
-## 1. Source of Truth
-
-- Tasks live under **`backlog/tasks/`** (drafts under **`backlog/drafts/`**).
-- Every implementation decision starts with reading the corresponding Markdown task file.
-- Project documentation is in **`backlog/docs/`**.
-- Project decisions are in **`backlog/decisions/`**.
-
-## 2. Defining Tasks
-
-### **Title**
-
-Use a clear brief title that summarizes the task.
-
-### **Description**: (The **"why"**)
-
-Provide a concise summary of the task purpose and its goal. Do not add implementation details here. It
-should explain the purpose and context of the task. Code snippets should be avoided.
-
-### **Acceptance Criteria**: (The **"what"**)
-
-List specific, measurable outcomes that define what means to reach the goal from the description. Use checkboxes (`- [ ]`) for tracking.
-When defining `## Acceptance Criteria` for a task, focus on **outcomes, behaviors, and verifiable requirements** rather
-than step-by-step implementation details.
-Acceptance Criteria (AC) define _what_ conditions must be met for the task to be considered complete.
-They should be testable and confirm that the core purpose of the task is achieved.
-**Key Principles for Good ACs:**
-
-- **Outcome-Oriented:** Focus on the result, not the method.
-- **Testable/Verifiable:** Each criterion should be something that can be objectively tested or verified.
-- **Clear and Concise:** Unambiguous language.
-- **Complete:** Collectively, ACs should cover the scope of the task.
-- **User-Focused (where applicable):** Frame ACs from the perspective of the end-user or the system's external behavior.
-  - _Good Example:_ "- [ ] User can successfully log in with valid credentials."
-  - _Good Example:_ "- [ ] System processes 1000 requests per second without errors."
-  - _Bad Example (Implementation Step):_ "- [ ] Add a new function `handleLogin()` in `auth.ts`."
-
-### Task file
-
-Once a task is created it will be stored in `backlog/tasks/` directory as a Markdown file with the format
-`task-<id> - <title>.md` (e.g. `task-42 - Add GraphQL resolver.md`).
-
-### Additional task requirements
-
-- Tasks must be **atomic** and **testable**. If a task is too large, break it down into smaller subtasks.
-  Each task should represent a single unit of work that can be completed in a single PR.
-
-- **Never** reference tasks that are to be done in the future or that are not yet created. You can only reference
-  previous
-  tasks (id < current task id).
-
-- When creating multiple tasks, ensure they are **independent** and they do not depend on future tasks.
-  Example of wrong tasks splitting: task 1: "Add API endpoint for user data", task 2: "Define the user model and DB
-  schema".
-  Example of correct tasks splitting: task 1: "Add system for handling API requests", task 2: "Add user model and DB
-  schema", task 3: "Add API endpoint for user data".
-
-## 3. Recommended Task Anatomy
-
-```markdown
-# task‑42 - Add GraphQL resolver
-
-## Description (the why)
-
-Short, imperative explanation of the goal of the task and why it is needed.
-
-## Acceptance Criteria (the what)
-
-- [ ] Resolver returns correct data for happy path
-- [ ] Error response matches REST
-- [ ] P95 latency ≤ 50 ms under 100 RPS
-
-## Implementation Plan (the how)
-
-1. Research existing GraphQL resolver patterns
-2. Implement basic resolver with error handling
-3. Add performance monitoring
-4. Write unit and integration tests
-5. Benchmark performance under load
-
-## Implementation Notes (only added after working on the task)
-
-- Approach taken
-- Features implemented or modified
-- Technical decisions and trade-offs
-- Modified or added files
-```
-
-## 6. Implementing Tasks
-
-Mandatory sections for every task:
-
-- **Implementation Plan**: (The **"how"**) Outline the steps to achieve the task. Because the implementation details may
-  change after the task is created, **the implementation notes must be added only after putting the task in progress**
-  and before starting working on the task.
-- **Implementation Notes**: Document your approach, decisions, challenges, and any deviations from the plan. This
-  section is added after you are done working on the task. It should summarize what you did and why you did it. Keep it
-  concise but informative.
-
-**IMPORTANT**: Do not implement anything else that deviates from the **Acceptance Criteria**. If you need to
-implement something that is not in the AC, update the AC first and then implement it or create a new task for it.
-
-## 2. Typical Workflow
-
-```bash
-# 1 Identify work
-backlog task list -s "To Do" --plain
-
-# 2 Read details & documentation
-backlog task 42 --plain
-# Read also all documentation files in `backlog/docs/` directory.
-# Read also all decision files in `backlog/decisions/` directory.
-
-# 3 Start work: assign yourself & move column
-backlog task edit 42 -a @{yourself} -s "In Progress"
-
-# 4 Add implementation plan before starting
-backlog task edit 42 --plan "1. Analyze current implementation\n2. Identify bottlenecks\n3. Refactor in phases"
-
-# 5 Break work down if needed by creating subtasks or additional tasks
-backlog task create "Refactor DB layer" -p 42 -a @{yourself} -d "Description" --ac "Tests pass,Performance improved"
-
-# 6 Complete and mark Done
-backlog task edit 42 -s Done --notes "Implemented GraphQL resolver with error handling and performance monitoring"
-```
-
-### 7. Final Steps Before Marking a Task as Done
-
-Always ensure you have:
-
-1. ✅ Marked all acceptance criteria as completed (change `- [ ]` to `- [x]`)
-2. ✅ Added an `## Implementation Notes` section documenting your approach
-3. ✅ Run all tests and linting checks
-4. ✅ Updated relevant documentation
-
-## 8. Definition of Done (DoD)
-
-A task is **Done** only when **ALL** of the following are complete:
-
-1. **Acceptance criteria** checklist in the task file is fully checked (all `- [ ]` changed to `- [x]`).
-2. **Implementation plan** was followed or deviations were documented in Implementation Notes.
-3. **Automated tests** (unit + integration) cover new logic.
-4. **Static analysis**: linter & formatter succeed.
-5. **Documentation**:
-   - All relevant docs updated (any relevant README file, backlog/docs, backlog/decisions, etc.).
-   - Task file **MUST** have an `## Implementation Notes` section added summarising:
-     - Approach taken
-     - Features implemented or modified
-     - Technical decisions and trade-offs
-     - Modified or added files
-6. **Review**: self review code.
-7. **Task hygiene**: status set to **Done** via CLI (`backlog task edit <id> -s Done`).
-8. **No regressions**: performance, security and licence checks green.
-
-⚠️ **IMPORTANT**: Never mark a task as Done without completing ALL items above.
-
-## 9. Handy CLI Commands
-
-| Purpose          | Command                                                                |
-| ---------------- | ---------------------------------------------------------------------- |
-| Create task      | `backlog task create "Add OAuth"`                                      |
-| Create with desc | `backlog task create "Feature" -d "Enables users to use this feature"` |
-| Create with AC   | `backlog task create "Feature" --ac "Must work,Must be tested"`        |
-| Create with deps | `backlog task create "Feature" --dep task-1,task-2`                    |
-| Create sub task  | `backlog task create -p 14 "Add Google auth"`                          |
-| List tasks       | `backlog task list --plain`                                            |
-| View detail      | `backlog task 7 --plain`                                               |
-| Edit             | `backlog task edit 7 -a @{yourself} -l auth,backend`                   |
-| Add plan         | `backlog task edit 7 --plan "Implementation approach"`                 |
-| Add AC           | `backlog task edit 7 --ac "New criterion,Another one"`                 |
-| Add deps         | `backlog task edit 7 --dep task-1,task-2`                              |
-| Add notes        | `backlog task edit 7 --notes "We added this and that feature because"` |
-| Mark as done     | `backlog task edit 7 -s "Done"`                                        |
-| Archive          | `backlog task archive 7`                                               |
-| Draft flow       | `backlog draft create "Spike GraphQL"` → `backlog draft promote 3.1`   |
-| Demote to draft  | `backlog task demote <task-id>`                                        |
-
-## 10. Tips for AI Agents
-
-- **Always use `--plain` flag** when listing or viewing tasks for AI-friendly text output instead of using Backlog.md
-  interactive UI.
-- When users mention to create a task, they mean to create a task using Backlog.md CLI tool.
-
-# === BACKLOG.MD GUIDELINES END ===

.devcontainer/Dockerfile.devcontainer

@@ -1,22 +1,38 @@
-# Primary image
-FROM mcr.microsoft.com/devcontainers/base:debian-12
-
-# apt caching & install packages
-RUN rm -f /etc/apt/apt.conf.d/docker-clean && \
+FROM ruby:3.2.0 AS native-gems
+RUN rm -f /etc/apt/apt.conf.d/docker-clean; \
     echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/keep-cache
-
-RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
+RUN \
+    --mount=type=cache,target=/var/cache/apt,sharing=locked \
     --mount=type=cache,target=/var/lib/apt,sharing=locked \
     apt-get update && \
-    apt-get install --no-install-recommends --no-install-suggests -qqy \
-    abiword \
+    apt-get install --no-install-recommends --no-install-suggests -y \
+    cmake
+
+WORKDIR /usr/src/app
+RUN gem install bundler -v '2.4.5'
+COPY gems gems
+WORKDIR /usr/src/app/gems/xdiff-rb
+RUN bundle install
+RUN rake compile
+WORKDIR /usr/src/app/gems/rb-bsdiff
+RUN bundle install
+RUN rake compile
+
+# Primary image
+FROM mcr.microsoft.com/devcontainers/ruby:1-3.2-bookworm
+
+# apt caching & install packages
+RUN rm -f /etc/apt/apt.conf.d/docker-clean; \
+    echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/keep-cache
+RUN \
+    --mount=type=cache,target=/var/cache/apt,sharing=locked \
+    --mount=type=cache,target=/var/lib/apt,sharing=locked \
+    apt-get update && \
+    apt-get install --no-install-recommends --no-install-suggests -y \
     autoconf \
     build-essential \
     ca-certificates \
     curl \
-    ffmpeg \
-    ffmpegthumbnailer \
-    file \
     gnupg \
     iputils-ping \
     libblas-dev \
@@ -25,73 +41,22 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
     libgdbm-dev \
     libgdbm6 \
     libgmp-dev \
-    libicu-dev \
     liblapack-dev \
     libncurses5-dev \
     libpq-dev \
     libreadline6-dev \
-    libreoffice \
-    libsqlite3-dev \
     libssl-dev \
+    libvips42 \
     libyaml-dev \
     patch \
-    pdftohtml \
-    pkg-config \
-    rustc \
     uuid-dev \
+    zlib1g-dev \
     watchman \
-    zlib1g-dev
-
-# Install vips dependencies
-RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
-    --mount=type=cache,target=/var/lib/apt,sharing=locked \
-    apt-get update && \
-    apt-get install --no-install-recommends --no-install-suggests -qqy \
-    automake \
-    gtk-doc-tools \
-    gobject-introspection \
-    libgirepository1.0-dev \
-    libglib2.0-dev \
-    libexpat1-dev \
-    libjpeg-dev \
-    libpng-dev \
-    libtiff5-dev \
-    libwebp-dev \
-    libheif-dev \
-    libexif-dev \
-    liblcms2-dev \
-    libxml2-dev \
-    libfftw3-dev \
-    liborc-0.4-dev \
-    libcgif-dev \
-    libjxl-dev \
-    libopenjp2-7-dev \
-    meson \
-    ninja-build
-
-
-# Install imagemagick from source
-RUN cd /tmp && \
-    wget -qO- https://imagemagick.org/archive/releases/ImageMagick-7.1.2-1.tar.xz | tar -xJ && \
-    cd ImageMagick-7.1.2-1 && \
-    ./configure && \
-    make -j$(nproc) && \
-    make install && \
-    ldconfig && \
-    cd / && \
-    rm -rf /tmp/ImageMagick-7.1.2-1*
-
-# Install vips from source
-RUN cd /tmp && \
-    wget -qO- https://github.com/libvips/libvips/releases/download/v8.17.1/vips-8.17.1.tar.xz | tar -xJ && \
-    cd vips-8.17.1 && \
-    meson setup build --prefix=/usr/local -Dcgif=enabled && \
-    cd build && \
-    ninja && \
-    ninja install && \
-    ldconfig && \
-    cd / && \
-    rm -rf /tmp/vips-8.17.1*
+    ffmpeg \
+    ffmpegthumbnailer \
+    abiword \
+    pdftohtml \
+    libreoffice
 
 # Install postgres 15 client
 RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
@@ -99,9 +64,9 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
     sudo install -d /usr/share/postgresql-common/pgdg && \
     curl -o /usr/share/postgresql-common/pgdg/apt.postgresql.org.asc --fail https://www.postgresql.org/media/keys/ACCC4CF8.asc && \
     sh -c 'echo "deb [signed-by=/usr/share/postgresql-common/pgdg/apt.postgresql.org.asc] https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list' && \
-    sudo apt update && \
-    sudo apt-get install --no-install-recommends --no-install-suggests -qqy \
-    postgresql-client-17
+    apt update && \
+    apt-get install --no-install-recommends --no-install-suggests -y \
+    postgresql-client-15
 
 # Install & configure delta diff tool
 RUN wget -O- https://github.com/dandavison/delta/releases/download/0.18.2/git-delta_0.18.2_amd64.deb > /tmp/git-delta.deb && \
@@ -110,38 +75,29 @@ RUN wget -O- https://github.com/dandavison/delta/releases/download/0.18.2/git-de
 
 RUN git config --system core.pager "delta" && \
     git config --system interactive.diffFilter "delta --color-only" && \
-    git config --system delta.navigate "true" && \
-    git config --system delta.dark "true" && \
-    git config --system delta.side-by-side "true" && \
+    git config --system delta.navigate true && \
+    git config --system delta.dark true && \
+    git config --system delta.side-by-side true && \
     git config --system merge.conflictstyle "zdiff3" && \
-    git config --system core.editor "cursor --wait" && \
-    git config --system diff.algorithm "histogram" && \
-    git config --system diff.colorMoved "default"
+    git config --system core.editor "cursor --wait"
 
-# Install ruby
-USER vscode
-RUN git clone https://github.com/rbenv/rbenv.git ~/.rbenv
-ENV PATH="/home/vscode/.rbenv/bin:/home/vscode/.rbenv/shims:$PATH"
-RUN echo 'eval "$(rbenv init - --no-rehash bash)"' >> ~/.bashrc
-RUN git clone https://github.com/rbenv/ruby-build.git "$(rbenv root)"/plugins/ruby-build
-RUN rbenv install 3.4.4 && \
-    rbenv global 3.4.4
+# Install native gems
+COPY --from=native-gems /usr/src/app/gems/xdiff-rb /gems/xdiff-rb
+COPY --from=native-gems /usr/src/app/gems/rb-bsdiff /gems/rb-bsdiff
 
 ENV RAILS_ENV development
 
-# Pre install gems to speed up container startup
-USER root
-RUN mkdir -p /tmp/bundle-install-cache && \
-    chown -R vscode:vscode /tmp/bundle-install-cache
-WORKDIR /tmp/bundle-install-cache
-USER vscode
-COPY Gemfile.lock Gemfile ./
-COPY gems/has_aux_table ./gems/has_aux_table
-RUN BUNDLE_FROZEN=true MAKE="make -j$(nproc)" bundle install --jobs $(nproc)
+# [Optional] Uncomment this line to install additional gems.
+RUN su vscode -c "gem install bundler -v '2.5.6'" && \
+    su vscode -c "gem install rake -v '13.0.6'" && \
+    su vscode -c "gem install ruby-lsp -v '0.22.1'"
 
 # install exo
-RUN curl -sL https://exo.deref.io/install | bash
+RUN su vscode -c "curl -sL https://exo.deref.io/install | bash"
 ENV PATH "/home/vscode/.exo/bin:$PATH"
 
 # install just (command runner)
-RUN curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | sudo bash -s -- --to /usr/local/bin
+RUN curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | bash -s -- --to /usr/local/bin
+
+RUN su vscode -c "source /usr/local/share/nvm/nvm.sh && nvm install 18 && nvm use 18 && npm install -g yarn" 2>&1
+ENV PATH /usr/local/share/nvm/current/bin:$PATH

.devcontainer/Dockerfile.postgres

@@ -1,7 +1,8 @@
-FROM postgres:17
+FROM postgres:15
 
 RUN apt-get update && apt-get install -y \
-    postgresql-17-pgvector \
+    postgresql-15-pgvector \
     && rm -rf /var/lib/apt/lists/*
 
-RUN echo "CREATE EXTENSION vector;" >> /docker-entrypoint-initdb.d/01-vector.sql
+COPY create-tablespaces.bash /docker-entrypoint-initdb.d/00-create-tablespaces.bash
+RUN echo "CREATE EXTENSION pgvector;" >> /docker-entrypoint-initdb.d/01-pgvector.sql

.devcontainer/create-tablespaces.bash

@@ -0,0 +1,9 @@
+#!/bin/bash -ex
+
+mkdir -p /tablespaces/mirai
+chown postgres:postgres /tablespaces/mirai
+chmod 750 /tablespaces/mirai
+psql -v ON_ERROR_STOP=1 \
+    --username "$POSTGRES_USER" \
+    --dbname "$POSTGRES_DB" \
+    -c "CREATE TABLESPACE mirai LOCATION '/tablespaces/mirai'"

.devcontainer/devcontainer.json

@@ -7,28 +7,7 @@
   "workspaceFolder": "/workspaces/${localWorkspaceFolderBasename}",
   "features": {
     "ghcr.io/meaningful-ooo/devcontainer-features/fish:1": {},
-    "ghcr.io/nikobockerman/devcontainer-features/fish-persistent-data:2": {},
-    "ghcr.io/devcontainers-extra/features/npm-package:1": {
-      "package": "backlog.md"
-    }
-  },
-  "customizations": {
-    "vscode": {
-      "extensions": [
-        "Shopify.ruby-extensions-pack",
-        "dbaeumer.vscode-eslint",
-        "aliariff.vscode-erb-beautify",
-        "bradlc.vscode-tailwindcss",
-        "KoichiSasada.vscode-rdbg",
-        "qwtel.sqlite-viewer",
-        "esbenp.prettier-vscode",
-        "ms-azuretools.vscode-docker",
-        "1YiB.rust-bundle",
-        "rust-lang.rust-analyzer",
-        "saoudrizwan.claude-dev",
-        "ritwickdey.LiveServer"
-      ]
-    }
+    "ghcr.io/nikobockerman/devcontainer-features/fish-persistent-data:2": {}
   },
   // Use 'forwardPorts' to make a list of ports inside the container available locally.
   // This can be used to network with other containers or the host.
@@ -37,8 +16,8 @@
   // "postCreateCommand": "bundle install && rake db:setup",
   "postCreateCommand": ".devcontainer/post-create.sh",
   "forwardPorts": [
-    3000, // rails
-    3001, // thrust
+    3000, // rails development
+    3001, // rails staging
     9394, // prometheus exporter
     "pgadmin:8080", // pgadmin
     "grafana:3100", // grafana

.devcontainer/docker-compose.yml

@@ -23,7 +23,8 @@ services:
       dockerfile: Dockerfile.postgres
     restart: unless-stopped
     volumes:
-      - postgres-17-data:/var/lib/postgresql/data
+      - postgres-data:/var/lib/postgresql/data
+      - postgres-data-tablespaces:/tablespaces
       - ./create-db-user.sql:/docker-entrypoint-initdb.d/create-db-user.sql
     environment:
       POSTGRES_USER: postgres
@@ -64,52 +65,10 @@ services:
     volumes:
       - devcontainer-redux-grafana-data:/var/lib/grafana
 
-  airvpn-netherlands-proxy:
-    image: qmcgaw/gluetun
-    cap_add:
-      - NET_ADMIN
-    devices:
-      - /dev/net/tun:/dev/net/tun
-    environment:
-      - HTTPPROXY=on
-      - SHADOWSOCKS=on
-      - HTTPPROXY_LOG=on
-      - VPN_SERVICE_PROVIDER=airvpn
-      - VPN_TYPE=wireguard
-      - WIREGUARD_PRIVATE_KEY=INLA6x1gUVLRPKcCBgRmfpJBCXhOpyq3SvRd5EvCE08=
-      - WIREGUARD_PRESHARED_KEY=DR6CBW9yG5y+D+qpo8TZCizo5WKOooC/UFBdWk6lGEg=
-      - WIREGUARD_ADDRESSES=10.165.87.232,fd7d:76ee:e68f:a993:4d1b:a77a:b471:a606
-      - SERVER_COUNTRIES=Netherlands
-
-  airvpn-san-jose-proxy:
-    image: qmcgaw/gluetun
-    cap_add:
-      - NET_ADMIN
-    devices:
-      - /dev/net/tun:/dev/net/tun
-    environment:
-      - HTTPPROXY=on
-      - SHADOWSOCKS=on
-      - HTTPPROXY_LOG=on
-      - VPN_SERVICE_PROVIDER=airvpn
-      - VPN_TYPE=wireguard
-      - WIREGUARD_PRIVATE_KEY=INLA6x1gUVLRPKcCBgRmfpJBCXhOpyq3SvRd5EvCE08=
-      - WIREGUARD_PRESHARED_KEY=DR6CBW9yG5y+D+qpo8TZCizo5WKOooC/UFBdWk6lGEg=
-      - WIREGUARD_ADDRESSES=10.165.87.232/32,fd7d:76ee:e68f:a993:4d1b:a77a:b471:a606/128
-      - SERVER_CITIES="San Jose California, Fremont California"
-
-  tor:
-    image: dockurr/tor
-    volumes:
-      - devcontainer-redux-tor-config:/etc/tor
-      - devcontainer-redux-tor-data:/var/lib/tor
-    restart: always
-
 volumes:
-  postgres-17-data:
+  postgres-data:
+  postgres-data-tablespaces:
   devcontainer-redux-gem-cache:
   devcontainer-redux-blob-files:
   devcontainer-redux-grafana-data:
   devcontainer-redux-prometheus-data:
-  devcontainer-redux-tor-config:
-  devcontainer-redux-tor-data:

.devcontainer/fish-shell-conf-d/bash-in-agent.fish

@@ -1,10 +0,0 @@
-# Agent detection - only activate minimal mode for actual agents
-if test -n "$npm_config_yes"; or test -n "$CI"; or not status --is-interactive
-    set -gx AGENT_MODE true
-else
-    set -gx AGENT_MODE false
-end
-
-if test $AGENT_MODE = true
-    # /usr/bin/bash -l
-end

.devcontainer/fish-shell-conf-d/binstubs.fish

@@ -1 +0,0 @@
-set -gx PATH "/workspaces/redux-scraper/bin" $PATH

.devcontainer/fish-shell-conf-d/rbenv.fish

@@ -1 +0,0 @@
-status --is-interactive; and rbenv init - --no-rehash fish | source

.devcontainer/fish-shell-conf-d/rustup.fish

@@ -0,0 +1 @@
+source "$HOME/.cargo/env.fish"

.devcontainer/fish-shell-conf-d/utils.fish

@@ -24,14 +24,4 @@ function blob-files-stats
     set -l files_dir (blob-files-dir || return 1)
     printf "apparent size: %s\n" (du -sh --apparent-size $files_dir)
     printf "actual size:   %s\n" (du -sh $files_dir)
-end
-
-function curl-fa-onion
-    curl \
-    --socks5-hostname tor:9050 \
-    --compressed \
-    -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:128.0) Gecko/20100101 Firefox/128.0" \
-    -H "Accept-Encoding: gzip, deflate" \
-    -H "Connection: keep-alive" \
-    "http://g6jy5jkx466lrqojcngbnksugrcfxsl562bzuikrka5rv7srgguqbjid.onion/$argv[1]"
-end
+end

.dockerignore

@@ -13,77 +13,3 @@ settings.json
 *.export
 .devcontainer
 user_scripts/dist
-backlog
-
-# Test directories (not needed in production)
-spec
-test
-
-# Development and CI/CD files
-.github
-.ruby-lsp
-.aider*
-.cursorignore
-.cursorrules
-.rspec
-.rspec_parallel
-.rubocop.yml
-.prettierrc
-TODO.md
-*.notes.md
-things-to-fix.notes.md
-mf-fitter-commands.notes.md
-.aiderignore
-
-# Sorbet type checking files (not needed in production)
-sorbet
-
-# Storage directory (contains uploaded files/cache)
-storage
-
-# Development database files
-db/*.sqlite3
-db/*.sqlite3-*
-
-# Core dump files
-core
-
-# Yarn/npm cache and lock files that might conflict
-yarn-error.log
-yarn-debug.log*
-.yarn-integrity
-package-lock.json
-
-# OS specific files
-.DS_Store
-Thumbs.db
-
-# Editor specific files
-*.swp
-*.swo
-*~
-
-# Local environment files
-.env.local
-.env.*.local
-
-# Compiled assets (will be rebuilt in Docker)
-public/assets
-public/packs
-public/packs-test
-app/assets/builds/*
-
-# Flame graph files
-flamegraph.svg
-
-# Procfile variants (only need production one)
-Procfile.dev
-Procfile.dev-static
-Procfile.staging
-Procfile.worker
-
-# Development scripts
-justfile
-
-# Documentation
-README.md

.gitignore

@@ -8,6 +8,9 @@ build
 tmp
 core
 *.bundle
+lib/xdiff
+ext/xdiff/Makefile
+ext/xdiff/xdiff
 user_scripts/dist
 migrated_files.txt
 
@@ -15,7 +18,7 @@ migrated_files.txt
 package-lock.json
 
 *.notes.md
-*.txt
+
 # Ignore bundler config.
 /.bundle
 
@@ -59,4 +62,3 @@ yarn-debug.log*
 .yarn-integrity
 .DS_Store
 *.export
-.aider*

.gitmodules

@@ -1,3 +0,0 @@
-[submodule "gems/has_aux_table"]
-	path = gems/has_aux_table
-	url = ssh://git@git.dy.mk:2221/dymk/has_aux_table.git

.prettierrc

@@ -4,10 +4,6 @@
   "trailingComma": "all",
   "arrowParens": "always",
   "singleQuote": true,
-  "semi": true,
-  "bracketSpacing": true,
-  "bracketSameLine": false,
-  "printWidth": 80,
   "plugins": [
     "prettier-plugin-tailwindcss",
     "@prettier/plugin-ruby",
@@ -15,13 +11,5 @@
     "@4az/prettier-plugin-html-erb"
   ],
   "xmlQuoteAttributes": "double",
-  "xmlWhitespaceSensitivity": "ignore",
-  "overrides": [
-    {
-      "files": ["*.ts", "*.tsx"],
-      "options": {
-        "parser": "typescript"
-      }
-    }
-  ]
+  "xmlWhitespaceSensitivity": "ignore"
 }

.rspec_parallel

@@ -1,2 +0,0 @@
---format progress
---format ParallelTests::RSpec::RuntimeLogger --out tmp/parallel_runtime_rspec.log

.ruby-version

@@ -1 +1 @@
-3.4.4
+system

.vscode/launch.json

@@ -3,8 +3,9 @@
   "configurations": [
     {
       "type": "rdbg",
-      "name": "rdbg - attach",
-      "request": "attach"
+      "name": "Attach rdbg",
+      "request": "attach",
+      "rdbgPath": "export GEM_HOME=/usr/local/rvm/gems/default && bundle exec rdbg"
     }
   ]
 }

.vscode/settings.json

@@ -5,14 +5,6 @@
   "workbench.preferredDarkColorTheme": "Spinel",
   "workbench.preferredLightColorTheme": "Spinel Light",
   "rubyLsp.formatter": "syntax_tree",
-  "rubyLsp.featureFlags": {
-    "fullTestDiscovery": true
-  },
-  "rubyLsp.addonSettings": {
-    "Ruby LSP RSpec": {
-      "debug": true
-    }
-  },
   "files.associations": {
     ".env-cmdrc": "json"
   },

Dockerfile

@@ -1,5 +1,25 @@
+FROM ruby:3.2.6 AS native-gems
+RUN rm -f /etc/apt/apt.conf.d/docker-clean; \
+    echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/keep-cache
+RUN \
+    --mount=type=cache,target=/var/cache/apt,sharing=locked \
+    --mount=type=cache,target=/var/lib/apt,sharing=locked \
+    apt-get update && \
+    apt-get install --no-install-recommends --no-install-suggests -y \
+    cmake
+
+WORKDIR /usr/src/app
+RUN gem install bundler -v '2.5.6'
+COPY gems gems
+WORKDIR /usr/src/app/gems/xdiff-rb
+RUN bundle _2.5.6_ install
+RUN rake compile
+WORKDIR /usr/src/app/gems/rb-bsdiff
+RUN bundle _2.5.6_ install
+RUN rake compile
+
 # Primary image
-FROM ruby:3.4.4
+FROM ruby:3.2.6
 USER root
 
 # apt caching & install packages
@@ -14,10 +34,10 @@ RUN \
     libblas-dev liblapack-dev
 
 # preinstall gems that take a long time to install
-RUN MAKE="make -j12" gem install bundler           -v '2.6.7'
-RUN MAKE="make -j12" gem install rice              -v '4.3.3'
-RUN MAKE="make -j12" gem install faiss             -v '0.3.2'
-RUN MAKE="make -j12" gem install rails_live_reload -v '0.3.6'
+RUN MAKE="make -j12" gem install bundler           -v '2.5.6' --verbose
+RUN MAKE="make -j12" gem install rice              -v '4.3.3' --verbose
+RUN MAKE="make -j12" gem install faiss             -v '0.3.2' --verbose
+RUN MAKE="make -j12" gem install rails_live_reload -v '0.3.6' --verbose
 RUN bundle config --global frozen 1
 
 # set up nodejs 18.x deb repo
@@ -32,6 +52,7 @@ RUN \
     --mount=type=cache,target=/var/lib/apt,sharing=locked \
     apt-get update && \
     apt-get install --no-install-recommends --no-install-suggests -y \
+    libvips42 \
     ca-certificates \
     curl \
     gnupg \
@@ -43,60 +64,12 @@ RUN \
     pdftohtml \
     libreoffice
 
-# Install vips dependencies
-RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
-    --mount=type=cache,target=/var/lib/apt,sharing=locked \
-    apt-get update && \
-    apt-get install --no-install-recommends --no-install-suggests -qqy \
-    automake \
-    gtk-doc-tools \
-    gobject-introspection \
-    libgirepository1.0-dev \
-    libglib2.0-dev \
-    libexpat1-dev \
-    libjpeg-dev \
-    libpng-dev \
-    libtiff5-dev \
-    libwebp-dev \
-    libheif-dev \
-    libexif-dev \
-    liblcms2-dev \
-    libxml2-dev \
-    libfftw3-dev \
-    liborc-0.4-dev \
-    libcgif-dev \
-    libjxl-dev \
-    libopenjp2-7-dev \
-    meson \
-    ninja-build
-
-# Install imagemagick from source
-RUN cd /tmp && \
-    wget -qO- https://imagemagick.org/archive/releases/ImageMagick-7.1.2-1.tar.xz | tar -xJ && \
-    cd ImageMagick-7.1.2-1 && \
-    ./configure && \
-    make -j$(nproc) && \
-    make install && \
-    ldconfig && \
-    cd / && \
-    rm -rf /tmp/ImageMagick-7.1.2-1*
-
-# Install vips from source
-RUN cd /tmp && \
-    wget -qO- https://github.com/libvips/libvips/releases/download/v8.17.1/vips-8.17.1.tar.xz | tar -xJ && \
-    cd vips-8.17.1 && \
-    meson setup build --prefix=/usr/local -Dcgif=enabled && \
-    cd build && \
-    ninja && \
-    ninja install && \
-    ldconfig && \
-    cd / && \
-    rm -rf /tmp/vips-8.17.1*
+COPY --from=native-gems /usr/src/app/gems/xdiff-rb /gems/xdiff-rb
+COPY --from=native-gems /usr/src/app/gems/rb-bsdiff /gems/rb-bsdiff
 
 WORKDIR /usr/src/app
 COPY Gemfile Gemfile.lock ./
-COPY gems/has_aux_table ./gems/has_aux_table
-RUN ls -lah gems && BUNDLE_FROZEN=true MAKE="make -j$(nproc)" bundle install --jobs $(nproc)
+RUN bundle _2.5.6_ install
 
 # install js dependencies
 COPY package.json yarn.lock ./

Gemfile

@@ -1,17 +1,18 @@
 source "https://rubygems.org"
 git_source(:github) { |repo| "https://github.com/#{repo}.git" }
 
-ruby "~> 3"
-gem "bundler", "~> 2.6.7"
+ruby "3.2.6"
+# ruby "3.0.3"
 
 # Bundle edge Rails instead: gem "rails", github: "rails/rails", branch: "main"
 gem "rails", "~> 7.2"
-gem "has_aux_table", path: "gems/has_aux_table"
 
 # The original asset pipeline for Rails [https://github.com/rails/sprockets-rails]
 gem "sprockets-rails"
 
+# Use sqlite3 as the database for Active Record
 gem "pg"
+gem "sqlite3", "~> 1.4"
 
 gem "pry"
 gem "pry-stack_explorer"
@@ -55,7 +56,7 @@ gem "bootsnap", require: false
 
 group :development, :test, :staging do
   # See https://guides.rubyonrails.org/debugging_rails_applications.html#debugging-with-the-debug-gem
-  gem "debug", "~> 1.11", platforms: %i[mri mingw x64_mingw], require: false
+  gem "debug", "~> 1.10", platforms: %i[mri mingw x64_mingw]
 end
 
 group :development, :staging do
@@ -65,8 +66,7 @@ group :development, :staging do
   gem "web-console"
 
   # Speed up commands on slow machines / big apps [https://github.com/rails/spring]
-  gem "spring"
-  gem "spring-commands-rspec"
+  # gem "spring"
 
   # Add speed badges [https://github.com/MiniProfiler/rack-mini-profiler]
   gem "memory_profiler"
@@ -100,9 +100,14 @@ end
 
 group :test, :development do
   gem "parallel_tests"
-  gem "spring-commands-parallel-tests"
 end
 
+gem "xdiff", path: "/gems/xdiff-rb"
+
+# for legacy import
+gem "diffy"
+gem "rb-bsdiff", path: "/gems/rb-bsdiff"
+
 gem "addressable"
 gem "colorize"
 gem "concurrent-ruby-edge", require: "concurrent-edge"
@@ -130,10 +135,6 @@ gem "ruby-bbcode"
 gem "dtext_rb",
     git: "https://github.com/e621ng/dtext_rb",
     ref: "5ef8fd7a5205c832f4c18197911717e7d491494e"
-gem "charlock_holmes"
-
-# Telegram Bot API
-gem "telegram-bot-ruby"
 
 # gem "pghero", git: "https://github.com/dymk/pghero", ref: "e314f99"
 gem "pghero", "~> 3.6"
@@ -147,11 +148,11 @@ gem "attr_json"
 
 group :production, :staging do
   gem "rails_semantic_logger", "~> 4.17"
-  gem "cloudflare-rails"
 end
 
 group :production do
   gem "sd_notify"
+  gem "cloudflare-rails"
 end
 
 gem "rack", "~> 2.2"
@@ -164,9 +165,6 @@ gem "timeout"
 group :development do
   gem "prettier_print"
   gem "syntax_tree", "~> 6.2"
-  gem "unicode_plot" # For terminal-based data visualization (Ruby API)
-  gem "rumale" # Professional machine learning library for Ruby
-  gem "ruby-lsp-rspec", require: false
 end
 
 gem "cssbundling-rails", "~> 1.4"
@@ -181,13 +179,7 @@ gem "pundit", "~> 2.4"
 # Monitoring
 gem "prometheus_exporter", "~> 2.2"
 
-SORBET_VERSION = "0.5.12221"
-gem "sorbet", SORBET_VERSION, group: :development
-gem "sorbet-runtime", SORBET_VERSION
-gem "tapioca", "0.16.6", require: false, group: %i[development test]
+gem "sorbet-static-and-runtime"
+gem "tapioca", require: false
 gem "rspec-sorbet", group: [:test]
 gem "sorbet-struct-comparable"
-
-gem "skyfall", "~> 0.6.0"
-
-gem "didkit", "~> 0.2.3"

Gemfile.lock

@@ -25,12 +25,14 @@ GIT
       websocket-driver
 
 PATH
-  remote: gems/has_aux_table
+  remote: /gems/rb-bsdiff
   specs:
-    has_aux_table (0.1.0)
-      activerecord (>= 7.0)
-      activesupport (>= 7.0)
-      sorbet-runtime (~> 0.5)
+    rb-bsdiff (0.1.0)
+
+PATH
+  remote: /gems/xdiff-rb
+  specs:
+    xdiff (0.0.1)
 
 GEM
   remote: https://rubygems.org/
@@ -110,11 +112,10 @@ GEM
       public_suffix (>= 2.0.2, < 7.0)
     attr_json (2.5.0)
       activerecord (>= 6.0.0, < 8.1)
-    base32 (0.3.4)
-    base64 (0.3.0)
+    base64 (0.2.0)
     bcrypt (3.1.20)
-    benchmark (0.4.1)
-    bigdecimal (3.2.2)
+    benchmark (0.4.0)
+    bigdecimal (3.1.9)
     bindex (0.8.1)
     binding_of_caller (1.0.1)
       debug_inspector (>= 1.2.0)
@@ -130,8 +131,6 @@ GEM
       rack-test (>= 0.6.3)
       regexp_parser (>= 1.5, < 3.0)
       xpath (~> 3.2)
-    cbor (0.5.9.9)
-    charlock_holmes (0.7.9)
     cloudflare-rails (6.2.0)
       actionpack (>= 7.1.0, < 8.1.0)
       activesupport (>= 7.1.0, < 8.1.0)
@@ -144,18 +143,17 @@ GEM
       concurrent-ruby (~> 1.3)
     concurrent-ruby-ext (1.3.4)
       concurrent-ruby (= 1.3.4)
-    connection_pool (2.5.3)
+    connection_pool (2.4.1)
     crass (1.0.6)
     cssbundling-rails (1.4.1)
       railties (>= 6.0.0)
-    csv (3.3.5)
     curb (1.0.6)
     daemons (1.4.1)
     date (3.4.1)
     db-query-matchers (0.14.0)
       activesupport (>= 4.0, < 8.1)
       rspec (>= 3.0)
-    debug (1.11.0)
+    debug (1.10.0)
       irb (~> 1.10)
       reline (>= 0.3.8)
     debug_inspector (1.2.0)
@@ -167,8 +165,8 @@ GEM
       warden (~> 1.2.3)
     dhash-vips (0.2.3.0)
       ruby-vips (~> 2.0, != 2.1.1, != 2.1.0)
-    didkit (0.2.3)
     diff-lcs (1.5.1)
+    diffy (3.4.3)
     discard (1.4.0)
       activerecord (>= 4.2, < 9.0)
     disco (0.5.1)
@@ -178,34 +176,10 @@ GEM
       nokogiri (~> 1.13, >= 1.13.0)
       rubyzip (~> 2.0)
     domain_name (0.6.20240107)
-    drb (2.2.3)
-    dry-core (1.1.0)
-      concurrent-ruby (~> 1.0)
-      logger
-      zeitwerk (~> 2.6)
-    dry-inflector (1.2.0)
-    dry-logic (1.6.0)
-      bigdecimal
-      concurrent-ruby (~> 1.0)
-      dry-core (~> 1.1)
-      zeitwerk (~> 2.6)
-    dry-struct (1.8.0)
-      dry-core (~> 1.1)
-      dry-types (~> 1.8, >= 1.8.2)
-      ice_nine (~> 0.11)
-      zeitwerk (~> 2.6)
-    dry-types (1.8.3)
-      bigdecimal (~> 3.0)
-      concurrent-ruby (~> 1.0)
-      dry-core (~> 1.0)
-      dry-inflector (~> 1.0)
-      dry-logic (~> 1.4)
-      zeitwerk (~> 2.6)
-    enumerable-statistics (2.0.8)
+    drb (2.2.1)
     erubi (1.13.1)
     et-orbi (1.2.11)
       tzinfo
-    eventmachine (1.2.7)
     execjs (2.10.0)
     factory_bot (6.5.0)
       activesupport (>= 5.0.0)
@@ -215,17 +189,6 @@ GEM
     faiss (0.3.2)
       numo-narray
       rice (>= 4.0.2)
-    faraday (2.13.4)
-      faraday-net_http (>= 2.0, < 3.5)
-      json
-      logger
-    faraday-multipart (1.1.1)
-      multipart-post (~> 2.0)
-    faraday-net_http (3.4.1)
-      net-http (>= 0.5.0)
-    faye-websocket (0.12.0)
-      eventmachine (>= 0.12.0)
-      websocket-driver (>= 0.8.0)
     ffi (1.17.1-aarch64-linux-gnu)
     ffi (1.17.1-aarch64-linux-musl)
     ffi (1.17.1-arm64-darwin)
@@ -247,22 +210,16 @@ GEM
       fugit (>= 1.11.0)
       railties (>= 6.1.0)
       thor (>= 1.0.0)
-    google-protobuf (4.31.1-aarch64-linux-gnu)
+    google-protobuf (4.29.2-aarch64-linux)
       bigdecimal
       rake (>= 13)
-    google-protobuf (4.31.1-aarch64-linux-musl)
+    google-protobuf (4.29.2-arm64-darwin)
       bigdecimal
       rake (>= 13)
-    google-protobuf (4.31.1-arm64-darwin)
+    google-protobuf (4.29.2-x86_64-darwin)
       bigdecimal
       rake (>= 13)
-    google-protobuf (4.31.1-x86_64-darwin)
-      bigdecimal
-      rake (>= 13)
-    google-protobuf (4.31.1-x86_64-linux-gnu)
-      bigdecimal
-      rake (>= 13)
-    google-protobuf (4.31.1-x86_64-linux-musl)
+    google-protobuf (4.29.2-x86_64-linux)
       bigdecimal
       rake (>= 13)
     htmlbeautifier (1.4.3)
@@ -275,9 +232,8 @@ GEM
     http-cookie (1.0.8)
       domain_name (~> 0.5)
     http-form_data (2.3.0)
-    i18n (1.14.7)
+    i18n (1.14.6)
       concurrent-ruby (~> 1.0)
-    ice_nine (0.11.2)
     io-console (0.8.0)
     irb (1.14.3)
       rdoc (>= 4.0.0)
@@ -285,7 +241,6 @@ GEM
     jbuilder (2.13.0)
       actionview (>= 5.0.0)
       activesupport (>= 5.0.0)
-    json (2.13.2)
     kaminari (1.2.2)
       activesupport (>= 4.1.0)
       kaminari-actionview (= 1.2.2)
@@ -298,9 +253,6 @@ GEM
       activerecord
       kaminari-core (= 1.2.2)
     kaminari-core (1.2.2)
-    language_server-protocol (3.17.0.5)
-    lbfgsb (0.6.0)
-      numo-narray (>= 0.9.1)
     libmf (0.4.0)
       ffi
     listen (3.9.0)
@@ -309,7 +261,7 @@ GEM
     llhttp-ffi (0.5.0)
       ffi-compiler (~> 1.0)
       rake (~> 13.0)
-    logger (1.7.0)
+    logger (1.6.4)
     loofah (2.23.1)
       crass (~> 1.0.2)
       nokogiri (>= 1.12.0)
@@ -323,15 +275,11 @@ GEM
     memory_profiler (1.1.0)
     method_source (1.1.0)
     mini_mime (1.1.5)
-    minitest (5.25.5)
-    mmh3 (1.2.0)
+    minitest (5.25.4)
     msgpack (1.7.5)
     multi_json (1.15.0)
-    multipart-post (2.4.1)
     neighbor (0.5.1)
       activerecord (>= 7)
-    net-http (0.6.0)
-      uri
     net-imap (0.5.4)
       date
       net-protocol
@@ -453,8 +401,6 @@ GEM
     rbi (0.2.2)
       prism (~> 1.0)
       sorbet-runtime (>= 0.5.9204)
-    rbs (3.9.4)
-      logger
     rdoc (6.10.0)
       psych (>= 4.0.0)
     react_on_rails (14.0.5)
@@ -498,12 +444,6 @@ GEM
     rspec-support (3.13.2)
     ruby-bbcode (2.1.1)
       activesupport (>= 4.2.2)
-    ruby-lsp (0.25.0)
-      language_server-protocol (~> 3.17.0)
-      prism (>= 1.2, < 2.0)
-      rbs (>= 3, < 5)
-    ruby-lsp-rspec (0.1.26)
-      ruby-lsp (~> 0.25.0)
     ruby-prof (1.7.1)
     ruby-prof-speedscope (0.3.0)
       ruby-prof (~> 1.0)
@@ -512,91 +452,6 @@ GEM
       logger
     rubyzip (2.3.2)
     rufo (0.18.0)
-    rumale (1.0.0)
-      numo-narray (>= 0.9.1)
-      rumale-clustering (~> 1.0.0)
-      rumale-core (~> 1.0.0)
-      rumale-decomposition (~> 1.0.0)
-      rumale-ensemble (~> 1.0.0)
-      rumale-evaluation_measure (~> 1.0.0)
-      rumale-feature_extraction (~> 1.0.0)
-      rumale-kernel_approximation (~> 1.0.0)
-      rumale-kernel_machine (~> 1.0.0)
-      rumale-linear_model (~> 1.0.0)
-      rumale-manifold (~> 1.0.0)
-      rumale-metric_learning (~> 1.0.0)
-      rumale-model_selection (~> 1.0.0)
-      rumale-naive_bayes (~> 1.0.0)
-      rumale-nearest_neighbors (~> 1.0.0)
-      rumale-neural_network (~> 1.0.0)
-      rumale-pipeline (~> 1.0.0)
-      rumale-preprocessing (~> 1.0.0)
-      rumale-tree (~> 1.0.0)
-    rumale-clustering (1.0.0)
-      numo-narray (>= 0.9.1)
-      rumale-core (~> 1.0.0)
-    rumale-core (1.0.0)
-      csv (>= 3.1.9)
-      numo-narray (>= 0.9.1)
-    rumale-decomposition (1.0.0)
-      numo-narray (>= 0.9.1)
-      rumale-core (~> 1.0.0)
-    rumale-ensemble (1.0.0)
-      numo-narray (>= 0.9.1)
-      rumale-core (~> 1.0.0)
-      rumale-linear_model (~> 1.0.0)
-      rumale-model_selection (~> 1.0.0)
-      rumale-preprocessing (~> 1.0.0)
-      rumale-tree (~> 1.0.0)
-    rumale-evaluation_measure (1.0.0)
-      numo-narray (>= 0.9.1)
-      rumale-core (~> 1.0.0)
-    rumale-feature_extraction (1.0.0)
-      mmh3 (~> 1.0)
-      numo-narray (>= 0.9.1)
-      rumale-core (~> 1.0.0)
-    rumale-kernel_approximation (1.0.0)
-      numo-narray (>= 0.9.1)
-      rumale-core (~> 1.0.0)
-    rumale-kernel_machine (1.0.0)
-      numo-narray (>= 0.9.1)
-      rumale-core (~> 1.0.0)
-    rumale-linear_model (1.0.0)
-      lbfgsb (>= 0.3.0)
-      numo-narray (>= 0.9.1)
-      rumale-core (~> 1.0.0)
-    rumale-manifold (1.0.0)
-      numo-narray (>= 0.9.1)
-      rumale-core (~> 1.0.0)
-      rumale-decomposition (~> 1.0.0)
-    rumale-metric_learning (1.0.0)
-      lbfgsb (>= 0.3.0)
-      numo-narray (>= 0.9.1)
-      rumale-core (~> 1.0.0)
-      rumale-decomposition (~> 1.0.0)
-    rumale-model_selection (1.0.0)
-      numo-narray (>= 0.9.1)
-      rumale-core (~> 1.0.0)
-      rumale-evaluation_measure (~> 1.0.0)
-      rumale-preprocessing (~> 1.0.0)
-    rumale-naive_bayes (1.0.0)
-      numo-narray (>= 0.9.1)
-      rumale-core (~> 1.0.0)
-    rumale-nearest_neighbors (1.0.0)
-      numo-narray (>= 0.9.1)
-      rumale-core (~> 1.0.0)
-    rumale-neural_network (1.0.0)
-      numo-narray (>= 0.9.1)
-      rumale-core (~> 1.0.0)
-    rumale-pipeline (1.0.0)
-      numo-narray (>= 0.9.1)
-      rumale-core (~> 1.0.0)
-    rumale-preprocessing (1.0.0)
-      numo-narray (>= 0.9.1)
-      rumale-core (~> 1.0.0)
-    rumale-tree (1.0.0)
-      numo-narray (>= 0.9.1)
-      rumale-core (~> 1.0.0)
     sanitize (6.1.3)
       crass (~> 1.0.2)
       nokogiri (>= 1.12.0)
@@ -616,21 +471,15 @@ GEM
       semantic_range (>= 2.3.0)
     shoulda-matchers (6.4.0)
       activesupport (>= 5.2.0)
-    skyfall (0.6.0)
-      base32 (~> 0.3, >= 0.3.4)
-      base64 (~> 0.1)
-      cbor (~> 0.5, >= 0.5.9.6)
-      eventmachine (~> 1.2, >= 1.2.7)
-      faye-websocket (~> 0.12)
-    sorbet (0.5.12221)
-      sorbet-static (= 0.5.12221)
-    sorbet-runtime (0.5.12221)
-    sorbet-static (0.5.12221-aarch64-linux)
-    sorbet-static (0.5.12221-universal-darwin)
-    sorbet-static (0.5.12221-x86_64-linux)
-    sorbet-static-and-runtime (0.5.12221)
-      sorbet (= 0.5.12221)
-      sorbet-runtime (= 0.5.12221)
+    sorbet (0.5.11711)
+      sorbet-static (= 0.5.11711)
+    sorbet-runtime (0.5.11711)
+    sorbet-static (0.5.11711-aarch64-linux)
+    sorbet-static (0.5.11711-universal-darwin)
+    sorbet-static (0.5.11711-x86_64-linux)
+    sorbet-static-and-runtime (0.5.11711)
+      sorbet (= 0.5.11711)
+      sorbet-runtime (= 0.5.11711)
     sorbet-struct-comparable (1.3.0)
       sorbet-runtime (>= 0.5)
     spoom (1.5.0)
@@ -638,11 +487,6 @@ GEM
       prism (>= 0.28.0)
       sorbet-static-and-runtime (>= 0.5.10187)
       thor (>= 0.19.2)
-    spring (4.3.0)
-    spring-commands-parallel-tests (1.0.1)
-      spring (>= 0.9.1)
-    spring-commands-rspec (1.0.4)
-      spring (>= 0.9.1)
     sprockets (4.2.1)
       concurrent-ruby (~> 1.0)
       rack (>= 2.2.4, < 4)
@@ -650,6 +494,10 @@ GEM
       actionpack (>= 6.1)
       activesupport (>= 6.1)
       sprockets (>= 3.0.0)
+    sqlite3 (1.7.3-aarch64-linux)
+    sqlite3 (1.7.3-arm64-darwin)
+    sqlite3 (1.7.3-x86_64-darwin)
+    sqlite3 (1.7.3-x86_64-linux)
     stackprof (0.2.26)
     stimulus-rails (1.3.4)
       railties (>= 6.0.0)
@@ -664,7 +512,7 @@ GEM
     tailwindcss-ruby (3.4.17-arm64-darwin)
     tailwindcss-ruby (3.4.17-x86_64-darwin)
     tailwindcss-ruby (3.4.17-x86_64-linux)
-    tapioca (0.16.6)
+    tapioca (0.16.5)
       bundler (>= 2.2.25)
       netrc (>= 0.11.0)
       parallel (>= 1.21.0)
@@ -673,11 +521,6 @@ GEM
       spoom (>= 1.2.0)
       thor (>= 1.2.0)
       yard-sorbet
-    telegram-bot-ruby (2.4.0)
-      dry-struct (~> 1.6)
-      faraday (~> 2.0)
-      faraday-multipart (~> 1.0)
-      zeitwerk (~> 2.6)
     thor (1.3.2)
     thruster (0.1.11-aarch64-linux)
     thruster (0.1.11-arm64-darwin)
@@ -689,9 +532,6 @@ GEM
       railties (>= 6.0.0)
     tzinfo (2.0.6)
       concurrent-ruby (~> 1.0)
-    unicode_plot (0.0.5)
-      enumerable-statistics (>= 2.0.1)
-    uri (1.0.3)
     useragent (0.16.11)
     warden (1.2.9)
       rack (>= 2.0.9)
@@ -706,8 +546,7 @@ GEM
       selenium-webdriver (~> 4.0, < 4.11)
     webrick (1.9.1)
     websocket (1.2.11)
-    websocket-driver (0.8.0)
-      base64
+    websocket-driver (0.7.6)
       websocket-extensions (>= 0.1.0)
     websocket-extensions (0.1.5)
     xpath (3.2.0)
@@ -733,9 +572,7 @@ DEPENDENCIES
   addressable
   attr_json
   bootsnap
-  bundler (~> 2.6.7)
   capybara
-  charlock_holmes
   cloudflare-rails
   colorize
   concurrent-ruby-edge
@@ -744,10 +581,10 @@ DEPENDENCIES
   curb
   daemons
   db-query-matchers (~> 0.14)
-  debug (~> 1.11)
+  debug (~> 1.10)
   devise (~> 4.9)
   dhash-vips
-  didkit (~> 0.2.3)
+  diffy
   discard
   disco
   docx
@@ -756,7 +593,6 @@ DEPENDENCIES
   faiss
   ffmpeg!
   good_job (~> 4.6)
-  has_aux_table!
   htmlbeautifier
   http (~> 5.2)
   http-cookie
@@ -785,49 +621,43 @@ DEPENDENCIES
   rails-controller-testing
   rails_live_reload!
   rails_semantic_logger (~> 4.17)
+  rb-bsdiff!
   react_on_rails
   ripcord
   rouge
   rspec-rails (~> 7.0)
   rspec-sorbet
   ruby-bbcode
-  ruby-lsp-rspec
   ruby-prof
   ruby-prof-speedscope
   ruby-vips
   rufo
-  rumale
   sanitize (~> 6.1)
   sd_notify
   selenium-webdriver
   shakapacker (~> 6.6)
   shoulda-matchers
-  skyfall (~> 0.6.0)
-  sorbet (= 0.5.12221)
-  sorbet-runtime (= 0.5.12221)
+  sorbet-static-and-runtime
   sorbet-struct-comparable
-  spring
-  spring-commands-parallel-tests
-  spring-commands-rspec
   sprockets-rails
+  sqlite3 (~> 1.4)
   stackprof
   stimulus-rails
   syntax_tree (~> 6.2)
   table_print
   tailwindcss-rails (~> 3.0)
-  tapioca (= 0.16.6)
-  telegram-bot-ruby
+  tapioca
   thruster
   timeout
   turbo-rails
   tzinfo-data
-  unicode_plot
   web-console
   webdrivers
+  xdiff!
   zstd-ruby
 
 RUBY VERSION
    ruby 3.2.6p234
 
 BUNDLED WITH
-   2.6.7
+   2.6.2

Procfile.dev

@@ -1,4 +1,4 @@
-rails: RAILS_ENV=development HTTP_PORT=3001 thrust ./bin/rails server
+rails: RAILS_ENV=development HTTP_PORT=3000 TARGET_PORT=3003 rdbg --command --nonstop --open -- thrust ./bin/rails server -p 3003
 wp-client: RAILS_ENV=development HMR=true ./bin/webpacker-dev-server
 wp-server: RAILS_ENV=development HMR=true SERVER_BUNDLE_ONLY=yes ./bin/webpacker --watch
 css: tailwindcss -c ./config/tailwind.config.js -i ./app/assets/stylesheets/application.tailwind.css -o ./app/assets/builds/tailwind.css --watch

Procfile.staging

@@ -1,5 +1,5 @@
-rails: RAILS_ENV=staging HTTP_PORT=3001 bundle exec thrust ./bin/rails server
+rails: RAILS_ENV=staging HTTP_PORT=3001 TARGET_PORT=3002 bundle exec thrust ./bin/rails server -p 3002
 wp-client: RAILS_ENV=development HMR=true ./bin/webpacker-dev-server
 wp-server: RAILS_ENV=development HMR=true SERVER_BUNDLE_ONLY=yes ./bin/webpacker --watch
 css: RAILS_ENV=development yarn "build:css[debug]" --watch
-prometheus-exporter: RAILS_ENV=staging bundle exec prometheus_exporter --bind 0.0.0.0 --prefix redux_ --label '{"environment": "staging"}'
+prometheus_exporter: RAILS_ENV=staging bundle exec prometheus_exporter --bind 0.0.0.0 --prefix redux_ --label '{"environment": "staging"}'

README.md

@@ -1,29 +1,3 @@
-# Redux Scraper
-
-A Ruby on Rails application for scraping and managing various content sources.
-
-## Setup
-
-This application is configured for optimal development and testing performance:
-
-### Performance Optimizations
-
-- **Bootsnap**: Accelerates gem loading and caching for faster boot times
-- **Spring**: Preloads the Rails application for faster command execution
-
-#### Rails Boot Performance
-
-- Development boot time: ~270ms (87% faster than without optimization)
-- Test environment startup: ~211ms (29% faster than without optimization)
-
-To use Spring-optimized commands:
-```bash
-# Use bin/ executables for Spring acceleration
-bin/rails console
-bin/rails runner "puts 'Hello'"
-bin/rspec spec/
-```
-
 # README
 
 This README would normally document whatever steps are necessary to get the

Rakefile

@@ -40,23 +40,22 @@ task periodic_tasks: %i[environment set_logger_stdout] do
   loop { sleep 10 }
 end
 
-# TODO - migrate to Domain::Post / Domain::User
-# namespace :db_sampler do
-#   task export: :environment do
-#     url_names = ENV["url_names"] || raise("need 'url_names' (comma-separated)")
-#     outfile = $stdout
-#     DbSampler.new(outfile).export(url_names.split(","))
-#   ensure
-#     outfile.close if outfile
-#   end
+namespace :db_sampler do
+  task export: :environment do
+    url_names = ENV["url_names"] || raise("need 'url_names' (comma-separated)")
+    outfile = $stdout
+    DbSampler.new(outfile).export(url_names.split(","))
+  ensure
+    outfile.close if outfile
+  end
 
-#   task import: [:environment] do
-#     infile = $stdin
-#     DbSampler.new(infile).import
-#   ensure
-#     infile.close if infile
-#   end
-# end
+  task import: [:environment] do
+    infile = $stdin
+    DbSampler.new(infile).import
+  ensure
+    infile.close if infile
+  end
+end
 
 task good_job: %i[environment set_ar_stdout set_logger_stdout] do
   env_hash = {
@@ -94,6 +93,138 @@ task :reverse_csv do
   out_csv.close
 end
 
+task migrate_to_domain: :environment do
+  only_user = ENV["only_user"]
+  allowed_domains = %w[e621 fa ib]
+  only_domains = (ENV["only_domains"] || "").split(",")
+  only_domains = allowed_domains if only_domains.empty?
+  if (only_domains - allowed_domains).any?
+    raise "only_domains must be a subset of #{allowed_domains.join(", ")}"
+  end
+
+  migrator = Domain::MigrateToDomain.new
+
+  if only_domains.include?("e621")
+    # migrator.migrate_e621_users(only_user: only_user)
+    # migrator.migrate_e621_posts(only_user: only_user)
+    migrator.migrate_e621_users_favs(only_user: only_user)
+  end
+
+  if only_domains.include?("fa")
+    # migrator.migrate_fa_users(only_user: only_user)
+    # migrator.migrate_fa_posts(only_user: only_user)
+    # migrator.migrate_fa_users_favs(only_user: only_user)
+    migrator.migrate_fa_users_followed_users(only_user: only_user)
+  end
+
+  if only_domains.include?("ib")
+    migrator.migrate_inkbunny_users(only_user: only_user)
+    migrator.migrate_inkbunny_posts(only_user: only_user)
+    migrator.migrate_inkbunny_pools(only_user: nil) if only_user.nil?
+  end
+end
+
+task infer_last_submission_log_entries: :environment do
+  only_fa_id = ENV["only_fa_id"]
+  start = ENV["start_at"]&.to_i || nil
+
+  if only_fa_id
+    relation = Domain::Fa::Post.where(fa_id: only_fa_id)
+  else
+    relation =
+      Domain::Fa::Post
+        .where(state: :ok)
+        .where(last_submission_page_id: nil)
+        .or(Domain::Fa::Post.where(state: :ok).where(posted_at: nil))
+  end
+
+  relation.find_each(batch_size: 10, start:) do |post|
+    parts = ["[id: #{post.id}]", "[fa_id: #{post.fa_id}]"]
+
+    log_entry = post.guess_last_submission_page
+    unless log_entry
+      parts << "[no log entry]"
+      next
+    end
+
+    contents = log_entry.response&.contents
+    unless contents
+      parts << "[no contents]"
+      next
+    end
+
+    parser = Domain::Fa::Parser::Page.new(contents)
+    if parser.submission_not_found?
+      parts << "[removed]"
+      post.state = :removed
+    else
+      posted_at = parser.submission.posted_date
+      post.posted_at ||= posted_at
+      parts << "[posted at: #{posted_at}]"
+    end
+
+    if post.last_submission_page_id.present? &&
+         log_entry.id != post.last_submission_page_id
+      parts << "[overwrite]"
+    end
+    post.last_submission_page_id = log_entry.id
+
+    parts << "[log entry: #{log_entry.id}]"
+    parts << "[uri: #{log_entry.uri.to_s}]"
+    post.save!
+  rescue => e
+    parts << "[error: #{e.message}]"
+  ensure
+    puts parts.join(" ")
+  end
+end
+
+task fix_fa_post_files: :environment do
+  file_ids = ENV["file_ids"]&.split(",") || raise("need 'file_ids'")
+  Domain::Fa::Post
+    .where(file_id: file_ids)
+    .find_each { |post| post.fix_file_by_uri! }
+end
+
+task fix_fa_post_files_by_csv: :environment do
+  require "csv"
+
+  csv_file = ENV["csv_file"] || raise("need 'csv_file'")
+  CSV
+    .open(csv_file, headers: true)
+    .each do |row|
+      id = row["id"].to_i
+      post = Domain::Fa::Post.find(id)
+      post.fix_file_by_uri!
+    end
+end
+
+task fix_buggy_fa_posts: :environment do
+  post_fa_ids = %w[7704069 7704068 6432347 6432346].map(&:to_i)
+
+  require "uri"
+
+  post_fa_ids.each do |fa_id|
+    post = Domain::Fa::Post.find_by(fa_id: fa_id)
+    next unless post&.file
+    post_file_url_str = Addressable::URI.parse(post.file_url_str).to_s
+    file_url_str = Addressable::URI.parse(CGI.unescape(post.file.uri.to_s)).to_s
+    hle = post.guess_last_submission_page
+
+    parser = Domain::Fa::Parser::Page.new(hle.response.contents)
+    if parser.submission_not_found?
+      post.file = nil
+      post.save!
+      puts "submission not found"
+    else
+      submission = parser.submission
+      full_res_img = Addressable::URI.parse(submission.full_res_img)
+      full_res_img.scheme = "https" if full_res_img.scheme.blank?
+      matches = full_res_img.to_s == post.file_url_str
+    end
+  end
+end
+
 task enqueue_fa_posts_missing_files: %i[environment set_logger_stdout] do
   Domain::Post::FaPost
     .where(state: "ok")
@@ -179,12 +310,12 @@ task perform_good_jobs: :environment do
         queue_name: job.queue_name,
         serialized_params: job.serialized_params,
         scheduled_at: job.scheduled_at,
-        created_at: Time.now,
-        updated_at: Time.now,
+        created_at: Time.current,
+        updated_at: Time.current,
         process_id: SecureRandom.uuid,
       )
 
-    start_time = Time.now
+    start_time = Time.current
 
     # Temporarily disable concurrency limits
     job_class = job.job_class.constantize
@@ -194,28 +325,28 @@ task perform_good_jobs: :environment do
     begin
       # Perform the job with deserialized arguments
       GoodJob::CurrentThread.job = job
-      job.update!(performed_at: Time.now)
+      job.update!(performed_at: Time.current)
       job_instance.arguments = deserialized_args
       job_instance.perform_now
 
       # Update execution and job records
       execution.update!(
-        finished_at: Time.now,
+        finished_at: Time.current,
         error: nil,
         error_event: nil,
-        duration: Time.now - start_time,
+        duration: Time.current - start_time,
       )
-      job.update!(finished_at: Time.now)
+      job.update!(finished_at: Time.current)
       puts "Job completed successfully"
     rescue => e
       puts "Job failed: #{e.message}"
       # Update execution and job records with error
       execution.update!(
-        finished_at: Time.now,
+        finished_at: Time.current,
         error: e.message,
         error_event: "execution_failed",
         error_backtrace: e.backtrace,
-        duration: Time.now - start_time,
+        duration: Time.current - start_time,
       )
       job.update!(
         error: "#{e.class}: #{e.message}",
@@ -276,6 +407,56 @@ rescue => e
   binding.pry
 end
 
+task fix_fa_user_avatars: :environment do
+  new_users_missing_avatar =
+    Domain::User::FaUser.where.missing(:avatar).select(:url_name)
+  old_users_with_avatar =
+    Domain::Fa::User
+      .where(url_name: new_users_missing_avatar)
+      .includes(:avatar)
+      .filter(&:avatar)
+
+  old_users_with_avatar.each do |old_user|
+    old_avatar = old_user.avatar
+    new_user = Domain::User::FaUser.find_by(url_name: old_user.url_name)
+
+    if old_avatar.log_entry.nil?
+      puts "enqueue fresh download for #{old_user.url_name}"
+      new_avatar = Domain::UserAvatar.new
+      new_user.avatar = new_avatar
+      new_user.save!
+      Domain::Fa::Job::UserAvatarJob.perform_now(avatar: new_avatar)
+      new_avatar.reload
+
+      binding.pry
+      next
+    end
+
+    new_avatar = Domain::UserAvatar.new
+    new_avatar.log_entry_id = old_avatar.log_entry_id
+    new_avatar.last_log_entry_id = old_avatar.log_entry_id
+    new_avatar.url_str = old_avatar.file_url_str
+    new_avatar.downloaded_at = old_avatar.log_entry&.created_at
+    new_avatar.state =
+      case old_avatar.state
+      when "ok"
+        old_avatar.log_entry_id.present? ? "ok" : "pending"
+      when "file_not_found"
+        new_avatar.error_message = old_avatar.state
+        "file_404"
+      else
+        new_avatar.error_message = old_avatar.state
+        "http_error"
+      end
+    new_user.avatar = new_avatar
+    new_user.save!
+    puts "migrated #{old_user.url_name}"
+  rescue => e
+    puts "error: #{e.message}"
+    binding.pry
+  end
+end
+
 task run_fa_user_avatar_jobs: :environment do
   avatars =
     Domain::UserAvatar
@@ -292,23 +473,109 @@ task run_fa_user_avatar_jobs: :environment do
   end
 end
 
-task create_post_file_fingerprints: %i[environment set_logger_stdout] do
-  task = Tasks::CreatePostFileFingerprintsTask.new
+task sample_migrated_favs: :environment do
+  new_user = Domain::User::FaUser.where.not(migrated_user_favs_at: nil).last
+  old_user = Domain::Fa::User.find_by(url_name: new_user.url_name)
 
-  mode =
-    if ENV["post_file_descending"].present?
-      Tasks::CreatePostFileFingerprintsTask::Mode::PostFileDescending
-    elsif ENV["posts_descending"].present?
-      Tasks::CreatePostFileFingerprintsTask::Mode::PostsDescending
-    elsif ENV["user"].present?
-      Tasks::CreatePostFileFingerprintsTask::Mode::User
-    elsif ENV["users_descending"].present?
-      Tasks::CreatePostFileFingerprintsTask::Mode::UsersDescending
-    else
-      raise "need one of: post_file_descending, posts_descending, user, users_descending"
+  puts "user: #{new_user.url_name}"
+  puts "old fav count: #{old_user.fav_posts.count}"
+  puts "new fav count: #{new_user.faved_posts.count}"
+end
+
+task create_post_file_fingerprints: :environment do
+  def migrate_posts_for_user(user)
+    puts "migrating posts for #{user.to_param}"
+    pb =
+      ProgressBar.create(
+        total: user.posts.count,
+        format: "%t: %c/%C %B %p%% %a %e",
+      )
+
+    user
+      .posts
+      .includes(:files)
+      .find_in_batches(batch_size: 64) do |batch|
+        ReduxApplicationRecord.transaction do
+          batch.each { |post| migrate_post(post) }
+          pb.progress = [pb.progress + 1, pb.total].min
+        end
+      end
+  end
+
+  def migrate_post(post)
+    puts "migrating #{post.id} / #{post.to_param} / '#{post.title_for_view}'"
+    ColorLogger.quiet do
+      post.files.each do |file|
+        migrate_post_file(file)
+      rescue StandardError => e
+        puts "error: #{e.message}"
+      end
     end
+  end
 
-  task.run(mode: mode, user_param: ENV["user"], start_at: ENV["start_at"])
+  def migrate_post_file(post_file)
+    job = Domain::PostFileThumbnailJob.new
+    ColorLogger.quiet do
+      job.perform({ post_file: })
+    rescue => e
+      puts "error: #{e.message}"
+    end
+  end
+
+  if ENV["post_file_descending"].present?
+    total = 49_783_962 # cache this value
+    pb = ProgressBar.create(total:, format: "%t: %c/%C %B %p%% %a %e")
+    i = 0
+    Domain::PostFile
+      .where(state: "ok")
+      .includes(:blob)
+      .find_each(
+        order: :desc,
+        batch_size: 32,
+        start: ENV["start_at"],
+      ) do |post_file|
+        i += 1
+        if i % 100 == 0
+          puts "migrating #{post_file.id} / #{post_file.post.title_for_view}"
+        end
+        migrate_post_file(post_file)
+        pb.progress = [pb.progress + 1, pb.total].min
+      end
+  elsif ENV["posts_descending"].present?
+    # total = Domain::Post.count
+    total = 66_431_808 # cache this value
+    pb = ProgressBar.create(total:, format: "%t: %c/%C %B %p%% %a %e")
+    Domain::Post.find_each(order: :desc) do |post|
+      migrate_post(post) unless post.is_a?(Domain::Post::InkbunnyPost)
+      pb.progress = [pb.progress + 1, pb.total].min
+    end
+  elsif ENV["user"].present?
+    for_user = ENV["user"] || raise("need 'user'")
+    user = DomainController.find_model_from_param(Domain::User, for_user)
+    raise "user '#{for_user}' not found" unless user
+    migrate_posts_for_user(user)
+  elsif ENV["users_descending"].present?
+    # all users with posts, ordered by post count descending
+    migrated_file = File.open("migrated_files.txt", "a+")
+    migrated_file.seek(0)
+    migrated_users = migrated_file.readlines.map(&:strip)
+    users =
+      Domain::User::FaUser.order(
+        Arel.sql("json_attributes->>'num_watched_by' DESC NULLS LAST"),
+      ).pluck(:id)
+
+    users.each do |user_id|
+      user = Domain::User::FaUser.find(user_id)
+      next if migrated_users.include?(user.to_param)
+      puts "migrating posts for #{user.to_param} (#{user.num_watched_by} watched by)"
+      migrate_posts_for_user(user)
+      migrated_file.write("#{user.to_param}\n")
+      migrated_file.flush
+    end
+    migrated_file.close
+  else
+    raise "need 'user' or 'users_descending'"
+  end
 end
 
 task enqueue_pending_post_files: :environment do
@@ -350,45 +617,3 @@ task find_post_files_with_empty_response: :environment do
     pb.progress = [pb.progress + 1, pb.total].min
   end
 end
-
-desc "Enqueue pending post file jobs"
-task enqueue_pending_post_file_jobs: :environment do
-  Tasks::EnqueueDuePostFileJobsTask.new.run
-end
-
-desc "Compute null counter caches for all users"
-task compute_null_user_counter_caches: :environment do
-  counter_caches = {
-    user_post_creations_count: :user_post_creations,
-    user_post_favs_count: :user_post_favs,
-    user_user_follows_from_count: :user_user_follows_from,
-    user_user_follows_to_count: :user_user_follows_to,
-  }
-
-  query =
-    Domain::User.where(
-      counter_caches.map { |col, _| "(\"#{col}\" IS NULL)" }.join(" OR "),
-    )
-  total = query.count
-  query = query.select(:id, *counter_caches.keys)
-
-  puts "computing #{counter_caches.keys.join(", ")} for #{total} users"
-  pb = ProgressBar.create(total:, format: "%t: %c/%C %B %p%% %a %e")
-
-  query.find_in_batches(batch_size: 32) do |batch|
-    ReduxApplicationRecord.transaction do
-      batch.each do |user|
-        nil_caches =
-          counter_caches.keys.filter { |cache| user.send(cache).nil? }
-        Domain::User.reset_counters(
-          user.id,
-          *nil_caches.map { |col| counter_caches[col] },
-        )
-        pb.progress = [pb.progress + 1, total].min
-      end
-    end
-  end
-end
-
-puts "set proc title to #{ARGV.first}"
-Process.setproctitle(ARGV.first) if $0.split("/").last == "rake"

TODO.md

@@ -38,10 +38,4 @@
 - [ ] fix for IDs that have a dot in them - e.g. https://refurrer.com/users/fa@jakke.
 - [ ] Rich inline links to e621 e.g. https://refurrer.com/posts/fa@60070060
 - [ ] Find FaPost that have favs recorded but no scan / file, enqueue scan
-- [x] Bunch of posts with empty responses: posts = Domain::Post.joins(files: :log_entry).where(files: { http_log_entries: { response_sha256: BlobFile::EMPTY_FILE_SHA256 }}).limit(10)
-- [ ] Create GlobalState entries for last FA id on browse page, periodic scan to scan from the newest FA ID to the stored one
-- [ ] GlobalState entries for long running backfill jobs, automatically restart them if they fail
-- [ ] Flag to pass to jobs to log HTTP requests / responses to a directory, HTTP mock helper to read from that directory
-- [ ] fix IP address incorrect for Cloudflare proxied requests
-- [ ] SOCKS5 proxy for additional workers
-- [ ] Backup FA scraper using foxbot & g6jy5jkx466lrqojcngbnksugrcfxsl562bzuikrka5rv7srgguqbjid.onion
+- [ ] Bunch of posts with empty responses: posts = Domain::Post.joins(files: :log_entry).where(files: { http_log_entries: { response_sha256: BlobFile::EMPTY_FILE_SHA256 }}).limit(10)

app/assets/stylesheets/application.tailwind.css

@@ -49,11 +49,11 @@
 }
 
 .log-entry-table-header-cell {
-  @apply bg-slate-50 py-1 text-xs font-medium uppercase tracking-wider text-slate-500;
+  @apply border-b border-slate-200 bg-slate-50 px-2 py-1 text-xs font-medium uppercase tracking-wider text-slate-500;
 }
 
 .log-entry-table-row-cell {
-  @apply flex items-center py-1 text-sm;
+  @apply flex items-center border-b border-slate-200 px-2 py-1 text-sm group-hover:bg-slate-50;
 }
 
 .rich-text-content blockquote {

app/controllers/blob_entries_controller.rb

@@ -29,6 +29,8 @@ class BlobEntriesController < ApplicationController
 
     if show_blob_file(sha256, thumb)
       return
+    elsif BlobFile.migrate_sha256!(sha256) && show_blob_file(sha256, thumb)
+      return
     else
       raise ActiveRecord::RecordNotFound
     end
@@ -51,20 +53,6 @@ class BlobEntriesController < ApplicationController
         file_ext = "gif"
       end
 
-      # content-container may be pre-thumbnailed, see if the file is on the disk
-      if thumb == "content-container" && file_ext == "jpeg"
-        thumbnail_path =
-          Domain::PostFile::Thumbnail.absolute_file_path(
-            sha256,
-            "content_container",
-            0,
-          )
-        if File.exist?(thumbnail_path)
-          send_file(thumbnail_path, type: "image/jpeg", disposition: "inline")
-          return true
-        end
-      end
-
       width, height = thumb_params
       filename = "thumb-#{sha256}-#{thumb}.#{file_ext}"
       cache_key = "vips:#{filename}"
@@ -156,50 +144,34 @@ class BlobEntriesController < ApplicationController
     ).returns(T.nilable([String, String]))
   end
   def thumbnail_image_file(blob_file, width, height, file_ext)
-    blob_file_path = blob_file.absolute_file_path
-
     if file_ext == "gif"
-      VipsUtil.try_load_gif(
-        blob_file_path,
-        load_gif: -> do
-          Rack::MiniProfiler.step("vips: load gif") do
-            # Use libvips' gifload with n=-1 to load all frames
-            image = Vips::Image.gifload(blob_file_path, n: -1)
-            num_frames = image.get("n-pages")
-            image_width, image_height = image.width, (image.height / num_frames)
+      Rack::MiniProfiler.step("vips: load gif") do
+        # Use libvips' gifload with n=-1 to load all frames
+        image = Vips::Image.gifload(blob_file.absolute_file_path, n: -1)
+        num_frames = image.get("n-pages")
+        image_width, image_height = image.width, (image.height / num_frames)
 
-            if width >= image_width && height >= image_height
-              logger.info(
-                "gif is already smaller than requested thumbnail size",
-              )
-              return File.binread(blob_file_path), "image/gif"
-            end
+        if width >= image_width && height >= image_height
+          logger.info("gif is already smaller than requested thumbnail size")
+          return [
+            File.read(blob_file.absolute_file_path, mode: "rb"),
+            "image/gif"
+          ]
+        end
 
-            Rack::MiniProfiler.step("vips: thumbnail gif") do
-              image = image.thumbnail_image(width, height: height)
-              image_buffer =
-                image.gifsave_buffer(
-                  dither: 1,
-                  effort: 1,
-                  interframe_maxerror: 16,
-                  interpalette_maxerror: 10,
-                  interlace: true,
-                )
-              [image_buffer, "image/gif"]
-            end
-          end
-        end,
-        on_load_failed: ->(detected_content_type) do
-          case detected_content_type
-          when %r{image/png}
-            thumbnail_image_file(blob_file, width, height, "png")
-          when %r{image/jpeg}, %r{image/jpg}
-            thumbnail_image_file(blob_file, width, height, "jpeg")
-          else
-            raise
-          end
-        end,
-      )
+        Rack::MiniProfiler.step("vips: thumbnail gif") do
+          image = image.thumbnail_image(width, height: height)
+          image_buffer =
+            image.gifsave_buffer(
+              dither: 1,
+              effort: 1,
+              interframe_maxerror: 16,
+              interpalette_maxerror: 10,
+              interlace: true,
+            )
+          [image_buffer, "image/gif"]
+        end
+      end
     else
       # Original static image thumbnailing logic
       image_buffer =
@@ -213,10 +185,7 @@ class BlobEntriesController < ApplicationController
 
       Rack::MiniProfiler.step("vips: thumbnail image") do
         logger.info("rendering thumbnail as jpeg")
-        [
-          T.let(image_buffer.jpegsave_buffer(interlace: true, Q: 95), String),
-          "image/jpeg",
-        ]
+        [image_buffer.jpegsave_buffer(interlace: true, Q: 95), "image/jpeg"]
       end
     end
   end

app/controllers/domain/fa/api_controller.rb

@@ -0,0 +1,218 @@
+# typed: true
+class Domain::Fa::ApiController < ApplicationController
+  skip_before_action :authenticate_user!
+  before_action :validate_api_token!
+
+  skip_before_action :verify_authenticity_token,
+                     only: %i[enqueue_objects object_statuses similar_users]
+
+  skip_before_action :validate_api_token!,
+                     only: %i[object_statuses similar_users]
+
+  def object_statuses
+    fa_ids = (params[:fa_ids] || []).reject(&:blank?).map(&:to_i)
+    url_names = (params[:url_names] || []).reject(&:blank?)
+
+    url_name_to_user =
+      Domain::User::FaUser
+        .where(url_name: url_names)
+        .map { |user| [T.must(user.url_name), user] }
+        .to_h
+
+    fa_id_to_post =
+      Domain::Post::FaPost
+        .includes(:file)
+        .where(fa_id: fa_ids)
+        .map { |post| [T.must(post.fa_id), post] }
+        .to_h
+
+    posts_response = {}
+    users_response = {}
+
+    fa_ids.each do |fa_id|
+      post = fa_id_to_post[fa_id]
+
+      if post
+        post_state =
+          if post.file.present?
+            "have_file"
+          elsif post.scanned_at?
+            "scanned_post"
+          else
+            post.state
+          end
+
+        post_response = {
+          state: post_state,
+          seen_at: time_ago_or_never(post.created_at),
+          object_url: request.base_url + helpers.domain_post_path(post),
+          post_scan: {
+            last_at: time_ago_or_never(post.scanned_at),
+            due_for_scan: !post.scanned_at?,
+          },
+          file_scan: {
+            last_at: time_ago_or_never(post.file&.created_at),
+            due_for_scan: !post.file&.created_at?,
+          },
+        }
+      else
+        post_response = { state: "not_seen" }
+      end
+
+      posts_response[fa_id] = post_response
+    end
+
+    url_names.each do |url_name|
+      user = url_name_to_user[url_name]
+
+      if user
+        user_response = {
+          created_at: time_ago_or_never(user.created_at),
+          state: user.state,
+          object_url: request.base_url + helpers.domain_user_path(user),
+          page_scan: {
+            last_at: time_ago_or_never(user.scanned_page_at),
+            due_for_scan: user.page_scan.due?,
+          },
+          gallery_scan: {
+            last_at: time_ago_or_never(user.gallery_scan.at),
+            due_for_scan: user.gallery_scan.due?,
+          },
+          favs_scan: {
+            last_at: time_ago_or_never(user.favs_scan.at),
+            due_for_scan: user.favs_scan.due?,
+          },
+        }
+      else
+        user_response = { state: "not_seen" }
+      end
+      users_response[url_name] = user_response
+    end
+
+    render json: { posts: posts_response, users: users_response }
+  end
+
+  def enqueue_objects
+    @enqueue_counts ||= Hash.new { |h, k| h[k] = 0 }
+
+    fa_ids = (params[:fa_ids] || []).map(&:to_i)
+    url_names = (params[:url_names] || [])
+    url_names_to_enqueue = Set.new(params[:url_names_to_enqueue] || [])
+
+    fa_id_to_post =
+      Domain::Fa::Post
+        .includes(:file)
+        .where(fa_id: fa_ids)
+        .map { |post| [post.fa_id, post] }
+        .to_h
+
+    url_name_to_user =
+      Domain::Fa::User
+        .where(url_name: url_names)
+        .map { |user| [user.url_name, user] }
+        .to_h
+
+    fa_ids.each do |fa_id|
+      post = fa_id_to_post[fa_id]
+      defer_post_scan(post, fa_id)
+    end
+
+    url_names.each do |url_name|
+      user = url_name_to_user[url_name]
+      defer_user_scan(user, url_name, url_names_to_enqueue.include?(url_name))
+    end
+
+    enqueue_deferred!
+
+    render json: {
+             post_scans: @enqueue_counts[Domain::Fa::Job::ScanPostJob],
+             post_files: @enqueue_counts[Domain::Fa::Job::ScanFileJob],
+             user_pages: @enqueue_counts[Domain::Fa::Job::UserPageJob],
+             user_galleries: @enqueue_counts[Domain::Fa::Job::UserGalleryJob],
+           }
+  end
+
+  private
+
+  def defer_post_scan(post, fa_id)
+    if !post || !post.scanned?
+      defer_manual(Domain::Fa::Job::ScanPostJob, { fa_id: fa_id }, -17)
+    end
+
+    if post && post.file_uri && !post.file.present?
+      return(
+        defer_manual(
+          Domain::Fa::Job::ScanFileJob,
+          { post: post },
+          -15,
+          "static_file",
+        )
+      )
+    end
+  end
+
+  def defer_user_scan(user, url_name, highpri)
+    if !user || user.due_for_page_scan?
+      defer_manual(
+        Domain::Fa::Job::UserPageJob,
+        { url_name: url_name },
+        highpri ? -16 : -6,
+      )
+      return
+    end
+
+    if !user || user.due_for_gallery_scan?
+      defer_manual(
+        Domain::Fa::Job::UserGalleryJob,
+        { url_name: url_name },
+        highpri ? -14 : -4,
+      )
+      return
+    end
+
+    false
+  end
+
+  def defer_manual(klass, args, priority, queue = "manual")
+    @@enqueue_deduper ||= Set.new
+    return unless @@enqueue_deduper.add?([klass, args, priority])
+
+    @deferred_jobs ||= []
+    @deferred_jobs << [klass, args, priority, queue]
+    @enqueue_counts[klass] += 1
+  end
+
+  def enqueue_deferred!
+    GoodJob::Bulk.enqueue do
+      while job = (@deferred_jobs || []).shift
+        klass, args, priority, queue = job
+        klass.set(priority: priority, queue: queue).perform_later(args)
+      end
+    end
+  end
+
+  def time_ago_or_never(time)
+    if time
+      helpers.time_ago_in_words(time, include_seconds: true) + " ago"
+    else
+      "never"
+    end
+  end
+
+  API_TOKENS = {
+    "a4eb03ac-b33c-439c-9b51-a834d1c5cf48" => "dymk",
+    "56cc81fe-8c00-4436-8981-4580eab00e66" => "taargus",
+    "9c38727f-f11d-41de-b775-0effd86d520c" => "xjal",
+    "e38c568f-a24d-4f26-87f0-dfcd898a359d" => "fyacin",
+    "41fa1144-d4cd-11ed-afa1-0242ac120002" => "soft_fox_lad",
+    "9b3cf444-5913-4efb-9935-bf26501232ff" => "syfaro",
+  }
+
+  def validate_api_token!
+    api_token = request.params[:api_token]
+    api_user_name = API_TOKENS[api_token]
+    return if api_user_name
+    return if VpnOnlyRouteConstraint.new.matches?(request)
+    render status: 403, json: { error: "not authenticated" }
+  end
+end

app/controllers/domain/posts_controller.rb

@@ -18,7 +18,7 @@ class Domain::PostsController < DomainController
                        visual_results
                      ]
   before_action :set_post!, only: %i[show]
-  before_action :set_user!, only: %i[user_created_posts]
+  before_action :set_user!, only: %i[user_favorite_posts user_created_posts]
   before_action :set_post_group!, only: %i[posts_in_group]
 
   class PostsIndexViewConfig < T::ImmutableStruct
@@ -65,12 +65,28 @@ class Domain::PostsController < DomainController
     authorize @post
   end
 
+  sig(:final) { void }
+  def user_favorite_posts
+    @posts_index_view_config =
+      PostsIndexViewConfig.new(
+        show_domain_filters: false,
+        show_creator_links: true,
+        index_type_header: "user_favorites",
+      )
+
+    @user = T.must(@user)
+    authorize @user
+    @posts = posts_relation(@user.faved_posts)
+    authorize @posts
+    render :index
+  end
+
   sig(:final) { void }
   def user_created_posts
     @posts_index_view_config =
       PostsIndexViewConfig.new(
         show_domain_filters: false,
-        show_creator_links: false,
+        show_creator_links: true,
         index_type_header: "user_created",
       )
 
@@ -124,47 +140,35 @@ class Domain::PostsController < DomainController
     authorize Domain::Post
 
     # Process the uploaded image or URL
-    file_result = process_image_input
-    return unless file_result
-    file_path, content_type = file_result
+    image_result = process_image_input
+    return unless image_result
+
+    image_path, content_type = image_result
 
     # Create thumbnail for the view if possible
-    tmp_dir = Dir.mktmpdir("visual-search")
-    thumbs_and_fingerprints =
-      helpers.generate_fingerprints(file_path, content_type, tmp_dir)
-    first_thumb_and_fingerprint = thumbs_and_fingerprints&.first
-    if thumbs_and_fingerprints.nil? || first_thumb_and_fingerprint.nil?
-      flash.now[:error] = "Error generating fingerprints"
-      render :visual_search
-      return
+    @uploaded_image_data_uri = create_thumbnail(image_path, content_type)
+    @uploaded_hash_value = generate_fingerprint(image_path)
+    @uploaded_detail_hash_value = generate_detail_fingerprint(image_path)
+    @post_file_fingerprints =
+      find_similar_fingerprints(@uploaded_hash_value).to_a
+    @post_file_fingerprints.sort! do |a, b|
+      helpers.calculate_similarity_percentage(
+        b.fingerprint_detail_value,
+        @uploaded_detail_hash_value,
+      ) <=>
+        helpers.calculate_similarity_percentage(
+          a.fingerprint_detail_value,
+          @uploaded_detail_hash_value,
+        )
     end
-    logger.info("generated #{thumbs_and_fingerprints.length} thumbs")
-
-    @uploaded_image_data_uri =
-      helpers.create_image_thumbnail_data_uri(
-        first_thumb_and_fingerprint.thumb_path,
-        "image/jpeg",
-      )
-    @uploaded_detail_hash_value = first_thumb_and_fingerprint.detail_fingerprint
-    before = Time.now
-
-    similar_fingerprints =
-      helpers.find_similar_fingerprints(
-        thumbs_and_fingerprints.map(&:to_fingerprint_and_detail),
-      ).take(10)
-
-    @time_taken = Time.now - before
-
-    @matches = similar_fingerprints
-    @good_matches =
-      similar_fingerprints.select { |f| f.similarity_percentage >= 80 }
-    @bad_matches =
-      similar_fingerprints.select { |f| f.similarity_percentage < 80 }
-
-    @matches = @good_matches if @good_matches.any?
+    @post_file_fingerprints = @post_file_fingerprints.take(10)
+    @posts = @post_file_fingerprints.map(&:post_file).compact.map(&:post)
   ensure
     # Clean up any temporary files
-    FileUtils.rm_rf(tmp_dir) if tmp_dir
+    if @temp_file
+      @temp_file.unlink
+      @temp_file = nil
+    end
   end
 
   private
@@ -228,6 +232,47 @@ class Domain::PostsController < DomainController
     nil
   end
 
+  # Create a thumbnail from the image and return the data URI
+  sig do
+    params(image_path: String, content_type: String).returns(T.nilable(String))
+  end
+  def create_thumbnail(image_path, content_type)
+    helpers.create_image_thumbnail_data_uri(image_path, content_type)
+  end
+
+  # Generate a fingerprint from the image path
+  sig { params(image_path: String).returns(String) }
+  def generate_fingerprint(image_path)
+    # Use the new from_file_path method to create a fingerprint
+    Domain::PostFile::BitFingerprint.from_file_path(image_path)
+  end
+
+  # Generate a detail fingerprint from the image path
+  sig { params(image_path: String).returns(String) }
+  def generate_detail_fingerprint(image_path)
+    Domain::PostFile::BitFingerprint.detail_from_file_path(image_path)
+  end
+
+  # Find similar images based on the fingerprint
+  sig { params(fingerprint_value: String).returns(ActiveRecord::Relation) }
+  def find_similar_fingerprints(fingerprint_value)
+    # Use the model's similar_to_fingerprint method directly
+
+    subquery = <<~SQL
+      (
+        select distinct on (post_file_id) *, (fingerprint_value <~> '#{ActiveRecord::Base.connection.quote_string(fingerprint_value)}') as distance
+        from #{Domain::PostFile::BitFingerprint.table_name}
+        order by post_file_id, distance asc
+      ) subquery
+    SQL
+
+    Domain::PostFile::BitFingerprint
+      .select("*")
+      .from(subquery)
+      .order("distance ASC")
+      .limit(32)
+  end
+
   sig { override.returns(DomainController::DomainParamConfig) }
   def self.param_config
     DomainController::DomainParamConfig.new(
@@ -238,17 +283,14 @@ class Domain::PostsController < DomainController
   end
 
   sig(:final) do
-    params(
-      starting_relation: ActiveRecord::Relation,
-      skip_ordering: T::Boolean,
-    ).returns(
+    params(starting_relation: ActiveRecord::Relation).returns(
       T.all(ActiveRecord::Relation, Kaminari::ActiveRecordRelationMethods),
     )
   end
-  def posts_relation(starting_relation, skip_ordering: false)
+  def posts_relation(starting_relation)
     relation = starting_relation
     relation = T.unsafe(policy_scope(relation)).page(params[:page]).per(50)
-    relation = relation.order("posted_at DESC NULLS LAST") unless skip_ordering
+    relation = relation.order(relation.klass.post_order_attribute => :desc)
     relation
   end
 end

app/controllers/domain/user_job_events_controller.rb

@@ -1,120 +0,0 @@
-# typed: strict
-class Domain::UserJobEventsController < DomainController
-  extend T::Sig
-
-  before_action :set_user!
-
-  sig { void }
-  def tracked_objects_kinds
-    @kinds =
-      T.let(
-        Domain::UserJobEvent::AddTrackedObject.kinds.keys.map(&:to_s),
-        T.nilable(T::Array[String]),
-      )
-
-    @kind_counts =
-      T.let(
-        @user&.add_tracked_objects&.reorder(nil)&.group(:kind)&.count,
-        T.nilable(T::Hash[String, Integer]),
-      )
-
-    @kinds_most_at =
-      T.let(
-        @user
-          &.add_tracked_objects
-          &.reorder(nil)
-          &.group(:kind)
-          &.maximum(:requested_at),
-        T.nilable(T::Hash[String, Time]),
-      )
-  end
-
-  sig { void }
-  def tracked_objects
-    set_and_validate_kind!
-    @tracked_objects =
-      T.let(
-        T
-          .must(@user)
-          .add_tracked_objects
-          .includes(:log_entry)
-          .where(kind: @kind)
-          .sort_by { |a| -a.requested_at.to_i },
-        T.untyped,
-      )
-  end
-
-  sig { void }
-  def backfill_scan_job
-    set_and_validate_kind!
-    @user = T.must(@user)
-
-    unless @user.is_a?(Domain::User::FaUser)
-      flash[:error] = "This user is not a FurAffinity user"
-      redirect_to tracked_objects_domain_user_job_events_path(
-                    @user,
-                    kind: @kind,
-                  )
-      return
-    end
-
-    case @kind
-    when "favs"
-      now = Time.now
-      stats = Domain::Fa::BackfillTrackedObjectUserFavs.new(user: @user).run
-      flash[
-        :success
-      ] = "Backfilled #{@user.url_name} favs, #{stats.total_created} favs scans created, #{stats.total_favs} favs, loaded #{stats.total_hles} logs, took #{helpers.distance_of_time_in_words_to_now(now, include_seconds: true)}"
-    end
-
-    redirect_to tracked_objects_domain_user_job_events_path(@user, kind: @kind)
-  end
-
-  sig { void }
-  def enqueue_scan_job
-    set_and_validate_kind!
-    @user = T.must(@user)
-
-    unless @user.is_a?(Domain::User::FaUser)
-      flash[:error] = "This user is not a FurAffinity user"
-      redirect_to tracked_objects_domain_user_job_events_path(
-                    @user,
-                    kind: @kind,
-                  )
-      return
-    end
-
-    case @kind
-    when "favs"
-      flash[:success] = "Enqueued scan job for #{@user.url_name} favs"
-      Domain::Fa::Job::FavsJob.set(queue: "manual").perform_later(
-        user: @user,
-        force_scan: true,
-      )
-    else
-      flash[:error] = "Unimplemented kind: #{@kind}"
-    end
-
-    redirect_to tracked_objects_domain_user_job_events_path(@user, kind: @kind)
-  end
-
-  private
-
-  sig { override.returns(DomainController::DomainParamConfig) }
-  def self.param_config
-    DomainController::DomainParamConfig.new(
-      user_id_param: :domain_user_id,
-      post_id_param: :domain_post_id,
-      post_group_id_param: :domain_post_group_id,
-    )
-  end
-
-  sig { void }
-  def set_and_validate_kind!
-    @kind = T.let(params[:kind], T.nilable(String))
-    raise ActionController::RoutingError, "Not Found" if @kind.blank?
-    unless Domain::UserJobEvent::AddTrackedObject.kinds.include?(@kind)
-      raise ActionController::RoutingError, "Not Found"
-    end
-  end
-end

app/controllers/domain/user_post_favs_controller.rb

@@ -1,29 +0,0 @@
-# typed: true
-# frozen_string_literal: true
-
-class Domain::UserPostFavsController < DomainController
-  before_action :set_user!, only: %i[favorites]
-
-  def self.param_config
-    DomainParamConfig.new(
-      post_id_param: :domain_post_id,
-      user_id_param: :domain_user_id,
-      post_group_id_param: :domain_post_group_id,
-    )
-  end
-
-  sig { void }
-  def favorites
-    @posts_index_view_config =
-      Domain::PostsController::PostsIndexViewConfig.new(
-        show_domain_filters: false,
-        show_creator_links: true,
-        index_type_header: "user_favorites",
-      )
-    user = T.cast(@user, Domain::User)
-    @user_post_favs =
-      user.user_post_favs.includes(:post).page(params[:page]).per(50)
-    authorize @user_post_favs
-    render :favorites
-  end
-end

app/controllers/domain/users_controller.rb

@@ -3,8 +3,7 @@ class Domain::UsersController < DomainController
   extend T::Sig
   extend T::Helpers
 
-  before_action :set_user!,
-                only: %i[show followed_by following monitor_bluesky_user]
+  before_action :set_user!, only: %i[show followed_by following]
   before_action :set_post!, only: %i[users_faving_post]
   skip_before_action :authenticate_user!,
                      only: %i[
@@ -76,24 +75,6 @@ class Domain::UsersController < DomainController
     authorize Domain::User
     name = params[:name]&.downcase
     name = ReduxApplicationRecord.sanitize_sql_like(name)
-
-    if name.starts_with?("did:plc:") || name.starts_with?("did:pkh:")
-      @user_search_names =
-        Domain::UserSearchName
-          .select(
-            "domain_user_search_names.*, domain_users.*, domain_users_bluesky_aux.did",
-          )
-          .select(
-            "levenshtein(domain_users_bluesky_aux.did, '#{name}') as distance",
-          )
-          .where(
-            user: Domain::User::BlueskyUser.where("did LIKE ?", "#{name}%"),
-          )
-          .joins(:user)
-          .limit(10)
-      return
-    end
-
     @user_search_names =
       Domain::UserSearchName
         .select("domain_user_search_names.*, domain_users.*")
@@ -186,23 +167,6 @@ class Domain::UsersController < DomainController
            }
   end
 
-  sig { void }
-  def monitor_bluesky_user
-    user = T.cast(@user, Domain::User::BlueskyUser)
-    authorize user
-    monitor = Domain::Bluesky::MonitoredObject.build_for_user(user)
-    if monitor.save
-      Domain::Bluesky::Job::ScanUserJob.perform_later(user:)
-      Domain::Bluesky::Job::ScanPostsJob.perform_later(user:)
-      flash[:notice] = "User is now being monitored"
-    else
-      flash[
-        :alert
-      ] = "Error monitoring user: #{monitor.errors.full_messages.join(", ")}"
-    end
-    redirect_to domain_user_path(user)
-  end
-
   private
 
   sig { override.returns(DomainController::DomainParamConfig) }
@@ -214,17 +178,16 @@ class Domain::UsersController < DomainController
     )
   end
 
-  # TODO - make a typed ImmutableStruct for the return type
   sig { params(user: Domain::User::FaUser).returns(T::Hash[Symbol, T.untyped]) }
   def user_to_similarity_entry(user)
     profile_thumb_url = user.avatar&.log_entry&.uri_str
     profile_thumb_url ||=
       begin
         pp_log_entry = get_best_user_page_http_log_entry_for(user)
-        if pp_log_entry
+        if pp_log_entry && (response_bytes = pp_log_entry.response_bytes)
           parser =
-            Domain::Fa::Parser::Page.from_log_entry(
-              pp_log_entry,
+            Domain::Fa::Parser::Page.new(
+              response_bytes,
               require_logged_in: false,
             )
           parser.user_page.profile_thumb_url

app/controllers/global_states_controller.rb

@@ -11,8 +11,6 @@ class GlobalStatesController < ApplicationController
 
   IB_COOKIE_KEYS = %w[inkbunny-username inkbunny-password inkbunny-sid].freeze
 
-  TELEGRAM_KEYS = %w[telegram-bot-token].freeze
-
   def index
     authorize GlobalState
     @global_states = policy_scope(GlobalState).order(:key)
@@ -184,50 +182,6 @@ class GlobalStatesController < ApplicationController
     end
   end
 
-  def telegram_config
-    authorize GlobalState
-    @telegram_config =
-      TELEGRAM_KEYS.map do |key|
-        GlobalState.find_by(key: key) ||
-          GlobalState.new(key: key, value_type: :string)
-      end
-  end
-
-  def edit_telegram_config
-    authorize GlobalState
-    @telegram_config =
-      TELEGRAM_KEYS.map do |key|
-        GlobalState.find_by(key: key) ||
-          GlobalState.new(key: key, value_type: :string)
-      end
-  end
-
-  def update_telegram_config
-    authorize GlobalState
-
-    begin
-      ActiveRecord::Base.transaction do
-        telegram_config_params.each do |key, value|
-          state = GlobalState.find_or_initialize_by(key: key)
-          state.value = value
-          state.value_type = :string
-          state.save!
-        end
-      end
-
-      redirect_to telegram_config_global_states_path,
-                  notice: "Telegram bot configuration was successfully updated."
-    rescue ActiveRecord::RecordInvalid => e
-      @telegram_config =
-        TELEGRAM_KEYS.map do |key|
-          GlobalState.find_by(key: key) ||
-            GlobalState.new(key: key, value_type: :string)
-        end
-      flash.now[:alert] = "Error updating Telegram bot configuration: #{e.message}"
-      render :edit_telegram_config, status: :unprocessable_entity
-    end
-  end
-
   private
 
   def set_global_state
@@ -247,8 +201,4 @@ class GlobalStatesController < ApplicationController
       *IB_COOKIE_KEYS.reject { |key| key == "inkbunny-sid" },
     )
   end
-
-  def telegram_config_params
-    params.require(:telegram_config).permit(*TELEGRAM_KEYS)
-  end
 end

app/controllers/telegram_bot_logs_controller.rb

@@ -1,103 +0,0 @@
-# typed: false
-class TelegramBotLogsController < ApplicationController
-  before_action :set_telegram_bot_log, only: %i[show]
-  after_action :verify_authorized
-
-  def index
-    authorize TelegramBotLog
-
-    # Start with policy scope
-    @telegram_bot_logs = policy_scope(TelegramBotLog)
-
-    # Apply filters
-    @telegram_bot_logs = apply_filters(@telegram_bot_logs)
-
-    # Order by most recent first
-    @telegram_bot_logs = @telegram_bot_logs.recent
-
-    # Paginate with Kaminari
-    @limit = (params[:limit] || 50).to_i.clamp(1, 500)
-    @telegram_bot_logs = @telegram_bot_logs.page(params[:page]).per(@limit)
-
-    # Load associations for display
-    @telegram_bot_logs = @telegram_bot_logs.includes(:processed_image)
-
-    # Set up filter options for the view
-    @status_options = TelegramBotLog.statuses.keys
-    @filter_params =
-      params.slice(
-        :telegram_user_id,
-        :status,
-        :start_date,
-        :end_date,
-        :min_results,
-        :max_results,
-        :slow_requests,
-      )
-  end
-
-  def show
-    authorize @telegram_bot_log
-
-    # The processed_image association will be loaded automatically when accessed
-  end
-
-  private
-
-  def set_telegram_bot_log
-    @telegram_bot_log =
-      TelegramBotLog.includes(:processed_image).find(params[:id])
-  rescue ActiveRecord::RecordNotFound
-    redirect_to telegram_bot_logs_path, alert: "Telegram bot log not found."
-  end
-
-  def apply_filters(scope)
-    # Filter by telegram user ID
-    if params[:telegram_user_id].present?
-      scope = scope.for_user(params[:telegram_user_id].to_i)
-    end
-
-    # Filter by status
-    if params[:status].present? && TelegramBotLog.statuses.key?(params[:status])
-      scope = scope.where(status: params[:status])
-    end
-
-    # Filter by date range
-    if params[:start_date].present?
-      begin
-        start_date = Date.parse(params[:start_date])
-        scope =
-          scope.where("request_timestamp >= ?", start_date.beginning_of_day)
-      rescue Date::Error
-        # Ignore invalid date
-      end
-    end
-
-    if params[:end_date].present?
-      begin
-        end_date = Date.parse(params[:end_date])
-        scope = scope.where("request_timestamp <= ?", end_date.end_of_day)
-      rescue Date::Error
-        # Ignore invalid date
-      end
-    end
-
-    # Filter by search results count
-    if params[:min_results].present?
-      scope =
-        scope.where("search_results_count >= ?", params[:min_results].to_i)
-    end
-
-    if params[:max_results].present?
-      scope =
-        scope.where("search_results_count <= ?", params[:max_results].to_i)
-    end
-
-    # Filter by performance metrics
-    if params[:slow_requests].present? && params[:slow_requests] == "true"
-      scope = scope.slow_requests
-    end
-
-    scope
-  end
-end

app/helpers/domain/bluesky_post_helper.rb

@@ -1,210 +0,0 @@
-# typed: strict
-# frozen_string_literal: true
-
-module Domain::BlueskyPostHelper
-  extend T::Sig
-  include ActionView::Helpers::UrlHelper
-  include HelpersInterface
-  include Domain::PostsHelper
-
-  class FacetPart < T::Struct
-    const :type, Symbol
-    const :value, String
-  end
-
-  sig do
-    params(text: String, facets: T.nilable(T::Array[T.untyped])).returns(
-      T.nilable(String),
-    )
-  end
-  def render_bsky_post_facets(text, facets = nil)
-    return text if facets.blank?
-
-    facets =
-      begin
-        facets.map { |facet| Bluesky::Text::Facet.from_hash(facet) }
-      rescue => e
-        Rails.logger.error("error parsing Bluesky facets: #{e.message}")
-        return text
-      end
-
-    result_parts = T.let([], T::Array[FacetPart])
-    last_end = 0
-
-    # Sort facets by start position to handle them in order
-    sorted_facets = facets.sort_by(&:byteStart)
-
-    sorted_facets.each do |facet|
-      if facet.byteStart < 0 || facet.byteEnd <= facet.byteStart ||
-           facet.byteEnd > text.bytesize
-        next
-      end
-
-      # Skip overlapping facets
-      next if facet.byteStart < last_end
-
-      # Add text before this facet
-      if facet.byteStart > last_end
-        before_text = text.byteslice(last_end, facet.byteStart - last_end)
-        if before_text
-          result_parts << FacetPart.new(type: :text, value: before_text)
-        end
-      end
-
-      # Extract the facet text using byteslice for accurate character extraction
-      facet_text =
-        text.byteslice(facet.byteStart, facet.byteEnd - facet.byteStart)
-      next unless facet_text # Skip if byteslice returns nil
-
-      # Process the facet
-      rendered_facet = render_facet(facet, facet_text)
-      result_parts << FacetPart.new(type: :facet, value: rendered_facet)
-
-      last_end = facet.byteEnd
-    end
-
-    # Add remaining text after the last facet
-    if last_end < text.bytesize
-      remaining_text = text.byteslice(last_end, text.bytesize - last_end)
-      if remaining_text
-        result_parts << FacetPart.new(type: :text, value: remaining_text)
-      end
-    end
-
-    result_parts
-      .map do |part|
-        case part.type
-        when :text
-          part.value.gsub("\n", "<br />")
-        when :facet
-          part.value
-        end
-      end
-      .join
-      .html_safe
-  end
-
-  private
-
-  sig do
-    params(facet: Bluesky::Text::Facet, facet_text: String).returns(String)
-  end
-  def render_facet(facet, facet_text)
-    return facet_text unless facet.features.any?
-
-    # Process the first feature (Bluesky facets typically have one feature per facet)
-    feature = facet.features.first
-    return facet_text unless feature.is_a?(Bluesky::Text::FacetFeature)
-
-    case feature
-    when Bluesky::Text::FacetFeatureMention
-      render_mention_facet(feature, facet_text)
-    when Bluesky::Text::FacetFeatureURI
-      render_link_facet(feature, facet_text)
-    when Bluesky::Text::FacetFeatureTag
-      render_tag_facet(feature, facet_text)
-    else
-      # Unknown facet type, return original text
-      facet_text
-    end
-  end
-
-  sig do
-    params(
-      feature: Bluesky::Text::FacetFeatureMention,
-      facet_text: String,
-    ).returns(String)
-  end
-  def render_mention_facet(feature, facet_text)
-    did = feature.did
-    return facet_text unless did.present?
-
-    # Try to find the user in the database
-    user = Domain::User::BlueskyUser.find_by(did: did)
-
-    if user
-      # Render the inline user partial
-      render(
-        partial: "domain/has_description_html/inline_link_domain_user",
-        locals: {
-          user: user,
-          link_text: facet_text,
-          visual_style: "description-section-link-light",
-        },
-      )
-    else
-      # Render external link to Bluesky profile
-      render(
-        partial: "domain/has_description_html/external_link",
-        locals: {
-          link_text: facet_text,
-          url: "https://bsky.app/profile/#{did}",
-        },
-      )
-    end
-  end
-
-  sig do
-    params(feature: Bluesky::Text::FacetFeatureURI, facet_text: String).returns(
-      String,
-    )
-  end
-  def render_link_facet(feature, facet_text)
-    uri = feature.uri
-    return facet_text unless uri.present?
-
-    source = link_for_source(uri)
-    if source.present? && (model = source.model)
-      case model
-      when Domain::Post
-        return(
-          render(
-            partial: "domain/has_description_html/inline_link_domain_post",
-            locals: {
-              post: model,
-              link_text: facet_text,
-              visual_style: "description-section-link-light",
-            },
-          )
-        )
-      when Domain::User
-        return(
-          render(
-            partial: "domain/has_description_html/inline_link_domain_user",
-            locals: {
-              user: model,
-              link_text: facet_text,
-              visual_style: "description-section-link-light",
-            },
-          )
-        )
-      end
-    end
-
-    render(
-      partial: "domain/has_description_html/external_link",
-      locals: {
-        link_text: facet_text,
-        url: uri,
-      },
-    )
-  end
-
-  sig do
-    params(feature: Bluesky::Text::FacetFeatureTag, facet_text: String).returns(
-      String,
-    )
-  end
-  def render_tag_facet(feature, facet_text)
-    tag = feature.tag
-    return facet_text unless tag.present?
-
-    render(
-      partial: "domain/has_description_html/external_link",
-      locals: {
-        link_text: facet_text,
-        url: "https://bsky.app/hashtag/#{tag}",
-      },
-    )
-  end
-end

app/helpers/domain/descriptions_helper.rb

@@ -57,16 +57,11 @@ module Domain::DescriptionsHelper
   end
 
   WEAK_URL_MATCHER_REGEX =
-    %r{(http(s)?:\/\/)?(www\.)?[-a-zA-Z0-9@:%._\+~#=]{1,256}\.[a-z]{2,6}\b([-a-zA-Z0-9@:%_\+.~#?&//=]*)}
-
-  sig { params(str: String).returns(T.nilable(String)) }
-  def extract_weak_url(str)
-    str.match(WEAK_URL_MATCHER_REGEX)&.[](0)
-  end
+    %r{(http(s)?:\/\/.)?(www\.)?[-a-zA-Z0-9@:%._\+~#=]{2,256}\.[a-z]{2,6}\b([-a-zA-Z0-9@:%_\+.~#?&//=]*)}
 
   ALLOWED_INFERRED_URL_DOMAINS =
     T.let(
-      %w[furaffinity.net inkbunny.net e621.net bsky.app]
+      %w[furaffinity.net inkbunny.net e621.net]
         .flat_map { |domain| [domain, "www.#{domain}"] }
         .freeze,
       T::Array[String],
@@ -77,16 +72,6 @@ module Domain::DescriptionsHelper
     html = model.description_html_for_view
     return nil if html.blank?
 
-    is_bsky_description = model.is_a?(Domain::User::BlueskyUser)
-    visual_style =
-      (
-        if model.is_a?(Domain::User::BlueskyUser)
-          "description-section-link-light"
-        else
-          "description-section-link"
-        end
-      )
-
     case model
     when Domain::Post::E621Post
       dtext_result = DText.parse(html)
@@ -110,23 +95,17 @@ module Domain::DescriptionsHelper
         next unless node.text?
         next unless node.ancestors("a").empty?
         next unless (node_text = T.cast(node.text, T.nilable(String)))
-        next unless (url_text = extract_weak_url(node_text))
-        next if url_text.blank?
+        next unless (match = node_text.match(WEAK_URL_MATCHER_REGEX))
+        next unless (url_text = match[0])
         unless (
                  uri =
                    try_parse_uri(model.description_html_base_domain, url_text)
                )
           next
         end
-        if is_bsky_description
-          unless ALLOWED_EXTERNAL_LINK_DOMAINS.any? { |domain|
-                   url_matches_domain?(domain, uri.host)
-                 }
-            next
-          end
-        elsif ALLOWED_PLAIN_TEXT_URL_DOMAINS.none? do |domain|
-              url_matches_domain?(domain, uri.host)
-            end
+        unless ALLOWED_PLAIN_TEXT_URL_DOMAINS.any? { |domain|
+                 url_matches_domain?(domain, uri.host)
+               }
           next
         end
 
@@ -178,12 +157,20 @@ module Domain::DescriptionsHelper
               when Domain::Post
                 [
                   "domain/has_description_html/inline_link_domain_post",
-                  { post: found_model, link_text: node.text, visual_style: },
+                  {
+                    post: found_model,
+                    link_text: node.text,
+                    visual_style: "description-section-link",
+                  },
                 ]
               when Domain::User
                 [
                   "domain/has_description_html/inline_link_domain_user",
-                  { user: found_model, link_text: node.text, visual_style: },
+                  {
+                    user: found_model,
+                    link_text: node.text,
+                    visual_style: "description-section-link",
+                  },
                 ]
               else
                 raise "Unknown model type: #{found_link.model.class}"
@@ -204,24 +191,14 @@ module Domain::DescriptionsHelper
               end
 
               replacements[node] = Nokogiri::HTML5.fragment(
-                if is_bsky_description
-                  render(
-                    partial: "domain/has_description_html/external_link",
-                    locals: {
-                      link_text: node.text,
-                      url: url.to_s,
-                    },
-                  )
-                else
-                  render(
-                    partial: "domain/has_description_html/inline_link_external",
-                    locals: {
-                      url: url.to_s,
-                      title:,
-                      icon_path: icon_path_for_domain(url.host),
-                    },
-                  )
-                end,
+                render(
+                  partial: "domain/has_description_html/inline_link_external",
+                  locals: {
+                    url: url.to_s,
+                    title:,
+                    icon_path: icon_path_for_domain(url.host),
+                  },
+                ),
               )
               next { node_whitelist: [node] }
             end
@@ -282,13 +259,6 @@ module Domain::DescriptionsHelper
         "rounded-md px-1 transition-all",
         "inline-flex items-center align-bottom",
       ].join(" ")
-    when "description-section-link-light"
-      [
-        "text-sky-600 border-slate-300",
-        "border border-transparent hover:border-slate-500 hover:text-sky-800 hover:bg-slate-200",
-        "rounded-md px-1 transition-all",
-        "inline-flex items-center align-bottom",
-      ].join(" ")
     else
       "blue-link"
     end
@@ -310,12 +280,12 @@ module Domain::DescriptionsHelper
       .cache
       .fetch(cache_key) do
         num_posts =
-          user.has_created_posts? ? user.user_post_creations.size : nil
+          user.has_created_posts? ? user.user_post_creations.count : nil
         registered_at = domain_user_registered_at_string_for_view(user)
         num_followed_by =
-          user.has_followed_by_users? ? user.user_user_follows_to.size : nil
+          user.has_followed_by_users? ? user.user_user_follows_to.count : nil
         num_followed =
-          user.has_followed_users? ? user.user_user_follows_from.size : nil
+          user.has_followed_users? ? user.user_user_follows_from.count : nil
         avatar_thumb_size = icon_size == "large" ? "64-avatar" : "32-avatar"
 
         {
@@ -345,27 +315,12 @@ module Domain::DescriptionsHelper
   end
 
   sig do
-    params(
-      post: Domain::Post,
-      link_text: String,
-      visual_style: String,
-      domain_icon: T::Boolean,
-      link_params: T::Hash[Symbol, T.untyped],
-    ).returns(T::Hash[Symbol, T.untyped])
+    params(post: Domain::Post, link_text: String, visual_style: String).returns(
+      T::Hash[Symbol, T.untyped],
+    )
   end
-  def props_for_post_hover_preview(
-    post,
-    link_text,
-    visual_style,
-    domain_icon: true,
-    link_params: {}
-  )
-    cache_key = [
-      post,
-      policy(post),
-      link_text,
-      "popover_inline_link_domain_post",
-    ]
+  def props_for_post_hover_preview(post, link_text, visual_style)
+    cache_key = [post, policy(post), "popover_inline_link_domain_post"]
     Rails
       .cache
       .fetch(cache_key) do
@@ -373,14 +328,10 @@ module Domain::DescriptionsHelper
           linkText: link_text,
           postId: post.to_param,
           postTitle: post.title,
-          postPath:
-            Rails.application.routes.url_helpers.domain_post_path(
-              post,
-              link_params,
-            ),
+          postPath: Rails.application.routes.url_helpers.domain_post_path(post),
           postThumbnailPath: thumbnail_for_post_path(post),
           postThumbnailAlt: "View on #{domain_name_for_model(post)}",
-          postDomainIcon: domain_icon ? domain_model_icon_path(post) : nil,
+          postDomainIcon: domain_model_icon_path(post),
         }.then do |props|
           if creator = post.primary_creator_for_view
             props[:creatorName] = creator.name_for_view

app/helpers/domain/domain_model_helper.rb

@@ -17,10 +17,6 @@ module Domain::DomainModelHelper
       "E621"
     when Domain::DomainType::Inkbunny
       "Inkbunny"
-    when Domain::DomainType::Sofurry
-      "Sofurry"
-    when Domain::DomainType::Bluesky
-      "Bluesky"
     end
   end
 
@@ -33,10 +29,18 @@ module Domain::DomainModelHelper
       "E621"
     when Domain::DomainType::Inkbunny
       "IB"
-    when Domain::DomainType::Sofurry
-      "SF"
-    when Domain::DomainType::Bluesky
-      "BSKY"
     end
   end
+
+  sig { params(model: Domain::Post).returns(String) }
+  def title_for_post_model(model)
+    case model
+    when Domain::Post::FaPost
+      model.title
+    when Domain::Post::E621Post
+      model.title
+    when Domain::Post::InkbunnyPost
+      model.title
+    end || "(unknown)"
+  end
 end

app/helpers/domain/domain_type.rb

@@ -5,7 +5,5 @@ class Domain::DomainType < T::Enum
     Fa = new
     E621 = new
     Inkbunny = new
-    Sofurry = new
-    Bluesky = new
   end
 end

app/helpers/domain/domains_helper.rb

@@ -11,7 +11,6 @@ module Domain::DomainsHelper
     e621.net
     furaffinity.net
     inkbunny.net
-    bsky.app
   ].freeze
 
   # If a link is detected in an anchor tag and is one of these domains,
@@ -46,10 +45,7 @@ module Domain::DomainsHelper
           redbubble.com
           spreadshirt.com
           spreadshirt.de
-          subscribestar.adult
-          linktr.ee
           t.me
-          trello.com
           tumblr.com
           twitch.tv
           twitter.com
@@ -57,8 +53,6 @@ module Domain::DomainsHelper
           weasyl.com
           x.com
           youtube.com
-          sofurry.com
-          aethy.com
         ] + ALLOWED_PLAIN_TEXT_URL_DOMAINS
       ).freeze,
       T::Array[String],
@@ -91,12 +85,9 @@ module Domain::DomainsHelper
         "t.me" => "telegram.png",
         "tumblr.com" => "tumblr.png",
         "twitter.com" => "x-twitter.png",
-        "trello.com" => "trello.png",
         "weasyl.com" => "weasyl.png",
         "wixmp.com" => "deviantart.png",
         "x.com" => "x-twitter.png",
-        "linktr.ee" => "linktree.png",
-        "aethy.com" => "aethy.png",
       }.freeze,
       T::Hash[String, String],
     )

app/helpers/domain/e621/posts_helper.rb

@@ -1,5 +1,5 @@
 # typed: false
-module Domain::Posts::E621PostsHelper
+module Domain::E621::PostsHelper
   def icon_asset_for_url(url)
     domain = extract_domain(url)
     return nil unless domain
@@ -81,7 +81,7 @@ module Domain::Posts::E621PostsHelper
     return unless %w[www.furaffinity.net furaffinity.net].include?(uri.host)
     fa_id = uri.path.match(%r{/view/(\d+)})[1]
     return unless fa_id
-    raise("not implemented")
+    Domain::Fa::Post.find_by(fa_id: fa_id)
   rescue StandardError
     nil
   end

app/helpers/domain/fa/posts_helper.rb

@@ -0,0 +1,165 @@
+# typed: strict
+module Domain::Fa::PostsHelper
+  extend T::Sig
+
+  include ActionView::Helpers::DateHelper
+  include ActionView::Helpers::SanitizeHelper
+  include ActionView::Helpers::RenderingHelper
+  include ActionView::Helpers::TagHelper
+
+  sig { params(post: Domain::Fa::Post).returns(String) }
+  def post_state_string(post)
+    if post.have_file?
+      "file"
+    elsif post.scanned?
+      "scanned"
+    else
+      post.state || "unknown"
+    end
+  end
+
+  sig do
+    params(
+      params:
+        T.any(ActionController::Parameters, T::Hash[T.untyped, T.untyped]),
+    ).returns(T.nilable(String))
+  end
+  def page_str(params)
+    if (params[:page] || 1).to_i > 1
+      "(page #{params[:page]})"
+    else
+      nil
+    end
+  end
+
+  sig { params(post: Domain::Fa::Post).returns(T.nilable(HttpLogEntry)) }
+  def guess_scanned_http_log_entry(post)
+    HttpLogEntry.find_all_by_uri(
+      "https://www.furaffinity.net/view/#{post.fa_id}",
+    ).first
+  end
+
+  sig { params(post: Domain::Fa::Post).returns(T.nilable(HttpLogEntry)) }
+  def guess_file_downloaded_http_log_entry(post)
+    if (uri = post.file_uri)
+      HttpLogEntry.find_all_by_uri(uri).first
+    end
+  end
+
+  sig { params(html: String).returns(String) }
+  def fa_post_description_sanitized(html)
+    fa_post_id_to_node = {}
+    fa_user_url_name_to_node = {}
+
+    sanitizer =
+      Sanitize.new(
+        elements: %w[br img b i span strong],
+        attributes: {
+          "span" => %w[style],
+        },
+        css: {
+          properties: %w[font-size color],
+        },
+        transformers: [
+          Kernel.lambda do |env|
+            # Only allow and transform FA links
+            if env[:node_name] == "a"
+              node = env[:node]
+
+              # by default, assume the host is www.furaffinity.net
+              href = node["href"]&.downcase || ""
+              href = "//" + href if href.match?(/^(www\.)?furaffinity\.net/)
+              uri =
+                begin
+                  URI.parse(href)
+                rescue URI::InvalidURIError
+                  nil
+                end
+
+              valid_type = !uri.is_a?(URI::MailTo)
+              next { node_whitelist: [node] } if uri.nil? || !valid_type
+
+              uri.host ||= "www.furaffinity.net"
+              uri.scheme ||= "https"
+              path = uri.path
+
+              fa_host_matcher = /^(www\.)?furaffinity\.net$/
+              fa_post_matcher = %r{^/view/(\d+)/?$}
+              fa_user_matcher = %r{^/user/(\w+)/?$}
+
+              if fa_host_matcher.match?(uri.host) && path
+                if match = path.match(fa_post_matcher)
+                  fa_id = match[1].to_i
+                  fa_post_id_to_node[fa_id] = node
+                  next { node_whitelist: [node] }
+                elsif match = path.match(fa_user_matcher)
+                  fa_url_name = match[1]
+                  fa_user_url_name_to_node[fa_url_name] = node
+                  next { node_whitelist: [node] }
+                end
+              end
+
+              # Don't allow any other links
+              node.replace(node.children)
+            end
+          end,
+        ],
+      )
+
+    fragment = Nokogiri::HTML5.fragment(sanitizer.send(:preprocess, html))
+    sanitizer.node!(fragment)
+
+    if fa_post_id_to_node.any?
+      # Batch load posts and their titles, ensuring fa_post_ids are strings
+      posts_by_id =
+        Domain::Fa::Post.where(fa_id: fa_post_id_to_node.keys).index_by(&:fa_id)
+
+      # Replace the link text with post titles if available
+      fa_post_id_to_node.each do |fa_id, node|
+        if (post = posts_by_id[fa_id])
+          node.replace(
+            Nokogiri::HTML5.fragment(
+              render(
+                partial: "domain/fa/posts/description_inline_link_fa_post",
+                locals: {
+                  post: post,
+                },
+              ),
+            ),
+          )
+        else
+          node.replace(node.children)
+        end
+      end
+    end
+
+    if fa_user_url_name_to_node.any?
+      # Batch load users and their names, ensuring fa_user_url_names are strings
+      users_by_url_name =
+        Domain::Fa::User
+          .where(url_name: fa_user_url_name_to_node.keys)
+          .includes(:avatar)
+          .index_by(&:url_name)
+
+      # Replace the link text with user names if available
+      fa_user_url_name_to_node.each do |fa_url_name, node|
+        if (user = users_by_url_name[fa_url_name])
+          node.replace(
+            Nokogiri::HTML5.fragment(
+              render(
+                partial: "domain/fa/posts/description_inline_link_fa_user",
+                locals: {
+                  user: user,
+                },
+              ),
+            ),
+          )
+        else
+          node.replace(node.children)
+        end
+      end
+    end
+
+    raw fragment.to_html(preserve_newline: true)
+  end
+end

app/helpers/domain/fa/users_helper.rb

@@ -0,0 +1,131 @@
+# typed: false
+module Domain::Fa::UsersHelper
+  extend T::Sig
+
+  def avatar_url(sha256, thumb: "32-avatar")
+    blob_path(HexUtil.bin2hex(sha256), format: "jpg", thumb: thumb)
+  end
+
+  def fa_user_avatar_path(user, thumb: nil)
+    if (sha256 = user.avatar&.file_sha256)
+      blob_path(HexUtil.bin2hex(sha256), format: "jpg", thumb: thumb)
+    else
+      # default / 'not found' avatar image
+      # "/blobs/9080fd4e7e23920eb2dccfe2d86903fc3e748eebb2e5aa8c657bbf6f3d941cdc/contents.jpg"
+      asset_path("user-circle.svg")
+    end
+  end
+
+  def sanitized_fa_user_profile_html(html)
+    # try to preload all the FA usernames in the profile
+    maybe_url_names =
+      Nokogiri
+        .HTML(html)
+        .css("a")
+        .flat_map do |node|
+          href = URI.parse(node["href"])
+          right_host = href.host.nil? || href.host == "www.furaffinity.net"
+          right_path = href.path =~ %r{/user/.+}
+          if right_host && right_path
+            [href]
+          else
+            []
+          end
+        end
+        .map { |href| href.path.split("/")[2]&.downcase }
+
+    preloaded_users =
+      Domain::Fa::User
+        .where(url_name: maybe_url_names)
+        .select(:id, :state, :state_detail, :log_entry_detail, :url_name)
+        .joins(:avatar)
+        .includes(:avatar)
+        .index_by(&:url_name)
+
+    raw Sanitize.fragment(
+          html,
+          elements: %w[br img b i span strong],
+          attributes: {
+            "span" => %w[style],
+            "a" => [],
+          },
+          css: {
+            properties: %w[font-size color],
+          },
+          transformers:
+            lambda do |env|
+              return unless env[:node_name] == "a"
+              node = env[:node]
+              href = URI.parse(node["href"])
+              unless href.host == nil || href.host == "www.furaffinity.net"
+                return
+              end
+              return unless href.path =~ %r{/user/.+}
+              url_name = href.path.split("/")[2]&.downcase
+              Sanitize.node!(
+                node,
+                { elements: %w[a], attributes: { "a" => %w[href] } },
+              )
+              node["href"] = domain_fa_user_path(url_name)
+              node["class"] = "text-slate-200 underline decoration-slate-200 " +
+                "decoration-dashed decoration-dashed decoration-1"
+
+              whitelist = [node]
+
+              user =
+                preloaded_users[url_name] ||
+                  Domain::Fa::User.find_by(url_name: url_name)
+              if user
+                img = Nokogiri::XML::Node.new("img", node.document)
+                img["class"] = "inline w-5"
+                img["src"] = fa_user_avatar_path(user, thumb: "32-avatar")
+                node.prepend_child(img)
+                whitelist << img
+              end
+
+              { node_allowlist: whitelist }
+            end,
+        )
+  end
+
+  # TODO - remove this once we've migrated similarity scores to new user model
+  sig do
+    params(
+      user: Domain::User::FaUser,
+      limit: Integer,
+      exclude_followed_by: T.nilable(Domain::User::FaUser),
+    ).returns(T::Array[Domain::User::FaUser])
+  end
+  def similar_users_by_followed(user, limit: 10, exclude_followed_by: nil)
+    factors = Domain::Factors::UserUserFollowToFactors.find_by(user: user)
+    return [] if factors.nil?
+
+    relation =
+      Domain::NeighborFinder
+        .find_neighbors(factors)
+        .limit(limit)
+        .includes(:user)
+    if exclude_followed_by
+      relation =
+        relation.where.not(
+          user_id: exclude_followed_by.followed_users.select(:to_id),
+        )
+    end
+
+    relation.map { |factor| factor.user }
+  end
+
+  def fa_user_account_status(user)
+    log_entry_id = user.log_entry_detail["last_user_page_id"]
+    return "unknown" if log_entry_id.nil?
+    log_entry = HttpLogEntry.find_by(id: log_entry_id)
+    return "unknown" if log_entry.nil?
+    parser =
+      Domain::Fa::Parser::Page.new(
+        log_entry.response.contents,
+        require_logged_in: false,
+      )
+    return "unknown" unless parser.probably_user_page?
+    parser.user_page.account_status
+  end
+end

app/helpers/domain/model_helper.rb

@@ -12,10 +12,11 @@ module Domain::ModelHelper
       partial: String,
       as: Symbol,
       expires_in: ActiveSupport::Duration,
+      cache_key: T.untyped,
     ).returns(T.nilable(String))
   end
-  def render_for_model(model, partial, as:, expires_in: 1.hour)
-    cache_key = [model, policy(model), partial].compact
+  def render_for_model(model, partial, as:, expires_in: 1.hour, cache_key: nil)
+    cache_key ||= [model, policy(model), partial]
     Rails
       .cache
       .fetch(cache_key, expires_in:) do

app/helpers/domain/pagination_helper.rb

@@ -68,18 +68,4 @@ module Domain::PaginationHelper
     path += "?#{uri.query}" if uri.query.present?
     path
   end
-
-  sig do
-    params(
-      params:
-        T.any(ActionController::Parameters, T::Hash[T.untyped, T.untyped]),
-    ).returns(T.nilable(String))
-  end
-  def page_str(params)
-    if (params[:page] || 1).to_i > 1
-      "(page #{params[:page]})"
-    else
-      nil
-    end
-  end
 end

app/helpers/domain/post_groups_helper.rb

@@ -6,8 +6,8 @@ module Domain::PostGroupsHelper
   abstract!
 
   sig { params(post_group: Domain::PostGroup).returns(String) }
-  def domain_post_group_posts_path(post_group)
-    "#{domain_post_groups_path}/#{post_group.to_param}/posts"
+  def domain_post_group_path(post_group)
+    "#{domain_post_groups_path}/#{post_group.to_param}"
   end
 
   sig { returns(String) }

app/helpers/domain/posts_helper.rb

@@ -43,16 +43,6 @@ module Domain::PostsHelper
             domain_icon_path: "domain-icons/inkbunny.png",
             domain_icon_title: "Inkbunny",
           ),
-        Domain::DomainType::Sofurry =>
-          DomainData.new(
-            domain_icon_path: "domain-icons/sofurry.png",
-            domain_icon_title: "SoFurry",
-          ),
-        Domain::DomainType::Bluesky =>
-          DomainData.new(
-            domain_icon_path: "domain-icons/bluesky.png",
-            domain_icon_title: "Bluesky",
-          ),
       },
       T::Hash[Domain::DomainType, DomainData],
     )
@@ -72,7 +62,7 @@ module Domain::PostsHelper
   def gallery_file_for_post(post)
     file = post.primary_file_for_view
     return nil unless file.present?
-    return nil unless file.state_ok? || file.last_status_code == 200
+    return nil unless file.state_ok?
     return nil unless file.log_entry_id.present?
     content_type = file.log_entry&.content_type
     return nil unless content_type.present?
@@ -80,12 +70,10 @@ module Domain::PostsHelper
     file
   end
 
-  sig { params(post: Domain::Post).returns(T.any(T.nilable(String), Symbol)) }
+  sig { params(post: Domain::Post).returns(T.nilable(String)) }
   def gallery_file_info_for_post(post)
-    return :post_pending if post.pending_scan?
     file = post.primary_file_for_view
     return nil unless file.present?
-    return :file_pending if file.state_pending?
     return nil unless file.state_ok?
     return nil unless file.log_entry_id.present?
     content_type = file.log_entry&.content_type || ""
@@ -194,86 +182,6 @@ module Domain::PostsHelper
     file.log_entry&.response_size&.then { |size| number_to_human_size(size) }
   end
 
-  sig do
-    params(
-      post_files: T::Array[Domain::PostFile],
-      initial_file_index: T.nilable(Integer),
-    ).returns(T::Hash[Symbol, T.untyped])
-  end
-  def props_for_post_files(post_files:, initial_file_index: nil)
-    files_data =
-      post_files.map.with_index do |post_file, index|
-        thumbnail_path = nil
-        content_html = nil
-        log_entry = post_file.log_entry
-
-        if log_entry && (log_entry.status_code == 200)
-          if (response_sha256 = log_entry.response_sha256)
-            thumbnail_path = {
-              type: "url",
-              value:
-                blob_path(
-                  HexUtil.bin2hex(response_sha256),
-                  format: "jpg",
-                  thumb: "small",
-                ),
-            }
-          end
-
-          # Generate content HTML
-          content_html =
-            ApplicationController.renderer.render(
-              partial: "log_entries/content_container",
-              locals: {
-                log_entry: log_entry,
-              },
-              assigns: {
-                current_user: nil,
-              },
-            )
-        elsif post_file.state_pending?
-          thumbnail_path = {
-            type: "icon",
-            value: "fa-solid fa-file-arrow-down",
-          }
-        end
-
-        {
-          id: post_file.id,
-          fileState: post_file.state,
-          thumbnailPath: thumbnail_path,
-          hasContent: post_file.log_entry&.status_code == 200,
-          index: index,
-          contentHtml: content_html,
-          fileDetails:
-            (
-              if log_entry
-                {
-                  contentType: log_entry.content_type,
-                  fileSize: log_entry.response_size,
-                  responseTimeMs: log_entry.response_time_ms,
-                  responseStatusCode: log_entry.status_code,
-                  postFileState: post_file.state,
-                  logEntryId: log_entry.id,
-                  logEntryPath: log_entry_path(log_entry),
-                }
-              else
-                nil
-              end
-            ),
-        }
-      end
-
-    # Validate initial_file_index
-    validated_initial_index = nil
-    if initial_file_index && initial_file_index >= 0 &&
-         initial_file_index < post_files.count
-      validated_initial_index = initial_file_index
-    end
-
-    { files: files_data, initialSelectedIndex: validated_initial_index }
-  end
-
   sig { params(url: String).returns(T.nilable(String)) }
   def icon_asset_for_url(url)
     domain = extract_domain(url)
@@ -366,10 +274,9 @@ module Domain::PostsHelper
   IB_HOSTS = %w[*.inkbunny.net inkbunny.net]
   IB_CDN_HOSTS = %w[*.ib.metapix.net ib.metapix.net]
   E621_HOSTS = %w[www.e621.net e621.net]
-  BLUESKY_HOSTS = %w[bsky.app]
 
   URL_SUFFIX_QUERY = T.let(<<-SQL.strip.chomp.freeze, String)
-    lower('url_str') = lower(?)
+    lower(json_attributes->>'url_str') = lower(?)
   SQL
 
   MATCHERS =
@@ -385,7 +292,10 @@ module Domain::PostsHelper
           ],
           find_proc: ->(helper, match, _) do
             if post = Domain::Post::FaPost.find_by(fa_id: match[1])
-              SourceResult.new(model: post, title: post.title_for_view)
+              SourceResult.new(
+                model: post,
+                title: helper.title_for_post_model(post),
+              )
             end
           end,
         ),
@@ -398,7 +308,7 @@ module Domain::PostsHelper
 
             post_file =
               Domain::PostFile.where(
-                "lower('url_str') IN (?, ?, ?, ?, ?, ?)",
+                "lower(json_attributes->>'url_str') IN (?, ?, ?, ?, ?, ?)",
                 "d.furaffinity.net#{url.host}/#{url.path}",
                 "//d.furaffinity.net#{url.host}/#{url.path}",
                 "https://d.furaffinity.net#{url.host}/#{url.path}",
@@ -408,7 +318,9 @@ module Domain::PostsHelper
               ).first
 
             if post_file && (post = post_file.post)
-              SourceResult.new(model: post, title: post.title_for_view)
+              title =
+                T.bind(self, Domain::PostsHelper).title_for_post_model(post)
+              SourceResult.new(model: post, title:)
             end
           end,
         ),
@@ -431,7 +343,8 @@ module Domain::PostsHelper
           patterns: [%r{/s/(\d+)/?}, %r{/submissionview\.php\?id=(\d+)/?}],
           find_proc: ->(helper, match, _) do
             if post = Domain::Post::InkbunnyPost.find_by(ib_id: match[1])
-              SourceResult.new(model: post, title: post.title_for_view)
+              title = helper.title_for_post_model(post)
+              SourceResult.new(model: post, title:)
             end
           end,
         ),
@@ -447,7 +360,8 @@ module Domain::PostsHelper
                    "ib.metapix.net#{url.path}",
                  ).first
               if post = post_file.post
-                SourceResult.new(model: post, title: post.title_for_view)
+                title = helper.title_for_post_model(post)
+                SourceResult.new(model: post, title:)
               end
             end
           end,
@@ -459,7 +373,7 @@ module Domain::PostsHelper
           find_proc: ->(_, match, _) do
             if user =
                  Domain::User::InkbunnyUser.where(
-                   "name = lower(?)",
+                   "lower(json_attributes->>'name') = lower(?)",
                    match[1],
                  ).first
               SourceResult.new(
@@ -475,7 +389,10 @@ module Domain::PostsHelper
           patterns: [%r{/posts/(\d+)/?}],
           find_proc: ->(helper, match, _) do
             if post = Domain::Post::E621Post.find_by(e621_id: match[1])
-              SourceResult.new(model: post, title: post.title_for_view)
+              SourceResult.new(
+                model: post,
+                title: helper.title_for_post_model(post),
+              )
             end
           end,
         ),
@@ -492,44 +409,6 @@ module Domain::PostsHelper
             end
           end,
         ),
-        # Bluesky posts
-        SourceMatcher.new(
-          hosts: BLUESKY_HOSTS,
-          patterns: [%r{/profile/([^/]+)/post/([^/]+)/?$}],
-          find_proc: ->(helper, match, _) do
-            handle_or_did = match[1]
-            post_rkey = match[2]
-            if handle_or_did.start_with?("did:")
-              did = handle_or_did
-            else
-              user = Domain::User::BlueskyUser.find_by(handle: handle_or_did)
-              did = user&.did
-            end
-            next unless did
-            at_uri = "at://#{did}/app.bsky.feed.post/#{post_rkey}"
-            post = Domain::Post::BlueskyPost.find_by(at_uri:)
-            SourceResult.new(model: post, title: post.title_for_view) if post
-          end,
-        ),
-        # Bluesky users
-        SourceMatcher.new(
-          hosts: BLUESKY_HOSTS,
-          patterns: [%r{/profile/([^/]+)\/?$}],
-          find_proc: ->(helper, match, _) do
-            handle_or_did = match[1]
-            user =
-              if handle_or_did.start_with?("did:")
-                Domain::User::BlueskyUser.find_by(did: handle_or_did)
-              else
-                Domain::User::BlueskyUser.find_by(handle: handle_or_did)
-              end
-            next unless user
-            SourceResult.new(
-              model: user,
-              title: user.name_for_view || handle_or_did,
-            )
-          end,
-        ),
       ],
       T::Array[SourceMatcher],
     )
@@ -539,7 +418,7 @@ module Domain::PostsHelper
     return nil if source.blank?
 
     # normalize the source to a lowercase string with a protocol
-    source = source.downcase
+    source.downcase!
     source = "https://" + source unless source.include?("://")
     begin
       uri = URI.parse(source)
@@ -594,67 +473,6 @@ module Domain::PostsHelper
     post.keywords.map(&:strip).reject(&:blank?).compact
   end
 
-  sig do
-    params(post: Domain::Post::InkbunnyPost).returns(
-      T.nilable(T::Array[String]),
-    )
-  end
-  def keywords_for_ib_post(post)
-    post.keywords&.map { |keyword| keyword["keyword_name"] }&.compact
-  end
-
-  sig do
-    params(time: T.nilable(T.any(ActiveSupport::TimeWithZone, Time))).returns(
-      String,
-    )
-  end
-  def time_ago_in_words_no_prefix(time)
-    return "never" if time.nil?
-    time = time.in_time_zone if time.is_a?(Time)
-    time_ago_in_words(time).delete_prefix("over ").delete_prefix("about ")
-  end
-
-  sig do
-    params(faved_at_type: Domain::UserPostFav::FavedAtType).returns(String)
-  end
-  def faved_at_type_icon(faved_at_type)
-    case faved_at_type
-    when Domain::UserPostFav::FavedAtType::PostedAt
-      "fa-clock" # Clock icon for fallback to posted_at
-    when Domain::UserPostFav::FavedAtType::Explicit
-      "fa-calendar-check" # Calendar check for explicitly set time
-    when Domain::UserPostFav::FavedAtType::Inferred
-      "fa-chart-line" # Chart line for inferred from regression model
-    when Domain::UserPostFav::FavedAtType::InferredNow
-      "fa-bolt" # Lightning bolt for computed on the fly
-    end
-  end
-
-  sig do
-    params(faved_at_type: Domain::UserPostFav::FavedAtType).returns(String)
-  end
-  def faved_at_type_tooltip(faved_at_type)
-    case faved_at_type
-    when Domain::UserPostFav::FavedAtType::PostedAt
-      "Estimated from posted date"
-    when Domain::UserPostFav::FavedAtType::Explicit
-      "Exact time recorded"
-    when Domain::UserPostFav::FavedAtType::Inferred
-      "Estimated from regression model"
-    when Domain::UserPostFav::FavedAtType::InferredNow
-      "Estimated in real-time from regression model"
-    end
-  end
-
-  sig { returns(T::Hash[Symbol, T.untyped]) }
-  def props_for_visual_search_form
-    {
-      actionUrl:
-        Rails.application.routes.url_helpers.visual_results_domain_posts_path,
-      csrfToken: form_authenticity_token,
-    }
-  end
-
   private
 
   sig { params(url: String).returns(T.nilable(String)) }
@@ -664,6 +482,8 @@ module Domain::PostsHelper
     nil
   end
 
+  private
+
   TAG_CATEGORY_ORDER =
     T.let(
       %i[artist copyright character species general meta lore invalid].freeze,

app/helpers/domain/user_job_event_helper.rb

@@ -1,36 +0,0 @@
-# typed: strict
-module Domain::UserJobEventHelper
-  extend T::Sig
-  include HelpersInterface
-
-  sig { params(kind: String).returns(String) }
-  def add_tracked_object_kind_for_view(kind)
-    case kind
-    when "favs"
-      "Favs"
-    else
-      kind.titleize
-    end
-  end
-
-  sig { params(kind: String).returns(String) }
-  def add_tracked_object_kind_event_name(kind)
-    case kind
-    when "favs"
-      "favs scan"
-    else
-      "#{kind} scan"
-    end
-  end
-
-  sig { params(duration: ActiveSupport::Duration).returns(String) }
-  def format_duration_since_last_scan(duration)
-    if duration.in_days >= 1
-      pluralize(duration.in_days.round(0), "day")
-    elsif duration.in_hours >= 1
-      pluralize(duration.in_hours.round(0), "hour")
-    else
-      pluralize(duration.in_minutes.round(0), "min")
-    end
-  end
-end

app/helpers/domain/users/fa_users_helper.rb

@@ -1,32 +0,0 @@
-# typed: true
-module Domain::Users::FaUsersHelper
-  extend T::Sig
-  include HelpersInterface
-
-  # TODO - remove this once we've migrated similarity scores to new user model
-  sig do
-    params(
-      user: Domain::User::FaUser,
-      limit: Integer,
-      exclude_followed_by: T.nilable(Domain::User::FaUser),
-    ).returns(T::Array[Domain::User::FaUser])
-  end
-  def similar_users_by_followed(user, limit: 10, exclude_followed_by: nil)
-    factors = Domain::Factors::UserUserFollowToFactors.find_by(user: user)
-    return [] if factors.nil?
-
-    relation =
-      Domain::NeighborFinder
-        .find_neighbors(factors)
-        .limit(limit)
-        .includes(:user)
-    if exclude_followed_by
-      relation =
-        relation.where.not(
-          user_id: exclude_followed_by.followed_users.select(:to_id),
-        )
-    end
-
-    relation.map { |factor| factor.user }
-  end
-end

app/helpers/domain/users_helper.rb

@@ -31,7 +31,7 @@ module Domain::UsersHelper
   end
   def domain_user_registered_at_ts_for_view(user)
     case user
-    when Domain::User::FaUser, Domain::User::E621User, Domain::User::BlueskyUser
+    when Domain::User::FaUser, Domain::User::E621User
       user.registered_at
     else
       nil
@@ -91,6 +91,20 @@ module Domain::UsersHelper
     end
   end
 
+  sig { params(user: Domain::User).returns(String) }
+  def site_name_for_user(user)
+    case user
+    when Domain::User::E621User
+      "E621"
+    when Domain::User::FaUser
+      "Furaffinity"
+    when Domain::User::InkbunnyUser
+      "Inkbunny"
+    else
+      Kernel.raise "Unknown user type: #{user.class}"
+    end
+  end
+
   sig { params(user: Domain::User).returns(String) }
   def site_icon_path_for_user(user)
     case user
@@ -100,10 +114,6 @@ module Domain::UsersHelper
       asset_path("domain-icons/fa.png")
     when Domain::User::InkbunnyUser
       asset_path("domain-icons/inkbunny.png")
-    when Domain::User::SofurryUser
-      asset_path("domain-icons/sofurry.png")
-    when Domain::User::BlueskyUser
-      asset_path("domain-icons/bluesky.png")
     else
       Kernel.raise "Unknown user type: #{user.class}"
     end
@@ -139,27 +149,6 @@ module Domain::UsersHelper
     "#{domain_user_path(user)}/following"
   end
 
-  sig { params(user: Domain::User, kind: String).returns(String) }
-  def tracked_objects_domain_user_job_events_path(user, kind:)
-    unless Domain::UserJobEvent::AddTrackedObject.kinds.include?(kind)
-      Kernel.raise "invalid kind: #{kind}"
-    end
-    "#{domain_user_path(user)}/job_events/tracked_objects/#{kind}"
-  end
-
-  sig { params(user: Domain::User).returns(String) }
-  def tracked_objects_domain_user_job_events_kinds_path(user)
-    "#{domain_user_path(user)}/job_events/tracked_objects"
-  end
-
-  sig { params(user: Domain::User, kind: String).returns(String) }
-  def enqueue_scan_job_domain_user_job_events_path(user, kind:)
-    unless Domain::UserJobEvent::AddTrackedObject.kinds.include?(kind)
-      Kernel.raise "invalid kind: #{kind}"
-    end
-    "#{domain_user_path(user)}/job_events/enqueue_scan_job/#{kind}"
-  end
-
   class StatRow < T::ImmutableStruct
     include T::Struct::ActsAsComparable
 
@@ -177,13 +166,13 @@ module Domain::UsersHelper
   def stat_rows_for_user(user)
     rows = T.let([], T::Array[StatRow])
     if user.has_faved_posts?
-      rows << StatRow.new(name: "Favorites", value: user.user_post_favs.size)
+      rows << StatRow.new(name: "Favorites", value: user.user_post_favs.count)
     end
     if user.has_followed_by_users?
       can_view_link = policy(user).followed_by?
       rows << StatRow.new(
         name: "Followed by",
-        value: user.user_user_follows_to.size,
+        value: user.user_user_follows_to.count,
         link_to: can_view_link ? domain_user_followed_by_path(user) : nil,
       )
     end
@@ -191,7 +180,7 @@ module Domain::UsersHelper
       can_view_link = policy(user).following?
       rows << StatRow.new(
         name: "Following",
-        value: user.user_user_follows_from.size,
+        value: user.user_user_follows_from.count,
         link_to: can_view_link ? domain_user_following_path(user) : nil,
       )
     end
@@ -203,27 +192,6 @@ module Domain::UsersHelper
         due_for_scan ? "fa-hourglass-half" : "fa-check"
       end
 
-    if user.is_a?(Domain::User::BlueskyUser) && can_view_timestamps
-      rows << StatRow.new(
-        name: "Page scanned",
-        value: user.profile_scan,
-        link_to:
-          user.last_scan_log_entry && log_entry_path(user.last_scan_log_entry),
-        fa_icon_class: icon_for.call(user.profile_scan.due?),
-        hover_title: user.profile_scan.interval.inspect,
-      )
-
-      rows << StatRow.new(
-        name: "Posts scanned",
-        value: user.posts_scan,
-        link_to:
-          user.last_posts_scan_log_entry &&
-            log_entry_path(user.last_posts_scan_log_entry),
-        fa_icon_class: icon_for.call(user.posts_scan.due?),
-        hover_title: user.posts_scan.interval.inspect,
-      )
-    end
-
     if user.is_a?(Domain::User::FaUser) && can_view_timestamps
       if can_view_log_entries && hle = user.guess_last_user_page_log_entry
         rows << StatRow.new(
@@ -245,8 +213,6 @@ module Domain::UsersHelper
       rows << StatRow.new(
         name: "Favs",
         value: user.favs_scan,
-        link_to:
-          tracked_objects_domain_user_job_events_path(user, kind: "favs"),
         fa_icon_class: icon_for.call(user.favs_scan.due?),
         hover_title: user.favs_scan.interval.inspect,
       )

app/helpers/domain/visual_search_helper.rb

@@ -1,15 +1,10 @@
-# typed: strict
+# typed: true
 module Domain
   module VisualSearchHelper
-    extend T::Sig
-
     # Calculate the similarity percentage between two fingerprint hash values
     # @param hash_value [String] The hash value to compare
     # @param reference_hash_value [String] The reference hash value to compare against
     # @return [Float] The similarity percentage between 0 and 100
-    sig do
-      params(hash_value: String, reference_hash_value: String).returns(Float)
-    end
     def calculate_similarity_percentage(hash_value, reference_hash_value)
       # Calculate hamming distance between the two hash values
       distance =
@@ -29,7 +24,6 @@ module Domain
     # Determine the background color class based on similarity percentage
     # @param similarity_percentage [Float] The similarity percentage between 0 and 100
     # @return [String] The Tailwind CSS background color class
-    sig { params(similarity_percentage: Float).returns(String) }
     def match_badge_bg_color(similarity_percentage)
       case similarity_percentage
       when 90..100
@@ -46,7 +40,6 @@ module Domain
     # Determine the text color class based on similarity percentage
     # @param similarity_percentage [Float] The similarity percentage between 0 and 100
     # @return [String] The Tailwind CSS text color class
-    sig { params(similarity_percentage: Float).returns(String) }
     def match_text_color(similarity_percentage)
       case similarity_percentage
       when 90..100
@@ -63,128 +56,8 @@ module Domain
     # Get the CSS classes for the match percentage badge
     # @param similarity_percentage [Float] The similarity percentage between 0 and 100
     # @return [String] The complete CSS classes for the match percentage badge
-    sig { params(similarity_percentage: Float).returns(String) }
     def match_badge_classes(similarity_percentage)
-      "#{match_badge_bg_color(similarity_percentage)} text-white text-xs rounded-full px-3 py-1 shadow-md"
-    end
-
-    class SimilarFingerprintResult < T::Struct
-      include T::Struct::ActsAsComparable
-      const :fingerprint, Domain::PostFile::BitFingerprint
-      const :similarity_percentage, Float
-    end
-
-    class FingerprintAndDetail < T::Struct
-      include T::Struct::ActsAsComparable
-      const :fingerprint, String
-      const :detail_fingerprint, String
-    end
-
-    # Find similar images based on the fingerprint
-    sig do
-      params(
-        fingerprints: T::Array[FingerprintAndDetail],
-        limit: Integer,
-        oversearch: Integer,
-        includes: T.untyped,
-      ).returns(T::Array[SimilarFingerprintResult])
-    end
-    def find_similar_fingerprints(
-      fingerprints,
-      limit: 32,
-      oversearch: 2,
-      includes: {}
-    )
-      ActiveRecord::Base.connection.execute("SET ivfflat.probes = 20")
-
-      results =
-        fingerprints.flat_map do |f|
-          Domain::PostFile::BitFingerprint
-            .order(
-              Arel.sql "(fingerprint_value <~> '#{ActiveRecord::Base.connection.quote_string(f.fingerprint)}')"
-            )
-            .limit(limit * oversearch)
-            .includes(includes)
-            .to_a
-            .uniq(&:post_file_id)
-            .map do |other_fingerprint|
-              SimilarFingerprintResult.new(
-                fingerprint: other_fingerprint,
-                similarity_percentage:
-                  calculate_similarity_percentage(
-                    f.detail_fingerprint,
-                    T.must(other_fingerprint.fingerprint_detail_value),
-                  ),
-              )
-            end
-            .sort { |a, b| b.similarity_percentage <=> a.similarity_percentage }
-            .take(limit)
-        end
-
-      results
-        .group_by { |s| T.must(s.fingerprint.post_file_id) }
-        .map do |post_file_id, similar_fingerprints|
-          T.must(similar_fingerprints.max_by(&:similarity_percentage))
-        end
-        .sort_by(&:similarity_percentage)
-        .reverse
-    end
-
-    class GenerateFingerprintsResult < T::Struct
-      extend T::Sig
-      include T::Struct::ActsAsComparable
-
-      const :thumb_path, String
-      const :fingerprint, String
-      const :detail_fingerprint, String
-
-      sig { returns(FingerprintAndDetail) }
-      def to_fingerprint_and_detail
-        FingerprintAndDetail.new(
-          fingerprint: fingerprint,
-          detail_fingerprint: detail_fingerprint,
-        )
-      end
-    end
-
-    # Generate a fingerprint from the image path
-    sig do
-      params(image_path: String, content_type: String, tmp_dir: String).returns(
-        T.nilable(T::Array[GenerateFingerprintsResult]),
-      )
-    end
-    def generate_fingerprints(image_path, content_type, tmp_dir)
-      # Use the new from_file_path method to create a fingerprint
-      media = LoadedMedia.from_file(content_type, image_path)
-      return nil unless media
-      thumbnail_options =
-        LoadedMedia::ThumbnailOptions.new(
-          width: 128,
-          height: 128,
-          quality: 95,
-          size: :force,
-          interlace: false,
-          for_frames: [0.0, 0.1, 0.5, 0.9, 1.0],
-        )
-      frame_nums =
-        thumbnail_options
-          .for_frames
-          .map do |frame_fraction|
-            (frame_fraction * (media.num_frames - 1)).to_i
-          end
-          .uniq
-          .sort
-      frame_nums.map do |frame_num|
-        tmp_file = File.join(tmp_dir, "frame-#{frame_num}.jpg")
-        media.write_frame_thumbnail(frame_num, tmp_file, thumbnail_options)
-        GenerateFingerprintsResult.new(
-          thumb_path: tmp_file,
-          fingerprint:
-            Domain::PostFile::BitFingerprint.from_file_path(tmp_file),
-          detail_fingerprint:
-            Domain::PostFile::BitFingerprint.detail_from_file_path(tmp_file),
-        )
-      end
+      "#{match_badge_bg_color(similarity_percentage)} text-white font-semibold text-xs rounded-full px-3 py-1 shadow-md"
     end
   end
 end

app/helpers/domain_source_helper.rb

@@ -8,7 +8,6 @@ module DomainSourceHelper
       "furaffinity" => "Domain::Post::FaPost",
       "e621" => "Domain::Post::E621Post",
       "inkbunny" => "Domain::Post::InkbunnyPost",
-      "bluesky" => "Domain::Post::BlueskyPost",
     }
   end
 

app/helpers/fa_uri_helper.rb

@@ -1,59 +0,0 @@
-# typed: strict
-
-module FaUriHelper
-  extend T::Sig
-
-  FA_CDN_HOSTS = %w[d.facdn.net d.furaffinity.net].freeze
-
-  class FaMediaUrlInfo < T::ImmutableStruct
-    extend T::Sig
-    include T::Struct::ActsAsComparable
-
-    const :url_name, String
-    const :original_file_posted, Integer
-    const :latest_file_posted, Integer
-    const :filename, String
-    const :filename_with_ts, String
-
-    sig { returns(Time) }
-    def original_file_posted_at
-      Time.at(original_file_posted)
-    end
-
-    sig { returns(Time) }
-    def latest_file_posted_at
-      Time.at(latest_file_posted)
-    end
-  end
-
-  sig { params(url_str: String).returns(T.nilable(FaMediaUrlInfo)) }
-  def self.parse_fa_media_url(url_str)
-    uri = Addressable::URI.parse(url_str)
-    return nil unless is_fa_cdn_host?(uri.host)
-    # paths are in the form of `art/<user.url_name>/<latest_file_ts>/<og_file_ts>.<rest_of_filename>`
-    # latest_file_ts is the timestamp of the most up to date file that has been uploaded for the post
-    # og_file_ts is the timestamp of when the post was originally made
-    path = uri.path
-    match =
-      path.match(
-        %r{/art/(?<url_name>[^/]+)/(stories/)?(?<latest_ts>\d+)/(?<original_ts>\d+)\.(?<filename>.*)},
-      )
-    return nil unless match
-    url_name = match[:url_name]
-    latest_ts = match[:latest_ts].to_i
-    original_ts = match[:original_ts].to_i
-    filename = match[:filename]
-    FaMediaUrlInfo.new(
-      url_name:,
-      original_file_posted: original_ts,
-      latest_file_posted: latest_ts,
-      filename:,
-      filename_with_ts: path.split("/").last,
-    )
-  end
-
-  sig { params(host: String).returns(T::Boolean) }
-  def self.is_fa_cdn_host?(host)
-    FA_CDN_HOSTS.include?(host)
-  end
-end

app/helpers/good_job_helper.rb

@@ -9,9 +9,9 @@ module GoodJobHelper
   class AnsiSegment < T::Struct
     include T::Struct::ActsAsComparable
 
-    prop :text, String
-    prop :class_names, T::Array[String], default: []
-    prop :url, T.nilable(String), default: nil
+    const :text, String
+    const :class_names, T::Array[String]
+    const :url, T.nilable(String)
   end
 
   # ANSI escape code pattern
@@ -20,7 +20,7 @@ module GoodJobHelper
 
   sig { params(text: String).returns(T::Array[AnsiSegment]) }
   def parse_ansi(text)
-    segments = T.let([], T::Array[AnsiSegment])
+    segments = []
     current_classes = T::Array[String].new
 
     # Split the text into parts based on ANSI codes
@@ -48,34 +48,24 @@ module GoodJobHelper
       end
     end
 
-    segments.each_with_index do |s0, idx|
-      s1 = segments[idx + 1] || next
-      if s0.text == "[hle " && s1.text.match(/\d+/)
-        segments[idx + 1] = AnsiSegment.new(
-          text: s1.text,
-          class_names: s1.class_names,
-          url: Rails.application.routes.url_helpers.log_entry_path(s1.text),
-        )
-      end
-    end
-
     # go through segments and detect UUIDs, splitting the segment at the uuid
     # and adding them to the segments array. Should result in a <before>, <uuid>,
     # <after> tuple.
     segments.flat_map do |segment|
-      if (idx = segment.text.index(UUID_REGEX))
+      if segment.text.match?(UUID_REGEX)
+        idx = segment.text.index(UUID_REGEX)
         [
           AnsiSegment.new(
-            text: segment.text[0...idx] || "",
+            text: segment.text[0...idx],
             class_names: segment.class_names,
           ),
           AnsiSegment.new(
-            text: segment.text[idx...idx + 36] || "",
+            text: segment.text[idx...idx + 36],
             class_names: ["log-uuid"],
             url: "/jobs/jobs/#{segment.text[idx...idx + 36]}",
           ),
           AnsiSegment.new(
-            text: segment.text[idx + 36..] || "",
+            text: segment.text[idx + 36..],
             class_names: segment.class_names,
           ),
         ]
@@ -93,18 +83,11 @@ module GoodJobHelper
 
   sig { params(job: GoodJob::Job).returns(T::Array[JobArg]) }
   def arguments_for_job(job)
-    begin
-      deserialized =
-        T.cast(
-          ActiveJob::Arguments.deserialize(job.serialized_params).to_h,
-          T::Hash[String, T.untyped],
-        )
-    rescue ActiveJob::DeserializationError => e
-      Rails.logger.error(
-        "error deserializing job arguments: #{e.class.name} - #{e.message}",
+    deserialized =
+      T.cast(
+        ActiveJob::Arguments.deserialize(job.serialized_params).to_h,
+        T::Hash[String, T.untyped],
       )
-      return [JobArg.new(key: :error, value: e.message, inferred: true)]
-    end
     args_hash =
       T.cast(deserialized["arguments"].first, T::Hash[Symbol, T.untyped])
     args =

app/helpers/log_entries_helper.rb

@@ -223,13 +223,6 @@ module LogEntriesHelper
     tempfile&.close
   end
 
-  sig { params(str: String).returns(String) }
-  def reencode_as_utf8_lossy(str)
-    str.encode("UTF-8", invalid: :replace, undef: :replace, replace: "")
-  rescue StandardError
-    str
-  end
-
   sig { params(rich_text_body: String).returns(String) }
   def try_convert_bbcode_to_html(rich_text_body)
     rich_text_body.bbcode_to_html(false)
@@ -237,12 +230,6 @@ module LogEntriesHelper
     rich_text_body
   end
 
-  sig { params(rich_text_body: String).returns(String) }
-  def try_detect_encoding(rich_text_body)
-    encoding = CharlockHolmes::EncodingDetector.detect(rich_text_body)
-    encoding ? encoding[:encoding] : "UTF-8"
-  end
-
   sig { params(log_entry: HttpLogEntry).returns(T.nilable(String)) }
   def render_rich_text_content(log_entry)
     content_type = log_entry.content_type
@@ -252,9 +239,8 @@ module LogEntriesHelper
     is_plain_text = content_type.starts_with?("text/plain")
 
     if is_plain_text
-      encoding_name = try_detect_encoding(rich_text_body)
-      rich_text_body = rich_text_body.force_encoding(encoding_name)
-      rich_text_body = reencode_as_utf8_lossy(rich_text_body)
+      # rich_text_body.gsub!(/(\r\n|\n|\r)+/, "<br />")
+      rich_text_body = rich_text_body.force_encoding("UTF-8")
       document_html = try_convert_bbcode_to_html(rich_text_body)
     elsif content_type.starts_with?("application/pdf")
       document_html = convert_with_pdftohtml(rich_text_body)
@@ -370,18 +356,4 @@ module LogEntriesHelper
     end
     raw fragment
   end
-
-  sig { params(performed_by: String).returns(String) }
-  def performed_by_to_short_code(performed_by)
-    case performed_by
-    when "direct"
-      "DR"
-    when "airvpn-1-netherlands"
-      "NL"
-    when "airvpn-2-san-jose"
-      "SJ"
-    else
-      "??"
-    end
-  end
 end

app/helpers/paths_helper.rb

@@ -6,6 +6,54 @@ module PathsHelper
   include HelpersInterface
   abstract!
 
+  # sig do
+  #   params(post: Domain::Post, params: T::Hash[Symbol, T.untyped]).returns(
+  #     String,
+  #   )
+  # end
+  # def domain_post_path(post, params = {})
+  #   to_path("#{domain_posts_path}/#{post.to_param}", params)
+  # end
+
+  # sig do
+  #   params(post: Domain::Post, params: T::Hash[Symbol, T.untyped]).returns(
+  #     String,
+  #   )
+  # end
+  # def domain_post_faved_by_path(post, params = {})
+  #   to_path("#{domain_post_path(post)}/faved_by", params)
+  # end
+
+  # sig { params(params: T::Hash[Symbol, T.untyped]).returns(String) }
+  # def domain_posts_path(params = {})
+  #   to_path("/posts", params)
+  # end
+
+  # sig do
+  #   params(
+  #     post_group: Domain::PostGroup,
+  #     params: T::Hash[Symbol, T.untyped],
+  #   ).returns(String)
+  # end
+  # def domain_post_group_posts_path(post_group, params = {})
+  #   to_path("#{domain_post_group_path(post_group)}/posts", params)
+  # end
+
+  # sig do
+  #   params(
+  #     post_group: Domain::PostGroup,
+  #     params: T::Hash[Symbol, T.untyped],
+  #   ).returns(String)
+  # end
+  # def domain_post_group_path(post_group, params = {})
+  #   to_path("#{domain_post_groups_path}/#{post_group.to_param}", params)
+  # end
+
+  # sig { params(params: T::Hash[Symbol, T.untyped]).returns(String) }
+  # def domain_post_groups_path(params = {})
+  #   to_path("/pools", params)
+  # end
+
   private
 
   sig do

app/helpers/source_helper.rb

@@ -0,0 +1,22 @@
+# typed: true
+module SourceHelper
+  def self.source_name_to_class_name
+    {
+      "furaffinity" => "Domain::Fa::Post",
+      "e621" => "Domain::E621::Post",
+      "inkbunny" => "Domain::Inkbunny::Post",
+    }
+  end
+
+  def self.all_source_names
+    source_name_to_class_name.keys
+  end
+
+  def self.source_names_to_class_names(list)
+    list.map { |source| source_name_to_class_name[source] }.compact
+  end
+
+  def self.has_all_sources?(list)
+    list.sort == all_source_names.sort
+  end
+end

app/helpers/telegram_bot_logs_helper.rb

@@ -1,47 +0,0 @@
-# typed: strict
-
-module TelegramBotLogsHelper
-  extend T::Sig
-
-  sig { params(telegram_bot_log: TelegramBotLog).returns(String) }
-  def status_color_class(telegram_bot_log)
-    case telegram_bot_log.status
-    when "processing"
-      "bg-blue-100 text-blue-800"
-    when "success"
-      "bg-green-100 text-green-800"
-    when "error"
-      "bg-red-100 text-red-800"
-    when "invalid_image"
-      "bg-orange-100 text-orange-800"
-    else
-      "bg-slate-100 text-slate-800"
-    end
-  end
-
-  sig { params(blob_file: T.nilable(BlobFile)).returns(String) }
-  def image_dimensions_for_blob_file(blob_file)
-    return "N/A" unless blob_file
-    return "N/A" unless blob_file.content_type&.start_with?("image/")
-
-    begin
-      media =
-        LoadedMedia.from_file(
-          T.must(blob_file.content_type),
-          blob_file.absolute_file_path,
-        )
-      if media.is_a?(LoadedMedia::StaticImage)
-        vips_image = media.instance_variable_get(:@vips_image)
-        "#{vips_image.width}×#{vips_image.height}"
-      elsif media.is_a?(LoadedMedia::Gif)
-        width = media.instance_variable_get(:@width)
-        height = media.instance_variable_get(:@height)
-        "#{width}×#{height}"
-      else
-        "N/A"
-      end
-    rescue StandardError
-      "Unable to determine"
-    end
-  end
-end

app/javascript/bundles/Main/components/DisplayedFile.tsx

@@ -1,43 +0,0 @@
-import * as React from 'react';
-import { FileData } from './PostFiles';
-import { FileDetails } from './FileDetails';
-
-interface DisplayedFileProps {
-  file: FileData;
-}
-
-export const DisplayedFile: React.FC<DisplayedFileProps> = ({ file }) => {
-  return (
-    <>
-      {/* File content display */}
-      <div className="file-content-display mb-4">
-        {file.contentHtml ? (
-          <div dangerouslySetInnerHTML={{ __html: file.contentHtml }} />
-        ) : (
-          <section className="flex grow justify-center text-slate-500">
-            <div>
-              <i className="fa-solid fa-file-arrow-down mr-2"></i>
-              {fileStateContent(file.fileState)}
-            </div>
-          </section>
-        )}
-      </div>
-
-      {/* File details */}
-      {file.fileDetails && <FileDetails {...file.fileDetails} />}
-    </>
-  );
-};
-
-function fileStateContent(fileState: FileData['fileState']) {
-  switch (fileState) {
-    case 'pending':
-      return 'File pending download';
-    case 'terminal_error':
-      return 'File download failed';
-  }
-
-  return 'No file content available';
-}
-
-export default DisplayedFile;

app/javascript/bundles/Main/components/FileCarousel.tsx

@@ -1,85 +0,0 @@
-import * as React from 'react';
-import { FileData } from './PostFiles';
-
-interface FileCarouselProps {
-  files: FileData[];
-  totalFiles: number;
-  selectedIndex: number;
-  onFileSelect: (fileId: number, index: number) => void;
-}
-
-export const FileCarousel: React.FC<FileCarouselProps> = ({
-  files,
-  totalFiles,
-  selectedIndex,
-  onFileSelect,
-}) => {
-  const handleFileClick = (file: FileData) => {
-    onFileSelect(file.id, file.index);
-  };
-
-  // Only render if there are multiple files
-  if (files.length <= 1) {
-    return null;
-  }
-
-  return (
-    <div className="mb-4">
-      <div className="flex gap-2 overflow-x-auto" id="file-carousel">
-        {files.map((file) => {
-          const isSelected = file.index === selectedIndex;
-          const buttonClasses = [
-            'flex-shrink-0',
-            'w-20',
-            'h-20',
-            'rounded-md',
-            'border-2',
-            'transition-all',
-            'duration-200',
-            'hover:border-blue-400',
-            isSelected ? 'border-blue-500' : 'border-gray-300',
-          ];
-
-          if (file.thumbnailPath?.type === 'url') {
-            buttonClasses.push('overflow-hidden');
-          } else {
-            buttonClasses.push(
-              'bg-gray-100',
-              'flex',
-              'items-center',
-              'justify-center',
-            );
-          }
-
-          const thumbnail =
-            file.thumbnailPath?.type === 'url' ? (
-              <img
-                src={file.thumbnailPath.value}
-                className="h-full w-full object-cover"
-                alt={`File ${file.index + 1}`}
-              />
-            ) : file.thumbnailPath?.type === 'icon' ? (
-              <i className={`${file.thumbnailPath.value} text-slate-500`}></i>
-            ) : (
-              <i className="fa-solid fa-file text-gray-400"></i>
-            );
-
-          return (
-            <button
-              key={file.id}
-              className={buttonClasses.join(' ')}
-              onClick={() => handleFileClick(file)}
-              data-file-id={file.id}
-              data-index={file.index}
-              title={`File ${file.index + 1} of ${totalFiles}`}
-            >
-              {thumbnail}
-            </button>
-          );
-        })}
-      </div>
-    </div>
-  );
-};
-
-export default FileCarousel;

app/javascript/bundles/Main/components/FileDetails.tsx

@@ -1,113 +0,0 @@
-import * as React from 'react';
-import { PostFileState } from './PostFiles';
-import { byteCountToHumanSize } from '../utils/byteCountToHumanSize';
-import SkySection from './SkySection';
-
-export interface FileDetailsProps {
-  contentType: string;
-  fileSize: number;
-  responseTimeMs: number;
-  responseStatusCode: number;
-  postFileState: PostFileState;
-  logEntryId: number;
-  logEntryPath: string;
-}
-
-export const FileDetails: React.FC<FileDetailsProps> = ({
-  contentType,
-  fileSize,
-  responseTimeMs,
-  responseStatusCode,
-  postFileState,
-  logEntryId,
-  logEntryPath,
-}) => {
-  return (
-    <SkySection
-      title="File Details"
-      contentClassName="grid grid-cols-3 sm:grid-cols-6 text-sm"
-    >
-      <TitleStat
-        label="Type"
-        value={contentType}
-        iconClass="fa-solid fa-file"
-      />
-      <TitleStat
-        label="Size"
-        value={byteCountToHumanSize(fileSize)}
-        iconClass="fa-solid fa-weight-hanging"
-      />
-      <TitleStat
-        label="Time"
-        value={responseTimeMs == -1 ? undefined : `${responseTimeMs}ms`}
-        iconClass="fa-solid fa-clock"
-      />
-      <TitleStat
-        label="Status"
-        value={responseStatusCode}
-        textClass={
-          responseStatusCode == 200 ? 'text-green-600' : 'text-red-600'
-        }
-        iconClass="fa-solid fa-signal"
-      />
-      <TitleStat
-        label="State"
-        value={postFileState}
-        textClass={postFileState == 'ok' ? 'text-green-600' : 'text-red-600'}
-        iconClass="fa-solid fa-circle-check"
-      />
-      <TitleStat label="Log Entry" iconClass="fa-solid fa-file-pen">
-        <a
-          href={logEntryPath}
-          target="_blank"
-          rel="noopener noreferrer"
-          className="font-medium text-blue-600 hover:text-blue-800"
-        >
-          #{logEntryId}
-        </a>
-      </TitleStat>
-    </SkySection>
-  );
-};
-
-const TitleStat: React.FC<{
-  label: string;
-  value?: string | number;
-  iconClass: string;
-  textClass?: string;
-  children?: React.ReactNode;
-}> = ({ label, value, iconClass, textClass = 'text-slate-600', children }) => {
-  function valueElement(value: string | number | undefined) {
-    const defaultTextClass = 'font-normal';
-    if (value === undefined) {
-      return <span className="text-slate-500">&mdash;</span>;
-    } else if (typeof value === 'number') {
-      return (
-        <span className={`${textClass} ${defaultTextClass}`}>
-          {value.toLocaleString()}
-        </span>
-      );
-    } else {
-      return (
-        <span className={`${textClass} ${defaultTextClass}`}>{value}</span>
-      );
-    }
-  }
-
-  const gridInnerBorderClasses =
-    'border-r border-b border-slate-300 last:border-r-0 sm:last:border-r-0 [&:nth-child(3)]:border-r-0 sm:[&:nth-child(3)]:border-r [&:nth-last-child(-n+3)]:border-b-0 sm:[&:nth-last-child(-n+6)]:border-b-0';
-
-  return (
-    <div
-      className={`flex flex-col justify-center px-2 py-1 ${gridInnerBorderClasses}`}
-    >
-      <div className="flex items-center gap-2 font-light text-slate-600">
-        <i className={iconClass}></i>
-        <span>{label}</span>
-      </div>
-      {children || valueElement(value)}
-    </div>
-  );
-};
-
-export default FileDetails;

app/javascript/bundles/Main/components/ListItem.tsx

@@ -10,7 +10,6 @@ const COMMON_LIST_ELEM_CLASSES = `
 
 interface PropTypes {
   value: string;
-  subvalue?: string;
   subtext?: string;
   thumb?: string;
   isLast: boolean;
@@ -22,7 +21,6 @@ interface PropTypes {
 
 export default function ListItem({
   value,
-  subvalue,
   thumb,
   isLast,
   selected,
@@ -31,11 +29,11 @@ export default function ListItem({
   subtext,
   domainIcon,
 }: PropTypes) {
-  const groupHoverClassName = 'group-hover:text-slate-200';
   const iconClassName = ['ml-2'];
   const textClassName = [
     COMMON_LIST_ELEM_CLASSES,
-    'group flex items-center justify-between',
+    'relative flex items-center justify-between',
+    'border-t-0',
     isLast && 'rounded-b-lg',
     style === 'item' && selected && 'bg-slate-700 text-slate-100',
     style === 'info' && 'text-slate-500 italic',
@@ -56,7 +54,7 @@ export default function ListItem({
       {style === 'error' && (
         <Icon type="exclamation-circle" className={iconClassName.join(' ')} />
       )}
-      <div className={`${textClassName.join(' ')}`}>
+      <div className={textClassName.join(' ')}>
         <div className="inline-block w-8">
           {thumb && (
             <img
@@ -66,44 +64,14 @@ export default function ListItem({
             />
           )}
         </div>
-        <div className="flex flex-grow flex-col pl-1">
-          <span
-            className={['text-lg font-light', subvalue && 'leading-tight']
-              .filter(Boolean)
-              .join(' ')}
-          >
-            {value}
-          </span>
-          {subvalue && (
-            <span
-              className={[
-                'text-sm font-normal group-hover:text-slate-200',
-                !selected && 'text-slate-500',
-                selected && 'text-slate-200',
-              ]
-                .filter(Boolean)
-                .join(' ')}
-            >
-              {subvalue}
-            </span>
-          )}
-        </div>
+        <div className="inline-block flex-grow pl-1">{value}</div>
         {subtext && (
-          <div
-            className={[
-              'vertical-align-middle float-right inline-block pl-1 text-sm italic',
-              !selected && 'text-slate-500',
-              selected && 'text-slate-300',
-              groupHoverClassName,
-            ]
-              .filter(Boolean)
-              .join(' ')}
-          >
+          <div className="vertical-align-middle float-right inline-block pl-1 text-sm italic text-slate-500">
             {subtext}
           </div>
         )}
         {domainIcon && (
-          <img src={domainIcon} alt="domain icon" className="ml-1 inline w-6" />
+          <img src={domainIcon} alt="domain icon" className="inline w-6 pl-1" />
         )}
       </div>
     </a>

app/javascript/bundles/Main/components/PolynomialEquation.tsx

@@ -1,89 +0,0 @@
-import * as React from 'react';
-
-interface PolynomialEquationProps {
-  coefficients: number[];
-  className?: string;
-}
-
-export const PolynomialEquation: React.FC<PolynomialEquationProps> = ({
-  coefficients,
-  className = '',
-}) => {
-  const renderTerm = (coeff: number, degree: number, isFirst: boolean) => {
-    if (Math.abs(coeff) < 1e-10) return null; // Skip near-zero coefficients
-
-    const absCoeff = Math.abs(coeff);
-    const isPositive = coeff >= 0;
-
-    // Determine coefficient display
-    let coeffDisplay = '';
-    if (degree === 0) {
-      // Constant term
-      coeffDisplay = absCoeff.toFixed(3);
-    } else if (absCoeff === 1) {
-      // Coefficient of 1, don't show it
-      coeffDisplay = '';
-    } else {
-      // Regular coefficient
-      coeffDisplay = absCoeff.toFixed(3);
-    }
-
-    // Determine variable display
-    let variableDisplay = null;
-    if (degree === 0) {
-      // No variable for constant term
-      variableDisplay = null;
-    } else if (degree === 1) {
-      // Linear term: just x
-      variableDisplay = <span>x</span>;
-    } else {
-      // Higher degree: x with superscript
-      variableDisplay = (
-        <span>
-          x<sup>{degree}</sup>
-        </span>
-      );
-    }
-
-    // Determine sign display
-    let signDisplay = null;
-    if (isFirst) {
-      // First term: only show minus if negative
-      signDisplay = isPositive ? null : <span>−</span>;
-    } else {
-      // Subsequent terms: always show sign with spacing
-      signDisplay = <span className="mx-1">{isPositive ? '+' : '−'}</span>;
-    }
-
-    return (
-      <React.Fragment key={degree}>
-        {signDisplay}
-        {coeffDisplay && <span>{coeffDisplay}</span>}
-        {variableDisplay}
-      </React.Fragment>
-    );
-  };
-
-  const terms = [];
-  let hasTerms = false;
-
-  // Render terms from highest to lowest degree
-  for (let i = coefficients.length - 1; i >= 0; i--) {
-    const term = renderTerm(coefficients[i], i, !hasTerms);
-    if (term !== null) {
-      terms.push(term);
-      hasTerms = true;
-    }
-  }
-
-  if (!hasTerms) {
-    return <span className={className}>y = 0</span>;
-  }
-
-  return (
-    <span className={className}>
-      <span>y = </span>
-      {terms}
-    </span>
-  );
-};

app/javascript/bundles/Main/components/PostFiles.tsx

@@ -1,123 +0,0 @@
-import * as React from 'react';
-import { useState, useEffect, useCallback } from 'react';
-import { FileCarousel } from './FileCarousel';
-import { DisplayedFile } from './DisplayedFile';
-import { FileDetailsProps } from './FileDetails';
-
-export type PostFileState =
-  | 'pending'
-  | 'ok'
-  | 'file_error'
-  | 'retryable_error'
-  | 'terminal_error'
-  | 'removed';
-
-export interface FileData {
-  id: number;
-  fileState: PostFileState;
-  thumbnailPath?: { type: 'icon' | 'url'; value: string };
-  hasContent: boolean;
-  index: number;
-  contentHtml?: string;
-  fileDetails?: FileDetailsProps;
-}
-
-interface PostFilesProps {
-  files: FileData[];
-  initialSelectedIndex?: number;
-}
-
-export const PostFiles: React.FC<PostFilesProps> = ({
-  files,
-  initialSelectedIndex,
-}) => {
-  if (initialSelectedIndex == null) {
-    initialSelectedIndex = files.findIndex((file) => file.fileState === 'ok');
-    if (initialSelectedIndex === -1) {
-      initialSelectedIndex = 0;
-    }
-  }
-  const [selectedIndex, setSelectedIndex] = useState(initialSelectedIndex);
-
-  // Update URL parameter when selected file changes
-  const updateUrlWithFileIndex = (index: number) => {
-    if (typeof window === 'undefined' || files.length <= 1) return;
-
-    const url = new URL(window.location.href);
-    url.searchParams.set('idx', index.toString());
-    window.history.replaceState({}, '', url.toString());
-  };
-
-  const handleFileSelect = (fileId: number, index: number) => {
-    setSelectedIndex(index);
-    updateUrlWithFileIndex(index);
-  };
-
-  const navigateToNextFile = () => {
-    if (files.length > 1) {
-      const nextIndex = (selectedIndex + 1) % files.length;
-      handleFileSelect(files[nextIndex].id, nextIndex);
-    }
-  };
-
-  const navigateToPreviousFile = () => {
-    if (files.length > 1) {
-      const prevIndex = (selectedIndex - 1 + files.length) % files.length;
-      handleFileSelect(files[prevIndex].id, prevIndex);
-    }
-  };
-
-  // Add keyboard navigation
-  useEffect(() => {
-    const handleKeyDown = (event: KeyboardEvent) => {
-      // Only handle arrow keys if we have multiple files
-      if (files.length <= 1) return;
-
-      switch (event.key) {
-        case 'ArrowLeft':
-          event.preventDefault();
-          navigateToPreviousFile();
-          break;
-        case 'ArrowRight':
-          event.preventDefault();
-          navigateToNextFile();
-          break;
-      }
-    };
-
-    // Add event listener to document
-    document.addEventListener('keydown', handleKeyDown);
-
-    // Cleanup event listener on unmount
-    return () => {
-      document.removeEventListener('keydown', handleKeyDown);
-    };
-  }, [selectedIndex, files.length]);
-
-  const selectedFile = files[selectedIndex];
-
-  return (
-    <section id="file-display-section">
-      {files.length == 0 && (
-        <div className="flex grow justify-center text-slate-500">
-          <div className="flex items-center gap-2">
-            <i className="fa-solid fa-file-circle-exclamation"></i>
-            No files
-          </div>
-        </div>
-      )}
-      {files.length > 1 && (
-        <FileCarousel
-          files={files}
-          totalFiles={files.length}
-          selectedIndex={selectedIndex}
-          onFileSelect={handleFileSelect}
-        />
-      )}
-
-      {selectedFile && <DisplayedFile file={selectedFile} />}
-    </section>
-  );
-};
-
-export default PostFiles;

app/javascript/bundles/Main/components/PostHoverPreviewWrapper.tsx

@@ -12,7 +12,7 @@ interface PostHoverPreviewWrapperProps {
   postPath: string;
   postThumbnailPath: string;
   postThumbnailAlt: string;
-  postDomainIcon?: string;
+  postDomainIcon: string;
   creatorName?: string;
   creatorAvatarPath?: string;
 }
@@ -43,14 +43,14 @@ export const PostHoverPreviewWrapper: React.FC<
         href={postPath}
         className={anchorClassNamesForVisualStyle(visualStyle, true)}
       >
-        {postDomainIcon && (
+        {visualStyle === 'description-section-link' && (
           <img
             src={postDomainIcon}
-            alt={postTitle || postThumbnailAlt}
+            alt={postTitle}
             className={iconClassNamesForSize('small')}
           />
         )}
-        <span className="truncate">{linkText}</span>
+        {linkText}
       </a>
     </PostHoverPreview>
   );

app/javascript/bundles/Main/components/SkySection.tsx

@@ -1,30 +0,0 @@
-import * as React from 'react';
-
-export interface SkySectionProps {
-  title: string;
-  children?: React.ReactNode;
-  contentClassName?: string;
-}
-
-export const SkySection: React.FC<SkySectionProps> = ({
-  title,
-  children,
-  contentClassName,
-}) => {
-  return (
-    <div className="sky-section w-full">
-      <SkySectionHeader title={title} />
-      <div className={contentClassName}>{children}</div>
-    </div>
-  );
-};
-
-export default SkySection;
-
-export const SkySectionHeader: React.FC<SkySectionProps> = ({ title }) => {
-  return (
-    <div className="section-header flex items-center justify-between border-b py-2">
-      <span>{title}</span>
-    </div>
-  );
-};

app/javascript/bundles/Main/components/SortableTable.tsx

@@ -1,167 +0,0 @@
-import * as React from 'react';
-
-export interface TableCell {
-  value: string;
-  sortKey: string | number;
-}
-
-export interface TableData {
-  id: string;
-  cells: TableCell[];
-}
-
-export interface TableHeader {
-  label: string;
-  key: string;
-  align: 'left' | 'right';
-}
-
-interface SortableTableProps {
-  headers: TableHeader[];
-  data: TableData[];
-  defaultSortKey: string;
-  defaultSortOrder: 'asc' | 'desc';
-}
-
-type SortOrder = 'asc' | 'desc';
-
-export const SortableTable: React.FC<SortableTableProps> = ({
-  headers,
-  data,
-  defaultSortKey,
-  defaultSortOrder,
-}) => {
-  const [sortKey, setSortKey] = React.useState<string>(defaultSortKey);
-  const [sortOrder, setSortOrder] = React.useState<SortOrder>(defaultSortOrder);
-
-  const handleSort = (headerKey: string) => {
-    if (sortKey === headerKey) {
-      setSortOrder(sortOrder === 'asc' ? 'desc' : 'asc');
-    } else {
-      setSortKey(headerKey);
-      setSortOrder('desc');
-    }
-  };
-
-  const sortedData = React.useMemo(() => {
-    const headerIndex = headers.findIndex((h) => h.key === sortKey);
-    if (headerIndex === -1) return data;
-
-    return [...data].sort((a, b) => {
-      const aValue = a.cells[headerIndex]?.sortKey;
-      const bValue = b.cells[headerIndex]?.sortKey;
-
-      if (typeof aValue === 'string' && typeof bValue === 'string') {
-        const comparison = aValue.localeCompare(bValue);
-        return sortOrder === 'asc' ? comparison : -comparison;
-      } else if (typeof aValue === 'number' && typeof bValue === 'number') {
-        const comparison = aValue - bValue;
-        return sortOrder === 'asc' ? comparison : -comparison;
-      }
-      return 0;
-    });
-  }, [data, sortKey, sortOrder, headers]);
-
-  const gridStyle: React.CSSProperties = {
-    borderRadius: '0.5rem',
-    border: '1px solid #e2e8f0',
-    backgroundColor: 'white',
-    boxShadow: '0 1px 2px 0 rgb(0 0 0 / 0.05)',
-    overflow: 'hidden',
-  };
-
-  const containerStyle: React.CSSProperties = {
-    display: 'grid',
-    gridTemplateColumns: '1fr auto auto',
-  };
-
-  const headerStyle: React.CSSProperties = {
-    backgroundColor: '#f8fafc',
-    padding: '0.75rem 1rem',
-    fontSize: '0.875rem',
-    fontWeight: 500,
-    color: '#334155',
-    cursor: 'pointer',
-    userSelect: 'none',
-  };
-
-  const cellStyle: React.CSSProperties = {
-    padding: '0.25rem 1rem',
-    borderRight: '1px solid #e2e8f0',
-  };
-
-  const getSortIndicator = (headerKey: string) => {
-    if (sortKey !== headerKey) {
-      return (
-        <span
-          style={{ opacity: 0.5, marginLeft: '0.25rem', fontSize: '0.75rem' }}
-        >
-          ▼
-        </span>
-      );
-    }
-    return (
-      <span style={{ marginLeft: '0.25rem', fontSize: '0.75rem' }}>
-        {sortOrder === 'asc' ? '▲' : '▼'}
-      </span>
-    );
-  };
-
-  return (
-    <div style={gridStyle}>
-      <div style={containerStyle}>
-        {/* Header Row */}
-        <div className="contents">
-          {headers.map((header, index) => (
-            <div
-              key={header.key}
-              style={{
-                ...headerStyle,
-                textAlign: header.align,
-                ...(index === headers.length - 1
-                  ? { borderRight: 'none' }
-                  : {}),
-              }}
-              onClick={() => handleSort(header.key)}
-              onMouseEnter={(e) => {
-                e.currentTarget.style.backgroundColor = '#f1f5f9';
-              }}
-              onMouseLeave={(e) => {
-                e.currentTarget.style.backgroundColor = '#f8fafc';
-              }}
-            >
-              {header.label}
-              {getSortIndicator(header.key)}
-            </div>
-          ))}
-        </div>
-
-        {/* Data Rows */}
-        {sortedData.map((row) => (
-          <div key={row.id} className="group contents">
-            {row.cells.map((cell, cellIndex) => (
-              <div
-                key={cellIndex}
-                style={{
-                  ...cellStyle,
-                  textAlign: headers[cellIndex].align,
-                  fontSize: '0.875rem',
-                  fontWeight: cellIndex === 0 ? 500 : 400,
-                  color: cellIndex === 0 ? '#0f172a' : '#64748b',
-                  ...(cellIndex === row.cells.length - 1
-                    ? { borderRight: 'none' }
-                    : {}),
-                }}
-                className="transition-colors duration-150 group-hover:bg-slate-50"
-              >
-                {cell.value}
-              </div>
-            ))}
-          </div>
-        ))}
-      </div>
-    </div>
-  );
-};
-
-export default SortableTable;

app/javascript/bundles/Main/components/StatsCard.tsx

@@ -1,46 +0,0 @@
-import * as React from 'react';
-
-interface StatsCardProps {
-  requestCount: number;
-  timeWindow: string;
-  requestsPerSecond: string;
-  totalBytes: string;
-  bytesPerSecond: string;
-}
-
-export const StatsCard: React.FC<StatsCardProps> = ({
-  requestCount,
-  timeWindow,
-  requestsPerSecond,
-  totalBytes,
-  bytesPerSecond,
-}) => {
-  const cardStyle: React.CSSProperties = {
-    borderRadius: '0.5rem',
-    border: '1px solid #e2e8f0',
-    backgroundColor: 'white',
-    padding: '1rem',
-    boxShadow: '0 1px 2px 0 rgb(0 0 0 / 0.05)',
-  };
-
-  return (
-    <div className="mt-6" style={cardStyle}>
-      <div className="text-xl font-bold text-slate-900">
-        {requestCount} requests
-        <span className="text-base font-normal text-slate-600">
-          {' '}
-          in last {timeWindow}
-        </span>
-      </div>
-      <div className="mt-1 text-slate-600">
-        <span className="font-medium">{requestsPerSecond}</span> requests/sec
-      </div>
-      <div className="mt-1 text-sm text-slate-600">
-        <span className="font-medium">{totalBytes}</span> transferred •{' '}
-        <span className="font-medium">{bytesPerSecond}</span>/sec
-      </div>
-    </div>
-  );
-};
-
-export default StatsCard;

app/javascript/bundles/Main/components/StatsPage.tsx

@@ -1,223 +0,0 @@
-import * as React from 'react';
-import { StatsCard } from './StatsCard';
-import { SortableTable, TableData } from './SortableTable';
-
-interface StatsPageProps {
-  timeWindow: number; // in seconds
-  lastWindowCount: number;
-  lastWindowBytes: number;
-  requestsPerSecond: string;
-  totalBytesFormatted: string;
-  bytesPerSecondFormatted: string;
-  timeWindowFormatted: string;
-  contentTypeCounts: Array<{
-    content_type: string;
-    count: number;
-    bytes: number;
-    countFormatted: string;
-    bytesFormatted: string;
-  }>;
-  byDomainCounts: Array<{
-    domain: string;
-    count: number;
-    bytes: number;
-    countFormatted: string;
-    bytesFormatted: string;
-  }>;
-  availableTimeWindows: Array<{
-    seconds: number;
-    label: string;
-    active: boolean;
-    path: string;
-  }>;
-}
-
-export const StatsPage: React.FC<StatsPageProps> = ({
-  lastWindowCount,
-  requestsPerSecond,
-  totalBytesFormatted,
-  bytesPerSecondFormatted,
-  timeWindowFormatted,
-  contentTypeCounts,
-  byDomainCounts,
-  availableTimeWindows,
-}) => {
-  const contentTypeData: TableData[] = contentTypeCounts.map((item) => ({
-    id: item.content_type,
-    cells: [
-      { value: item.content_type, sortKey: item.content_type },
-      { value: item.countFormatted, sortKey: item.count },
-      { value: item.bytesFormatted, sortKey: item.bytes },
-    ],
-  }));
-
-  const domainData: TableData[] = byDomainCounts.map((item) => ({
-    id: item.domain,
-    cells: [
-      { value: item.domain, sortKey: item.domain },
-      { value: item.countFormatted, sortKey: item.count },
-      { value: item.bytesFormatted, sortKey: item.bytes },
-    ],
-  }));
-
-  return (
-    <div className="mx-auto mt-8 max-w-7xl px-4">
-      {/* Header Section */}
-      <div className="overflow-hidden rounded-xl border border-slate-200 bg-white shadow-sm">
-        {/* Top Bar */}
-        <div className="border-b border-slate-200 bg-slate-50 px-6 py-4">
-          <div className="flex items-center justify-between">
-            <h1 className="text-2xl font-bold text-slate-900">
-              HTTP Request Analytics
-            </h1>
-            <a
-              href="/log_entries"
-              className="inline-flex items-center gap-2 rounded-lg border border-slate-300 px-4 py-2 text-sm font-medium text-slate-600 transition-colors hover:bg-white hover:text-slate-900 hover:shadow-sm"
-            >
-              <i className="fas fa-arrow-left" />
-              Back to Log Entries
-            </a>
-          </div>
-        </div>
-
-        {/* Stats Summary */}
-        <div className="border-t border-slate-200 bg-gradient-to-br from-blue-50 to-indigo-50 px-6 py-6">
-          <div className="mb-4 text-center">
-            <h3 className="mb-1 text-lg font-semibold text-slate-900">
-              Summary for {timeWindowFormatted}
-            </h3>
-          </div>
-
-          <div className="grid grid-cols-1 gap-4 sm:grid-cols-2 lg:grid-cols-4">
-            {/* Total Requests */}
-            <div className="rounded-lg border border-slate-200 bg-white p-4 shadow-sm">
-              <div className="flex items-center justify-between">
-                <div>
-                  <p className="text-xs font-medium uppercase tracking-wide text-slate-500">
-                    Total Requests
-                  </p>
-                  <p className="mt-1 text-2xl font-bold text-slate-900">
-                    {lastWindowCount.toLocaleString()}
-                  </p>
-                </div>
-                <div className="flex h-10 w-10 items-center justify-center rounded-lg bg-blue-100">
-                  <i className="fas fa-chart-bar text-blue-600" />
-                </div>
-              </div>
-            </div>
-
-            {/* Requests per Second */}
-            <div className="rounded-lg border border-slate-200 bg-white p-4 shadow-sm">
-              <div className="flex items-center justify-between">
-                <div>
-                  <p className="text-xs font-medium uppercase tracking-wide text-slate-500">
-                    Requests/sec
-                  </p>
-                  <p className="mt-1 text-2xl font-bold text-slate-900">
-                    {requestsPerSecond}
-                  </p>
-                </div>
-                <div className="flex h-10 w-10 items-center justify-center rounded-lg bg-green-100">
-                  <i className="fas fa-bolt text-green-600" />
-                </div>
-              </div>
-            </div>
-
-            {/* Total Data */}
-            <div className="rounded-lg border border-slate-200 bg-white p-4 shadow-sm">
-              <div className="flex items-center justify-between">
-                <div>
-                  <p className="text-xs font-medium uppercase tracking-wide text-slate-500">
-                    Total Data
-                  </p>
-                  <p className="mt-1 text-2xl font-bold text-slate-900">
-                    {totalBytesFormatted}
-                  </p>
-                </div>
-                <div className="flex h-10 w-10 items-center justify-center rounded-lg bg-purple-100">
-                  <i className="fas fa-database text-purple-600" />
-                </div>
-              </div>
-            </div>
-
-            {/* Data per Second */}
-            <div className="rounded-lg border border-slate-200 bg-white p-4 shadow-sm">
-              <div className="flex items-center justify-between">
-                <div>
-                  <p className="text-xs font-medium uppercase tracking-wide text-slate-500">
-                    Data/sec
-                  </p>
-                  <p className="mt-1 text-2xl font-bold text-slate-900">
-                    {bytesPerSecondFormatted}
-                  </p>
-                </div>
-                <div className="flex h-10 w-10 items-center justify-center rounded-lg bg-orange-100">
-                  <i className="fas fa-download text-orange-600" />
-                </div>
-              </div>
-            </div>
-          </div>
-        </div>
-
-        {/* Time Window Selector */}
-        <div className="bg-gradient-to-br from-blue-50 to-indigo-50 px-4 py-4">
-          <div className="text-center">
-            <div className="inline-flex flex-wrap justify-center gap-1 rounded-lg border border-white/50 bg-white/70 p-1 shadow-md backdrop-blur-sm">
-              {availableTimeWindows.map((timeWindowOption, index) => (
-                <React.Fragment key={timeWindowOption.seconds}>
-                  {timeWindowOption.active ? (
-                    <span className="rounded-md bg-blue-600 px-4 py-2 text-sm font-semibold text-white shadow-sm">
-                      {timeWindowOption.label}
-                    </span>
-                  ) : (
-                    <a
-                      href={timeWindowOption.path}
-                      className="rounded-md border border-transparent px-4 py-2 text-sm font-medium text-slate-600 transition-colors hover:border-slate-200 hover:bg-white hover:text-slate-900 hover:shadow-sm"
-                    >
-                      {timeWindowOption.label}
-                    </a>
-                  )}
-                </React.Fragment>
-              ))}
-            </div>
-          </div>
-        </div>
-      </div>
-
-      {/* Tables Grid - 2 columns */}
-      <div className="my-8 grid grid-cols-1 gap-8 lg:grid-cols-2">
-        <div>
-          <h2 className="mb-3 text-xl font-bold text-slate-900">
-            By Content Type
-          </h2>
-          <SortableTable
-            headers={[
-              { label: 'Content Type', key: 'content_type', align: 'left' },
-              { label: 'Requests', key: 'count', align: 'right' },
-              { label: 'Transferred', key: 'bytes', align: 'right' },
-            ]}
-            data={contentTypeData}
-            defaultSortKey="count"
-            defaultSortOrder="desc"
-          />
-        </div>
-
-        <div>
-          <h2 className="mb-3 text-xl font-bold text-slate-900">By Domain</h2>
-          <SortableTable
-            headers={[
-              { label: 'Domain', key: 'domain', align: 'left' },
-              { label: 'Requests', key: 'count', align: 'right' },
-              { label: 'Transferred', key: 'bytes', align: 'right' },
-            ]}
-            data={domainData}
-            defaultSortKey="bytes"
-            defaultSortOrder="desc"
-          />
-        </div>
-      </div>
-    </div>
-  );
-};
-
-export default StatsPage;

app/javascript/bundles/Main/components/TrackedObjectsChart.tsx

@@ -1,226 +0,0 @@
-import * as React from 'react';
-import { useEffect, useRef, useState } from 'react';
-import {
-  calculateTrendLines,
-  calculateChartStatistics,
-  type ChartStatistics,
-} from '../utils/chartStatistics';
-import { PolynomialEquation } from './PolynomialEquation';
-
-declare global {
-  interface Window {
-    Chart: any;
-  }
-}
-
-interface TrackedObjectData {
-  x: number;
-  y: number;
-  scanDate: string;
-  durationFormatted: string;
-}
-
-interface TrackedObjectsChartProps {
-  data: TrackedObjectData[];
-  objectKind: string;
-}
-
-export const TrackedObjectsChart: React.FC<TrackedObjectsChartProps> = ({
-  data,
-  objectKind,
-}) => {
-  const canvasRef = useRef<HTMLCanvasElement>(null);
-  const chartRef = useRef<any>(null);
-  const [statistics, setStatistics] = useState<ChartStatistics | null>(null);
-  const [isInitialized, setIsInitialized] = useState(false);
-
-  useEffect(() => {
-    if (!canvasRef.current || data.length === 0 || !window.Chart) {
-      return;
-    }
-
-    // Add a small delay to ensure DOM is stable
-    const initTimer = setTimeout(() => {
-      if (!canvasRef.current) return;
-
-      // Calculate trend lines and statistics
-      const trendLines = calculateTrendLines(data);
-      const chartStatistics = calculateChartStatistics(data);
-      setStatistics(chartStatistics);
-
-      // Create trend line datasets for all polynomial fits
-      const minX = Math.min(...data.map((p) => p.x));
-      const maxX = Math.max(...data.map((p) => p.x));
-      const numPoints = 50; // More points for smooth curve
-
-      const trendLineDatasets = trendLines.map((trendLine) => {
-        const trendLineData = [];
-
-        for (let i = 0; i <= numPoints; i++) {
-          const x = minX + (maxX - minX) * (i / numPoints);
-          // Calculate y using polynomial: y = c₀ + c₁x + c₂x² + ...
-          let y = 0;
-          for (let j = 0; j < trendLine.coeffs.length; j++) {
-            y += trendLine.coeffs[j] * Math.pow(x, j);
-          }
-          trendLineData.push({ x, y });
-        }
-
-        return {
-          label: `${trendLine.degree === 1 ? 'Linear' : 'Quadratic'} Trend (R² = ${trendLine.rSquared.toFixed(3)})`,
-          data: trendLineData,
-          type: 'line',
-          borderColor: trendLine.color,
-          backgroundColor: trendLine.color.replace('1)', '0.1)'),
-          borderWidth: 2,
-          pointRadius: 2,
-          pointHoverRadius: 2,
-          pointBackgroundColor: 'transparent',
-          pointBorderColor: 'transparent',
-          pointHoverBackgroundColor: trendLine.color,
-          pointHoverBorderColor: trendLine.color,
-          fill: false,
-          tension: 0,
-        };
-      });
-
-      const ctx = canvasRef.current.getContext('2d');
-
-      // Destroy existing chart if it exists
-      if (chartRef.current) {
-        chartRef.current.destroy();
-      }
-
-      chartRef.current = new window.Chart(ctx, {
-        type: 'scatter',
-        data: {
-          datasets: [
-            {
-              label: `${objectKind} Added`,
-              data: data,
-              backgroundColor: 'rgba(59, 130, 246, 0.6)',
-              borderColor: 'rgba(59, 130, 246, 1)',
-              borderWidth: 1,
-              pointRadius: 5,
-              pointHoverRadius: 7,
-            },
-            ...trendLineDatasets,
-          ],
-        },
-        options: {
-          responsive: true,
-          maintainAspectRatio: false,
-          animation: {
-            duration: 0, // Disable animations to prevent jitter
-          },
-          scales: {
-            x: {
-              title: {
-                display: true,
-                text: `${objectKind} Added`,
-              },
-              min: 0,
-            },
-            y: {
-              title: {
-                display: true,
-                text: 'Days Since Last Scan',
-              },
-              min: 0,
-            },
-          },
-          plugins: {
-            tooltip: {
-              intersect: false,
-              mode: 'nearest',
-              callbacks: {
-                title: function (context: any) {
-                  const dataPoint = context[0];
-                  if (dataPoint.dataset.label?.includes('Trend')) {
-                    return `Predict ${context[0].parsed.x.toFixed(0)} ${objectKind.toLowerCase()} added`;
-                  }
-                  return data[dataPoint.dataIndex]?.scanDate || '';
-                },
-                label: function (context: any) {
-                  if (context.dataset.label?.includes('Trend')) {
-                    return `${context.parsed.y.toFixed(1)} days`;
-                  }
-                  return `${context.parsed.x} ${objectKind.toLowerCase()} added`;
-                },
-                afterLabel: function (context: any) {
-                  if (!context.dataset.label?.includes('Trend')) {
-                    const dataPoint = data[context.dataIndex];
-                    return `Time gap: ${dataPoint?.durationFormatted || 'N/A'}`;
-                  }
-                  return '';
-                },
-              },
-            },
-            legend: {
-              display: true,
-              position: 'top' as const,
-            },
-          },
-        },
-      });
-
-      setIsInitialized(true);
-    }, 100); // 100ms delay
-
-    return () => {
-      clearTimeout(initTimer);
-      if (chartRef.current) {
-        chartRef.current.destroy();
-      }
-    };
-  }, [data, objectKind]);
-
-  return (
-    <section className="rounded-md border border-slate-300 bg-white p-2">
-      <div className="h-96 w-full">
-        <canvas ref={canvasRef} />
-      </div>
-      {statistics && (
-        <div className="mt-2 space-y-2 text-center text-sm text-slate-600">
-          <div className="flex justify-center gap-6">
-            <div>
-              <span className="font-medium">{objectKind} per day: </span>
-              <span className="font-mono">
-                {statistics.favsPerDay > 0
-                  ? statistics.favsPerDay.toFixed(2)
-                  : 'N/A'}
-              </span>
-            </div>
-          </div>
-          {statistics.trendLines.length > 0 ? (
-            <div className="space-y-1">
-              {statistics.trendLines.map((trendLine, index) => (
-                <div key={index} className="flex justify-center gap-4 text-xs">
-                  <div>
-                    <span
-                      className="font-medium"
-                      style={{ color: trendLine.color }}
-                    >
-                      {trendLine.degree === 1 ? 'Linear' : 'Quadratic'} R²:
-                    </span>
-                    <span className="ml-1 font-mono">
-                      {trendLine.rSquared.toFixed(3)}
-                    </span>
-                  </div>
-                  <PolynomialEquation
-                    coefficients={trendLine.coeffs}
-                    className="font-mono text-slate-500"
-                  />
-                </div>
-              ))}
-            </div>
-          ) : (
-            <div className="text-xs text-slate-500">
-              Need at least 2 data points for trend analysis
-            </div>
-          )}
-        </div>
-      )}
-    </section>
-  );
-};

app/javascript/bundles/Main/components/UserMenu.tsx

@@ -0,0 +1,144 @@
+import * as React from 'react';
+import { useRef, useEffect, useState } from 'react';
+
+interface UserMenuProps {
+  userEmail: string;
+  userRole?: 'admin' | 'moderator';
+  editProfilePath: string;
+  signOutPath: string;
+  csrfToken: string;
+  globalStatesPath: string;
+  goodJobPath: string;
+  grafanaPath: string;
+  prometheusPath: string;
+}
+
+export const UserMenu: React.FC<UserMenuProps> = ({
+  userEmail,
+  userRole,
+  editProfilePath,
+  signOutPath,
+  csrfToken,
+  globalStatesPath,
+  goodJobPath,
+  grafanaPath,
+  prometheusPath,
+}) => {
+  const [isOpen, setIsOpen] = useState(false);
+  const menuRef = useRef<HTMLDivElement>(null);
+
+  useEffect(() => {
+    const handleClickOutside = (event: MouseEvent) => {
+      if (menuRef.current && !menuRef.current.contains(event.target as Node)) {
+        setIsOpen(false);
+      }
+    };
+
+    document.addEventListener('mousedown', handleClickOutside);
+    return () => document.removeEventListener('mousedown', handleClickOutside);
+  }, []);
+
+  const handleSignOut = (e: React.FormEvent) => {
+    e.preventDefault();
+    const form = document.createElement('form');
+    form.method = 'POST';
+    form.action = signOutPath;
+    form.style.display = 'none';
+
+    const methodInput = document.createElement('input');
+    methodInput.type = 'hidden';
+    methodInput.name = '_method';
+    methodInput.value = 'delete';
+
+    const csrfInput = document.createElement('input');
+    csrfInput.type = 'hidden';
+    csrfInput.name = 'authenticity_token';
+    csrfInput.value = csrfToken;
+
+    form.appendChild(methodInput);
+    form.appendChild(csrfInput);
+    document.body.appendChild(form);
+    form.submit();
+  };
+
+  return (
+    <div className="relative" ref={menuRef}>
+      <button
+        className="flex items-center space-x-2 text-slate-600 hover:text-slate-900 focus:outline-none"
+        onClick={() => setIsOpen(!isOpen)}
+      >
+        <i className="fas fa-user-circle text-2xl" />
+        <i className="fas fa-chevron-down text-xs" />
+      </button>
+
+      <div
+        className={`absolute right-0 z-50 mt-2 w-48 rounded-md bg-white py-1 shadow-lg ring-1 ring-black ring-opacity-5 transition-all duration-200 ${
+          isOpen ? 'visible opacity-100' : 'invisible opacity-0'
+        }`}
+      >
+        <div className="border-b border-slate-200 px-4 py-2 text-sm text-slate-700">
+          <div className="font-medium">{userEmail}</div>
+          {userRole === 'admin' && (
+            <span className="inline-flex items-center rounded-full bg-red-100 px-2 py-0.5 text-xs font-medium text-red-800">
+              Admin
+            </span>
+          )}
+          {userRole === 'moderator' && (
+            <span className="inline-flex items-center rounded-full bg-blue-100 px-2 py-0.5 text-xs font-medium text-blue-800">
+              Mod
+            </span>
+          )}
+        </div>
+
+        {userRole === 'admin' && (
+          <>
+            <a
+              href={globalStatesPath}
+              className="flex w-full items-center px-4 py-2 text-sm text-slate-700 hover:bg-slate-100"
+            >
+              <i className="fas fa-cogs mr-2 w-5" />
+              <span>Global State</span>
+            </a>
+            <a
+              href={goodJobPath}
+              className="flex w-full items-center px-4 py-2 text-sm text-slate-700 hover:bg-slate-100"
+            >
+              <i className="fas fa-tasks mr-2 w-5" />
+              <span>Jobs Queue</span>
+            </a>
+            <a
+              href={grafanaPath}
+              className="flex w-full items-center px-4 py-2 text-sm text-slate-700 hover:bg-slate-100"
+            >
+              <i className="fas fa-chart-line mr-2 w-5" />
+              <span>Grafana</span>
+            </a>
+            <a
+              href={prometheusPath}
+              className="flex w-full items-center px-4 py-2 text-sm text-slate-700 hover:bg-slate-100"
+            >
+              <i className="fas fa-chart-bar mr-2 w-5" />
+              <span>Prometheus</span>
+            </a>
+          </>
+        )}
+
+        <a
+          href={editProfilePath}
+          className="flex w-full items-center px-4 py-2 text-sm text-slate-700 hover:bg-slate-100"
+        >
+          <i className="fas fa-cog mr-2 w-5" />
+          <span>Edit Profile</span>
+        </a>
+
+        <button
+          onClick={handleSignOut}
+          className="flex w-full items-center px-4 py-2 text-left text-sm text-slate-700 hover:bg-slate-100"
+        >
+          <i className="fas fa-sign-out-alt mr-2 w-5" />
+          <span>Sign Out</span>
+        </button>
+      </div>
+    </div>
+  );
+};

app/javascript/bundles/Main/components/UserSearchBar.tsx

@@ -3,6 +3,7 @@ import * as React from 'react';
 import { useCallback, useRef, useState } from 'react';
 import Icon from './Icon';
 import ListItem from './ListItem';
+import Trie, { TrieNode } from '../lib/Trie';
 
 // 1. Group related constants
 const CONFIG = {
@@ -17,7 +18,7 @@ const STYLES = {
   ],
   SVG_BASE_CLASSNAME: `stroke-slate-500 fill-slate-500`,
   SVG_FOCUSABLE_CLASSNAME: `stroke-slate-500 fill-slate-500 group-focus-within:stroke-slate-800 group-focus-within:fill-slate-800`,
-  INPUT_CLASSNAME: `text-slate-500 focus:text-slate-800 placeholder-slate-500 group-focus-within:placeholder-slate-800 placeholder:font-extralight`,
+  INPUT_CLASSNAME: `text-slate-500 group-focus-within:text-slate-800 placeholder-slate-500 group-focus-within:placeholder-slate-800 placeholder:font-extralight`,
 } as const;
 
 // 2. Simplify logging
@@ -33,7 +34,6 @@ interface PropTypes {
 interface User {
   id: number;
   name: string;
-  name_secondary?: string;
   thumb?: string;
   show_path: string;
   num_posts?: number;
@@ -46,6 +46,10 @@ interface ServerResponse {
   users: User[];
 }
 
+type TrieValue = [number, string];
+type TrieType = Trie<TrieValue>;
+type TrieNodeType = TrieNode<TrieValue>;
+
 export default function UserSearchBar({ isServerRendered }: PropTypes) {
   isServerRendered = !!isServerRendered;
   const [pendingRequest, setPendingRequest] = useState<AbortController | null>(
@@ -115,6 +119,7 @@ export default function UserSearchBar({ isServerRendered }: PropTypes) {
           }
         } catch (err) {
           if (!err.message.includes('aborted')) {
+            log.error('error loading user trie: ', err);
             setState((s) => ({
               ...s,
               errorMessage: `error loading users: ` + err.message,
@@ -143,6 +148,7 @@ export default function UserSearchBar({ isServerRendered }: PropTypes) {
 
   const searchForUserDebounced = useCallback(
     debounce(async (userName) => {
+      log.info('sending search for ', userName);
       setState((s) => ({ ...s, typingSettled: true }));
       searchForUser(userName);
     }, 250),
@@ -152,6 +158,7 @@ export default function UserSearchBar({ isServerRendered }: PropTypes) {
   function invokeIdx(idx) {
     const user = state.userList[idx];
     if (user) {
+      log.info('selecting user: ', user);
       setState((s) => ({ ...s, userName: user.name }));
       inputRef.current.value = user.name;
       window.location.href = user.show_path;
@@ -178,7 +185,9 @@ export default function UserSearchBar({ isServerRendered }: PropTypes) {
 
   function UserSearchBarItems() {
     return (
-      <div className="divide-y divide-slate-300 overflow-hidden border border-slate-300 bg-slate-50 shadow-lg sm:rounded-xl">
+      <div
+        className={`${anyShown || 'border-b-0'} divide-y divide-inherit rounded-b-lg border border-t-0 border-inherit`}
+      >
         {visibility.error ? (
           <ListItem
             key="error"
@@ -199,24 +208,13 @@ export default function UserSearchBar({ isServerRendered }: PropTypes) {
         ) : null}
         {visibility.items
           ? state.userList.map(
-              (
-                {
-                  name,
-                  name_secondary,
-                  thumb,
-                  show_path,
-                  num_posts,
-                  domain_icon,
-                },
-                idx,
-              ) => (
+              ({ name, thumb, show_path, num_posts, domain_icon }, idx) => (
                 <ListItem
                   key={'name-' + name}
                   isLast={idx == state.userList.length - 1}
                   selected={idx == state.selectedIdx}
                   style="item"
                   value={name}
-                  subvalue={name_secondary}
                   thumb={thumb}
                   href={show_path}
                   subtext={num_posts ? `${num_posts.toString()} posts` : ''}
@@ -269,7 +267,7 @@ export default function UserSearchBar({ isServerRendered }: PropTypes) {
   return (
     <div
       className={[
-        'relative mx-auto w-full p-2 transition-colors duration-1000 sm:rounded-xl',
+        'group mx-auto w-full p-2 transition-colors duration-1000 sm:rounded-xl',
         'focus-within:border-slate-400 sm:max-w-md',
         'border-slate-300 bg-slate-50 p-2 shadow-lg',
       ].join(' ')}
@@ -292,6 +290,7 @@ export default function UserSearchBar({ isServerRendered }: PropTypes) {
             STYLES.INPUT_CLASSNAME,
             'rounded-lg outline-none',
             'bg-slate-50 placeholder:italic',
+            anyShown && 'rounded-b-none',
           ]
             .filter(Boolean)
             .join(' ')}
@@ -307,11 +306,7 @@ export default function UserSearchBar({ isServerRendered }: PropTypes) {
           ref={inputRef}
         />
       </label>
-      {anyShown && (
-        <div className="absolute left-0 right-0 top-full mt-1">
-          <UserSearchBarItems />
-        </div>
-      )}
+      <UserSearchBarItems />
     </div>
   );
 }

app/javascript/bundles/Main/components/VisualSearchForm.tsx

@@ -1,858 +0,0 @@
-import * as React from 'react';
-import { useCallback, useEffect, useRef, useState } from 'react';
-
-interface VisualSearchFormProps {
-  actionUrl: string;
-  csrfToken: string;
-}
-
-interface FeedbackMessage {
-  text: string;
-  type: 'success' | 'error' | 'warning';
-}
-
-interface ImageState {
-  file: File;
-  previewUrl: string;
-  originalFileSize: number | null;
-  thumbnailFile?: File; // For video thumbnails
-  isVideo?: boolean;
-}
-
-const ACCEPTED_IMAGE_TYPES = [
-  'image/png',
-  'image/jpeg',
-  'image/jpg',
-  'image/gif',
-  'image/webp',
-  'video/mp4',
-];
-
-const ACCEPTED_EXTENSIONS =
-  'image/png,image/jpeg,image/jpg,image/gif,image/webp,video/mp4';
-
-// Feedback Message Component
-interface FeedbackMessageProps {
-  message: FeedbackMessage;
-}
-
-function FeedbackMessageDisplay({ message }: FeedbackMessageProps) {
-  const getClassName = (type: FeedbackMessage['type']) => {
-    switch (type) {
-      case 'success':
-        return 'text-green-600';
-      case 'error':
-        return 'text-red-600';
-      case 'warning':
-        return 'text-amber-600';
-      default:
-        return 'text-slate-600';
-    }
-  };
-
-  return (
-    <p className={`text-sm ${getClassName(message.type)} mt-2`}>
-      {message.text}
-    </p>
-  );
-}
-
-// Image Preview Component
-interface ImagePreviewProps {
-  imageState: ImageState;
-  onRemove: () => void;
-}
-
-// Helper function to format file size
-function formatFileSize(bytes: number): string {
-  if (bytes === 0) return '0 Bytes';
-
-  const k = 1024;
-  const sizes = ['Bytes', 'KB', 'MB', 'GB'];
-  const i = Math.floor(Math.log(bytes) / Math.log(k));
-
-  return parseFloat((bytes / Math.pow(k, i)).toFixed(1)) + ' ' + sizes[i];
-}
-
-// Helper function to resize images larger than 2MB
-async function resizeImageIfNeeded(file: File): Promise<File> {
-  const MAX_SIZE_MB = 2;
-  const MAX_SIZE_BYTES = MAX_SIZE_MB * 1024 * 1024;
-
-  if (file.size <= MAX_SIZE_BYTES) {
-    return file;
-  }
-
-  return new Promise((resolve) => {
-    const canvas = document.createElement('canvas');
-    const ctx = canvas.getContext('2d')!;
-    const img = new Image();
-
-    img.onload = () => {
-      // Calculate compression ratio based on file size
-      const compressionRatio = Math.sqrt(MAX_SIZE_BYTES / file.size);
-
-      // Calculate new dimensions maintaining aspect ratio
-      const newWidth = Math.floor(img.width * compressionRatio);
-      const newHeight = Math.floor(img.height * compressionRatio);
-
-      // Set canvas dimensions
-      canvas.width = newWidth;
-      canvas.height = newHeight;
-
-      // Draw resized image
-      ctx.drawImage(img, 0, 0, newWidth, newHeight);
-
-      // Convert to blob with quality adjustment
-      canvas.toBlob(
-        (blob) => {
-          if (blob) {
-            const resizedFile = new File([blob], file.name, {
-              type: file.type,
-              lastModified: Date.now(),
-            });
-            resolve(resizedFile);
-          } else {
-            resolve(file); // Fallback to original if resize fails
-          }
-        },
-        file.type,
-        0.85, // Quality setting (85%)
-      );
-    };
-
-    img.onerror = () => {
-      resolve(file); // Fallback to original if image load fails
-    };
-
-    img.src = URL.createObjectURL(file);
-  });
-}
-
-// Helper function to generate thumbnail from video file
-async function generateVideoThumbnail(file: File): Promise<File> {
-  return new Promise((resolve) => {
-    const video = document.createElement('video');
-    const canvas = document.createElement('canvas');
-    const ctx = canvas.getContext('2d')!;
-
-    video.onloadedmetadata = () => {
-      // Set canvas dimensions to match video
-      canvas.width = video.videoWidth;
-      canvas.height = video.videoHeight;
-
-      // Seek to 1 second into the video (or 10% through if shorter)
-      const seekTime = Math.min(1, video.duration * 0.1);
-      video.currentTime = seekTime;
-    };
-
-    video.onseeked = () => {
-      // Draw the current frame to canvas
-      ctx.drawImage(video, 0, 0, canvas.width, canvas.height);
-
-      // Convert to blob as JPEG
-      canvas.toBlob(
-        (blob) => {
-          if (blob) {
-            // Create a new file with thumbnail data but keep original video file name base
-            const thumbnailName =
-              file.name.replace(/\.[^/.]+$/, '') + '_thumbnail.jpg';
-            const thumbnailFile = new File([blob], thumbnailName, {
-              type: 'image/jpeg',
-              lastModified: Date.now(),
-            });
-            resolve(thumbnailFile);
-          } else {
-            resolve(file); // Fallback to original file
-          }
-        },
-        'image/jpeg',
-        0.8, // Quality setting (80%)
-      );
-    };
-
-    video.onerror = () => {
-      resolve(file); // Fallback to original file if video processing fails
-    };
-
-    // Load the video file
-    video.src = URL.createObjectURL(file);
-    video.load();
-  });
-}
-
-function ImagePreview({ imageState, onRemove }: ImagePreviewProps) {
-  const isVideo = imageState.isVideo;
-
-  return (
-    <div className="flex items-center gap-4">
-      <div className="relative max-h-32 max-w-32 flex-shrink-0">
-        <img
-          src={imageState.previewUrl}
-          alt={isVideo ? 'Video thumbnail' : 'Selected image thumbnail'}
-          className="max-h-32 max-w-32 rounded-md object-cover shadow-sm"
-        />
-        {isVideo && (
-          <div className="absolute inset-0 flex items-center justify-center">
-            <div className="rounded-full bg-black bg-opacity-70 p-2">
-              <svg
-                className="h-4 w-4 text-white"
-                fill="currentColor"
-                viewBox="0 0 20 20"
-              >
-                <path d="M6.3 2.841A1.5 1.5 0 004 4.11V15.89a1.5 1.5 0 002.3 1.269l9.344-5.89a1.5 1.5 0 000-2.538L6.3 2.84z" />
-              </svg>
-            </div>
-          </div>
-        )}
-      </div>
-      <div className="flex min-w-0 flex-1 flex-col justify-center gap-1">
-        <h3 className="text-sm font-medium text-green-700">
-          {isVideo ? 'Selected Video' : 'Selected Image'}
-        </h3>
-        <p
-          className="max-w-32 truncate text-xs text-green-600"
-          title={imageState.file.name}
-        >
-          {imageState.file.name}
-        </p>
-        {isVideo ? (
-          <p className="text-xs text-slate-500">
-            {formatFileSize(imageState.file.size)} (thumbnail generated)
-          </p>
-        ) : imageState.originalFileSize ? (
-          <div className="text-xs text-slate-500">
-            <div>Original: {formatFileSize(imageState.originalFileSize)}</div>
-            <div>Resized: {formatFileSize(imageState.file.size)}</div>
-          </div>
-        ) : (
-          <p className="text-xs text-slate-500">
-            {formatFileSize(imageState.file.size)}
-          </p>
-        )}
-        <button
-          type="button"
-          onClick={(e) => {
-            e.stopPropagation();
-            onRemove();
-          }}
-          className="mt-1 self-start rounded bg-slate-600 px-2 py-1 text-xs font-medium text-slate-100 transition-colors hover:bg-red-600 focus:bg-red-600 focus:outline-none"
-          title={isVideo ? 'Clear video' : 'Clear image'}
-        >
-          {isVideo ? 'Remove Video' : 'Remove Image'}
-        </button>
-      </div>
-    </div>
-  );
-}
-
-// Empty Drop Zone Component
-interface EmptyDropZoneProps {
-  isMobile: boolean;
-}
-
-function EmptyDropZone({ isMobile }: EmptyDropZoneProps) {
-  return (
-    <div className="m-4 flex flex-col items-center justify-center gap-2">
-      <svg
-        xmlns="http://www.w3.org/2000/svg"
-        className="h-10 w-10 text-slate-400"
-        fill="none"
-        viewBox="0 0 24 24"
-        stroke="currentColor"
-      >
-        <path
-          strokeLinecap="round"
-          strokeLinejoin="round"
-          strokeWidth="2"
-          d="M4 16l4.586-4.586a2 2 0 012.828 0L16 16m-2-2l1.586-1.586a2 2 0 012.828 0L20 14m-6-6h.01M6 20h12a2 2 0 002-2V6a2 2 0 00-2-2H6a2 2 0 00-2 2v12a2 2 0 002 2z"
-        />
-      </svg>
-      <p className="hidden font-medium text-slate-600 sm:block">
-        Drag and drop an image or video here
-      </p>
-      <p className="block font-medium text-slate-600 sm:hidden">
-        tap here to paste an image or video from the clipboard
-      </p>
-      <p className="text-xs text-slate-500">or use one of the options below</p>
-    </div>
-  );
-}
-
-// Image Drop Zone Component
-interface ImageDropZoneProps {
-  imageState: ImageState | null;
-  isDragOver: boolean;
-  isMobile: boolean;
-  feedbackMessage: FeedbackMessage | null;
-  onClearImage: () => void;
-  onDragEnter: (e: React.DragEvent) => void;
-  onDragLeave: (e: React.DragEvent) => void;
-  onDragOver: (e: React.DragEvent) => void;
-  onDrop: (e: React.DragEvent) => void;
-  onClick: () => void;
-  onKeyDown: (e: React.KeyboardEvent) => void;
-  onPaste: (e: ClipboardEvent) => void | Promise<void>;
-  pasteInputRef: React.RefObject<HTMLInputElement>;
-  dropZoneRef: React.RefObject<HTMLDivElement>;
-}
-
-function ImageDropZone({
-  imageState,
-  isDragOver,
-  isMobile,
-  feedbackMessage,
-  onClearImage,
-  onDragEnter,
-  onDragLeave,
-  onDragOver,
-  onDrop,
-  onClick,
-  onKeyDown,
-  onPaste,
-  pasteInputRef,
-  dropZoneRef,
-}: ImageDropZoneProps) {
-  return (
-    <div
-      ref={dropZoneRef}
-      onClick={onClick}
-      onDragEnter={onDragEnter}
-      onDragLeave={onDragLeave}
-      onDragOver={onDragOver}
-      onDrop={onDrop}
-      onKeyDown={onKeyDown}
-      className={`relative mb-4 rounded-lg border-2 border-dashed p-2 text-center transition-colors duration-200 focus:border-blue-500 ${
-        isDragOver ? 'border-blue-500 bg-blue-50' : 'border-slate-300'
-      }`}
-      tabIndex={0}
-    >
-      <input
-        ref={pasteInputRef}
-        type="text"
-        className="pointer-events-none absolute opacity-0"
-        style={{ left: '-9999px' }}
-        autoComplete="off"
-        onPaste={async (e) => await onPaste(e.nativeEvent)}
-        onContextMenu={(e) => isMobile && e.stopPropagation()}
-      />
-
-      <div className="flex flex-col items-center justify-center gap-2">
-        {imageState ? (
-          <ImagePreview imageState={imageState} onRemove={onClearImage} />
-        ) : (
-          <EmptyDropZone isMobile={isMobile} />
-        )}
-      </div>
-
-      {feedbackMessage && <FeedbackMessageDisplay message={feedbackMessage} />}
-    </div>
-  );
-}
-
-// File Upload Section Component
-interface FileUploadSectionProps {
-  fileInputRef: React.RefObject<HTMLInputElement>;
-  onFileChange: (file: File | null) => Promise<void>;
-}
-
-function FileUploadSection({
-  fileInputRef,
-  onFileChange,
-}: FileUploadSectionProps) {
-  return (
-    <div className="flex flex-1 flex-col items-center justify-center">
-      <h3 className="text-lg font-medium text-slate-500">
-        Upload an image or video
-      </h3>
-      <div className="flex flex-col gap-1">
-        <label
-          htmlFor="image-file-input"
-          className="text-sm font-medium text-slate-700"
-        >
-          Choose an image or video file
-        </label>
-        <input
-          ref={fileInputRef}
-          id="image-file-input"
-          name="image_file"
-          type="file"
-          accept={ACCEPTED_EXTENSIONS}
-          className="w-full rounded-md border-slate-300 text-sm"
-          onChange={async (e) => {
-            const file = e.target.files?.[0];
-            await onFileChange(file || null);
-          }}
-        />
-        <p className="text-xs text-slate-500">
-          Supported formats: JPG, PNG, GIF, WebP, MP4
-        </p>
-      </div>
-    </div>
-  );
-}
-
-// URL Upload Section Component
-interface UrlUploadSectionProps {
-  imageUrl: string;
-  onUrlChange: (url: string) => void;
-}
-
-function UrlUploadSection({ imageUrl, onUrlChange }: UrlUploadSectionProps) {
-  return (
-    <div className="flex flex-1 flex-col items-center justify-center">
-      <h3 className="text-lg font-medium text-slate-500">
-        Provide image or video URL
-      </h3>
-      <div className="flex flex-col gap-1">
-        <label
-          htmlFor="image-url-input"
-          className="text-sm font-medium text-slate-700"
-        >
-          Image or Video URL
-        </label>
-        <input
-          id="image-url-input"
-          name="image_url"
-          type="url"
-          value={imageUrl}
-          onChange={(e) => onUrlChange(e.target.value)}
-          className="w-full rounded-md border-slate-300 text-sm"
-          placeholder="https://example.com/image.jpg or https://example.com/video.mp4"
-        />
-        <p className="text-xs text-slate-500">
-          Enter the direct URL to an image or video
-        </p>
-      </div>
-    </div>
-  );
-}
-
-// Upload Options Component
-interface UploadOptionsProps {
-  imageUrl: string;
-  isFileSelected: boolean;
-  fileInputRef: React.RefObject<HTMLInputElement>;
-  onFileChange: (file: File | null) => Promise<void>;
-  onUrlChange: (url: string) => void;
-}
-
-function UploadOptions({
-  imageUrl,
-  isFileSelected,
-  fileInputRef,
-  onFileChange,
-  onUrlChange,
-}: UploadOptionsProps) {
-  return (
-    <div
-      className={`flex flex-col justify-between gap-4 sm:flex-row ${
-        isFileSelected ? 'hidden' : ''
-      }`}
-    >
-      <FileUploadSection
-        fileInputRef={fileInputRef}
-        onFileChange={onFileChange}
-      />
-      <div className="flex flex-col items-center justify-center">
-        <h3 className="text-lg font-medium text-slate-500">or</h3>
-      </div>
-      <UrlUploadSection imageUrl={imageUrl} onUrlChange={onUrlChange} />
-    </div>
-  );
-}
-
-// Submit Button Component
-function SubmitButton() {
-  return (
-    <button
-      type="submit"
-      className="w-full rounded-md bg-blue-600 px-4 py-2 text-sm font-medium text-white transition-colors hover:bg-blue-700 disabled:cursor-not-allowed disabled:opacity-50"
-    >
-      Search for Similar Images
-    </button>
-  );
-}
-
-export default function VisualSearchForm({
-  actionUrl,
-  csrfToken,
-}: VisualSearchFormProps) {
-  const [imageState, setImageState] = useState<ImageState | null>(null);
-  const [imageUrl, setImageUrl] = useState<string>('');
-  const [isDragOver, setIsDragOver] = useState<boolean>(false);
-  const [isMobile, setIsMobile] = useState<boolean>(false);
-  const [feedbackMessage, setFeedbackMessage] =
-    useState<FeedbackMessage | null>(null);
-
-  const dragDropRef = useRef<HTMLDivElement>(null);
-  const fileInputRef = useRef<HTMLInputElement>(null);
-  const pasteInputRef = useRef<HTMLInputElement>(null);
-  const formRef = useRef<HTMLFormElement>(null);
-
-  // Detect mobile device
-  useEffect(() => {
-    const detectMobile = () => {
-      const userAgent =
-        /Android|webOS|iPhone|iPad|iPod|BlackBerry|IEMobile|Opera Mini/i.test(
-          navigator.userAgent,
-        );
-      const touchPoints =
-        navigator.maxTouchPoints && navigator.maxTouchPoints > 2;
-      return userAgent || touchPoints;
-    };
-    setIsMobile(detectMobile());
-  }, []);
-
-  // Cleanup object URL on unmount
-  useEffect(() => {
-    return () => {
-      if (imageState?.previewUrl) {
-        URL.revokeObjectURL(imageState.previewUrl);
-      }
-    };
-  }, [imageState?.previewUrl]);
-
-  // Show feedback message with auto-dismiss
-  const showFeedback = useCallback(
-    (text: string, type: FeedbackMessage['type']) => {
-      setFeedbackMessage({ text, type });
-      setTimeout(() => setFeedbackMessage(null), 5000);
-    },
-    [],
-  );
-
-  const clearFeedback = useCallback(() => {
-    setFeedbackMessage(null);
-  }, []);
-
-  // Clear selected image
-  const clearImage = useCallback(() => {
-    if (imageState?.previewUrl) {
-      URL.revokeObjectURL(imageState.previewUrl);
-    }
-    setImageState(null);
-    if (fileInputRef.current) {
-      fileInputRef.current.value = '';
-    }
-  }, [imageState?.previewUrl]);
-
-  // Handle image file selection
-  const handleImageFile = useCallback(
-    async (file: File) => {
-      if (!ACCEPTED_IMAGE_TYPES.includes(file.type)) {
-        showFeedback(
-          'Please select a valid image or video file (JPG, PNG, GIF, WebP, MP4)',
-          'error',
-        );
-        return;
-      }
-
-      // Clean up previous preview URL
-      if (imageState?.previewUrl) {
-        URL.revokeObjectURL(imageState.previewUrl);
-      }
-
-      // Show processing message for large files or videos
-      const originalSize = file.size;
-      const isLargeFile = originalSize > 2 * 1024 * 1024; // 2MB
-      const isVideo = file.type.startsWith('video/');
-
-      if (isLargeFile || isVideo) {
-        showFeedback(
-          isVideo
-            ? 'Generating video thumbnail...'
-            : 'Processing large image...',
-          'warning',
-        );
-      }
-
-      try {
-        let processedFile: File;
-        let thumbnailFile: File | undefined;
-        let previewUrl: string;
-
-        if (isVideo) {
-          // For video files, generate thumbnail for preview but keep original for upload
-          thumbnailFile = await generateVideoThumbnail(file);
-          processedFile = file; // Keep original video for upload
-          previewUrl = URL.createObjectURL(thumbnailFile);
-
-          // Set the original video file in the file input for form submission
-          const dataTransfer = new DataTransfer();
-          dataTransfer.items.add(file);
-          fileInputRef.current!.files = dataTransfer.files;
-        } else {
-          // For image files, process as before
-          processedFile = await resizeImageIfNeeded(file);
-          previewUrl = URL.createObjectURL(processedFile);
-
-          const dataTransfer = new DataTransfer();
-          dataTransfer.items.add(processedFile);
-          fileInputRef.current!.files = dataTransfer.files;
-        }
-
-        clearFeedback();
-
-        // Track original size if image was resized
-        const wasResized = !isVideo && processedFile.size < originalSize;
-
-        // Set all image state at once
-        setImageState({
-          file: processedFile,
-          previewUrl,
-          originalFileSize: wasResized ? originalSize : null,
-          thumbnailFile,
-          isVideo,
-        });
-
-        // Visual feedback
-        setIsDragOver(true);
-        setTimeout(() => setIsDragOver(false), 1000);
-      } catch (error) {
-        showFeedback(
-          isVideo
-            ? 'Error processing video. Please try another file.'
-            : 'Error processing image. Please try another file.',
-          'error',
-        );
-      }
-    },
-    [showFeedback, imageState?.previewUrl],
-  );
-
-  // File change handler
-  const handleFileChange = useCallback(
-    async (file: File | null) => {
-      if (file) {
-        await handleImageFile(file);
-      } else {
-        clearImage();
-      }
-    },
-    [handleImageFile, clearImage],
-  );
-
-  // Drag and drop handlers
-  const handleDragEnter = useCallback((e: React.DragEvent) => {
-    e.preventDefault();
-    e.stopPropagation();
-    setIsDragOver(true);
-  }, []);
-
-  const handleDragLeave = useCallback((e: React.DragEvent) => {
-    e.preventDefault();
-    e.stopPropagation();
-    setIsDragOver(false);
-  }, []);
-
-  const handleDragOver = useCallback((e: React.DragEvent) => {
-    e.preventDefault();
-    e.stopPropagation();
-  }, []);
-
-  const handleDrop = useCallback(
-    async (e: React.DragEvent) => {
-      e.preventDefault();
-      e.stopPropagation();
-      setIsDragOver(false);
-
-      const files = e.dataTransfer.files;
-      if (files.length > 0) {
-        const file = files[0];
-        if (file.type.match('image.*') || file.type.match('video.*')) {
-          await handleImageFile(file);
-        } else {
-          showFeedback(
-            `Please drop an image or video file (got ${file.type})`,
-            'error',
-          );
-        }
-      }
-    },
-    [handleImageFile, showFeedback],
-  );
-
-  // Modern Clipboard API handler
-  const tryClipboardAPIRead = useCallback(async (): Promise<boolean> => {
-    try {
-      if (!navigator.clipboard || !navigator.clipboard.read) {
-        return false;
-      }
-
-      const clipboardItems = await navigator.clipboard.read();
-      for (const clipboardItem of clipboardItems) {
-        for (const type of clipboardItem.types) {
-          if (type.startsWith('image/')) {
-            const blob = await clipboardItem.getType(type);
-            const file = new File([blob], 'clipboard-image.png', {
-              type: blob.type,
-            });
-            await handleImageFile(file);
-            return true;
-          }
-        }
-      }
-
-      showFeedback(
-        'No image or video found in clipboard. Copy an image or video first, then try again.',
-        'warning',
-      );
-      return false;
-    } catch (err) {
-      return false;
-    }
-  }, [handleImageFile, showFeedback]);
-
-  // Paste event handler
-  const handlePaste = useCallback(
-    async (e: ClipboardEvent) => {
-      // Allow normal paste behavior for text input fields
-      const target = e.target as HTMLElement;
-      if (
-        target &&
-        (target.tagName === 'INPUT' || target.tagName === 'TEXTAREA')
-      ) {
-        // Let the input handle the paste normally
-        return;
-      }
-
-      e.preventDefault();
-
-      const clipboardItems = e.clipboardData?.items;
-      if (!clipboardItems) return;
-
-      let imageFile: File | null = null;
-
-      // Look for image data in clipboard
-      for (let i = 0; i < clipboardItems.length; i++) {
-        const item = clipboardItems[i];
-        if (item.type.indexOf('image') !== -1) {
-          imageFile = item.getAsFile();
-          break;
-        }
-        if (item.type.indexOf('video') !== -1) {
-          imageFile = item.getAsFile();
-          break;
-        }
-      }
-
-      if (imageFile) {
-        await handleImageFile(imageFile);
-        // focus the drag/drop zone
-        dragDropRef.current?.focus();
-      } else {
-        showFeedback(
-          'No image or video found in clipboard. Copy an image or video first, then paste here.',
-          'warning',
-        );
-      }
-    },
-    [handleImageFile, showFeedback],
-  );
-
-  // Mobile paste instruction
-  const showMobilePasteInstruction = useCallback(() => {
-    showFeedback('Tap this area and select "Paste" from the menu', 'warning');
-  }, [showFeedback]);
-
-  // Click handler for drag-drop area
-  const handleDragDropClick = useCallback(async () => {
-    if (isMobile) {
-      pasteInputRef.current?.focus();
-      const success = await tryClipboardAPIRead();
-      if (!success) {
-        showMobilePasteInstruction();
-      }
-    } else {
-      dragDropRef.current?.focus();
-    }
-  }, [isMobile, tryClipboardAPIRead, showMobilePasteInstruction]);
-
-  // Keyboard event handler
-  const handleKeyDown = useCallback(
-    (e: React.KeyboardEvent) => {
-      if (e.key === 'Enter') {
-        e.preventDefault();
-        formRef.current?.submit();
-      } else if (e.key === ' ') {
-        e.preventDefault();
-        if (isMobile) {
-          pasteInputRef.current?.focus();
-          showMobilePasteInstruction();
-        }
-      }
-    },
-    [isMobile, showMobilePasteInstruction],
-  );
-
-  // Set up paste event listeners
-  useEffect(() => {
-    const pasteHandler = async (e: ClipboardEvent) => await handlePaste(e);
-    document.addEventListener('paste', pasteHandler);
-    return () => document.removeEventListener('paste', pasteHandler);
-  }, [handlePaste]);
-
-  // Mobile touch support
-  useEffect(() => {
-    if (!isMobile || !dragDropRef.current) return;
-
-    const handleTouchStart = () => {
-      setTimeout(() => {
-        pasteInputRef.current?.focus();
-      }, 100);
-    };
-
-    const dragDropElement = dragDropRef.current;
-    dragDropElement.addEventListener('touchstart', handleTouchStart, {
-      passive: true,
-    });
-
-    return () => {
-      dragDropElement.removeEventListener('touchstart', handleTouchStart);
-    };
-  }, [isMobile, dragDropRef, pasteInputRef]);
-
-  return (
-    <div className="overflow-hidden border border-slate-300 bg-white shadow-sm sm:rounded-lg">
-      <form
-        ref={formRef}
-        method="post"
-        action={actionUrl}
-        encType="multipart/form-data"
-        className="flex flex-col gap-4 p-4 sm:p-6"
-      >
-        <input type="hidden" name="authenticity_token" value={csrfToken} />
-
-        <ImageDropZone
-          imageState={imageState}
-          isDragOver={isDragOver}
-          isMobile={isMobile}
-          feedbackMessage={feedbackMessage}
-          onClearImage={clearImage}
-          onDragEnter={handleDragEnter}
-          onDragLeave={handleDragLeave}
-          onDragOver={handleDragOver}
-          onDrop={handleDrop}
-          onClick={handleDragDropClick}
-          onKeyDown={handleKeyDown}
-          onPaste={handlePaste}
-          pasteInputRef={pasteInputRef}
-          dropZoneRef={dragDropRef}
-        />
-
-        <UploadOptions
-          imageUrl={imageUrl}
-          isFileSelected={!!imageState}
-          fileInputRef={fileInputRef}
-          onFileChange={handleFileChange}
-          onUrlChange={setImageUrl}
-        />
-
-        <div className="my-2 border-t border-slate-200"></div>
-
-        <SubmitButton />
-      </form>
-    </div>
-  );
-}

app/javascript/bundles/Main/lib/Trie.ts

@@ -0,0 +1,91 @@
+interface SerializedTrie<T> {
+  // terminal node?
+  t: 1 | 0;
+  // value of the node
+  v: T;
+  // optional children
+  c?: { [s: string]: SerializedTrie<T> };
+}
+
+export class TrieNode<T> {
+  public terminal: boolean;
+  public value: T;
+  public children: Map<string, TrieNode<T>>;
+  public serialized: SerializedTrie<T>;
+
+  constructor(ser: SerializedTrie<T>) {
+    this.terminal = ser.t == 1;
+    this.value = ser.v;
+    this.children = new Map();
+    this.serialized = ser;
+
+    if (ser.c != null) {
+      for (const [key, value] of Object.entries(ser.c)) {
+        this.children.set(key, new TrieNode(value));
+      }
+    }
+  }
+}
+
+export default class Trie<T> {
+  public root: TrieNode<T>;
+  constructor(ser: SerializedTrie<T>) {
+    this.root = new TrieNode(ser);
+  }
+
+  public nodeForPrefix(key: string): {
+    chain: string[];
+    node: TrieNode<T> | null;
+  } {
+    let chain = [];
+    let node = this.root;
+    let remaining = key;
+    while (node && remaining.length > 0) {
+      let exactChild = null;
+      console.log('remaining: ', remaining);
+
+      for (const [childKey, child] of node.children.entries()) {
+        if (remaining.startsWith(childKey)) {
+          console.log('exact match for: ', childKey);
+          exactChild = child;
+          chain.push(childKey);
+          remaining = remaining.slice(childKey.length);
+          break;
+        }
+      }
+
+      // if an exact match was found, continue iterating
+      if (exactChild) {
+        node = exactChild;
+        continue;
+      }
+
+      console.log('looking for partial match for ', remaining);
+      for (const [childKey, child] of node.children.entries()) {
+        const startsWith = childKey.startsWith(remaining);
+        console.log(
+          'test ',
+          childKey,
+          ' against ',
+          remaining,
+          ': ',
+          startsWith,
+          ' ',
+          child.serialized,
+        );
+        if (startsWith) {
+          console.log('partial match for: ', remaining, ': ', child.serialized);
+          chain.push(childKey);
+          return { chain, node: child };
+        }
+      }
+
+      console.log('did not find partial, bailing!');
+      return { chain, node: null };
+    }
+
+    // // return remaining.length === 0 && node && node.terminal ? node : null;
+    console.log('returning child ', node, ' for remaining ', remaining);
+    return { chain, node };
+  }
+}

app/javascript/bundles/Main/utils/byteCountToHumanSize.ts

@@ -1,23 +0,0 @@
-/**
- * Converts a byte count to a human-readable size string.
- *
- * @param bytes - The number of bytes to convert
- * @param decimals - Number of decimal places to show (default: 1)
- * @returns A human-readable size string (e.g., "1.2 KB", "3.4 MB")
- */
-export function byteCountToHumanSize(
-  bytes: number,
-  decimals: number = 1,
-): string {
-  if (bytes === 0) return '0 B';
-  if (bytes < 0) return '0 B';
-
-  const k = 1024;
-  const dm = decimals < 0 ? 0 : decimals;
-  const sizes = ['B', 'KB', 'MB', 'GB', 'TB', 'PB', 'EB', 'ZB', 'YB'];
-
-  const i = Math.floor(Math.log(bytes) / Math.log(k));
-  const size = parseFloat((bytes / Math.pow(k, i)).toFixed(dm));
-
-  return `${size} ${sizes[i]}`;
-}

app/javascript/bundles/Main/utils/chartStatistics.ts

@@ -1,283 +0,0 @@
-interface DataPoint {
-  x: number;
-  y: number;
-}
-
-export interface TrendLineResult {
-  // Polynomial coefficients [c, b, a] for y = ax² + bx + c
-  // Index 0 = constant term, Index 1 = linear term, Index 2 = quadratic term, etc.
-  coeffs: number[];
-  rSquared: number;
-  degree: number;
-  equation: string;
-  color: string; // For display purposes
-}
-
-export interface ChartStatistics {
-  trendLines: TrendLineResult[];
-  favsPerDay: number;
-}
-
-/**
- * Calculate linear regression for given data points
- */
-const calculateLinearRegression = (data: DataPoint[]): TrendLineResult => {
-  const n = data.length;
-  const sumX = data.reduce((sum, point) => sum + point.x, 0);
-  const sumY = data.reduce((sum, point) => sum + point.y, 0);
-  const sumXY = data.reduce((sum, point) => sum + point.x * point.y, 0);
-  const sumX2 = data.reduce((sum, point) => sum + point.x * point.x, 0);
-
-  const denominator = n * sumX2 - sumX * sumX;
-
-  // Check for singular case (all x values are the same)
-  if (Math.abs(denominator) < 1e-10) {
-    throw new Error(
-      'Cannot calculate linear regression - all x values are identical',
-    );
-  }
-
-  const slope = (n * sumXY - sumX * sumY) / denominator;
-  const intercept = (sumY - slope * sumX) / n;
-
-  // Validate coefficients
-  if (!isFinite(slope) || !isFinite(intercept)) {
-    throw new Error('Invalid linear regression coefficients calculated');
-  }
-
-  const coeffs = [intercept, slope]; // [c, b] for y = bx + c
-  const rSquared = calculateRSquared(data, coeffs);
-  const equation = formatPolynomialEquation(coeffs);
-
-  return {
-    coeffs,
-    rSquared,
-    degree: 1,
-    equation,
-    color: 'rgba(34, 197, 94, 1)', // Green
-  };
-};
-
-/**
- * Calculate second-order polynomial regression for given data points
- * Fits y = ax² + bx + c using least squares method
- */
-const calculateQuadraticRegression = (data: DataPoint[]): TrendLineResult => {
-  const n = data.length;
-
-  // Calculate sums needed for polynomial regression
-  const sumX = data.reduce((sum, point) => sum + point.x, 0);
-  const sumY = data.reduce((sum, point) => sum + point.y, 0);
-  const sumX2 = data.reduce((sum, point) => sum + point.x * point.x, 0);
-  const sumX3 = data.reduce(
-    (sum, point) => sum + point.x * point.x * point.x,
-    0,
-  );
-  const sumX4 = data.reduce((sum, point) => sum + Math.pow(point.x, 4), 0);
-  const sumXY = data.reduce((sum, point) => sum + point.x * point.y, 0);
-  const sumX2Y = data.reduce(
-    (sum, point) => sum + point.x * point.x * point.y,
-    0,
-  );
-
-  // Set up matrix for normal equations
-  // [n    sumX   sumX2 ] [c]   [sumY  ]
-  // [sumX sumX2  sumX3 ] [b] = [sumXY ]
-  // [sumX2 sumX3 sumX4] [a]   [sumX2Y]
-
-  const matrix = [
-    [n, sumX, sumX2],
-    [sumX, sumX2, sumX3],
-    [sumX2, sumX3, sumX4],
-  ];
-
-  const vector = [sumY, sumXY, sumX2Y];
-
-  // Solve using Cramer's rule
-  const det = (m: number[][]) =>
-    m[0][0] * (m[1][1] * m[2][2] - m[1][2] * m[2][1]) -
-    m[0][1] * (m[1][0] * m[2][2] - m[1][2] * m[2][0]) +
-    m[0][2] * (m[1][0] * m[2][1] - m[1][1] * m[2][0]);
-
-  const detA = det(matrix);
-
-  // Check for singular matrix (determinant near zero)
-  if (Math.abs(detA) < 1e-10) {
-    throw new Error('Matrix is singular - cannot solve quadratic regression');
-  }
-
-  // Calculate coefficients
-  const detC = det([
-    [vector[0], sumX, sumX2],
-    [vector[1], sumX2, sumX3],
-    [vector[2], sumX3, sumX4],
-  ]);
-
-  const detB = det([
-    [n, vector[0], sumX2],
-    [sumX, vector[1], sumX3],
-    [sumX2, vector[2], sumX4],
-  ]);
-
-  const detA_coeff = det([
-    [n, sumX, vector[0]],
-    [sumX, sumX2, vector[1]],
-    [sumX2, sumX3, vector[2]],
-  ]);
-
-  const c = detC / detA;
-  const b = detB / detA;
-  const a = detA_coeff / detA;
-
-  // Validate coefficients
-  if (!isFinite(a) || !isFinite(b) || !isFinite(c)) {
-    throw new Error('Invalid coefficients calculated');
-  }
-
-  const coeffs = [c, b, a]; // [constant, linear, quadratic]
-  const rSquared = calculateRSquared(data, coeffs);
-  const equation = formatPolynomialEquation(coeffs);
-
-  return {
-    coeffs,
-    rSquared,
-    degree: 2,
-    equation,
-    color: 'rgba(239, 68, 68, 1)', // Red
-  };
-};
-
-/**
- * Calculate R-squared (coefficient of determination) for polynomial fit
- */
-export const calculateRSquared = (
-  data: DataPoint[],
-  coeffs: number[],
-): number => {
-  if (data.length < 2) {
-    return 0; // Cannot calculate meaningful R² with fewer than 2 points
-  }
-
-  const meanY = data.reduce((sum, point) => sum + point.y, 0) / data.length;
-
-  let ssRes = 0; // Sum of squares of residuals
-  let ssTot = 0; // Total sum of squares
-
-  data.forEach((point) => {
-    // Calculate predicted Y using polynomial: y = c₀ + c₁x + c₂x² + ...
-    let predictedY = 0;
-    for (let i = 0; i < coeffs.length; i++) {
-      predictedY += coeffs[i] * Math.pow(point.x, i);
-    }
-    ssRes += Math.pow(point.y - predictedY, 2);
-    ssTot += Math.pow(point.y - meanY, 2);
-  });
-
-  // Handle edge case where all y values are the same
-  if (Math.abs(ssTot) < 1e-10) {
-    return ssRes < 1e-10 ? 1.0 : 0.0; // Perfect fit if residuals are also zero
-  }
-
-  const rSquared = 1 - ssRes / ssTot;
-
-  // Validate result
-  if (!isFinite(rSquared)) {
-    return 0;
-  }
-
-  return rSquared;
-};
-
-/**
- * Generate polynomial equation string from coefficients
- */
-const formatPolynomialEquation = (coeffs: number[]): string => {
-  const terms: string[] = [];
-
-  for (let i = coeffs.length - 1; i >= 0; i--) {
-    const coeff = coeffs[i];
-    if (Math.abs(coeff) < 1e-10) continue; // Skip near-zero coefficients
-
-    let term = '';
-    const absCoeff = Math.abs(coeff);
-
-    if (i === 0) {
-      // Constant term
-      term = absCoeff.toFixed(3);
-    } else if (i === 1) {
-      // Linear term
-      if (absCoeff === 1) {
-        term = 'x';
-      } else {
-        term = `${absCoeff.toFixed(3)}x`;
-      }
-    } else {
-      // Higher order terms
-      if (absCoeff === 1) {
-        term = `x^${i}`;
-      } else {
-        term = `${absCoeff.toFixed(3)}x^${i}`;
-      }
-    }
-
-    if (terms.length === 0) {
-      // First term: include negative sign directly, no plus for positive
-      terms.push(coeff >= 0 ? term : `-${term}`);
-    } else {
-      // Subsequent terms: always include sign with proper spacing
-      terms.push(coeff >= 0 ? ` + ${term}` : ` - ${term}`);
-    }
-  }
-
-  return terms.length > 0 ? `y = ${terms.join('')}` : 'y = 0';
-};
-
-/**
- * Calculate trend lines for given data points, only including fits with sufficient data
- */
-export const calculateTrendLines = (data: DataPoint[]): TrendLineResult[] => {
-  const trendLines: TrendLineResult[] = [];
-
-  // Linear regression requires at least 2 points
-  if (data.length >= 2) {
-    trendLines.push(calculateLinearRegression(data));
-  }
-
-  // Quadratic regression requires at least 3 points
-  if (data.length >= 3) {
-    try {
-      trendLines.push(calculateQuadraticRegression(data));
-    } catch (e) {
-      console.error(e);
-    }
-  }
-
-  return trendLines;
-};
-
-/**
- * Calculate statistics for chart data
- */
-export const calculateChartStatistics = (
-  data: DataPoint[],
-): ChartStatistics => {
-  const trendLines = calculateTrendLines(data);
-
-  // Calculate favs per day using linear regression (for interpretability)
-  const linearTrendLine = trendLines.find((tl) => tl.degree === 1);
-  const linearSlope = linearTrendLine?.coeffs[1] || 0; // coefficient of x in linear equation
-
-  // Only calculate favsPerDay if we have a valid linear slope
-  let favsPerDay = 0;
-  if (linearSlope !== 0 && isFinite(linearSlope)) {
-    favsPerDay = 1 / linearSlope;
-    if (!isFinite(favsPerDay)) {
-      favsPerDay = 0;
-    }
-  }
-
-  return {
-    trendLines,
-    favsPerDay,
-  };
-};

app/javascript/bundles/Main/utils/hoverPreviewUtils.ts

@@ -17,7 +17,7 @@ export function iconClassNamesForSize(size: IconSize) {
     case 'large':
       return 'h-8 w-8 flex-shrink-0 rounded-md';
     case 'small':
-      return 'h-6 w-6 flex-shrink-0 rounded-sm';
+      return 'h-5 w-5 flex-shrink-0 rounded-sm';
   }
 }
 
@@ -227,8 +227,7 @@ export function anchorClassNamesForVisualStyle(
   visualStyle: string,
   hasIcon: boolean = false,
 ) {
-  // let classNames = ['truncate', 'gap-1'];
-  let classNames = ['gap-1', 'min-w-0'];
+  let classNames = ['truncate', 'gap-1'];
   if (hasIcon) {
     classNames.push('flex items-center');
   }

app/javascript/bundles/UI/userMenu.ts

@@ -1,165 +0,0 @@
-/**
- * User Menu Handler
- * Manages the responsive user menu that works for both desktop and mobile layouts
- */
-
-interface UserMenuElements {
-  button: HTMLButtonElement;
-  menu: HTMLDivElement;
-  menuIcon: SVGElement | null;
-  closeIcon: SVGElement | null;
-  chevronIcon: HTMLElement | null;
-}
-
-class UserMenu {
-  private elements: UserMenuElements | null = null;
-
-  constructor() {
-    this.initialize();
-  }
-
-  private initialize(): void {
-    const userMenuButton = document.querySelector(
-      '#user-menu-button',
-    ) as HTMLButtonElement;
-    const userMenu = document.querySelector('#user-menu') as HTMLDivElement;
-    const menuIcon = document.querySelector('.menu-icon') as SVGElement | null;
-    const closeIcon = document.querySelector(
-      '.close-icon',
-    ) as SVGElement | null;
-    const chevronIcon = document.querySelector(
-      '.chevron-icon',
-    ) as HTMLElement | null;
-
-    if (!userMenuButton || !userMenu) {
-      return; // User menu not present (e.g., guest user)
-    }
-
-    this.elements = {
-      button: userMenuButton,
-      menu: userMenu,
-      menuIcon,
-      closeIcon,
-      chevronIcon,
-    };
-
-    this.attachEventListeners();
-  }
-
-  private attachEventListeners(): void {
-    if (!this.elements) return;
-
-    // Toggle menu on button click
-    this.elements.button.addEventListener(
-      'click',
-      this.handleMenuToggle.bind(this),
-    );
-
-    // Close menu when clicking outside
-    document.addEventListener('click', this.handleClickOutside.bind(this));
-
-    // Close menu on escape key
-    document.addEventListener('keydown', this.handleEscapeKey.bind(this));
-  }
-
-  private handleMenuToggle(): void {
-    if (!this.elements) return;
-
-    const isOpen = !this.elements.menu.classList.contains('hidden');
-
-    if (isOpen) {
-      this.closeMenu();
-    } else {
-      this.openMenu();
-    }
-  }
-
-  private handleClickOutside(event: MouseEvent): void {
-    if (!this.elements) return;
-
-    const target = event.target as Node;
-
-    if (
-      !this.elements.button.contains(target) &&
-      !this.elements.menu.contains(target)
-    ) {
-      this.closeMenu();
-    }
-  }
-
-  private handleEscapeKey(event: KeyboardEvent): void {
-    if (!this.elements) return;
-
-    if (
-      event.key === 'Escape' &&
-      !this.elements.menu.classList.contains('hidden')
-    ) {
-      this.closeMenu();
-    }
-  }
-
-  private openMenu(): void {
-    if (!this.elements) return;
-
-    // Show menu but keep it in initial animation state
-    this.elements.menu.classList.remove('hidden');
-    this.elements.button.setAttribute('aria-expanded', 'true');
-
-    // Ensure initial state is set
-    this.elements.menu.classList.add('opacity-0', '-translate-y-2');
-    this.elements.menu.classList.remove('opacity-100', 'translate-y-0');
-
-    // Force a reflow to ensure the initial state is applied
-    this.elements.menu.offsetHeight;
-
-    // Trigger animation by transitioning to visible state
-    requestAnimationFrame(() => {
-      this.elements!.menu.classList.remove('opacity-0', '-translate-y-2');
-      this.elements!.menu.classList.add('opacity-100', 'translate-y-0');
-    });
-
-    // Update mobile icons (hamburger -> close)
-    if (this.elements.menuIcon && this.elements.closeIcon) {
-      this.elements.menuIcon.classList.add('hidden');
-      this.elements.closeIcon.classList.remove('hidden');
-    }
-
-    // Update desktop chevron (rotate down)
-    if (this.elements.chevronIcon) {
-      this.elements.chevronIcon.classList.add('rotate-180');
-    }
-  }
-
-  private closeMenu(): void {
-    if (!this.elements) return;
-
-    // Animate out: fade and slide up
-    this.elements.menu.classList.remove('opacity-100', 'translate-y-0');
-    this.elements.menu.classList.add('opacity-0', '-translate-y-2');
-
-    // Hide menu after CSS animation completes
-    setTimeout(() => {
-      if (this.elements) {
-        this.elements.menu.classList.add('hidden');
-      }
-    }, 150); // Match the CSS transition duration
-
-    this.elements.button.setAttribute('aria-expanded', 'false');
-
-    // Update mobile icons (close -> hamburger)
-    if (this.elements.menuIcon && this.elements.closeIcon) {
-      this.elements.menuIcon.classList.remove('hidden');
-      this.elements.closeIcon.classList.add('hidden');
-    }
-
-    // Update desktop chevron (rotate back up)
-    if (this.elements.chevronIcon) {
-      this.elements.chevronIcon.classList.remove('rotate-180');
-    }
-  }
-}
-
-// Initialize user menu when DOM is loaded
-export function initUserMenu(): void {
-  new UserMenu();
-}

app/javascript/packs/application-bundle.js

@@ -1,30 +1,22 @@
 import ReactOnRails from 'react-on-rails';
 
 import UserSearchBar from '../bundles/Main/components/UserSearchBar';
+import { UserMenu } from '../bundles/Main/components/UserMenu';
 import { PostHoverPreviewWrapper } from '../bundles/Main/components/PostHoverPreviewWrapper';
 import { UserHoverPreviewWrapper } from '../bundles/Main/components/UserHoverPreviewWrapper';
-import { TrackedObjectsChart } from '../bundles/Main/components/TrackedObjectsChart';
-import { PostFiles } from '../bundles/Main/components/PostFiles';
 import { initCollapsibleSections } from '../bundles/UI/collapsibleSections';
 import { IpAddressInput } from '../bundles/UI/components';
-import { StatsPage } from '../bundles/Main/components/StatsPage';
-import VisualSearchForm from '../bundles/Main/components/VisualSearchForm';
-import { initUserMenu } from '../bundles/UI/userMenu';
 
 // This is how react_on_rails can see the components in the browser.
 ReactOnRails.register({
   UserSearchBar,
+  UserMenu,
   PostHoverPreviewWrapper,
   UserHoverPreviewWrapper,
-  TrackedObjectsChart,
-  PostFiles,
   IpAddressInput,
-  StatsPage,
-  VisualSearchForm,
 });
 
-// Initialize UI components
+// Initialize collapsible sections
 document.addEventListener('DOMContentLoaded', function () {
   initCollapsibleSections();
-  initUserMenu();
 });

app/javascript/packs/server-bundle.js

@@ -1,14 +1,14 @@
 import ReactOnRails from 'react-on-rails';
 
 import UserSearchBar from '../bundles/Main/components/UserSearchBarServer';
+import { UserMenu } from '../bundles/Main/components/UserMenu';
 import { PostHoverPreviewWrapper } from '../bundles/Main/components/PostHoverPreviewWrapper';
 import { UserHoverPreviewWrapper } from '../bundles/Main/components/UserHoverPreviewWrapper';
-import { PostFiles } from '../bundles/Main/components/PostFiles';
 
 // This is how react_on_rails can see the UserSearchBar in the browser.
 ReactOnRails.register({
+  UserMenu,
   UserSearchBar,
   PostHoverPreviewWrapper,
   UserHoverPreviewWrapper,
-  PostFiles,
 });

app/javascript/server/buildUsersTrie.js

@@ -0,0 +1,140 @@
+function buildUsersTrie(users) {
+    const rootNode = new trie();
+    users.forEach(([id, name]) => {
+        rootNode.insert(name.toLowerCase(), [id, name]);
+    });
+    return JSON.stringify(rootNode.serialize());
+}
+class trie_node {
+    constructor() {
+        this.terminal = false;
+        this.children = new Map();
+    }
+    serialize() {
+        const { terminal, value, children } = this;
+        let mapped = {};
+        let numChildren = 0;
+        Object.keys(Object.fromEntries(children)).forEach((childKey) => {
+            numChildren += 1;
+            mapped[childKey] = children.get(childKey).serialize();
+        });
+        return {
+            t: this.terminal ? 1 : 0,
+            v: value,
+            c: numChildren > 0 ? mapped : undefined,
+        };
+    }
+}
+class trie {
+    constructor() {
+        this.root = new trie_node();
+        this.elements = 0;
+    }
+    serialize() {
+        return this.root.serialize();
+    }
+    get length() {
+        return this.elements;
+    }
+    get(key) {
+        const node = this.getNode(key);
+        if (node) {
+            return node.value;
+        }
+        return null;
+    }
+    contains(key) {
+        const node = this.getNode(key);
+        return !!node;
+    }
+    insert(key, value) {
+        let node = this.root;
+        let remaining = key;
+        while (remaining.length > 0) {
+            let child = null;
+            for (const childKey of node.children.keys()) {
+                const prefix = this.commonPrefix(remaining, childKey);
+                if (!prefix.length) {
+                    continue;
+                }
+                if (prefix.length === childKey.length) {
+                    // enter child node
+                    child = node.children.get(childKey);
+                    remaining = remaining.slice(childKey.length);
+                    break;
+                }
+                else {
+                    // split the child
+                    child = new trie_node();
+                    child.children.set(childKey.slice(prefix.length), node.children.get(childKey));
+                    node.children.delete(childKey);
+                    node.children.set(prefix, child);
+                    remaining = remaining.slice(prefix.length);
+                    break;
+                }
+            }
+            if (!child && remaining.length) {
+                child = new trie_node();
+                node.children.set(remaining, child);
+                remaining = "";
+            }
+            node = child;
+        }
+        if (!node.terminal) {
+            node.terminal = true;
+            this.elements += 1;
+        }
+        node.value = value;
+    }
+    remove(key) {
+        const node = this.getNode(key);
+        if (node) {
+            node.terminal = false;
+            this.elements -= 1;
+        }
+    }
+    map(prefix, func) {
+        const mapped = [];
+        const node = this.getNode(prefix);
+        const stack = [];
+        if (node) {
+            stack.push([prefix, node]);
+        }
+        while (stack.length) {
+            const [key, node] = stack.pop();
+            if (node.terminal) {
+                mapped.push(func(key, node.value));
+            }
+            for (const c of node.children.keys()) {
+                stack.push([key + c, node.children.get(c)]);
+            }
+        }
+        return mapped;
+    }
+    getNode(key) {
+        let node = this.root;
+        let remaining = key;
+        while (node && remaining.length > 0) {
+            let child = null;
+            for (let i = 1; i <= remaining.length; i += 1) {
+                child = node.children.get(remaining.slice(0, i));
+                if (child) {
+                    remaining = remaining.slice(i);
+                    break;
+                }
+            }
+            node = child;
+        }
+        return remaining.length === 0 && node && node.terminal ? node : null;
+    }
+    commonPrefix(a, b) {
+        const shortest = Math.min(a.length, b.length);
+        let i = 0;
+        for (; i < shortest; i += 1) {
+            if (a[i] !== b[i]) {
+                break;
+            }
+        }
+        return a.slice(0, i);
+    }
+}

app/javascript/server/buildUsersTrie.ts

@@ -0,0 +1,163 @@
+type UserRow = [number, string];
+
+function buildUsersTrie(users: UserRow[]): string {
+  const rootNode = new trie<[number, string]>();
+  users.forEach(([id, name]) => {
+    rootNode.insert(name.toLowerCase(), [id, name]);
+  });
+  return JSON.stringify(rootNode.serialize());
+}
+
+class trie_node<T> {
+  public terminal: boolean;
+  public value: T;
+  public children: Map<string, trie_node<T>>;
+
+  constructor() {
+    this.terminal = false;
+    this.children = new Map();
+  }
+
+  public serialize(): Object {
+    const { terminal, value, children } = this;
+    let mapped = {};
+    let numChildren = 0;
+    Object.keys(Object.fromEntries(children)).forEach((childKey) => {
+      numChildren += 1;
+      mapped[childKey] = children.get(childKey).serialize();
+    });
+    return {
+      t: this.terminal ? 1 : 0,
+      v: value,
+      c: numChildren > 0 ? mapped : undefined,
+    };
+  }
+}
+
+class trie<T> {
+  public root: trie_node<T>;
+  public elements: number;
+
+  constructor() {
+    this.root = new trie_node<T>();
+    this.elements = 0;
+  }
+
+  public serialize(): Object {
+    return this.root.serialize();
+  }
+
+  public get length(): number {
+    return this.elements;
+  }
+
+  public get(key: string): T | null {
+    const node = this.getNode(key);
+    if (node) {
+      return node.value;
+    }
+    return null;
+  }
+
+  public contains(key: string): boolean {
+    const node = this.getNode(key);
+    return !!node;
+  }
+
+  public insert(key: string, value: T): void {
+    let node = this.root;
+    let remaining = key;
+    while (remaining.length > 0) {
+      let child: trie_node<T> = null;
+      for (const childKey of node.children.keys()) {
+        const prefix = this.commonPrefix(remaining, childKey);
+        if (!prefix.length) {
+          continue;
+        }
+        if (prefix.length === childKey.length) {
+          // enter child node
+          child = node.children.get(childKey);
+          remaining = remaining.slice(childKey.length);
+          break;
+        } else {
+          // split the child
+          child = new trie_node<T>();
+          child.children.set(
+            childKey.slice(prefix.length),
+            node.children.get(childKey)
+          );
+          node.children.delete(childKey);
+          node.children.set(prefix, child);
+          remaining = remaining.slice(prefix.length);
+          break;
+        }
+      }
+      if (!child && remaining.length) {
+        child = new trie_node<T>();
+        node.children.set(remaining, child);
+        remaining = "";
+      }
+      node = child;
+    }
+    if (!node.terminal) {
+      node.terminal = true;
+      this.elements += 1;
+    }
+    node.value = value;
+  }
+
+  public remove(key: string): void {
+    const node = this.getNode(key);
+    if (node) {
+      node.terminal = false;
+      this.elements -= 1;
+    }
+  }
+
+  public map<U>(prefix: string, func: (key: string, value: T) => U): U[] {
+    const mapped = [];
+    const node = this.getNode(prefix);
+    const stack: [string, trie_node<T>][] = [];
+    if (node) {
+      stack.push([prefix, node]);
+    }
+    while (stack.length) {
+      const [key, node] = stack.pop();
+      if (node.terminal) {
+        mapped.push(func(key, node.value));
+      }
+      for (const c of node.children.keys()) {
+        stack.push([key + c, node.children.get(c)]);
+      }
+    }
+    return mapped;
+  }
+
+  private getNode(key: string): trie_node<T> | null {
+    let node = this.root;
+    let remaining = key;
+    while (node && remaining.length > 0) {
+      let child = null;
+      for (let i = 1; i <= remaining.length; i += 1) {
+        child = node.children.get(remaining.slice(0, i));
+        if (child) {
+          remaining = remaining.slice(i);
+          break;
+        }
+      }
+      node = child;
+    }
+    return remaining.length === 0 && node && node.terminal ? node : null;
+  }
+
+  private commonPrefix(a: string, b: string): string {
+    const shortest = Math.min(a.length, b.length);
+    let i = 0;
+    for (; i < shortest; i += 1) {
+      if (a[i] !== b[i]) {
+        break;
+      }
+    }
+    return a.slice(0, i);
+  }
+}

app/jobs/domain/bluesky/job/base.rb

@@ -1,90 +0,0 @@
-# typed: strict
-class Domain::Bluesky::Job::Base < Scraper::JobBase
-  abstract!
-  include HasBulkEnqueueJobs
-
-  queue_as :bluesky
-  discard_on ActiveJob::DeserializationError
-
-  sig { override.returns(Symbol) }
-  def self.http_factory_method
-    :get_generic_http_client
-  end
-
-  protected
-
-  sig { params(user: Domain::User::BlueskyUser).void }
-  def enqueue_scan_posts_job_if_due(user)
-    if user.posts_scan.due? || force_scan?
-      logger.info(
-        format_tags(
-          "enqueue posts scan",
-          make_tags(posts_scan: user.posts_scan.ago_in_words),
-        ),
-      )
-      defer_job(Domain::Bluesky::Job::ScanPostsJob, { user: })
-    else
-      logger.info(
-        format_tags(
-          "skipping enqueue of posts scan",
-          make_tags(scanned_at: user.posts_scan.ago_in_words),
-        ),
-      )
-    end
-  end
-
-  sig { params(user: Domain::User::BlueskyUser).void }
-  def enqueue_scan_user_job_if_due(user)
-    if user.profile_scan.due? || force_scan?
-      logger.info(
-        format_tags(
-          "enqueue user scan",
-          make_tags(profile_scan: user.profile_scan.ago_in_words),
-        ),
-      )
-      defer_job(Domain::Bluesky::Job::ScanUserJob, { user: })
-    else
-      logger.info(
-        format_tags(
-          "skipping enqueue of user scan",
-          make_tags(scanned_at: user.profile_scan.ago_in_words),
-        ),
-      )
-    end
-  end
-
-  sig { returns(T.nilable(Domain::User::BlueskyUser)) }
-  def user_from_args
-    if (user = arguments[0][:user]).is_a?(Domain::User::BlueskyUser)
-      user
-    elsif (did = arguments[0][:did]).present?
-      Domain::User::BlueskyUser.find_or_create_by(did:) do |user|
-        resolver = DIDKit::Resolver.new
-        if (resolved = resolver.resolve_did(did))
-          user.handle = resolved.get_validated_handle
-        end
-      end
-    elsif (handle = arguments[0][:handle]).present?
-      resolver = DIDKit::Resolver.new
-      did = resolver.resolve_handle(handle)&.did
-      fatal_error("failed to resolve handle: #{handle}") if did.nil?
-      user = Domain::User::BlueskyUser.find_or_initialize_by(did:)
-      user.handle = handle
-      user.save!
-      user
-    else
-      nil
-    end
-  end
-
-  sig { returns(Domain::User::BlueskyUser) }
-  def user_from_args!
-    T.must(user_from_args)
-  end
-
-  sig { params(user: Domain::User::BlueskyUser).returns(T::Boolean) }
-  def buggy_user?(user)
-    # Add any known problematic handles/DIDs here
-    false
-  end
-end

app/jobs/domain/bluesky/job/scan_posts_job.rb

@@ -1,258 +0,0 @@
-# typed: strict
-class Domain::Bluesky::Job::ScanPostsJob < Domain::Bluesky::Job::Base
-  MEDIA_EMBED_TYPES = %w[app.bsky.embed.images app.bsky.embed.video]
-
-  self.default_priority = -10
-
-  sig { override.params(args: T::Hash[Symbol, T.untyped]).returns(T.untyped) }
-  def perform(args)
-    user = user_from_args!
-    logger.push_tags(make_arg_tag(user))
-    logger.info(format_tags("starting posts scan"))
-
-    return if buggy_user?(user)
-    unless user.state_ok?
-      logger.error(
-        format_tags("skipping posts scan", make_tags(state: user.state)),
-      )
-      return
-    end
-
-    if !user.posts_scan.due? && !force_scan?
-      logger.info(
-        format_tags(
-          "skipping posts scan",
-          make_tags(scanned_at: user.posts_scan.ago_in_words),
-        ),
-      )
-      return
-    end
-
-    scan_user_posts(user)
-    user.last_posts_scan_log_entry = first_log_entry
-    user.touch
-    logger.info(format_tags("completed posts scan"))
-  ensure
-    user.save! if user
-  end
-
-  private
-
-  sig do
-    params(
-      user: Domain::User::BlueskyUser,
-      record_data: T::Hash[String, T.untyped],
-    ).returns(T::Boolean)
-  end
-  def should_record_post?(user, record_data)
-    # Check for quotes first - skip quotes of other users' posts
-    quote_uri = extract_quote_uri(record_data)
-    if quote_uri
-      # Extract DID from the quoted post URI
-      quoted_did = quote_uri.split("/")[2]
-      return false unless quoted_did == user.did
-    end
-
-    # Check for replies - only record if it's a root post or reply to user's own post
-    return true unless record_data.dig("value", "reply")
-
-    # For replies, check if the root post is by the same user
-    reply_data = record_data.dig("value", "reply")
-    root_uri = reply_data.dig("root", "uri")
-
-    return true unless root_uri # If we can't determine root, allow it
-
-    # Extract DID from the root post URI
-    # AT URI format: at://did:plc:xyz/app.bsky.feed.post/rkey
-    root_did = root_uri.split("/")[2]
-
-    # Only record if the root post is by the same user
-    root_did == user.did
-  end
-
-  sig { params(record: T::Hash[String, T.untyped]).returns(T.nilable(String)) }
-  def extract_quote_uri(record)
-    # Check for quote in embed data
-    embed = record["embed"]
-    return nil unless embed
-
-    case embed["$type"]
-    when "app.bsky.embed.record"
-      # Direct quote - check if it's actually a quote of a post
-      record_data = embed["record"]
-      if record_data && record_data["uri"]&.include?("app.bsky.feed.post")
-        record_data["uri"]
-      end
-    when "app.bsky.embed.recordWithMedia"
-      # Quote with media
-      record_data = embed.dig("record", "record")
-      if record_data && record_data["uri"]&.include?("app.bsky.feed.post")
-        record_data["uri"]
-      end
-    else
-      nil
-    end
-  end
-
-  sig { params(user: Domain::User::BlueskyUser).void }
-  def scan_user_posts(user)
-    # Use AT Protocol API to list user's posts
-    posts_url =
-      "https://bsky.social/xrpc/com.atproto.repo.listRecords?repo=#{user.did}&collection=app.bsky.feed.post&limit=100"
-
-    cursor = T.let(nil, T.nilable(String))
-    num_processed_posts = 0
-    num_posts_with_media = 0
-    num_filtered_posts = 0
-    num_created_posts = 0
-    num_pages = 0
-    posts_scan = Domain::UserJobEvent::PostsScan.create!(user:)
-
-    loop do
-      url = cursor ? "#{posts_url}&cursor=#{cursor}" : posts_url
-      response = http_client.get(url)
-      posts_scan.update!(log_entry: response.log_entry) if num_pages == 0
-
-      num_pages += 1
-      if response.status_code == 400
-        error = JSON.parse(response.body)["error"]
-        if error == "InvalidRequest"
-          logger.error(format_tags("account is disabled / does not exist"))
-          user.state = "account_disabled"
-          return
-        end
-      end
-
-      if response.status_code != 200
-        fatal_error(
-          format_tags(
-            "failed to get user posts",
-            make_tags(status_code: response.status_code),
-          ),
-        )
-      end
-
-      begin
-        data = JSON.parse(response.body)
-
-        if data["error"]
-          logger.error(
-            format_tags("posts API error", make_tags(error: data["error"])),
-          )
-          break
-        end
-
-        records = data["records"] || []
-
-        records.each do |record_data|
-          num_processed_posts += 1
-
-          embed_type = record_data.dig("value", "embed", "$type")
-          unless MEDIA_EMBED_TYPES.include?(embed_type)
-            logger.info(
-              format_tags(
-                "skipping post, non-media embed type",
-                make_tags(embed_type:),
-              ),
-            )
-            next
-          end
-
-          # Only process posts with media
-          num_posts_with_media += 1
-
-          # Skip posts that are replies to other users or quotes
-          unless should_record_post?(user, record_data)
-            num_filtered_posts += 1
-            next
-          end
-
-          if process_historical_post(user, record_data, response.log_entry)
-            num_created_posts += 1
-          end
-        end
-
-        cursor = data["cursor"]
-        break if cursor.nil? || records.empty?
-      rescue JSON::ParserError => e
-        logger.error(
-          format_tags(
-            "failed to parse posts JSON",
-            make_tags(error: e.message),
-          ),
-        )
-        break
-      end
-    end
-
-    user.scanned_posts_at = Time.current
-    posts_scan.update!(
-      total_posts_seen: num_processed_posts,
-      new_posts_seen: num_created_posts,
-    )
-    logger.info(
-      format_tags(
-        "scanned posts",
-        make_tags(
-          num_processed_posts:,
-          num_posts_with_media:,
-          num_filtered_posts:,
-          num_created_posts:,
-          num_pages:,
-        ),
-      ),
-    )
-  end
-
-  sig do
-    params(
-      user: Domain::User::BlueskyUser,
-      record_data: T::Hash[String, T.untyped],
-      log_entry: HttpLogEntry,
-    ).returns(T::Boolean)
-  end
-  def process_historical_post(user, record_data, log_entry)
-    at_uri = record_data["uri"]
-
-    # Check if we already have this post
-    existing_post = user.posts.find_by(at_uri:)
-    if existing_post
-      enqueue_pending_files_job(existing_post)
-      return false
-    end
-
-    # Extract reply and quote URIs from the raw post data
-    reply_to_uri = record_data.dig("value", "reply", "root", "uri")
-    quote_uri = extract_quote_uri(record_data)
-
-    post =
-      Domain::Post::BlueskyPost.build(
-        state: "ok",
-        at_uri: at_uri,
-        first_seen_entry: log_entry,
-        text: record_data.dig("value", "text") || "",
-        posted_at: Time.parse(record_data.dig("value", "createdAt")),
-        post_raw: record_data.dig("value"),
-        reply_to_uri: reply_to_uri,
-        quote_uri: quote_uri,
-      )
-    post.creator = user
-    post.save!
-
-    # Process media if present
-    embed = record_data.dig("value", "embed")
-    helper = Bluesky::ProcessPostHelper.new(@deferred_job_sink)
-    helper.process_post_media(post, embed, user.did!) if embed
-    logger.debug(format_tags("created post", make_tags(at_uri:)))
-    true
-  end
-
-  sig { params(post: Domain::Post::BlueskyPost).void }
-  def enqueue_pending_files_job(post)
-    post.files.each do |post_file|
-      if post_file.state_pending?
-        defer_job(Domain::StaticFileJob, { post_file: }, { queue: "bluesky" })
-      end
-    end
-  end
-end

app/jobs/domain/bluesky/job/scan_user_follows_job.rb

@@ -1,266 +0,0 @@
-# typed: strict
-# frozen_string_literal: true
-
-class Domain::Bluesky::Job::ScanUserFollowsJob < Domain::Bluesky::Job::Base
-  self.default_priority = -10
-
-  sig { override.params(args: T::Hash[Symbol, T.untyped]).returns(T.untyped) }
-  def perform(args)
-    user = user_from_args!
-
-    last_follows_scan = user.follows_scans.where(state: "completed").last
-    if (ca = last_follows_scan&.created_at) && (ca > 1.month.ago) &&
-         !force_scan?
-      logger.info(
-        format_tags(
-          "skipping user #{user.did} follows scan",
-          make_tags(
-            ago: time_ago_in_words(ca),
-            last_scan_id: last_follows_scan.id,
-          ),
-        ),
-      )
-    else
-      perform_scan_type(
-        user,
-        "follows",
-        bsky_method: "app.bsky.graph.getFollows",
-        bsky_field: "follows",
-        edge_name: :user_user_follows_from,
-        user_attr: :from_id,
-        other_attr: :to_id,
-      )
-    end
-
-    last_followed_by_scan =
-      user.followed_by_scans.where(state: "completed").last
-    if (ca = last_followed_by_scan&.created_at) && (ca > 1.month.ago) &&
-         !force_scan?
-      logger.info(
-        format_tags(
-          "skipping user #{user.did} followed by scan",
-          make_tags(
-            ago: time_ago_in_words(ca),
-            last_scan_id: last_followed_by_scan.id,
-          ),
-        ),
-      )
-    else
-      perform_scan_type(
-        user,
-        "followed_by",
-        bsky_method: "app.bsky.graph.getFollowers",
-        bsky_field: "followers",
-        edge_name: :user_user_follows_to,
-        user_attr: :to_id,
-        other_attr: :from_id,
-      )
-    end
-  end
-
-  private
-
-  sig do
-    params(
-      user: Domain::User::BlueskyUser,
-      kind: String,
-      bsky_method: String,
-      bsky_field: String,
-      edge_name: Symbol,
-      user_attr: Symbol,
-      other_attr: Symbol,
-    ).void
-  end
-  def perform_scan_type(
-    user,
-    kind,
-    bsky_method:,
-    bsky_field:,
-    edge_name:,
-    user_attr:,
-    other_attr:
-  )
-    scan = Domain::UserJobEvent::FollowScan.create!(user:, kind:)
-    cursor = T.let(nil, T.nilable(String))
-    page = 0
-    subjects_data = T.let([], T::Array[Bluesky::Graph::Subject])
-
-    loop do
-      # get followers
-      xrpc_url =
-        "https://public.api.bsky.app/xrpc/#{bsky_method}?actor=#{user.did!}&limit=100"
-      xrpc_url = "#{xrpc_url}&cursor=#{cursor}" if cursor
-
-      response = http_client.get(xrpc_url)
-      scan.update!(log_entry: response.log_entry) if page == 0
-      page += 1
-
-      if response.status_code != 200
-        fatal_error(
-          format_tags(
-            "failed to get user #{kind}",
-            make_tags(status_code: response.status_code),
-          ),
-        )
-      end
-
-      data = JSON.parse(response.body)
-      if data["error"]
-        fatal_error(
-          format_tags(
-            "failed to get user #{kind}",
-            make_tags(error: data["error"]),
-          ),
-        )
-      end
-      subjects_data.concat(
-        data[bsky_field].map do |subject_data|
-          Bluesky::Graph::Subject.from_json(subject_data)
-        end,
-      )
-      cursor = data["cursor"]
-      break if cursor.nil?
-    end
-
-    handle_subjects_data(
-      user,
-      subjects_data,
-      scan,
-      edge_name:,
-      user_attr:,
-      other_attr:,
-    )
-    scan.update!(state: "completed", completed_at: Time.current)
-    logger.info(
-      format_tags(
-        "completed user #{kind} scan",
-        make_tags(num_subjects: subjects_data.size),
-      ),
-    )
-  rescue => e
-    scan.update!(state: "error", completed_at: Time.current) if scan
-    raise e
-  end
-
-  sig do
-    params(
-      user: Domain::User::BlueskyUser,
-      subjects: T::Array[Bluesky::Graph::Subject],
-      scan: Domain::UserJobEvent::FollowScan,
-      edge_name: Symbol,
-      user_attr: Symbol,
-      other_attr: Symbol,
-    ).void
-  end
-  def handle_subjects_data(
-    user,
-    subjects,
-    scan,
-    edge_name:,
-    user_attr:,
-    other_attr:
-  )
-    subjects_by_did =
-      T.cast(subjects.index_by(&:did), T::Hash[String, Bluesky::Graph::Subject])
-
-    users_by_did =
-      T.cast(
-        Domain::User::BlueskyUser.where(did: subjects_by_did.keys).index_by(
-          &:did
-        ),
-        T::Hash[String, Domain::User::BlueskyUser],
-      )
-
-    missing_user_dids = subjects_by_did.keys - users_by_did.keys
-    missing_user_dids.each do |did|
-      subject = subjects_by_did[did] || next
-      users_by_did[did] = create_user_from_subject(subject)
-    end
-
-    users_by_id = users_by_did.values.map { |u| [T.must(u.id), u] }.to_h
-
-    existing_subject_ids =
-      T.cast(user.send(edge_name).pluck(other_attr), T::Array[Integer])
-
-    new_user_ids = users_by_did.values.map(&:id).compact - existing_subject_ids
-    removed_user_ids =
-      existing_subject_ids - users_by_did.values.map(&:id).compact
-
-    follow_upsert_attrs = []
-    unfollow_upsert_attrs = []
-    referenced_user_ids = Set.new([user.id])
-
-    new_user_ids.each do |new_user_id|
-      new_user_did = users_by_id[new_user_id]&.did
-      followed_at = new_user_did && subjects_by_did[new_user_did]&.created_at
-      referenced_user_ids.add(new_user_id)
-      follow_upsert_attrs << {
-        user_attr => user.id,
-        other_attr => new_user_id,
-        :followed_at => followed_at,
-        :removed_at => nil,
-      }
-    end
-
-    removed_at = Time.current
-    removed_user_ids.each do |removed_user_id|
-      referenced_user_ids.add(removed_user_id)
-      unfollow_upsert_attrs << {
-        user_attr => user.id,
-        other_attr => removed_user_id,
-        :removed_at => removed_at,
-      }
-    end
-
-    Domain::User.transaction do
-      follow_upsert_attrs.each_slice(5000) do |slice|
-        Domain::UserUserFollow.upsert_all(slice, unique_by: %i[from_id to_id])
-      end
-      unfollow_upsert_attrs.each_slice(5000) do |slice|
-        Domain::UserUserFollow.upsert_all(slice, unique_by: %i[from_id to_id])
-      end
-    end
-
-    # reset counter caches
-    Domain::User.transaction do
-      referenced_user_ids.each do |user_id|
-        Domain::User.reset_counters(
-          user_id,
-          :user_user_follows_from,
-          :user_user_follows_to,
-        )
-      end
-    end
-
-    update_attrs = {
-      num_created_users: missing_user_dids.size,
-      num_existing_assocs: existing_subject_ids.size,
-      num_new_assocs: new_user_ids.size,
-      num_removed_assocs: removed_user_ids.size,
-      num_total_assocs: subjects.size,
-    }
-
-    logger.info(
-      format_tags("updated user #{edge_name}", make_tags(update_attrs)),
-    )
-    scan.update_json_attributes!(update_attrs)
-    user.touch
-  end
-
-  sig do
-    params(subject: Bluesky::Graph::Subject).returns(Domain::User::BlueskyUser)
-  end
-  def create_user_from_subject(subject)
-    user =
-      Domain::User::BlueskyUser.create!(
-        did: subject.did,
-        handle: subject.handle,
-        display_name: subject.display_name,
-        description: subject.description,
-      )
-    avatar = user.create_avatar(url_str: subject.avatar)
-    defer_job(Domain::Bluesky::Job::ScanUserJob, { user: }, { priority: 0 })
-    defer_job(Domain::UserAvatarJob, { avatar: }, { priority: -1 })
-    user
-  end
-end

app/jobs/domain/bluesky/job/scan_user_job.rb

@@ -1,232 +0,0 @@
-# typed: strict
-class Domain::Bluesky::Job::ScanUserJob < Domain::Bluesky::Job::Base
-  self.default_priority = -20
-
-  sig { override.params(args: T::Hash[Symbol, T.untyped]).returns(T.untyped) }
-  def perform(args)
-    user = user_from_args!
-    logger.push_tags(make_arg_tag(user))
-    logger.info(format_tags("starting profile scan"))
-
-    if user.state_account_disabled? && !force_scan?
-      logger.info(format_tags("account is disabled, skipping profile scan"))
-      return
-    end
-
-    if !user.profile_scan.due? && !force_scan?
-      logger.info(
-        format_tags(
-          "skipping profile scan",
-          make_tags(scanned_at: user.profile_scan.ago_in_words),
-        ),
-      )
-      return
-    end
-
-    scan_user_profile(user)
-    user.scanned_profile_at = Time.zone.now
-    logger.info(format_tags("completed profile scan"))
-  ensure
-    user.save! if user
-  end
-
-  private
-
-  sig { params(user: Domain::User::BlueskyUser).void }
-  def scan_user_profile(user)
-    logger.info(format_tags("scanning user profile"))
-    profile_scan = Domain::UserJobEvent::ProfileScan.create!(user:)
-
-    # Use Bluesky Actor API to get user profile
-    profile_url =
-      "https://public.api.bsky.app/xrpc/app.bsky.actor.getProfile?actor=#{user.did}"
-
-    response = http_client.get(profile_url)
-    user.last_scan_log_entry = response.log_entry
-    profile_scan.update!(log_entry: response.log_entry)
-
-    if response.status_code == 400
-      error = JSON.parse(response.body)["error"]
-      if error == "InvalidRequest"
-        logger.error(format_tags("account is disabled / does not exist"))
-        user.state = "account_disabled"
-        return
-      end
-    end
-
-    if response.status_code != 200
-      fatal_error(
-        format_tags(
-          "failed to get user profile",
-          make_tags(status_code: response.status_code),
-        ),
-      )
-    end
-
-    begin
-      profile_data = JSON.parse(response.body)
-    rescue JSON::ParserError => e
-      fatal_error(
-        format_tags(
-          "failed to parse profile JSON",
-          make_tags(error: e.message),
-        ),
-      )
-    end
-
-    if profile_data["error"]
-      fatal_error(
-        format_tags(
-          "profile API error",
-          make_tags(error: profile_data["error"]),
-        ),
-      )
-    end
-
-    # The getProfile endpoint returns the profile data directly, not wrapped in "value"
-    record = profile_data
-    if record
-      # Update user profile information
-      user.description = record["description"]
-      user.display_name = record["displayName"]
-      user.profile_raw = record
-
-      # Set registration time from profile createdAt
-      if record["createdAt"]
-        user.registered_at = Time.parse(record["createdAt"]).in_time_zone("UTC")
-        logger.info(
-          format_tags(
-            "set user registration time",
-            make_tags(registered_at: user.registered_at),
-          ),
-        )
-      end
-
-      # Process avatar if present
-      process_user_avatar_url(user, record["avatar"]) if record["avatar"]
-    end
-  end
-
-  sig do
-    params(
-      user: Domain::User::BlueskyUser,
-      avatar_data: T::Hash[String, T.untyped],
-    ).void
-  end
-  def process_user_avatar(user, avatar_data)
-    logger.debug(format_tags("processing user avatar", make_tags(avatar_data:)))
-    return unless avatar_data["ref"]
-
-    user_did = user.did
-    return unless user_did
-
-    avatar_url =
-      Bluesky::ProcessPostHelper.construct_blob_url(
-        user_did,
-        avatar_data["ref"]["$link"],
-      )
-    logger.debug(format_tags("extract avatar url", make_tags(avatar_url:)))
-
-    # Check if avatar already exists and is downloaded
-    existing_avatar = user.avatar
-    if existing_avatar.present?
-      logger.debug(
-        format_tags(
-          "existing avatar found",
-          make_tags(state: existing_avatar.state),
-        ),
-      )
-      # Only enqueue if the avatar URL has changed or it's not downloaded yet
-      if existing_avatar.url_str != avatar_url
-        avatar = user.avatars.create!(url_str: avatar_url)
-        logger.info(
-          format_tags(
-            "avatar url changed, creating new avatar",
-            make_arg_tag(avatar),
-          ),
-        )
-        defer_job(
-          Domain::UserAvatarJob,
-          { avatar: avatar },
-          { queue: "bluesky", priority: -30 },
-        )
-      elsif existing_avatar.state_pending?
-        defer_job(
-          Domain::UserAvatarJob,
-          { avatar: existing_avatar },
-          { queue: "bluesky", priority: -30 },
-        )
-        logger.info(format_tags("re-enqueued pending avatar download"))
-      end
-    else
-      # Create new avatar and enqueue download
-      avatar = user.avatars.create!(url_str: avatar_url)
-      defer_job(
-        Domain::UserAvatarJob,
-        { avatar: },
-        { queue: "bluesky", priority: -30 },
-      )
-      logger.info(
-        format_tags(
-          "created avatar and enqueued download",
-          make_arg_tag(avatar),
-        ),
-      )
-    end
-  end
-
-  sig { params(user: Domain::User::BlueskyUser, avatar_url: String).void }
-  def process_user_avatar_url(user, avatar_url)
-    logger.debug(
-      format_tags("processing user avatar url", make_tags(avatar_url:)),
-    )
-    return if avatar_url.blank?
-
-    # Check if avatar already exists and is downloaded
-    existing_avatar = user.avatar
-    if existing_avatar.present?
-      logger.debug(
-        format_tags(
-          "existing avatar found",
-          make_tags(state: existing_avatar.state),
-        ),
-      )
-      # Only enqueue if the avatar URL has changed or it's not downloaded yet
-      if existing_avatar.url_str != avatar_url
-        avatar = user.avatars.create!(url_str: avatar_url)
-        logger.info(
-          format_tags(
-            "avatar url changed, creating new avatar",
-            make_arg_tag(avatar),
-          ),
-        )
-        defer_job(
-          Domain::UserAvatarJob,
-          { avatar: avatar },
-          { queue: "bluesky", priority: -30 },
-        )
-      elsif existing_avatar.state_pending?
-        defer_job(
-          Domain::UserAvatarJob,
-          { avatar: existing_avatar },
-          { queue: "bluesky", priority: -30 },
-        )
-        logger.info(format_tags("re-enqueued pending avatar download"))
-      end
-    else
-      # Create new avatar and enqueue download
-      avatar = user.avatars.create!(url_str: avatar_url)
-      defer_job(
-        Domain::UserAvatarJob,
-        { avatar: },
-        { queue: "bluesky", priority: -30 },
-      )
-      logger.info(
-        format_tags(
-          "created avatar and enqueued download",
-          make_arg_tag(avatar),
-        ),
-      )
-    end
-  end
-end

app/jobs/domain/e621/job/scan_post_favs_job.rb

@@ -78,7 +78,7 @@ class Domain::E621::Job::ScanPostFavsJob < Domain::E621::Job::Base
       breaker += 1
     end
 
-    post.scanned_post_favs_at = Time.current
+    post.scanned_post_favs_at = DateTime.current
     post.save!
   end
 end

app/jobs/domain/e621/job/scan_user_favs_job.rb

@@ -40,7 +40,7 @@ class Domain::E621::Job::ScanUserFavsJob < Domain::E621::Job::Base
       if response.status_code == 403 &&
            response.body.include?("This users favorites are hidden")
         user.favs_are_hidden = true
-        user.scanned_favs_at = Time.now
+        user.scanned_favs_at = DateTime.current
         user.save!
         break
       end
@@ -121,19 +121,13 @@ class Domain::E621::Job::ScanUserFavsJob < Domain::E621::Job::Base
     logger.info "upserting #{post_ids.size} favs"
     post_ids.each_slice(1000) do |slice|
       ReduxApplicationRecord.transaction do
-        Domain::UserPostFav::E621UserPostFav.upsert_all(
-          slice.map do |post_id|
-            { user_id: user.id, post_id: post_id, removed: false }
-          end,
-          unique_by: %i[user_id post_id],
+        Domain::UserPostFav.upsert_all(
+          slice.map { |post_id| { user_id: user.id, post_id: post_id } },
+          unique_by: :index_domain_user_post_favs_on_user_id_and_post_id,
         )
       end
     end
 
-    # Use reset_counters to update the counter cache after using upsert_all
-    Domain::User.reset_counters(user.id, :user_post_favs)
-    logger.info("[reset user_post_favs counter cache for user: #{user.id}]")
-
     logger.info(
       [
         "[favs scanned: #{post_ids.size.to_s.bold}]",
@@ -144,7 +138,7 @@ class Domain::E621::Job::ScanUserFavsJob < Domain::E621::Job::Base
     )
 
     user.scanned_favs_ok!
-    user.scanned_favs_at = Time.now
+    user.scanned_favs_at = DateTime.current
     user.save!
   rescue StandardError
     logger.error("error scanning user favs: #{user&.e621_id}")

app/jobs/domain/e621/job/static_file_job.rb

@@ -10,6 +10,16 @@ class Domain::E621::Job::StaticFileJob < Domain::E621::Job::Base
         T.cast(file, Domain::PostFile)
       elsif (post = args[:post]) && post.is_a?(Domain::Post::E621Post)
         T.must(post.file)
+      elsif (post = args[:post]) && post.is_a?(Domain::E621::Post)
+        post =
+          Domain::Post::E621Post.find_by(e621_id: post.e621_id) ||
+            fatal_error(
+              format_tags(
+                "post with not found",
+                make_tag("e621_id", post.e621_id),
+              ),
+            )
+        T.must(post.file)
       else
         fatal_error(":file or :post is required")
       end

app/jobs/domain/fa/job/base.rb

@@ -11,8 +11,7 @@ class Domain::Fa::Job::Base < Scraper::JobBase
 
   protected
 
-  BUGGY_USER_URL_NAMES =
-    T.let(["click here", "..", ".", "<i class="], T::Array[String])
+  BUGGY_USER_URL_NAMES = T.let(["click here", "..", "."], T::Array[String])
 
   sig { params(user: Domain::User::FaUser).returns(T::Boolean) }
   def buggy_user?(user)
@@ -37,6 +36,8 @@ class Domain::Fa::Job::Base < Scraper::JobBase
     post = args[:post]
     if post.is_a?(Domain::Post::FaPost)
       return post
+    elsif post.is_a?(Domain::Fa::Post)
+      return Domain::Post::FaPost.find_by!(fa_id: post.fa_id)
     elsif fa_id = args[:fa_id]
       if build_post
         Domain::Post::FaPost.find_or_initialize_by(fa_id: fa_id)
@@ -57,9 +58,12 @@ class Domain::Fa::Job::Base < Scraper::JobBase
       return avatar
     elsif user.is_a?(Domain::User::FaUser)
       return T.must(user.avatar)
+    elsif user.is_a?(Domain::Fa::User)
+      user = Domain::User::FaUser.find_by(url_name: user.url_name)
+      return T.must(user&.avatar)
     else
       fatal_error(
-        "arg 'avatar' must be a Domain::UserAvatar or user must be a Domain::User::FaUser",
+        "arg 'avatar' must be a Domain::UserAvatar or user must be a Domain::Fa::User",
       )
     end
   end
@@ -70,6 +74,8 @@ class Domain::Fa::Job::Base < Scraper::JobBase
     user = args[:user]
     if user.is_a?(Domain::User::FaUser)
       user
+    elsif user.is_a?(Domain::Fa::User)
+      Domain::User::FaUser.find_by!(url_name: user.url_name)
     elsif url_name = args[:url_name]
       if create_if_missing
         user =
@@ -91,7 +97,7 @@ class Domain::Fa::Job::Base < Scraper::JobBase
       end
     else
       fatal_error(
-        "arg 'user' must be a Domain::User::FaUser, or url_name must be provided",
+        "arg 'user' must be a Domain::User::FaUser or Domain::Fa::User, or url_name must be provided",
       )
     end
   end
@@ -363,17 +369,18 @@ class Domain::Fa::Job::Base < Scraper::JobBase
     ).returns(T.nilable(Domain::Fa::Parser::Page))
   end
   def update_user_from_user_page(user, response)
+    disabled_or_not_found = user_disabled_or_not_found?(user, response)
     user.scanned_page_at = Time.current
     user.last_user_page_log_entry = response.log_entry
-    return nil if user_disabled_or_not_found?(user, response)
+    return nil if disabled_or_not_found
 
-    page = Domain::Fa::Parser::Page.from_log_entry(response.log_entry)
+    page = Domain::Fa::Parser::Page.new(response.body)
     return nil unless page.probably_user_page?
 
     user_page = page.user_page
     user.state_ok!
     user.name = user_page.name
-    user.registered_at = user_page.registered_since&.in_time_zone("UTC")
+    user.registered_at = user_page.registered_since
     user.num_pageviews = user_page.num_pageviews
     user.num_submissions = user_page.num_submissions
     user.num_comments_recieved = user_page.num_comments_recieved
@@ -522,7 +529,6 @@ class Domain::Fa::Job::Base < Scraper::JobBase
     T.let(
       [
         /User ".+" has voluntarily disabled access/,
-        /User .+ has voluntarily disabled access/,
         /The page you are trying to reach is currently pending deletion/,
       ],
       T::Array[Regexp],
@@ -559,8 +565,7 @@ class Domain::Fa::Job::Base < Scraper::JobBase
     ).returns(T::Boolean)
   end
   def user_disabled_or_not_found?(user, response)
-    # HTTP 400 is returned when the user is not found
-    if response.status_code != 200 && response.status_code != 400
+    if response.status_code != 200
       fatal_error(
         "http #{response.status_code}, log entry #{response.log_entry.id}",
       )

app/jobs/domain/fa/job/browse_page_job.rb

@@ -57,15 +57,15 @@ class Domain::Fa::Job::BrowsePageJob < Domain::Fa::Job::Base
       )
     end
 
-    page = Domain::Fa::Parser::Page.from_log_entry(response.log_entry)
+    enqueue_jobs_from_found_links(response.log_entry)
+
+    page = Domain::Fa::Parser::Page.new(response.body)
     listing_page_stats =
       update_and_enqueue_posts_from_listings_page(
         ListingPageType::BrowsePage.new(page_number: @page_number),
         page_parser: page,
       )
 
-    enqueue_jobs_from_found_links(response.log_entry)
-
     @total_num_new_posts_seen += listing_page_stats.new_posts.count
     @total_num_posts_seen += listing_page_stats.all_posts.count
     listing_page_stats.new_posts.count > 0

app/jobs/domain/fa/job/favs_job.rb

@@ -29,11 +29,6 @@ class Domain::Fa::Job::FavsJob < Domain::Fa::Job::Base
       return
     end
 
-    if user.state_account_disabled?
-      logger.warn(format_tags("user is disabled, skipping"))
-      return
-    end
-
     faved_post_ids = T.let(Set.new, T::Set[Integer])
     existing_faved_post_ids =
       T.let(Set.new(user.user_post_favs.pluck(:post_id)), T::Set[Integer])
@@ -48,7 +43,7 @@ class Domain::Fa::Job::FavsJob < Domain::Fa::Job::Base
 
     while true
       ret = scan_next_page(user: user)
-      break if ret.is_a?(ScanPageResult::Stop)
+      return if ret.is_a?(ScanPageResult::Stop)
 
       faved_post_ids += ret.faved_post_ids_on_page
       new_faved_post_ids_on_page =
@@ -86,16 +81,30 @@ class Domain::Fa::Job::FavsJob < Domain::Fa::Job::Base
     end
 
     faved_post_ids_to_add = faved_post_ids - existing_faved_post_ids
-    user.upsert_new_favs(
-      faved_post_ids_to_add.to_a,
-      log_entry: causing_log_entry!,
-    )
+    upsert_faved_post_ids(user:, post_ids: faved_post_ids_to_add)
   ensure
     user.save! if user
   end
 
   private
 
+  sig { params(user: Domain::User::FaUser, post_ids: T::Set[Integer]).void }
+  def upsert_faved_post_ids(user:, post_ids:)
+    ReduxApplicationRecord.transaction do
+      if post_ids.any?
+        post_ids.each_slice(1000) do |slice|
+          Domain::UserPostFav.upsert_all(
+            slice.map { |id| { user_id: user.id, post_id: id } },
+            unique_by: %i[user_id post_id],
+          )
+        end
+      end
+
+      user.scanned_favs_at = Time.zone.now
+    end
+    logger.info(format_tags(make_tag("total new favs", post_ids.size)))
+  end
+
   module ScanPageResult
     extend T::Sig
 
@@ -130,9 +139,11 @@ class Domain::Fa::Job::FavsJob < Domain::Fa::Job::Base
     end
 
     disabled_or_not_found = user_disabled_or_not_found?(user, response)
+    user.scanned_favs_at = Time.current
+
     return ScanPageResult::Stop.new if disabled_or_not_found
 
-    page_parser = Domain::Fa::Parser::Page.from_log_entry(response.log_entry)
+    page_parser = Domain::Fa::Parser::Page.new(response.body)
     return ScanPageResult::Stop.new unless page_parser.probably_listings_page?
 
     listing_page_stats =
@@ -140,11 +151,6 @@ class Domain::Fa::Job::FavsJob < Domain::Fa::Job::Base
         ListingPageType::FavsPage.new(page_number: @page_id, user:),
         page_parser:,
       )
-
-    ReduxApplicationRecord.transaction do
-      self.class.update_favs_and_dates(user:, page_parser:)
-    end
-
     @page_id = page_parser.favorites_next_button_id
     ScanPageResult::Ok.new(
       faved_post_ids_on_page:
@@ -153,127 +159,4 @@ class Domain::Fa::Job::FavsJob < Domain::Fa::Job::Base
       keep_scanning: @page_id.present?,
     )
   end
-
-  class FavsAndDatesStats < T::ImmutableStruct
-    extend T::Sig
-    include T::Struct::ActsAsComparable
-    NH = NumberHelper
-
-    const :num_updated_with_fav_fa_id, Integer, default: 0
-    const :num_updated_with_date, Integer, default: 0
-    const :num_updated_total, Integer, default: 0
-
-    sig { params(other: FavsAndDatesStats).returns(FavsAndDatesStats) }
-    def +(other)
-      FavsAndDatesStats.new(
-        num_updated_with_fav_fa_id:
-          num_updated_with_fav_fa_id + other.num_updated_with_fav_fa_id,
-        num_updated_with_date:
-          num_updated_with_date + other.num_updated_with_date,
-        num_updated_total: num_updated_total + other.num_updated_total,
-      )
-    end
-
-    sig { returns(FavsAndDatesStats) }
-    def self.zero
-      FavsAndDatesStats.new()
-    end
-
-    sig { returns(String) }
-    def to_s
-      [
-        num_updated_with_fav_fa_id,
-        num_updated_with_date,
-        num_updated_total,
-      ].map { |n| (NH.number_with_delimiter(n) || "(none)") }.join(" / ")
-    end
-  end
-
-  class FavUpsertData < T::ImmutableStruct
-    extend T::Sig
-    include T::Struct::ActsAsComparable
-    const :post_id, Integer
-    const :fav_id, Integer
-  end
-
-  # Creates or updates Domain::UserPostFav::FaUserPostFav records for the user's favs
-  # and dates. The first faved post is special-cased to be the most recent
-  # faved post, so we can use the date from the page parser to set its date.
-  # The rest of the faved posts are updated with the fav_fa_id from the
-  # page parser.
-  sig do
-    params(
-      user: Domain::User::FaUser,
-      page_parser: Domain::Fa::Parser::Page,
-    ).returns(FavsAndDatesStats)
-  end
-  def self.update_favs_and_dates(user:, page_parser:)
-    num_updated_with_fav_fa_id = 0
-    num_updated_with_date = 0
-    num_updated_total = 0
-
-    fa_id_to_post_id =
-      T.let(
-        Domain::Post::FaPost
-          .where(fa_id: page_parser.submissions_parsed.map(&:id))
-          .pluck(:fa_id, :id)
-          .to_h,
-        T::Hash[Integer, Integer],
-      )
-
-    page_parser
-      .submissions_parsed
-      .filter { |sub| fa_id_to_post_id[T.must(sub.id)].nil? }
-      .each do |sub|
-        fa_id = T.must(sub.id)
-        model = Domain::Post::FaPost.find_or_create_by!(fa_id:)
-        fa_id_to_post_id[fa_id] = T.must(model.id)
-      end
-
-    first_faved = page_parser.submissions_parsed[0]
-    if (fa_id = first_faved&.id) && (post_id = fa_id_to_post_id[fa_id]) &&
-         (fav_id = first_faved.fav_id) &&
-         (explicit_time = page_parser.most_recent_faved_at_time)
-      num_updated_with_date += 1
-      num_updated_total += 1
-      user.update_fav_model(post_id:, fav_id:, explicit_time:)
-    end
-
-    user_post_favs_with_fav_id =
-      (page_parser.submissions_parsed[1..] || [])
-        .filter_map do |sub_data|
-          post_id = (id = sub_data.id) && fa_id_to_post_id[id]
-          next if post_id.nil?
-          fav_id = sub_data.fav_id
-          next if fav_id.nil?
-
-          FavUpsertData
-            .new(post_id:, fav_id:)
-            .tap do
-              num_updated_with_fav_fa_id += 1
-              num_updated_total += 1
-            end
-        end
-        .group_by(&:post_id)
-        .values
-        .filter_map { |data_arr| data_arr.max_by(&:fav_id) }
-        .map do |data|
-          {
-            user_id: T.must(user.id),
-            post_id: data.post_id,
-            fa_fav_id: data.fav_id,
-          }
-        end
-
-    Domain::UserPostFav::FaUserPostFav.upsert_all(
-      user_post_favs_with_fav_id,
-      unique_by: %i[user_id post_id],
-    )
-
-    FavsAndDatesStats.new(
-      num_updated_with_fav_fa_id:,
-      num_updated_with_date:,
-      num_updated_total:,
-    )
-  end
 end

app/jobs/domain/fa/job/scan_file_job.rb

@@ -19,11 +19,13 @@ class Domain::Fa::Job::ScanFileJob < Domain::Fa::Job::Base
             return
           end
           post =
-            if post.is_a?(Domain::Post::FaPost)
+            if post.is_a?(Domain::Fa::Post)
+              Domain::Post::FaPost.find_by!(fa_id: post.fa_id)
+            elsif post.is_a?(Domain::Post::FaPost)
               post
             else
               fatal_error(
-                "invalid post model: #{post.class}, expected Domain::Post::FaPost",
+                "invalid post model: #{post.class}, expected Domain::Fa::Post or Domain::Post::FaPost",
               )
             end
           post.file