redux-scraper/TODO.md

Project TODO List

• Add bookmarking feature for posts across different domains
• Add search feature to search FA descriptions, tags, E621 descriptions, tags
• Get inkbunny index scan job working
• Attach logs to jobs, page to view jobs and their logs
• Standardize all the embeddings tables to use the same schema (item_id, embedding)
• Bluesky scraper
• Download favs / votes for E621 users
• Automatically enqueue jobs for FA users to do incremental scans of profiles
• Fix FA posts that start with "Font size adjustment: smallerlarger"
• Convert logger .prefix=... into .tagged(...)
• make_tag should be smart about the objects it takes
• Convert all state: string attributes to enums in ActiveRecord models
• Create belongs_to_log_entry macro for ActiveRecord models
• Use StaticFileJobHelper for Domain::Fa::Job::ScanFileJob
• Unify HTTP client configs for all domains, so the same job type can be used for different domains
• put abstract external_url_for_view in a module
• backfill descriptions on inkbunny posts
• store deep update json on inkbunny posts
• limit number of users, or paginate for "users who favorited this post" page
• manual good job runner does not indicate if the job threw an exception - check return value of #perform, maybe?
• FA user favs job should stop when in incremental mode when all posts on the page are already known favs (e.g. pages with only 47 posts are not a false positive)
• Factor out FA listings page enqueue logic into common location; use in Gallery and Favs jobs
• Add followers / following to FA user show page
• Parse E621 source url for inkbunny posts & users
• Parse E621 source url for fa users
• Parse BBCode in post descriptions
  • example post with bbcode: https://refurrer.com/posts/ib/3452498
• Show tags on fa posts, ib posts
• Sofurry implmentation
• Make unified Static file job
• Make unified Avatar file job
• ko-fi domain icon
• tumblr domain icon
• Do PCA on user factors table to display a 2D plot of users
• Use links found in descriptions to indicate re-scanning a post? (e.g. for comic next/prev links)
• fix for IDs that have a dot in them - e.g. https://refurrer.com/users/fa@jakke.
• Rich inline links to e621 e.g. https://refurrer.com/posts/fa@60070060
• Find FaPost that have favs recorded but no scan / file, enqueue scan
• Bunch of posts with empty responses: posts = Domain::Post.joins(files: :log_entry).where(files: { http_log_entries: { response_sha256: BlobFile::EMPTY_FILE_SHA256 }}).limit(10)