dymk/redux-scraper
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add avatar downloading to Bluesky scan user job

Browse Source 
- Modified process_user_avatar method to enqueue Domain::UserAvatarJob for avatar downloads
- Made Domain::UserAvatarJob concrete (removed abstract!) with generic HTTP client
- Added smart avatar management: handles new avatars, URL changes, and pending re-enqueues
- Added comprehensive test coverage for all avatar scenarios
- Updated HTTP mocking in specs to use HttpClientMockHelpers pattern
- Fixed caused_by_entry handling for chained HTTP requests
- Updated .cursorrules with proper HTTP mocking documentation including caused_by_entry: :any

The job now automatically downloads user avatars when scanning Bluesky users.
This commit is contained in:
Dylan Knutson
2025-08-09 01:23:16 +00:00
parent f2f8a9c34a
commit 5c71fc6b15
6 changed files with 245 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files

46
.cursorrules
View File

@@ -10,6 +10,7 @@
- For instance, if you modify `app/models/domain/post.rb`, run `bin/rspec spec/models/domain/post_spec.rb`. If you modify `app/views/domain/users/index.html.erb`, run `bin/rspec spec/controllers/domain/users_controller_spec.rb`.
- At the end of a long series of changes, run `just test`.
- If specs are failing, then fix the failures, and rerun with `bin/rspec <path_to_spec_file>`.
- If you need to add logging to a Job to debug it, set `quiet: false` on the spec you are debugging.
# Typescript Development
@@ -17,6 +18,51 @@
- Styling is done with Tailwind CSS and FontAwesome.
- Put new typescript files in `app/javascript/bundles/Main/components/`
# HTTP Mocking in Job Specs
When writing specs for jobs that make HTTP requests, use `HttpClientMockHelpers.init_http_client_mock()` instead of manually creating doubles:
```ruby
# CORRECT: Use HttpClientMockHelpers
let(:client_mock_config) do
[
{
uri: "https://example.com/api/first-endpoint",
status_code: 200,
content_type: "application/json",
contents: first_response_body,
},
{
uri: "https://example.com/api/second-endpoint",
status_code: 200,
content_type: "application/json",
contents: second_response_body,
caused_by_entry: :any, # Use this for chained requests
},
]
end
before do
@log_entries =
HttpClientMockHelpers.init_http_client_mock(
http_client_mock,
client_mock_config,
)
end
# WRONG: Don't create doubles manually
expect(http_client_mock).to receive(:get).and_return(
double(status_code: 200, body: response_body, log_entry: double),
)
```
This pattern:
- Creates real HttpLogEntry objects that can be serialized by ActiveJob
- Follows the established codebase pattern
- Avoids "Unsupported argument type: RSpec::Mocks::Double" errors
- Use `caused_by_entry: :any` for HTTP requests that are chained (where one request's log entry becomes the `caused_by_entry` for the next request)
# === BACKLOG.MD GUIDELINES START ===
# Instructions for the usage of Backlog.md CLI Tool

1
app/helpers/domain_source_helper.rb
View File

@@ -8,6 +8,7 @@ module DomainSourceHelper
"furaffinity" => "Domain::Post::FaPost",
"e621" => "Domain::Post::E621Post",
"inkbunny" => "Domain::Post::InkbunnyPost",
"bluesky" => "Domain::Post::BlueskyPost",
}
end

37
app/jobs/domain/bluesky/job/scan_user_job.rb
View File

@@ -275,18 +275,43 @@ class Domain::Bluesky::Job::ScanUserJob < Domain::Bluesky::Job::Base
).void
end
def process_user_avatar(user, avatar_data)
return if user.avatar.present?
logger.debug("process_user_avatar called for user: #{user.handle}")
return unless avatar_data["ref"]
user_did = user.did
return unless user_did
user.create_avatar!(
url_str: construct_blob_url(user_did, avatar_data["ref"]["$link"]),
)
avatar_url = construct_blob_url(user_did, avatar_data["ref"]["$link"])
logger.debug("Avatar URL: #{avatar_url}")
# Enqueue avatar download job if we had one
logger.debug("Created avatar for user: #{user.handle}")
# Check if avatar already exists and is downloaded
existing_avatar = user.avatar
if existing_avatar.present?
logger.debug("Existing avatar found with state: #{existing_avatar.state}")
# Only enqueue if the avatar URL has changed or it's not downloaded yet
if existing_avatar.url_str != avatar_url
existing_avatar.update!(url_str: avatar_url, state: "pending")
defer_result =
defer_job(Domain::UserAvatarJob, { avatar: existing_avatar })
logger.debug(
"Updated avatar URL and enqueued download for user: #{user.handle}, defer_result: #{defer_result}",
)
elsif existing_avatar.state_pending?
defer_result =
defer_job(Domain::UserAvatarJob, { avatar: existing_avatar })
logger.debug(
"Re-enqueued pending avatar download for user: #{user.handle}, defer_result: #{defer_result}",
)
end
else
# Create new avatar and enqueue download
logger.debug("Creating new avatar for user: #{user.handle}")
avatar = user.create_avatar!(url_str: avatar_url, state: "pending")
defer_result = defer_job(Domain::UserAvatarJob, { avatar: avatar })
logger.debug(
"Created avatar and enqueued download for user: #{user.handle}, defer_result: #{defer_result}",
)
end
end
sig { params(did: String, cid: String).returns(String) }

6
app/jobs/domain/user_avatar_job.rb
View File

@@ -1,8 +1,12 @@
# typed: strict
class Domain::UserAvatarJob < Scraper::JobBase
abstract!
queue_as :static_file
sig { override.returns(Symbol) }
def self.http_factory_method
:get_generic_http_client
end
sig { override.params(args: T::Hash[Symbol, T.untyped]).returns(T.untyped) }
def perform(args)
avatar = avatar_from_args!

2
app/models/domain/user/bluesky_user.rb
View File

@@ -61,7 +61,7 @@ class Domain::User::BlueskyUser < Domain::User
sig { override.returns(T.nilable(String)) }
def name_for_view
display_name || handle
"@#{handle}"
end
sig { override.returns(String) }

184
spec/jobs/domain/bluesky/job/scan_user_job_spec.rb
View File

@@ -83,30 +83,32 @@ RSpec.describe Domain::Bluesky::Job::ScanUserJob do
}.to_json
end
before do
# Mock profile API call
expect(http_client_mock).to receive(:get).with(
"https://bsky.social/xrpc/com.atproto.repo.getRecord?repo=#{user.did}&collection=app.bsky.actor.profile&rkey=self",
anything,
).and_return(
double(
let(:client_mock_config) do
[
{
uri:
"https://bsky.social/xrpc/com.atproto.repo.getRecord?repo=#{user.did}&collection=app.bsky.actor.profile&rkey=self",
status_code: 200,
body: profile_response_body,
log_entry: double,
),
)
content_type: "application/json",
contents: profile_response_body,
},
{
uri:
"https://bsky.social/xrpc/com.atproto.repo.listRecords?repo=#{user.did}&collection=app.bsky.feed.post&limit=100",
status_code: 200,
content_type: "application/json",
contents: posts_response_body,
caused_by_entry: :any, # Accept any caused_by_entry for the posts call
},
]
end
# Mock posts API call
expect(http_client_mock).to receive(:get).with(
"https://bsky.social/xrpc/com.atproto.repo.listRecords?repo=#{user.did}&collection=app.bsky.feed.post&limit=100",
anything,
).and_return(
double(
status_code: 200,
body: posts_response_body,
log_entry: double,
),
)
before do
@log_entries =
HttpClientMockHelpers.init_http_client_mock(
http_client_mock,
client_mock_config,
)
# Mock static file job enqueueing - allow it but don't require it
allow(Domain::StaticFileJob).to receive(:perform_later)
@@ -123,7 +125,7 @@ RSpec.describe Domain::Bluesky::Job::ScanUserJob do
expect(user.state).to eq("ok")
end
it "creates avatar for user" do
it "creates avatar for user with pending state" do
expect { perform_now({ user: user }) }.to change {
user.reload.avatar.present?
}.from(false).to(true)
@@ -132,6 +134,21 @@ RSpec.describe Domain::Bluesky::Job::ScanUserJob do
expect(avatar.url_str).to eq(
"https://bsky.social/xrpc/com.atproto.sync.getBlob?did=#{user.did}&cid=bafkreiavatar123",
)
expect(avatar.state).to eq("pending")
end
it "enqueues Domain::UserAvatarJob when creating new avatar" do
# Clear any existing enqueued jobs first
SpecUtil.clear_enqueued_jobs!
perform_now({ user: user })
avatar = user.reload.avatar
expect(avatar).to be_present
# Check that UserAvatarJob was enqueued
enqueued_jobs = SpecUtil.enqueued_job_args(Domain::UserAvatarJob)
expect(enqueued_jobs).to contain_exactly(hash_including(avatar: avatar))
end
it "creates posts with media and associated files" do
@@ -166,6 +183,127 @@ RSpec.describe Domain::Bluesky::Job::ScanUserJob do
end
end
context "avatar handling scenarios" do
let(:profile_response_body) do
{
"uri" => "at://#{user.did}/app.bsky.actor.profile/self",
"cid" => "bafyreiabc123",
"value" => {
"displayName" => "Test User",
"description" => "A test user profile",
"avatar" => {
"ref" => {
"$link" => "bafkreiavatar123",
},
"mimeType" => "image/jpeg",
"size" => 50_000,
},
},
}.to_json
end
let(:posts_response_body) { { "records" => [], "cursor" => nil }.to_json }
let(:client_mock_config) do
[
{
uri:
"https://bsky.social/xrpc/com.atproto.repo.getRecord?repo=#{user.did}&collection=app.bsky.actor.profile&rkey=self",
status_code: 200,
content_type: "application/json",
contents: profile_response_body,
},
{
uri:
"https://bsky.social/xrpc/com.atproto.repo.listRecords?repo=#{user.did}&collection=app.bsky.feed.post&limit=100",
status_code: 200,
content_type: "application/json",
contents: posts_response_body,
caused_by_entry: :any, # Accept any caused_by_entry for the posts call
},
]
end
before do
@log_entries =
HttpClientMockHelpers.init_http_client_mock(
http_client_mock,
client_mock_config,
)
end
context "when user has existing avatar with same URL" do
let!(:existing_avatar) do
user.create_avatar!(
url_str:
"https://bsky.social/xrpc/com.atproto.sync.getBlob?did=#{user.did}&cid=bafkreiavatar123",
state: "ok",
)
end
it "does not enqueue job for already downloaded avatar with same URL" do
perform_now({ user: user })
# Avatar should remain unchanged
existing_avatar.reload
expect(existing_avatar.state).to eq("ok")
# Should not enqueue any avatar job
enqueued_jobs = SpecUtil.enqueued_job_args(Domain::UserAvatarJob)
expect(enqueued_jobs).to be_empty
end
end
context "when user has existing avatar with different URL" do
let!(:existing_avatar) do
user.create_avatar!(
url_str:
"https://bsky.social/xrpc/com.atproto.sync.getBlob?did=#{user.did}&cid=oldavatar456",
state: "ok",
)
end
it "updates avatar URL and enqueues job" do
perform_now({ user: user })
existing_avatar.reload
expect(existing_avatar.url_str).to eq(
"https://bsky.social/xrpc/com.atproto.sync.getBlob?did=#{user.did}&cid=bafkreiavatar123",
)
expect(existing_avatar.state).to eq("pending")
# Should enqueue avatar job with the existing avatar
enqueued_jobs = SpecUtil.enqueued_job_args(Domain::UserAvatarJob)
expect(enqueued_jobs).to contain_exactly(
hash_including(avatar: existing_avatar),
)
end
end
context "when user has existing avatar in pending state" do
let!(:existing_avatar) do
user.create_avatar!(
url_str:
"https://bsky.social/xrpc/com.atproto.sync.getBlob?did=#{user.did}&cid=bafkreiavatar123",
state: "pending",
)
end
it "re-enqueues job for pending avatar with same URL" do
perform_now({ user: user })
existing_avatar.reload
expect(existing_avatar.state).to eq("pending")
# Should re-enqueue avatar job
enqueued_jobs = SpecUtil.enqueued_job_args(Domain::UserAvatarJob)
expect(enqueued_jobs).to contain_exactly(
hash_including(avatar: existing_avatar),
)
end
end
end
context "when user already scanned recently" do
before do
user.update!(scanned_profile_at: 1.day.ago, scanned_posts_at: 1.day.ago)