Bluesky Scraper

Bluesky Scraper

Scrape Bluesky through its public AT Protocol XRPC API. Two modes:

• Search — pass a searchQuery keyword and get matching public posts. Bluesky has closed post search to signed-out clients (app.bsky.feed.searchPosts answers 403 for every unauthenticated request, from any IP), so this mode needs a Bluesky handle + app password in blueskyIdentifier / blueskyAppPassword.
• Authors — pass authorHandles (e.g. bsky.app) and get each author's profile (followers, bio, post count) plus their recent posts.

Input

Provide at least one of searchQuery or authorHandles. Each mode is independent: if search is unavailable, the author handles still return rows.

Output

Each post row:

{
  "ok": true,
  "type": "post",
  "uri": "at://did:plc:.../app.bsky.feed.post/3kxyz...",
  "postUrl": "https://bsky.app/profile/bsky.app/post/3kxyz...",
  "authorHandle": "bsky.app",
  "authorName": "Bluesky",
  "authorDid": "did:plc:...",
  "text": "…",
  "createdAt": "2024-01-01T00:00:00.000Z",
  "likeCount": 0,
  "repostCount": 0,
  "replyCount": 0,
  "quoteCount": 0,
  "langs": ["en"]
}

In author mode a profile row (type: "profile") is also emitted per handle, with did, handle, displayName, description, followersCount, followsCount, postsCount, avatar, banner, createdAt, and profileUrl.

Posts are deduplicated by uri. The rkey used in postUrl is the last path segment of the post uri.

Nullable fields. Some fields can be null when the API omits them: on posts, postUrl, authorHandle, authorName, authorDid, and createdAt (counts default to 0, text to "", langs to []); on profiles, handle, displayName, description, avatar, banner, createdAt, and profileUrl (counts default to 0).

Diagnostics

If the run fails or returns nothing, a single ok:false row is pushed with an errorCode (BAD_INPUT, NO_RESULTS, RATE_LIMITED, SERVER_ERROR, NETWORK, …) and a human-readable error message. Diagnostic rows are never charged.

Troubleshooting. If you get a BAD_INPUT row, set searchQuery to a keyword or add at least one handle to authorHandles. A NO_RESULTS row means the API answered but had nothing for that query/author — Bluesky's public index is smaller and sparser than Twitter/X, so broad keywords may return few posts. If you see RATE_LIMITED from many parallel runs, enable the optional proxy or lower the volume.

Billing

$2.00 per 1,000 posts ($0.002 each), with no run-start fee. Flat rate — no volume tiers, no plan gates — and you are charged only for posts actually returned.

Charged per unique post returned (post event). Profile rows and diagnostic rows are not charged.

API

Built on the public host https://api.bsky.app:

• app.bsky.feed.searchPosts
• app.bsky.feed.getAuthorFeed
• app.bsky.actor.getProfile

All are public, cursor-paginated GET/JSON endpoints. We hit api.bsky.app directly rather than the documented public.api.bsky.app alias: the alias is fronted by BunnyCDN, which intermittently returns 403 for searchPosts in some regions, whereas api.bsky.app is the same public AppView served directly and is more reliable.