arXiv Scraper

arXiv Scraper

Search arXiv via the official arXiv API and get clean, structured paper metadata. No API key, no login, no anti-bot — just polite, paginated requests to export.arxiv.org.

$2.00 per 1,000 papers ($0.002 each), with no run-start fee. Flat rate — no volume tiers, no plan gates — and you are charged only for papers actually returned.

What it does

Given an arXiv search query, the actor fetches the Atom feed from the arXiv API, parses every <entry>, dedupes by arXiv ID, and returns one clean record per paper.

Input

Search query syntax

arXiv uses field prefixes you can combine with AND / OR / ANDNOT:

• all:transformer — search all fields
• cat:cs.AI — papers in a category (e.g. cs.CL, cs.LG, stat.ML)
• ti:attention — title
• au:hinton — author
• abs:diffusion — abstract
• ti:attention AND au:vaswani — combine

Examples: all:large language models, cat:cs.CL, ti:"retrieval augmented generation".

Output

One record per paper:

{
  "ok": true,
  "arxivId": "2401.12345",
  "title": "Paper title",
  "abstract": "Whitespace-collapsed abstract…",
  "authors": ["First Author", "Second Author"],
  "primaryCategory": "cs.CL",
  "categories": ["cs.CL", "cs.AI"],
  "publishedAt": "2024-01-23T18:00:00Z",
  "updatedAt": "2024-02-01T12:00:00Z",
  "doi": "10.1000/xyz123",
  "absUrl": "http://arxiv.org/abs/2401.12345v1",
  "pdfUrl": "http://arxiv.org/pdf/2401.12345v1"
}

Nullable fields: doi is null when the paper has no registered DOI. primaryCategory, publishedAt, updatedAt, and absUrl can also be null if the arXiv feed omits them for an entry. pdfUrl is derived from the arXiv ID when the feed doesn't include a PDF link, so it is null only when the ID itself is missing.

Results are deduplicated by arxivId, falling back to absUrl and then title when a record has no arXiv ID.

Notes

• Be polite: the actor pauses ~3 seconds between pages, as arXiv requests.
• arXiv hard-limits the total reachable results per query to about 30000.
• On an error, empty result, or missing query, the actor writes a single diagnostic row (ok: false, with an errorCode such as NO_RESULTS, BAD_INPUT, RATE_LIMITED, or NETWORK) and does not charge. The run still finishes cleanly so you can inspect the reason in the dataset.

Pricing

There is no run-start fee — a run that returns nothing costs nothing. Diagnostic rows (ok: false) and empty results are never charged, so a mistyped query is free. 50 papers = $0.10. 1,000 papers = $2.00.