Auto Paper Harvester

Batch-download paper PDFs (and supplementary files) by DOI. Routes each DOI through publisher TDM APIs → open-access aggregators → optional institutional-browser fallback, so you actually get the PDFs your institution is paying for instead of a wall of 403s.

Publisher TDM APIs (Wiley / Elsevier / Springer)
        │  (on failure)
        ▼
Crossref / OpenAlex / Unpaywall  (open-access fallback)
        │  (still failing, with --use-browser-fallback)
        ▼
Playwright + Chromium  (reuses your institutional cookies)

Each article ends up in downloads/pdfs/<doi-slug>/<doi-slug>.pdf with any supplementary PDFs detected on the landing page saved alongside it. Throughput is throttled to satisfy publisher TDM rate limits (≥ 1 s/file by default).

v0.2.0 highlights: 24 DOI-prefix routing table covering 19 publisher families (see docs/SUPPORTED_PUBLISHERS.md); new --use-browser-fallback Playwright pass for paywalled publishers without a public TDM API (ACS, RSC, IEEE, AIP, IOP, APS, ...); failed-DOI tracking with structured residual-failure summary; pre-packaged agent skill at .claude/skills/paper-download/.

---

🚀 Quick start

git clone https://github.com/jxtse/auto-paper-harvester.git
cd auto-paper-harvester
pip install -e .                           # core (API + OA pipeline)
cp .env.example .env                       # then edit — set at least CROSSREF_MAILTO

# Single DOI
python .claude/skills/paper-download/scripts/download_by_doi.py \
  --doi 10.1038/s41586-020-2649-2

# Or batch from a Web of Science export
python -m auto_paper_download --savedrecs savedrecs.xls

For paywalled publishers without a TDM API (ACS, RSC, IEEE, AIP, IOP, APS):

pip install -e '.[browser]' && playwright install chromium    # ~150 MB, one-time
python -m auto_paper_download --savedrecs savedrecs.xls --use-browser-fallback

PDFs land under downloads/pdfs/<doi-slug>/. Re-running skips files already on disk.

The repo distinguishes two audiences — the rest of this README has one section for each. Both share the same install and the same .env; the only thing that differs is how you invoke the tool.

---

👤 For Humans

You're a researcher or developer who wants to run this on your own DOI list.

Install

git clone https://github.com/jxtse/auto-paper-harvester.git
cd auto-paper-harvester
pip install -e .                           # requires pip ≥ 21.3 for PEP 660 editable
# OR with browser fallback:
pip install -e '.[browser]' && playwright install chromium

Alternative: uv sync if you prefer uv over pip.

Configure credentials

cp .env.example .env
$EDITOR .env

At minimum set ONE of CROSSREF_MAILTO / OPENALEX_MAILTO to a real email — public APIs require this for polite-pool access. Other credentials are all optional:

Variable	Used for	Free to get?
CROSSREF_MAILTO / OPENALEX_MAILTO	Polite-pool access (any email works)	✅
UNPAYWALL_EMAIL	OA fallback (any email works)	✅
WILEY_TDM_TOKEN	Wiley TDM API	✅ (apply via Wiley)
ELSEVIER_API_KEY	Elsevier TDM API	✅ (apply via Elsevier)
SPRINGER_API_KEY	Springer OA API	✅ (apply via Springer)
CROSSREF_REQUEST_DELAY / WILEY_REQUEST_DELAY	Optional throttling overrides	—