---
name: research-toolkit
title: Research Toolkit
description: "External data discovery and research-paper production: arXiv/RSS feeds, prediction markets, job boards, experiment design, paper drafting, review response, and submission templates."
version: 2.0.0
author: Hermes Agent
license: MIT
platforms: [linux, macos, windows]
metadata:
  hermes:
    tags: [Research, arXiv, RSS, Feeds, Polymarket, Prediction-Markets, Jobs, Career, API, Paper-Writing, ML, AI, Experiments, LaTeX, Citations, Statistical-Analysis, NeurIPS, ICML, ICLR, ACL, AAAI, COLM]
    category: research
    related_skills: [llm-wiki, job-search-toolkit]
---

# Research Toolkit

External data discovery and research-paper production. Covers:

1. **External data discovery:** arXiv papers, RSS/Atom feeds, Polymarket prediction markets, job boards, and LLM/provider comparisons.
2. **Research paper production:** End-to-end pipeline for ML/AI papers (experiment design → execution → analysis → drafting → self-review → revision → submission), including LaTeX templates for NeurIPS, ICML, ICLR, ACL, AAAI, and COLM.

This skill is the umbrella for reusable research workflows previously split across `research-toolkit` and `research-paper-writing`.

> **Related skills:** For building a persistent knowledge base from discovered sources, load `llm-wiki`. For job-search research, load `job-search-toolkit`.

---

## 1. External Data Discovery

### 1.1 arXiv — Academic Paper Search

Search and retrieve papers from arXiv via their free REST API. No API key required.

### Search Papers

```bash
# Keyword search
curl "https://export.arxiv.org/api/query?search_query=all:attention+mechanism+transformer&max_results=5&sortBy=submittedDate&sortOrder=descending"

# Author search
curl "https://export.arxiv.org/api/query?search_query=au:Hinton&max_results=10"

# Category search (cs.CL = Computation and Language)
curl "https://export.arxiv.org/api/query?search_query=cat:cs.CL&max_results=20"

# Recent submissions (today)
curl "https://export.arxiv.org/api/query?search_query=submittedDate:[20260618000000+TO+20260619235959]&max_results=50"
```

### Parse with Python

```python
import xml.etree.ElementTree as ET

xml = """...API response..."""
root = ET.fromstring(xml)
ns = {'atom': 'http://www.w3.org/2005/Atom'}

for entry in root.findall('.//atom:entry', ns):
    title = entry.find('atom:title', ns).text
    authors = [a.find('atom:name', ns).text for a in entry.findall('atom:author', ns)]
    abstract = entry.find('atom:summary', ns).text
    link = entry.find('atom:id', ns).text
    pdf_url = link.replace('/abs/', '/pdf/') + '.pdf'
    published = entry.find('atom:published', ns).text[:10]
    print(f"[{published}] {title}\n  Authors: {', '.join(authors)}\n  PDF: {pdf_url}")
```

### Fetch Specific Paper

```bash
# By ID
curl "https://export.arxiv.org/api/query?id_list=2402.03300"
```

### Read via Hermes

```python
# Abstract (web page)
web_extract(urls=["https://arxiv.org/abs/2402.03300"])

# Full PDF
web_extract(urls=["https://arxiv.org/pdf/2402.03300"])
```

### Categories

| Code | Field |
|------|-------|
| cs.AI | Artificial Intelligence |
| cs.CL | Computation and Language (NLP) |
| cs.CV | Computer Vision |
| cs.LG | Machine Learning |
| cs.RO | Robotics |
| stat.ML | Statistics: Machine Learning |
| q-fin.ST | Quantitative Finance: Statistical Finance |

---

### 1.2 Blogwatcher — RSS/Atom Feed Monitoring

Track blog and RSS/Atom feeds with the `blogwatcher-cli` tool.

### Install

```bash
go install github.com/JulienTant/blogwatcher-cli/cmd/blogwatcher-cli@latest

# Or binary (Linux amd64)
curl -sL https://github.com/JulienTant/blogwatcher-cli/releases/latest/download/blogwatcher-cli_linux_amd64.tar.gz | tar xz -C /usr/local/bin blogwatcher-cli
```

### Manage Feeds

```bash
# Add a feed
blogwatcher-cli add https://blog.example.com/feed.xml --name "Example Blog"

# Add with auto-discovery (tries to find RSS from HTML page)
blogwatcher-cli add https://blog.example.com --discover

# List feeds
blogwatcher-cli list

# Remove a feed
blogwatcher-cli remove "Example Blog"
```

### Read Articles

```bash
# Fetch latest articles (checks all feeds)
blogwatcher-cli fetch

# Show unread articles
blogwatcher-cli articles --unread

# Show articles from a specific feed
blogwatcher-cli articles --feed "Example Blog" --limit 10

# Mark as read
blogwatcher-cli read "Article Title"

# Search articles
blogwatcher-cli search "keyword"
```

### OPML Import/Export

```bash
# Export feeds to OPML (for importing into other readers)
blogwatcher-cli export > feeds.opml

# Import from OPML
blogwatcher-cli import feeds.opml
```

### HTML Scraping Fallback

When a site doesn't publish an RSS feed, `blogwatcher-cli` can scrape the HTML page directly:

```bash
blogwatcher-cli add https://site-without-rss.com --scrape --selector "article h2 a"
```

The `--selector` specifies CSS selector for article links.

---

### 1.3 Polymarket — Prediction Market Data

Query prediction market data from Polymarket's public REST API. Read-only, no authentication.

### Key Concepts

- **Events** contain one or more **Markets** (1:many)
- **Markets** are binary outcomes with Yes/No prices (0.00–1.00)
- Prices ARE probabilities: 0.65 = 65% likelihood
- `outcomePrices` field: `["0.80", "0.20"]` for Yes/No

### Markets and Events

```bash
# Active markets (paginated)
curl -s "https://gamma-api.polymarket.com/markets?active=true&limit=20" | python3 -m json.tool

# Specific market
curl -s "https://gamma-api.polymarket.com/markets/0x..." | python3 -m json.tool

# Events
curl -s "https://gamma-api.polymarket.com/events?active=true&limit=20" | python3 -m json.tool

# Market by slug
curl -s "https://gamma-api.polymarket.com/markets?slug=market-slug-here" | python3 -m json.tool
```

### Orderbook

```bash
# Get orderbook for a specific outcome
MARKET_ID="0x..."
OUTCOME_ID=0  # 0 = Yes, 1 = No

curl -s "https://gamma-api.polymarket.com/orderbook?market=$MARKET_ID&outcomeId=$OUTCOME_ID"
```

### Price History

```bash
# Timeseries price data
curl -s "https://gamma-api.polymarket.com/markets/0x.../prices"

# With interval
curl -s "https://gamma-api.polymarket.com/markets/0x.../prices?interval=1h&startDate=2026-06-01&endDate=2026-06-19"
```

### Parsing with Python

```python
import requests, json

def get_markets(active=True, limit=20):
    url = f"https://gamma-api.polymarket.com/markets?active={active}&limit={limit}"
    return requests.get(url).json()

for m in get_markets(limit=5):
    yes_price, no_price = json.loads(m['outcomePrices'])
    print(f"{m['question']}: Yes {yes_price} / No {no_price}")
```

### Full API Reference

See `references/research-toolkit-polymarket-api-endpoints.md` for all endpoints, parameters, and curl examples.

---

### 1.4 Career Opportunity Analysis

Analyze external job boards and company career pages to find opportunity matches.

### Workflow

#### 1. Load User Profile from Memory

Read `memory(user)` before searching. Verify:
- Current role/seniority
- Location constraints (remote/onsite/hybrid, timezone)
- Tech stack
- Availability timeline
- Salary expectations

#### 2. Navigate and Survey the Site

```bash
# Use browser tools to load and explore
browser_navigate(url="https://company.com/careers")
browser_click(ref="@e5")  # accept cookies if blocking
browser_snapshot(full=true)
```

#### 3. Extract Listing Data

**Strategy A — Static Snapshot:**
- `browser_snapshot(full=true)` → read the page
- `browser_scroll` if paginated

**Strategy B — Deep Dive (per listing):**
- Click individual listings
- Extract requirements, tech stack, compensation, application instructions
- Build a structured table

#### 4. Analysis and Matching

Rank listings by fit score based on:
- Tech stack overlap (weight: 30%)
- Seniority match (weight: 25%)
- Location/remote match (weight: 20%)
- Compensation range match (weight: 15%)
- Company stage/culture fit (weight: 10%)

Present top 5 matches with:
- Role, company, location
- Tech stack overlap score
- Application deadline (if visible)
- Direct application link
- Why it's a strong match

### Supported Platforms

| Platform | URL Pattern | Strategy |
|----------|-------------|----------|
| LinkedIn | linkedin.com/jobs | Search + deep dive |
| Indeed | indeed.com | Search + pagination |
| Glassdoor | glassdoor.com | Search + salary data |
| AngelList/Wellfound | wellfound.com | Startup-focused |
| Company careers | /careers or /jobs | Site-specific scraping |
| Greenhouse | boards.greenhouse.io | ATS scrape |
| Lever | jobs.lever.co | ATS scrape |
| Workday | myworkdayjobs.com | Complex, often requires search |

### Pitfalls

- **ATS variability:** Greenhouse, Lever, Workday all have different DOM structures. Adapt selectors per platform.
- **Login walls:** LinkedIn limits anonymous views. Try adding `?trk=public_jobs` to URLs.
- **Stale listings:** Some jobs are listed but already filled. Note the posting date and treat older listings as lower confidence.
- **Salary transparency:** Not all listings show compensation. Note when it's absent rather than assuming.
- **Job aggregation sites** (Indeed, ZipRecruiter) may have duplicate listings across sources. Dedupe by company+role+location.

### Template: Opportunity Summary Table

| # | Company | Role | Location | Stack Match | Salary | Deadline | Link |
|---|---------|------|----------|-------------|--------|----------|------|
| 1 | Acme | Senior Backend | Remote | 85% | $150-180K | Open | [Apply](...) |

---

### 1.6 Workflow: Research a Topic End-to-End

```
1. Search arXiv for recent papers (Section 1.1)
2. Monitor relevant researcher blogs (Section 1.2)
3. Check prediction market sentiment on the topic (Section 1.3)
4. Compile findings into llm-wiki markdown pages
5. Cross-reference all sources for contradictions and confirmations
```

---

## 2. Research Paper Production

End-to-end pipeline for producing publication-ready ML/AI research papers targeting **NeurIPS, ICML, ICLR, ACL, AAAI, and COLM**. Covers the full research lifecycle: experiment design, execution, monitoring, analysis, paper writing, review, revision, and submission.

This is **not a linear pipeline** — it is an iterative loop. Results trigger new experiments. Reviews trigger new analysis. The agent must handle these feedback loops.

### 2.1 Core philosophy

1. **Be proactive.** Deliver complete drafts, not questions. Scientists are busy — produce something concrete they can react to, then iterate.
2. **Never hallucinate citations.** AI-generated citations have ~40% error rate. Always fetch programmatically. Mark unverifiable citations as `[CITATION NEEDED]`.
3. **Paper is a story, not a collection of experiments.** Every paper needs one clear contribution stated in a single sentence. If you can't do that, the paper isn't ready.
4. **Experiments serve claims.** Every experiment must explicitly state which claim it supports. Never run experiments that don't connect to the paper's narrative.
5. **Commit early, commit often.** Every completed experiment batch, every paper draft update — commit with descriptive messages. Git log is the experiment history.

### 2.2 When to use the paper-production section

- Starting a new research paper from an existing codebase or idea
- Designing and running experiments to support paper claims
- Writing or revising any section of a research paper
- Preparing for submission to a specific conference or workshop
- Responding to reviews with additional experiments or revisions
- Converting a paper between conference formats
- Writing non-empirical papers: theory, survey, benchmark, or position papers (see `references/paper-types.md`)
- Designing human evaluations for NLP, HCI, or alignment research
- Preparing post-acceptance deliverables: posters, talks, code releases

### 2.3 High-level phases

| Phase | Purpose |
|-------|---------|
| Setup | Establish workspace, understand existing work, identify contribution |
| Literature Review | Find and synthesize related work; fetch real citations |
| Experiment Design | Choose baselines, metrics, datasets, compute budgets, stopping criteria |
| Execution & Monitoring | Run experiments, track progress, detect failures, commit often |
| Analysis | Statistical tests, ablations, visualization, claim mapping |
| Paper Drafting | Write each section with a clear narrative and claim order |
| Self-Review & Revision | Checklist-based review, simulate reviewer questions |
| Submission | Format, compile, upload, and complete checklist |

### 2.4 Key references for paper production

- `references/writing-guide.md` — Section-level writing recipes and style conventions.
- `references/experiment-patterns.md` — Reusable experimental designs and statistical tests.
- `references/checklists.md` — Pre-submission and self-review checklists.
- `references/paper-types.md` — Theory, survey, benchmark, and position paper recipes.
- `references/autoreason-methodology.md` — Structured reasoning methodology for paper arguments.
- `references/human-evaluation.md` — Designing and analyzing human evaluations.
- `references/citation-workflow.md` — Fetching, validating, and formatting citations without hallucination.
- `references/reviewer-guidelines.md` — Simulated reviewer rubrics and red-flag checks.
- `references/sources.md` — Curated starter paper lists and data sources.

### 2.5 LaTeX submission templates

Copy the appropriate conference directory from `templates/`:

| Conference | Template path |
|------------|---------------|
| NeurIPS 2025 | `templates/neurips2025/` |
| ICML 2026 | `templates/icml2026/` |
| ICLR 2026 | `templates/iclr2026/` |
| ACL | `templates/acl/` |
| AAAI 2026 | `templates/aaai2026/` |
| COLM 2025 | `templates/colm2025/` |

Each template includes the `.tex` scaffold, `.sty`/`.bst` style files, sample `.bib` files, and a README.

### 2.6 Paper production pitfalls

- **Never claim statistical significance without a test.** Report p-values or confidence intervals.
- **Never present results you cannot reproduce.** Save random seeds, hyperparameters, and exact commands.
- **Avoid metric soup.** Use at most 3 primary metrics; everything else goes to appendix.
- **Cite the original source, not the blog post.** Use Semantic Scholar / Crossref / arXiv APIs.
- **Match the target venue's story expectations.** ICML favors rigorous methodology; ICLR favors novelty and insight; NeurIPS tolerates both if clearly framed.
- **Checklist compliance:** Many venues require reproducibility, ethics, or broader-impact statements. See `references/checklists.md`.

---

## Support Files

### Discovery scripts
- `scripts/research-toolkit-arxiv-search.py` — Python arXiv search utility
- `scripts/research-toolkit-polymarket.py` — Polymarket API client

### Paper-production references
- `references/writing-guide.md`
- `references/experiment-patterns.md`
- `references/checklists.md`
- `references/paper-types.md`
- `references/autoreason-methodology.md`
- `references/human-evaluation.md`
- `references/citation-workflow.md`
- `references/reviewer-guidelines.md`
- `references/sources.md`

### Discovery references
- `references/research-toolkit-polymarket-api-endpoints.md`
- `references/research-toolkit-single-listing-deep-dive.md`
- `references/research-toolkit-regional-junior-webdev-hunt.md`
- `references/research-toolkit-llm-cloud-models.md`

### Paper templates
- `templates/neurips2025/`
- `templates/icml2026/`
- `templates/iclr2026/`
- `templates/acl/`
- `templates/aaai2026/`
- `templates/colm2025/`

---

### 1.5 LLM Model & Provider Research

Research and compare large language models across model catalogs (e.g., Ollama Cloud), API providers, and benchmark leaderboards.

### Typical workflow

1. Load the catalog filtered by the relevant capability (cloud, coding, vision, tools).
2. Open each candidate model's page to collect:
   - Parameter count, active parameters, architecture (dense / MoE)
   - Context window length
   - Capabilities: tools, thinking, vision, multimodal
   - Quantization tags and local/cloud availability
   - License and pull count / recency
3. Extract benchmark claims from the model card and any linked technical report.
4. Cross-check with independent comparisons on sites like LLMReference, Aider, or coding fleet benchmarks.
5. Check pricing: per-token rates or GPU-time plans; factor model size into usage budgets.
6. Summarize with a comparison table keyed to the user's actual use case, not just the highest headline score.

### Key coding benchmarks to collect

| Benchmark | What it measures |
|---|---|
| SWE-Bench Verified | Real GitHub issue resolution with verified tests |
| SWE-Bench Pro | Harder repository-level engineering tasks |
| Terminal-Bench 2.0 | Long-horizon terminal / shell-based coding work |
| LiveCodeBench v6 | Live/competitive programming problems |
| Codeforces ELO | Competitive programming rating proxy |
| MCP Atlas / MCP Mark Verified | Multi-step tool calling and MCP agent workflows |
| NL2Repo | Natural-language to full repository generation |
| SkillsBench | Long-context skill execution |

### Pitfalls

- **Vendor-reported numbers**: Most benchmark scores are self-reported by the model maker with their own agent harness. Treat them as directional, not gospel.
- **Different harnesses**: A model may win on SWE-Bench Pro but feel worse in your IDE because the harness differs.
- **Cloud tag availability**: Catalogs like Ollama add `:cloud` tags gradually. Verify the exact tag exists before telling a user to run it.
- **GPU-time vs per-token pricing**: Ollama Cloud bills by GPU time, not per token. Large MoE models consume allowance faster than small dense ones.
- **Recency bias**: Newly released models often lack independent head-to-heads. Flag when a comparison is mostly press-release data.

### Reference

See `references/research-toolkit-llm-cloud-models.md` for an example comparison built with this workflow.

---

## Workflow: Research a Topic End-to-End

```
1. Search arXiv for recent papers (Section 1)
2. Monitor relevant researcher blogs (Section 2)
3. Check prediction market sentiment on the topic (Section 3)
4. Compile findings into llm-wiki markdown pages
5. Cross-reference all sources for contradictions and confirmations
```