Everything about evaluating and comparing Apify actors with Audit Tools — how it works, what it measures, which platforms it covers, and how AI agents use it.
Audit Tools evaluates Apify actors empirically and tells you which one to use for your task. You paste actor IDs — up to 5 at once — choose evaluation criteria, and get evidence-backed scores with full receipts. Every score traces to a probe result or a measured signal. No scores without evidence.
Anyone who needs to pick an Apify actor for a scraping or data extraction task — developers building pipelines, AI agents selecting tools autonomously, teams comparing alternatives before committing spend, and anyone who has been burned by an actor that looked fine in the Store listing but failed on real inputs.
No. Audit Tools is an independent evaluation layer. It reads public Apify Store data and runs actors via the standard Apify API. Not affiliated with Apify.
The full Apify Store catalog — 39,000+ actors. Not a curated subset. If an actor is publicly listed on Apify Store, it can be evaluated.
The Apify Store has 39,000+ actors and no good way to pick between them. Store search ranks by popularity. ApifyForge covers a curated subset. Apify's own quality score is publisher-facing — it tells actors developers how to rank higher, not buyers which actor to choose. Audit Tools fills the selection gap: run real probes, get ground-truth evidence, make a decision.
Yes, for funded criteria. The actor executes on real inputs and Audit Tools inspects what comes back — output shape, field completeness, null rates, latency, cost. The free metadata tier reads public Apify platform stats and static signals without executing actors.
apify/instagram-scraper), or Apify Store URLMetadata mode is free and instant. It checks public signals — 30-day reliability from Apify platform stats, rot check (does the actor exist and respond), listing freshness, pricing transparency, publisher health, x402 payability, and permission posture. No actor runs required.
Full mode executes live probes: runs the actor on real inputs, checks whether advertised fields actually appear in output, measures true cost-per-result, tests latency, checks memory fit, and verifies run-to-run consistency. Requires x402 payment (USDC on Base) or an Apify API token.
Every score in Audit Tools links to the evidence behind it — the actual probe result, the measured stat, the API response. You can see exactly why an actor scored the way it did. Scores without receipts are opinions; scores with receipts are measurements.
Audit Tools evaluates 16 criteria across six families. Default criteria run on every evaluation. Funded criteria require live actor execution.
| Criterion | What it measures | Free or funded | Default | Weight |
|---|---|---|---|---|
| Rot Check | Actor still exists; last build date; how stale it is | Free | Yes | 1.0 |
| Marketplace 30-Day Reliability | Succeeded runs ÷ total runs (last 30 days), weighted by volume | Free | Yes | 1.0 |
| x402 Payability | Can an agent call this actor autonomously via USDC on Base? | Free | Yes | 1.0 |
| Advertised-Field Truth | Do the fields the listing advertises actually appear in output? | Funded | Yes | 1.0 |
| Cost Per Usable Item | True cost per non-null result (compute + fees ÷ good results) | Funded | Yes | 1.0 |
| Run-to-Run Consistency | Same input twice: result overlap, shape and count stability | Funded | Yes | 0.9 |
| Memory Fit | Completes on default memory without OOM errors | Funded | Yes | 0.9 |
| Listing Freshness | Days since last modification; 180+ days flagged | Free | Yes | 0.8 |
| End-to-End Latency | Wall-clock ms from request to usable dataset on real input | Funded | Yes | 0.8 |
| Permission Posture | Limited vs full-permission; full = human approval required = agent blocker | Free | Yes | 0.8 |
| Pricing Transparency | Actor has a defined public pricing model; PPE = most agent-compatible | Free | Yes | 0.7 |
| Publisher Pulse | Publisher's portfolio health across all their actors | Free | Yes | 0.6 |
| User-Base Trajectory | Monthly ÷ total users ratio — growing, flat, or dying | Free | Yes | 0.5 |
| Machine-Readable Contract | Stable JSON schema, typed fields, deterministic error codes | Free | No | 0.7 |
| Schema / Shape Validity | Actor declares a structured output schema in its listing | Free | No | 0.6 |
| Adoption Momentum | Total users + weekly active users; low with no weekly = abandoned | Free | No | 0.5 |
This is the most common form of silent failure on Apify. An actor's listing advertises fields — email, phone number, price, image URL — that come back null or missing on real inputs. The actor succeeds (status: SUCCEEDED) but the data you needed isn't there. Advertised-Field Truth runs a live probe and checks whether the specific fields claimed in the listing actually appear in output. An actor can have 99% reliability and fail this check entirely.
Sticker price is the per-event fee listed on the actor's Store page. True cost per result is: (compute time cost + memory cost + per-event fee) ÷ number of results that actually pass your quality bar. An actor priced at $0.004/event that returns 5 useful results costs $0.0008 per result. An actor priced at $0.002/event that returns 1 useful result costs $0.002 per result. The cheaper sticker price loses on cost-per-result. Audit Tools measures this directly from live runs.
Above 95% (30-day success rate) with at least 100 runs in that period is strong. 80–95% with high run volume is acceptable. Below 80% is a warning. A new actor with 99% over 8 runs is statistically weaker than an established actor with 94% over 50,000 runs — Audit Tools weights reliability by run volume for this reason.
Audit Tools evaluates all actors in the Apify Store for any platform. Go to /evaluate and paste the actor IDs to compare. The answers below reflect what to look for when evaluating scrapers for each platform — run a current evaluation for live scores.
Google Maps scrapers are among the most competitive on Apify (490K+ users on the top actor). Key differentiators: whether the actor extracts reviews alongside listings, field completeness for address/phone/hours/website, and reliability against Google's anti-bot measures. Evaluate on Advertised-Field Truth and Run-to-Run Consistency — Google Maps actors are especially prone to returning empty reviews or truncated results under load.
The official Apify Instagram Scraper (apify/instagram-scraper, 317K+ users) is the benchmark. When comparing alternatives, check Advertised-Field Truth for engagement metrics (likes, comments count), since Instagram frequently changes its internal API and fields go stale. Also check Permission Posture — some Instagram actors require full permissions.
LinkedIn scrapers carry the highest breakage risk of any major platform — LinkedIn aggressively blocks scraping and frequently changes its structure. Listing Freshness and Rot Check are more important here than on most platforms. Check the actor's last modification date and run 30-day reliability — actors not maintained in the last 60 days are very likely broken.
TikTok scrapers vary significantly in what they actually return. Evaluate Advertised-Field Truth specifically for: video URL, play count, like count, and author data. Some actors advertise these but return nulls or partial data. Run-to-Run Consistency matters for TikTok because hashtag result sets change between calls.
For Amazon, the most important criterion is Advertised-Field Truth for price, ASIN, and review count. Cost Per Usable Item varies significantly between Amazon actors — some charge per page crawled rather than per product record. End-to-End Latency also matters; some Amazon actors are slow because they navigate multiple pages per product. Evaluate multiple actors with identical ASINs to compare directly.
Reddit has a public JSON API (append .json to any URL), which means well-maintained Reddit actors can be very reliable. Key factors: whether the actor respects rate limits cleanly, whether it handles pagination correctly for large subreddits, and whether comment depth is configurable. Check Adoption Momentum — the Reddit scraper ecosystem has been in flux since Reddit API changes.
X (Twitter) scrapers are the most volatile category on Apify due to frequent API changes and aggressive bot detection. Listing Freshness is critical — actors unmaintained for more than 30 days on X are likely broken. Always check the Issues tab on the actor listing before using. Run a full evaluation with live probes before committing to any X scraper.
Real estate scrapers: check Advertised-Field Truth for price, address, bed/bath count, and listing URL. Zillow actors vary widely in whether they return active vs. sold listings. Check whether the actor supports filtering by zip code or region. End-to-End Latency matters for real estate pipelines that need fresh daily data.
SERP actors compete with paid services like SerpAPI. Evaluate on Cost Per Usable Item (cost per ranked URL returned) and Run-to-Run Consistency (does the same query return consistent results?). Also check whether the actor returns organic results only or includes ads, local pack, and featured snippets.
YouTube actors should be evaluated on Advertised-Field Truth for: view count, subscriber count, transcript availability, and comment pagination. YouTube's anti-bot measures mean latency is high — set expectations on End-to-End Latency before building pipelines that need fast turnaround.
For B2B lead gen, the critical criterion is Advertised-Field Truth for email and phone — these fields are frequently advertised and frequently null. Run a live probe on a known company to verify. Also evaluate Cost Per Usable Item against lead enrichment services like Apollo or Clay; Apify actors can be cheaper but require verification.
Run-to-Run Consistency is the most important criterion for price monitoring — you need stable output structure across daily runs. Listing Freshness matters because e-commerce sites change layouts frequently and unmaintained actors break silently. Evaluate End-to-End Latency if you need intraday pricing data.
For rank tracking, evaluate Run-to-Run Consistency on the same keywords across multiple runs. Some SERP actors return slightly different rankings between calls due to Google's personalization — this is expected, but large variance is a signal of unreliability. Cost Per Usable Item: calculate cost per tracked keyword per day across your portfolio.
Brand monitoring requires high Run-to-Run Consistency and reliable pagination. Check whether the actor supports date-range filtering, keyword filtering, and whether it handles rate limits gracefully (failed runs due to rate limits vs. genuine actor failure — these should be distinguished in reliability stats).
Email extraction actors must pass Advertised-Field Truth with flying colors or they're worthless. Run a live probe on domains you know have public emails. Also check whether the actor validates email format (no null@domain.com entries) and whether it deduplicates across pages.
For actors called by AI agents: x402 Payability and Permission Posture are prerequisite filters. Only limited-permission, Pay-Per-Event actors with x402 support can be called in a fully autonomous pipeline without human approval. Machine-Readable Contract matters for agent reliability — actors with stable JSON schemas and typed error codes are far more reliable in agent loops than actors that return unstructured error messages.
Yes. Audit Tools is x402-native. Agents submit actor IDs and criteria to /evaluate, pay per evaluation via USDC on Base, and receive structured scores with evidence. No Apify account, no OAuth, no human in the loop required.
The standard pattern: before an agent selects and runs an Apify actor for a task, it calls Audit Tools with the candidate actors and a criteria set weighted for its use case. Audit Tools returns a ranked list with confidence scores. The agent selects the top-ranked actor that passes the x402 Payability and Permission Posture criteria, then proceeds.
x402 is a payment protocol for autonomous agents. Agents pay per API call in USDC on Base with no human approval step. It enables fully automated tool selection and usage in agent pipelines. Audit Tools uses x402 so AI agents can evaluate actors without being blocked on payment authorization.
Apify actors run in two permission modes. Limited permission means the actor can run autonomously with a pay-per-event payment — compatible with x402 and fully agent-callable. Full permission requires a human to approve the run in the Apify Console — a hard blocker for any autonomous agent pipeline. Audit Tools flags full-permission actors so you never build a pipeline around an actor that will stall waiting for human approval.
Priority order for agent pipelines: (1) x402 Payability — must be true or the actor can't be called autonomously, (2) Permission Posture — must be limited-permission, (3) Marketplace 30-Day Reliability — agents can't tolerate frequent failures, (4) Machine-Readable Contract — stable schema is essential for reliable parsing, (5) Run-to-Run Consistency — pipeline outputs must be predictable.
The full free metadata tier includes: Rot Check, Marketplace 30-Day Reliability, Listing Freshness, Pricing Transparency, Publisher Pulse, x402 Payability, Permission Posture, User-Base Trajectory, Schema Validity, and Adoption Momentum. No account, no payment, no Apify token required.
Live probe criteria require actor execution: Advertised-Field Truth, Cost Per Usable Item, End-to-End Latency, Run-to-Run Consistency, and Memory Fit. These are funded via x402 (USDC on Base) or an Apify API token charged to your Apify account.
Cost depends on which actors are evaluated and how long they take to run. The evaluation overhead is $0.20 or less for most actor comparisons — comparable to the cost of running the actor once yourself, but returning structured evidence across all criteria rather than a single unverified run.
Apify Store search returns actors ranked by relevance and popularity. It doesn't tell you which actor is most reliable, what it actually costs per result, whether the advertised fields really appear in output, or whether an agent can call it autonomously. It's a discovery tool; Audit Tools is an evaluation tool.
ApifyForge Recommender uses keyword matching against a curated catalog of ~300 actors, ranked by 30-day success rate and user popularity — all static, no live probes. It covers a deliberately limited set of well-known actors. Audit Tools covers all 39,000+ actors in the Apify Store, runs live probes to generate ground-truth evidence, and measures true cost-per-result rather than inferring quality from aggregate stats.
Apify's quality score is designed for actor publishers — it measures how well-optimized an actor's listing is for Store discovery. It's a publisher metric. Audit Tools is a buyer metric — it measures whether the actor actually does what you need, at the cost you can afford, reliably enough to build on. Different job, different seat at the table.
ToolRate is a runtime reliability oracle for generic AI tool APIs (Stripe, PayPal, etc.). It wraps live API calls and reports reliability from collective production usage. Audit Tools is Apify-specific and pre-flight — you evaluate actors before you commit to using them, using Apify-specific signals that ToolRate doesn't model (output schema, advertised-field truth, x402 payability, permission posture).
Audit Tools · Independent evaluation layer for the Apify Store · Not affiliated with Apify · llms.txt · Methodology · Evaluate an actor