← Audit Tools · FAQ · Criteria catalog
Every score in Audit Tools traces to a specific probe result or measured signal. This page explains the evaluation architecture, scoring model, evidence classes, and the reasoning behind each criterion.
The Apify Store has 39,000+ actors. Every listing has a title, a description, a star rating (often zero), a user count, and a pricing model. None of these tell you:
Audit Tools measures these things directly. The evaluation model is built around one question: would I build a production pipeline on this actor?
Audit Tools uses four probe types to gather evidence:
Reads public data from the Apify Store API without executing the actor: listing details, build timestamps, run statistics (30-day success rate, total runs), pricing configuration, output schema declaration, publisher portfolio stats. No actor credits consumed.
Executes the actor on controlled real inputs and inspects what comes back. Measures end-to-end latency, actual cost, output field completeness, and whether advertised fields appear in the dataset. Requires x402 payment (USDC on Base) or an Apify API token.
Runs the actor twice on identical inputs and computes result overlap. Measures run-to-run consistency — the Jaccard similarity of result sets, output shape stability, and count variance. Used for the Run-to-Run Consistency criterion.
Extracts field claims from the actor's README and Store listing, runs the actor on real input, and checks whether each claimed field is populated in the output dataset. This is the Advertised-Field Truth criterion — the most valuable single signal for deciding whether an actor is safe to use.
This is the most common form of silent actor failure. It happens because:
Audit Tools' Claims-vs-Reality probe runs the actor on known-good inputs — URLs or queries where the claimed fields are expected to exist — and checks the output. An actor that advertises email, phone, and website and returns all three populated scores 100% on Advertised-Field Truth. An actor that returns null for email 80% of the time fails this criterion regardless of its 30-day success rate.
Apify actors have two cost components:
Sticker price comparison ignores compute cost, which often dominates for actors with long runtimes. An actor priced at $0.002/start that runs for 2 minutes on 1GB costs approximately $0.002 + (0.004 × 2/60) = $0.0033 per start.
But cost-per-start also ignores result count. The cost that matters is:
cost_per_usable_item = (event_fee + compute_cost) ÷ count_of_non_null_resultsAn actor priced at $0.004/start returning 50 good results costs $0.00008 per result. An actor priced at $0.002/start returning 8 good results costs $0.00025 per result — 3× more expensive per useful record. Audit Tools measures this directly from live runs.
The Apify Store shows a 30-day success rate for each actor. This number is trustworthy only at scale. An actor with 99% success rate over 10 runs has demonstrated very little — one failure in its history. An actor with 94% success rate over 50,000 runs has proven reliability across diverse inputs, geographies, and time periods.
Audit Tools weights the 30-day reliability signal by log-scaled run volume:
Audit Tools includes a family of criteria specifically relevant to AI agents calling Apify actors autonomously:
Three conditions must all be true: (1) the actor is whitelisted for agentic payments on Apify, (2) it uses Pay-Per-Event pricing, (3) it runs with limited permissions. When all three are true, an agent can call the actor autonomously via USDC on Base without any human approval step.
Apify's limited-permission mode allows actors to run with scoped access to resources. Full-permission mode requires explicit human approval in the Apify Console before the actor can run. Full-permission actors are a hard blocker for autonomous agent pipelines — the agent loop stalls waiting for a human. Audit Tools surfaces this as a binary flag.
Agents parse actor output programmatically. Actors with stable JSON schemas, typed output fields, and deterministic error codes (rather than unstructured error messages) are far more reliable in agent loops. This criterion checks for: output schema declaration, field typing consistency, and pricing model explicitness.
Individual actor stats tell you about one actor's past. Publisher Pulse looks at the actor developer's full portfolio — every actor they've published — and computes aggregate health signals: average failure rate across their portfolio, percentage of their actors that are actively maintained, and whether they've been responding to issues. A publisher with a history of publishing and abandoning actors is a risk signal even for their newest actor, which may not yet show failures in its individual stats.
Each criterion contributes a normalized 0–100 score weighted by its default weight. The composite score is a weighted average:
composite = Σ (criterion_score × criterion_weight) ÷ Σ (criterion_weight)Criteria with binary outcomes (Rot Check, x402 Payability, Memory Fit) score 100 (pass) or 0 (fail). Ratio criteria (Reliability, Consistency) score proportionally. Cost criteria (Cost Per Usable Item, Latency) are inverted and normalized against the distribution of evaluated actors.
When comparing multiple actors, scores are relative — Actor A's cost-per-result score is computed relative to the other actors in the same evaluation, not against a global benchmark. This surfaces which actor wins on each criterion within the comparison set.
| Evidence class | Meaning | Criteria |
|---|---|---|
| Measured | Directly observed from a probe result or API stat | All 16 criteria |
| Free | No actor execution required; from public metadata | Rot Check, Reliability, Freshness, Pricing, Publisher Pulse, x402, Permission, Trajectory, Schema, Contract, Momentum |
| Funded | Requires live actor execution; costs credits | Cost Per Item, Latency, Consistency, Memory Fit, Advertised-Field Truth |
Audit Tools · Independent evaluation layer for the Apify Store · Not affiliated with Apify · llms.txt · FAQ · Criteria catalog · Evaluate an actor