Bosco Labs
Deduplicate & Merge Apify Datasets: remove duplicate items
Coming soon to the StoreDeduplicate and merge one or more Apify datasets into one, removing duplicate items by any fields you choose (for example url, email or address.city), or by the whole item. Built for people who run the same scraper many times, or several scrapers, and need one clean list. It streams the data, so millions of items run with constant memory.
Pricing (pay per event): $0.00015 per unique item written ($0.15 per 1,000). Duplicates are free, reading is free, there is no start fee, and the run stops at your max charge limit.
Questions agents ask
- “Merge the results of my last three Google Maps runs and remove duplicate places by placeId.”
- “Deduplicate this list of leads by email and keep only name, email and website.”
- “Keep only the latest price per SKU from these two product datasets.”
Example
Input
{
"datasetIds": ["aBcD1234efGH5678i", "jKlM9012noPQ3456r"],
"fields": ["url"],
"keep": "first",
"outputFields": ["url", "title", "price"]
}Output
[{"url": "https://a.com/1", "title": "A", "price": 10},
{"url": "https://a.com/2", "title": "B", "price": 12},
{"url": "https://a.com/3", "title": "C", "price": 7}]What you get
Items are copied unchanged from the input datasets (or reduced to outputFields).
Coverage
Works on the Apify datasets you pick in the input, or a pasted JSON array. For every distinct key the Actor keeps a 16-byte hash in memory, never the items: 10 million distinct keys need roughly 1-2 GB.
Use with AI agents (MCP)
Add this URL to your MCP client (Claude, Cursor, VS Code, ...) to load the Actor as a tool through the Apify hosted MCP server. It signs you in with OAuth or accepts Authorization: Bearer <APIFY_TOKEN>. It works once the Actor is live on the Apify Store.
https://mcp.apify.com?tools=bosco_labs/dataset-dedupFor “Merge the results of my last three Google Maps runs and remove duplicate places by placeId.” the agent passes:
{"datasetIds": ["<dataset ID 1>", "<dataset ID 2>", "<dataset ID 3>"], "fields": ["placeId"]}the unique items, unchanged and in input order, in the run's default dataset. The Actor adds no fields of its own (no schema_version or source link), since the items are your data; the run log ends with read, duplicates and written counts.
Or call the Apify API: POST https://api.apify.com/v2/acts/bosco_labs~dataset-dedup/run-sync-get-dataset-items with the input as JSON body. Cost: $0.00015 per unique item written (duplicates are free).
Input fields
| Field | Type | Meaning |
|---|---|---|
datasetIds | array | Apify dataset IDs to merge, in this order, e.g. ["aBcD1234efGH5678i"]: a run's defaultDatasetId (Storage tab). Optional if you pass items. |
items | array | Items to deduplicate directly, as a JSON array of objects, e.g. [{"url": "https://example.com/a"}]. Processed before the datasets. Handy for small lists and quick tests. |
fields | array | Fields that form the duplicate key: items with equal values in all of them are duplicates. Dot paths work, e.g. ["url"] or ["address.city", "emails.0"]. Default [] = compare whole items. Items that have none of these fields are always kept. |
keep | string | "first" (default) keeps the first occurrence of each key; "last" keeps the last one and reads the datasets twice. |
outputFields | array | Top-level fields to keep in the output, e.g. ["url", "title"]. Empty = keep every field. |
Sources, licence and limits
How big can the datasets be? For every distinct key the Actor keeps a 16-byte hash in memory, never the items. 10 million distinct keys need roughly 1-2 GB, so most runs fit in the default memory.
Is keep: last slower? It reads the input datasets twice (once to find the last occurrence of each key, once to write), still without holding the items in memory.
What about items that don't have the dedupe field? They are always kept, never merged with each other, and counted as kept_without_key in the log. A field that is null counts as missing.
Are 1 and "1" the same? No. Values are compared exactly as stored (type included). Whole-item comparison ignores the order of keys.
Which datasets can it read? Only the ones you pick in the input. The Actor runs with limited permissions, so it can't see any other storage in your account.
What if the run is restarted by the platform? It starts over, skips the items it already wrote, and doesn't charge them again.