Bosco LabsRemove the duplicate URLs from this list and keep the newest entry for each.
items takes a pasted JSON array instead of a dataset. With fields url and keep last, the output has one item per URL and keeps the last occurrence, here A (updated) and B. The Actor keeps only a 16-byte hash per key in memory, so the same input works for millions of items.
Tool input
{
"items": [
{
"url": "https://example.com/a",
"name": "A"
},
{
"url": "https://example.com/b",
"name": "B"
},
{
"url": "https://example.com/a",
"name": "A (updated)"
}
],
"fields": [
"url"
],
"keep": "last"
}Ask it from an AI agent (MCP)
Add the Apify MCP server with this tool to your MCP client (Claude, Cursor, VS Code, ...); it signs you in with OAuth or Authorization: Bearer <APIFY_TOKEN>. It works once the Actor is live on the Apify Store.
https://mcp.apify.com?tools=bosco_labs/dataset-dedupThen ask:
Use the bosco_labs/dataset-dedup tool to answer: Remove the duplicate URLs from this list and keep the newest entry for each.Call the Apify API
Runs the Actor and returns the rows in one request, billed to your Apify account ($0.00015 per unique item written (duplicates are free)). It works once the Actor is live on the Apify Store.
curl https://api.apify.com/v2/acts/bosco_labs~dataset-dedup/run-sync-get-dataset-items \
-H "Authorization: Bearer $APIFY_TOKEN" \
-H "Content-Type: application/json" \
-d '{"items": [{"url": "https://example.com/a", "name": "A"}, {"url": "https://example.com/b", "name": "B"}, {"url": "https://example.com/a", "name": "A (updated)"}], "fields": ["url"], "keep": "last"}'Example output
Example output from the tool's documentation (not a run of this exact question).
[{"url": "https://a.com/1", "title": "A", "price": 10},
{"url": "https://a.com/2", "title": "B", "price": 12},
{"url": "https://a.com/3", "title": "C", "price": 7}]