How Do I Process Large Lists in Make Without Timeouts or Duplicates?

A practical field guide from Automation Ace.

The short answer

Don't try to process a huge list in one Make run. Split the work into batches, keep a status flag or checkpoint for each record, and let a scheduled scenario pick up the next batch each run. Use a unique key so each record is created only once, slow down for API rate limits, and send failures to a review list instead of stopping the whole job.

Why large lists break

• Run time limits: Make limits how long a single scenario run can last, so a long loop can be cut off partway.
• Rate limits: calling an API thousands of times in a row triggers 429 errors. See 429 rate limit errors.
• Partial failures: if record 3,217 fails, you need to know which records were already done before you rerun.
• Search limits: search and list modules return a limited number of results by default, so some records are silently skipped.
• Usage: every bundle through every module counts toward your plan. See why a scenario uses too many operations.

1. Split the work into batches

Keep the full list in a table, such as Airtable, a Google Sheet, a database, or a Make data store. Add a status field (pending, done, failed) to each record. A scheduled scenario then:

1. Searches for the next batch of pending records, for example 100 at a time.
2. Processes each record.
3. Marks each record done or failed as soon as it finishes.

Each run is short, and the next run continues where the last one stopped. See where to keep scenario data.

2. Page through APIs with a checkpoint

When the list comes from an API, save your place after each page: the last page number, cursor, or updated-at timestamp. Store it in a data store so the next run starts from the checkpoint instead of the beginning. See calling APIs with the HTTP module.

3. Prevent duplicates

• Give every record a stable unique key, such as the source ID, and search for it before creating anything.
• Mark records done in the same run that processes them, not in a separate cleanup step.
• Turn on sequential processing for webhook-triggered scenarios that must handle items one at a time.

See safely reprocessing records without duplicates and why a webhook fires twice.

4. Pace requests and handle errors

• Add a Sleep module between calls, or lower the batch size, to stay under the API's rate limit.
• Use an error handler that marks the record as failed and continues, so one bad record doesn't stop the batch. See Make error handling.
• Retry failed records in a later run, and send records that fail twice to a person for review.

5. Fan out to a second scenario when needed

For heavy work per record, one scenario can find the records and send each one to a second scenario through a webhook. Each record then runs on its own, so a slow or failed record doesn't hold up the rest. Use this with care: it multiplies runs, so check your usage and the receiving API's limits first.

Example: enriching 20,000 contacts

A team needs to enrich 20,000 CRM contacts with company data from an API that allows a limited number of requests per minute. Instead of one long run, the contacts get a status field. Every 15 minutes, a scenario takes 200 pending contacts, calls the API with a short pause between requests, writes the results, and marks each contact done or failed. The job finishes over a day without timeouts, rate-limit errors, or duplicates, and failed contacts are listed for review. See long-running automations.

Frequently asked questions

How many records can a Make scenario process in one run?

It depends on your plan's run time limit, the modules involved, and the APIs you call. Rather than finding the maximum, process a fixed batch per run and let a schedule work through the rest.

How do I stop a large job from creating duplicates?

Give each record a unique key, search for it before creating, and mark each record done as soon as it is processed. Then a rerun skips anything already finished.

What should happen when one record fails?

Mark it as failed and continue with the rest of the batch. Retry failed records in a later run, and send repeat failures to a person for review.

Should I use a Make data store or a separate database for the queue?

A data store keeps everything inside Make and works well for checkpoints and modest lists. For large lists, or when people need to review and edit records, Airtable or a database is easier to manage.