My job’s outcome is “killed by oom”. What happened?
Asked and answered by the Zyte team, as a reference for the community.
The operating system killed the job because it used more memory than its units provide. Each unit gives 1 GB.
Two ways to fix it:
- Run the job with more units, which gives it more memory.
- Lower the memory your spider uses. A good first setting to try is
SCRAPER_SLOT_MAX_ACTIVE_SIZE, which limits how many response bodies Scrapy holds in memory at one time.
If the memory climbs steadily through the run, also look for things your own code keeps, such as large lists, or items that hold references to responses.