Table of contents
- Use case 1: Too many jobs are scheduled
- Use case 2: Running out of memory
Use case 1: Too many jobs are scheduled
Issue: The user schedulers a lot of jobs, which causes the process takes too long time and the scheduler is run too often. For example, when an incoming configuration takes 10 minutes to complete it is scheduled every 5 minutes.
Solution: The user can check the System > the Queue page. There they can see how long a process took from the moment it was scheduled to the moment it was finished. If they see a process took 10 minutes they can configure the scheduler to run for example every 15 minutes.
Use case 2: Running out of memory
Jobs can run out of memory. The default maximum is 384MB per process.
Case 1: Users handle large files.
Subscribing to files in remote filesystems is one of the standard integration approaches. However, users sometimes need to fetch large files, and it can cause the process to run out of memory.
Solution: The user can configure the JSON, XML, etc deserializers to process data line by line. We provide a functionality to incrementally read the files. This will reduce the memory usage.
Case 2: HTTP subscriber returns a large number of data
Due to users’ requirements, it is sometimes required to fetch a list of records from a web service. Retrieving a large number of data (response) from a single API call can consume a lot of memory.
Solution: If the API endpoint supports pagination, we recommend configuring the HTTP request to limit the number of data fetched in a single API call. Then, let the incoming configuration fetch the next page of the data in the next run of the incoming configuration. In addition, we also provide functionality to read the data incrementally.
Case 3: Users enabled real-time processing
An incoming configuration is connected to a route that has real-time processing enabled. Real-time processing can make a process of incoming configuration consume more memory because the process will also execute the outgoing jobs (transformation and publishing).
Solution: Disable real-time processing and configure a separate scheduler to run the route.
Case 4: A Route exports a large number of tasks at a time
Users can configure a scheduler of a route to export as many tasks as possible. However, more tasks to export by a single process means more memory to consume.
Solution: Users can try reducing the number of tasks to export until the route can export the tasks successfully without failing.
Note: As an alternative to resolve the out-of-memory issue, you have the option to upgrade the memory limit at any time. For further details, please reach out to your Customer Success Manager.