Intro
I recently had to write a coding standards document for my (now ex) company. The document was obviously copy and pasted to a SKILL.md file (sigh) and not actually distributed to any human engineers. I wrote a section on data handling that contains what I think can be a simple and useful framework. This might be obvious to most engineers, but I've seen it ignored time after time with disastrous effects.
Bounded data
Data is bounded when the exact maximum size, meaning amount of items times the size of each item, is known. There are a ton of advantages to handling bounded data. The maximum can be tested, it can be reasoned about, you can know in advance if trying to load it into RAM will cause an OOM etc.
Unbounded data
Data is unbounded when the exact maximum size is not known (not necessarily unknowable). One cannot guarantee anything when handling unbounded data. Will it cause an OOM? Maybe. Will it choke the CPU for seconds, minutes or hours? Maybe. Will I get a stack overflow exception? Maybe. Will the user complain? Probably.
Turning unbounded data into bounded data
Finding out
In some cases there are limitations imposed on the size of data by underlying technologies. If you use PostgreSQL varchar column, if you use fastify, etc. You will have a default (or custom) max size. If you're very very lucky everything will be bounded.
Hard limits
Sometimes data is theoretically unbounded, but in practical use cases it's actually very small. In these cases you can reject all requests (commands, etc.) that will increase the data beyond a reasonable limit, or requests (commands, etc.) containing large amounts of data. DO NOT ASSUME that data will remain small, actually limit it. Be aware that this solution can lead to a bad experience for users, as what they see as legitimate usage can differ from your preconceptions.
Pagination/Chunking
Splitting unbounded data, or bounded data that is too large, is the most usually applicable solution. Always process chunks with a known maximum size. If the data is stored in a database, query it in pages. If the data is stored in a file, read it in chunks. If the data is transferred over a network accept it in chunks. Just make sure you are not accumulating an unbounded amount of chunks/pages in memory at the same time.
Design
Sometimes it's possible to design processes in a way that avoids unbounded data. For example you can avoid free text fields in favor of enumerable options. You could use a hash function if the exact contents are not what you need. You could move logic from one service to another to avoid sending unbounded data between them. Get creative and think from first principles: what am I trying to accomplish and how do I accomplish it in a predictable manner?
Outro
If you're working on an existing project and you are in panic mode, I think setting hard limits to stop the bleeding and working on pagination/chunking is your best bet. If you're working on a new project, it will save you so much headache to design your system in a way that minimizes the amount of data being handled and working on pagination/chunking from the get-go.