> Allocating all required memory is, IMHO, a great practice for almost any type of program.
Do you know any good resources to get into that mode of thinking, including common patterns, for sombody who is used to malloc everything? I could easily come up with lots of examples in my current project where "it isn't that easy" and "you cannot know this up-front", even when re-designing the whole architecture, and trying to solve those problems feels like re-inventing a hundred wheels that have been invented before.
I could only read the second article for now (the first one is blocked at work), but while interesting it's not quite what my question was aiming at. That is, it solves the malloc/free hassle in an elegant way, but it doesn't solve the fundamental problem to know in advance how much memory you'll need.
That is, the second article basically says: (pre-arena) It would be so easy if we could just allocate everything on the stack, because lifetimes are nestable. But the article doesn't say how to know the required size of the stack in advance, to prevent stack overflows. That was the part I was interested in.
edit: I could read the first article through a proxy, but it also just talks about different allocators. I'd like to achieve what com2kid was talking about: "Allocating all required memory is, IMHO, a great practice for almost any type of program."
Late reply, but the most common ‘advice’ (usually in the form of post-mortums) I have seen is that knowing the total allocation size for a program is an engineering analysis based on memory availability for a given system and estimated worst-case bounds for memory usage. The system is then designed to work within the constraints generated by that analysis.
Pre-allocating certainly requires a system/application designer to consider the most likely worst case resource usage and then enforce that estimate throughout the design.
Further, for systems that have modern memory sized resources (i.e. 16gb and up) the system is designed to work as though memory was a stack and then that stack is allocated from system resources using something like a growable array, or slab style allocator. So that if the estimated wort case usage is reached, the ‘static memory allocation’ can be relocated and enlarged or added onto with a new slab.
Your comments imply the sticking point may be the estimate of program memory usage, and that is a very fact/situational analysis.
> Do you know any good resources to get into that mode of thinking, including common patterns, for sombody who is used to malloc everything? I could easily come up with lots of examples in my current project where "it isn't that easy" and "you cannot know this up-front", even when re-designing the whole architecture, and trying to solve those problems feels like re-inventing a hundred wheels that have been invented before.
At some point you have to accept you will have hard limits in place. You have to signal back to whoever is sending you data that you cannot receive any more, they need to chunk it up smaller.
But, key lesson, you should already be doing that, but 99.999% of the time we get away without doing that because memory on modern systems is so large.
Drop down to embedded, and now your bit of code gets a 512 byte buffer to work with, and your paycheck depends on making it work, well, you will figure out a way to make it work.
For data coming in over the wire, generally it consists of packet size fields. It also means waiting for data to be processed until you accept more.
For data being passed around locally, well someone somewhere has a buffer. Careful tracking of ownership means you reduce unnecessary copying of data, especially if data is going to be processed and then discarded.
Strings are a separate problem, typically you will have some sort of string allocator dedicated just to strings because strings suck. But even then you will have a max length, but string sizes vary so much just using max length for every string on a screen will blow your memory budget away.
It took me awhile to get into the proper headspace, and it hurt my heart a little bit when I went back to GC'd languages where memory is thrown away willy nilly!
Do you know any good resources to get into that mode of thinking, including common patterns, for sombody who is used to malloc everything? I could easily come up with lots of examples in my current project where "it isn't that easy" and "you cannot know this up-front", even when re-designing the whole architecture, and trying to solve those problems feels like re-inventing a hundred wheels that have been invented before.