Mark Jame wrote:At the moment I am looking at how to deal with huge numbers of objects, too many to fit into memory (pretend my memory will never be big enough) and files that are too large to fit into memory.
Mark Jame wrote:Perhaps another example may help. I may have a program that polls a data directory looking for files (of serialized objects) that are created by an external program. My program has access to the API (classes only, that define the objects, a Widget class for example) so I know what the objects look like and how to get information about them (getters for example). The job of my program is to deserialize the data files, recording some information about each object then discard them. My program has no idea how big the files will be or how many objects they may contain.
Winston Gutkowski wrote:If the things you're deserializating are fairly simple and don't need much in the way of context in order to "record" what you need, the chances are that you can just deal with them individually in batches; but if there's anything more needed...
Mark Jame wrote:For example you may work on a system where one part creates files and another part reads and processes thoses files. In this case you know before you start to design the reading part that the size and/or number of files created are within certain limits.
I know for example I cannot create a simple object, add it to a list and repeat until there are billiions of simple objects in the list because I would run out of memory before the list is complete and before I can serialize it.
Mark Jame wrote:I have then tried to deserialize that large object on a (simulated) machine with much less memory, and as I thought it ran out of memory. This was one of the examples I was trying to look into. In this case I do not believe there is any way to deserialize the file?
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime. |