Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #74622 > unrolled thread

Memory and error management approaches

Back to article view | Back to comp.lang.c

Contents

  Memory and error management approaches "Morten W. Petersen" <morphex@gmail.com> - 2015-11-02 10:17 +0100
    Re: Memory and error management approaches Ian Collins <ian-news@hotmail.com> - 2015-11-02 22:25 +1300
      Re: Memory and error management approaches "Morten W. Petersen" <morphex@gmail.com> - 2015-11-02 10:36 +0100
        Re: Memory and error management approaches Ian Collins <ian-news@hotmail.com> - 2015-11-02 22:41 +1300
          Re: Memory and error management approaches "Morten W. Petersen" <morphex@gmail.com> - 2015-11-02 11:05 +0100
            Re: Memory and error management approaches Jorgen Grahn <grahn+nntp@snipabacken.se> - 2015-11-02 12:15 +0000
              Re: Memory and error management approaches "Morten W. Petersen" <morphex@gmail.com> - 2015-11-02 20:31 +0100
            Re: Memory and error management approaches Ian Collins <ian-news@hotmail.com> - 2015-11-03 09:48 +1300
              Re: Memory and error management approaches supercat@casperkitty.com - 2015-11-02 13:17 -0800
                Re: Memory and error management approaches Ian Collins <ian-news@hotmail.com> - 2015-11-03 10:29 +1300
                  Re: Memory and error management approaches supercat@casperkitty.com - 2015-11-02 13:48 -0800
              Re: Memory and error management approaches "Morten W. Petersen" <morphex@gmail.com> - 2015-11-02 22:31 +0100
                Re: Memory and error management approaches Ian Collins <ian-news@hotmail.com> - 2015-11-03 11:02 +1300
                  Re: Memory and error management approaches supercat@casperkitty.com - 2015-11-02 15:21 -0800
                Re: Memory and error management approaches <william@wilbur.25thandClement.com> - 2015-11-03 17:24 -0800
                  Re: Memory and error management approaches Ian Collins <ian-news@hotmail.com> - 2015-11-04 14:38 +1300
                    Re: Memory and error management approaches <william@wilbur.25thandClement.com> - 2015-11-04 10:48 -0800
                      Re: Memory and error management approaches supercat@casperkitty.com - 2015-11-04 11:28 -0800
                      Re: Memory and error management approaches James Kuyper <jameskuyper@verizon.net> - 2015-11-04 14:54 -0500
                        Re: Memory and error management approaches Gareth Owen <gwowen@gmail.com> - 2015-11-06 07:50 +0000
                          Re: Memory and error management approaches <william@wilbur.25thandClement.com> - 2015-11-06 11:40 -0800
                            Re: Memory and error management approaches Ian Collins <ian-news@hotmail.com> - 2015-11-07 09:38 +1300
                            Re: Memory and error management approaches Jorgen Grahn <grahn+nntp@snipabacken.se> - 2015-11-06 20:55 +0000
                              Re: Memory and error management approaches Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-11-07 02:23 -0800
                                Re: Memory and error management approaches Jorgen Grahn <grahn+nntp@snipabacken.se> - 2015-11-07 11:29 +0000
                                  Re: Memory and error management approaches Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-11-07 04:18 -0800
                                    Re: Memory and error management approaches asetofsymbols@gmail.com - 2015-11-07 05:41 -0800
                                      Re: Memory and error management approaches asetofsymbols@gmail.com - 2015-11-07 06:27 -0800
                                    Re: Memory and error management approaches Sjouke Burry <burrynulnulfour@ppllaanneett.nnll> - 2015-11-07 18:20 +0100
                                      Re: Memory and error management approaches Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-11-07 10:28 -0800
                                        Re: Memory and error management approaches gazelle@shell.xmission.com (Kenny McCormack) - 2015-11-07 19:01 +0000
                                Re: Memory and error management approaches raltbos@xs4all.nl (Richard Bos) - 2015-11-10 16:14 +0000
                                Re: Memory and error management approaches Gareth Owen <gwowen@gmail.com> - 2015-11-10 21:13 +0000
                      Re: Memory and error management approaches Jorgen Grahn <grahn+nntp@snipabacken.se> - 2015-11-10 19:26 +0000
                  Re: Memory and error management approaches Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-11-03 18:25 -0800
                  Re: Memory and error management approaches "Morten W. Petersen" <morphex@gmail.com> - 2015-11-06 10:05 +0100
      Re: Memory and error management approaches Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-11-02 02:15 -0800
        Re: Memory and error management approaches Jorgen Grahn <grahn+nntp@snipabacken.se> - 2015-11-02 12:01 +0000
          Re: Memory and error management approaches Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-11-02 06:05 -0800
            Re: Memory and error management approaches Jorgen Grahn <grahn+nntp@snipabacken.se> - 2015-11-02 17:18 +0000
              Re: Memory and error management approaches Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-11-02 09:27 -0800
                Re: Memory and error management approaches Jorgen Grahn <grahn+nntp@snipabacken.se> - 2015-11-02 23:02 +0000
                Re: Memory and error management approaches Ian Collins <ian-news@hotmail.com> - 2015-11-03 12:07 +1300
                  Re: Memory and error management approaches Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-11-02 15:23 -0800
              Re: Memory and error management approaches supercat@casperkitty.com - 2015-11-02 10:28 -0800
        Re: Memory and error management approaches Ian Collins <ian-news@hotmail.com> - 2015-11-03 09:34 +1300
          Re: Memory and error management approaches Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-11-02 14:05 -0800
            Re: Memory and error management approaches Ian Collins <ian-news@hotmail.com> - 2015-11-03 11:27 +1300
              Re: Memory and error management approaches Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-11-03 18:15 -0800

Page 3 of 3 — ← Prev page 1 2 [3]

#74670

On Monday, November 2, 2015 at 5:18:53 PM UTC, Jorgen Grahn wrote:
> On Mon, 2015-11-02, Malcolm McLean wrote:
> 
> > you've also got the problem of needing to handle failure on every
> > call.
> 
> That's always a problem in C though, no matter what you replace
> malloc with.
> 
Often you can establish a fairly tight upper bound on the allocations, based on
the input (e.g., we might know that we can never allocate more memory in total
than then size of the xml file, and never have more allocations than the number of
'<' characters in the file, we also know that apart from a few space characters and controls,
most of the file will be data that is allocated, and apart from a few '<' characters that are
escaped or comments etc, all the '<'s will represent nodes we need to allocate, so our
upper bound is tight). So then you can set up the custom allocator and, 
if successful in reserving memory, you know that you can never fail.

[toc] | [prev] | [next] | [standalone]

#74715

On Mon, 2015-11-02, Malcolm McLean wrote:
> On Monday, November 2, 2015 at 5:18:53 PM UTC, Jorgen Grahn wrote:
>> On Mon, 2015-11-02, Malcolm McLean wrote:
>> 
>> > you've also got the problem of needing to handle failure on every
>> > call.
>> 
>> That's always a problem in C though, no matter what you replace
>> malloc with.
>> 
> Often you can establish a fairly tight upper bound on the
> allocations, based on the input (e.g., we might know that we can
> never allocate more memory in total than then size of the xml file,
> and never have more allocations than the number of '<' characters in
> the file, we also know that apart from a few space characters and
> controls, most of the file will be data that is allocated, and apart
> from a few '<' characters that are escaped or comments etc, all the
> '<'s will represent nodes we need to allocate, so our upper bound is
> tight). So then you can set up the custom allocator and, if
> successful in reserving memory, you know that you can never fail.

I thought about that, but it seemed to me to be hard to find a /tight/
upper bound.

Depends on the style of parser he's aiming for, I guess, and I didn't
pay attention to that.  If the schema for a file is "a sequence of Foo
objects" and I get fed one Foo at a time, then I don't want to reserve
space for a million Foo objects.

If I instead get the tree to navigate at will, then I see two
allocations: one huge chunk which contains the whole file (or perhaps
an mmap area in certain OSes), and one containing the tree structure.

/Jorgen

-- 
  // Jorgen Grahn <grahn@  Oo  o.   .     .
\X/     snipabacken.se>   O  o   .

[toc] | [prev] | [next] | [standalone]

#74719

Malcolm McLean wrote:
> On Monday, November 2, 2015 at 5:18:53 PM UTC, Jorgen Grahn wrote:
>> On Mon, 2015-11-02, Malcolm McLean wrote:
>>
>>> you've also got the problem of needing to handle failure on every
>>> call.
>>
>> That's always a problem in C though, no matter what you replace
>> malloc with.
>>
> Often you can establish a fairly tight upper bound on the allocations, based on
> the input (e.g., we might know that we can never allocate more memory in total
> than then size of the xml file, and never have more allocations than the number of
> '<' characters in the file, we also know that apart from a few space characters and controls,
> most of the file will be data that is allocated, and apart from a few '<' characters that are
> escaped or comments etc, all the '<'s will represent nodes we need to allocate, so our
> upper bound is tight). So then you can set up the custom allocator and,
> if successful in reserving memory, you know that you can never fail.

Not for streamed input.

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#74722

On Monday, November 2, 2015 at 11:07:14 PM UTC, Ian Collins wrote:
> Malcolm McLean wrote:
> 
> > Often you can establish a fairly tight upper bound on the allocations, 
> > based on the input (e.g., we might know that we can never allocate 
> > more memory in total than then size of the xml file, and never have 
> > more allocations than the number of '<' characters in the file, we 
> > also know that apart from a few space characters and controls,
> > most of the file will be data that is allocated, and apart from a 
> > few '<' characters that are escaped or comments etc, all the '<'s 
> > will represent nodes we need to allocate, so our upper bound is tight).
> > So then you can set up the custom allocator and, if successful in
> > reserving memory, you know that you can never fail.
> 
> Not for streamed input.
> 
Yes, you need to be able to stat the file or load it into memory
for that technique to work. Which maybe isn't a good thing to require
of a general-purpose XML parser.
On the other hand, you can also write what I believe is called in
XML circles a "pull parser". That keeps only a small section of 
the file in memory, and allows you to read files which are too big 
to fit into main memory. Obviously you only get random access
in a small window. 
 

[toc] | [prev] | [next] | [standalone]

#74678

On Monday, November 2, 2015 at 11:18:53 AM UTC-6, Jorgen Grahn wrote:
> That's always a problem in C though, no matter what you replace
> malloc with.

If one is willing to pass a a "context" object through the call stack, one
could document that a memory API will do a longjmp via jmp_buff maintained
by that context object.  An API function could then allow user code to add
its own intermediate jmp_buff code in case any mid-level logic would require
cleanup in case of failure.

[toc] | [prev] | [next] | [standalone]

#74695

Malcolm McLean wrote:
> On Monday, November 2, 2015 at 9:26:06 AM UTC, Ian Collins wrote:
>> Morten W. Petersen wrote:
>>
>>> Any thoughts on this?  A while ago someone wrote on comp.lang.c about
>>> writing a custom memory manager for performance; is that something that
>>> could be coupled with all of this?
>>
>> Most of the platforms I'm familiar with provide specialised memory
>> allocator libraries, you shouldn't have to roll yet another one for
>> something as simple as XML parsing.
>>
> In an ideal world, that would be good advice. In the reality, it's bad advice.

In this case, it is good advice.  How many straight forward (I'd include 
XML parsing here) applications require a custom allocator?  Virtually none.

> An XML parser is just a medium-complexity component. But it's likely to handle
> huge amounts of memory, certainly it's likely that the programmer cannot control
> the size of data file he will be passed, at compile time. It's also likely to want to
> split that memory up into a huge number of tiny sections.
> So the case for some sort of custom scheme, rather than malloc() . free(), is strong.

I've been using and growing the same core XML DOM parser for 15 years or 
so and to date I haven't seen the need to mess with memory allocation. 
I've profiled it with typical files (in my case, SOAP) and with large 
Open Office documents (possibly the best stress and completeness tests 
for an XML parser).  memory allocation has never seen enough of an issue 
for me to change it.

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#74709

On Monday, November 2, 2015 at 8:34:32 PM UTC, Ian Collins wrote:
> Malcolm McLean wrote:
> > On Monday, November 2, 2015 at 9:26:06 AM UTC, Ian Collins wrote:
> >> Morten W. Petersen wrote:
> >>
> >>> Any thoughts on this?  A while ago someone wrote on comp.lang.c about
> >>> writing a custom memory manager for performance; is that something that
> >>> could be coupled with all of this?
> >>
> >> Most of the platforms I'm familiar with provide specialised memory
> >> allocator libraries, you shouldn't have to roll yet another one for
> >> something as simple as XML parsing.
> >>
> > In an ideal world, that would be good advice. In the reality, it's
> > bad advice.
> 
> In this case, it is good advice.  How many straight forward (I'd include 
> XML parsing here) applications require a custom allocator?  Virtually 
> none.
>
The bad advice isn't to stick with malloc() until proved inadequate.
It's to introduce a third party dependency with a ready-made special-
purpose allocator.
  

[toc] | [prev] | [next] | [standalone]

#74711

Malcolm McLean wrote:
> On Monday, November 2, 2015 at 8:34:32 PM UTC, Ian Collins wrote:
>> Malcolm McLean wrote:
>>> On Monday, November 2, 2015 at 9:26:06 AM UTC, Ian Collins wrote:
>>>> Morten W. Petersen wrote:
>>>>
>>>>> Any thoughts on this?  A while ago someone wrote on comp.lang.c about
>>>>> writing a custom memory manager for performance; is that something that
>>>>> could be coupled with all of this?
>>>>
>>>> Most of the platforms I'm familiar with provide specialised memory
>>>> allocator libraries, you shouldn't have to roll yet another one for
>>>> something as simple as XML parsing.
>>>>
>>> In an ideal world, that would be good advice. In the reality, it's
>>> bad advice.
>>
>> In this case, it is good advice.  How many straight forward (I'd include
>> XML parsing here) applications require a custom allocator?  Virtually
>> none.
>>
> The bad advice isn't to stick with malloc() until proved inadequate.
> It's to introduce a third party dependency with a ready-made special-
> purpose allocator.

I didn't suggest introducing a third party dependency.

You would choose the allocator based on how an application uses the 
parser library.  One application might use multiple threads to parse 
many documents in parallel, another might be parsing a single monster 
document.  The allocator requirements for these would be very different. 
  For the former you might want to consider an allocator optimised for 
threading and for the latter the standard allocator.  A decent platform 
will give you that choice.

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#74793

On Monday, November 2, 2015 at 10:27:20 PM UTC, Ian Collins wrote:
> Malcolm McLean wrote:
>
> I didn't suggest introducing a third party dependency.
> 
> You would choose the allocator based on how an application uses the 
> parser library.  One application might use multiple threads to parse 
> many documents in parallel, another might be parsing a single monster 
> document.  The allocator requirements for these would be very different. 
>   For the former you might want to consider an allocator optimised for 
> threading and for the latter the standard allocator.  A decent platform 
> will give you that choice.
> 
If by "custom allocator" you mean something that is called malloc()
but has a tweakable algorithm, I agree, that's the best of both
worlds. Or overloading the C++ new operator (done that, but it was
more fiddly than it was worth).

[toc] | [prev] | [standalone]

Page 3 of 3 — ← Prev page 1 2 [3]

Back to top | Article view | comp.lang.c

csiph-web