Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.lang.c++ > #9464 > unrolled thread
| Started by | Paul <pchristor@yahoo.co.uk> |
|---|---|
| First post | 2011-08-29 23:26 -0700 |
| Last post | 2011-08-30 16:45 -0700 |
| Articles | 20 on this page of 197 — 20 participants |
Back to article view | Back to comp.lang.c++
bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-29 23:26 -0700
Re: bad alloc Krice <paulkp@mbnet.fi> - 2011-08-30 02:14 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 06:26 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-08-30 14:34 +0100
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 06:54 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-08-30 15:02 +0100
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-08-30 15:06 +0100
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 07:20 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-08-30 15:31 +0100
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 07:47 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-08-30 15:52 +0100
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 08:00 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-08-30 16:15 +0100
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 10:31 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-08-30 18:58 +0100
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-08-30 19:18 +0100
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 16:17 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-08-31 00:56 +0100
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 17:11 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-08-31 01:24 +0100
Re: bad alloc Ian Collins <ian-news@hotmail.com> - 2011-08-31 12:33 +1200
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 18:38 -0700
Re: bad alloc Noah Roberts <roberts.noah@gmail.com> - 2011-08-30 20:07 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-31 04:04 -0700
Re: bad alloc Noah Roberts <roberts.noah@gmail.com> - 2011-08-31 09:17 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-08-31 14:59 +0100
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-01 00:50 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-09-01 12:41 +0100
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-01 05:00 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-09-01 13:59 +0100
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-01 06:35 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-09-01 14:50 +0100
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-01 13:51 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-09-01 21:58 +0100
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-01 14:05 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-09-01 22:08 +0100
Re: bad alloc Nick Keighley <nick_keighley_nospam@hotmail.com> - 2011-09-02 03:16 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-02 04:12 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-09-02 15:08 +0100
Re: bad alloc Nick Keighley <nick_keighley_nospam@hotmail.com> - 2011-09-02 03:07 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-02 04:15 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 11:21 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 10:46 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-08-30 18:58 +0100
Re: bad alloc "Waldek M." <wm@localhost.localdomain> - 2011-08-31 20:58 +0200
Re: bad alloc "io_x" <a@b.c.invalid> - 2011-09-03 10:13 +0200
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-05 18:53 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 16:28 -0700
Re: bad alloc Nick Keighley <nick_keighley_nospam@hotmail.com> - 2011-09-01 00:49 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-01 04:50 -0700
Re: bad alloc Nick Keighley <nick_keighley_nospam@hotmail.com> - 2011-09-02 03:17 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-02 04:21 -0700
Re: bad alloc James Kanze <james.kanze@gmail.com> - 2011-09-01 16:23 -0700
Re: bad alloc Nick Keighley <nick_keighley_nospam@hotmail.com> - 2011-09-02 03:19 -0700
Re: bad alloc James Kanze <james.kanze@gmail.com> - 2011-09-03 15:37 -0700
Re: bad alloc Dombo <dombo@disposable.invalid> - 2011-09-04 09:25 +0200
Re: bad alloc James Kanze <james.kanze@gmail.com> - 2011-09-01 16:20 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-08-30 06:54 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 07:12 -0700
Re: bad alloc Dombo <dombo@disposable.invalid> - 2011-08-30 22:58 +0200
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 15:56 -0700
Re: bad alloc yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2011-08-31 09:28 +0000
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-31 04:20 -0700
Re: bad alloc yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2011-09-02 14:57 +0000
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-02 08:35 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-08-31 04:25 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-08-30 03:26 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 05:54 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-08-30 14:30 +0100
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 07:00 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-08-31 00:37 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-31 04:26 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-08-30 07:03 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 07:57 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-08-30 16:04 +0100
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 08:15 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-08-30 16:20 +0100
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 10:37 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-08-30 18:41 +0100
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 10:49 -0700
Re: bad alloc Asger-P <junk@asger-p.dk> - 2011-08-31 10:33 +0200
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-31 04:22 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-08-30 16:04 -0700
Re: bad alloc Noah Roberts <roberts.noah@gmail.com> - 2011-08-30 20:02 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-08-30 20:33 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-08-31 01:30 -0700
Re: bad alloc yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2011-08-31 09:23 +0000
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-08-31 04:21 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-08-31 04:47 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-31 05:35 -0700
Re: bad alloc Nick Keighley <nick_keighley_nospam@hotmail.com> - 2011-09-01 01:08 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-01 03:56 -0700
Re: bad alloc Nick Keighley <nick_keighley_nospam@hotmail.com> - 2011-09-02 04:18 -0700
Re: bad alloc Dombo <dombo@disposable.invalid> - 2011-09-03 18:37 +0200
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-03 13:39 -0700
Re: bad alloc red floyd <no.spam.here@its.invalid> - 2011-09-03 13:45 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-03 13:51 -0700
Re: bad alloc Sjouke Burry <burryNULNULFOUR@PPLLAANNEETT.NNLL> - 2011-09-04 00:14 +0200
Re: bad alloc James Kanze <james.kanze@gmail.com> - 2011-09-03 15:51 -0700
Re: bad alloc Ian Collins <ian-news@hotmail.com> - 2011-09-04 12:15 +1200
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-04 05:00 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-05 18:13 -0700
Re: bad alloc James Kanze <james.kanze@gmail.com> - 2011-09-03 15:45 -0700
Re: bad alloc yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2011-09-02 15:40 +0000
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-02 09:33 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-02 14:06 -0700
Re: bad alloc "io_x" <a@b.c.invalid> - 2011-09-03 10:12 +0200
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-08-31 04:21 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-08-31 01:22 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-08-31 05:18 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-08-31 06:36 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-08-31 09:33 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-09-01 00:20 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-01 04:25 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-09-01 12:51 +0100
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-01 06:04 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-09-01 06:26 -0700
Re: bad alloc James Kanze <james.kanze@gmail.com> - 2011-09-01 16:58 -0700
Re: bad alloc Joshua Maurice <joshuamaurice@gmail.com> - 2011-09-01 17:25 -0700
Re: bad alloc James Kanze <james.kanze@gmail.com> - 2011-09-03 16:02 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-05 18:20 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-05 19:13 -0700
Re: bad alloc James Kanze <james.kanze@gmail.com> - 2011-09-06 15:39 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-06 17:26 -0700
Re: bad alloc James Kanze <james.kanze@gmail.com> - 2011-09-06 15:35 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-06 17:28 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-01 21:01 -0700
Re: bad alloc James Kanze <james.kanze@gmail.com> - 2011-09-03 16:14 -0700
Re: bad alloc "io_x" <a@b.c.invalid> - 2011-09-05 10:31 +0200
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-05 18:36 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-06 04:09 -0700
Re: bad alloc James Kanze <james.kanze@gmail.com> - 2011-09-06 15:49 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-06 17:11 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-09-07 00:24 -0700
Re: bad alloc Joshua Maurice <joshuamaurice@gmail.com> - 2011-08-31 09:59 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-08-31 11:24 -0700
Re: bad alloc Joshua Maurice <joshuamaurice@gmail.com> - 2011-09-01 16:29 -0700
Re: bad alloc Joshua Maurice <joshuamaurice@gmail.com> - 2011-09-01 16:42 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-01 21:37 -0700
Re: bad alloc Ian Collins <ian-news@hotmail.com> - 2011-09-02 17:45 +1200
Re: bad alloc James Kanze <james.kanze@gmail.com> - 2011-09-03 16:20 -0700
Re: bad alloc Ian Collins <ian-news@hotmail.com> - 2011-09-04 12:03 +1200
Re: bad alloc James Kanze <james.kanze@gmail.com> - 2011-09-04 10:56 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-05 18:50 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-09-06 00:12 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-06 03:04 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-09-07 00:08 -0700
Re: bad alloc Nick Keighley <nick_keighley_nospam@hotmail.com> - 2011-09-07 02:33 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-07 04:28 -0700
Re: bad alloc yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2011-09-06 13:43 +0000
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-06 07:37 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-09-07 00:11 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-07 04:29 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-09-07 05:38 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-07 18:17 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-09-08 00:45 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-06 06:58 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-06 09:21 -0700
Re: bad alloc "Fred Zwarts \(KVI\)" <F.Zwarts@KVI.nl> - 2011-09-07 09:10 +0200
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-07 04:18 -0700
Re: bad alloc yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2011-09-07 10:34 +0000
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-07 04:00 -0700
Re: bad alloc yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2011-09-08 12:11 +0000
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-08 06:27 -0700
Re: bad alloc James Kanze <james.kanze@gmail.com> - 2011-09-06 15:52 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-06 17:16 -0700
Re: bad alloc Nick Keighley <nick_keighley_nospam@hotmail.com> - 2011-09-07 02:39 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-07 04:14 -0700
Re: bad alloc Ian Collins <ian-news@hotmail.com> - 2011-09-06 17:00 +1200
Re: bad alloc Miles Bader <miles@gnu.org> - 2011-09-06 14:51 +0900
Re: bad alloc James Kanze <james.kanze@gmail.com> - 2011-09-06 15:55 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-05 18:44 -0700
Re: bad alloc yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2011-09-07 10:20 +0000
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-07 03:40 -0700
Re: bad alloc yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2011-09-13 17:06 +0000
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-13 12:09 -0700
Re: bad alloc "Paul" <pchristor@yahoo.co.uk> - 2011-09-14 00:18 +0100
Re: bad alloc yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2011-09-14 15:11 +0000
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-14 11:17 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-09-15 04:17 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-15 06:50 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-09-15 07:31 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-15 07:59 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-09-16 00:19 -0700
Re: bad alloc Noah Roberts <roberts.noah@gmail.com> - 2011-09-17 10:13 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-09-18 10:07 -0700
Re: bad alloc yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2011-09-16 15:41 +0000
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-16 11:26 -0700
Re: bad alloc Joshua Maurice <joshuamaurice@gmail.com> - 2011-09-16 15:38 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-17 04:25 -0700
Re: bad alloc Joshua Maurice <joshuamaurice@gmail.com> - 2011-09-01 23:17 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-02 04:03 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-02 04:37 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-09-02 15:15 +0100
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-02 07:35 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-08-30 16:06 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 16:45 -0700
Page 9 of 10 — ← Prev page 1 … 7 8 [9] 10 Next page →
| From | yatremblay@bel1lin202.(none) (Yannick Tremblay) |
|---|---|
| Date | 2011-09-07 10:34 +0000 |
| Message-ID | <j47hbq$l9b$1@dont-email.me> |
| In reply to | #9807 |
In article <b5f2c1fd-e60f-43ca-bcf0-88a9ba5a7f42@e8g2000vbc.googlegroups.com>, Adam Skutt <askutt@gmail.com> wrote: >On Sep 6, 9:43am, yatremblay@bel1lin202.(none) (Yannick Tremblay) >wrote: >> In article <2ab097e7-57e7-4e01-9d05-5630fe255...@f41g2000yqh.googlegroups.com>, >> Adam Skutt <ask...@gmail.com> wrote: >> So, if I understand you correctly, you are saying that you must always >> setup some artificial limits to the external inputs and set >> artificially low so that no matter what is happening in the rest of >> the system, the program will never run out of resources.... > >Yes, if you want to isolate failure processing one request from >another (esp. in a threaded system), you set limits on how much input >can be provided with each request. You reject requests that exceed >the limit. > >However, this doesn't mean the limits are set artificially low. >Usually memory isn't your bounding constraint, so you'll run out of >database handles, CPU, etc. long before you run out of memory. Per- >request memory limits can be generous and not create an issue. > >Of course, I'm personally fine with unbounded input as long as the >user understands the system will break at some point and they get to >keep both of the pieces. > >> This seems like a very bad proposition to me. The only way to win is >> to reserve and grab at startup time all of the resources you might >> potentially ever need in order to meet the worse case scenario of your >> inputs. > >No, I'm not sure why you think this follows in the least. I also >think I've explained why this isn't the case several times already. >If you have per-request bounds and the OS can't give you memory when >you ask for it you either need to rewrite your code (so you'll be >terminating anyway) or the OS is likely to terminate (so you'll be >terminating anyway). Given a multitasking OS running on a more than one process, there is no point in time where you can know how much resources are available. What was true a clock cycle ago may now be false. So assuming that processing the inputs require some resources that varies with the input complexity, there is no point in time that you can *know* that there will be enough resource to process even the simplest of input, unless processing it does not require you to acquire any new resources at all. Ergo: in order to validate an input and be sure that you will be able to process it, you need to already have acquired all the resources you will ever need to process this input. Ergo: in order to document the input limits, you need to have already acquired the resources that you will ever need in order to process the limit. Yan
[toc] | [prev] | [next] | [standalone]
| From | Adam Skutt <askutt@gmail.com> |
|---|---|
| Date | 2011-09-07 04:00 -0700 |
| Message-ID | <45436526-6273-4e83-b074-488db5914edb@a7g2000yqb.googlegroups.com> |
| In reply to | #9885 |
On Sep 7, 6:34 am, yatremblay@bel1lin202.(none) (Yannick Tremblay) wrote: > In article <b5f2c1fd-e60f-43ca-bcf0-88a9ba5a7...@e8g2000vbc.googlegroups.com>, Adam Skutt <ask...@gmail.com> wrote: >> No, I'm not sure why you think this follows in the least. I also >> think I've explained why this isn't the case several times already. >> If you have per-request bounds and the OS can't give you memory when >> you ask for it you either need to rewrite your code (so you'll be >> terminating anyway) or the OS is likely to terminate (so you'll be >> terminating anyway). > > Given a multitasking OS running on a more than one process, there is > no point in time where you can know how much resources are available. > What was true a clock cycle ago may now be false. There's no way for the application to generally know. There's plenty of ways for the system designer to know or make more than reasonable assurances, if it comes to that. The reality of the matter is one can grant sufficient resources to the system such that exhaustion would be considered catastrophic. Hence my second reason why you terminate: the machine ran out of resources and some human being needs to add more resources. That's the best you can do, even preallocating resources. You don't know if the kernel will overcommit, you don't know if the kernel will run out of resources for itself, after all. Preallocation only works if everything your program requires to run preallocates too. Modern, mainstream operating system kernels do not preallocate most of their resources. You should take that in to serious consideration when deciding what applications should do. > Ergo: in order to document the input limits, you need to have already > acquired the resources that you will ever need in order to process the > limit. And may prevent the program from running altogether. If your goal is to serve as many requests as possible before failure, then is worse than what I suggested, not better. If there's not enough memory, then you'll serve 0. I'll serve some number between 0 and all of them. Adam
[toc] | [prev] | [next] | [standalone]
| From | yatremblay@bel1lin202.(none) (Yannick Tremblay) |
|---|---|
| Date | 2011-09-08 12:11 +0000 |
| Message-ID | <j4abdc$j2e$1@dont-email.me> |
| In reply to | #9888 |
In article <45436526-6273-4e83-b074-488db5914edb@a7g2000yqb.googlegroups.com>, Adam Skutt <askutt@gmail.com> wrote: >On Sep 7, 6:34am, yatremblay@bel1lin202.(none) (Yannick Tremblay) >wrote: >> In article <b5f2c1fd-e60f-43ca-bcf0-88a9ba5a7...@e8g2000vbc.googlegroups.com>, >Adam Skutt <ask...@gmail.com> wrote: >>> No, I'm not sure why you think this follows in the least. I also >>> think I've explained why this isn't the case several times already. >>> If you have per-request bounds and the OS can't give you memory when >>> you ask for it you either need to rewrite your code (so you'll be >>> terminating anyway) or the OS is likely to terminate (so you'll be >>> terminating anyway). >> >> Given a multitasking OS running on a more than one process, there is >> no point in time where you can know how much resources are available. >> What was true a clock cycle ago may now be false. > >There's no way for the application to generally know. There's plenty >of ways for the system designer to know or make more than reasonable >assurances, if it comes to that. The reality of the matter is one can >grant sufficient resources to the system such that exhaustion would be >considered catastrophic. Hence my second reason why you terminate: >the machine ran out of resources and some human being needs to add >more resources. > >That's the best you can do, even preallocating resources. You don't >know if the kernel will overcommit, you don't know if the kernel will >run out of resources for itself, after all. Preallocation only works >if everything your program requires to run preallocates too. Modern, >mainstream operating system kernels do not preallocate most of their >resources. You should take that in to serious consideration when >deciding what applications should do. > >> Ergo: in order to document the input limits, you need to have already >> acquired the resources that you will ever need in order to process the >> limit. > >And may prevent the program from running altogether. If your goal is >to serve as many requests as possible before failure, then is worse >than what I suggested, not better. If there's not enough memory, then >you'll serve 0. I'll serve some number between 0 and all of them. I am confused. you are the one that said that in order to avoid OOM error due to input complexity, all you need to do is to setup upfront limit on allowable input. (since this is what other engineering discipline do) My reply simply highlighted that this is impossible to do correctly unless you pre-acquire all resources you will ever need. Are you now suggesting that you should not set upfront limits?
[toc] | [prev] | [next] | [standalone]
| From | Adam Skutt <askutt@gmail.com> |
|---|---|
| Date | 2011-09-08 06:27 -0700 |
| Message-ID | <c5837c6a-51ed-4ce4-92f7-b53da5796044@1g2000vbu.googlegroups.com> |
| In reply to | #9919 |
On Sep 8, 8:11 am, yatremblay@bel1lin202.(none) (Yannick Tremblay) wrote: > In article <45436526-6273-4e83-b074-488db5914...@a7g2000yqb.googlegroups.com>, > Adam Skutt <ask...@gmail.com> wrote: > > > > > > > > > > >On Sep 7, 6:34am, yatremblay@bel1lin202.(none) (Yannick Tremblay) > >wrote: > >> In article <b5f2c1fd-e60f-43ca-bcf0-88a9ba5a7...@e8g2000vbc.googlegroups.com>, > >Adam Skutt <ask...@gmail.com> wrote: > >>> No, I'm not sure why you think this follows in the least. I also > >>> think I've explained why this isn't the case several times already. > >>> If you have per-request bounds and the OS can't give you memory when > >>> you ask for it you either need to rewrite your code (so you'll be > >>> terminating anyway) or the OS is likely to terminate (so you'll be > >>> terminating anyway). > > >> Given a multitasking OS running on a more than one process, there is > >> no point in time where you can know how much resources are available. > >> What was true a clock cycle ago may now be false. > > >There's no way for the application to generally know. There's plenty > >of ways for the system designer to know or make more than reasonable > >assurances, if it comes to that. The reality of the matter is one can > >grant sufficient resources to the system such that exhaustion would be > >considered catastrophic. Hence my second reason why you terminate: > >the machine ran out of resources and some human being needs to add > >more resources. > > >That's the best you can do, even preallocating resources. You don't > >know if the kernel will overcommit, you don't know if the kernel will > >run out of resources for itself, after all. Preallocation only works > >if everything your program requires to run preallocates too. Modern, > >mainstream operating system kernels do not preallocate most of their > >resources. You should take that in to serious consideration when > >deciding what applications should do. > > >> Ergo: in order to document the input limits, you need to have already > >> acquired the resources that you will ever need in order to process the > >> limit. > > >And may prevent the program from running altogether. If your goal is > >to serve as many requests as possible before failure, then is worse > >than what I suggested, not better. If there's not enough memory, then > >you'll serve 0. I'll serve some number between 0 and all of them. > > I am confused. you are the one that said that in order to avoid OOM > error due to input complexity, all you need to do is to setup upfront > limit on allowable input. (since this is what other engineering > discipline do) > > My reply simply highlighted that this is impossible to do correctly > unless you pre-acquire all resources you will ever need. > And my reply demonstrated how and why this is incorrect. > Are you now suggesting that you should not set upfront limits? No, I'm suggesting trying to preallocate the resources for those limits is normally pointless. I'm saying that if you choose to serve a max of 100 connections transferring no more than 5MB of data (so roughly a 500MB limit) and then run it on a system that doesn't have at least that much memory, you deserve what you get. This is obvious and goes without saying, even if I didn't mention it explicitly before. Preventing OOM from occurring is a systems-level problem. You cannot meaningfully solve it while discussing a single software application removed from the system it runs on. You have to be able to make certain assumptions about systems-level behavior in order to come up with a workable solution. Adam
[toc] | [prev] | [next] | [standalone]
| From | James Kanze <james.kanze@gmail.com> |
|---|---|
| Date | 2011-09-06 15:52 -0700 |
| Message-ID | <9f7f742b-2971-420e-a1d3-0785de9d90ec@t3g2000vbe.googlegroups.com> |
| In reply to | #9773 |
On Sep 6, 2:50 am, Adam Skutt <ask...@gmail.com> wrote: > On Sep 4, 1:56 pm, James Kanze <james.ka...@gmail.com> wrote: > > On Sep 4, 1:03 am, Ian Collins <ian-n...@hotmail.com> wrote: > > > On 09/ 4/11 11:20 AM, James Kanze wrote: > > > > On Sep 2, 6:45 am, Ian Collins<ian-n...@hotmail.com> wrote: > > > >> On 09/ 2/11 04:37 PM, Adam Skutt wrote: > > > > [...] > > > >> I agree. On a descent hosted environment, memory exhaustion is usually > > > >> down to either a system wide problem, or a programming error. > > > > Or an overly complex client request. > > > Not spotting those is a programming (or specification) error! > > And the way you spot them is by catching bad_alloc:-). > No, you set upfront bounds on allowable inputs. That's another possible solution. Not acceptable for all applications, and sometimes more difficult to implement than handling OOM. Applications vary. > This is what other > engineering disciplines do, Actually, they don't. There's a good reason why soldiers are required to break step when crossing a bridge. -- James Kanze
[toc] | [prev] | [next] | [standalone]
| From | Adam Skutt <askutt@gmail.com> |
|---|---|
| Date | 2011-09-06 17:16 -0700 |
| Message-ID | <a7273fdc-f3af-43ed-b06d-545bcd2eb278@f41g2000yqh.googlegroups.com> |
| In reply to | #9844 |
On Sep 6, 6:52 pm, James Kanze <james.ka...@gmail.com> wrote: > That's another possible solution. Not acceptable for all > applications, and sometimes more difficult to implement than > handling OOM. Applications vary. Then provide an example of when it's more difficult. I'll wait. > > > This is what other > > engineering disciplines do, > > Actually, they don't. There's a good reason why soldiers are > required to break step when crossing a bridge. And if you think that reason is a counterexample to what I said, then you're simply crazy. Walking over a bridge where you don't know how much wait it is designed to support (nor can you) isn't relevant here; we're the ones building the bridge, so we get to set the limits and tell the world as necessary. Please don't be so disingenuous. Adam
[toc] | [prev] | [next] | [standalone]
| From | Nick Keighley <nick_keighley_nospam@hotmail.com> |
|---|---|
| Date | 2011-09-07 02:39 -0700 |
| Message-ID | <ba8b2b38-8239-452a-b04d-4c8f457bd790@o26g2000vbi.googlegroups.com> |
| In reply to | #9854 |
On Sep 7, 1:16 am, Adam Skutt <ask...@gmail.com> wrote: > On Sep 6, 6:52 pm, James Kanze <james.ka...@gmail.com> wrote: > > > [limiting inputs] is what other engineering disciplines do, > > > Actually, they don't. There's a good reason why soldiers are > > required to break step when crossing a bridge. > > And if you think that reason is a counterexample to what I said, then > you're simply crazy. Walking over a bridge where you don't know how > much wait it is designed to support (nor can you) isn't relevant here; it's not about weight it's about resonance. Though how the officer estimates the bridge's resonant frequency is beyond me... > we're the ones building the bridge, so we get to set the limits and > tell the world as necessary. Please don't be so disingenuous.
[toc] | [prev] | [next] | [standalone]
| From | Adam Skutt <askutt@gmail.com> |
|---|---|
| Date | 2011-09-07 04:14 -0700 |
| Message-ID | <b5ba0e29-128c-4bf5-861a-ad977f37b4c0@d25g2000yqh.googlegroups.com> |
| In reply to | #9879 |
On Sep 7, 5:39 am, Nick Keighley <nick_keighley_nos...@hotmail.com> wrote: > On Sep 7, 1:16 am, Adam Skutt <ask...@gmail.com> wrote: > > > On Sep 6, 6:52 pm, James Kanze <james.ka...@gmail.com> wrote: > > > > [limiting inputs] is what other engineering disciplines do, > > > > Actually, they don't. There's a good reason why soldiers are > > > required to break step when crossing a bridge. > > > And if you think that reason is a counterexample to what I said, then > > you're simply crazy. Walking over a bridge where you don't know how > > much wait it is designed to support (nor can you) isn't relevant here; > > it's not about weight it's about resonance. That's a common statement, but it's not really, really true. Perhaps counter-intuitively, when bridges start to sway, people tend to match the swaying, potentially exacerbating the problem anyway. While it is certainly possible for marching cadence to cause resonance on a badly built or damaged bridge, it's possible for it to happen even when step is broken due to human nature. As a result, bridges have collapsed on troops, most likely due to resonance, even when they heeded this advice. > Though how the officer > estimates the bridge's resonant frequency is beyond me... If the swaying is getting worse, you have a problem. Unless the bridge is about to structurally fail anyway (in which case it won't support the dynamic load), it will get pretty severe before the failure. Unsurprisingly, when this has happened to marching troops in history, there were plenty of warning signs before the actual collapse. Nevertheless, it's still not the least bit relevant to my point. Adam
[toc] | [prev] | [next] | [standalone]
| From | Ian Collins <ian-news@hotmail.com> |
|---|---|
| Date | 2011-09-06 17:00 +1200 |
| Message-ID | <9clnn7F72aU2@mid.individual.net> |
| In reply to | #9738 |
On 09/ 5/11 05:56 AM, James Kanze wrote: > On Sep 4, 1:03 am, Ian Collins<ian-n...@hotmail.com> wrote: >> On 09/ 4/11 11:20 AM, James Kanze wrote: > >>> On Sep 2, 6:45 am, Ian Collins<ian-n...@hotmail.com> wrote: >>>> On 09/ 2/11 04:37 PM, Adam Skutt wrote: > >>> [...] >>>> I agree. On a descent hosted environment, memory exhaustion is usually >>>> down to either a system wide problem, or a programming error. > >>> Or an overly complex client request. > >> Not spotting those is a programming (or specification) error! > > And the way you spot them is by catching bad_alloc:-). > > Seriously, the problem is very much like that of a compiler. > Nest parentheses too deep, and the compiler will run out of > memory. There are two solutions: specify an artificial nesting > limit, which you know you can handle (regardless of how many > connections are active, etc.), or react when you run out of > resources. There are valid arguments for both solutions, and > I've used both, in different applications. I have also seen both in compilers. For example last time I played, g++ didn't have a recursion limit for templates (as used in meta-programming) while Sun CC does. -- Ian Collins
[toc] | [prev] | [next] | [standalone]
| From | Miles Bader <miles@gnu.org> |
|---|---|
| Date | 2011-09-06 14:51 +0900 |
| Message-ID | <buoipp6w759.fsf@dhlpc061.dev.necel.com> |
| In reply to | #9775 |
Ian Collins <ian-news@hotmail.com> writes: > I have also seen both in compilers. For example last time I played, g++ > didn't have a recursion limit for templates (as used in > meta-programming) while Sun CC does. [It does now tho...] -miles -- Discriminate, v.i. To note the particulars in which one person or thing is, if possible, more objectionable than another.
[toc] | [prev] | [next] | [standalone]
| From | James Kanze <james.kanze@gmail.com> |
|---|---|
| Date | 2011-09-06 15:55 -0700 |
| Message-ID | <ae79f956-d0ad-47ce-8bed-a4244863e523@o9g2000vbo.googlegroups.com> |
| In reply to | #9775 |
On Sep 6, 6:00 am, Ian Collins <ian-n...@hotmail.com> wrote:
> On 09/ 5/11 05:56 AM, James Kanze wrote:
[...]
> > Seriously, the problem is very much like that of a compiler.
> > Nest parentheses too deep, and the compiler will run out of
> > memory. There are two solutions: specify an artificial nesting
> > limit, which you know you can handle (regardless of how many
> > connections are active, etc.), or react when you run out of
> > resources. There are valid arguments for both solutions, and
> > I've used both, in different applications.
> I have also seen both in compilers. For example last time I played, g++
> didn't have a recursion limit for templates (as used in
> meta-programming) while Sun CC does.
What you mean, of course, is that the recursion limit in g++ is
determined by the available resources; not that there isn't
one:-).
There are arguments for both strategies. I've used both,
depending on the application. There is no one "correct"
solution.
--
James Kanze
[toc] | [prev] | [next] | [standalone]
| From | Adam Skutt <askutt@gmail.com> |
|---|---|
| Date | 2011-09-05 18:44 -0700 |
| Message-ID | <1cdb8d42-faa0-4001-91f3-fbb58309f0ee@d25g2000yqh.googlegroups.com> |
| In reply to | #9715 |
On Sep 3, 7:20 pm, James Kanze <james.ka...@gmail.com> wrote: > That's the correct response to a programming error or a system > wide problem. For a request which is part of a DOS attack, it > seems to be playing into the attacker's hand, and for a simply > unreasonable client request, it does seem unnecessary to abort > all other client connections. One, it's not a given that it aborts other client connections, after all. There could be a higher level mechanism that provides the illusion of a persistent connection even after failover. Second, it may be unnecessary, but just because it's unnecessary it doesn't follow that: 1) The value of trying to handle OOM, instead of terminating, exceeds it cost. 2) OOM can be handled in a robust fashion. > What's wrong with just responding > with an "insufficient resources" error? > That's not enough to get what you want, which is isolation of the failure. > > That's why robust servers run on dedicated machines (where they > are the only "application"). No, that's not a requirement for robustness. Frequently, it makes the situation worse. Trivial example: separating your Kerberos5 and LDAP servers rarely does anything for robustness. Especially if you stored your K5 database in LDAP. Adam
[toc] | [prev] | [next] | [standalone]
| From | yatremblay@bel1lin202.(none) (Yannick Tremblay) |
|---|---|
| Date | 2011-09-07 10:20 +0000 |
| Message-ID | <j47gi0$fms$1@dont-email.me> |
| In reply to | #9771 |
In article <1cdb8d42-faa0-4001-91f3-fbb58309f0ee@d25g2000yqh.googlegroups.com>, Adam Skutt <askutt@gmail.com> wrote: >On Sep 3, 7:20pm, James Kanze <james.ka...@gmail.com> wrote: > >> That's the correct response to a programming error or a system >> wide problem. For a request which is part of a DOS attack, it >> seems to be playing into the attacker's hand, and for a simply >> unreasonable client request, it does seem unnecessary to abort >> all other client connections. > >One, it's not a given that it aborts other client connections, after >all. There could be a higher level mechanism that provides the >illusion of a persistent connection even after failover. Second, it >may be unnecessary, but just because it's unnecessary it doesn't >follow that: >1) The value of trying to handle OOM, instead of terminating, exceeds >it cost. >2) OOM can be handled in a robust fashion. This arguments can go both ways which you seem to refuse to accept: It doesn't follow that: 1) The cost of trying to handle *some* OOM error, instead of terminating, exceeds the value. 2) That no OOM errors whatsoever can be handled in a robust fashion. Yannick
[toc] | [prev] | [next] | [standalone]
| From | Adam Skutt <askutt@gmail.com> |
|---|---|
| Date | 2011-09-07 03:40 -0700 |
| Message-ID | <56e11c1e-de9e-4e8e-8d71-a9a501e4ce17@b20g2000vbz.googlegroups.com> |
| In reply to | #9883 |
On Sep 7, 6:20 am, yatremblay@bel1lin202.(none) (Yannick Tremblay) wrote: > In article <1cdb8d42-faa0-4001-91f3-fbb58309f...@d25g2000yqh.googlegroups.com>, > Adam Skutt <ask...@gmail.com> wrote: > >On Sep 3, 7:20pm, James Kanze <james.ka...@gmail.com> wrote: > > >> That's the correct response to a programming error or a system > >> wide problem. For a request which is part of a DOS attack, it > >> seems to be playing into the attacker's hand, and for a simply > >> unreasonable client request, it does seem unnecessary to abort > >> all other client connections. > > >One, it's not a given that it aborts other client connections, after > >all. There could be a higher level mechanism that provides the > >illusion of a persistent connection even after failover. Second, it > >may be unnecessary, but just because it's unnecessary it doesn't > >follow that: > >1) The value of trying to handle OOM, instead of terminating, exceeds > >it cost. > >2) OOM can be handled in a robust fashion. > > This arguments can go both ways which you seem to refuse to accept: It > doesn't follow that: > > 1) The cost of trying to handle *some* OOM error, instead of > terminating, exceeds the value. Then provide a reliable mechanism to distinguish OOM errors. Thus far, you have been unable to do so. Otherwise, a reliable mechanism must treat them all and treat them as the worse case situation. > 2) That no OOM errors whatsoever can be handled in a robust fashion. I have never at any point said that or anything close to that. All I've said is that it's rarely, if ever, worth it, and that it's not nearly as easy to do robustly as most people here seem to believe. Adam
[toc] | [prev] | [next] | [standalone]
| From | yatremblay@bel1lin202.(none) (Yannick Tremblay) |
|---|---|
| Date | 2011-09-13 17:06 +0000 |
| Message-ID | <j4o2i1$nhe$1@dont-email.me> |
| In reply to | #9887 |
In article <56e11c1e-de9e-4e8e-8d71-a9a501e4ce17@b20g2000vbz.googlegroups.com>,
Adam Skutt <askutt@gmail.com> wrote:
>On Sep 7, 6:20am, yatremblay@bel1lin202.(none) (Yannick Tremblay)
>wrote:
>> In article <1cdb8d42-faa0-4001-91f3-fbb58309f...@d25g2000yqh.googlegroups.com>,
>> Adam Skutt <ask...@gmail.com> wrote:
>> >On Sep 3, 7:20pm, James Kanze <james.ka...@gmail.com> wrote:
>>
>> >> That's the correct response to a programming error or a system
>> >> wide problem. For a request which is part of a DOS attack, it
>> >> seems to be playing into the attacker's hand, and for a simply
>> >> unreasonable client request, it does seem unnecessary to abort
>> >> all other client connections.
>>
>> >One, it's not a given that it aborts other client connections, after
>> >all. There could be a higher level mechanism that provides the
>> >illusion of a persistent connection even after failover. Second, it
>> >may be unnecessary, but just because it's unnecessary it doesn't
>> >follow that:
>> >1) The value of trying to handle OOM, instead of terminating, exceeds
>> >it cost.
>> >2) OOM can be handled in a robust fashion.
>>
>> This arguments can go both ways which you seem to refuse to accept: It
>> doesn't follow that:
>>
>> 1) The cost of trying to handle *some* OOM error, instead of
>> terminating, exceeds the value.
>
>Then provide a reliable mechanism to distinguish OOM errors. Thus
>far, you have been unable to do so. Otherwise, a reliable mechanism
>must treat them all and treat them as the worse case situation.
Sorry for the long discussion below but short simplified and not fully
explicit answers have been previously met with dismissal based on
generalities:
Thus far, I have never attempted to provide a mechanism to
automatically distinguish OOM errors with no other information
whatsoever. Thus far I have posted a small code sample that catch()
following a new that was known to potentially be large. You simply
questionned that you didn't know what "large" was hence the example
was invalid. My answer is that I know it is potentially large because
it was designed that way.
Someone has posted experiences of observing an application recovering
from OOM errors. I have also done the same and repetitively tested it
by purposefully triggering OOM errors (yes feeding purposefully
desgined inputs to a real application in such a way that the application
eventually uses all of the available memory on a system and making
sure it still recovers).
The key is design. Design your application so that you *know*
where are the safe points of failure. You *know* how to cancel a job
safely. You *know* where the application will attempt to allocate a
large amount of memory and design this area in such a way that
recovery is possible *if* the failure is due to the requested
allocation being too large *and* much larger than what is normal.
You are the designer. You should be able to design the code in such a
way that you can ensure that "large" *potentially* recoverable
allocation happens at a known location in the code.
You keep saying "how can you distinguish recoverable vs
non-recoverable allocation?" The answer is design. You design your
application in such a way that you *plan* where the recoverable
allocations will happen.
So here is an compilable example that recovers safely from bad_alloc
----------------------------------------------
#include <iostream>
#include <stdexcept>
size_t const multiplier = 1000000;
bool doIt(size_t size)
{
int * p = 0;
try
{
p = new int[multiplier * size];
}
catch(std::bad_alloc &e)
{
// NOTE: I purposefully use iostream here.
// I realise that this may allocate memory underneath
// and is not a guarantee nothrow operation but here
// it highlights that memory is avaiable.
// In practice, you would probably not do it and just
// try to log in a *safe* way and
//start unrolling the stack
std::cout << "Error: " << e.what() << std::endl;
// allocate a bit anyway.
int *z = new int[10];
std::cout << "Wow, new[] still works" << std::endl;
delete z;
return false;
}
int * q = new int[5];
delete[] q;
delete[] p;
std::cout << "Did it " << size << std::endl;
return true;
}
int main()
{
size_t size;
for(;;)
{
std::cout << "Please enter allocation size: " << std::endl;
std::cin >> size;
if(!doIt(size))
{
std::cout << "Error happened for " << size << std::endl;
}
else
{
std::cout << "Job done for " << size << std::endl;
}
}
}
-------------------------------------------
This is a simplistic example but the principles are usable in a larger
application and even in a multithreaded application and it would not
matter if the try/catch was 5 function up the stack or directly around
the new.
Note that if the std::bad_alloc happens anywhere outside the
purposefully *designed* *potentially recoverable* area (i.e. when
allocating for q, z or within iostream even including the one in the
catch), then the program will terminate. This is also as-designed.
So the result:
The program can recover from OOM errors if they occur at a
particular location in the code where the designer planned to do
large allocations.
The program behave as you argued for when the OOM error occurs
elsewhere and terminate
The program can process all inputs while only being limited by the
actual current resource limits on the host (not some artificial
limits).
The program can *potentially* recover from OOM error if they are due
to input complexity. If recovery is succesfull, it can continue
processing new, less complex inputs safely. If recovery is not
successful, it will simply terminate.
The program will terminate on OOM error that happen elsewhere are are
*really* unexpected and the designer could not know how to handle.
IMO, the value of implementing this purposefully localised and
targetted OOM error recovery exceeds its cost.
>> 2) That no OOM errors whatsoever can be handled in a robust fashion.
>
>I have never at any point said that or anything close to that. All
>I've said is that it's rarely, if ever, worth it, and that it's not
>nearly as easy to do robustly as most people here seem to believe.
IMNSHO, this is not as difficult as you suggest if you carefully
design the system for this purpose.
Yannick
[toc] | [prev] | [next] | [standalone]
| From | Adam Skutt <askutt@gmail.com> |
|---|---|
| Date | 2011-09-13 12:09 -0700 |
| Message-ID | <b840f667-ae2f-4d55-a63c-bb2c77a8bf25@u19g2000vbm.googlegroups.com> |
| In reply to | #10078 |
On Sep 13, 1:06 pm, yatremblay@bel1lin202.(none) (Yannick Tremblay) wrote: > In article <56e11c1e-de9e-4e8e-8d71-a9a501e4c...@b20g2000vbz.googlegroups.com>, > Adam Skutt <ask...@gmail.com> wrote: > >Then provide a reliable mechanism to distinguish OOM errors. Thus > >far, you have been unable to do so. Otherwise, a reliable mechanism > >must treat them all and treat them as the worse case situation. > > > Thus far, I have never attempted to provide a mechanism to > automatically distinguish OOM errors with no other information > whatsoever. Thus far I have posted a small code sample that catch() > following a new that was known to potentially be large. You simply > questionned that you didn't know what "large" was hence the example > was invalid. My answer is that I know it is potentially large because > it was designed that way. Which is what I told you. It's still not sufficiently general to support the counterargument, "You only have to handle some of them". Besides, the reality of that matter is only having to handle some of them does not make things one iota easier, and I'm not sure why you think it would make things easier. The difficulty is in figuring out what goes inside the catch block, not in figuring out where to place the damn thing (though it's hardly as simple as you think it is). Regardless, like it or not, the burden on you to support your claim is as I described. It's not my fault you're making claims you cannot support. > > The key is design. Design your application so that you *know* > where are the safe points of failure. You *know* how to cancel a job > safely. There are plenty of systems where such things are impossible or simply more difficult than they are worth. This is why we have multiple levels of exception safety guarantees. There may not be such thing as a "safe point of failure". All of my exception safety guarantees may be weak. > You *know* where the application will attempt to allocate a > large amount of memory and design this area in such a way that > recovery is possible *if* the failure is due to the requested > allocation being too large *and* much larger than what is normal. > > You are the designer. You should be able to design the code in such a > way that you can ensure that "large" *potentially* recoverable > allocation happens at a known location in the code. I don't know why you keep returning to this point. 'Too large' is not the only reason to see an allocation failure. On modern operating systems with virtual memory, it's may not even be the primary reason to see an allocation failure[1]. Heap fragmentation / allocator limitations are far more likely to cause an OOM condition. Such issues equally effect small and large allocations: it depends entirely on the algorithm used to allocate memory. Do you know what your std::allocator does? It probably doesn't do what you think it does. Likewise, malloc() probably doesn't behave the way you think it does either. I may never, ever see an error because the allocation was "too large". It's poor justification for going through the effort of writing an OOM handler. Size alone does not tell me which allocations in the application will fail. > You keep saying "how can you distinguish recoverable vs > non-recoverable allocation?" The answer is design. You design your > application in such a way that you *plan* where the recoverable > allocations will happen. As I've stated many times: where one thinks the allocation failures will happen and where they will actually happen are two entirely different things. Especially when executing threaded code. Simply *saying* design does not tell me what I need to know. You haven't yet told me all the factors I need to consider in my design. There are plainly more factors than how much memory a particular request makes and why the request is being made. It's no different from optimization: just because you claim the hotspot is 'X' does not actually mean the hotspot is 'X'. Just because you say, "The program will run out of memory here" does not mean that will ever actually happen in practice. > // NOTE: I purposefully use iostream here. > // I realise that this may allocate memory underneath > // and is not a guarantee nothrow operation but here > // it highlights that memory is avaiable. No, it doesn't highlight that memory is available in general. It highlights memory was available in whatever asinine test cases you came up with, or that memory was not allocated in that particular situation. I'm not sure why you think an inductive proof has any value here whatsoever. I'm also not sure why you think such a simplistic example has any value whatsoever. The goal is to improve robustness by handling OOM. It was already stated that using iostreams as-is will not do this, so I'm not sure what you hoped to prove by writing this example. Write an example that actually improves robustness. And since you claimed this can be done in a multi-threaded program without impacting the other threads, do that too. Otherwise, you haven't demonstrated anything of any value whatsoever. Just for grins, try allocating all that space one byte at a time (go ahead and leak it), so you actually fill up the freestore before making the failed allocation. Then see how much space you have available, if you don't outright crash your computer[2][3]. > > This is a simplistic example but the principles are usable in a larger > application and even in a multithreaded application and it would not > matter if the try/catch was 5 function up the stack or directly around > the new. Such a simplistic handler will not protect other threads from failing during stack unwind for the original std::bad_alloc. How can it possibly do so? > The program can process all inputs while only being limited by the > actual current resource limits on the host (not some artificial > limits). Nope. Again, do you know what your std::allocator does? > IMNSHO, this is not as difficult as you suggest if you carefully > design the system for this purpose. You haven't yet designed a system for this purpose, so your opinion is worth nothing. Adam [1] On a modern 64-bit operating system, it's possible to never see an allocation failure for this reason. The address space of your process almost certainly exceeds your commit limit by several orders of magnitude (40 to 48-bits vs. 3X bits). Your process has good chance of just crashing the system or getting stopped by an OOM killer instead of seeing the failure. What will exactly happen depends on a huge number of factors not worth listing here (tbh, I'm not 100% confident I can even list them all). [2] When I ran such a test on one system I have access to, I could not allocate even a single int on the freestore once std::bad_alloc was thrown. It's also worth noting that the operating system claimed the process had filled the virtual address space, but the amount of memory my code allocated was considerably less (3GiB vs. ~800MiB). Asking for quarter pages (1KiB on my system) raised the amount returned to my code to very close to the virtual address space limit. This simply goes to further that making assumptions about how operator new/malloc behave can be quite dangerous. It means it is quite possible for "big" allocations to succeed when "little" allocations fail (proof left as exercise to reader)! [3] This is of course one reason that restarting is inherently superior to handling OOM: if your program does leak memory, then restarting the program will get that memory back. Plus, you will eventually have to take it out of service anyway to plug the leak for good.
[toc] | [prev] | [next] | [standalone]
| From | "Paul" <pchristor@yahoo.co.uk> |
|---|---|
| Date | 2011-09-14 00:18 +0100 |
| Message-ID | <j4ood2$11up$1@adenine.netfront.net> |
| In reply to | #10083 |
"Adam Skutt" <askutt@gmail.com> wrote in message news:b840f667-ae2f-4d55-a63c-bb2c77a8bf25@u19g2000vbm.googlegroups.com... On Sep 13, 1:06 pm, yatremblay@bel1lin202.(none) (Yannick Tremblay) wrote: > In article > <56e11c1e-de9e-4e8e-8d71-a9a501e4c...@b20g2000vbz.googlegroups.com>, > Adam Skutt <ask...@gmail.com> wrote: > >Then provide a reliable mechanism to distinguish OOM errors. Thus > >far, you have been unable to do so. Otherwise, a reliable mechanism > >must treat them all and treat them as the worse case situation. > > > Thus far, I have never attempted to provide a mechanism to > automatically distinguish OOM errors with no other information > whatsoever. Thus far I have posted a small code sample that catch() > following a new that was known to potentially be large. You simply > questionned that you didn't know what "large" was hence the example > was invalid. My answer is that I know it is potentially large because > it was designed that way. Which is what I told you. It's still not sufficiently general to support the counterargument, "You only have to handle some of them". Besides, the reality of that matter is only having to handle some of them does not make things one iota easier, and I'm not sure why you think it would make things easier. The difficulty is in figuring out what goes inside the catch block, not in figuring out where to place the damn thing (though it's hardly as simple as you think it is). Regardless, like it or not, the burden on you to support your claim is as I described. It's not my fault you're making claims you cannot support. +++++++++++++++++++++++++++++++++++++++++++++++++ There is a good debate about this here: http://stackoverflow.com/questions/333736/is-out-of-memory-a-recoverable-error I think the burden of proof lies with proving why programs should just terminate. That is if a side must be picked for the burden of proof . If the program must die then it should at least die gracefully. It wasn't so long ago average PC's had 32MB RAM, and historiclaly programs have been designed to handle OOM and recover, or die gracefully if the system was not capable of running the program. Consumers have come to expect this behavour from a program and they do not expect it to die every time it sniffs an low memory condition. Nowadays we have average 2GB RAM so fair enough if you make a program that is expected to use 16MB max or something then obviously there is a serious problem if you see a low memory condition. Termination in this case would probably be the answer but there is still no reason not to terminate gracefully. --- Posted via news://freenews.netfront.net/ - Complaints to news@netfront.net ---
[toc] | [prev] | [next] | [standalone]
| From | yatremblay@bel1lin202.(none) (Yannick Tremblay) |
|---|---|
| Date | 2011-09-14 15:11 +0000 |
| Message-ID | <j4qg7k$gh6$1@dont-email.me> |
| In reply to | #10083 |
In article <b840f667-ae2f-4d55-a63c-bb2c77a8bf25@u19g2000vbm.googlegroups.com>, Adam Skutt <askutt@gmail.com> wrote: >On Sep 13, 1:06pm, yatremblay@bel1lin202.(none) (Yannick Tremblay) >wrote: >Which is what I told you. It's still not sufficiently general to >support the counterargument, "You only have to handle some of them". You are being obtuse and refuse to even try to understand. I design an application. I specifically do the design in such a way that I know where "large" memory allocation occurs. In fact I designed it so that I purposefully do one large allocation rather than drip-by-drip. There is not need to be able to recover from all allocation failure. You seem to be the one that claims that because it is impossible to recover from all possible allocation failures, then you should never ever under any circumstances even consider attempting to recover from any allocation failure whatsoever. I disagree and the code sample posted demonstrate that this is possible. You said it yourself, if memory is really exhausted and a small allocation really fails, then attempting to recover is really hard. So you are correct that attempting to recover from all allocation failures is a bad idea. However, I can design an application so that it can recover from some specific allocation failure if I use my brain to design it correctly. >Beside, the reality of that matter is only having to handle some of >them does not make things one iota easier, and I'm not sure why you >think it would make things easier. The difficulty is in figuring out >what goes inside the catch block, not in figuring out where to place >the damn thing (though it's hardly as simple as you think it is). See code sample supplied that you purposefully ignore (well almost ignore). 1- It works 2- Some things is being done in the catch block No problem. Not difficult. I am not sure what is your problems with it. >Regardless, like it or not, the burden on you to support your claim is >as I described. It's not my fault you're making claims you cannot >support. FFS! Code supplied. Claim supported. Did you run it? It works. More than one person have told you that they have tested a similar thing in the real world. My claim that it is possible to recover from some allocation failures is fully supported. I have no idea what claim you claim that I made that you claim that I cannot support. Clarification before you try to claim that I claim other things: My claims: "It is possible to recover from some allocation failures" "It is possible to design an application in such a way so that there are suitable recovery points" "It is possible to design an application in such a way that you plan to do potentially large allocation in one particular area and given this knowledge and this planning, it is safe to *attempt* to recover" "On a case by case basis, the cost of handling some OOM error may be justified by the value" As far as I can understand, you seem to claim the generic that recovering from any allocation failures whatsoever is impossible to do and never worth doing. This is invalid. You can disprove an "it's impossible to do" by simply supplying one example where it works. BTW: I am not disputing that it is very difficult (or virtually impossible) to write a generic allocation failure handler that will always succesfully recover from all allocation failure regardless of the cause. >> The key is design. Design your application so that you *know* >> where are the safe points of failure. You *know* how to cancel a job >> safely. > >There are plenty of systems where such things are impossible or simply >more difficult than they are worth. This is why we have multiple >levels of exception safety guarantees. There may not be such thing as >a "safe point of failure". All of my exception safety guarantees may >be weak. I never claimed that it is always possible for all system. I claim that one can design a system where it is possible in restricted cases. The fact that some system may not have any "safe point of failure" does not mean that it is impossible to design a specific system to have safe points of failure. Are you claiming that it is impossible to design any system with safe points of failure? >> You *know* where the application will attempt to allocate a >> large amount of memory and design this area in such a way that >> recovery is possible *if* the failure is due to the requested >> allocation being too large *and* much larger than what is normal. >> >> You are the designer. You should be able to design the code in such a >> way that you can ensure that "large" *potentially* recoverable >> allocation happens at a known location in the code. > >I don't know why you keep returning to this point. Because it seems you are refusing to understand. >'Too large' is not >the only reason to see an allocation failure. In the system as designed, you *know* that in this particular location, the allocation size is possibly too large (because it depends on external input). You *know* that because you designed it that way. Hence two possibilities: 1- The allocation failure was due to the requested allocation being too large. The the recovery attempt will succeed and it is fine to continue. 2- The allocation failure was not due to the requested allocation being too large and instead due to allocation being totally impossible on the system now. Then the recovery attempt will fail and the program will terminate. Essentially you are advocating assuming that you are always and will always ever be in situation #2. I am advocating that given good design, good expertise and knowledge, you can know where it is worth checking if you are in situation #1 and it may be worth attempting to recover. Of course, that also leave allocation failures that happens elsewhere in the program. In this case, the result will be as you advocate and the application will terminate. As designed. >On modern operating >systems with virtual memory, it's may not even be the primary reason >to see an allocation failure[1]. Heap fragmentation / allocator >limitations are far more likely to cause an OOM condition. Such >issues equally effect small and large allocations: it depends entirely >on the algorithm used to allocate memory. Do you know what your >std::allocator does? It probably doesn't do what you think it does. >Likewise, malloc() probably doesn't behave the way you think it does >either. Can we at least agree that the application can't know if an allocation failed because of heap fragmentation, allocator limitation, maximum per-process OS enforced limits or OS actually having run out of memory altogether? The visible result for the application will be the same. So this is irrelavent to the discussion. For what it's worth, the posted code will work as advertised on a system with little physical RAM and disabled virtual memory. Included code demonstate that on modern operating system, it is possible to design an application that may have to allocate memory depending on external input (unknown at compile time) in a system with unknown current available resources and that it can be design in such a way so that in some/one location in the code, allocation failure are likely to be primarily caused by "large" allocation requests. The posted code does not in any way attempt to demonstrate that the application will always be able to recover from all possible causes of OOM errors at any possible places in the code. >I may never, ever see an error because the allocation was "too >large". It's poor justification for going through the effort of >writing an OOM handler. Size alone does not tell me which allocations >in the application will fail. For your application, this may be the case. For other applications, this may not be the case. >> You keep saying "how can you distinguish recoverable vs >> non-recoverable allocation?" The answer is design. You design your >> application in such a way that you *plan* where the recoverable >> allocations will happen. > >As I've stated many times: where one thinks the allocation failures >will happen and where they will actually happen are two entirely >different things. You choose to quit at the first hurdle. I choose to at least attempt to jump. If I fail, no loss. I am no worse than you. If I succeed, I live to fight another day. See posted code. The application is designed so that allocation failure due to input complexity happen where planned. Other allocation failures may happen elsewhere. The way an allocation failure happening elsewhere is handled will be different than the way it will be treated if it happened in the purposefully designed "recoverable" section. > Especially when executing threaded code. Simply >*saying* design does not tell me what I need to know. You haven't yet >told me all the factors I need to consider in my design. There are >plainly more factors than how much memory a particular request makes >and why the request is being made. I am not the designer of your application so I can't know what are all the factors that need to be considered for *your* design. I have no ideas of *your* requirements. I will leave it to you as an exercise to modify the code previously posted so that it can run multithreaded and have recovery points. >It's no different from optimization: just because you claim the >hotspot is 'X' does not actually mean the hotspot is 'X'. Just >because you say, "The program will run out of memory here" does not >mean that will ever actually happen in practice. Do you like trying to put claims in other peoples mouths? I am saying: I design the program so that potentially large allocation happens here and given this design I choose to attempt to recover *if* the allocation failure happened there. Question: In the posted simplitic example, does the hotspot happen where planned? If the program has an allocation failure elsewhere, the result is the same as what you preach: terminate. So no loss whatsoever for other allocation failure. Gain for some specific allocation failures. >> // NOTE: I purposefully use iostream here. >> // I realise that this may allocate memory underneath >> // and is not a guarantee nothrow operation but here >> // it highlights that memory is avaiable. > >No, it doesn't highlight that memory is available in general. It >highlights memory was available in whatever asinine test cases you >came up with, or that memory was not allocated in that particular >situation. I'm not sure why you think an inductive proof has any >value here whatsoever. I'm also not sure why you think such a >simplistic example has any value whatsoever. A simplistic example can disprove a generality. You argue against ever attempting to recover from OOM because it's so much more difficult than anyone can imagine. That it's so difficult that it never ever worth even attempting. The simplistic example demonstrate that it is perfectly possible and not necessarily complex in specific cases if you design your application this way. If you choose to design your application in such a way that nowhere is it possible to recover from an allocation failure, it is *your* choice. >The goal is to improve robustness by handling OOM. It was already >stated that using iostreams as-is will not do this, so I'm not sure >what you hoped to prove by writing this example. Write an example >that actually improves robustness. And since you claimed this can be >done in a multi-threaded program without impacting the other threads, >do that too. Otherwise, you haven't demonstrated anything of any >value whatsoever. Bullshit! (sorry for the rudeness but you asked for it) The example supply improved robustness since even after an OOM error, the program recovers and can keep processing further inputs. The example can be extended to multithread using the same principle. It's simple to do. Can you do it? You keep claiming that everyone else are giving unsupported claims. I gave you an example supporting my claims. The example demonstrate that it is possible to recover from some OOM errors. What about you proving your claims? >Just for grins, try allocating all that space one byte at a time (go >ahead and leak it), so you actually fill up the freestore before >making the failed allocation. Then see how much space you have >available, if you don't outright crash your computer[2][3]. So you are advocating that I should design my application in such a way that OOM error are always fatal. IN such a way that I purposefully micro-allocate lots and lots of memory so that failure will most likely happen at totally random places. Euh?!? Well the consequences will be that OOM errors will always be fatal. That the way the application is designed. I chose to design the application so that some OOM errors can be handled. Is there a law against good design and an obligation to always enforcing stupid design? I am not sure I understand why, in order to support you arguments that it is never possible to ever recover from an OOM error, everyone should always design their application so that they are purposefully designed not to be able to recover from an OOM error. The key to the design is precisely that it does not allocate memory one byte at a time. Because of this design, the application will never run out of memory altogether purely due to input complexity. It doesn't matter if it is multi-threaded or single threaded. The application uses malloc/new/allocator in a similar way as you suggested to use predefined arbitrary limits: instead of checking that the input are of lower complexity than X, it "check that I have enough resources to process this input and if so, reserve them immediately". The std::bad_alloc is just the allocator answering "no" to the first question. The failed "new" does not affect the available resources, it fails to acquire them. >> This is a simplistic example but the principles are usable in a larger >> application and even in a multithreaded application and it would not >> matter if the try/catch was 5 function up the stack or directly around >> the new. > >Such a simplistic handler will not protect other threads from failing >during stack unwind for the original std::bad_alloc. How can it >possibly do so? Given your claimed superior knowledge of allocators, please enlight us on why the same pattern would fail on a multithreaded setup? Can you clarify why thread #1 *failing* to allocate a large block of memory directly stop thread #2 from being able to allocate a small block of memory? Are your allocator not-thread safe? It will work as advertised. Assuming you design your multithreaded application intelligently and build in places where recovery is possible. If allocation failure was due to the size of allocation, the other threads will keep working fine. If the allocation failure was due to other reason, the program will terminate. You choose to believe it is not possible. I know it is possible (in limited circumstance, for specific cases, if you design carefully, even in multithreaded applications). Obviously, if you design you application to leak memory one byte at a time, the application is likely to fail at any random point and will most probably not be able to recover. That's your design and your choice. >> IMNSHO, this is not as difficult as you suggest if you carefully >> design the system for this purpose. > >You haven't yet designed a system for this purpose, so your opinion is >worth nothing. This would be worth an rude reply but I'll skip. BTW: all your claims so far are unsupported. Kettle, pot, black? >[3] This is of course one reason that restarting is inherently >superior to handling OOM: if your program does leak memory, then >restarting the program will get that memory back. Plus, you will >eventually have to take it out of service anyway to plug the leak for >good. If you program leaks memory, you should fix it. Not rely on periodic restarts. Sorry but IMO, crashing and restarting is inherently inferior to not crashing and staying in a fully stable state. We will have to agree to disagree but I doubt users seeing the apps crash will be particularly happy. Are you now recommending that just in case an application may have been written by an incompetent programmer that leaks memory, every persistent service application in the world should always be restarted periodically? BTW: designing your application so that it attempts to recover from some error does not mean that you can't also have a monitor that restart the application if it crashes. Sorry about the tone of some of my comments but the style of your answer annoyed me. I think the discussion is worthwhile and interesting. Yannick
[toc] | [prev] | [next] | [standalone]
| From | Adam Skutt <askutt@gmail.com> |
|---|---|
| Date | 2011-09-14 11:17 -0700 |
| Message-ID | <6908aff2-da9b-4b31-b37c-e6577d954798@s2g2000vby.googlegroups.com> |
| In reply to | #10100 |
On Sep 14, 11:11 am, yatremblay@bel1lin202.(none) (Yannick Tremblay) wrote: > In article <b840f667-ae2f-4d55-a63c-bb2c77a8b...@u19g2000vbm.googlegroups.com>, > Adam Skutt <ask...@gmail.com> wrote: > > >On Sep 13, 1:06pm, yatremblay@bel1lin202.(none) (Yannick Tremblay) > >wrote: > >Which is what I told you. It's still not sufficiently general to > >support the counterargument, "You only have to handle some of them". > > You are being obtuse and refuse to even try to understand. > No, I understand your code and your point perfectly. You're simply wrong, because you have zero evidence that any real application will behave as your little example. Real applications can behave in a myriad of ways, including in the two I suggested: total failure to allocate memory memory and where "little" allocations fail while "large" allocations do not. Your example plainly does not work as intended in either of those two situations. More importantly, it does not achieve your original stated goal: to improve the robustness of the application and isolate the failure of one processing from impacting the others. Your lack of understanding about how programs, runtimes, and operating systems allocate memory and refusal to consider the situations I've posed does not make me obtuse. The fact you're unwilling or unable to accept that the code may never throw std::bad_alloc where you claim it will is simply gobsmacking. You've provided zero evidence to believe that the code will fail where you claim it will with any regularity. As such, we have no reason whatsoever to believe your design will improve robustness. Even if we believed you have managed to place the catch block correctly (you didn't), the entire discussion started over the difficulty of writing what goes inside the catch block! What you've written is plainly insufficient, so you still haven't met the burden of proof upon you. Until you actually present something that will demonstrably improve robustness in a real application under a wide variety of failure cases (or demonstrate their irrelevance), there's no reason to discuss this any further than this e-mail. You've been presented with plenty of reasoning as to why your views are invalid and your example will not work in a real application. If what you've been given isn't sufficient proof, then I doubt anything will be sufficient proof. > There is not need to be able to recover from all allocation failure. > You seem to be the one that claims that because it is impossible to > recover from all possible allocation failures, then you should never > ever under any circumstances even consider attempting to recover from > any allocation failure whatsoever. No, I say that only handling some failures doesn't buy you anything because you have no way to tell which allocations will fail. Writing a catch handler that never runs does nothing but waste my time. You have yet to tell me how to determine which allocations will fail and which allocations will not. You keep harping on "large", but I've already demonstrated that size enough isn't sufficient information. If your allocator is pooling and doesn't return memory to the operating system, one failed allocation might be enough to prevent all future allocations from succeeding, if they use a different allocator. > > 1- It works > 2- Some things is being done in the catch block > > No problem. Not difficult. I am not sure what is your problems with > it. It doesn't work if there is no memory remaining, because the iostream might allocate behind your back. Ergo, it is not robust. Just because you believe your code proves there must be memory remaining does not mean it has actually done so. You have not tested all the possibilities yet. Inductive fallacies are not proof, and they are just another reason I see no point in discussing this topic with you further. > I have no idea what claim you claim that I made that you claim that I > cannot support. All of them, ignoring the fact you've seriously backed off your original claims. Which is what I still want to see, because they would actually be interesting as opposed to what you are trying to do now. > As far as I can understand, you seem to claim the generic that > recovering from any allocation failures whatsoever is impossible to do > and never worth doing. I've never claimed any such thing. I've claimed it's almost never worth doing because it's simply too hard to do robustly, and it's far easier to improve robustness in other ways. I've also claimed that it is considerably harder than its proponents suggest and demonstrated the issues with the proposed solutions and why they are not actually robust. > In the system as designed, you *know* that in this particular location, > the allocation size is possibly too large (because it depends on external > input). You *know* that because you designed it that way. > > Hence two possibilities: > > 1- The allocation failure was due to the requested allocation being > too large. The the recovery attempt will succeed and it is fine to > continue. > > 2- The allocation failure was not due to the requested allocation > being too large and instead due to allocation being totally impossible > on the system now. Then the recovery attempt will fail and the > program will terminate. > Those are not the only two possibilities, which is part of the problem with your reasoning. Consider: 3. The allocation succeeds but then causes all future allocations to fail due to its size (i.e., it filled the freestore). This should be plainly be handled in the same manner as the first possibility, since the size of the user's request caused the failure. However, your code example will not handle this possibility correctly. > Essentially you are advocating assuming that you are always and will > always ever be in situation #2. Nope, I'm advocating that you cannot tell what situation you are in and that you may well be in a situation where recovery is impossible or pointless, generally speaking. It's obviously possible to figure these things out, but it requires substantially more work than you, or any other advocate, believe it does. Figuring out the situation typically requires knowing low-level runtime and operating system details so you can reason about what situations are possible and when they can occur. Such details aren't trivial by any stretch of the imagination. > Can we at least agree that the application can't know if an allocation > failed because of heap fragmentation, allocator limitation, maximum > per-process OS enforced limits or OS actually having run out of memory > altogether? The visible result for the application will be the same. Your first statement is right and your second statement is wrong. The visible result for the application will plainly not be the same. If you've reached the commit limit, then all requests for more memory from the OS will fail until the limit is increased or memory is returned the operating system. If a request fails because it's simply too large (e..g, it's bigger than your virtual address space), then reasonable requests will still succeed. Why the allocation failed matters, whether you like it or not. It matters because it determines whether there's any value in spending the time on writing a catch handler. > >As I've stated many times: where one thinks the allocation failures > >will happen and where they will actually happen are two entirely > >different things. > > You choose to quit at the first hurdle. I choose to at least attempt > to jump. If I fail, no loss. I am no worse than you. If I succeed, > I live to fight another day. No, you're far worse off because you've spend a large amount of money designing, implementing, and (badly) testing code that's effectively dead. I'm better off because I got to keep all that money. Writing code is not free. > > I am not the designer of your application so I can't know what are all > the factors that need to be considered for *your* design. I have no > ideas of *your* requirements. Then you cannot possibly claim it's generally worthwhile to handle OOM conditions, yet you've attempted to do precisely that several times over! > >Just for grins, try allocating all that space one byte at a time (go > >ahead and leak it), so you actually fill up the freestore before > >making the failed allocation. Then see how much space you have > >available, if you don't outright crash your computer[2][3]. > So you are advocating that I should design my application in such a > way that OOM error are always fatal. IN such a way that I > purposefully micro-allocate lots and lots of memory so that failure > will most likely happen at totally random places. Euh?!? No, I'm suggesting you try test cases that better exercise all of the conditions under which OOM can occur. Many applications include many small allocations to go along with their large allocations. Many application may never make singular large allocations, perhaps they structure their data using a linked list or a tree instead of an array. Your example is simply not realistic, and I gave one example of how it is unrealistic. The large allocation may be the last allocation in a long string of small allocations. It may succeed, filling the address space, causing all future allocations to fail. > Given your claimed superior knowledge of allocators, please enlight us > on why the same pattern would fail on a multithreaded setup? > Can you clarify why thread #1 *failing* to allocate a large block of > memory directly stop thread #2 from being able to allocate a small > block of memory? Are your allocator not-thread safe? If the first allocation failed because the virtual address space is full, the second allocation will fail for the same reason. Your original claim was that you could isolate failure in one thread from failure in others simply by catching OOM in the first thread. This is obviously not possible when the reason for failure has nothing to do with the size of the request. Sure, your trivial example won't fail, but an example that considers only one allocation isn't particularly interesting. Very few applications are written to process their requests with only one contiguous dynamic allocation. The problem is that you have no way of knowing that it was the size of the request. If the only reason my program will ever fail is because virtual address space is full (i.e., all requests are "reasonable") then my program will be terminating. That makes writing the catch handler pointless. Remember, this issue originally came up because someone claimed that stack unwinding would free memory, allowing the other threads to proceed. That's not true because allocators may not return memory to the OS or a global pool, and different threads and/or data types may use different allocators. It's also not true because there just may not be enough resources left to be freed. > >[3] This is of course one reason that restarting is inherently > >superior to handling OOM: if your program does leak memory, then > >restarting the program will get that memory back. Plus, you will > >eventually have to take it out of service anyway to plug the leak for > >good. > If you program leaks memory, you should fix it. Yes, I agree. Doing that means restarting the program. That means I must have built a system that can tolerate restarting the program. That seriously diminishes the value of writing code with the sole purpose of avoiding program restarts. Nevermind the fact my program may leak due to factors entirely outside my control, like buggy operating system or runtime libraries. Adam
[toc] | [prev] | [next] | [standalone]
| From | Goran <goran.pusic@gmail.com> |
|---|---|
| Date | 2011-09-15 04:17 -0700 |
| Message-ID | <0a7ef0ca-d0bd-452e-94e6-b380d9c9b4ff@h6g2000vbc.googlegroups.com> |
| In reply to | #10105 |
On Sep 14, 8:17 pm, Adam Skutt <ask...@gmail.com> wrote:
> > You choose to quit at the first hurdle. I choose to at least attempt
> > to jump. If I fail, no loss. I am no worse than you. If I succeed,
> > I live to fight another day.
>
> No, you're far worse off because you've spend a large amount of money
> designing, implementing, and (badly) testing code that's effectively
> dead. I'm better off because I got to keep all that money. Writing
> code is not free.
We're talking about bad_alloc, hence exceptions. In that light, the
above is wrong. In exceptions-based code cost you're mentioning is
virtually non-existent^^^, because:
* in good C++ code, number of try/catch statements is __seriously__
small, and that number of "bad_alloc" catches is even smaller.
* in such code, there's absolutely no need to do anything particular
WRT OOM, __except__ in a catch(const bad_alloc&) {}.
In fact, hereby I am putting this to you: you don't know how to work
with exceptions effectively. You are missing the necessary mindset
change, from:
* trying to guess, all the time, whether there can be an error
somewhere, and what are the consequences (bad),
to
* assigning exception safety levels to bits of code (good).
In the former case, indeed, cost of handling OOM is immense, because
number of explicit failure points that must be handled is immense.
In the latter, however, you just don't care about OOM in particular.
You think about your exception safety guarantees, and OOM is in the
same bag with all other exceptions - ultimately, you don't even see
it.
Goran.
^^^ The cost is merely making sure that what's in a (rare) catch(const
bad_alloc&) { ---HERE--- } is either a no-throw operation, either not
the last catch there is (which is a pretty trivial affair).
[toc] | [prev] | [next] | [standalone]
Page 9 of 10 — ← Prev page 1 … 7 8 [9] 10 Next page →
Back to top | Article view | comp.lang.c++
csiph-web