Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.lang.c++ > #9464 > unrolled thread
| Started by | Paul <pchristor@yahoo.co.uk> |
|---|---|
| First post | 2011-08-29 23:26 -0700 |
| Last post | 2011-08-30 16:45 -0700 |
| Articles | 20 on this page of 197 — 20 participants |
Back to article view | Back to comp.lang.c++
bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-29 23:26 -0700
Re: bad alloc Krice <paulkp@mbnet.fi> - 2011-08-30 02:14 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 06:26 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-08-30 14:34 +0100
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 06:54 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-08-30 15:02 +0100
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-08-30 15:06 +0100
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 07:20 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-08-30 15:31 +0100
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 07:47 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-08-30 15:52 +0100
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 08:00 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-08-30 16:15 +0100
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 10:31 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-08-30 18:58 +0100
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-08-30 19:18 +0100
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 16:17 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-08-31 00:56 +0100
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 17:11 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-08-31 01:24 +0100
Re: bad alloc Ian Collins <ian-news@hotmail.com> - 2011-08-31 12:33 +1200
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 18:38 -0700
Re: bad alloc Noah Roberts <roberts.noah@gmail.com> - 2011-08-30 20:07 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-31 04:04 -0700
Re: bad alloc Noah Roberts <roberts.noah@gmail.com> - 2011-08-31 09:17 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-08-31 14:59 +0100
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-01 00:50 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-09-01 12:41 +0100
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-01 05:00 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-09-01 13:59 +0100
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-01 06:35 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-09-01 14:50 +0100
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-01 13:51 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-09-01 21:58 +0100
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-01 14:05 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-09-01 22:08 +0100
Re: bad alloc Nick Keighley <nick_keighley_nospam@hotmail.com> - 2011-09-02 03:16 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-02 04:12 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-09-02 15:08 +0100
Re: bad alloc Nick Keighley <nick_keighley_nospam@hotmail.com> - 2011-09-02 03:07 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-02 04:15 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 11:21 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 10:46 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-08-30 18:58 +0100
Re: bad alloc "Waldek M." <wm@localhost.localdomain> - 2011-08-31 20:58 +0200
Re: bad alloc "io_x" <a@b.c.invalid> - 2011-09-03 10:13 +0200
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-05 18:53 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 16:28 -0700
Re: bad alloc Nick Keighley <nick_keighley_nospam@hotmail.com> - 2011-09-01 00:49 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-01 04:50 -0700
Re: bad alloc Nick Keighley <nick_keighley_nospam@hotmail.com> - 2011-09-02 03:17 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-02 04:21 -0700
Re: bad alloc James Kanze <james.kanze@gmail.com> - 2011-09-01 16:23 -0700
Re: bad alloc Nick Keighley <nick_keighley_nospam@hotmail.com> - 2011-09-02 03:19 -0700
Re: bad alloc James Kanze <james.kanze@gmail.com> - 2011-09-03 15:37 -0700
Re: bad alloc Dombo <dombo@disposable.invalid> - 2011-09-04 09:25 +0200
Re: bad alloc James Kanze <james.kanze@gmail.com> - 2011-09-01 16:20 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-08-30 06:54 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 07:12 -0700
Re: bad alloc Dombo <dombo@disposable.invalid> - 2011-08-30 22:58 +0200
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 15:56 -0700
Re: bad alloc yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2011-08-31 09:28 +0000
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-31 04:20 -0700
Re: bad alloc yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2011-09-02 14:57 +0000
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-02 08:35 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-08-31 04:25 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-08-30 03:26 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 05:54 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-08-30 14:30 +0100
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 07:00 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-08-31 00:37 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-31 04:26 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-08-30 07:03 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 07:57 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-08-30 16:04 +0100
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 08:15 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-08-30 16:20 +0100
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 10:37 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-08-30 18:41 +0100
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 10:49 -0700
Re: bad alloc Asger-P <junk@asger-p.dk> - 2011-08-31 10:33 +0200
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-31 04:22 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-08-30 16:04 -0700
Re: bad alloc Noah Roberts <roberts.noah@gmail.com> - 2011-08-30 20:02 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-08-30 20:33 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-08-31 01:30 -0700
Re: bad alloc yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2011-08-31 09:23 +0000
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-08-31 04:21 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-08-31 04:47 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-31 05:35 -0700
Re: bad alloc Nick Keighley <nick_keighley_nospam@hotmail.com> - 2011-09-01 01:08 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-01 03:56 -0700
Re: bad alloc Nick Keighley <nick_keighley_nospam@hotmail.com> - 2011-09-02 04:18 -0700
Re: bad alloc Dombo <dombo@disposable.invalid> - 2011-09-03 18:37 +0200
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-03 13:39 -0700
Re: bad alloc red floyd <no.spam.here@its.invalid> - 2011-09-03 13:45 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-03 13:51 -0700
Re: bad alloc Sjouke Burry <burryNULNULFOUR@PPLLAANNEETT.NNLL> - 2011-09-04 00:14 +0200
Re: bad alloc James Kanze <james.kanze@gmail.com> - 2011-09-03 15:51 -0700
Re: bad alloc Ian Collins <ian-news@hotmail.com> - 2011-09-04 12:15 +1200
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-04 05:00 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-05 18:13 -0700
Re: bad alloc James Kanze <james.kanze@gmail.com> - 2011-09-03 15:45 -0700
Re: bad alloc yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2011-09-02 15:40 +0000
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-02 09:33 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-02 14:06 -0700
Re: bad alloc "io_x" <a@b.c.invalid> - 2011-09-03 10:12 +0200
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-08-31 04:21 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-08-31 01:22 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-08-31 05:18 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-08-31 06:36 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-08-31 09:33 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-09-01 00:20 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-01 04:25 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-09-01 12:51 +0100
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-01 06:04 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-09-01 06:26 -0700
Re: bad alloc James Kanze <james.kanze@gmail.com> - 2011-09-01 16:58 -0700
Re: bad alloc Joshua Maurice <joshuamaurice@gmail.com> - 2011-09-01 17:25 -0700
Re: bad alloc James Kanze <james.kanze@gmail.com> - 2011-09-03 16:02 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-05 18:20 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-05 19:13 -0700
Re: bad alloc James Kanze <james.kanze@gmail.com> - 2011-09-06 15:39 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-06 17:26 -0700
Re: bad alloc James Kanze <james.kanze@gmail.com> - 2011-09-06 15:35 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-06 17:28 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-01 21:01 -0700
Re: bad alloc James Kanze <james.kanze@gmail.com> - 2011-09-03 16:14 -0700
Re: bad alloc "io_x" <a@b.c.invalid> - 2011-09-05 10:31 +0200
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-05 18:36 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-06 04:09 -0700
Re: bad alloc James Kanze <james.kanze@gmail.com> - 2011-09-06 15:49 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-06 17:11 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-09-07 00:24 -0700
Re: bad alloc Joshua Maurice <joshuamaurice@gmail.com> - 2011-08-31 09:59 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-08-31 11:24 -0700
Re: bad alloc Joshua Maurice <joshuamaurice@gmail.com> - 2011-09-01 16:29 -0700
Re: bad alloc Joshua Maurice <joshuamaurice@gmail.com> - 2011-09-01 16:42 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-01 21:37 -0700
Re: bad alloc Ian Collins <ian-news@hotmail.com> - 2011-09-02 17:45 +1200
Re: bad alloc James Kanze <james.kanze@gmail.com> - 2011-09-03 16:20 -0700
Re: bad alloc Ian Collins <ian-news@hotmail.com> - 2011-09-04 12:03 +1200
Re: bad alloc James Kanze <james.kanze@gmail.com> - 2011-09-04 10:56 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-05 18:50 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-09-06 00:12 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-06 03:04 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-09-07 00:08 -0700
Re: bad alloc Nick Keighley <nick_keighley_nospam@hotmail.com> - 2011-09-07 02:33 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-07 04:28 -0700
Re: bad alloc yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2011-09-06 13:43 +0000
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-06 07:37 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-09-07 00:11 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-07 04:29 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-09-07 05:38 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-07 18:17 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-09-08 00:45 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-06 06:58 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-06 09:21 -0700
Re: bad alloc "Fred Zwarts \(KVI\)" <F.Zwarts@KVI.nl> - 2011-09-07 09:10 +0200
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-07 04:18 -0700
Re: bad alloc yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2011-09-07 10:34 +0000
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-07 04:00 -0700
Re: bad alloc yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2011-09-08 12:11 +0000
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-08 06:27 -0700
Re: bad alloc James Kanze <james.kanze@gmail.com> - 2011-09-06 15:52 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-06 17:16 -0700
Re: bad alloc Nick Keighley <nick_keighley_nospam@hotmail.com> - 2011-09-07 02:39 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-07 04:14 -0700
Re: bad alloc Ian Collins <ian-news@hotmail.com> - 2011-09-06 17:00 +1200
Re: bad alloc Miles Bader <miles@gnu.org> - 2011-09-06 14:51 +0900
Re: bad alloc James Kanze <james.kanze@gmail.com> - 2011-09-06 15:55 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-05 18:44 -0700
Re: bad alloc yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2011-09-07 10:20 +0000
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-07 03:40 -0700
Re: bad alloc yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2011-09-13 17:06 +0000
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-13 12:09 -0700
Re: bad alloc "Paul" <pchristor@yahoo.co.uk> - 2011-09-14 00:18 +0100
Re: bad alloc yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2011-09-14 15:11 +0000
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-14 11:17 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-09-15 04:17 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-15 06:50 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-09-15 07:31 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-15 07:59 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-09-16 00:19 -0700
Re: bad alloc Noah Roberts <roberts.noah@gmail.com> - 2011-09-17 10:13 -0700
Re: bad alloc Goran <goran.pusic@gmail.com> - 2011-09-18 10:07 -0700
Re: bad alloc yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2011-09-16 15:41 +0000
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-16 11:26 -0700
Re: bad alloc Joshua Maurice <joshuamaurice@gmail.com> - 2011-09-16 15:38 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-17 04:25 -0700
Re: bad alloc Joshua Maurice <joshuamaurice@gmail.com> - 2011-09-01 23:17 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-09-02 04:03 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-02 04:37 -0700
Re: bad alloc Leigh Johnston <leigh@i42.co.uk> - 2011-09-02 15:15 +0100
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-09-02 07:35 -0700
Re: bad alloc Adam Skutt <askutt@gmail.com> - 2011-08-30 16:06 -0700
Re: bad alloc Paul <pchristor@yahoo.co.uk> - 2011-08-30 16:45 -0700
Page 7 of 10 — ← Prev page 1 … 5 6 [7] 8 9 10 Next page →
| From | Adam Skutt <askutt@gmail.com> |
|---|---|
| Date | 2011-09-05 18:20 -0700 |
| Message-ID | <503015a0-ee7e-44a1-a948-a79253eef526@d25g2000yqh.googlegroups.com> |
| In reply to | #9713 |
On Sep 3, 7:02 pm, James Kanze <james.ka...@gmail.com> wrote: > On Sep 2, 1:25 am, Joshua Maurice <joshuamaur...@gmail.com> wrote: > > For most processes which are not the ones > > being "abusive" but merely innocent bystanders, including system > > processes, I suspect they will behave similarly. With overcommit on, > > they will be killed with great prejudice. With overcommit off, when > > they get the malloc error, most will respond just the same and die a > > quick death. > > That would be a very poorly written service that died just > because of a malloc failure. Then are you willing to make the claim most Linux, UNIX, OS X and Windows services are poorly written? Because that's what you just said, merely using different words. > > All services, critical or not, should have some sort of > reasonable response to all possible error conditions. Crashing is a perfectly reasonable response to an error condition, really most error conditions. I'm not sure why anyone would think otherwise even for a second. > All of the Unices I know do implement per-process limits. Which > can be useful in specific cases as well. You only know a few then. POSIX specifies some crude (and useless) per-/user/ limits, but per-process limits are not standard by any stretch of the imagine. As I already explained, they also do nothing to solve the problem, nor can they really. Adam
[toc] | [prev] | [next] | [standalone]
| From | Adam Skutt <askutt@gmail.com> |
|---|---|
| Date | 2011-09-05 19:13 -0700 |
| Message-ID | <b9760e77-5d65-426d-9187-0830b3d7b1cf@w28g2000yqw.googlegroups.com> |
| In reply to | #9769 |
On Sep 5, 9:20 pm, Adam Skutt <ask...@gmail.com> wrote: > On Sep 3, 7:02 pm, James Kanze <james.ka...@gmail.com> wrote: > > All of the Unices I know do implement per-process limits. Which > > can be useful in specific cases as well. > > You only know a few then. POSIX specifies some crude (and useless) > per-/user/ limits, but per-process limits are not standard by any > stretch of the imagine. As I already explained, they also do nothing > to solve the problem, nor can they really. > I should clarify this part. All Unicies provide some limits through ulimit, which provides mostly per-process limits that are user discretionary. Few provide the ability to enforce limits on all invocations of binary X, which is what I assumed you were talking about and what would be necessary. ulimit isn't helpful because the limits have to be high enough for the largest program the user wishes to run, and you cannot prevent the user from setting those limits for running other binaries. Adam
[toc] | [prev] | [next] | [standalone]
| From | James Kanze <james.kanze@gmail.com> |
|---|---|
| Date | 2011-09-06 15:39 -0700 |
| Message-ID | <347d86a2-b1db-4513-9495-723d5d3ba376@y21g2000yqy.googlegroups.com> |
| In reply to | #9774 |
On Sep 6, 3:13 am, Adam Skutt <ask...@gmail.com> wrote: > On Sep 5, 9:20 pm, Adam Skutt <ask...@gmail.com> wrote: > > On Sep 3, 7:02 pm, James Kanze <james.ka...@gmail.com> wrote: > > > All of the Unices I know do implement per-process limits. Which > > > can be useful in specific cases as well. > > You only know a few then. POSIX specifies some crude (and useless) > > per-/user/ limits, but per-process limits are not standard by any > > stretch of the imagine. As I already explained, they also do nothing > > to solve the problem, nor can they really. > I should clarify this part. All Unicies provide some limits through > ulimit, which provides mostly per-process limits that are user > discretionary. Few provide the ability to enforce limits on all > invocations of binary X, which is what I assumed you were talking > about and what would be necessary. That's not what I was talking about, and that's absolutely unnecessary. Any reliable server will be running on a dedicated machine, in a dedicated environment. It's impossible to have any sort of reliable system otherwise. > ulimit isn't helpful because the > limits have to be high enough for the largest program the user wishes > to run, and you cannot prevent the user from setting those limits for > running other binaries. That's just bullshit. A reliable server will be started from a shell script, which will set ulimits just before starting it (and will do nothing else which might use significant amounts of memory). -- James Kanze
[toc] | [prev] | [next] | [standalone]
| From | Adam Skutt <askutt@gmail.com> |
|---|---|
| Date | 2011-09-06 17:26 -0700 |
| Message-ID | <4ac8a950-3876-4c56-bcd6-afe2f2ea2cf1@s20g2000yql.googlegroups.com> |
| In reply to | #9840 |
On Sep 6, 6:39 pm, James Kanze <james.ka...@gmail.com> wrote: > On Sep 6, 3:13 am, Adam Skutt <ask...@gmail.com> wrote: > > > On Sep 5, 9:20 pm, Adam Skutt <ask...@gmail.com> wrote: > > > On Sep 3, 7:02 pm, James Kanze <james.ka...@gmail.com> wrote: > > > > All of the Unices I know do implement per-process limits. Which > > > > can be useful in specific cases as well. > > > You only know a few then. POSIX specifies some crude (and useless) > > > per-/user/ limits, but per-process limits are not standard by any > > > stretch of the imagine. As I already explained, they also do nothing > > > to solve the problem, nor can they really. > > I should clarify this part. All Unicies provide some limits through > > ulimit, which provides mostly per-process limits that are user > > discretionary. Few provide the ability to enforce limits on all > > invocations of binary X, which is what I assumed you were talking > > about and what would be necessary. > > That's not what I was talking about, and that's absolutely > unnecessary. Then what is necessary? You clearly believe they provide something (which is simply wrong, ulimit is indeed what is standard) and you clearly believe whatever is provided is necessary. So stop stalling and answer, or be a grownup and admit you were wrong twice over. > Any reliable server will be running on a dedicated > machine, in a dedicated environment. It's cute that you believe that, but that's hardly the case anymore. > It's impossible to have > any sort of reliable system otherwise. All those very old IBM mainframe operators and systems would like to have many words with you. IBM (along with many others) have been doing virtualization, and doing it well, before the PC even existed. I'm not sure why you believe dedicated hardware is necessary. I'm not sure why you believe it improves robustness. > > > ulimit isn't helpful because the > > limits have to be high enough for the largest program the user wishes > > to run, and you cannot prevent the user from setting those limits for > > running other binaries. > > That's just bullshit. No, it's not. It's a fundamental limitation of discretionary access control. I can assign the same limits to /bin/bash as I can to /usr/ bin/firefox, even though the two don't need the same amount of resources. > A reliable server will be started from a > shell script, which will set ulimits just before starting it > (and will do nothing else which might use significant amounts of > memory). Who said we were talking about just servers? I'm not sure as hell not talking about just servers, nor have I ever been talking about just servers. Hell, you're not even talking about just servers and it would be disingenuous of you to claim otherwise. UNIX doesn't provide the resource controls you think it does, but even if it did, they wouldn't help the problem we're discussing. Adam
[toc] | [prev] | [next] | [standalone]
| From | James Kanze <james.kanze@gmail.com> |
|---|---|
| Date | 2011-09-06 15:35 -0700 |
| Message-ID | <efc2a7bb-cc7b-403c-b2b5-5f2201d95df8@dp9g2000vbb.googlegroups.com> |
| In reply to | #9769 |
On Sep 6, 2:20 am, Adam Skutt <ask...@gmail.com> wrote: > On Sep 3, 7:02 pm, James Kanze <james.ka...@gmail.com> wrote: > > On Sep 2, 1:25 am, Joshua Maurice <joshuamaur...@gmail.com> wrote: > > > For most processes which are not the ones > > > being "abusive" but merely innocent bystanders, including system > > > processes, I suspect they will behave similarly. With overcommit on, > > > they will be killed with great prejudice. With overcommit off, when > > > they get the malloc error, most will respond just the same and die a > > > quick death. > > That would be a very poorly written service that died just > > because of a malloc failure. > Then are you willing to make the claim most Linux, UNIX, OS X and > Windows services are poorly written? Because that's what you just > said, merely using different words. I don't know about OS X, since I've never used it, but it's true that Linux and Windows services are far from robust. Neither are really what I'd take as a model. (I wouldn't use either Linux or Windows if I needed a reliable server.) > > All services, critical or not, should have some sort of > > reasonable response to all possible error conditions. > Crashing is a perfectly reasonable response to an error condition, > really most error conditions. I'm not sure why anyone would think > otherwise even for a second. It depends on the application. For most systems, crashing is the only acceptable response for a detected programming error. But depending on the application, running out of memory isn't necessary a programming error. -- James Kanze
[toc] | [prev] | [next] | [standalone]
| From | Adam Skutt <askutt@gmail.com> |
|---|---|
| Date | 2011-09-06 17:28 -0700 |
| Message-ID | <db418f00-f330-4b4e-b180-ba3cfe06238f@x11g2000yqc.googlegroups.com> |
| In reply to | #9839 |
On Sep 6, 6:35 pm, James Kanze <james.ka...@gmail.com> wrote: > On Sep 6, 2:20 am, Adam Skutt <ask...@gmail.com> wrote: > > > > All services, critical or not, should have some sort of > > > reasonable response to all possible error conditions. > > Crashing is a perfectly reasonable response to an error condition, > > really most error conditions. I'm not sure why anyone would think > > otherwise even for a second. > > It depends on the application. For most systems, crashing is > the only acceptable response for a detected programming error. > But depending on the application, running out of memory isn't > necessary a programming error. However, there's generally no way to tell. Plus, your implication 'If it's not a programming error, I can handle it' simply does not follow. Adam
[toc] | [prev] | [next] | [standalone]
| From | Adam Skutt <askutt@gmail.com> |
|---|---|
| Date | 2011-09-01 21:01 -0700 |
| Message-ID | <d816cc21-78a9-479c-b595-e4c26d1099d8@eb1g2000vbb.googlegroups.com> |
| In reply to | #9636 |
On Sep 1, 7:58 pm, James Kanze <james.ka...@gmail.com> wrote: > If you can't write your output, then logging will fail. But > that's a different problem from running out of memory. > Not particularly. Our goal here is to never fail. It's by definition impossible. > > > I tried, a long time ago, to eat all my memory and then proceed to > > > disk I/O. This works on e.g. Unix and windows. Why wouldn't it? > > Sure, if you're using read(2) and write(2) (or equivalents) and have > > already allocated your buffers, then being out of memory won't require > > any additional allocations on the part of your process. Of course, > > performing I/O requires more effort than just the read and write > > calls, and many (most?) people don't write code that uses such low- > > level interfaces. Those interfaces frequently do not (e.g., C++ > > iostreams) make it easy or even possible to ensure that any given I/O > > operation will not cause memory allocation to occur. > > It's very simple to ensure that a write to an ostream doesn't do > any allocations, if you design your streambuf correctly. > Writing my own streambuf is more work than I ever want to do, and really more work than anyone should ever have to do, in order to handle an exception. Which was my whole point initially. C++ doesn't provide the facilities out of the box for doing anything other than some sort of termination on an OOM condition. > Yes, but if you're logging an out of memory condition, you don't > need any of those conversions (or you know which ones you need, > and you can use static or pre-allocated memory for them). > Yes, but that's not the only behavior that was advocated in an OOM condition. Attempting to save program state was advocated, and that may well require converting state. In fact, in context, it started as discussion about trying to save after std::bad_alloc was thrown, not just merely log the OOM condition. > Those due to handling a specific request. One obvious example > is parsing filters in LDAP; the filter can contain an > arbitrarily complex expression, which must be represented in > memory. If you run out of memory to represent it, you abort the > request (freeing the memory) with an "insufficient resources" > error. That doesn't mean that you can't handle more reasonable > requests. It doesn't mean you can, either. And it hardly justifies the effort involved in handling the OOM condition. > > If the operating system's virtual memory allows for memory allocation > > by other processes to cause allocation failure in my own, then > > ultimately I may be forced to crash anyway. Many operating systems > > kernel panic (i.e., stop completely) if they reach their commit limit > > and have no way of raising the limit (e.g., adding swap automatically > > or expanding an existing file). Talking about other processes when > > all mainstream systems provide robust virtual memory systems is > > tomfoolery. > > All mainstream systems except Linux (and I think Windows, and > some versions of AIX, and I think some versions of HP/UX as > well), you mean. The default configuration of Linux will start > killing random processes when memory gets tight (rather than > returning an error from the system request for memory). > I'm not sure how what you wrote has anything whatsoever to do with what I said. Adam
[toc] | [prev] | [next] | [standalone]
| From | James Kanze <james.kanze@gmail.com> |
|---|---|
| Date | 2011-09-03 16:14 -0700 |
| Message-ID | <ccf6af63-55e7-4ae7-a875-c7887187bf79@fi7g2000vbb.googlegroups.com> |
| In reply to | #9642 |
On Sep 2, 5:01 am, Adam Skutt <ask...@gmail.com> wrote: > On Sep 1, 7:58 pm, James Kanze <james.ka...@gmail.com> wrote: > > If you can't write your output, then logging will fail. But > > that's a different problem from running out of memory. > Not particularly. Our goal here is to never fail. It's by > definition impossible. Logging is not normally part of the "essential" job of the application. > > > > I tried, a long time ago, to eat all my memory and then proceed to > > > > disk I/O. This works on e.g. Unix and windows. Why wouldn't it? > > > Sure, if you're using read(2) and write(2) (or equivalents) and have > > > already allocated your buffers, then being out of memory won't require > > > any additional allocations on the part of your process. Of course, > > > performing I/O requires more effort than just the read and write > > > calls, and many (most?) people don't write code that uses such low- > > > level interfaces. Those interfaces frequently do not (e.g., C++ > > > iostreams) make it easy or even possible to ensure that any given I/O > > > operation will not cause memory allocation to occur. > > It's very simple to ensure that a write to an ostream doesn't do > > any allocations, if you design your streambuf correctly. > Writing my own streambuf is more work than I ever want to do, and > really more work than anyone should ever have to do, in order to > handle an exception. I've yet to see a logging system in a large system which didn't use custom streambuf. (For that matter, at least on large scale servers, I'd say that outputting to custom streambuf's is far more frequent than outputting to the standard streambuf's.) > Which was my whole point initially. C++ doesn't > provide the facilities out of the box for doing anything other than > some sort of termination on an OOM condition. I'm not sure what you mean by "the facilities out of the box". No language that I know provides the logging facilities necessary for a large scale server "out of the box"; C++ actually does far better than most in this regard. And no language, to my knowledge, provides any sort of transaction management (e.g. with rollback). > > Yes, but if you're logging an out of memory condition, you don't > > need any of those conversions (or you know which ones you need, > > and you can use static or pre-allocated memory for them). > Yes, but that's not the only behavior that was advocated in an OOM > condition. Attempting to save program state was advocated, and that > may well require converting state. Or may not, if it is designed correctly. (This is one of the rare cases where just dumping the memory image of certain struct's might be appropriate.) > In fact, in context, it started as discussion about trying to > save after std::bad_alloc was thrown, not just merely log the > OOM condition. I've not followed the discussion completely, but most of what I've seen seems to concern returning an error for the request triggering the problem, and continuing to handle other requests. > > Those due to handling a specific request. One obvious example > > is parsing filters in LDAP; the filter can contain an > > arbitrarily complex expression, which must be represented in > > memory. If you run out of memory to represent it, you abort the > > request (freeing the memory) with an "insufficient resources" > > error. That doesn't mean that you can't handle more reasonable > > requests. > It doesn't mean you can, either. Unless the reason you've run out of memory is a memory leak, you can. > And it hardly justifies the effort involved in handling the > OOM condition. The effort isn't that hard, and it's a basic requirement for some applications. If you don't handle OOM correctly, you're application isn't correct. > > > If the operating system's virtual memory allows for memory allocation > > > by other processes to cause allocation failure in my own, then > > > ultimately I may be forced to crash anyway. Many operating systems > > > kernel panic (i.e., stop completely) if they reach their commit limit > > > and have no way of raising the limit (e.g., adding swap automatically > > > or expanding an existing file). Talking about other processes when > > > all mainstream systems provide robust virtual memory systems is > > > tomfoolery. > > All mainstream systems except Linux (and I think Windows, and > > some versions of AIX, and I think some versions of HP/UX as > > well), you mean. The default configuration of Linux will start > > killing random processes when memory gets tight (rather than > > returning an error from the system request for memory). > I'm not sure how what you wrote has anything whatsoever to do with > what I said. I'm not sure either. But it is something that must be kept in mind when you are writing an application which has to handle OOM. -- James Kanze
[toc] | [prev] | [next] | [standalone]
| From | "io_x" <a@b.c.invalid> |
|---|---|
| Date | 2011-09-05 10:31 +0200 |
| Message-ID | <4e64884c$0$44200$4fafbaef@reader1.news.tin.it> |
| In reply to | #9714 |
"James Kanze" <james.kanze@gmail.com> ha scritto nel messaggio news:ccf6af63-55e7-4ae7-a875-c7887187bf79@fi7g2000vbb.googlegroups.com... > On Sep 2, 5:01 am, Adam Skutt <ask...@gmail.com> wrote: >> On Sep 1, 7:58 pm, James Kanze <james.ka...@gmail.com> wrote: > >> > If you can't write your output, then logging will fail. But >> > that's a different problem from running out of memory. > >> Not particularly. Our goal here is to never fail. It's by >> definition impossible. > > Logging is not normally part of the "essential" job of the > application. > >> > > > I tried, a long time ago, to eat all my memory and then proceed to >> > > > disk I/O. This works on e.g. Unix and windows. Why wouldn't it? >> > > Sure, if you're using read(2) and write(2) (or equivalents) and have >> > > already allocated your buffers, then being out of memory won't require >> > > any additional allocations on the part of your process. Of course, >> > > performing I/O requires more effort than just the read and write >> > > calls, and many (most?) people don't write code that uses such low- >> > > level interfaces. Those interfaces frequently do not (e.g., C++ >> > > iostreams) make it easy or even possible to ensure that any given I/O >> > > operation will not cause memory allocation to occur. > >> > It's very simple to ensure that a write to an ostream doesn't do >> > any allocations, if you design your streambuf correctly. > >> Writing my own streambuf is more work than I ever want to do, and >> really more work than anyone should ever have to do, in order to >> handle an exception. i not have enought experience for talk about above but because i'm in talk mode... the right language for a tcp server is ... assembly :) and use one library where *is know* who use global variables if the functions are thread safe who and where allocate memory etc. a library that is know to cpu instructions it is not sufficient above there would be a easy way to find leak and errors i don't say it is not possible in cpp too but could be more heavy work to be right and because there is something eye not see, can be more difficult for find errors or could be easier to do errors > I've yet to see a logging system in a large system which didn't > use custom streambuf. (For that matter, at least on large scale > servers, I'd say that outputting to custom streambuf's is far > more frequent than outputting to the standard streambuf's.) this is because one has to know where memory is request and thread safe of functions >> Which was my whole point initially. C++ doesn't >> provide the facilities out of the box for doing anything other than >> some sort of termination on an OOM condition. > > I'm not sure what you mean by "the facilities out of the box". > No language that I know provides the logging facilities > necessary for a large scale server "out of the box"; C++ > actually does far better than most in this regard. And no > language, to my knowledge, provides any sort of transaction > management (e.g. with rollback). >> > Yes, but if you're logging an out of memory condition, you don't >> > need any of those conversions (or you know which ones you need, >> > and you can use static or pre-allocated memory for them).
[toc] | [prev] | [next] | [standalone]
| From | Adam Skutt <askutt@gmail.com> |
|---|---|
| Date | 2011-09-05 18:36 -0700 |
| Message-ID | <573a5bcf-3819-4f92-b499-f9d512921e14@w28g2000yqw.googlegroups.com> |
| In reply to | #9714 |
On Sep 3, 7:14 pm, James Kanze <james.ka...@gmail.com> wrote: > On Sep 2, 5:01 am, Adam Skutt <ask...@gmail.com> wrote: > > > On Sep 1, 7:58 pm, James Kanze <james.ka...@gmail.com> wrote: > > > If you can't write your output, then logging will fail. But > > > that's a different problem from running out of memory. > > Not particularly. Our goal here is to never fail. It's by > > definition impossible. > > Logging is not normally part of the "essential" job of the > application. > If your goal is to improve robustness by logging the OOM condition, then it's not just essential, it's mandatory. If you fail to do it, you failed to improve robustness. > I've yet to see a logging system in a large system which didn't > use custom streambuf. (For that matter, at least on large scale > servers, I'd say that outputting to custom streambuf's is far > more frequent than outputting to the standard streambuf's.) > > I'm not sure what you mean by "the facilities out of the box". > No language that I know provides the logging facilities > necessary for a large scale server "out of the box"; C++ > actually does far better than most in this regard. And no > language, to my knowledge, provides any sort of transaction > management (e.g. with rollback). Then you don't know many languages! Java, Python, and many others provide robust, enterprise-grade logging facilities out of the box. Haskell, Erlang, and many others provide all sorts of transactional facilities, depending on exactly what you want, out of the box. > > Or may not, if it is designed correctly. (This is one of the > rare cases where just dumping the memory image of certain > struct's might be appropriate.) You're just making the cost/value proposition worse, not better. That's even more code I have to write! > I've not followed the discussion completely, but most of what > I've seen seems to concern returning an error for the request > triggering the problem, and continuing to handle other requests. > No, that's not even how the discussion started. One of the major advocates for handling OOM suggested this was not only possible, but trivial. > Unless the reason you've run out of memory is a memory leak, > you can. Nope. All the other requests can die while you're trying to handle the OOM condition. Or the other side could drop the request because they got tired of waiting. The reality of the matter is that both will happen. > The effort isn't that hard, Yes, it is. It requires me to rewrite a considerable number of language and sometimes even OS facilities, something you have admitted yourself! The entire reason I'm using a programming language is because it provides useful facilities for me As a result, it isn't the least bit unreasonable to conclude that rewriting language facilities is hard. If I wanted to be doing to be writing language facilities, then I'd just write my own damn programming language in the first place! > and it's a basic requirement for > some applications. If you don't handle OOM correctly, you're > application isn't correct. > Applications that require a response to OOM other than terminate are an unsubstantial minority. Systems that cannot permit termination as an OOM response are almost certainly broken. > I'm not sure either. But it is something that must be kept in > mind when you are writing an application which has to handle > OOM. And it makes justifying handling OOM only harder, not easier! You're making my case for me! Adam
[toc] | [prev] | [next] | [standalone]
| From | Paul <pchristor@yahoo.co.uk> |
|---|---|
| Date | 2011-09-06 04:09 -0700 |
| Message-ID | <6650498e-bbd5-4d80-86ad-6fb2b2f848f1@l4g2000vbv.googlegroups.com> |
| In reply to | #9770 |
On Sep 6, 2:36 am, Adam Skutt <ask...@gmail.com> wrote: > On Sep 3, 7:14 pm, James Kanze <james.ka...@gmail.com> wrote: > > > On Sep 2, 5:01 am, Adam Skutt <ask...@gmail.com> wrote: > > > > On Sep 1, 7:58 pm, James Kanze <james.ka...@gmail.com> wrote: > > > > If you can't write your output, then logging will fail. But > > > > that's a different problem from running out of memory. > > > Not particularly. Our goal here is to never fail. It's by > > > definition impossible. > > > Logging is not normally part of the "essential" job of the > > application. > > If your goal is to improve robustness by logging the OOM condition, > then it's not just essential, it's mandatory. If you fail to do it, > you failed to improve robustness. > > > I've yet to see a logging system in a large system which didn't > > use custom streambuf. (For that matter, at least on large scale > > servers, I'd say that outputting to custom streambuf's is far > > more frequent than outputting to the standard streambuf's.) > > > I'm not sure what you mean by "the facilities out of the box". > > No language that I know provides the logging facilities > > necessary for a large scale server "out of the box"; C++ > > actually does far better than most in this regard. And no > > language, to my knowledge, provides any sort of transaction > > management (e.g. with rollback). > > Then you don't know many languages! Java, Python, and many others > provide robust, enterprise-grade logging facilities out of the box. > Haskell, Erlang, and many others provide all sorts of transactional > facilities, depending on exactly what you want, out of the box. > > > > > Or may not, if it is designed correctly. (This is one of the > > rare cases where just dumping the memory image of certain > > struct's might be appropriate.) > > You're just making the cost/value proposition worse, not better. > That's even more code I have to write! > > > I've not followed the discussion completely, but most of what > > I've seen seems to concern returning an error for the request > > triggering the problem, and continuing to handle other requests. > > No, that's not even how the discussion started. One of the major > advocates for handling OOM suggested this was not only possible, but > trivial. > > > Unless the reason you've run out of memory is a memory leak, > > you can. > > Nope. All the other requests can die while you're trying to handle > the OOM condition. Or the other side could drop the request because > they got tired of waiting. The reality of the matter is that both > will happen. > > > The effort isn't that hard, > > Yes, it is. It requires me to rewrite a considerable number of > language and sometimes even OS facilities, something you have admitted > yourself! The entire reason I'm using a programming language is > because it provides useful facilities for me As a result, it isn't > the least bit unreasonable to conclude that rewriting language > facilities is hard. If I wanted to be doing to be writing language > facilities, then I'd just write my own damn programming language in > the first place! But how much effort is it to just enclose all allocations in a try- catch block? It's only one try block and this is all provided as a language feature. > > > and it's a basic requirement for > > some applications. If you don't handle OOM correctly, you're > > application isn't correct. > > Applications that require a response to OOM other than terminate are > an unsubstantial minority. Systems that cannot permit termination as > an OOM response are almost certainly broken. Looking at a user application on windows, termination on low memory condition is permitted but if the program were just to terminate at the slightest whiff of low memory it wouldn't be a very well written program IMHO. At least give an error message and inform the user why you are closing before termination, or just terminate the process that the memeory was requested for and keep the program alive thus allowing the user to close other apps and free up some memory. If some other program is leaking memory, your program will get the blame as being a rogue program because it just crashed in an unproffessional way. This alone is a good reason for error checking.
[toc] | [prev] | [next] | [standalone]
| From | James Kanze <james.kanze@gmail.com> |
|---|---|
| Date | 2011-09-06 15:49 -0700 |
| Message-ID | <862f5750-c735-41b6-8993-97bc8f6eed1f@hb5g2000vbb.googlegroups.com> |
| In reply to | #9770 |
On Sep 6, 2:36 am, Adam Skutt <ask...@gmail.com> wrote:
> On Sep 3, 7:14 pm, James Kanze <james.ka...@gmail.com> wrote:
[...]
> > I'm not sure what you mean by "the facilities out of the box".
> > No language that I know provides the logging facilities
> > necessary for a large scale server "out of the box"; C++
> > actually does far better than most in this regard. And no
> > language, to my knowledge, provides any sort of transaction
> > management (e.g. with rollback).
> Then you don't know many languages! Java, Python, and many others
> provide robust, enterprise-grade logging facilities out of the box.
> Haskell, Erlang, and many others provide all sorts of transactional
> facilities, depending on exactly what you want, out of the box.
I'm familiar with Java and Python, and neither provide any sort
of transactional management, nor adequate logging for large
scale applications.
> > Or may not, if it is designed correctly. (This is one of the
> > rare cases where just dumping the memory image of certain
> > struct's might be appropriate.)
> You're just making the cost/value proposition worse, not better.
> That's even more code I have to write!
You have to write it anyway, since it's part of making the
application robust.
> > I've not followed the discussion completely, but most of what
> > I've seen seems to concern returning an error for the request
> > triggering the problem, and continuing to handle other requests.
> No, that's not even how the discussion started. One of the major
> advocates for handling OOM suggested this was not only possible, but
> trivial.
Nothing in a large scale application is trivial, but handling
out of memory isn't more difficult than any number of other
things you have to do.
> > Unless the reason you've run out of memory is a memory leak,
> > you can.
> Nope. All the other requests can die while you're trying to handle
> the OOM condition. Or the other side could drop the request because
> they got tired of waiting. The reality of the matter is that both
> will happen.
The reality of the matter is that neither happens, if you
program correctly. I've written robust applications which
handled out of memory, and they worked.
> > The effort isn't that hard,
> Yes, it is.
It's no harder than a lot of other things necessary to write
correct software.
> It requires me to rewrite a considerable number of
> language and sometimes even OS facilities, something you have admitted
> yourself!
But so do logging, and transaction management, and a lot of
other things.
> The entire reason I'm using a programming language is
> because it provides useful facilities for me As a result, it isn't
> the least bit unreasonable to conclude that rewriting language
> facilities is hard. If I wanted to be doing to be writing language
> facilities, then I'd just write my own damn programming language in
> the first place!
But no languate has adequate logging facilities, nor transaction
management facilities, nor a lot of other things you need.
> > and it's a basic requirement for
> > some applications. If you don't handle OOM correctly, you're
> > application isn't correct.
> Applications that require a response to OOM other than terminate are
> an unsubstantial minority.
Finally something I can agree with. They're definitely a
minority. But they do exist. (I'd guess, on the whole, they
represent less than 10% of the applications I've worked on. But
it's hard to quantify, since a lot of lower level applications
I've worked on in the past didn't use dynamic memory, period.)
> Systems that cannot permit termination as
> an OOM response are almost certainly broken.
A system is broken if it doesn't meet its requirements.
> > I'm not sure either. But it is something that must be kept in
> > mind when you are writing an application which has to handle
> > OOM.
> And it makes justifying handling OOM only harder, not easier! You're
> making my case for me!
You're just ignoring the facts. Some applications (a minority)
have to handle OOM, at least in certain cases or configurations.
If they don't they're broken.
--
James Kanze
[toc] | [prev] | [next] | [standalone]
| From | Adam Skutt <askutt@gmail.com> |
|---|---|
| Date | 2011-09-06 17:11 -0700 |
| Message-ID | <28cbe59e-dd78-4e69-a21e-b120dec08c52@j1g2000yqn.googlegroups.com> |
| In reply to | #9842 |
On Sep 6, 6:49 pm, James Kanze <james.ka...@gmail.com> wrote: > On Sep 6, 2:36 am, Adam Skutt <ask...@gmail.com> wrote: > > On Sep 3, 7:14 pm, James Kanze <james.ka...@gmail.com> wrote: > > > I'm not sure what you mean by "the facilities out of the box". > > > No language that I know provides the logging facilities > > > necessary for a large scale server "out of the box"; C++ > > > actually does far better than most in this regard. And no > > > language, to my knowledge, provides any sort of transaction > > > management (e.g. with rollback). > > Then you don't know many languages! Java, Python, and many others > > provide robust, enterprise-grade logging facilities out of the box. > > Haskell, Erlang, and many others provide all sorts of transactional > > facilities, depending on exactly what you want, out of the box. > > I'm familiar with Java and Python, and neither provide any sort > of transactional management, nor adequate logging for large > scale applications. They do provide robust, enterprise grade logging in java.util.logging and simply logging OOB. Both provide what just about every other logging framework on the planet provide; and both provide something that's worlds better than what C++ provides out of the box, which is / nothing/. If you believe otherwise, then you either don't know what's actually done for large-scale applications or you don't actually know the languages. I'll leave that decision up to you. > > > Or may not, if it is designed correctly. (This is one of the > > > rare cases where just dumping the memory image of certain > > > struct's might be appropriate.) > > You're just making the cost/value proposition worse, not better. > > That's even more code I have to write! > > You have to write it anyway, since it's part of making the > application robust. Just because you claim it makes the application more robust doesn't mean that actually makes the application more robust. All I see is more code to test and debug, and therefore more points where things could fail. Until you give concrete reasoning as to why we should believe a program that terminates under OOM is less robust than one that does not, you have no basis for claiming I have to write an OOM handler of any sort. It's not a tautology that fewer terminations equal improved robustness, no matter how much you wish it were. So I would start there, honestly. Good luck with that. > > > > I've not followed the discussion completely, but most of what > > > I've seen seems to concern returning an error for the request > > > triggering the problem, and continuing to handle other requests. > > No, that's not even how the discussion started. One of the major > > advocates for handling OOM suggested this was not only possible, but > > trivial. > > Nothing in a large scale application is trivial, but handling > out of memory isn't more difficult than any number of other > things you have to do. Yes, it is obviously considerably harder, seeing as I literally have to do all the work myself. Just about everything else discussed has frameworks and libraries to make my life easier. Handling OOM does not. > > > Unless the reason you've run out of memory is a memory leak, > > > you can. > > Nope. All the other requests can die while you're trying to handle > > the OOM condition. Or the other side could drop the request because > > they got tired of waiting. The reality of the matter is that both > > will happen. > > The reality of the matter is that neither happens, if you > program correctly. I can't stop the world on Linux, OS X, Windows, or Solaris. Doing that would require special language runtime and kernel support. So no, even if I 'program correctly', the things I mentioned still can happen. It's unavoidable: you cannot make the other threads stop executing while you handle the OOM condition. That means they're vulnerable to the OOM condition as well, as it's a /per process/ condition. > I've written robust applications which > handled out of memory, and they worked. > You mean they've worked in whatever conditions you managed to test them under. But without source, given your track record of basic factual errors, I don't really care what you claim you've coded. > It's no harder than a lot of other things necessary to write > correct software. Show me the library I can download to handle OOM conditions on Linux, Windows, and OS/X. I'll wait. > But so do logging, and transaction management, and a lot of > other things. Nope. If the language doesn't provide them, I can download libraries that provide them for me. The same is not the case for OOM. > Finally something I can agree with. They're definitely a > minority. But they do exist. If you actually believed this, you wouldn't be making the arguments you're making. There's a logical contradiction between the claims: 1) Handling OOM improves robustness 2) Handling OOM is easy 3) Handling OOM is only necessary, and only done, for a minority of applications. Statement 3 is inherently contradictory with the first two. Moreover, if you believed they were a minority, you wouldn't be arguing with me at all, since my argument is inherently a cost/benefit argument. So no, I don't believe you. Plus, you've contradicted yourself on this point elsewhere. > > Systems that cannot permit termination as > > an OOM response are almost certainly broken. > > A system is broken if it doesn't meet its requirements. Few systems specify, "You cannot terminate under OOM conditions" because it's a senseless and pointless requirement. > > > > I'm not sure either. But it is something that must be kept in > > > mind when you are writing an application which has to handle > > > OOM. > > And it makes justifying handling OOM only harder, not easier! You're > > making my case for me! > > You're just ignoring the facts. I'm not ignoring any facts. If the OS might indiscriminately kill my process anyway, then my precious OOM handler may never even run. That makes it even harder to justify writing the damn thing in the first place, not easier! Ergo, you're making my argument for me. Adam
[toc] | [prev] | [next] | [standalone]
| From | Goran <goran.pusic@gmail.com> |
|---|---|
| Date | 2011-09-07 00:24 -0700 |
| Message-ID | <40496a34-6e12-42ed-a6f4-8b5574678da4@j1g2000yqn.googlegroups.com> |
| In reply to | #9853 |
On Sep 7, 2:11 am, Adam Skutt <ask...@gmail.com> wrote: > On Sep 6, 6:49 pm, James Kanze <james.ka...@gmail.com> wrote: > > On Sep 6, 2:36 am, Adam Skutt <ask...@gmail.com> wrote: > > > On Sep 3, 7:14 pm, James Kanze <james.ka...@gmail.com> wrote: > > > > I'm not sure what you mean by "the facilities out of the box". > > > > No language that I know provides the logging facilities > > > > necessary for a large scale server "out of the box"; C++ > > > > actually does far better than most in this regard. And no > > > > language, to my knowledge, provides any sort of transaction > > > > management (e.g. with rollback). > > > Then you don't know many languages! Java, Python, and many others > > > provide robust, enterprise-grade logging facilities out of the box. > > > Haskell, Erlang, and many others provide all sorts of transactional > > > facilities, depending on exactly what you want, out of the box. > > > I'm familiar with Java and Python, and neither provide any sort > > of transactional management, nor adequate logging for large > > scale applications. > > They do provide robust, enterprise grade logging in java.util.logging > and simply logging OOB. Both provide what just about every other > logging framework on the planet provide; and both provide something > that's worlds better than what C++ provides out of the box, which is / > nothing/. If you believe otherwise, then you either don't know what's > actually done for large-scale applications or you don't actually know > the languages. I'll leave that decision up to you. > > > > > Or may not, if it is designed correctly. (This is one of the > > > > rare cases where just dumping the memory image of certain > > > > struct's might be appropriate.) > > > You're just making the cost/value proposition worse, not better. > > > That's even more code I have to write! > > > You have to write it anyway, since it's part of making the > > application robust. > > Just because you claim it makes the application more robust doesn't > mean that actually makes the application more robust. All I see is > more code to test and debug, and therefore more points where things > could fail. > > Until you give concrete reasoning as to why we should believe a > program that terminates under OOM is less robust than one that does > not, you have no basis for claiming I have to write an OOM handler of > any sort. The reasoning has been given to you on several occasions in this very thread. You are just refusing to acknowledge it. Here's a pretty typical situation (repeating oneself): code embarks on a task said task requires resources code is chugging along, allocating resources on the way, calculating (rinse, repeat) calculation finished, code frees all _temporary_ resources that were needed for calculation and are not part of result or state change Now... See that rinse, repeat part? Well, imagine that, for whatever reason, a resource was unavailable. Calculation is dead in the water. What's a good code to do? Just as for a "happy" path, it should free all that temporary resources. It should probably also revert all these state changes. All that frees resources. Once out, there's plenty breathing space. And all you can say "I've hit OOM, I should die"? Yeah, right. Further, it might be that said operation is the only operation you do. In that case (and I said this in my very first post), yeah, dying on OOM is just as good as rolling the stack and state down and exiting cleanly. In any other situation, dying on OOM is much less appealing. Goran.
[toc] | [prev] | [next] | [standalone]
| From | Joshua Maurice <joshuamaurice@gmail.com> |
|---|---|
| Date | 2011-08-31 09:59 -0700 |
| Message-ID | <9dbad43a-f341-44ac-b3cb-14cb8f3eb909@l2g2000vbn.googlegroups.com> |
| In reply to | #9561 |
On Aug 31, 5:18 am, Adam Skutt <ask...@gmail.com> wrote: > On Aug 31, 4:22 am, Goran <goran.pu...@gmail.com> wrote: > > > I disagree (obviously). Here's the way I see it: it all depends on the > > number of functions program executes. A simple programs who only does > > one thing (e.g. a "main" function with no "until something external > > says stop" in the call chain) in C++ benefits slightly from "die on > > OOM" approach (in C, or something else without exceptions, benefit is > > greater because there, error checking is very labor-demanding). In > > fact, it benefits from "die on any problem" approach. > > > Programs that do more than one function are at a net loss with "die on > > OOM" approach, and the loss is bigger the more the functions there are > > (and the more important they are. Imagine an image processing program. > > So you apply a transformation, and that OOMs. You die, your user loses > > his latest changes that worked. But if you go back the stack, clean > > all those resources transformation needed and say "sorry, OOM", he > > could have saved (heck, you could have done it for the user, given > > that we hit OOM). And... Dig this: trying to do the same at the spot > > you hit OOM is a __mighty__ bad idea. Why? Because memory, and other > > resources, are likely already scarce, and an attempt to do anything > > might fail do to that. > > Trying to do it at any point is a mighty bad idea. Rolling back the > stack and deallocating memory _does not ensure_ future allocations > will succeed. Once you reach OOM, you're not assured the user will be > able to save; you're not assured you can do anything at all. Trying > may very well be pointless. > > It's considerably smarter, and absurdly easier, to save off the file > before attempting the operation in the first place. This way, your > code can die in a clean fashion, and the user has a recovery file with > the changes up to the failed operation. Trying to create the recovery > file under or after an OOM condition is very difficult. However, > trying to create the recovery file before the OOM condition is > trivial. This is disingenuous reasoning. I agree that once you hit OOM you're not guaranteed that you can save. However, even before you hit OOM you're not guaranteed that you can save. By this fallacious pedantic reasoning, "you're never guaranteed of anything, so why try?". I do agree that the way you achieve reliability in practice, specifically the way you achieve fault tolerance, is through fault isolation. For C++ on modern desktops, the first good layer of fault isolation is the process. After that is separate hardware. Then there's other points of failure like shared hardware ala networks, power supply, and so on. I would hazard a guess that a fair share of programs could reasonably attempt to recover from an OOM because they will free memory when they start freeing the stack. I also think that any reasonable program written to recover from OOM would be able to report it to the user without allocating memory. You'd have to use some non-portable OS functions, possibly another process to send the error through a pipe, but it's relatively easy to do, if annoying. You are doing a common mistake IMHO. You are automatically assuming that the unix / Windows process is the default, best, and perhaps only level of fault tolerance in a system. You seem to think it's ok to try to not report an error when a process dies when it hits OOM. I don't follow this line of reasoning at all. Does that mean that whenever I hit any error I should never log a report? That is absurd. So, you're going to assert that I should only log an error when X happens but not Y? That's incredibly silly. I like having as much longing as possible to debug issues when they happen. > In all of the counter-examples everyone's provided so far, the correct > way to ensure reliability is not to attempt to avoid crashing on OOM. > The correct way is to write your code so that crashing on OOM becomes > irrelevant. This has the added bonus that your code will still be > reliable even if your process is never told of the OOM condition. As > I mentioned before, in some languages you're not promised to be told > about allocation failure, and other have mentioned that on some > operating systems your process may simply die instead of being told > about the failure. This is a C++ newsgroup. Also, to be pedantic, those OSs have an option to turn off overcommit. It's an unfortunate reality due to the fork-exec bug in POSIX (fork as the only process creation is the bug), but we deal with what we have. [...] > > First off, when you actually hit the top- > > level exception handler, chances are, you will have freed some memory. > > Which doesn't mean you can make use of it. When you get > std::bad_alloc, you must assume there's 0 bytes available. No new, or > malloc, etc., anywhere. Why? Again, see the earlier reference to fallacious reasoning. You must always assume that there's 0 bytes available, both before and after any std::bad_alloc errors. > > There is difficulty, I agree with that, but it's actually trivial: keep in mind > > that, once you hit that OOM handler (most likely, some top-level > > exception handler not necessarily tied to OOM), you have all you might > > need prepared upfront. > > Which is exceptionally hard. Let's say I'm going to attempt your file > writing idea. I have to have the stream itself already open and > ready. I have to make sure the internal streambuf has enough memory > to buffer my I/O (or is unbuffered). I have to make sure my data > structures are constructed in such a fashion so that no memory > allocation occurs when I access the data, and that the data is > accessible when the handler runs. This means no copying of anything, > which is not impossible to ensure but difficult. > > You gloss over the difficultly, especially when you don't know what > code will allocate memory, nor when it will do it. Memory allocation > can and does occur in places you don't expect, so ensuring memory > allocation doesn't happen means being intimately familiar with the > implementation of every type involved in your OOM handler. > > Personally, I have better things to do then familiarize my self with > the inner guts of my C++ runtime implementation. Agreed. I don't mean to gloss over the difficulty. You have to abandon basically all portable code at this point and use the system APIs, like POSIX write. Annoying, but doable.
[toc] | [prev] | [next] | [standalone]
| From | Adam Skutt <askutt@gmail.com> |
|---|---|
| Date | 2011-08-31 11:24 -0700 |
| Message-ID | <6bfba27a-a5bd-42df-b3b9-d1d393958665@fd21g2000vbb.googlegroups.com> |
| In reply to | #9571 |
On Aug 31, 12:59 pm, Joshua Maurice <joshuamaur...@gmail.com> wrote: > > Trying to do it at any point is a mighty bad idea. Rolling back the > > stack and deallocating memory _does not ensure_ future allocations > > will succeed. Once you reach OOM, you're not assured the user will be > > able to save; you're not assured you can do anything at all. Trying > > may very well be pointless. > > > It's considerably smarter, and absurdly easier, to save off the file > > before attempting the operation in the first place. This way, your > > code can die in a clean fashion, and the user has a recovery file with > > the changes up to the failed operation. Trying to create the recovery > > file under or after an OOM condition is very difficult. However, > > trying to create the recovery file before the OOM condition is > > trivial. > > This is disingenuous reasoning. I agree that once you hit OOM you're > not guaranteed that you can save. However, even before you hit OOM > you're not guaranteed that you can save. By this fallacious pedantic > reasoning, "you're never guaranteed of anything, so why try?". Except that isn't my reasoning. Try reading what I said again, taking note of the fact I said, 'one is easy' and 'one is hard'. > I would hazard a guess that a fair share of programs could reasonably > attempt to recover from an OOM because they will free memory when they > start freeing the stack. No, that doesn't guarantee you a thing, especially if your goal is to retry the operation you failed in the first place. You'll simply reallocate the same amount of memory, again, and fail in the same place, again. The OOM handler must go out of its way to ensure additional memory gets freed if a retry is desirable (generally, if anything other than termination is desirable). > I also think that any reasonable program > written to recover from OOM would be able to report it to the user > without allocating memory. The key part of your statement is "written to recover from OOM". The entire argument I'm making is that is impressively difficult and rarely worth the effort. It is much, much harder than anyone here seems to believe it is. It also gains you very little. Users tend to notice crashed programs, after all. > You'd have to use some non-portable OS > functions, possibly another process to send the error through a pipe, > but it's relatively easy to do, if annoying. Non-portable OS functions are a non-starter for many codebases, and it's not easy to do when compared to using an I/O library. Reliably ensuring that process is around is not easy to do. That's far more complicated then keeping a file and buffers open! If you're going for easy compared to other suggestions, you're going in the wrong direction. > > You are doing a common mistake IMHO. You are automatically assuming > that the unix / Windows process is the default, best, and perhaps only > level of fault tolerance in a system. I've not done anything of the sort. You are assuming a relatively UNIX / Windows centric existence by suggesting the use of pipes as a way to log an OOM condition, though. > You seem to think it's ok to try > to not report an error when a process dies when it hits OOM. Yes, because the effort involved in doing so is quite substantial. > I don't > follow this line of reasoning at all. Does that mean that whenever I > hit any error I should never log a report? Quite possibly. It depends on what you're doing, but even if I attempted to log something whenever my program is about to exit, I certainly wouldn't normally bother in ensuring that logging works under an OOM condition. Nor would I worry too terribly much about what happens if the logging fails due to an I/O problem. Neither is easy to fix, especially in the situation in question. > That is absurd. So, you're > going to assert that I should only log an error when X happens but not > Y? That's incredibly silly. Yes, in general, you should only log things that are useful to the user / administrator / whomever. Whether "I ran out of memory" is helpful or not depends on the situation. Even when it is helpful, the effort involved in ensuring such a message is guaranteed to be noted is rarely worth it. > This is a C++ newsgroup. And? Good programming advice is often language independent. Many languages have exception semantics very similar to those of C++, so good exception handling behavior should be cross-language. Moreover, the larger point is this: some very popular languages by design make what you want to do impossible. As a result, it stands to reason you're grossly overstating the importance of trying to handle OOM conditions in any way but termination. > Also, to be pedantic, those OSs have an > option to turn off overcommit. Turning off overcommit doesn't ensure your process stays alive. Even when it does, you may have traded the survival of your process for the loss of the whole system. > It's an unfortunate reality due to the > fork-exec bug in POSIX (fork as the only process creation is the bug), > but we deal with what we have. fork()/exec() is not the primary reason for overcommit support in Linux and other operating systems. Overcommit comes about because many application do use allocation strategies that ask for more memory from the operating system then they ever use. fork()/exec() is really a corner case. > Why? Again, see the earlier reference to fallacious reasoning. You > must always assume that there's 0 bytes available, both before and > after any std::bad_alloc errors. No, that does not follow at all. You need to consider the goal of the exception handler: succeed no matter what state the application is in vs. the goals of the rest of the program: succeed only if possible. The latter means no memory allocation. > Agreed. I don't mean to gloss over the difficulty. You have to abandon > basically all portable code at this point and use the system APIs, > like POSIX write. Annoying, but doable. And a non-starter, especially for a C++ newsgroup! Adam
[toc] | [prev] | [next] | [standalone]
| From | Joshua Maurice <joshuamaurice@gmail.com> |
|---|---|
| Date | 2011-09-01 16:29 -0700 |
| Message-ID | <dec7cdd4-1f03-4656-84c2-430c58b4c46e@u7g2000pro.googlegroups.com> |
| In reply to | #9572 |
On Aug 31, 11:24 am, Adam Skutt <ask...@gmail.com> wrote: > On Aug 31, 12:59 pm, Joshua Maurice <joshuamaur...@gmail.com> wrote: > > I would hazard a guess that a fair share of programs could reasonably > > attempt to recover from an OOM because they will free memory when they > > start freeing the stack. > > No, that doesn't guarantee you a thing, especially if your goal is to > retry the operation you failed in the first place. You'll simply > reallocate the same amount of memory, again, and fail in the same > place, again. The OOM handler must go out of its way to ensure > additional memory gets freed if a retry is desirable (generally, if > anything other than termination is desirable). Again, I don't think anyone is suggesting a naive retry of a failed job on OOM. We're suggesting logging the error first, and possibly more sophisticated schemes of retry. [...] > > Also, to be pedantic, those OSs have an > > option to turn off overcommit. > > Turning off overcommit doesn't ensure your process stays alive. Even > when it does, you may have traded the survival of your process for the > loss of the whole system. > > > It's an unfortunate reality due to the > > fork-exec bug in POSIX (fork as the only process creation is the bug), > > but we deal with what we have. > > fork()/exec() is not the primary reason for overcommit support in > Linux and other operating systems. Overcommit comes about because > many application do use allocation strategies that ask for more memory > from the operating system then they ever use. fork()/exec() is really > a corner case. Interesting. I've tried to google for more information on this topic, but there's not much I can find offhand about the actual initial motivations for overcommit, and the current motivations for leaving overcommit in. Furthermore, the current situation is even more broken than I thought. It seems there's no good way to limit (virtual) memory usage on a per- user basis in Linux (and presumably other unix-like OSes (?)). Thus, your options are 1- overcommit and the OOM killer on a relatively random process, or 2- no overcommit and an OOM failure return code or exception from malloc et al in a relatively random process. This is completely broken. I often wonder how such states can persist for so long. It's not just me, right? Other people do see how this is broken, right? Why is no one fixing this? It can't be that hard to implement per-user limits. With this in mind, I give slightly more credence to your approach, or at least I'm more open minded as my approach has significant limitations as well. > > Why? Again, see the earlier reference to fallacious reasoning. You > > must always assume that there's 0 bytes available, both before and > > after any std::bad_alloc errors. > > No, that does not follow at all. You need to consider the goal of the > exception handler: succeed no matter what state the application is in > vs. the goals of the rest of the program: succeed only if possible. > The latter means no memory allocation. Why do you claim that those are their goals? Has a stakeholder ever given you such requirements? > > Agreed. I don't mean to gloss over the difficulty. You have to abandon > > basically all portable code at this point and use the system APIs, > > like POSIX write. Annoying, but doable. > > And a non-starter, especially for a C++ newsgroup! The distinction is that you were discussing strategies for other languages, which is why I rightly noted that it's irrelevant to the conversation and the newsgroup. I will now perhaps acknowledge that you were trying to shed some light on the current situation by referencing how other languages do it (in an omitted quote). I was possibly incorrect for jumping to "off topic". However, in the above quote, I talked about using non-portable mechanisms to achieve some degree of reliability, which is relevant to C++ and this newsgroup. This newsgroup is not purely C++ standard - that's comp.std.c++. This newsgroup does have a strong bias towards portable programming, but if the only reasonable solution is to use POSIX or win32 APIs, then it is appropriate to suggest that here.
[toc] | [prev] | [next] | [standalone]
| From | Joshua Maurice <joshuamaurice@gmail.com> |
|---|---|
| Date | 2011-09-01 16:42 -0700 |
| Message-ID | <636d210e-5f3a-45b0-a505-10d085af34d0@p25g2000pri.googlegroups.com> |
| In reply to | #9631 |
On Sep 1, 4:29 pm, Joshua Maurice <joshuamaur...@gmail.com> wrote: > On Aug 31, 11:24 am, Adam Skutt <ask...@gmail.com> wrote: > > > On Aug 31, 12:59 pm, Joshua Maurice <joshuamaur...@gmail.com> wrote: > > > It's an unfortunate reality due to the > > > fork-exec bug in POSIX (fork as the only process creation is the bug), > > > but we deal with what we have. > > > fork()/exec() is not the primary reason for overcommit support in > > Linux and other operating systems. Overcommit comes about because > > many application do use allocation strategies that ask for more memory > > from the operating system then they ever use. fork()/exec() is really > > a corner case. > > Interesting. I've tried to google for more information on this topic, > but there's not much I can find offhand about the actual initial > motivations for overcommit, and the current motivations for leaving > overcommit in. > > Furthermore, the current situation is even more broken than I thought. > It seems there's no good way to limit (virtual) memory usage on a per- > user basis in Linux (and presumably other unix-like OSes (?)). Thus, > your options are 1- overcommit and the OOM killer on a relatively > random process, or 2- no overcommit and an OOM failure return code or > exception from malloc et al in a relatively random process. This is > completely broken. I often wonder how such states can persist for so > long. It's not just me, right? Other people do see how this is broken, > right? Why is no one fixing this? It can't be that hard to implement > per-user limits. > > With this in mind, I give slightly more credence to your approach, or > at least I'm more open minded as my approach has significant > limitations as well. Heck, I don't know what the fix is. I do know this is broken, and it seems there's got to be something better than the current situation.
[toc] | [prev] | [next] | [standalone]
| From | Adam Skutt <askutt@gmail.com> |
|---|---|
| Date | 2011-09-01 21:37 -0700 |
| Message-ID | <a8ea8bd6-6402-4893-9ddc-9594c488cb70@m18g2000vbe.googlegroups.com> |
| In reply to | #9631 |
On Sep 1, 7:29 pm, Joshua Maurice <joshuamaur...@gmail.com> wrote: > Again, I don't think anyone is suggesting a naive retry of a failed > job on OOM. That's precisely what you suggested when you claimed that all we had to do is rollback down the stack! > We're suggesting logging the error first, and possibly > more sophisticated schemes of retry. Then you need to define these "more sophisticated schemes", and define such a scheme that's actually worth it to implement most of the time. > Interesting. I've tried to google for more information on this topic, > but there's not much I can find offhand about the actual initial > motivations for overcommit, and the current motivations for leaving > overcommit in. > > Furthermore, the current situation is even more broken than I thought. > It seems there's no good way to limit (virtual) memory usage on a per- > user basis in Linux (and presumably other unix-like OSes (?)). Yes, there is. You can set the maximize size of the VAS per-process, and the maximum number of processes, which provides a hard upper bound. If you want finer grained controls, you have to patch the Linux kernel; there are various patches out there that can accomplish what you want. Some other UNIX systems (e.g., Solaris) provide finer grain controls out of the box. There are other ways, such as containers and virtualization, to accomplish similar feats. They're rarely worth it. > Thus, > your options are 1- overcommit and the OOM killer on a relatively > random process, or 2- no overcommit and an OOM failure return code or > exception from malloc et al in a relatively random process. This is > completely broken. Compared to what alternative? The traditional alternative is a kernel panic, and that's not necessarily better (nor worse). For better or ill, we've become accustomed to the assumption we can treat memory as an endless resource. Most of the time, that assumption works out pretty well. When it falls apart, it's not shocking the resulting consequences are pretty terrible. > I often wonder how such states can persist for so > long. It's not just me, right? Other people do see how this is broken, > right? Why is no one fixing this? It can't be that hard to implement > per-user limits. Because per-user limits don't fix the problem, unless you limit every user on the system in such a fashion to never exceed your commit limit. Even then, you're still not promised the "memory hog" is the one that's going to be told no-more memory. That's why we don't bother: identifying the "memory hog" is much too hard for a computer. It may well be a broken situation, but there's also not a good solution. > > > No, that does not follow at all. You need to consider the goal of the > > exception handler: succeed no matter what state the application is in > > vs. the goals of the rest of the program: succeed only if possible. > > The latter means no memory allocation. > Why do you claim that those are their goals? Has a stakeholder ever > given you such requirements? Yes, plenty of people here seem to be convinced handling OOM (by not crashing) is a requirement for writing "robust" software; that means being able to perform some sort of action (such as logging) and continue onward after the OOM condition. However, some people here also seem to believe that performing I/O doesn't effect program state, so their requirements are probably not worth fulfilling. Still, this plays into my larger point: even if you have a situation where you can respond to an OOM condition in some meaningful fashion other than termination, you're still not assured of success. Put more plainly, handling OOM doesn't ipso facto ensure additional robustness. You need to be able to handle the OOM condition and have a reasonable assurance that your response will actually succeed. It may be an overstatement on my part to say, "Succeed no matter what", but making the software more "robust" certainly means tending closer to that extreme than the opposite. As a concrete example, writing all this OOM handling code does me no good if when my process finally hits a OOM condition, my whole computer is going to die anyway. For many applications, this is one of two reasons why they'll ever see an OOM condition. Of course, the reality is that all of this rarely makes software more robust, because actual robust systems generally are pretty tolerant of things like program termination. As I've said before, frequently it's even preferable to terminate, even when it would be possible to recover from the error. This makes the value proposition of handling OOM even less worthwhile. > However, in the above quote, I talked about using non-portable > mechanisms to achieve some degree of reliability, which is relevant to > C++ and this newsgroup. Yes, I know. And my whole point from the start is that handling OOM is too much of a pain to bother. Having to give up writing portable code definitely falls under "too much of a pain to bother" for lots of code and lots of programmers. None of this should be controversial in the least or require this much discussion. Adam
[toc] | [prev] | [next] | [standalone]
| From | Ian Collins <ian-news@hotmail.com> |
|---|---|
| Date | 2011-09-02 17:45 +1200 |
| Message-ID | <9cb8rmFfcoU1@mid.individual.net> |
| In reply to | #9643 |
On 09/ 2/11 04:37 PM, Adam Skutt wrote: > On Sep 1, 7:29 pm, Joshua Maurice<joshuamaur...@gmail.com> wrote: >> >>> No, that does not follow at all. You need to consider the goal of the >>> exception handler: succeed no matter what state the application is in >>> vs. the goals of the rest of the program: succeed only if possible. >>> The latter means no memory allocation. > >> Why do you claim that those are their goals? Has a stakeholder ever >> given you such requirements? > > Yes, plenty of people here seem to be convinced handling OOM (by not > crashing) is a requirement for writing "robust" software; that means > being able to perform some sort of action (such as logging) and > continue onward after the OOM condition. However, some people here > also seem to believe that performing I/O doesn't effect program state, > so their requirements are probably not worth fulfilling. > > Still, this plays into my larger point: even if you have a situation > where you can respond to an OOM condition in some meaningful fashion > other than termination, you're still not assured of success. Put more > plainly, handling OOM doesn't ipso facto ensure additional > robustness. You need to be able to handle the OOM condition and have > a reasonable assurance that your response will actually succeed. It > may be an overstatement on my part to say, "Succeed no matter what", > but making the software more "robust" certainly means tending closer > to that extreme than the opposite. I agree. On a descent hosted environment, memory exhaustion is usually down to either a system wide problem, or a programming error. If you require an application to be "robust" then you can use an external entity to manage and if necessary, restart it. This is common practice for system processes. I have done this in the past where an application ran in a resource limited Solaris zone and was monitored from outside of the zone. On a general purpose system, it is difficult and often impossible for an application to determine how much free memory the system will be able to provide. On systems with over commit any guess will be too optimistic and on system with processes then release memory on demand, too pessimistic. In a resource constrained environment, critical processes can perform all their allocations up front before any other processes start. > As a concrete example, writing all this OOM handling code does me no > good if when my process finally hits a OOM condition, my whole > computer is going to die anyway. For many applications, this is > one of two reasons why they'll ever see an OOM condition. I agree. > Of course, the reality is that all of this rarely makes software more > robust, because actual robust systems generally are pretty tolerant of > things like program termination. As I've said before, frequently it's > even preferable to terminate, even when it would be possible to > recover from the error. This makes the value proposition of handling > OOM even less worthwhile. Indeed, robustness goes beyond a single application. -- Ian Collins
[toc] | [prev] | [next] | [standalone]
Page 7 of 10 — ← Prev page 1 … 5 6 [7] 8 9 10 Next page →
Back to top | Article view | comp.lang.c++
csiph-web