Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.lang.c > #63176 > unrolled thread
| Started by | "Morten W. Petersen" <morphex@gmail.com> |
|---|---|
| First post | 2015-06-10 22:32 -0700 |
| Last post | 2015-06-10 23:07 -0700 |
| Articles | 20 on this page of 182 — 25 participants |
Back to article view | Back to comp.lang.c
Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-10 22:32 -0700
Re: Reason for segmentation violation Xavier Roche <xroche@free.fr.NOSPAM.invalid> - 2015-06-11 07:52 +0200
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-11 07:19 -0700
Re: Reason for segmentation violation Richard Heathfield <rjh@cpax.org.uk> - 2015-06-11 15:46 +0100
Re: Reason for segmentation violation Barry Schwarz <schwarzb@dqel.com> - 2015-06-11 11:23 -0700
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-11 12:08 -0700
Re: Reason for segmentation violation James Kuyper <jameskuyper@verizon.net> - 2015-06-11 15:44 -0400
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-11 12:55 -0700
Re: Reason for segmentation violation James Kuyper <jameskuyper@verizon.net> - 2015-06-11 16:29 -0400
Re: Reason for segmentation violation gordonb.h0fbo@burditt.org (Gordon Burditt) - 2015-06-17 23:51 -0500
Re: Reason for segmentation violation Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-06-17 23:10 -0700
Re: Reason for segmentation violation gordonb.0vo9r@burditt.org (Gordon Burditt) - 2015-06-18 03:30 -0500
Re: Reason for segmentation violation Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-06-18 02:40 -0700
Re: Reason for segmentation violation Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-06-11 21:00 +0100
Re: Reason for segmentation violation Barry Schwarz <schwarzb@dqel.com> - 2015-06-11 19:58 -0700
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-11 23:42 -0700
Re: Reason for segmentation violation Richard Heathfield <rjh@cpax.org.uk> - 2015-06-12 08:33 +0100
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-12 02:36 -0700
Re: Reason for segmentation violation Richard Heathfield <rjh@cpax.org.uk> - 2015-06-12 11:07 +0100
Re: Reason for segmentation violation gwowen <gwowen@gmail.com> - 2015-06-12 04:55 -0700
Re: Reason for segmentation violation Richard Heathfield <rjh@cpax.org.uk> - 2015-06-12 13:17 +0100
Re: Reason for segmentation violation Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-06-12 13:32 +0100
Re: Reason for segmentation violation Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-06-12 13:20 +0100
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-13 23:55 -0700
Restrictions on inline functions (Was: Reason for segmentation violation) Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-06-14 10:30 +0100
Re: Reason for segmentation violation James Kuyper <jameskuyper@verizon.net> - 2015-06-12 11:10 -0400
Re: Reason for segmentation violation David Thompson <dave.thompson2@verizon.net> - 2015-06-21 12:46 -0400
Re: Reason for segmentation violation Keith Thompson <kst-u@mib.org> - 2015-06-12 08:35 -0700
Re: Reason for segmentation violation James Kuyper <jameskuyper@verizon.net> - 2015-06-12 12:21 -0400
Re: Reason for segmentation violation Keith Thompson <kst-u@mib.org> - 2015-06-12 10:48 -0700
Re: Reason for segmentation violation Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-06-12 19:12 +0100
Re: Reason for segmentation violation David Kleinecke <dkleinecke@gmail.com> - 2015-06-12 08:49 -0700
Re: Reason for segmentation violation Keith Thompson <kst-u@mib.org> - 2015-06-12 09:13 -0700
Re: Reason for segmentation violation David Kleinecke <dkleinecke@gmail.com> - 2015-06-13 08:55 -0700
Re: Reason for segmentation violation Keith Thompson <kst-u@mib.org> - 2015-06-12 08:56 -0700
Re: Reason for segmentation violation glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2015-06-12 09:41 +0000
Re: Reason for segmentation violation Keith Thompson <kst-u@mib.org> - 2015-06-12 08:29 -0700
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-14 00:03 -0700
Re: Reason for segmentation violation Barry Schwarz <schwarzb@dqel.com> - 2015-06-14 00:58 -0700
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-14 02:14 -0700
Re: Reason for segmentation violation Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-06-14 04:26 -0700
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-14 06:29 -0700
Re: Reason for segmentation violation Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-06-14 15:06 +0100
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-14 07:25 -0700
Re: Reason for segmentation violation Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-06-14 16:59 +0100
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-14 11:51 -0700
Re: Reason for segmentation violation Richard Damon <Richard@Damon-Family.org> - 2015-06-14 16:43 -0400
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-14 14:16 -0700
Re: Reason for segmentation violation Richard Damon <Richard@Damon-Family.org> - 2015-06-14 20:35 -0400
Re: Reason for segmentation violation Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-06-15 01:19 -0700
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-15 03:57 -0700
Re: Reason for segmentation violation Bartc <bc@freeuk.com> - 2015-06-15 12:26 +0100
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-15 04:47 -0700
Re: Reason for segmentation violation Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-06-15 15:02 +0100
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-15 07:08 -0700
Re: Reason for segmentation violation Bartc <bc@freeuk.com> - 2015-06-15 18:56 +0100
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-15 20:09 -0700
Re: Reason for segmentation violation Bartc <bc@freeuk.com> - 2015-06-16 09:31 +0100
Re: Reason for segmentation violation Richard Heathfield <rjh@cpax.org.uk> - 2015-06-16 10:09 +0100
Re: Reason for segmentation violation Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-06-16 02:36 -0700
Re: Reason for segmentation violation Richard Heathfield <rjh@cpax.org.uk> - 2015-06-16 11:01 +0100
Re: Reason for segmentation violation Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-06-16 12:37 +0100
Re: Reason for segmentation violation Richard Damon <Richard@Damon-Family.org> - 2015-06-16 08:40 -0400
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-16 10:32 -0700
Re: Reason for segmentation violation Richard Damon <Richard@Damon-Family.org> - 2015-06-17 07:54 -0400
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-17 05:28 -0700
Re: Reason for segmentation violation Richard Damon <Richard@Damon-Family.org> - 2015-06-18 00:34 -0400
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-18 09:13 +0200
Re: Reason for segmentation violation Richard Damon <Richard@Damon-Family.org> - 2015-06-19 21:17 -0400
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-20 23:28 +0200
Re: Reason for segmentation violation "Rick C. Hodgin" <rick.c.hodgin@gmail.com> - 2015-06-15 05:26 -0700
Re: Reason for segmentation violation Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-06-15 06:20 -0700
Re: Reason for segmentation violation Richard Heathfield <rjh@cpax.org.uk> - 2015-06-15 14:25 +0100
Re: Reason for segmentation violation Keith Thompson <kst-u@mib.org> - 2015-06-15 07:52 -0700
Re: Reason for segmentation violation Richard Heathfield <rjh@cpax.org.uk> - 2015-06-15 18:29 +0100
Re: Reason for segmentation violation Keith Thompson <kst-u@mib.org> - 2015-06-15 11:15 -0700
Re: Reason for segmentation violation raltbos@xs4all.nl (Richard Bos) - 2015-06-18 20:41 +0000
Re: Reason for segmentation violation Bartc <bc@freeuk.com> - 2015-06-15 18:48 +0100
Re: Reason for segmentation violation Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-06-15 11:04 -0700
Re: Reason for segmentation violation raltbos@xs4all.nl (Richard Bos) - 2015-06-18 20:44 +0000
Re: Reason for segmentation violation David Brown <david.brown@hesbynett.no> - 2015-06-19 10:39 +0200
Re: Reason for segmentation violation gwowen <gwowen@gmail.com> - 2015-06-16 00:58 -0700
Re: Reason for segmentation violation raltbos@xs4all.nl (Richard Bos) - 2015-06-18 20:40 +0000
Re: Reason for segmentation violation James Kuyper <jameskuyper@verizon.net> - 2015-06-18 17:06 -0400
Re: Reason for segmentation violation Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-06-18 14:56 -0700
Re: Reason for segmentation violation Richard Heathfield <rjh@cpax.org.uk> - 2015-06-18 23:04 +0100
Re: Reason for segmentation violation James Kuyper <jameskuyper@verizon.net> - 2015-06-18 18:18 -0400
Re: Reason for segmentation violation David Brown <david.brown@hesbynett.no> - 2015-06-19 10:26 +0200
Re: Reason for segmentation violation David Kleinecke <dkleinecke@gmail.com> - 2015-06-19 08:57 -0700
Re: Reason for segmentation violation Richard Damon <Richard@Damon-Family.org> - 2015-06-15 08:54 -0400
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-15 06:24 -0700
Re: Reason for segmentation violation Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-06-15 14:32 +0100
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-15 05:43 -0700
Re: Reason for segmentation violation Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-06-14 23:28 +0100
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-14 16:31 -0700
Re: Reason for segmentation violation Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-06-15 01:27 +0100
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-15 05:34 -0700
Re: Reason for segmentation violation Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-06-15 15:00 +0100
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-15 20:37 -0700
Re: Reason for segmentation violation Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-06-16 09:58 +0100
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-16 10:37 -0700
Re: Reason for segmentation violation luser droog <luser.droog@gmail.com> - 2015-06-15 17:51 -0700
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-15 21:02 -0700
Re: Reason for segmentation violation James Kuyper <jameskuyper@verizon.net> - 2015-06-16 00:51 -0400
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-15 22:30 -0700
Re: Reason for segmentation violation James Kuyper <jameskuyper@verizon.net> - 2015-06-17 00:35 -0400
Re: Reason for segmentation violation Rosario19 <Ros@invalid.invalid> - 2015-06-19 18:51 +0200
Re: Reason for segmentation violation luser droog <luser.droog@gmail.com> - 2015-06-15 22:40 -0700
Re: Reason for segmentation violation luser droog <luser.droog@gmail.com> - 2015-06-15 23:23 -0700
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-16 11:07 -0700
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-16 23:54 -0700
Re: Reason for segmentation violation Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-06-17 10:51 +0100
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-17 03:46 -0700
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-17 04:17 -0700
Re: Reason for segmentation violation Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-06-17 14:02 +0100
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-17 06:15 -0700
Re: Reason for segmentation violation Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-06-17 15:34 +0100
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-17 07:58 -0700
Re: Reason for segmentation violation Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-06-17 16:04 +0100
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-17 08:16 -0700
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-18 14:57 +0200
Re: Reason for segmentation violation Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-06-18 15:35 +0100
Re: Reason for segmentation violation Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-06-18 15:44 +0100
Re: Reason for segmentation violation Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-06-18 08:18 -0700
Re: Reason for segmentation violation Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-06-18 16:42 +0100
Re: Reason for segmentation violation Keith Thompson <kst-u@mib.org> - 2015-06-17 08:19 -0700
Re: Reason for segmentation violation Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-06-17 16:35 +0100
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-17 08:49 -0700
Re: Reason for segmentation violation James Kuyper <jameskuyper@verizon.net> - 2015-06-17 11:56 -0400
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-18 09:46 +0200
Re: Reason for segmentation violation Keith Thompson <kst-u@mib.org> - 2015-06-17 09:14 -0700
Re: Reason for segmentation violation gordonb.446bm@burditt.org (Gordon Burditt) - 2015-06-18 01:56 -0500
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-18 09:54 +0200
Re: Reason for segmentation violation Keith Thompson <kst-u@mib.org> - 2015-06-17 09:08 -0700
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-22 16:53 +0200
Re: Reason for segmentation violation Richard Heathfield <rjh@cpax.org.uk> - 2015-06-22 16:32 +0100
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-22 18:21 +0200
Re: Reason for segmentation violation Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-06-22 16:33 +0100
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-22 18:20 +0200
Re: Reason for segmentation violation James Kuyper <jameskuyper@verizon.net> - 2015-06-22 13:05 -0400
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-22 19:47 +0200
Re: Reason for segmentation violation Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-06-17 13:52 +0100
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-17 06:05 -0700
Re: Reason for segmentation violation Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-06-17 15:39 +0100
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-17 08:02 -0700
Re: Reason for segmentation violation Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-06-17 16:12 +0100
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-18 10:46 +0200
Re: Reason for segmentation violation Richard Heathfield <rjh@cpax.org.uk> - 2015-06-17 16:12 +0100
Re: Reason for segmentation violation Tim Rentsch <txr@alumni.caltech.edu> - 2015-07-11 08:28 -0700
Re: Reason for segmentation violation Ian Collins <ian-news@hotmail.com> - 2015-06-18 07:32 +1200
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-17 22:02 +0200
Re: Reason for segmentation violation Keith Thompson <kst-u@mib.org> - 2015-06-17 13:16 -0700
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-17 22:32 +0200
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-23 06:52 +0200
Re: Reason for segmentation violation James Kuyper <jameskuyper@verizon.net> - 2015-06-17 16:16 -0400
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-17 22:44 +0200
Re: Reason for segmentation violation James Kuyper <jameskuyper@verizon.net> - 2015-06-17 17:39 -0400
Re: Reason for segmentation violation Robert Wessel <robertwessel2@yahoo.com> - 2015-06-18 01:57 -0500
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-18 09:29 +0200
Re: Reason for segmentation violation Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-06-17 13:57 -0700
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-17 23:10 +0200
Re: Reason for segmentation violation Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-06-17 14:22 -0700
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-17 23:41 +0200
Re: Reason for segmentation violation Richard Damon <Richard@Damon-Family.org> - 2015-06-18 00:40 -0400
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-18 09:18 +0200
Re: Reason for segmentation violation Keith Thompson <kst-u@mib.org> - 2015-06-17 08:52 -0700
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-17 09:27 -0700
Re: Reason for segmentation violation Keith Thompson <kst-u@mib.org> - 2015-06-17 11:00 -0700
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-14 07:48 -0700
Re: Reason for segmentation violation Bartc <bc@freeuk.com> - 2015-06-14 16:53 +0100
Re: Reason for segmentation violation Barry Schwarz <schwarzb@dqel.com> - 2015-06-14 10:59 -0700
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-14 11:45 -0700
Re: Reason for segmentation violation Richard Heathfield <rjh@cpax.org.uk> - 2015-06-14 20:01 +0100
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-14 12:40 -0700
Re: Reason for segmentation violation Richard Heathfield <rjh@cpax.org.uk> - 2015-06-14 21:00 +0100
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-14 13:13 -0700
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-14 13:17 -0700
Re: Reason for segmentation violation James Kuyper <jameskuyper@verizon.net> - 2015-06-14 16:36 -0400
Re: Reason for segmentation violation "Morten W. Petersen" <morphex@gmail.com> - 2015-06-14 13:43 -0700
Re: Reason for segmentation violation James Kuyper <jameskuyper@verizon.net> - 2015-06-14 14:50 -0400
Re: Reason for segmentation violation Barry Schwarz <schwarzb@dqel.com> - 2015-06-12 03:21 -0700
Re: Reason for segmentation violation Barry Schwarz <schwarzb@dqel.com> - 2015-06-10 23:07 -0700
Page 3 of 10 — ← Prev page 1 2 [3] 4 5 … 10 Next page →
| From | Malcolm McLean <malcolm.mclean5@btinternet.com> |
|---|---|
| Date | 2015-06-14 04:26 -0700 |
| Message-ID | <f47d5bfd-1b8f-44c4-b418-fc5690b1e950@googlegroups.com> |
| In reply to | #63234 |
On Sunday, June 14, 2015 at 10:14:52 AM UTC+1, Morten W. Petersen wrote: > > Yes, this makes sense, that calls to a function shouldn't terminate the caller. It is conceivable > that a user tries to import a huge XML file, which could ultimately lead to memory issues. > > Failures to (re)allocate memory is a fatal error, which means that it isn't any use trying to > figure things out. Therefore, I'm thinking that a setting a flag somewhere & a message + a > goto that jumps directly to the upmost local calling function, for example parse_xml_file > could be the right thing to do, because there is no point trying to fix anything. > > The internal storage structure should be something like a set of linked lists, which means > that by using a couple of variables it should be possible to free most (all?) memory allocated > using malloc. The only remaining issue then is how the stack etc. looks. > > ..but now I googled a bit and found that goto was only local and setjmp and longjmp were > the things I was looking for. > > What I'm trying to achieve here is a lazy and simple approach to handling fatal errors like > memory errors. > If there is a legitimate use for setjmp and longjmp, it's for breaking out of error conditions in parsers. But they're not functions which are really in the spirit of C, many maintaining programmers will be surprised to see them used and may mishandle them. The technique is use is the sticky error. You have a error structure which you pass to every function. Initially it's clear. However it never overwrites an error,and usually for a simple parser, it just store the first error. Attempts to set subsequent errors are ignored. You write the parse in such a way that you can attempt to parse erroneous input for quite a few steps without getting the parser into an irrecoverable mess. Obviously it will produce nonsense, but it still steps through (so an input like x + / (2 *3) might parse the sub-expression (2*3), before rejecting the line as a whole as erroneous). You abort and pass back up where it is convenient to check for an error, so the parser eventually bubbles back up to the top of its recursive call stack, with an error flag set. You then go to cleanup and report the error to caller. Normally user can easily pass you a malformed file, and he wants some feedback telling him why the file is malformed. However the system can also run out of memory. xml was mis-specified, and the "barrel of laughs" exploit allows a short xml file to demand excessive memory - you have to handle out of memory. Usually it's best just to treat it as another parse error (you have to be careful that reporting the out of memory error doesn't itself require a memory allocation). it's not really a parse error, but usually it's effectively such, a modern hosted system is very unlikely to genuinely run out of memory on a legitimate small input.
[toc] | [prev] | [next] | [standalone]
| From | "Morten W. Petersen" <morphex@gmail.com> |
|---|---|
| Date | 2015-06-14 06:29 -0700 |
| Message-ID | <0c9c5583-e00a-4e37-94d9-b14c1de34ced@googlegroups.com> |
| In reply to | #63236 |
On Sunday, June 14, 2015 at 1:27:06 PM UTC+2, Malcolm McLean wrote: > On Sunday, June 14, 2015 at 10:14:52 AM UTC+1, Morten W. Petersen wrote: > > What I'm trying to achieve here is a lazy and simple approach to handling fatal errors like > > memory errors. > > > If there is a legitimate use for setjmp and longjmp, it's for breaking out of error conditions > in parsers. But they're not functions which are really in the spirit of C, many maintaining > programmers will be surprised to see them used and may mishandle them. > > The technique is use is the sticky error. You have a error structure which you pass to every > function. Initially it's clear. However it never overwrites an error,and usually for a simple > parser, it just store the first error. Attempts to set subsequent errors are ignored. OK. Well, then I'd have to pass along a reference to an error object on every function call. Which made me wonder how functions are inlined and values are passed. If I inline a function, is it then better to pass all arguments as pointers and work on those? Isn't it simpler and faster to have a global error variable and go-to place for fatal errors? For this parser, the idea is to do as little error correction as possible. If an error pops up, the parser should just stop, with the line number and/or position of the unicode character where it failed and a description of what it expected or rather, what didn't compute. I think that's a good focus for a parser that should be fast and secure, that it doesn't try to do anything else than what comes naturally out of that original focus. -Morten
[toc] | [prev] | [next] | [standalone]
| From | Ben Bacarisse <ben.usenet@bsb.me.uk> |
|---|---|
| Date | 2015-06-14 15:06 +0100 |
| Message-ID | <871the4c6m.fsf@bsb.me.uk> |
| In reply to | #63237 |
"Morten W. Petersen" <morphex@gmail.com> writes: > On Sunday, June 14, 2015 at 1:27:06 PM UTC+2, Malcolm McLean wrote: >> On Sunday, June 14, 2015 at 10:14:52 AM UTC+1, Morten W. Petersen wrote: > >> > What I'm trying to achieve here is a lazy and simple approach to >> > handling fatal errors like >> > memory errors. >> > >> If there is a legitimate use for setjmp and longjmp, it's for >> breaking out of error conditions >> in parsers. But they're not functions which are really in the spirit >> of C, many maintaining >> programmers will be surprised to see them used and may mishandle them. >> >> The technique is use is the sticky error. You have a error structure >> which you pass to every >> function. Initially it's clear. However it never overwrites an >> error,and usually for a simple >> parser, it just store the first error. Attempts to set subsequent >> errors are ignored. > > OK. Well, then I'd have to pass along a reference to an error object > on every function call. Which made me wonder how functions are > inlined and values are passed. If I inline a function, is it then > better to pass all arguments as pointers and work on those? There's no hard and fast rule, though you must pass a pointer to any object which the function might have to modify. Other than that it's going to depend on a lot of factors, one of which is that probably it won't matter for functions that are inlined! But in general, pass large objects as a pointer to a const object (often rather confusingly called a "const pointer"). By the way, *you* don't inline functions. The best you can do is mark a function as "inline". The compiler may decide to inline function calls that are not so marked, and it may decide not to inline some that are. > Isn't it simpler and faster to have a global error variable and go-to > place for fatal errors? Probably simpler and faster, yes, but only simpler initially. Libraries with "global" objects are often restrictive -- think possible future threading and recursive parses[1]. Given that a parse has state (line number, next token, error condition etc.) you are probably best passing a pointer to a "parser state" object in every call. As for faster, most parses will be IO bound, so what you pass around and what you make global is likely to be simple noise in the run time. -- Ben.
[toc] | [prev] | [next] | [standalone]
| From | "Morten W. Petersen" <morphex@gmail.com> |
|---|---|
| Date | 2015-06-14 07:25 -0700 |
| Message-ID | <59d38c7a-ccc4-4389-bd03-d826b27ea308@googlegroups.com> |
| In reply to | #63238 |
On Sunday, June 14, 2015 at 4:06:21 PM UTC+2, Ben Bacarisse wrote: > "Morten W. Petersen" <morphex@gmail.com> writes: > > Isn't it simpler and faster to have a global error variable and go-to > > place for fatal errors? > > Probably simpler and faster, yes, but only simpler initially. Libraries > with "global" objects are often restrictive -- think possible future > threading and recursive parses[1]. Given that a parse has state (line > number, next token, error condition etc.) you are probably best passing > a pointer to a "parser state" object in every call. > > As for faster, most parses will be IO bound, so what you pass around and > what you make global is likely to be simple noise in the run time. Ah yes. Given that I'm going to work with 32-bit Unicode symbols and XML files, adding an option to work on compressed files has been something I've considered. On SSD disks things are blazingly fast already, but on slower disks a light compression might give better overall processing speed. So I was thinking of maybe putting a real 2 GB XML file on a RAM-disk and profile that performance. -Morten
[toc] | [prev] | [next] | [standalone]
| From | Ben Bacarisse <ben.usenet@bsb.me.uk> |
|---|---|
| Date | 2015-06-14 16:59 +0100 |
| Message-ID | <87pp4y2sdv.fsf@bsb.me.uk> |
| In reply to | #63240 |
"Morten W. Petersen" <morphex@gmail.com> writes: > On Sunday, June 14, 2015 at 4:06:21 PM UTC+2, Ben Bacarisse wrote: >> "Morten W. Petersen" <morphex@gmail.com> writes: >> > Isn't it simpler and faster to have a global error variable and go-to >> > place for fatal errors? >> >> Probably simpler and faster, yes, but only simpler initially. Libraries >> with "global" objects are often restrictive -- think possible future >> threading and recursive parses[1]. I realise I forgot to explain this. Parsing is, of course, very often inherently recursive. What I meant was "top-level" re-entry. For example, you've just parsed a string as part of a program in language X, but now you want to parse the string as a program in language X. <snip> >> As for faster, most parses will be IO bound, so what you pass around and >> what you make global is likely to be simple noise in the run time. > > Ah yes. Given that I'm going to work with 32-bit Unicode symbols and > XML files, adding an option to work on compressed files has been > something I've considered. (Not that it matters, but you need no more than 21 bits for Unicode. The magic of the 21 is that it needs no more than 4 bytes in UTF-8.) On a side issue, the code seemed to be reading a 32-bit encoding for the data. That's not wrong just rather odd -- it's a rare encoding and it usually needs "endianness" to be taking into account. > On SSD disks things are blazingly fast already, but on slower disks a > light compression might give better overall processing speed. Yes, I wonder, though, if they are fast enough to make the details of the error handling matter. I suspect not (even now) but I have no data to support that. You could write the parser two ways a let we'd know (at least of one case)! <snip> -- Ben.
[toc] | [prev] | [next] | [standalone]
| From | "Morten W. Petersen" <morphex@gmail.com> |
|---|---|
| Date | 2015-06-14 11:51 -0700 |
| Message-ID | <456652aa-93c5-4a69-8284-a07436b6241d@googlegroups.com> |
| In reply to | #63243 |
On Sunday, June 14, 2015 at 5:59:17 PM UTC+2, Ben Bacarisse wrote: > "Morten W. Petersen" <morphex@gmail.com> writes: > > > On Sunday, June 14, 2015 at 4:06:21 PM UTC+2, Ben Bacarisse wrote: > >> "Morten W. Petersen" <morphex@gmail.com> writes: > >> > Isn't it simpler and faster to have a global error variable and go-to > >> > place for fatal errors? > >> > >> Probably simpler and faster, yes, but only simpler initially. Libraries > >> with "global" objects are often restrictive -- think possible future > >> threading and recursive parses[1]. > > I realise I forgot to explain this. Parsing is, of course, very often > inherently recursive. What I meant was "top-level" re-entry. For > example, you've just parsed a string as part of a program in language X, > but now you want to parse the string as a program in language X. > > <snip> > >> As for faster, most parses will be IO bound, so what you pass around and > >> what you make global is likely to be simple noise in the run time. > > > > Ah yes. Given that I'm going to work with 32-bit Unicode symbols and > > XML files, adding an option to work on compressed files has been > > something I've considered. > > (Not that it matters, but you need no more than 21 bits for Unicode. > The magic of the 21 is that it needs no more than 4 bytes in UTF-8.) > > On a side issue, the code seemed to be reading a 32-bit encoding for the > data. That's not wrong just rather odd -- it's a rare encoding and it > usually needs "endianness" to be taking into account. Well, I thought an XML library that bases itself on the 32-bit encoding which doesn't discriminate against any characters, or rather groups of characters, is a good idea. I was thinking of just creating a custom 24-bit-LE encoding without the empty byte, because as you say, Unicode characters fit in 21 bits. Might do that, but put it on hold to not offend anyone. :) > > On SSD disks things are blazingly fast already, but on slower disks a > > light compression might give better overall processing speed. > > Yes, I wonder, though, if they are fast enough to make the details of > the error handling matter. I suspect not (even now) but I have no data > to support that. You could write the parser two ways a let we'd know > (at least of one case)! Well, squeezing out as much performance as possible is a goal, and will be interesting and fun to do once things are working. I guess a set of macros and define statements could make the code compile as either passing an error message or with jmp and would be a good basis for a comparison. -Morten
[toc] | [prev] | [next] | [standalone]
| From | Richard Damon <Richard@Damon-Family.org> |
|---|---|
| Date | 2015-06-14 16:43 -0400 |
| Message-ID | <1Ilfx.116981$HH2.35252@fx20.iad> |
| In reply to | #63247 |
On 6/14/15 2:51 PM, Morten W. Petersen wrote: > On Sunday, June 14, 2015 at 5:59:17 PM UTC+2, Ben Bacarisse wrote: >> "Morten W. Petersen" <morphex@gmail.com> writes: >> >>> On Sunday, June 14, 2015 at 4:06:21 PM UTC+2, Ben Bacarisse wrote: >>>> "Morten W. Petersen" <morphex@gmail.com> writes: >>>>> Isn't it simpler and faster to have a global error variable and go-to >>>>> place for fatal errors? >>>> >>>> Probably simpler and faster, yes, but only simpler initially. Libraries >>>> with "global" objects are often restrictive -- think possible future >>>> threading and recursive parses[1]. >> >> I realise I forgot to explain this. Parsing is, of course, very often >> inherently recursive. What I meant was "top-level" re-entry. For >> example, you've just parsed a string as part of a program in language X, >> but now you want to parse the string as a program in language X. >> >> <snip> >>>> As for faster, most parses will be IO bound, so what you pass around and >>>> what you make global is likely to be simple noise in the run time. >>> >>> Ah yes. Given that I'm going to work with 32-bit Unicode symbols and >>> XML files, adding an option to work on compressed files has been >>> something I've considered. >> >> (Not that it matters, but you need no more than 21 bits for Unicode. >> The magic of the 21 is that it needs no more than 4 bytes in UTF-8.) >> >> On a side issue, the code seemed to be reading a 32-bit encoding for the >> data. That's not wrong just rather odd -- it's a rare encoding and it >> usually needs "endianness" to be taking into account. > > Well, I thought an XML library that bases itself on the 32-bit > encoding which doesn't discriminate against any characters, or rather > groups of characters, is a good idea. UCS-4 (32-bit Unicode) really isn't designed as a "storage format", as both UTF-8 and UTF-16 can store the same data more compactly, (and NEVER take more space to store a character/string). For most application, UTF-8 is the most compact externally, and has the advantage of not worrying about endianness. UTF-16 can have an advantage when dealing with Asian (in particular the CJK) sets, as these will take 1 UTF-16 encoding unit, but 3 UTF-8 encoding units. The "doesn't descriminate" comment reminds me of a question put forth by a CS professor when I was in school (this was when programs were submitted on cards for batch processing). Given that 60 programs are submitted at 8:00, each will take 1 minute of processor time to run. Which is the better scheduling algorithm: 1) Run each program for a second, and then move to the next, cycling through all the programs, then back to the first and so on. Thus all programs will finish at about 9:00. 2) Run the first program to completion, then the next, then the next. Thus the first program finishes at 8:01, the next at 8:02, then 8:03, ... till the last program finishes at 9:00. Amazingly, many people thought the first was better as it was "fair". UCS-4 can be a good format for internal processing, especially if being able to index by code-points in useful, and UTF-8 and UTF-16 were designed to make it easy to do that conversion if needed. > > I was thinking of just creating a custom 24-bit-LE encoding without > the empty byte, because as you say, Unicode characters fit in 21 bits. > > Might do that, but put it on hold to not offend anyone. :) The big problem with a 24 bit format is that processor just don't deal with 3 byte objects well, and tend to need to treat it as 3 separate 1 byte objects and put them together (operations are optimized for power of 2 sized objects). > >>> On SSD disks things are blazingly fast already, but on slower disks a >>> light compression might give better overall processing speed. >> >> Yes, I wonder, though, if they are fast enough to make the details of >> the error handling matter. I suspect not (even now) but I have no data >> to support that. You could write the parser two ways a let we'd know >> (at least of one case)! > > Well, squeezing out as much performance as possible is a goal, and > will be interesting and fun to do once things are working. > > I guess a set of macros and define statements could make the code > compile as either passing an error message or with jmp and would be a > good basis for a comparison. > > -Morten > My first guess would be that for XML, processing the file as UTF-8 would give the best performance in most cases, you have less data to process, and the grammar for XML really only needs to understand the ascii characters, and everything else can just be considers data/identifiers that just need to be passed through or compared for equality. Unless you need to handle "Unicode normalization" (things like 'a with circle above' being encode as a single character vs a and 'combining circle above' being the same, or needing to detect mal-encoded files, you can likely be Unicode unaware.
[toc] | [prev] | [next] | [standalone]
| From | "Morten W. Petersen" <morphex@gmail.com> |
|---|---|
| Date | 2015-06-14 14:16 -0700 |
| Message-ID | <881964cf-df83-479f-82fb-95e6bdd81187@googlegroups.com> |
| In reply to | #63259 |
On Sunday, June 14, 2015 at 10:43:49 PM UTC+2, Richard Damon wrote: > On 6/14/15 2:51 PM, Morten W. Petersen wrote: [...] > > Well, I thought an XML library that bases itself on the 32-bit > > encoding which doesn't discriminate against any characters, or rather > > groups of characters, is a good idea. > > UCS-4 (32-bit Unicode) really isn't designed as a "storage format", as > both UTF-8 and UTF-16 can store the same data more compactly, (and NEVER > take more space to store a character/string). For most application, > UTF-8 is the most compact externally, and has the advantage of not > worrying about endianness. UTF-16 can have an advantage when dealing > with Asian (in particular the CJK) sets, as these will take 1 UTF-16 > encoding unit, but 3 UTF-8 encoding units. OK. Maybe neutral is a better word. There are compression formats out there that should be free of any restrictions such as patents and so on that can be used.. These should help keep the size smaller than an uncompressed UTF-8 file would be. > The "doesn't descriminate" comment reminds me of a question put forth by > a CS professor when I was in school (this was when programs were > submitted on cards for batch processing). Given that 60 programs are > submitted at 8:00, each will take 1 minute of processor time to run. > Which is the better scheduling algorithm: > > 1) Run each program for a second, and then move to the next, cycling > through all the programs, then back to the first and so on. Thus all > programs will finish at about 9:00. > > 2) Run the first program to completion, then the next, then the next. > Thus the first program finishes at 8:01, the next at 8:02, then 8:03, > ... till the last program finishes at 9:00. > > Amazingly, many people thought the first was better as it was "fair". > > UCS-4 can be a good format for internal processing, especially if being > able to index by code-points in useful, and UTF-8 and UTF-16 were > designed to make it easy to do that conversion if needed. I'm going to run with the 32-bit storage and most likely 32- or 24-bit internal storage format for now, to keep things simple. I might also look at storing characters as 21 bits later, and the right way to do that is using a struct, right? The 4 or 3 byte internal storage will be based on global defines, so it will be possible to compare the two approaches precisely. > > I was thinking of just creating a custom 24-bit-LE encoding without > > the empty byte, because as you say, Unicode characters fit in 21 bits. > > > > Might do that, but put it on hold to not offend anyone. :) > > The big problem with a 24 bit format is that processor just don't deal > with 3 byte objects well, and tend to need to treat it as 3 separate 1 > byte objects and put them together (operations are optimized for power > of 2 sized objects). OK. Hm, well it does give a significant saving in required memory, but maybe other ways of storing the data like dictionaries, word counters and so on is smarter then. Are there any classic ways of reducing the size of many different strings or "tokens" while still being fast for example for searching? There isn't some way of expanding 24-bit values to 32-bit values without a lot of overhead using something like a bitwise operator and remembering the position one is at? > > Well, squeezing out as much performance as possible is a goal, and > > will be interesting and fun to do once things are working. > > > > I guess a set of macros and define statements could make the code > > compile as either passing an error message or with jmp and would be a > > good basis for a comparison. [...] > My first guess would be that for XML, processing the file as UTF-8 would > give the best performance in most cases, you have less data to process, > and the grammar for XML really only needs to understand the ascii > characters, and everything else can just be considers data/identifiers > that just need to be passed through or compared for equality. Unless you > need to handle "Unicode normalization" (things like 'a with circle > above' being encode as a single character vs a and 'combining circle > above' being the same, or needing to detect mal-encoded files, you can > likely be Unicode unaware. Again being neutral is important, and disk storage can be compressed. So it's really a matter of working efficiently with characters from them being read into the internal storage, worked on / used until they are written out again. -Morten
[toc] | [prev] | [next] | [standalone]
| From | Richard Damon <Richard@Damon-Family.org> |
|---|---|
| Date | 2015-06-14 20:35 -0400 |
| Message-ID | <o5pfx.77838$hH6.62666@fx22.iad> |
| In reply to | #63260 |
On 6/14/15 5:16 PM, Morten W. Petersen wrote: > On Sunday, June 14, 2015 at 10:43:49 PM UTC+2, Richard Damon wrote: >> On 6/14/15 2:51 PM, Morten W. Petersen wrote: > [...] >>> Well, I thought an XML library that bases itself on the 32-bit >>> encoding which doesn't discriminate against any characters, or rather >>> groups of characters, is a good idea. >> >> UCS-4 (32-bit Unicode) really isn't designed as a "storage format", as >> both UTF-8 and UTF-16 can store the same data more compactly, (and NEVER >> take more space to store a character/string). For most application, >> UTF-8 is the most compact externally, and has the advantage of not >> worrying about endianness. UTF-16 can have an advantage when dealing >> with Asian (in particular the CJK) sets, as these will take 1 UTF-16 >> encoding unit, but 3 UTF-8 encoding units. > > OK. Maybe neutral is a better word. There are compression formats > out there that should be free of any restrictions such as patents and > so on that can be used.. These should help keep the size smaller than > an uncompressed UTF-8 file would be. Why is this type of "Neutral" desired? It isn't that we have a trade off that one format does A better, while the other does B better, UTF-8 and UTF-16 both do EVERY character at least as good as USC-4. You are basically saying that ASCII can't be more efficient then Romulan, so you are going to add overhead to ASCII to make it just as big as Romulan. (This doesn't help Romulan). If you are going to apply compression, I would suspect that compressed UTF-8 (OR UTF-16) would be on par with compressed UCS-4 (after all, the data has the same amount of information in it). The big advantage of going to UTF-8 would be that the file gets most of the compression, but is still in a format that many tools can directly handle. > >> The "doesn't descriminate" comment reminds me of a question put forth by >> a CS professor when I was in school (this was when programs were >> submitted on cards for batch processing). Given that 60 programs are >> submitted at 8:00, each will take 1 minute of processor time to run. >> Which is the better scheduling algorithm: >> >> 1) Run each program for a second, and then move to the next, cycling >> through all the programs, then back to the first and so on. Thus all >> programs will finish at about 9:00. >> >> 2) Run the first program to completion, then the next, then the next. >> Thus the first program finishes at 8:01, the next at 8:02, then 8:03, >> ... till the last program finishes at 9:00. >> >> Amazingly, many people thought the first was better as it was "fair". >> >> UCS-4 can be a good format for internal processing, especially if being >> able to index by code-points in useful, and UTF-8 and UTF-16 were >> designed to make it easy to do that conversion if needed. > > I'm going to run with the 32-bit storage and most likely 32- or > 24-bit internal storage format for now, to keep things simple. I > might also look at storing characters as 21 bits later, and the right > way to do that is using a struct, right? A struct won't help you make the items not by byte aligned. The only practical way to do that is to use a "bit array" with access functions that extract a given bit string out of an array of bytes with masking and shifting. (Your 24 bit format will basically need the same thing, only you don't need the masking, and all the shifts are multiple of 8. > > The 4 or 3 byte internal storage will be based on global defines, so > it will be possible to compare the two approaches precisely. > For internal storage, if you do need to use UCS-4, I would just use 4 bytes, as the overhead for building the native type from 24 bits is going to be costly. Unless you data file is so big that you risk running out of memory, the savings of 3 byte encoding are going to be lost. If you really are going to be memory limited, you probably want to look at reworking the algorithm to not need all the data in RAM at once. (But see also my discussing that doing the processing in UTF-8) >>> I was thinking of just creating a custom 24-bit-LE encoding without >>> the empty byte, because as you say, Unicode characters fit in 21 bits. >>> >>> Might do that, but put it on hold to not offend anyone. :) >> >> The big problem with a 24 bit format is that processor just don't deal >> with 3 byte objects well, and tend to need to treat it as 3 separate 1 >> byte objects and put them together (operations are optimized for power >> of 2 sized objects). > > OK. Hm, well it does give a significant saving in required memory, > but maybe other ways of storing the data like dictionaries, word > counters and so on is smarter then. Are there any classic ways of > reducing the size of many different strings or "tokens" while still > being fast for example for searching? What method can be done to compress the storage of data depends a LOT on what you are doing with it. Again, UTF-8 will typically give you T least as much of a savings unless you data has a lot of esoteric characters. > > There isn't some way of expanding 24-bit values to 32-bit values > without a lot of overhead using something like a bitwise operator and > remembering the position one is at? > The simple way would be something like: uint8_t c24[some size]; /* Fill c24, each character uses 3 entries */ uint32_t c32 = c24[i] | (c24[i+1] << 8) | (c24[i+2] << 16); Note that this will take at least 3 instructions to fetch the data on an optimal machine, and could easily take 7 instructions (3 load, 2 shifts, 2 ors) on a typical machine (compared to a single instruction for a 32 bit load). >>> Well, squeezing out as much performance as possible is a goal, and >>> will be interesting and fun to do once things are working. >>> >>> I guess a set of macros and define statements could make the code >>> compile as either passing an error message or with jmp and would be a >>> good basis for a comparison. > [...] >> My first guess would be that for XML, processing the file as UTF-8 would >> give the best performance in most cases, you have less data to process, >> and the grammar for XML really only needs to understand the ascii >> characters, and everything else can just be considers data/identifiers >> that just need to be passed through or compared for equality. Unless you >> need to handle "Unicode normalization" (things like 'a with circle >> above' being encode as a single character vs a and 'combining circle >> above' being the same, or needing to detect mal-encoded files, you can >> likely be Unicode unaware. > > Again being neutral is important, and disk storage can be > compressed. So it's really a matter of working efficiently with > characters from them being read into the internal storage, worked on > / used until they are written out again. > > -Morten > Again WHY is being neutral important for files. Also, is the a big reason you can't just work in UTF-8 internally. UCS-4 has it place, and some applications do benefit with using it internally. These would be applications that need to be able to randomly access code points in a buffer (not from memorized location), or know how many code points are in a string. From what you have described so far, I don't see any of these applying. You seem to be wanting to tie your hands on the requirements for some nebulous "Neutrality" issue.
[toc] | [prev] | [next] | [standalone]
| From | Malcolm McLean <malcolm.mclean5@btinternet.com> |
|---|---|
| Date | 2015-06-15 01:19 -0700 |
| Message-ID | <147d4a4c-a956-4774-8d88-917e06edd55a@googlegroups.com> |
| In reply to | #63264 |
On Monday, June 15, 2015 at 1:35:40 AM UTC+1, Richard Damon wrote: > > You seem to be wanting to tie your hands on the requirements for some > nebulous "Neutrality" issue. > It's a political rather than engineering consideration. Some Indians opposed UTF-8 because Hindi characters were given the "minor language" slots with 4 byte codes, which implicitly declares that Hindi isn't such an important language.
[toc] | [prev] | [next] | [standalone]
| From | "Morten W. Petersen" <morphex@gmail.com> |
|---|---|
| Date | 2015-06-15 03:57 -0700 |
| Message-ID | <5c43b2a8-ab0e-458a-9374-a98a6a4ef996@googlegroups.com> |
| In reply to | #63268 |
On Monday, June 15, 2015 at 10:19:52 AM UTC+2, Malcolm McLean wrote: > On Monday, June 15, 2015 at 1:35:40 AM UTC+1, Richard Damon wrote: > > > > You seem to be wanting to tie your hands on the requirements for some > > nebulous "Neutrality" issue. > > > It's a political rather than engineering consideration. > Some Indians opposed UTF-8 because Hindi characters were given the > "minor language" slots with 4 byte codes, which implicitly declares > that Hindi isn't such an important language. Yes, this is an important point. I don't want to encourage any language or character set over others. You could call it political, pragmatical or neutral. :) If you want to put a romantic tint on it, different languages and character sets are part of a rich cultural heritage we and those who come after us can all enjoy and learn from. Some things it's easier to get away from at this point, like certain character value ranges being tested before others while checking for validity, if it's a big deal or not I'll know later. -Morten
[toc] | [prev] | [next] | [standalone]
| From | Bartc <bc@freeuk.com> |
|---|---|
| Date | 2015-06-15 12:26 +0100 |
| Message-ID | <mlmcma$qnm$1@dont-email.me> |
| In reply to | #63271 |
On 15/06/2015 11:57, Morten W. Petersen wrote: > On Monday, June 15, 2015 at 10:19:52 AM UTC+2, Malcolm McLean wrote: >> On Monday, June 15, 2015 at 1:35:40 AM UTC+1, Richard Damon wrote: >>> >>> You seem to be wanting to tie your hands on the requirements for some >>> nebulous "Neutrality" issue. >>> >> It's a political rather than engineering consideration. >> Some Indians opposed UTF-8 because Hindi characters were given the >> "minor language" slots with 4 byte codes, which implicitly declares >> that Hindi isn't such an important language. (It can also mean the opposite, being a richer, bigger language.) > > Yes, this is an important point. I don't want to encourage any language or character set over others. You could call it political, pragmatical or neutral. :) If you want to put a romantic tint on it, different languages and character sets are part of a rich cultural heritage we and those who come after us can all enjoy and learn from. Actually I think it's a ridiculous one! Are people really going to peer inside your source code to see if shorter, more common codepoints are processed more efficiently than more obscure ones? And then take offence if their local dialect takes a few microseconds longer? > Some things it's easier to get away from at this point, like certain character value ranges being tested before others while checking for validity, if it's a big deal or not I'll know later. You will have to take care with sorting. You don't want Australia being seen as having priority over Zimbabwe for example, nor the other way around. -- Bartc
[toc] | [prev] | [next] | [standalone]
| From | "Morten W. Petersen" <morphex@gmail.com> |
|---|---|
| Date | 2015-06-15 04:47 -0700 |
| Message-ID | <40a69f07-2369-46ea-b1fa-f0578392f085@googlegroups.com> |
| In reply to | #63272 |
On Monday, June 15, 2015 at 1:26:23 PM UTC+2, Bart wrote: > On 15/06/2015 11:57, Morten W. Petersen wrote: > > On Monday, June 15, 2015 at 10:19:52 AM UTC+2, Malcolm McLean wrote: > >> On Monday, June 15, 2015 at 1:35:40 AM UTC+1, Richard Damon wrote: > >>> > >>> You seem to be wanting to tie your hands on the requirements for some > >>> nebulous "Neutrality" issue. > >>> > >> It's a political rather than engineering consideration. > >> Some Indians opposed UTF-8 because Hindi characters were given the > >> "minor language" slots with 4 byte codes, which implicitly declares > >> that Hindi isn't such an important language. > > (It can also mean the opposite, being a richer, bigger language.) > > > > > Yes, this is an important point. I don't want to encourage any language or character set over others. You could call it political, pragmatical or neutral. :) If you want to put a romantic tint on it, different languages and character sets are part of a rich cultural heritage we and those who come after us can all enjoy and learn from. > > Actually I think it's a ridiculous one! Are people really going to peer > inside your source code to see if shorter, more common codepoints are > processed more efficiently than more obscure ones? And then take offence > if their local dialect takes a few microseconds longer? Well, say you process terabytes of XML files every day or more, if the amount of data is big enough, how fast the program runs in total will be important. Same if it is a big XML file and the results are needed quickly. It's hard to say which parts of the program will be important with regards to speed, but profiling will reveal this. Again, a neutral standpoint is a good place to start. > > Some things it's easier to get away from at this point, like certain character value ranges being tested before others while checking for validity, if it's a big deal or not I'll know later. > > You will have to take care with sorting. You don't want Australia being > seen as having priority over Zimbabwe for example, nor the other way around. Hehe. Good thing we're not dealing with currencies.. ;) It might be that a macro and some defines could rearrange the order of how characters are evaluated, in the Asian region, Russia and so on, the data processed should be mostly non-ASCII characters. -Morten
[toc] | [prev] | [next] | [standalone]
| From | Ben Bacarisse <ben.usenet@bsb.me.uk> |
|---|---|
| Date | 2015-06-15 15:02 +0100 |
| Message-ID | <877fr51345.fsf@bsb.me.uk> |
| In reply to | #63273 |
"Morten W. Petersen" <morphex@gmail.com> writes: <snip> > Well, say you process terabytes of XML files every day or more, if the > amount of data is big enough, how fast the program runs in total will > be important. Same if it is a big XML file and the results are needed > quickly. > > It's hard to say which parts of the program will be important with > regards to speed, but profiling will reveal this. > > Again, a neutral standpoint is a good place to start. Your standpoint is not neutral -- it favours files encoded with a particular encoding that is very rare "in the wild". By all means measure, but since it is common to find UTF-8 encoded XML files that never need more than single bytes per character, processing four times the required data will almost certainly cost you speed in the vast majority of cases. This is a very general point. A fast implementation of anything is usually faster with respect to some data and slower with respect to other data. To be called simply "fast" you need to know the distribution of likely data. Rather the trying to be neutral, try to be flexible. If your program is well-designed you will be able to switch between encodings relatively simply. That way you can either offer all sorts of input format or, at the very least, test a variety of inputs to see what it best for your expected data. <snip> -- Ben.
[toc] | [prev] | [next] | [standalone]
| From | "Morten W. Petersen" <morphex@gmail.com> |
|---|---|
| Date | 2015-06-15 07:08 -0700 |
| Message-ID | <7c2b0ab0-dc8f-4d4a-8f70-78442555daa8@googlegroups.com> |
| In reply to | #63286 |
On Monday, June 15, 2015 at 4:02:42 PM UTC+2, Ben Bacarisse wrote: > "Morten W. Petersen" <morphex@gmail.com> writes: > <snip> > > Well, say you process terabytes of XML files every day or more, if the > > amount of data is big enough, how fast the program runs in total will > > be important. Same if it is a big XML file and the results are needed > > quickly. > > > > It's hard to say which parts of the program will be important with > > regards to speed, but profiling will reveal this. > > > > Again, a neutral standpoint is a good place to start. [...] > Rather the trying to be neutral, try to be flexible. If your program is > well-designed you will be able to switch between encodings relatively > simply. That way you can either offer all sorts of input format or, at > the very least, test a variety of inputs to see what it best for your > expected data. Yes, I agree with this. Liberal yet strict with regards to what comes in, then storing 32-bit+compression on files by default. I'm going to try that approach.. :) For simplicity though I'm going with 32-bit LE encoding now, and will come back later to re-encoding input. -Morten
[toc] | [prev] | [next] | [standalone]
| From | Bartc <bc@freeuk.com> |
|---|---|
| Date | 2015-06-15 18:56 +0100 |
| Message-ID | <mln3ia$oga$1@dont-email.me> |
| In reply to | #63287 |
On 15/06/2015 15:08, Morten W. Petersen wrote: > On Monday, June 15, 2015 at 4:02:42 PM UTC+2, Ben Bacarisse wrote: >> Rather the trying to be neutral, try to be flexible. If your program is >> well-designed you will be able to switch between encodings relatively >> simply. That way you can either offer all sorts of input format or, at >> the very least, test a variety of inputs to see what it best for your >> expected data. > For simplicity though I'm going with 32-bit LE encoding now, and will come back later to re-encoding input. I think the Python language uses a scheme strings are encoded internally as either 8, 16 or 32-bit characters depending on what the content requires. So ASCII-only needs only 8 bits. This means that most of the time only 8 or 16 bits would be used. But it does mean that you only need one rare character out of a billion to cost you an extra 3GB. The advantage is that it gives you the ability to randomly index individual code-points or characters within a string, compared with UTF8 for example. (Also, you don't have to keep reminding yourself exactly what the difference is between code-points and characters..) -- Bartc
[toc] | [prev] | [next] | [standalone]
| From | "Morten W. Petersen" <morphex@gmail.com> |
|---|---|
| Date | 2015-06-15 20:09 -0700 |
| Message-ID | <d86430b8-df37-4a14-8f5f-d5b28643d840@googlegroups.com> |
| In reply to | #63301 |
On Monday, June 15, 2015 at 7:56:47 PM UTC+2, Bart wrote: [...] > I think the Python language uses a scheme strings are encoded internally > as either 8, 16 or 32-bit characters depending on what the content > requires. So ASCII-only needs only 8 bits. Yep, this document describes how that's implemented: https://www.python.org/dev/peps/pep-0393/ [...] > The advantage is that it gives you the ability to randomly index > individual code-points or characters within a string, compared with UTF8 > for example. (Also, you don't have to keep reminding yourself exactly > what the difference is between code-points and characters..) Well, with some abstractions it would be possible to have any kind of internal representation of the Unicode characters. I'm not sure what you mean when you say randomly index individual characters within a string? -Morten
[toc] | [prev] | [next] | [standalone]
| From | Bartc <bc@freeuk.com> |
|---|---|
| Date | 2015-06-16 09:31 +0100 |
| Message-ID | <mlomrg$b6j$1@dont-email.me> |
| In reply to | #63326 |
On 16/06/2015 04:09, Morten W. Petersen wrote: > On Monday, June 15, 2015 at 7:56:47 PM UTC+2, Bart wrote: > [...] >> I think the Python language uses a scheme strings are encoded internally >> as either 8, 16 or 32-bit characters depending on what the content >> requires. So ASCII-only needs only 8 bits. > > Yep, this document describes how that's implemented: > > https://www.python.org/dev/peps/pep-0393/ > > [...] >> The advantage is that it gives you the ability to randomly index >> individual code-points or characters within a string, compared with UTF8 >> for example. > I'm not sure what you mean when you say randomly index individual characters within a string? If you have a char array A containing an ASCII string for example, then the 417th character will be in A[416] (indexed from 0). If the same array contained UTF8, then you won't know where the 417th character is without scanning the string from the beginning. (And then, might require combining several consecutive elements.) With an int array B containing UCS4, the 417th character is also in B[416]. -- Bartc
[toc] | [prev] | [next] | [standalone]
| From | Richard Heathfield <rjh@cpax.org.uk> |
|---|---|
| Date | 2015-06-16 10:09 +0100 |
| Message-ID | <mlop1b$j33$1@dont-email.me> |
| In reply to | #63326 |
On 16/06/15 04:09, Morten W. Petersen wrote:
<snip>
> I'm not sure what you mean when you say randomly index individual characters within a string?
Given a string:
char mystring[] = "Now is the time to bring a bottle to the party.";
we can obtain any single character we like, using mystring[idx], without
regard for the size of individual characters.
This arbitrary ("random") access to characters works because every
element of the string, every character, occupies the same amount of
memory. We don't have to do any special counting to find out where a
character starts.
If the amount of memory taken up by the character depends on the
character, though, mystring[idx] will only work on /some/ of the
characters. For example, you couldn't loop like this:
for(idx = 0; idx < n; idx++)
{
draw_glyph(mystring[idx]);
}
because for /some/ characters, the characters whose code points are made
up from multiple bytes, that would print the wrong thing.
--
Richard Heathfield
Email: rjh at cpax dot org dot uk
"Usenet is a strange place" - dmr 29 July 1999
Sig line 4 vacant - apply within
[toc] | [prev] | [next] | [standalone]
| From | Malcolm McLean <malcolm.mclean5@btinternet.com> |
|---|---|
| Date | 2015-06-16 02:36 -0700 |
| Message-ID | <14b72114-6ead-4f2f-abc8-f1a6f24c88d1@googlegroups.com> |
| In reply to | #63338 |
On Tuesday, June 16, 2015 at 10:09:21 AM UTC+1, Richard Heathfield wrote:
>
> If the amount of memory taken up by the character depends on the
> character, though, mystring[idx] will only work on /some/ of the
> characters. For example, you couldn't loop like this:
>
> for(idx = 0; idx < n; idx++)
> {
> draw_glyph(mystring[idx]);
> }
>
> because for /some/ characters, the characters whose code points are made
> up from multiple bytes, that would print the wrong thing.
>
Looping through isn't a problem
instead of
for( idx = 0; idx < n; idx++)
you write
for(ptr = start; ptr < end; ptr += utf8_skip(ptr))
the problem occurs when you want to slice from say, position 13 to position
19. You've got to go through from the seat, counting, until you reach the 13th
character, then go through again for four more. It's a lot more fiddly and slower
than just doing an index calculation and copying four bytes. Often you'll also
need dynamic memory because you don't know the size of the buffer you'll
be taking.
However the reality is that most string operations involve starting at the
beginning and working through, just as humans read them sequentially
from the start. So the non-fixed width characteristic of UTF-8 is only a
minor disadvantage.
[toc] | [prev] | [next] | [standalone]
Page 3 of 10 — ← Prev page 1 2 [3] 4 5 … 10 Next page →
Back to top | Article view | comp.lang.c
csiph-web