Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c++ > #79727 > unrolled thread

recovering from std::bad_alloc in std::string reserve

Started byLynn McGuire <lynnmcguire5@gmail.com>
First post2021-05-24 20:46 -0500
Last post2021-06-01 17:35 -0700
Articles 12 on this page of 112 — 21 participants

Back to article view | Back to comp.lang.c++

Contents

  recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-24 20:46 -0500
    Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-24 20:50 -0500
      Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-24 20:52 -0500
        Re: recovering from std::bad_alloc in std::string reserve Paavo Helde <myfirstname@osa.pri.ee> - 2021-05-25 07:47 +0300
          Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-25 12:20 -0500
            Re: recovering from std::bad_alloc in std::string reserve Christian Gollwitzer <auriocus@gmx.de> - 2021-05-26 08:36 +0200
              Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-26 13:28 -0500
        Re: recovering from std::bad_alloc in std::string reserve Bonita Montero <Bonita.Montero@gmail.com> - 2021-05-25 12:56 +0200
          Re: recovering from std::bad_alloc in std::string reserve scott@slp53.sl.home (Scott Lurndal) - 2021-05-25 14:55 +0000
            Re: recovering from std::bad_alloc in std::string reserve Paavo Helde <myfirstname@osa.pri.ee> - 2021-05-25 18:42 +0300
              Re: recovering from std::bad_alloc in std::string reserve MrSpook_Ann1u@d0v_5eh1bqgd.com - 2021-05-25 16:10 +0000
                Re: recovering from std::bad_alloc in std::string reserve Bonita Montero <Bonita.Montero@gmail.com> - 2021-05-25 18:23 +0200
                Re: recovering from std::bad_alloc in std::string reserve Paavo Helde <myfirstname@osa.pri.ee> - 2021-05-25 19:57 +0300
                  Re: recovering from std::bad_alloc in std::string reserve MrSpook_ddZgr4t4@okc9_pd48oig5.info - 2021-05-26 07:15 +0000
                    Re: recovering from std::bad_alloc in std::string reserve Bonita Montero <Bonita.Montero@gmail.com> - 2021-05-26 09:22 +0200
                    Re: recovering from std::bad_alloc in std::string reserve Paavo Helde <myfirstname@osa.pri.ee> - 2021-05-26 10:54 +0300
                      Re: recovering from std::bad_alloc in std::string reserve MrSpook_dw5lvA4g@kak0_42x.edu - 2021-05-26 08:29 +0000
              Re: recovering from std::bad_alloc in std::string reserve Nikolaj Lazic <nlazicBEZ_OVOGA@mudrac.ffzg.hr> - 2021-05-25 16:21 +0000
                Re: recovering from std::bad_alloc in std::string reserve Paavo Helde <myfirstname@osa.pri.ee> - 2021-05-25 20:04 +0300
                  Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-25 12:46 -0500
                    Re: recovering from std::bad_alloc in std::string reserve Paavo Helde <myfirstname@osa.pri.ee> - 2021-05-25 20:58 +0300
                      Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-25 13:48 -0500
                    Re: recovering from std::bad_alloc in std::string reserve Nikolaj Lazic <nlazicBEZ_OVOGA@mudrac.ffzg.hr> - 2021-05-25 18:23 +0000
                      Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-25 13:49 -0500
                        Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-26 21:07 -0500
            Re: recovering from std::bad_alloc in std::string reserve Bonita Montero <Bonita.Montero@gmail.com> - 2021-05-25 18:23 +0200
    Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-24 22:33 -0500
      Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-24 22:35 -0500
        Re: recovering from std::bad_alloc in std::string reserve Juha Nieminen <nospam@thanks.invalid> - 2021-05-25 05:16 +0000
          Re: recovering from std::bad_alloc in std::string reserve Bonita Montero <Bonita.Montero@gmail.com> - 2021-05-25 13:00 +0200
          Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-25 12:49 -0500
    Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-27 14:08 -0500
      Re: recovering from std::bad_alloc in std::string reserve scott@slp53.sl.home (Scott Lurndal) - 2021-05-27 21:59 +0000
        Re: recovering from std::bad_alloc in std::string reserve Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-05-27 15:33 -0700
          Re: recovering from std::bad_alloc in std::string reserve Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-05-27 15:58 -0700
            Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-27 18:17 -0500
              Re: recovering from std::bad_alloc in std::string reserve Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-05-27 16:58 -0700
                Re: recovering from std::bad_alloc in std::string reserve Öö Tiib <ootiib@hot.ee> - 2021-05-27 18:36 -0700
                  Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-27 21:38 -0500
                    Re: recovering from std::bad_alloc in std::string reserve Paavo Helde <myfirstname@osa.pri.ee> - 2021-05-28 08:53 +0300
                    Re: recovering from std::bad_alloc in std::string reserve Bo Persson <bo@bo-persson.se> - 2021-05-28 09:47 +0200
                  Re: recovering from std::bad_alloc in std::string reserve Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-05-27 19:54 -0700
                    Re: recovering from std::bad_alloc in std::string reserve Öö Tiib <ootiib@hot.ee> - 2021-05-27 21:02 -0700
                      Re: recovering from std::bad_alloc in std::string reserve Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-05-28 12:09 -0700
                        Re: recovering from std::bad_alloc in std::string reserve scott@slp53.sl.home (Scott Lurndal) - 2021-05-28 20:26 +0000
                        Re: recovering from std::bad_alloc in std::string reserve Öö Tiib <ootiib@hot.ee> - 2021-05-28 17:10 -0700
                        Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-05-29 11:47 +0200
                          Re: recovering from std::bad_alloc in std::string reserve "daniel...@gmail.com" <danielaparker@gmail.com> - 2021-05-30 15:15 -0700
                            Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-05-31 08:33 +0200
                              Re: recovering from std::bad_alloc in std::string reserve "daniel...@gmail.com" <danielaparker@gmail.com> - 2021-05-31 08:21 -0700
                                Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-05-31 18:13 +0200
                              Re: recovering from std::bad_alloc in std::string reserve Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-05-31 14:43 -0700
                                Re: recovering from std::bad_alloc in std::string reserve Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-05-31 14:44 -0700
                                Re: recovering from std::bad_alloc in std::string reserve "daniel...@gmail.com" <danielaparker@gmail.com> - 2021-05-31 15:20 -0700
                                  Re: recovering from std::bad_alloc in std::string reserve Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-06-01 11:17 -0700
                                    Re: recovering from std::bad_alloc in std::string reserve Manfred <noname@add.invalid> - 2021-06-01 20:25 +0200
                                      Re: recovering from std::bad_alloc in std::string reserve Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-06-01 16:55 -0700
                                    Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-06-02 09:06 +0200
                                      Re: recovering from std::bad_alloc in std::string reserve Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-06-05 17:50 -0700
                                        Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-06-06 11:10 +0200
                                          Re: recovering from std::bad_alloc in std::string reserve Öö Tiib <ootiib@hot.ee> - 2021-06-06 04:07 -0700
                                          Re: recovering from std::bad_alloc in std::string reserve Richard Damon <Richard@Damon-Family.org> - 2021-06-06 07:49 -0400
                                  Re: recovering from std::bad_alloc in std::string reserve Manfred <noname@add.invalid> - 2021-06-01 20:21 +0200
                                    Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-06-02 09:29 +0200
                                      Re: recovering from std::bad_alloc in std::string reserve Manfred <noname@add.invalid> - 2021-06-02 18:43 +0200
                                        Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-06-02 23:32 +0200
                                          Re: recovering from std::bad_alloc in std::string reserve Manfred <noname@add.invalid> - 2021-06-03 00:49 +0200
                                            Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-06-03 09:08 +0200
                                              Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-06-03 10:53 +0200
                                              Re: recovering from std::bad_alloc in std::string reserve Manfred <noname@add.invalid> - 2021-06-03 16:20 +0200
                                              Re: recovering from std::bad_alloc in std::string reserve James Kuyper <jameskuyper@alumni.caltech.edu> - 2021-06-03 11:20 -0400
                                                Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-06-03 19:04 +0200
                                                  Re: recovering from std::bad_alloc in std::string reserve scott@slp53.sl.home (Scott Lurndal) - 2021-06-03 22:24 +0000
                                                    Re: recovering from std::bad_alloc in std::string reserve Öö Tiib <ootiib@hot.ee> - 2021-06-03 16:27 -0700
                                                      Re: recovering from std::bad_alloc in std::string reserve scott@slp53.sl.home (Scott Lurndal) - 2021-06-04 00:36 +0000
                                                        Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-06-04 08:46 +0200
                                                          Re: recovering from std::bad_alloc in std::string reserve Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-06-05 22:02 -0700
                                                            Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-06-06 11:17 +0200
                                                              Re: recovering from std::bad_alloc in std::string reserve Manfred <noname@add.invalid> - 2021-06-06 18:39 +0200
                                                              Re: recovering from std::bad_alloc in std::string reserve Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-06-06 16:33 -0700
                                                                Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-06-07 08:41 +0200
                                                            Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-06-06 17:46 +0200
                                                            Re: recovering from std::bad_alloc in std::string reserve MrSpook_Qfmtjdmi@nc6.com - 2021-06-06 14:11 +0000
                                          Re: recovering from std::bad_alloc in std::string reserve Manfred <noname@add.invalid> - 2021-06-03 16:43 +0200
                                  Re: recovering from std::bad_alloc in std::string reserve Manfred <noname@add.invalid> - 2021-06-01 18:00 +0200
                                    Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-06-01 19:15 +0200
                                  Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-06-01 15:55 +0200
                                  Re: recovering from std::bad_alloc in std::string reserve Bo Persson <bo@bo-persson.se> - 2021-06-01 13:59 +0200
                                    Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-06-01 13:26 -0500
                                Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-06-01 08:29 +0200
                                  Re: recovering from std::bad_alloc in std::string reserve Bo Persson <bo@bo-persson.se> - 2021-06-01 14:05 +0200
                                    Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-06-01 15:58 +0200
                                      Re: recovering from std::bad_alloc in std::string reserve James Kuyper <jameskuyper@alumni.caltech.edu> - 2021-06-01 11:33 -0400
                                        Re: recovering from std::bad_alloc in std::string reserve Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-06-01 10:52 -0700
              Re: recovering from std::bad_alloc in std::string reserve James Kuyper <jameskuyper@alumni.caltech.edu> - 2021-05-27 23:16 -0400
              Re: recovering from std::bad_alloc in std::string reserve scott@slp53.sl.home (Scott Lurndal) - 2021-05-28 14:34 +0000
            Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-27 18:19 -0500
          Re: recovering from std::bad_alloc in std::string reserve Bonita Montero <Bonita.Montero@gmail.com> - 2021-05-28 03:47 +0200
        Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-27 17:45 -0500
          Re: recovering from std::bad_alloc in std::string reserve scott@slp53.sl.home (Scott Lurndal) - 2021-05-28 14:38 +0000
            Re: recovering from std::bad_alloc in std::string reserve Bonita Montero <Bonita.Montero@gmail.com> - 2021-05-28 16:44 +0200
            Re: recovering from std::bad_alloc in std::string reserve Bonita Montero <Bonita.Montero@gmail.com> - 2021-05-28 16:42 +0200
            Re: recovering from std::bad_alloc in std::string reserve Paavo Helde <myfirstname@osa.pri.ee> - 2021-05-28 18:22 +0300
        Re: recovering from std::bad_alloc in std::string reserve Paavo Helde <myfirstname@osa.pri.ee> - 2021-05-28 08:47 +0300
    Re: recovering from std::bad_alloc in std::string reserve Bonita Montero <Bonita.Montero@gmail.com> - 2021-05-26 06:16 +0200
      Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-26 13:30 -0500
        Re: recovering from std::bad_alloc in std::string reserve Bonita Montero <Bonita.Montero@gmail.com> - 2021-05-26 20:37 +0200
          Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-26 14:09 -0500
    Re: recovering from std::bad_alloc in std::string reserve "Alf P. Steinbach" <alf.p.steinbach@gmail.com> - 2021-05-26 18:33 +0200
      Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-26 13:32 -0500
        Re: recovering from std::bad_alloc in std::string reserve Christian Gollwitzer <auriocus@gmx.de> - 2021-05-27 07:50 +0200
    Re: recovering from std::bad_alloc in std::string reserve "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2021-06-01 17:35 -0700

Page 6 of 6 — ← Prev page 1 2 3 4 5 [6]

#79885

FromBonita Montero <Bonita.Montero@gmail.com>
Date2021-05-28 16:44 +0200
Message-ID<s8qvlc$qtq$1@dont-email.me>
In reply to#79884
Am 28.05.2021 um 16:42 schrieb Bonita Montero:
>> If windows did not define an off_t equivelent, then the windows API
>> is insufficient.
> 
> https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-getfilesize 

This is more convenient:
https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-getfilesizeex

> A file-size under Windows isn't a special type but just a 64-bit number.
> And this isn't insufficient.
>

[toc] | [prev] | [next] | [standalone]

#79886

FromBonita Montero <Bonita.Montero@gmail.com>
Date2021-05-28 16:42 +0200
Message-ID<s8qvg1$olg$1@dont-email.me>
In reply to#79884
> If windows did not define an off_t equivelent, then the windows API
> is insufficient.

https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-getfilesize

A file-size under Windows isn't a special type but just a 64-bit number.
And this isn't insufficient.

[toc] | [prev] | [next] | [standalone]

#79892

FromPaavo Helde <myfirstname@osa.pri.ee>
Date2021-05-28 18:22 +0300
Message-ID<s8r1r1$boj$1@dont-email.me>
In reply to#79884
28.05.2021 17:38 Scott Lurndal kirjutas:
> File sizes use the 'off_t' type.
> 
> If windows did not define an off_t equivelent, then the windows API
> is insufficient.

The size of off_t is implementation and macro dependent. So it is not 
quite clear what you mean by "off_t equivalent", and if it would be a 
good idea to have such equivalent.

Windows API aims to use fixed sized types, for better binary 
compatibility. This means that as there exist files over 4GB, the chosen 
fixed size type in the Win32 API for the file sizes and offsets must be 
64-bit. And indeed, this is how the relevant functions in Windows API 
are defined:

DWORD GetFileSize(HANDLE hFile, LPDWORD lpFileSizeHigh);

This returns the file size in two 32-bit pieces. Ugly as hell, but at 
least the result is always 64-bit. There is a newer function with a 
better interface:

BOOL GetFileSizeEx(HANDLE hFile, PLARGE_INTEGER lpFileSize);

Here, LARGE_INTEGER is defined as strictly 64-bit.

One can see from the names that MS has tried to avoid binding them to 
specific bit sizes, but this has failed and nowadays these types are 
documented as having fixed size. As part of that failure, DWORD is 
nowadays usually half a word, not double word.

[toc] | [prev] | [next] | [standalone]

#79844

FromPaavo Helde <myfirstname@osa.pri.ee>
Date2021-05-28 08:47 +0300
Message-ID<s8q04m$tac$1@dont-email.me>
In reply to#79807
28.05.2021 00:59 Scott Lurndal kirjutas:
> Lynn McGuire <lynnmcguire5@gmail.com> writes:
>> On 5/27/2021 12:50 AM, Christian Gollwitzer wrote:
>>> Am 26.05.21 um 20:32 schrieb Lynn McGuire: - Alf
>>>>
>>>> I have already replaced the fell code with _ftelli64.
>>>>
>>>> //  get the size of the output file
>>>> fseek (pOutputFile, 0, SEEK_END);
>>>> __int64 outputFileLength = _ftelli64 (pOutputFile) + 42;  // give it
>>>> some slop
>>>> int outputFileLengthInt = (int) outputFileLength;
>>>
>>> ...and here you restrict it to 2GB again, or worse, retrieve a negative
>>> file size for sizes between 2GB and 4GB.
>>>
>>>
>>> To prepare for a 64bit move, you should replace all size variables with
>>> size_t for unsigned or ptrdiff_t for signed. That will correspond to a
>>> 32bit integer in 32 bit and a 64 bit integer in 64 bit.
>>>
>>>       Christian
>>
>> Done.  With checking against SIZE_MAX before casting the variable to size_t.


> Why?   size_t is guaranteed to hold the size of any object, which implies that
> it must be large enough to accomodate an object the size of the virtual address
> space.   Generally it's minimum size in bits is the same as long.

It's true that in C++ there cannot be any in-memory contiguous object 
with a size larger than what fits in size_t. But that's not relevant 
here, the file is not in the memory, but on the disk.

A file can be 300 GB or whatever; an int64 ought to be able to express 
the size of any file for the foreseeable future. OTOH, size_t in a 
32-bit program can only express up to 4 GB. So it means one indeed needs 
to check the number before casting to size_t.

[toc] | [prev] | [next] | [standalone]

#79958

FromBonita Montero <Bonita.Montero@gmail.com>
Date2021-05-26 06:16 +0200
Message-ID<s8ki3b$1th$1@dont-email.me>
In reply to#79727
Maybe it would be an idea to process your file in pieces ?

[toc] | [prev] | [next] | [standalone]

#80003

FromLynn McGuire <lynnmcguire5@gmail.com>
Date2021-05-26 13:30 -0500
Message-ID<s8m43o$g6f$2@dont-email.me>
In reply to#79958
On 5/25/2021 11:16 PM, Bonita Montero wrote:
> Maybe it would be an idea to process your file in pieces ?

I have thought about that.  Not today.  I am thinking about trying the 
large address space switch though.

Lynn

[toc] | [prev] | [next] | [standalone]

#80006

FromBonita Montero <Bonita.Montero@gmail.com>
Date2021-05-26 20:37 +0200
Message-ID<s8m4hr$jb3$1@dont-email.me>
In reply to#80003
>> Maybe it would be an idea to process your file in pieces ?

> I have thought about that.  Not today. 
> I am thinking about trying the  large address space switch though.

If you're accessing the file lienary consider file-mapping.
File-mapping is slower for random accesses since the pages
have to be mapped on demand, but with linear accesses pre-
fetching of your drive and the operating-system take effect.

[toc] | [prev] | [next] | [standalone]

#80008

FromLynn McGuire <lynnmcguire5@gmail.com>
Date2021-05-26 14:09 -0500
Message-ID<s8m6cp$11o$1@dont-email.me>
In reply to#80006
On 5/26/2021 1:37 PM, Bonita Montero wrote:
>>> Maybe it would be an idea to process your file in pieces ?
> 
>> I have thought about that.  Not today. I am thinking about trying the  
>> large address space switch though.
> 
> If you're accessing the file lienary consider file-mapping.
> File-mapping is slower for random accesses since the pages
> have to be mapped on demand, but with linear accesses pre-
> fetching of your drive and the operating-system take effect.

I store a compressed copy of the output file in our binary file so that 
when the user sends it to us so we get a copy of exactly what happened. 
  It is not a crisis if it is not there.  It is a crisis if the file 
processing / storage causes our program to crash.

Thanks,
Lynn

[toc] | [prev] | [next] | [standalone]

#80000

From"Alf P. Steinbach" <alf.p.steinbach@gmail.com>
Date2021-05-26 18:33 +0200
Message-ID<s8lt9d$tf0$1@dont-email.me>
In reply to#79727
On 2021-05-25 03:46, Lynn McGuire wrote:
> I am getting std::bad_alloc from the following code when I try to 
> reserve a std::string of size 937,180,144:
> 
> std::string filename = getFormsMainOwner () -> getOutputFileName ();
> FILE * pOutputFile = nullptr;
> errno_t err = fopen_s_UTF8 ( & pOutputFile, filename.c_str (), "rt");
> if (err == 0)
> {
>      std::string outputFileBuffer;
>          //  need to preallocate the space in case the output file is a 
> gigabyte or more, PMR 6408
>      fseek (pOutputFile, 0, SEEK_END);
>      size_t outputFileLength = ftell (pOutputFile) + 42;  // give it 
> some slop
>      fseek (pOutputFile, 0, SEEK_SET);
>      outputFileBuffer.reserve (outputFileLength);

[snip]

In the above code `ftell` will fail in Windows if the file is 2GB or 
more, because in Windows, even in 64-bit Windows, the `ftell` return 
type `long` is just 32 bits.

However, the C++ level iostreams can report the file size correctly:


----------------------------------------------------------------------------
#include <stdio.h>      // fopen, fseek, ftell, fclose
#include <stdlib.h>     // EXIT_...

#include <iostream>
#include <fstream>
#include <stdexcept>        // runtime_error
using namespace std;

auto hopefully( const bool e ) -> bool { return e; }
auto fail( const char* s ) -> bool { throw runtime_error( s ); }

struct Is_zero {};
auto operator>>( int x, Is_zero ) -> bool { return x == 0; }

const auto& filename = "large_file";

void c_level_check()
{
     struct C_file
     {
         FILE* handle;
         ~C_file() { if( handle != 0 ) { fclose( handle ); } }
     };

     auto const f = C_file{ fopen( ::filename, "rb" ) };
     hopefully( !!f.handle )
         or fail( "fopen failed" );
     fseek( f.handle, 0, SEEK_END )
         >> Is_zero()
         or fail( "fseek failed, probably rather biggus filus" );
     const long pos = ftell( f.handle );
     hopefully( pos >= 0 )
         or fail( "ftell failed" );
     cout << "`ftell` says the file is " << pos << " byte(s)." << endl;
}

void cpp_level_check()
{
     auto f = ifstream( ::filename, ios::in | ios::binary );
     f.seekg( 0, ios::end );
     const ifstream::pos_type pos = f.tellg();
     hopefully( pos != -1 )
         or fail( "ifstream::tellg failed" );
     cout << "`ifstream::tellg` says the file is " << pos << " bytes." 
<< endl;
}

void cpp_main()
{
     try {
         c_level_check();
     } catch( const exception& x ) {
         cerr << "!" << x.what() << endl;
         cpp_level_check();
     }
}

auto main() -> int
{
     try {
         cpp_main();
         return EXIT_SUCCESS;
     } catch( const exception& x ) {
         cerr << "!" << x.what() << endl;
     }
     return EXIT_FAILURE;
}
-------------------------------------------------------------------------------

When I tested this with `large_file` as a copy of the roughly 4GB 
"Bad.Boys.for.Life.2020.1080p.WEB-DL.DD5.1.H264-FGT.mkv", I got


[c:\root\dev\explore\filesize]
 > b
!ftell failed
`ifstream::tellg` says the file is 4542682554 bytes.


- Alf

[toc] | [prev] | [next] | [standalone]

#80004

FromLynn McGuire <lynnmcguire5@gmail.com>
Date2021-05-26 13:32 -0500
Message-ID<s8m48i$g6f$3@dont-email.me>
In reply to#80000
On 5/26/2021 11:33 AM, Alf P. Steinbach wrote:
> On 2021-05-25 03:46, Lynn McGuire wrote:
>> I am getting std::bad_alloc from the following code when I try to 
>> reserve a std::string of size 937,180,144:
>>
>> std::string filename = getFormsMainOwner () -> getOutputFileName ();
>> FILE * pOutputFile = nullptr;
>> errno_t err = fopen_s_UTF8 ( & pOutputFile, filename.c_str (), "rt");
>> if (err == 0)
>> {
>>      std::string outputFileBuffer;
>>          //  need to preallocate the space in case the output file is 
>> a gigabyte or more, PMR 6408
>>      fseek (pOutputFile, 0, SEEK_END);
>>      size_t outputFileLength = ftell (pOutputFile) + 42;  // give it 
>> some slop
>>      fseek (pOutputFile, 0, SEEK_SET);
>>      outputFileBuffer.reserve (outputFileLength);
> 
> [snip]
> 
> In the above code `ftell` will fail in Windows if the file is 2GB or 
> more, because in Windows, even in 64-bit Windows, the `ftell` return 
> type `long` is just 32 bits.
> 
> However, the C++ level iostreams can report the file size correctly:
> 
> 
> ---------------------------------------------------------------------------- 
> 
> #include <stdio.h>      // fopen, fseek, ftell, fclose
> #include <stdlib.h>     // EXIT_...
> 
> #include <iostream>
> #include <fstream>
> #include <stdexcept>        // runtime_error
> using namespace std;
> 
> auto hopefully( const bool e ) -> bool { return e; }
> auto fail( const char* s ) -> bool { throw runtime_error( s ); }
> 
> struct Is_zero {};
> auto operator>>( int x, Is_zero ) -> bool { return x == 0; }
> 
> const auto& filename = "large_file";
> 
> void c_level_check()
> {
>      struct C_file
>      {
>          FILE* handle;
>          ~C_file() { if( handle != 0 ) { fclose( handle ); } }
>      };
> 
>      auto const f = C_file{ fopen( ::filename, "rb" ) };
>      hopefully( !!f.handle )
>          or fail( "fopen failed" );
>      fseek( f.handle, 0, SEEK_END )
>          >> Is_zero()
>          or fail( "fseek failed, probably rather biggus filus" );
>      const long pos = ftell( f.handle );
>      hopefully( pos >= 0 )
>          or fail( "ftell failed" );
>      cout << "`ftell` says the file is " << pos << " byte(s)." << endl;
> }
> 
> void cpp_level_check()
> {
>      auto f = ifstream( ::filename, ios::in | ios::binary );
>      f.seekg( 0, ios::end );
>      const ifstream::pos_type pos = f.tellg();
>      hopefully( pos != -1 )
>          or fail( "ifstream::tellg failed" );
>      cout << "`ifstream::tellg` says the file is " << pos << " bytes." 
> << endl;
> }
> 
> void cpp_main()
> {
>      try {
>          c_level_check();
>      } catch( const exception& x ) {
>          cerr << "!" << x.what() << endl;
>          cpp_level_check();
>      }
> }
> 
> auto main() -> int
> {
>      try {
>          cpp_main();
>          return EXIT_SUCCESS;
>      } catch( const exception& x ) {
>          cerr << "!" << x.what() << endl;
>      }
>      return EXIT_FAILURE;
> }
> ------------------------------------------------------------------------------- 
> 
> 
> When I tested this with `large_file` as a copy of the roughly 4GB 
> "Bad.Boys.for.Life.2020.1080p.WEB-DL.DD5.1.H264-FGT.mkv", I got
> 
> 
> [c:\root\dev\explore\filesize]
>  > b
> !ftell failed
> `ifstream::tellg` says the file is 4542682554 bytes.
> 
> 
> - Alf

I have already replaced the fell code with _ftelli64.

//  get the size of the output file
fseek (pOutputFile, 0, SEEK_END);
__int64 outputFileLength = _ftelli64 (pOutputFile) + 42;  // give it 
some slop
int outputFileLengthInt = (int) outputFileLength;
fseek (pOutputFile, 0, SEEK_SET);

Thanks,
Lynn

[toc] | [prev] | [next] | [standalone]

#80026

FromChristian Gollwitzer <auriocus@gmx.de>
Date2021-05-27 07:50 +0200
Message-ID<s8nbvd$5ah$1@dont-email.me>
In reply to#80004
Am 26.05.21 um 20:32 schrieb Lynn McGuire: - Alf
> 
> I have already replaced the fell code with _ftelli64.
> 
> //  get the size of the output file
> fseek (pOutputFile, 0, SEEK_END);
> __int64 outputFileLength = _ftelli64 (pOutputFile) + 42;  // give it 
> some slop
> int outputFileLengthInt = (int) outputFileLength;

...and here you restrict it to 2GB again, or worse, retrieve a negative 
file size for sizes between 2GB and 4GB.


To prepare for a 64bit move, you should replace all size variables with 
size_t for unsigned or ptrdiff_t for signed. That will correspond to a 
32bit integer in 32 bit and a 64 bit integer in 64 bit.

	Christian

[toc] | [prev] | [next] | [standalone]

#80061

From"Chris M. Thomasson" <chris.m.thomasson.1@gmail.com>
Date2021-06-01 17:35 -0700
Message-ID<s96jo0$1uma$1@gioia.aioe.org>
In reply to#79727
On 5/24/2021 6:46 PM, Lynn McGuire wrote:
> I am getting std::bad_alloc from the following code when I try to 
> reserve a std::string of size 937,180,144:
> 
> std::string filename = getFormsMainOwner () -> getOutputFileName ();
> FILE * pOutputFile = nullptr;
> errno_t err = fopen_s_UTF8 ( & pOutputFile, filename.c_str (), "rt");
> if (err == 0)
> {
>      std::string outputFileBuffer;
>          //  need to preallocate the space in case the output file is a 
> gigabyte or more, PMR 6408
>      fseek (pOutputFile, 0, SEEK_END);
>      size_t outputFileLength = ftell (pOutputFile) + 42;  // give it 
> some slop
>      fseek (pOutputFile, 0, SEEK_SET);
>      outputFileBuffer.reserve (outputFileLength);
> 
> Any thoughts here on how to handle the std::bad_alloc in std::string 
> reserve ?

Fwiw, way back when I was working with C server code using WinNT and 
IOCP, if a malloc failed, I would put the server into a so-called 
"panic" mode that would trigger the event loops to dump resources. A 
dumped resource would be a connection that has not responded and in the 
timeout detection logic. It would do other things like free buffers in 
connections, ect... When a malloc fails, well, it was not as bad as 
non-paged pool memory crapping out!

The failed malloc would be deferred to try again.

[toc] | [prev] | [standalone]

Page 6 of 6 — ← Prev page 1 2 3 4 5 [6]

Back to top | Article view | comp.lang.c++

csiph-web