Groups > comp.lang.c++ > #79727 > unrolled thread

recovering from std::bad_alloc in std::string reserve

Started by	Lynn McGuire <lynnmcguire5@gmail.com>
First post	2021-05-24 20:46 -0500
Last post	2021-06-01 17:35 -0700
Articles	20 on this page of 112 — 21 participants

Back to article view | Back to comp.lang.c++

  recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-24 20:46 -0500
    Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-24 20:50 -0500
      Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-24 20:52 -0500
        Re: recovering from std::bad_alloc in std::string reserve Paavo Helde <myfirstname@osa.pri.ee> - 2021-05-25 07:47 +0300
          Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-25 12:20 -0500
            Re: recovering from std::bad_alloc in std::string reserve Christian Gollwitzer <auriocus@gmx.de> - 2021-05-26 08:36 +0200
              Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-26 13:28 -0500
        Re: recovering from std::bad_alloc in std::string reserve Bonita Montero <Bonita.Montero@gmail.com> - 2021-05-25 12:56 +0200
          Re: recovering from std::bad_alloc in std::string reserve scott@slp53.sl.home (Scott Lurndal) - 2021-05-25 14:55 +0000
            Re: recovering from std::bad_alloc in std::string reserve Paavo Helde <myfirstname@osa.pri.ee> - 2021-05-25 18:42 +0300
              Re: recovering from std::bad_alloc in std::string reserve MrSpook_Ann1u@d0v_5eh1bqgd.com - 2021-05-25 16:10 +0000
                Re: recovering from std::bad_alloc in std::string reserve Bonita Montero <Bonita.Montero@gmail.com> - 2021-05-25 18:23 +0200
                Re: recovering from std::bad_alloc in std::string reserve Paavo Helde <myfirstname@osa.pri.ee> - 2021-05-25 19:57 +0300
                  Re: recovering from std::bad_alloc in std::string reserve MrSpook_ddZgr4t4@okc9_pd48oig5.info - 2021-05-26 07:15 +0000
                    Re: recovering from std::bad_alloc in std::string reserve Bonita Montero <Bonita.Montero@gmail.com> - 2021-05-26 09:22 +0200
                    Re: recovering from std::bad_alloc in std::string reserve Paavo Helde <myfirstname@osa.pri.ee> - 2021-05-26 10:54 +0300
                      Re: recovering from std::bad_alloc in std::string reserve MrSpook_dw5lvA4g@kak0_42x.edu - 2021-05-26 08:29 +0000
              Re: recovering from std::bad_alloc in std::string reserve Nikolaj Lazic <nlazicBEZ_OVOGA@mudrac.ffzg.hr> - 2021-05-25 16:21 +0000
                Re: recovering from std::bad_alloc in std::string reserve Paavo Helde <myfirstname@osa.pri.ee> - 2021-05-25 20:04 +0300
                  Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-25 12:46 -0500
                    Re: recovering from std::bad_alloc in std::string reserve Paavo Helde <myfirstname@osa.pri.ee> - 2021-05-25 20:58 +0300
                      Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-25 13:48 -0500
                    Re: recovering from std::bad_alloc in std::string reserve Nikolaj Lazic <nlazicBEZ_OVOGA@mudrac.ffzg.hr> - 2021-05-25 18:23 +0000
                      Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-25 13:49 -0500
                        Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-26 21:07 -0500
            Re: recovering from std::bad_alloc in std::string reserve Bonita Montero <Bonita.Montero@gmail.com> - 2021-05-25 18:23 +0200
    Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-24 22:33 -0500
      Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-24 22:35 -0500
        Re: recovering from std::bad_alloc in std::string reserve Juha Nieminen <nospam@thanks.invalid> - 2021-05-25 05:16 +0000
          Re: recovering from std::bad_alloc in std::string reserve Bonita Montero <Bonita.Montero@gmail.com> - 2021-05-25 13:00 +0200
          Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-25 12:49 -0500
    Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-27 14:08 -0500
      Re: recovering from std::bad_alloc in std::string reserve scott@slp53.sl.home (Scott Lurndal) - 2021-05-27 21:59 +0000
        Re: recovering from std::bad_alloc in std::string reserve Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-05-27 15:33 -0700
          Re: recovering from std::bad_alloc in std::string reserve Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-05-27 15:58 -0700
            Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-27 18:17 -0500
              Re: recovering from std::bad_alloc in std::string reserve Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-05-27 16:58 -0700
                Re: recovering from std::bad_alloc in std::string reserve Öö Tiib <ootiib@hot.ee> - 2021-05-27 18:36 -0700
                  Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-27 21:38 -0500
                    Re: recovering from std::bad_alloc in std::string reserve Paavo Helde <myfirstname@osa.pri.ee> - 2021-05-28 08:53 +0300
                    Re: recovering from std::bad_alloc in std::string reserve Bo Persson <bo@bo-persson.se> - 2021-05-28 09:47 +0200
                  Re: recovering from std::bad_alloc in std::string reserve Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-05-27 19:54 -0700
                    Re: recovering from std::bad_alloc in std::string reserve Öö Tiib <ootiib@hot.ee> - 2021-05-27 21:02 -0700
                      Re: recovering from std::bad_alloc in std::string reserve Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-05-28 12:09 -0700
                        Re: recovering from std::bad_alloc in std::string reserve scott@slp53.sl.home (Scott Lurndal) - 2021-05-28 20:26 +0000
                        Re: recovering from std::bad_alloc in std::string reserve Öö Tiib <ootiib@hot.ee> - 2021-05-28 17:10 -0700
                        Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-05-29 11:47 +0200
                          Re: recovering from std::bad_alloc in std::string reserve "daniel...@gmail.com" <danielaparker@gmail.com> - 2021-05-30 15:15 -0700
                            Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-05-31 08:33 +0200
                              Re: recovering from std::bad_alloc in std::string reserve "daniel...@gmail.com" <danielaparker@gmail.com> - 2021-05-31 08:21 -0700
                                Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-05-31 18:13 +0200
                              Re: recovering from std::bad_alloc in std::string reserve Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-05-31 14:43 -0700
                                Re: recovering from std::bad_alloc in std::string reserve Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-05-31 14:44 -0700
                                Re: recovering from std::bad_alloc in std::string reserve "daniel...@gmail.com" <danielaparker@gmail.com> - 2021-05-31 15:20 -0700
                                  Re: recovering from std::bad_alloc in std::string reserve Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-06-01 11:17 -0700
                                    Re: recovering from std::bad_alloc in std::string reserve Manfred <noname@add.invalid> - 2021-06-01 20:25 +0200
                                      Re: recovering from std::bad_alloc in std::string reserve Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-06-01 16:55 -0700
                                    Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-06-02 09:06 +0200
                                      Re: recovering from std::bad_alloc in std::string reserve Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-06-05 17:50 -0700
                                        Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-06-06 11:10 +0200
                                          Re: recovering from std::bad_alloc in std::string reserve Öö Tiib <ootiib@hot.ee> - 2021-06-06 04:07 -0700
                                          Re: recovering from std::bad_alloc in std::string reserve Richard Damon <Richard@Damon-Family.org> - 2021-06-06 07:49 -0400
                                  Re: recovering from std::bad_alloc in std::string reserve Manfred <noname@add.invalid> - 2021-06-01 20:21 +0200
                                    Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-06-02 09:29 +0200
                                      Re: recovering from std::bad_alloc in std::string reserve Manfred <noname@add.invalid> - 2021-06-02 18:43 +0200
                                        Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-06-02 23:32 +0200
                                          Re: recovering from std::bad_alloc in std::string reserve Manfred <noname@add.invalid> - 2021-06-03 00:49 +0200
                                            Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-06-03 09:08 +0200
                                              Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-06-03 10:53 +0200
                                              Re: recovering from std::bad_alloc in std::string reserve Manfred <noname@add.invalid> - 2021-06-03 16:20 +0200
                                              Re: recovering from std::bad_alloc in std::string reserve James Kuyper <jameskuyper@alumni.caltech.edu> - 2021-06-03 11:20 -0400
                                                Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-06-03 19:04 +0200
                                                  Re: recovering from std::bad_alloc in std::string reserve scott@slp53.sl.home (Scott Lurndal) - 2021-06-03 22:24 +0000
                                                    Re: recovering from std::bad_alloc in std::string reserve Öö Tiib <ootiib@hot.ee> - 2021-06-03 16:27 -0700
                                                      Re: recovering from std::bad_alloc in std::string reserve scott@slp53.sl.home (Scott Lurndal) - 2021-06-04 00:36 +0000
                                                        Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-06-04 08:46 +0200
                                                          Re: recovering from std::bad_alloc in std::string reserve Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-06-05 22:02 -0700
                                                            Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-06-06 11:17 +0200
                                                              Re: recovering from std::bad_alloc in std::string reserve Manfred <noname@add.invalid> - 2021-06-06 18:39 +0200
                                                              Re: recovering from std::bad_alloc in std::string reserve Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-06-06 16:33 -0700
                                                                Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-06-07 08:41 +0200
                                                            Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-06-06 17:46 +0200
                                                            Re: recovering from std::bad_alloc in std::string reserve MrSpook_Qfmtjdmi@nc6.com - 2021-06-06 14:11 +0000
                                          Re: recovering from std::bad_alloc in std::string reserve Manfred <noname@add.invalid> - 2021-06-03 16:43 +0200
                                  Re: recovering from std::bad_alloc in std::string reserve Manfred <noname@add.invalid> - 2021-06-01 18:00 +0200
                                    Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-06-01 19:15 +0200
                                  Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-06-01 15:55 +0200
                                  Re: recovering from std::bad_alloc in std::string reserve Bo Persson <bo@bo-persson.se> - 2021-06-01 13:59 +0200
                                    Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-06-01 13:26 -0500
                                Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-06-01 08:29 +0200
                                  Re: recovering from std::bad_alloc in std::string reserve Bo Persson <bo@bo-persson.se> - 2021-06-01 14:05 +0200
                                    Re: recovering from std::bad_alloc in std::string reserve David Brown <david.brown@hesbynett.no> - 2021-06-01 15:58 +0200
                                      Re: recovering from std::bad_alloc in std::string reserve James Kuyper <jameskuyper@alumni.caltech.edu> - 2021-06-01 11:33 -0400
                                        Re: recovering from std::bad_alloc in std::string reserve Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-06-01 10:52 -0700
              Re: recovering from std::bad_alloc in std::string reserve James Kuyper <jameskuyper@alumni.caltech.edu> - 2021-05-27 23:16 -0400
              Re: recovering from std::bad_alloc in std::string reserve scott@slp53.sl.home (Scott Lurndal) - 2021-05-28 14:34 +0000
            Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-27 18:19 -0500
          Re: recovering from std::bad_alloc in std::string reserve Bonita Montero <Bonita.Montero@gmail.com> - 2021-05-28 03:47 +0200
        Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-27 17:45 -0500
          Re: recovering from std::bad_alloc in std::string reserve scott@slp53.sl.home (Scott Lurndal) - 2021-05-28 14:38 +0000
            Re: recovering from std::bad_alloc in std::string reserve Bonita Montero <Bonita.Montero@gmail.com> - 2021-05-28 16:44 +0200
            Re: recovering from std::bad_alloc in std::string reserve Bonita Montero <Bonita.Montero@gmail.com> - 2021-05-28 16:42 +0200
            Re: recovering from std::bad_alloc in std::string reserve Paavo Helde <myfirstname@osa.pri.ee> - 2021-05-28 18:22 +0300
        Re: recovering from std::bad_alloc in std::string reserve Paavo Helde <myfirstname@osa.pri.ee> - 2021-05-28 08:47 +0300
    Re: recovering from std::bad_alloc in std::string reserve Bonita Montero <Bonita.Montero@gmail.com> - 2021-05-26 06:16 +0200
      Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-26 13:30 -0500
        Re: recovering from std::bad_alloc in std::string reserve Bonita Montero <Bonita.Montero@gmail.com> - 2021-05-26 20:37 +0200
          Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-26 14:09 -0500
    Re: recovering from std::bad_alloc in std::string reserve "Alf P. Steinbach" <alf.p.steinbach@gmail.com> - 2021-05-26 18:33 +0200
      Re: recovering from std::bad_alloc in std::string reserve Lynn McGuire <lynnmcguire5@gmail.com> - 2021-05-26 13:32 -0500
        Re: recovering from std::bad_alloc in std::string reserve Christian Gollwitzer <auriocus@gmx.de> - 2021-05-27 07:50 +0200
    Re: recovering from std::bad_alloc in std::string reserve "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2021-06-01 17:35 -0700

Page 1 of 6 [1] 2 3 4 5 6 Next page →

#79727 — recovering from std::bad_alloc in std::string reserve

From	Lynn McGuire <lynnmcguire5@gmail.com>
Date	2021-05-24 20:46 -0500
Subject	recovering from std::bad_alloc in std::string reserve
Message-ID	<s8hkt2$liv$1@dont-email.me>

I am getting std::bad_alloc from the following code when I try to 
reserve a std::string of size 937,180,144:

std::string filename = getFormsMainOwner () -> getOutputFileName ();
FILE * pOutputFile = nullptr;
errno_t err = fopen_s_UTF8 ( & pOutputFile, filename.c_str (), "rt");
if (err == 0)
{
	std::string outputFileBuffer;
		//  need to preallocate the space in case the output file is a 
gigabyte or more, PMR 6408
	fseek (pOutputFile, 0, SEEK_END);
	size_t outputFileLength = ftell (pOutputFile) + 42;  // give it some slop
	fseek (pOutputFile, 0, SEEK_SET);
	outputFileBuffer.reserve (outputFileLength);

Any thoughts here on how to handle the std::bad_alloc in std::string 
reserve ?

Thanks,
Lynn

[toc] | [next] | [standalone]

#79729

From	Lynn McGuire <lynnmcguire5@gmail.com>
Date	2021-05-24 20:50 -0500
Message-ID	<s8hl58$mvm$1@dont-email.me>
In reply to	#79727

On 5/24/2021 8:46 PM, Lynn McGuire wrote:
> I am getting std::bad_alloc from the following code when I try to 
> reserve a std::string of size 937,180,144:
> 
> std::string filename = getFormsMainOwner () -> getOutputFileName ();
> FILE * pOutputFile = nullptr;
> errno_t err = fopen_s_UTF8 ( & pOutputFile, filename.c_str (), "rt");
> if (err == 0)
> {
>      std::string outputFileBuffer;
>          //  need to preallocate the space in case the output file is a 
> gigabyte or more, PMR 6408
>      fseek (pOutputFile, 0, SEEK_END);
>      size_t outputFileLength = ftell (pOutputFile) + 42;  // give it 
> some slop
>      fseek (pOutputFile, 0, SEEK_SET);
>      outputFileBuffer.reserve (outputFileLength);
> 
> Any thoughts here on how to handle the std::bad_alloc in std::string 
> reserve ?
> 
> Thanks,
> Lynn

I am using Visual Studio C++ 2015 on a Windows 7 x86 PC with 16 GB of 
ram.  I am probably using a lot of ram already in my program.

Thanks,
Lynn

[toc] | [prev] | [next] | [standalone]

#79730

From	Lynn McGuire <lynnmcguire5@gmail.com>
Date	2021-05-24 20:52 -0500
Message-ID	<s8hl8c$mvm$2@dont-email.me>
In reply to	#79729

On 5/24/2021 8:50 PM, Lynn McGuire wrote:
> On 5/24/2021 8:46 PM, Lynn McGuire wrote:
>> I am getting std::bad_alloc from the following code when I try to 
>> reserve a std::string of size 937,180,144:
>>
>> std::string filename = getFormsMainOwner () -> getOutputFileName ();
>> FILE * pOutputFile = nullptr;
>> errno_t err = fopen_s_UTF8 ( & pOutputFile, filename.c_str (), "rt");
>> if (err == 0)
>> {
>>      std::string outputFileBuffer;
>>          //  need to preallocate the space in case the output file is 
>> a gigabyte or more, PMR 6408
>>      fseek (pOutputFile, 0, SEEK_END);
>>      size_t outputFileLength = ftell (pOutputFile) + 42;  // give it 
>> some slop
>>      fseek (pOutputFile, 0, SEEK_SET);
>>      outputFileBuffer.reserve (outputFileLength);
>>
>> Any thoughts here on how to handle the std::bad_alloc in std::string 
>> reserve ?
>>
>> Thanks,
>> Lynn
> 
> I am using Visual Studio C++ 2015 on a Windows 7 x86 PC with 16 GB of 
> ram.  I am probably using a lot of ram already in my program.
> 
> Thanks,
> Lynn

And I am building a Win32 program.  Not a Win64 program.

Thanks,
Lynn

[toc] | [prev] | [next] | [standalone]

#79746

From	Paavo Helde <myfirstname@osa.pri.ee>
Date	2021-05-25 07:47 +0300
Message-ID	<s8hvgi$uia$1@dont-email.me>
In reply to	#79730

25.05.2021 04:52 Lynn McGuire kirjutas:
> On 5/24/2021 8:50 PM, Lynn McGuire wrote:
>> On 5/24/2021 8:46 PM, Lynn McGuire wrote:
>>> I am getting std::bad_alloc from the following code when I try to 
>>> reserve a std::string of size 937,180,144:
>>>
>>> std::string filename = getFormsMainOwner () -> getOutputFileName ();
>>> FILE * pOutputFile = nullptr;
>>> errno_t err = fopen_s_UTF8 ( & pOutputFile, filename.c_str (), "rt");
>>> if (err == 0)
>>> {
>>>      std::string outputFileBuffer;
>>>          //  need to preallocate the space in case the output file is 
>>> a gigabyte or more, PMR 6408
>>>      fseek (pOutputFile, 0, SEEK_END);
>>>      size_t outputFileLength = ftell (pOutputFile) + 42;  // give it 
>>> some slop
>>>      fseek (pOutputFile, 0, SEEK_SET);
>>>      outputFileBuffer.reserve (outputFileLength);
>>>
>>> Any thoughts here on how to handle the std::bad_alloc in std::string 
>>> reserve ?
>>>
>>> Thanks,
>>> Lynn
>>
>> I am using Visual Studio C++ 2015 on a Windows 7 x86 PC with 16 GB of 
>> ram.  I am probably using a lot of ram already in my program.
>>
>> Thanks,
>> Lynn
> 
> And I am building a Win32 program.  Not a Win64 program.

The usable memory space in a Windows 32-bit program is limited to 2GB. 
It can be increased to 3GB, but this does not buy you much.

Catching std::bad_alloc as you have done in other responses is trivial, 
but now what? Your program still does not work as expected.

If you are dealing with strings in GB range then you really should start 
thinking of switching over to x64 compilation (this will involve some 
64-bit bugfixing if this is your first time). Either that, or you need 
to redesign your code to read and write files in smaller pieces, which 
might be a lot of work.

Also, ftell() returns a signed 32-bit value in Windows so what you have 
written here will cease to work when your files grow larger than 2 GB. 
Suggesting to always use 64-bit alternatives for handling file sizes and 
positions, even in 32-bit programs. Unfortunately these alternatives are 
not portable and one needs to take some extra care for supporting 
different OS-es.

It's also strange to base the program logic on the size of an *output* 
file. But there are probably reasons.

[toc] | [prev] | [next] | [standalone]

#79911

From	Lynn McGuire <lynnmcguire5@gmail.com>
Date	2021-05-25 12:20 -0500
Message-ID	<s8jbkh$i5b$1@dont-email.me>
In reply to	#79746

On 5/24/2021 11:47 PM, Paavo Helde wrote:
> 25.05.2021 04:52 Lynn McGuire kirjutas:
>> On 5/24/2021 8:50 PM, Lynn McGuire wrote:
>>> On 5/24/2021 8:46 PM, Lynn McGuire wrote:
>>>> I am getting std::bad_alloc from the following code when I try to 
>>>> reserve a std::string of size 937,180,144:
>>>>
>>>> std::string filename = getFormsMainOwner () -> getOutputFileName ();
>>>> FILE * pOutputFile = nullptr;
>>>> errno_t err = fopen_s_UTF8 ( & pOutputFile, filename.c_str (), "rt");
>>>> if (err == 0)
>>>> {
>>>>      std::string outputFileBuffer;
>>>>          //  need to preallocate the space in case the output file 
>>>> is a gigabyte or more, PMR 6408
>>>>      fseek (pOutputFile, 0, SEEK_END);
>>>>      size_t outputFileLength = ftell (pOutputFile) + 42;  // give it 
>>>> some slop
>>>>      fseek (pOutputFile, 0, SEEK_SET);
>>>>      outputFileBuffer.reserve (outputFileLength);
>>>>
>>>> Any thoughts here on how to handle the std::bad_alloc in std::string 
>>>> reserve ?
>>>>
>>>> Thanks,
>>>> Lynn
>>>
>>> I am using Visual Studio C++ 2015 on a Windows 7 x86 PC with 16 GB of 
>>> ram.  I am probably using a lot of ram already in my program.
>>>
>>> Thanks,
>>> Lynn
>>
>> And I am building a Win32 program.  Not a Win64 program.
> 
> The usable memory space in a Windows 32-bit program is limited to 2GB. 
> It can be increased to 3GB, but this does not buy you much.
> 
> Catching std::bad_alloc as you have done in other responses is trivial, 
> but now what? Your program still does not work as expected.
> 
> If you are dealing with strings in GB range then you really should start 
> thinking of switching over to x64 compilation (this will involve some 
> 64-bit bugfixing if this is your first time). Either that, or you need 
> to redesign your code to read and write files in smaller pieces, which 
> might be a lot of work.
> 
> Also, ftell() returns a signed 32-bit value in Windows so what you have 
> written here will cease to work when your files grow larger than 2 GB. 
> Suggesting to always use 64-bit alternatives for handling file sizes and 
> positions, even in 32-bit programs. Unfortunately these alternatives are 
> not portable and one needs to take some extra care for supporting 
> different OS-es.
> 
> It's also strange to base the program logic on the size of an *output* 
> file. But there are probably reasons.

Thanks, that is a good idea to move to the 64 bit version of ftell.

My 450,000 lines of C++ program is so tied to the Win32 API that it is 
not funny.  My calculation engine is 850,000 lines of F77 and about 
20,000 lines of C++ but it is still portable to the Unix boxen, probably 
mainframes too if any engineers ran them anymore.

Lynn

[toc] | [prev] | [next] | [standalone]

#79967

From	Christian Gollwitzer <auriocus@gmx.de>
Date	2021-05-26 08:36 +0200
Message-ID	<s8kq8j$9cd$1@dont-email.me>
In reply to	#79911

Am 25.05.21 um 19:20 schrieb Lynn McGuire:
> My 450,000 lines of C++ program is so tied to the Win32 API that it is 
> not funny.  My calculation engine is 850,000 lines of F77 and about 
> 20,000 lines of C++ but it is still portable to the Unix boxen, probably 
> mainframes too if any engineers ran them anymore.

Have you actually tried to recompile in 64bit? The Win32-API is the same 
AFAIUI. Are you casting pointers to ints/longs? I haven't ported 
million-LOC programs to 64bit, but in smaller projects there was 
surprisingly little to do to make it work. I have no idea what goes 
wrong when you link to F77, though.

	Christian

[toc] | [prev] | [next] | [standalone]

#80005

From	Lynn McGuire <lynnmcguire5@gmail.com>
Date	2021-05-26 13:28 -0500
Message-ID	<s8m416$g6f$1@dont-email.me>
In reply to	#79967

On 5/26/2021 1:36 AM, Christian Gollwitzer wrote:
> Am 25.05.21 um 19:20 schrieb Lynn McGuire:
>> My 450,000 lines of C++ program is so tied to the Win32 API that it is 
>> not funny.  My calculation engine is 850,000 lines of F77 and about 
>> 20,000 lines of C++ but it is still portable to the Unix boxen, 
>> probably mainframes too if any engineers ran them anymore.
> 
> Have you actually tried to recompile in 64bit? The Win32-API is the same 
> AFAIUI. Are you casting pointers to ints/longs? I haven't ported 
> million-LOC programs to 64bit, but in smaller projects there was 
> surprisingly little to do to make it work. I have no idea what goes 
> wrong when you link to F77, though.
> 
>      Christian

We have casts all over the place that are killing us now.  One of my 
programmers is currently converting us from ASCII to UNICODE at the 
moment and having all kinds of problems due to the casts.  This program 
originated in 1987 with Windows 2.0 and C coding.  The Win16 to Win32 
port was a freaking disaster and took three of us 18 months to complete. 
  Of course, a portion of our software was Smalltalk which was converted 
to C++ in that port.

The calculation engine still runs as a separate program so the F77 code 
does not matter.

Lynn

[toc] | [prev] | [next] | [standalone]

#79829

From	Bonita Montero <Bonita.Montero@gmail.com>
Date	2021-05-25 12:56 +0200
Message-ID	<s8il57$5mt$3@dont-email.me>
In reply to	#79730

You've got a 2GB address-space and not a contignous piece of
memory which fits to your 900MB.

[toc] | [prev] | [next] | [standalone]

#79869

From	scott@slp53.sl.home (Scott Lurndal)
Date	2021-05-25 14:55 +0000
Message-ID	<Hd8rI.461376$2A5.409452@fx45.iad>
In reply to	#79829

Bonita Montero <Bonita.Montero@gmail.com> writes:
>You've got a 2GB address-space and not a contignous piece of
>memory which fits to your 900MB.

It doesn't need to be contiguous.  But it needs to exist either
as real memory or as configured backing store (i.e. swap space).

[toc] | [prev] | [next] | [standalone]

#79876

From	Paavo Helde <myfirstname@osa.pri.ee>
Date	2021-05-25 18:42 +0300
Message-ID	<s8j5u0$8mn$1@dont-email.me>
In reply to	#79869

25.05.2021 17:55 Scott Lurndal kirjutas:
> Bonita Montero <Bonita.Montero@gmail.com> writes:
>> You've got a 2GB address-space and not a contignous piece of
>> memory which fits to your 900MB.
> 
> It doesn't need to be contiguous.  But it needs to exist either
> as real memory or as configured backing store (i.e. swap space).

For fitting a 900 MB std::string into a memory one needs 900 MB of 
contiguous address space and Bonita is right in pointing out this might 
be problematic because of memory fragmentation.

The OP has 16 GB of RAM so there is plenty of physical memory, it's just 
a problem with the limited address space in 32-bit programs.

[toc] | [prev] | [next] | [standalone]

#79880

From	MrSpook_Ann1u@d0v_5eh1bqgd.com
Date	2021-05-25 16:10 +0000
Message-ID	<s8j7i0$g2a$1@gioia.aioe.org>
In reply to	#79876

On Tue, 25 May 2021 18:42:54 +0300
Paavo Helde <myfirstname@osa.pri.ee> wrote:
>25.05.2021 17:55 Scott Lurndal kirjutas:
>> Bonita Montero <Bonita.Montero@gmail.com> writes:
>>> You've got a 2GB address-space and not a contignous piece of
>>> memory which fits to your 900MB.
>> 
>> It doesn't need to be contiguous.  But it needs to exist either
>> as real memory or as configured backing store (i.e. swap space).
>
>For fitting a 900 MB std::string into a memory one needs 900 MB of 
>contiguous address space and Bonita is right in pointing out this might 

Why? Only the virtual memory address space needs to be contiguous, the
real memory pages storing the string could be all over the place.

[toc] | [prev] | [next] | [standalone]

#79888

From	Bonita Montero <Bonita.Montero@gmail.com>
Date	2021-05-25 18:23 +0200
Message-ID	<s8j8a1$pvk$1@dont-email.me>
In reply to	#79880

> Why? Only the virtual memory address space needs to be contiguous,
> the real memory pages storing the string could be all over the place.

OMG, what a stupid statement.

[toc] | [prev] | [next] | [standalone]

#79900

From	Paavo Helde <myfirstname@osa.pri.ee>
Date	2021-05-25 19:57 +0300
Message-ID	<s8jaab$8tv$1@dont-email.me>
In reply to	#79880

25.05.2021 19:10 MrSpook_Ann1u@d0v_5eh1bqgd.com kirjutas:
> On Tue, 25 May 2021 18:42:54 +0300
> Paavo Helde <myfirstname@osa.pri.ee> wrote:
>> 25.05.2021 17:55 Scott Lurndal kirjutas:
>>> Bonita Montero <Bonita.Montero@gmail.com> writes:
>>>> You've got a 2GB address-space and not a contignous piece of
>>>> memory which fits to your 900MB.
>>>
>>> It doesn't need to be contiguous.  But it needs to exist either
>>> as real memory or as configured backing store (i.e. swap space).
>>
>> For fitting a 900 MB std::string into a memory one needs 900 MB of
>> contiguous address space and Bonita is right in pointing out this might
> 
> Why? Only the virtual memory address space needs to be contiguous, the
> real memory pages storing the string could be all over the place.

Exactly. And if the memory allocator cannot find a free range of 
contiguous 900M addresses, guess what happens.

[toc] | [prev] | [next] | [standalone]

#79968

From	MrSpook_ddZgr4t4@okc9_pd48oig5.info
Date	2021-05-26 07:15 +0000
Message-ID	<s8ksj0$tni$1@gioia.aioe.org>
In reply to	#79900

On Tue, 25 May 2021 19:57:47 +0300
Paavo Helde <myfirstname@osa.pri.ee> wrote:
>25.05.2021 19:10 MrSpook_Ann1u@d0v_5eh1bqgd.com kirjutas:
>> On Tue, 25 May 2021 18:42:54 +0300
>> Paavo Helde <myfirstname@osa.pri.ee> wrote:
>>> 25.05.2021 17:55 Scott Lurndal kirjutas:
>>>> Bonita Montero <Bonita.Montero@gmail.com> writes:
>>>>> You've got a 2GB address-space and not a contignous piece of
>>>>> memory which fits to your 900MB.
>>>>
>>>> It doesn't need to be contiguous.  But it needs to exist either
>>>> as real memory or as configured backing store (i.e. swap space).
>>>
>>> For fitting a 900 MB std::string into a memory one needs 900 MB of
>>> contiguous address space and Bonita is right in pointing out this might
>> 
>> Why? Only the virtual memory address space needs to be contiguous, the
>> real memory pages storing the string could be all over the place.
>
>Exactly. And if the memory allocator cannot find a free range of 
>contiguous 900M addresses, guess what happens.

Things would have to be pretty badly FUBARed for the VM to run out of virtual 
memory address space on a 64 bit system given the 16 exabyte max size!

[toc] | [prev] | [next] | [standalone]

#79969

From	Bonita Montero <Bonita.Montero@gmail.com>
Date	2021-05-26 09:22 +0200
Message-ID	<s8kt0a$no8$1@dont-email.me>
In reply to	#79968

> Things would have to be pretty badly FUBARed for the VM to run out of virtual
> memory address space on a 64 bit system given the 16 exabyte max size!

Actually AMD64 supports "only" 48 bit page-tables where the lower 47
bit fall into user-space. Intel 64 has a recent change to 56 bit page
-tables, but I think that's rather for systems with large file-mappings.

[toc] | [prev] | [next] | [standalone]

#79970

From	Paavo Helde <myfirstname@osa.pri.ee>
Date	2021-05-26 10:54 +0300
Message-ID	<s8kurk$33c$1@dont-email.me>
In reply to	#79968

26.05.2021 10:15 MrSpook_ddZgr4t4@okc9_pd48oig5.info kirjutas:
> On Tue, 25 May 2021 19:57:47 +0300
> Paavo Helde <myfirstname@osa.pri.ee> wrote:
>> 25.05.2021 19:10 MrSpook_Ann1u@d0v_5eh1bqgd.com kirjutas:
>>> On Tue, 25 May 2021 18:42:54 +0300
>>> Paavo Helde <myfirstname@osa.pri.ee> wrote:
>>>> 25.05.2021 17:55 Scott Lurndal kirjutas:
>>>>> Bonita Montero <Bonita.Montero@gmail.com> writes:
>>>>>> You've got a 2GB address-space and not a contignous piece of
>>>>>> memory which fits to your 900MB.
>>>>>
>>>>> It doesn't need to be contiguous.  But it needs to exist either
>>>>> as real memory or as configured backing store (i.e. swap space).
>>>>
>>>> For fitting a 900 MB std::string into a memory one needs 900 MB of
>>>> contiguous address space and Bonita is right in pointing out this might
>>>
>>> Why? Only the virtual memory address space needs to be contiguous, the
>>> real memory pages storing the string could be all over the place.
>>
>> Exactly. And if the memory allocator cannot find a free range of
>> contiguous 900M addresses, guess what happens.
> 
> Things would have to be pretty badly FUBARed for the VM to run out of virtual
> memory address space on a 64 bit system given the 16 exabyte max size!

It looks like you have overlooked the small fact that the OP is having a 
32-bit program and does not want to upgrade to 64-bit at this moment.

[toc] | [prev] | [next] | [standalone]

#79971

From	MrSpook_dw5lvA4g@kak0_42x.edu
Date	2021-05-26 08:29 +0000
Message-ID	<s8l0u6$usd$1@gioia.aioe.org>
In reply to	#79970

On Wed, 26 May 2021 10:54:27 +0300
Paavo Helde <myfirstname@osa.pri.ee> wrote:
>26.05.2021 10:15 MrSpook_ddZgr4t4@okc9_pd48oig5.info kirjutas:
>> Things would have to be pretty badly FUBARed for the VM to run out of virtual
>
>> memory address space on a 64 bit system given the 16 exabyte max size!
>
>It looks like you have overlooked the small fact that the OP is having a 
>32-bit program and does not want to upgrade to 64-bit at this moment.

Fair enough. In which case running out of address space would be pretty
easy given modern application sizes.

[toc] | [prev] | [next] | [standalone]

#79887

From	Nikolaj Lazic <nlazicBEZ_OVOGA@mudrac.ffzg.hr>
Date	2021-05-25 16:21 +0000
Message-ID	<slrnsaq90p.vil.nlazicBEZ_OVOGA@mudrac.ffzg.hr>
In reply to	#79876

Dana Tue, 25 May 2021 18:42:54 +0300, Paavo Helde <myfirstname@osa.pri.ee> napis'o:
> 25.05.2021 17:55 Scott Lurndal kirjutas:
>> Bonita Montero <Bonita.Montero@gmail.com> writes:
>>> You've got a 2GB address-space and not a contignous piece of
>>> memory which fits to your 900MB.
>> 
>> It doesn't need to be contiguous.  But it needs to exist either
>> as real memory or as configured backing store (i.e. swap space).
>
> For fitting a 900 MB std::string into a memory one needs 900 MB of 
> contiguous address space and Bonita is right in pointing out this might 
> be problematic because of memory fragmentation.
>
> The OP has 16 GB of RAM so there is plenty of physical memory, it's just 
> a problem with the limited address space in 32-bit programs.

Win7 32bit cannot access 16G. Limit is 3.5G.
She needs 64bit Windows to use it.

[toc] | [prev] | [next] | [standalone]

#79904

From	Paavo Helde <myfirstname@osa.pri.ee>
Date	2021-05-25 20:04 +0300
Message-ID	<s8jams$dlb$1@dont-email.me>
In reply to	#79887

25.05.2021 19:21 Nikolaj Lazic kirjutas:
> Dana Tue, 25 May 2021 18:42:54 +0300, Paavo Helde <myfirstname@osa.pri.ee> napis'o:
>> 25.05.2021 17:55 Scott Lurndal kirjutas:
>>> Bonita Montero <Bonita.Montero@gmail.com> writes:
>>>> You've got a 2GB address-space and not a contignous piece of
>>>> memory which fits to your 900MB.
>>>
>>> It doesn't need to be contiguous.  But it needs to exist either
>>> as real memory or as configured backing store (i.e. swap space).
>>
>> For fitting a 900 MB std::string into a memory one needs 900 MB of
>> contiguous address space and Bonita is right in pointing out this might
>> be problematic because of memory fragmentation.
>>
>> The OP has 16 GB of RAM so there is plenty of physical memory, it's just
>> a problem with the limited address space in 32-bit programs.
> 
> Win7 32bit cannot access 16G. Limit is 3.5G.
> She needs 64bit Windows to use it.

OP was a bit unclear in this point. But there would be no point to have 
a 32-bit Windows 7 on a machine with 16 GB RAM, so I hope he has got a 
64-bit OS after all.

[toc] | [prev] | [next] | [standalone]

#79915

From	Lynn McGuire <lynnmcguire5@gmail.com>
Date	2021-05-25 12:46 -0500
Message-ID	<s8jd5q$vlv$1@dont-email.me>
In reply to	#79904

On 5/25/2021 12:04 PM, Paavo Helde wrote:
> 25.05.2021 19:21 Nikolaj Lazic kirjutas:
>> Dana Tue, 25 May 2021 18:42:54 +0300, Paavo Helde 
>> <myfirstname@osa.pri.ee> napis'o:
>>> 25.05.2021 17:55 Scott Lurndal kirjutas:
>>>> Bonita Montero <Bonita.Montero@gmail.com> writes:
>>>>> You've got a 2GB address-space and not a contignous piece of
>>>>> memory which fits to your 900MB.
>>>>
>>>> It doesn't need to be contiguous.  But it needs to exist either
>>>> as real memory or as configured backing store (i.e. swap space).
>>>
>>> For fitting a 900 MB std::string into a memory one needs 900 MB of
>>> contiguous address space and Bonita is right in pointing out this might
>>> be problematic because of memory fragmentation.
>>>
>>> The OP has 16 GB of RAM so there is plenty of physical memory, it's just
>>> a problem with the limited address space in 32-bit programs.
>>
>> Win7 32bit cannot access 16G. Limit is 3.5G.
>> She needs 64bit Windows to use it.
> 
> OP was a bit unclear in this point. But there would be no point to have 
> a 32-bit Windows 7 on a machine with 16 GB RAM, so I hope he has got a 
> 64-bit OS after all.

Yes, I have Windows 7 x64 Pro.  I cannot convert to win64 at this time. 
  When we do convert, it will be a steep hill as we started this code in 
Win16 in 1987.  The Win32 port was a very steep hill in 2000.

Lynn

[toc] | [prev] | [next] | [standalone]

Page 1 of 6 [1] 2 3 4 5 6 Next page →

csiph-web

recovering from std::bad_alloc in std::string reserve

Contents

#79727 — recovering from std::bad_alloc in std::string reserve

#79729

#79730

#79746

#79911

#79967

#80005

#79829

#79869

#79876

#79880

#79888

#79900

#79968

#79969

#79970

#79971

#79887

#79904

#79915