Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c++ > #70957 > unrolled thread

Write to file in 8 places simultaneously

Back to article view | Back to comp.lang.c++

Contents

  Write to file in 8 places simultaneously Frederick Gotham <cauldwell.thomas@gmail.com> - 2020-01-02 01:46 -0800
    Re: Write to file in 8 places simultaneously Paavo Helde <myfirstname@osa.pri.ee> - 2020-01-02 12:46 +0200
      Re: Write to file in 8 places simultaneously Frederick Gotham <cauldwell.thomas@gmail.com> - 2020-01-02 03:34 -0800
        Re: Write to file in 8 places simultaneously Jorgen Grahn <grahn+nntp@snipabacken.se> - 2020-01-03 10:57 +0000
    Re: Write to file in 8 places simultaneously Frederick Gotham <cauldwell.thomas@gmail.com> - 2020-01-02 03:32 -0800
    Re: Write to file in 8 places simultaneously Bonita Montero <Bonita.Montero@gmail.com> - 2020-01-03 13:41 +0100

#70957 — Write to file in 8 places simultaneously

First, let me give an example of code I wrote this morning to write to RAM in 8 places simultaneously:

unsigned constexpr g_amount_threads = 8;

#include <cstddef>
using std::size_t;

#include <cstdint>  /* uint8_t, uint64_t */

#include <numeric>  /* iota */

#include <thread>  /* thread */

typedef std::uint8_t Element_Type;

size_t constexpr g_total_bytes = 4294967296ull;  // Four gigabytes

size_t constexpr g_total_elements = g_total_bytes / sizeof(Element_Type);

size_t constexpr g_elements_per_thread = g_total_elements / g_amount_threads;

Element_Type *g_p_mem = nullptr;

void Thread_Entry_Point_Set_Memory(unsigned const thread_number)
{
	Element_Type *const p = g_p_mem + (g_elements_per_thread * thread_number);
	
	Element_Type *const q = p + g_elements_per_thread;
	
	std::iota(p, q, static_cast<Element_Type>(g_elements_per_thread * thread_number));
}

#include <iostream>
using std::cout;
using std::endl;

auto main(void) -> int
{
	g_p_mem = new Element_Type[g_total_elements];
	
	std::thread *ptrs_threads[g_amount_threads];
	
	for (unsigned i = 0; i != g_amount_threads; ++i)
	{
		ptrs_threads[i] = new std::thread(Thread_Entry_Point_Set_Memory,i);
	}
	
	// Now join all the threads
	
	for (auto &p : ptrs_threads)
	{
		p->join();
		
		delete p;
	}

	delete [] g_p_mem;
}


I have measured the differences in execution time with 1 thread, with 2 threads, with 4 threads, etc..

What I want to do now is create a very large file (4 GB) on the hard disk and write to it in 8 different places simultaneously. I know that I can create 8 different files and deal with them independently, but I just want one file.

I think this would be possible under Unix using the POSIX functions open, pwrite, writenv. Has anyone tried it?

[toc] | [next] | [standalone]

#70961

On 2.01.2020 11:46, Frederick Gotham wrote:
>
> First, let me give an example of code I wrote this morning to write to RAM in 8 places simultaneously:
>
> unsigned constexpr g_amount_threads = 8;
>
> #include <cstddef>
> using std::size_t;
>
> #include <cstdint>  /* uint8_t, uint64_t */
>
> #include <numeric>  /* iota */
>
> #include <thread>  /* thread */
>
> typedef std::uint8_t Element_Type;
>
> size_t constexpr g_total_bytes = 4294967296ull;  // Four gigabytes
>
> size_t constexpr g_total_elements = g_total_bytes / sizeof(Element_Type);
>
> size_t constexpr g_elements_per_thread = g_total_elements / g_amount_threads;
>
> Element_Type *g_p_mem = nullptr;
>
> void Thread_Entry_Point_Set_Memory(unsigned const thread_number)
> {
> 	Element_Type *const p = g_p_mem + (g_elements_per_thread * thread_number);
> 	
> 	Element_Type *const q = p + g_elements_per_thread;
> 	
> 	std::iota(p, q, static_cast<Element_Type>(g_elements_per_thread * thread_number));
> }
>
> #include <iostream>
> using std::cout;
> using std::endl;
>
> auto main(void) -> int
> {
> 	g_p_mem = new Element_Type[g_total_elements];
> 	
> 	std::thread *ptrs_threads[g_amount_threads];
> 	
> 	for (unsigned i = 0; i != g_amount_threads; ++i)
> 	{
> 		ptrs_threads[i] = new std::thread(Thread_Entry_Point_Set_Memory,i);
> 	}
> 	
> 	// Now join all the threads
> 	
> 	for (auto &p : ptrs_threads)
> 	{
> 		p->join();
> 		
> 		delete p;
> 	}
>
> 	delete [] g_p_mem;
> }
>
>
> I have measured the differences in execution time with 1 thread, with 2 threads, with 4 threads, etc..

This is basically testing the main memory throughput. For curiosity, 
what where the results?

>
> What I want to do now is create a very large file (4 GB) on the hard disk and write to it in 8 different places simultaneously. I know that I can create 8 different files and deal with them independently, but I just want one file.

And this is testing either main memory throughput or hard disk 
throughput, depending on the amount of free memory in the computer.

>
> I think this would be possible under Unix using the POSIX functions open, pwrite, writenv. Has anyone tried it?

If you aim for max speed and are willing to use non-portable code, you 
should probably consider mmap.

[toc] | [prev] | [next] | [standalone]

#70968

On Thursday, January 2, 2020 at 10:46:53 AM UTC, Paavo Helde wrote:

> This is basically testing the main memory throughput. For curiosity, 
> what where the results?


One thread is 12 seconds.

Eight threads is 1.8 seconds.


> And this is testing either main memory throughput or hard disk 
> throughput, depending on the amount of free memory in the computer.


And depending on where the bottle necks are.


> If you aim for max speed and are willing to use non-portable code, you 
> should probably consider mmap.


I'm gonna play with the POSIX functions so more before I look up mmap.

[toc] | [prev] | [next] | [standalone]

#71009

On Thu, 2020-01-02, Frederick Gotham wrote:
> On Thursday, January 2, 2020 at 10:46:53 AM UTC, Paavo Helde wrote:
...
>> If you aim for max speed and are willing to use non-portable code, you 
>> should probably consider mmap.
>
> I'm gonna play with the POSIX functions so more before I look up mmap.

mmap is a POSIX function, too.

/Jorgen

-- 
  // Jorgen Grahn <grahn@  Oo  o.   .     .
\X/     snipabacken.se>   O  o   .

[toc] | [prev] | [next] | [standalone]

#70966

On Thursday, January 2, 2020 at 9:47:01 AM UTC, Frederick Gotham wrote:

> I think this would be possible under Unix using the POSIX functions open, pwrite, writenv. Has anyone tried it?


Making a few changes to my original code, here's what I have so far for writing to disk. I'm not getting consistent behaviour yet (the output files have a different sha256 sum depending on how many cores I use). Anyway here's what I have so far:

unsigned constexpr g_amount_threads = 8;

char const g_filepath[] = "/home/fgotham/special.txt";

#include <cstddef>
using std::size_t;

#include <cstdint>  /* uint8_t, uint64_t */

#include <numeric>  /* iota */

#include <thread>  /* thread */

#include <stdio.h>  /* fopen, fseek, fputc, fclose */

#include <iostream>
using std::cout;
using std::endl;

/* The next three lines are for POSIX functions: open, pwrite, close */
#include <unistd.h>
#include <sys/stat.h>
#include <fcntl.h>

struct Thread_Local_Stuff {
	std::thread *ptr_thread;
	int fd;
};

Thread_Local_Stuff g_thread_stuffs[g_amount_threads];

typedef std::uint8_t Element_Type;

size_t constexpr g_total_bytes = 4294967296ull;  // Four gigabytes

size_t constexpr g_bytes_per_thread = g_total_bytes / g_amount_threads;

size_t constexpr g_total_elements = g_total_bytes / sizeof(Element_Type);

size_t constexpr g_elements_per_thread = g_total_elements / g_amount_threads;

Element_Type *g_p_mem = nullptr;

void Thread_Entry_Point(unsigned const thread_number)
{
	Element_Type *const p = g_p_mem + (g_elements_per_thread * thread_number);
	
	Element_Type *const q = p + g_elements_per_thread;
	
	std::iota(p, q, static_cast<Element_Type>(g_elements_per_thread * thread_number));
	
	pwrite( g_thread_stuffs[thread_number].fd, p, g_bytes_per_thread, p - g_p_mem );
}

auto main(void) -> int
{
	// First create the very large file
	
	FILE *pfile = fopen(g_filepath, "w");
	fseek(pfile, g_total_bytes - 1u, SEEK_SET);
	fputc('\0', pfile);
	fclose(pfile);
	
	// Now start the threads
	
	g_p_mem = new Element_Type[g_total_elements];
	
	for (unsigned i = 0; i != g_amount_threads; ++i)
	{
		g_thread_stuffs[i].fd = open(g_filepath, O_WRONLY|O_SYNC);
		g_thread_stuffs[i].ptr_thread = new std::thread(Thread_Entry_Point,i);
	}
	
	// Now join all the threads
	
	for (auto &e : g_thread_stuffs)
	{
		e.ptr_thread->join();
		
		delete e.ptr_thread;
		
		close(e.fd);
	}
	
	delete [] g_p_mem;
}

[toc] | [prev] | [next] | [standalone]

#71010

> 	std::thread *ptrs_threads[g_amount_threads];
> 	for (unsigned i = 0; i != g_amount_threads; ++i)
> 	{
> 		ptrs_threads[i] = new std::thread(Thread_Entry_Point_Set_Memory,i);
> 	}

Use
         vector<thread> threads( g_amount_threads ).
         for( thread &t : threads )
             t = move( thread( threadFn, params ) );

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.c++

csiph-web